diff --git a/.changeset/calm-oracles-check-loss.md b/.changeset/calm-oracles-check-loss.md new file mode 100644 index 0000000000..ec0fcb7153 --- /dev/null +++ b/.changeset/calm-oracles-check-loss.md @@ -0,0 +1,17 @@ +--- +'@tanstack/db': patch +'@tanstack/db-sqlite-persistence-core': patch +'@tanstack/electric-db-collection': patch +--- + +Harden Electric resume and lifecycle handling so partial updates cannot materialize unknown or moved-out rows, stale async work and waiters cannot cross cleanup or restart—including automatic garbage collection—and valid batches behave the same across callback partitions and persistence hydration. + +Preserve hydrated baseline rows during persistence reloads, accept complete-row updates from explicit full-replica resumes, retain committed match evidence until reset, and restart persisted resumes when hydration completion cannot be verified. + +Replace stale cached rows atomically when an invalid resume falls back to a fresh snapshot. Keep subset acquisitions from restoring logically removed rows, and isolate utilities and tag visibility when collection options are reused while preserving compatible same-collection resume state. + +Accept partial updates to complete rows published independently by persistence, while preserving pending removal and reset boundaries. Avoid copying all applied keys at startup or each subset acquisition; presence checks overlay queued writes and buffered messages once per stream callback. Warn once when an older persistence adapter cannot verify hydration for safe resume. + +Keep buffered tag move-outs inside the progressive snapshot's existing transaction so later live updates are not discarded behind an orphaned truncate. + +Keep copied materialized configs and reentrant match callbacks scoped to their owning collection session. Cold tagged or legacy persisted state now recovers with a full snapshot behind cached rows, including in on-demand mode. Keep the reset marker through interrupted recovery and publish the replacement only after the full snapshot completes; known untagged and compatible warm resumes retain their saved offset. diff --git a/.changeset/curly-planets-lead.md b/.changeset/curly-planets-lead.md new file mode 100644 index 0000000000..8f0b2cbeb7 --- /dev/null +++ b/.changeset/curly-planets-lead.md @@ -0,0 +1,6 @@ +--- +'@tanstack/powersync-db-collection': minor +--- + +Add attachments support via `TanStackDBAttachmentQueue`. This extends the PowerSync SDK's `AttachmentQueue` and backs it with +a TanStack DB collection, so attachment metadata and related rows commit atomically. Local files and remote uploads/deletes are managed separately. diff --git a/.changeset/finish-review-recovery.md b/.changeset/finish-review-recovery.md new file mode 100644 index 0000000000..16d050dae5 --- /dev/null +++ b/.changeset/finish-review-recovery.md @@ -0,0 +1,11 @@ +--- +'@tanstack/db': patch +'@tanstack/db-sqlite-persistence-core': patch +'@tanstack/powersync-db-collection': patch +--- + +Preserve native values, arbitrary class references, and draft cycles during mutation detachment; keep transaction persistence receipts settled after publication errors and avoid restoring an acknowledged direct insert over its server row. Keep a delete/reinsert visible when the old synced row has not yet been replaced. + +Retire replaced ordered prefixes without interrupting successful-load bookkeeping if release throws. Retry automatic ordered repair at most twice while retaining stale results and exposing the error; cleanup cancels retries and explicit window retry remains available. + +Keep persisted acquisitions independent, avoid retaining one-shot refreshes as permanent demand, and reject upstream load failures without discarding cached rows. Restore PowerSync readiness only after the recovered baseline also removes rows deleted or moved outside active filters during the tracking outage. diff --git a/.changeset/fix-optimistic-field-reconciliation.md b/.changeset/fix-optimistic-field-reconciliation.md new file mode 100644 index 0000000000..0d96935419 --- /dev/null +++ b/.changeset/fix-optimistic-field-reconciliation.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db': patch +--- + +Preserve whole-row optimistic snapshots through sync and truncate. Fix insert-dependent update settlement, local origin tracking, and rollback publication while sibling requests remain pending. Keep source updates beneath an optimistic live-query delete when queued sync batches apply, without changing sync queue timing. diff --git a/.changeset/fix-trailbase-stream-lifetimes.md b/.changeset/fix-trailbase-stream-lifetimes.md new file mode 100644 index 0000000000..ad4806160b --- /dev/null +++ b/.changeset/fix-trailbase-stream-lifetimes.md @@ -0,0 +1,5 @@ +--- +'@tanstack/trailbase-db-collection': patch +--- + +Fix unhandled rejections and resource leaks when a TrailBase subscription closes, fails, or is cleaned up. Drain buffered events before releasing the reader and prevent a canceled startup from canceling a replacement sync session. diff --git a/.changeset/lucky-donkeys-repeat.md b/.changeset/lucky-donkeys-repeat.md new file mode 100644 index 0000000000..91cf7f7438 --- /dev/null +++ b/.changeset/lucky-donkeys-repeat.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db': patch +--- + +Reclaim collections that start syncing without subscribers, releasing unused live-query subscriptions after a minimum 50ms grace period. Keep pending preloads alive until they settle and refresh retention when preloading ready data; `gcTime: 0` continues to disable automatic GC. Keep detached observer snapshots fresh after empty reloads and allow Node processes to exit while background collection cleanup is pending. diff --git a/.changeset/powersync-attachment-startup-ownership.md b/.changeset/powersync-attachment-startup-ownership.md new file mode 100644 index 0000000000..d30adce3d6 --- /dev/null +++ b/.changeset/powersync-attachment-startup-ownership.md @@ -0,0 +1,5 @@ +--- +'@tanstack/powersync-db-collection': patch +--- + +Load attachment IDs before save/delete in eager and on-demand collections. Preserve existing files when a duplicate save is rejected, reject overlapping saves of the same ID across queues sharing a database, and clean up partial local writes. diff --git a/.changeset/reject-ignored-page-callback.md b/.changeset/reject-ignored-page-callback.md new file mode 100644 index 0000000000..e44a9400bd --- /dev/null +++ b/.changeset/reject-ignored-page-callback.md @@ -0,0 +1,7 @@ +--- +'@tanstack/react-db': minor +'@tanstack/vue-db': minor +'@tanstack/svelte-db': minor +--- + +Remove the ignored `getNextPageParam` option from `useLiveInfiniteQuery` and reject it with a clear error when passed at runtime. Delete this callback from your config; for server pagination, use an on-demand Query Collection whose `queryFn` fulfills `meta.loadSubsetOptions`. Document fixed-server-page loading and clarify that `initialPageParam` labels result pages rather than setting a server cursor. diff --git a/.github/SSR_RELEASE_PLAN.md b/.github/SSR_RELEASE_PLAN.md new file mode 100644 index 0000000000..8df2f4fbbf --- /dev/null +++ b/.github/SSR_RELEASE_PLAN.md @@ -0,0 +1,69 @@ +# TanStack DB SSR Release Plan + +## Release Goal + +Ship TanStack DB SSR as a single coherent story: + +- explicit collection-row hydration and live-query result snapshots through + `DbClient` +- React and Svelte provider and descriptor resolution +- derived live query identity with `queryKey` only when necessary +- backwards-compatible dependency arrays with dev warnings until 1.0 +- a working TanStack Start demo and E2E proof + +## Pre-release Validation + +- Run `pnpm --filter @tanstack/db test`. +- Run `pnpm --filter @tanstack/react-db test`. +- Run `pnpm --filter @tanstack/svelte-db test`. +- Run `pnpm --filter @tanstack/react-router-with-db test` (includes type + tests). +- Run `pnpm --filter @tanstack/query-db-collection test`. +- Run `pnpm --filter @tanstack/db-sqlite-persistence-core test`. +- Run `pnpm --filter @tanstack/db-example-react-start-ssr-e2e test:e2e`. +- Run `pnpm --filter @tanstack/db-example-react-next-ssr-e2e test:e2e`. +- Run `pnpm test:docs`. +- Run `pnpm test:sherif`. +- Run `pnpm build`. + +## Demo + +- Live URL: https://tanstack-db-ssr-demo.netlify.app/ssr-db +- Deploy `examples/react/start-ssr-e2e` to an SSR-capable host. +- Verify the deployed `/ssr-db` route serves SSR HTML with hydrated rows. +- Verify browser hydration succeeds without console/page errors. +- Verify the streamed collection chunk updates the live query. +- Verify `/ssr-db-stream` streams a projected result, omits source-only data, + and hands off to browser sync. +- Run `PLAYWRIGHT_BASE_URL=https://tanstack-db-ssr-demo.netlify.app pnpm --filter @tanstack/db-example-react-start-ssr-e2e test:e2e:hosted`. +- Add the live URL to the PR description and release notes. + +## Docs + +- Publish the [SSR and Hydration guide](../docs/guides/ssr.md). +- Link the guide from overview, quick start, live queries, and React overview. +- Regenerate API reference docs in a dedicated docs-maintenance pass if broad + TypeDoc output churn is acceptable. +- Confirm docs explain when `queryKey` is necessary and when it should be + omitted. +- Confirm docs say dependency arrays warn now and are removed in 1.0. + +## Migration Messaging + +- Lead with: explicit collection preloads transport normalized rows; live-query + preloads transport only their result snapshot. +- Emphasize that existing apps keep working. +- State that `createCollection(...)` remains available, but SSR apps should use + `collectionOptions(...)` plus `DbClient`. +- Explain that React dependency arrays are deprecated with a 1.0 removal path. +- Show `queryKey` only for opaque functional query logic or hot render paths. + +## Announcement Checklist + +- PR description includes high-level summary, migration cheat sheet, and test + commands. +- Release notes include a "No removals in this release" compatibility section. +- Discord announcement links the SSR guide and live demo. +- Example migration diff is available from the Start SSR demo. +- Follow-up issues are filed for the remaining framework adapters and API + reference generation if they are not part of the shipping PR. diff --git a/.github/workflows/e2e-tests.yml b/.github/workflows/e2e-tests.yml index 3678cd19f2..a340502b46 100644 --- a/.github/workflows/e2e-tests.yml +++ b/.github/workflows/e2e-tests.yml @@ -45,6 +45,7 @@ jobs: run: | pnpm --filter @tanstack/db-ivm build pnpm --filter @tanstack/db build + pnpm --filter @tanstack/react-db build pnpm --filter @tanstack/electric-db-collection build pnpm --filter @tanstack/offline-transactions build pnpm --filter @tanstack/query-db-collection build @@ -68,6 +69,21 @@ jobs: env: ELECTRIC_URL: http://localhost:3000 + - name: Install Playwright browsers + run: | + cd examples/react/start-ssr-e2e + pnpm exec playwright install --with-deps chromium + + - name: Run React Start SSR E2E tests + run: | + cd examples/react/start-ssr-e2e + pnpm test:e2e + + - name: Run Next.js SSR E2E tests + run: | + cd examples/react/next-ssr-e2e + pnpm test:e2e + - name: Run Node SQLite persisted collection E2E tests run: | cd packages/node-db-sqlite-persistence diff --git a/.gitignore b/.gitignore index 4ad9ee25d4..3bfb31bb9b 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,8 @@ yarn.lock build coverage dist +playwright-report +test-results # misc .DS_Store @@ -19,6 +21,7 @@ dist .env.test.local .env.production.local .next +next-env.d.ts npm-debug.log* yarn-debug.log* diff --git a/AGENTS.md b/AGENTS.md index a92ff46761..aa7ffc7f1e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,6 +2,17 @@ This guide provides principles and patterns for AI agents contributing to the TanStack DB codebase. These guidelines are derived from PR review patterns and reflect the quality standards expected in this project. +## Required reading: live-query materialization + +Before reading, analyzing, or modifying correlated live-query materialization +code under `packages/db/src/query/live/`, read +`packages/db/src/query/live/ARCHITECTURE.md` in full. Read it before changing +the related includes oracle tests as well. + +Treat that document's component boundaries and normative laws as constraints. +If a change intentionally revises an architectural contract, update the +architecture document in the same pull request. + ## Table of Contents 1. [Type Safety](#type-safety) @@ -351,7 +362,8 @@ const dependentBuilders = [] // Accurately describes dependents ### Always Add Tests for Bugs -**Key Principle:** If you're fixing a bug, add a unit test that reproduces the bug before fixing it. This ensures: +**Key Principle:** Reproduce a bug in a test before fixing it. Prefer extending +an oracle as described below over adding an isolated unit test. This ensures: - The bug is actually fixed - The bug doesn't regress in the future @@ -368,6 +380,60 @@ test('ignores snapshot that resolves after up-to-date message', async () => { }) ``` +### Treat Every Review Bug as a Test Gap + +When a reviewer agent confirms a bug, it must also ask why the existing tests +did not catch it. The finding should name the missing test law, state +transition, generator dimension, adapter boundary, or assertion. If a test or +oracle should already have caught the bug, identify the false-green model, +classifier, fixture, or assertion that let it pass. Use that analysis to suggest +the smallest test or oracle improvement that would catch the same class of bug, +not only the reported example. + +### Keep Oracles Independent + +An oracle is useful only when its expected result comes from a source independent +of the implementation under test. Do not translate production branches, state +machines, classifiers, or helper functions into a second implementation and call +that an oracle. Both copies can encode the same wrong assumption. + +- Derive expected behavior from public contracts, documented prior behavior, + mathematical laws, or a separately specified reference model. +- Keep the reference model structurally different from production. Do not import + the production helper or reuse its classifications to compute expected results. +- Preserve existing contract tests unless a product or design decision explicitly + changes the contract. Rewriting a passing expectation to match new production + behavior is a design review, not routine test maintenance. +- When production work suggests an oracle change, compare the old and new + semantics with counterexamples before editing the oracle. +- Use hostile mutants to prove the oracle rejects plausible wrong designs, + including the mistake production currently makes. A green oracle without a + demonstrated kill is weak evidence. +- Use process grammar to explore lifecycle paths, and design grammar to challenge + the oracle's reference semantics. More generated traces cannot repair a wrong + reference model. + +### Prefer Oracle Coverage Over Isolated Regressions + +An oracle that checks general laws across generated states and histories is a +stronger form of coverage than a unit test for one specific example. Prefer +extending an existing oracle when it can cover the behavior. Add the missing +model rule, generator dimension, state transition, or observable assertion; +adding more pinned examples alone does not generalize the oracle. + +Use a focused regression to isolate and shrink a failure, then keep it as a +replay example for the broader oracle where possible. Verify that the expanded +oracle fails without the fix and passes with it. Keep valuable unit tests, but +do not treat them as a substitute for applicable oracle coverage. If an oracle +is not practical for the behavior, explain why a focused test is sufficient. + +### Name Tests After Behavior + +Test names should state the behavior they prove. Do not put issue or pull +request numbers in test names; those references become stale and make the test +suite harder to read. When an external report contains essential context that +the test cannot express, link it in a nearby comment instead. + ### Test Corner Cases Common corner cases to consider: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000000..cc6f65a9e7 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,146 @@ +# Contributing + +## Questions + +If you have questions about implementation details, help, or support, please use our dedicated community forum at [GitHub Discussions](https://github.com/TanStack/db/discussions). **PLEASE NOTE:** If you choose to open an issue for your question instead, your issue may be closed and redirected to the forum. + +## Reporting issues + +If you have found what you think is a bug, please [file an issue](https://github.com/TanStack/db/issues/new/choose). **PLEASE NOTE:** Issues that are identified as implementation questions or non-issues may be closed and redirected to [GitHub Discussions](https://github.com/TanStack/db/discussions). + +## Suggesting new features + +If you are here to suggest a feature, first create an issue if it does not already exist. From there, we can discuss use cases for the feature and how it could be implemented. + +## Development + +If you have been assigned to fix an issue or develop a new feature, please follow these steps to get started: + +- Fork this repository. +- Use the Node.js version mentioned in `.nvmrc`. + + ```bash + nvm use + ``` + +- Enable [Corepack](https://nodejs.org/api/corepack.html) so the [pnpm](https://pnpm.io/) version mentioned in `package.json` is used. + + ```bash + corepack enable + ``` + +- Install dependencies. + + ```bash + pnpm install + ``` + +- Build all packages. + + ```bash + pnpm build + ``` + +- Run tests. + + ```bash + pnpm test + ``` + +- Run linting. + + ```bash + pnpm lint + ``` + +- Implement your changes and tests in the relevant package or example. +- Document your changes in the appropriate doc page. +- Git stage your required changes and commit them. +- Submit a PR for review. + +### Editing the docs locally and previewing changes + +The documentation for all TanStack projects is hosted on [tanstack.com](https://tanstack.com), which is a TanStack Start application (https://github.com/TanStack/tanstack.com). You need to run this app locally to preview your changes in the `TanStack/db` docs. + +> [!NOTE] +> The website fetches doc pages from GitHub in production, and searches for them at `../db/docs` in development. Your local clone of `TanStack/db` needs to be in the same directory as the local clone of `TanStack/tanstack.com`. + +You can follow these steps to set up the docs for local development: + +1. Make a new directory called `tanstack`. + +```sh +mkdir tanstack +``` + +2. Enter that directory and clone the [`TanStack/db`](https://github.com/TanStack/db) and [`TanStack/tanstack.com`](https://github.com/TanStack/tanstack.com) repos. + +```sh +cd tanstack +git clone git@github.com:TanStack/db.git +# We probably don't need all the branches and commit history +# from the `tanstack.com` repo, so let's just create a shallow +# clone of the latest version of the `main` branch. +# Read more about shallow clones here: +# https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/#user-content-shallow-clones +git clone git@github.com:TanStack/tanstack.com.git --depth=1 --single-branch --branch=main +``` + +> [!NOTE] +> Your `tanstack` directory should look like this: +> +> ```text +> tanstack/ +> | +> +-- db/ (<-- this directory cannot be called anything else!) +> | +> +-- tanstack.com/ +> ``` + +3. Enter the `tanstack/tanstack.com` directory, install the dependencies, and run the app in dev mode. + +```sh +cd tanstack.com +pnpm i +# The app will run on http://localhost:3000 by default +pnpm dev +``` + +4. Visit http://localhost:3000/db/latest/docs/overview in the browser and see the changes you make in `tanstack/db/docs` there. + +> [!WARNING] +> You will need to update `docs/config.json` if you add a new documentation page. + +### Running examples + +- Make sure you've installed dependencies in the repo's root directory. + + ```bash + pnpm install + ``` + +- If you want to run an example against your local changes, run the relevant package build/watch command from the repo root if needed. Otherwise, examples may run against the latest published TanStack DB release. + +- Run the example from the selected example directory. + + ```bash + pnpm dev + ``` + +#### Note on standalone execution + +If you want to run an example without installing dependencies for the whole repo, follow the instructions from the example's README.md file. It will then run against the latest TanStack DB release. + +## Changesets + +This repo uses [Changesets](https://github.com/changesets/changesets) to automate releases. If your PR should release a new package version (patch, minor, or major), please run `pnpm changeset` and commit the generated file. If your PR affects docs, examples, styles, etc., you probably don't need to generate a changeset. + +## Pull requests + +Maintainers merge pull requests by squashing all commits and editing the commit message if necessary using the GitHub user interface. + +Use an appropriate commit type. Be especially careful with breaking changes. + +## Releases + +For each new commit added to `main`, a GitHub Workflow is triggered which runs the [Changesets Action](https://github.com/changesets/action). This generates a preview PR showing the impact of all changesets. When this PR is merged, the package will be published to npm. diff --git a/README.md b/README.md index 01e50bf5c1..7bc46637df 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,19 @@
- + + + + TanStack DB +

diff --git a/docs/collections/electric-collection.md b/docs/collections/electric-collection.md index f5a475bb9e..f363661925 100644 --- a/docs/collections/electric-collection.md +++ b/docs/collections/electric-collection.md @@ -2,8 +2,6 @@ title: Electric Collection --- -# Electric Collection - Electric collections provide seamless integration between TanStack DB and ElectricSQL, enabling real-time data synchronization with your Postgres database through Electric's sync engine. ## Overview @@ -335,6 +333,46 @@ await todosCollection.utils.awaitMatch( ) ``` +### Cleanup and resume safety + +Transaction evidence and pending `awaitTxId`/`awaitMatch` calls belong to one +collection lifecycle. Explicit cleanup and automatic garbage collection reject +pending waits with `StreamAbortedError`; callbacks from the retired stream cannot +settle waits in a restarted collection. Observe these promises even when the +component or collection may be disposed before they settle. + +Persisted resumes wait for the cached row baseline to finish hydrating. If the +persistence wrapper cannot verify hydration completion, Electric starts a fresh +snapshot instead of using the saved offset and handle. It warns once per options +descriptor; update the persistence adapter alongside Electric to enable safe resume. +Fresh eager snapshots wait for hydration, then replace the cached rows at the +snapshot's commit boundary. Rows omitted from that snapshot do not survive in +the collection or its persisted cache, even when the fresh snapshot is empty. + +Tag membership is kept in memory, not restored from cached row headers. A cold +restart therefore fetches a full snapshot when the saved state needs tags, or +comes from an older version that did not record whether tags were used. Untagged +shapes can still resume from their saved offset. Cached rows remain visible until +the replacement snapshot completes; a partial batch or subset completion cannot +publish that replacement early. Interrupting recovery leaves a durable reset +marker so the next start still refetches. This recovery also requests a full shape +snapshot in on-demand mode, at the cost of fetching more than the active subsets. + +An eager or progressive resume cannot apply a partial update to an unknown row. +The adapter rejects that batch, enters an error state, and records a reset so the +next sync starts from a full snapshot. This does not silently retry the failed +stream. Complete updates from an explicit `replica: 'full'` stream remain valid. +On-demand streams can observe updates outside their loaded subsets; unknown +partial rows are ignored, while transaction acknowledgement evidence is retained. +Complete rows published by persistence reloads or another tab are valid baselines +for subsequent partial updates. Pending deletions and resets still take precedence +over an older row that remains publicly visible. + +Reusing Electric collection options, including a spread of those options, does +not share transaction waiters or tag visibility between collections. Tag state +survives a compatible resume of the same collection and clears on a fresh +snapshot or `must-refetch`. + ### Helper Functions The package exports helper functions for use in custom match functions: diff --git a/docs/collections/local-only-collection.md b/docs/collections/local-only-collection.md index 17bf51ef4b..145ba72683 100644 --- a/docs/collections/local-only-collection.md +++ b/docs/collections/local-only-collection.md @@ -2,8 +2,6 @@ title: LocalOnly Collection --- -# LocalOnly Collection - LocalOnly collections are designed for in-memory client data or UI state that doesn't need to persist across browser sessions or sync across tabs. ## Overview @@ -192,10 +190,12 @@ export const modalStateCollection = createCollection( // Use in component function UserProfileModal() { - const { data: modals } = useLiveQuery((q) => - q.from({ modal: modalStateCollection }) - .where(({ modal }) => eq(modal.id, 'user-profile')) - ) + const { data: modals } = useLiveQuery({ + query: (q) => + q + .from({ modal: modalStateCollection }) + .where(({ modal }) => eq(modal.id, 'user-profile')), + }) const modalState = modals[0] @@ -248,10 +248,12 @@ export const formDraftsCollection = createCollection( // Use in component function CreatePostForm() { - const { data: drafts } = useLiveQuery((q) => - q.from({ draft: formDraftsCollection }) - .where(({ draft }) => eq(draft.id, 'new-post')) - ) + const { data: drafts } = useLiveQuery({ + query: (q) => + q + .from({ draft: formDraftsCollection }) + .where(({ draft }) => eq(draft.id, 'new-post')), + }) const currentDraft = drafts[0] diff --git a/docs/collections/local-storage-collection.md b/docs/collections/local-storage-collection.md index 171e5cb9b4..36eecfe3f1 100644 --- a/docs/collections/local-storage-collection.md +++ b/docs/collections/local-storage-collection.md @@ -2,8 +2,6 @@ title: LocalStorage Collection --- -# LocalStorage Collection - LocalStorage collections store small amounts of local-only state that persists across browser sessions and syncs across browser tabs in real-time. ## Overview @@ -263,10 +261,12 @@ export const userPreferencesCollection = createCollection( // Use in component function SettingsPanel() { - const { data: prefs } = useLiveQuery((q) => - q.from({ pref: userPreferencesCollection }) - .where(({ pref }) => eq(pref.id, 'current-user')) - ) + const { data: prefs } = useLiveQuery({ + query: (q) => + q + .from({ pref: userPreferencesCollection }) + .where(({ pref }) => eq(pref.id, 'current-user')), + }) const currentPrefs = prefs[0] diff --git a/docs/collections/powersync-collection.md b/docs/collections/powersync-collection.md index c8ddbabbbe..b2a1ff8529 100644 --- a/docs/collections/powersync-collection.md +++ b/docs/collections/powersync-collection.md @@ -2,8 +2,6 @@ title: PowerSync Collection --- -# PowerSync Collection - PowerSync collections provide seamless integration between TanStack DB and [PowerSync](https://powersync.com), enabling automatic synchronization between your in-memory TanStack DB collections and PowerSync's SQLite database. This gives you offline-ready persistence, real-time sync capabilities, and powerful conflict resolution. ## Overview @@ -1099,4 +1097,177 @@ const liveQuery = createLiveQueryCollection({ completed: todo.completed, })), }) -``` \ No newline at end of file +``` + +## Attachments + +`@tanstack/powersync-db-collection` ships `TanStackDBAttachmentQueue`, an [`AttachmentQueue`](https://docs.powersync.com/usage/use-case-examples/attachments-files) that commits attachment metadata and related collection mutations (for example, setting `lists.photo_id`) in one database transaction. File I/O is separate: a failed save attempts to remove its local file, while the SDK performs remote uploads and deletes later. + +The queue extends PowerSync's `AttachmentQueue`, so the generic concepts are unchanged and documented once in the SDK. + +> This section only covers what is specific to the TanStack DB integration. For storage adapters (local and remote), the `AttachmentTable` schema primitive, error-handling/retry semantics, and the `startSync()` / `stopSync()` lifecycle, see the [PowerSync attachments documentation](https://docs.powersync.com/usage/use-case-examples/attachments-files). + +### Prerequisites + +These are standard PowerSync attachment requirements. See the SDK attachments docs for details. + +- An `AttachmentTable` in your schema: + + ```ts + import { AttachmentTable, Schema } from "@powersync/web" + + const APP_SCHEMA = new Schema({ + // ...your tables + attachments: new AttachmentTable(), + }) + ``` + +- A local storage adapter (such as `IndexDBFileSystemStorageAdapter` on web) and a remote storage adapter (an implementation of the SDK's `RemoteStorageAdapter`, for example backed by Supabase Storage). Both are generic to all attachment users. See the SDK docs for the available adapters and the remote-adapter contract. + +### 1. Create the attachments collection + +This is the piece that makes the integration TanStack-aware: a normal PowerSync collection over the attachments table. The queue reads and writes attachment records through it. + +Both eager and on-demand collections work. Before `save` or `delete` opens its mutation, the queue loads the attachment ID through a temporary live query and retains that query until the transaction is confirmed. It does not call `preload()` inside a mutation function or require loading the entire table. + +An existing ID, or a concurrent save of that ID through the same PowerSync database object, is rejected before writing the file. This is an in-process guard, not a lock across separate database handles, SDK queues, tabs, or processes. File names retain the SDK's ID-based convention so restart can find files after the app's storage directory moves. + +```ts +import { createCollection } from "@tanstack/react-db" +import { powerSyncCollectionOptions } from "@tanstack/powersync-db-collection" + +const attachmentsCollection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.attachments, + }) +) +``` + +### 2. Construct the queue + +Pass your collection as `attachmentsCollection` alongside the standard `AttachmentQueue` options. Only `attachmentsCollection` and `watchAttachments` (below) are specific to this package; `db`, `localStorage`, `remoteStorage`, and `errorHandler` are the usual SDK options. + +```ts +import { TanStackDBAttachmentQueue } from "@tanstack/powersync-db-collection" + +const attachmentQueue = new TanStackDBAttachmentQueue({ + db, + attachmentsCollection, // TanStack DB collection over your AttachmentTable + localStorage, // SDK local storage adapter + remoteStorage, // your RemoteStorageAdapter (see SDK docs) + watchAttachments, // see step 3 + errorHandler, // standard AttachmentQueue error handler (see SDK docs) +}) +``` + +Start and stop syncing with the standard `attachmentQueue.startSync()` / `attachmentQueue.stopSync()` lifecycle (see SDK docs), typically inside a React effect or provider. + +### 3. Tell the queue which attachments exist (`watchAttachments`) + +`watchAttachments` reports the set of attachment IDs your data currently references, so the queue knows what to download and what to archive. With TanStack DB you drive it from a live query: emit the initial state, then re-emit the complete set on every change, and clean up on abort. + +```ts +import { + createCollection, + isNull, + liveQueryCollectionOptions, + not, +} from "@tanstack/db" +import { WatchedAttachmentItem } from "@powersync/web" + +const watchAttachments = async (onUpdate, abortSignal) => { + // Every row in your data model that references an attachment. + const livePhotoIds = createCollection( + liveQueryCollectionOptions({ + query: (q) => + q + .from({ document: listsCollection }) + .where(({ document }) => not(isNull(document.photo_id))) + .select(({ document }) => ({ photo_id: document.photo_id })), + }) + ) + + const mapper = (item) => + ({ + id: item.photo_id, + fileExtension: "jpg", + }) satisfies WatchedAttachmentItem + + // 1. Report the initial set of referenced attachment IDs. + const initialState = await livePhotoIds.stateWhenReady() + onUpdate(Array.from(initialState.values()).map(mapper)) + + // 2. Re-emit the whole set on every change (the queue expects the holistic state). + livePhotoIds.subscribeChanges(() => { + onUpdate(livePhotoIds.map(mapper)) + }) + + // 3. Clean up when sync stops. + abortSignal.addEventListener("abort", () => livePhotoIds.cleanup(), { + once: true, + }) +} +``` + +### 4. Save an attachment atomically with related data + +`save` writes the file, inserts the attachment record into your collection, and runs your `updateHook` mutations in the same transaction. Use the hook to insert or update the row that references the new attachment, so both land together or not at all. + +```ts +await attachmentQueue.save({ + data, // file bytes (ArrayBuffer / base64, per your local adapter) + fileExtension: "jpg", + updateHook: (attachmentRecord) => { + // Runs in the same transaction as the attachment insert. + listsCollection.insert({ + id: crypto.randomUUID(), + name, + created_at: new Date(), + owner_id: userID, + photo_id: attachmentRecord.id, // associate the row with the attachment + }) + }, +}) +``` + +> `updateHook` must be synchronous, it runs inside the transaction's synchronous `mutate()` block and its return value is not awaited, so any mutation after an `await` escapes the transaction. Do asynchronous work before calling `save` or `delete`. + +### 5. Delete an attachment and detach it from the row + +`delete` queues the file for deletion and runs your `updateHook` in the same transaction. Clear the foreign key so the row and the attachment stay consistent. As with `save`, the hook must be synchronous. + +**Upstream limitation:** the SDK version used by this PR can overwrite a queued deletion when an already-running upload succeeds or fails. The related row is detached, but the SDK can lose the remote deletion or retry the obsolete upload. This integration does not work around that SDK completion race. The [attachment oracle notes](../../packages/powersync-db-collection/tests/ATTACHMENT-ORACLE.md) include runnable native-SDK and integration repros; a green ordinary suite does not establish safety for this overlap. + +```ts +await attachmentQueue.delete({ + id: photo_id, + updateHook: () => { + listsCollection.update(listId, (draft) => { + draft.photo_id = null + }) + }, +}) +``` + +### 6. Display attachments via a live-query join + +Join your attachments collection into a live query to read the local URI (the locally cached file path) alongside your domain rows: + +```ts +import { eq } from "@tanstack/db" + +const { data } = useLiveQuery((q) => + q + .from({ lists: listsCollection }) + .leftJoin({ attachment: attachmentsCollection }, ({ lists, attachment }) => + eq(lists.photo_id, attachment.id) + ) + .select(({ lists, attachment }) => ({ + id: lists.id, + name: lists.name, + photo_id: lists.photo_id, + attachment_local_uri: attachment?.local_uri, + })) +) +``` diff --git a/docs/collections/query-collection.md b/docs/collections/query-collection.md index 73f3511edc..25b5f565de 100644 --- a/docs/collections/query-collection.md +++ b/docs/collections/query-collection.md @@ -2,8 +2,6 @@ title: Query Collection --- -# Query Collection - Query collections provide seamless integration between TanStack DB and TanStack Query, enabling automatic synchronization between your local database and remote data sources. ## Overview @@ -25,22 +23,26 @@ npm install @tanstack/query-db-collection @tanstack/query-core @tanstack/db ```typescript import { QueryClient } from "@tanstack/query-core" -import { createCollection } from "@tanstack/db" +import { DbClient, collectionOptions } from "@tanstack/db" import { queryCollectionOptions } from "@tanstack/query-db-collection" const queryClient = new QueryClient() +const db = new DbClient({ queryClient }) -const todosCollection = createCollection( +const todosCollection = collectionOptions("todos", (client) => queryCollectionOptions({ + id: "todos", queryKey: ["todos"], queryFn: async () => { const response = await fetch("/api/todos") return response.json() }, - queryClient, + queryClient: client.requireDependency("queryClient"), getKey: (item) => item.id, }) ) + +const todos = db.collection(todosCollection) ``` ## Configuration Options @@ -54,27 +56,225 @@ The `queryCollectionOptions` function accepts the following options: - `queryClient`: TanStack Query client instance - `getKey`: Function to extract the unique key from an item +### Request-scoped QueryClient + +`queryCollectionOptions` needs a `queryClient`. In SSR, TanStack Start, tests, +or multi-tenant apps, that client is request-local rather than module-global. +Put it on `DbClient`, then resolve it inside the collection descriptor factory: + +```typescript +import { QueryClient } from "@tanstack/query-core" +import { DbClient, collectionOptions } from "@tanstack/db" +import { queryCollectionOptions } from "@tanstack/query-db-collection" + +interface Todo { + id: string + title: string +} + +export const todoCollection = collectionOptions("todos", (client) => + queryCollectionOptions({ + id: "todos", + queryKey: ["todos"], + queryFn: async () => { + const response = await fetch("/api/todos") + return response.json() as Promise> + }, + queryClient: client.requireDependency("queryClient"), + getKey: (todo) => todo.id, + }) +) + +export function createRequestClients() { + const queryClient = new QueryClient() + const dbClient = new DbClient({ queryClient }) + return { queryClient, dbClient } +} +``` + +`dbClient.collection(todoCollection)` memoizes one collection instance for that +descriptor and client. A second `DbClient` materializes fresh adapter state and +uses its own `QueryClient`. + +Passing `queryClient` directly to `queryCollectionOptions` remains supported for +`createCollection(...)` and existing apps. When a descriptor is materialized, +an explicit `DbClient` dependency takes precedence; the configured +`queryClient` is the backwards-compatible fallback. + +### Business-Scoped Collection Factories + +A tenant, project, account, or route parameter can define a **business scope**: +the server resource that a collection represents. Include the scope in the +descriptor id, Query key, and `queryFn`. This extends the +[request-scoped QueryClient pattern](#request-scoped-queryclient) with an +explicit scope parameter: + +```typescript +interface Todo { + id: string + title: string + projectId: string +} + +async function fetchProjectTodos(projectId: string): Promise> { + const response = await fetch(`/api/projects/${projectId}/todos`) + return response.json() +} + +function createProjectTodosDescriptor( + projectId: string, +) { + return collectionOptions(`project:${projectId}:todos`, (client) => + queryCollectionOptions({ + id: `project:${projectId}:todos`, + queryKey: ["projects", projectId, "todos"], + queryFn: () => fetchProjectTodos(projectId), + queryClient: client.requireDependency("queryClient"), + getKey: (todo) => todo.id, + }) + ) +} +``` + +The scope is part of the descriptor identity. `DbClient` resolves separately +created descriptors with the same id to the same collection, so a React hook +can create the descriptor from its current parameters: + +```typescript +export function useProjectTodos(projectId: string) { + return useDbClient().collection(createProjectTodosDescriptor(projectId)) +} +``` + +Only the first descriptor for an id is materialized. Include every scope value +that changes the collection in both its descriptor id and Query key. Call +`await dbClient.cleanup()` when the client scope ends. + +A business scope is separate from a **relational subset** requested by a live query. With `syncMode: "on-demand"`, `LoadSubsetOptions` describes predicates, ordering, limits, and offsets within one business-scoped collection. These options reach `queryFn` through `ctx.meta.loadSubsetOptions` and determine the subset Query keys. See [QueryFn and Predicate Push-Down](#queryfn-and-predicate-push-down). + +Do not create a collection for each `where`, `orderBy`, or `limit`. Reuse the business-scoped collection and let on-demand loading represent those subsets. Create separate collections only for distinct server resources. + ### Query Options -- `select`: Function that lets extract array items when they're wrapped with metadata +Query Collections use TanStack Query internally and expose supported Query observer options as top-level `queryCollectionOptions` fields. + +The following top-level Query Collection options are forwarded to the underlying Query observer: + +- `select`: Function that extracts the row array TanStack DB materializes from a wrapped Query response - `enabled`: Whether the query should automatically run (default: `true`) -- `refetchInterval`: Refetch interval in milliseconds (default: 0 — set an interval to enable polling refetching) +- `refetchInterval`: Refetch interval in milliseconds - `retry`: Retry configuration for failed queries - `retryDelay`: Delay between retries - `staleTime`: How long data is considered fresh -- `meta`: Optional metadata that will be passed to the query function context +- `gcTime`: How long unused query data stays in the Query cache +- `refetchOnWindowFocus`: Whether to refetch when the window regains focus +- `refetchOnReconnect`: Whether to refetch when the network reconnects +- `refetchOnMount`: Whether to refetch when the observer mounts +- `networkMode`: Query network mode +- `initialData`: Initial Query response for eager collections +- `initialDataUpdatedAt`: Timestamp used by TanStack Query to determine initial data freshness +- `meta`: Metadata passed to the query function context. Query Collections may add `loadSubsetOptions` for on-demand queries. + +```ts +const todosCollection = createCollection( + queryCollectionOptions({ + queryKey: ["todos"], + queryFn: fetchTodos, + queryClient, + getKey: (todo) => todo.id, + refetchOnWindowFocus: true, + refetchOnReconnect: true, + refetchOnMount: "always", + networkMode: "online", + }) +) +``` + +Top-level `meta` is always merged by Query Collection so it can add on-demand `loadSubsetOptions`. Other supported top-level Query options are only passed to TanStack Query when you define them. If you omit them, `QueryClient.defaultOptions` can still apply. + +Some fields are owned or reinterpreted by the collection adapter rather than treated as ordinary Query option pass-through: + +- `queryKey`: Identifies the Query cache entry and, in on-demand mode, may be built from load-subset options. +- `queryFn`: Fetches the complete collection state or the requested on-demand subset. +- `select`: Extracts array rows from wrapped responses before they are stored in the collection. This is not the same contract as TanStack Query's `select` option. +- `queryClient`: Supplies the Query client instance used by the collection. +- `syncMode`: Controls whether the collection syncs eagerly or on demand. +- `getKey`: Extracts each row's stable TanStack DB key. +- Mutation handlers such as `onInsert`, `onUpdate`, and `onDelete`. + +Some TanStack Query fields are owned or reinterpreted by Query Collection and are intentionally not exposed as ordinary Query observer options: + +- `queryKey`, `queryFn`, and `queryClient` +- `select` (Query Collection uses this for row extraction, not TanStack Query's observer-level `select` contract) +- `meta` (merged by Query Collection so on-demand `loadSubsetOptions` can be included) +- `subscribed` (Query Collection owns the observer subscription lifecycle) +- `structuralSharing` and `notifyOnChangeProps` (managed by Query Collection synchronization) + +`placeholderData` is intentionally unsupported. TanStack Query treats placeholder data as observer-local presentation state rather than cached Query data. Materializing it would expose temporary UI data as collection-wide normalized rows. Render placeholders in the consuming UI instead. + +### Request Cancellation with `QueryFunctionContext.signal` + +TanStack Query passes an `AbortSignal` to `queryFn` through the query function +context. Forward `ctx.signal` to `fetch` or another abortable client to make the +request cancellable: + +```typescript +const todosCollection = createCollection( + queryCollectionOptions({ + queryKey: ["todos"], + queryFn: async (ctx) => { + const response = await fetch("/api/todos", { + signal: ctx.signal, + }) + + if (!response.ok) { + throw new Error("Failed to fetch todos") + } + + return response.json() as Promise> + }, + queryClient, + getKey: (todo) => todo.id, + }), +) +``` + +Explicit collection cleanup cancels each exact Query key the collection is +currently tracking before removing it from the Query cache: + +```typescript +await todosCollection.cleanup() +``` + +The underlying request is aborted only when its client consumes `ctx.signal`. +A client that ignores the signal may continue its request even though the +collection has been cleaned up. + +An unloaded on-demand subset is no longer tracked. A later explicit collection +cleanup does not revisit its Query key. + +Query cache entries are shared within a `QueryClient`. Explicit cleanup can +affect other consumers using the same exact Query keys. + +On-demand subset unloading does not explicitly call +`queryClient.cancelQueries()`. It removes the subset's Query observer. If this +was the final observer and the query function consumed `ctx.signal`, TanStack +Query aborts the request. If the signal was ignored, or another observer still +uses the same exact Query key, the request may finish and remain cached until +`gcTime`. ### Using with `queryOptions(...)` -If your app already uses TanStack Query's `queryOptions` helper (e.g. from `@tanstack/react-query`), you can spread those options into `queryCollectionOptions`. Note that `queryFn` must be explicitly provided since query collections require it both in types and at runtime: +If your app already uses TanStack Query's `queryOptions` helper (e.g. from `@tanstack/react-query`), you can spread compatible top-level options into `queryCollectionOptions`. Note that `queryFn` must be explicitly provided since query collections require it both in types and at runtime, and Query Collection's `select` option is for row extraction rather than TanStack Query observer-level selection: ```typescript import { QueryClient } from "@tanstack/query-core" -import { createCollection } from "@tanstack/db" +import { DbClient, collectionOptions } from "@tanstack/db" import { queryCollectionOptions } from "@tanstack/query-db-collection" import { queryOptions } from "@tanstack/react-query" const queryClient = new QueryClient() +const db = new DbClient({ queryClient }) const listOptions = queryOptions({ queryKey: ["todos"], @@ -84,18 +284,114 @@ const listOptions = queryOptions({ }, }) -const todosCollection = createCollection( +const todosCollection = collectionOptions("todos", (client) => queryCollectionOptions({ + id: "todos", ...listOptions, queryFn: (context) => listOptions.queryFn!(context), - queryClient, + queryClient: client.requireDependency("queryClient"), getKey: (item) => item.id, }), ) + +const todos = db.collection(todosCollection) ``` If `queryFn` is missing at runtime, `queryCollectionOptions` throws `QueryFnRequiredError`. +### Initial Data + +Eager Query Collections support TanStack Query's `initialData` and +`initialDataUpdatedAt` options. Initial data has the original Query response +shape, is stored in the Query cache, and is immediately materialized as +normalized collection rows. TanStack Query uses `initialDataUpdatedAt` together +with `staleTime` to decide whether to fetch. + +```typescript +const serverRenderedAt = Date.now() +const initialTodos = [ + { id: "1", title: "Write documentation" }, + { id: "2", title: "Ship initial data support" }, +] + +const todosCollection = createCollection( + queryCollectionOptions({ + queryKey: ["todos"], + queryFn: fetchTodos, + queryClient, + getKey: (todo) => todo.id, + initialData: initialTodos, + initialDataUpdatedAt: serverRenderedAt, + staleTime: 60_000, + }), +) +``` + +An existing cached or hydrated Query response takes precedence over +`initialData`. Query keys remain the cache identity: two collections using the +same QueryClient and exact Query key observe one shared Query document, and a +later collection's initializer does not replace it. Use distinct Query keys for +independent documents. + +Initial data is supported only for eager collections. A collection-wide value +cannot establish row membership for arbitrary on-demand predicates, ordering, +limits, and offsets. For `syncMode: "on-demand"`, seed or hydrate the exact +derived Query cache entries instead. + +If a stale initial response triggers a fetch, the initial rows remain available +while it is in flight. A successful response reconciles them through the normal +row ownership pipeline; an error retains the initial rows. Direct writes use the +same Query cache-patching rules as fetched data, and a later successful server +response may reconcile or replace those writes. + +### Selecting Rows from Wrapped Responses + +Many APIs return rows inside a response envelope that also contains metadata such as pagination cursors, totals, or request information. Use `select` to extract the row array that TanStack DB should materialize: + +```typescript +interface TodosResponse { + items: Array<{ id: string; title: string }> + nextCursor?: string + total: number +} + +const todosCollection = createCollection( + queryCollectionOptions({ + queryKey: ["todos"], + queryFn: async (): Promise => { + const response = await fetch("/api/todos") + return response.json() + }, + initialData: { + items: [{ id: "1", title: "Initial todo" }], + nextCursor: undefined, + total: 1, + }, + select: (response) => response.items, + queryClient, + getKey: (item) => item.id, + }), +) +``` + +`select` is a query-db-collection row extraction hook. It tells TanStack DB which rows to materialize while the TanStack Query cache keeps the original query response shape. In the example above, `queryClient.getQueryData(["todos"])` still returns the full `TodosResponse`, including `nextCursor` and `total`. + +The same projection applies to `initialData`: provide the complete response +envelope, and Query Collection materializes the rows returned by `select` while +preserving the envelope in the Query cache. + +This differs from TanStack Query's observer-level `select`: query-db-collection uses this option to bridge Query's response object into DB's normalized row store. + +Direct write utilities such as `writeInsert`, `writeUpdate`, and `writeDelete` make a best-effort attempt to update the matching row array inside wrapped Query cache entries while preserving wrapper metadata. + +This works automatically for simple wrappers such as: + +- `{ data: [...] }` +- `{ items: [...] }` +- `{ results: [...] }` + +Derived projections, such as `select: (response) => response.edges.map((edge) => edge.node)`, are read-side row extraction only. query-db-collection cannot generally reconstruct the original response envelope from updated rows. Refetch or invalidate the query if the wrapped cache must exactly reflect direct writes for a derived projection. + ### Collection Options - `id`: Unique identifier for the collection @@ -113,6 +409,13 @@ If `queryFn` is missing at runtime, `queryCollectionOptions` throws `QueryFnRequ The `meta` option allows you to pass additional metadata to your query function. By default, Query Collections automatically include `loadSubsetOptions` in the meta object, which contains filtering, sorting, and pagination options for on-demand queries. +Treat `ctx.meta.loadSubsetOptions` and its nested request data as read-only. +Do not edit expression nodes, ordering options, Dates, byte arrays, or membership +arrays. Build separate API parameters instead. Core retains request data without +cloning it; changing submitted data can make the request disagree with its cache +key. To change a query constant, supply a new value rather than mutating the old +one. Cancellation through the request's `AbortSignal` remains supported. + ### Type-Safe Meta Access The `ctx.meta.loadSubsetOptions` property is automatically typed as `LoadSubsetOptions` without requiring any additional imports or type assertions: @@ -445,22 +748,99 @@ const todosCollection = createCollection( todosCollection.insert({ text: "Buy milk", completed: false }) ``` -### Example: Large Dataset Pagination +### Server pagination with live queries + +`useLiveInfiniteQuery` in React, Vue, and Svelte grows a local ordered query +window. It does not run TanStack Query's `InfiniteQueryObserver`. Query +Collections use `QueryObserver`, so `queryFn` receives +`meta.loadSubsetOptions`, not `pageParam`. + +The previously ignored `getNextPageParam` option has been removed. Delete it +from your hook config; passing it at runtime now throws a clear error. +`initialPageParam` labels result pages only. It does not set a remote offset +or server cursor. + +For server loading, use `syncMode: 'on-demand'` and make `queryFn` fulfill the +requested filter, order, offset, and limit. Use a deterministic total order +(for example, a timestamp followed by a unique ID). The loader may request a +prefix, a suffix, a tie group, or the full filtered source. A request is not +necessarily one UI page: the hook fetches an extra row to determine +`hasNextPage`. Returning one capped endpoint page can incorrectly make the +query appear exhausted even when the server has more rows. + +#### Endpoints with fixed-size pages + +If your endpoint uses page numbers, drain enough server pages to fulfill each +request. This example assumes a zero-based page API with a fixed size of 50. +The endpoint must apply the supplied filters and sorts **before** pagination, +keep a consistent ordered result while its pages are read, and return +`nextPage: null` only when it has authoritatively exhausted that result. +This example uses offset-based pagination. `api.listPosts` translates the full +`where` expression and `orderBy` options into the endpoint's syntax, and rejects +unsupported expressions. The separate `cursor` hints are deliberately unused; +cursor-based adapters must handle those hints alongside `where`, not treat them +as already included in it. See [QueryFn and Predicate Push-Down](#queryfn-and-predicate-push-down) +for translation helpers. Do not drop predicates or filter after paginating: +either changes the requested window. ```typescript -// Load additional pages without refetching existing data -const loadMoreTodos = async (page) => { - const newTodos = await api.getTodos({ page, limit: 50 }) +import { createCollection } from '@tanstack/db' +import { queryCollectionOptions } from '@tanstack/query-db-collection' - // Add new items without affecting existing ones - todosCollection.utils.writeBatch(() => { - newTodos.forEach((todo) => { - todosCollection.utils.writeInsert(todo) - }) - }) -} +type Post = { id: number; createdAt: number; title: string } +const serverPageSize = 50 + +const postsCollection = createCollection( + queryCollectionOptions({ + queryKey: ['posts'], + queryClient, + syncMode: 'on-demand', + getKey: (post: Post) => post.id, + queryFn: async (ctx): Promise> => { + const { where, orderBy, offset = 0, limit } = ctx.meta?.loadSubsetOptions ?? {} + const skip = offset % serverPageSize + let page: number | null = Math.floor(offset / serverPageSize) + const gathered: Array = [] + + while (page !== null && (limit === undefined || gathered.length < skip + limit)) { + ctx.signal.throwIfAborted() + const response: { rows: Array; nextPage: number | null } = + await api.listPosts({ + page, + pageSize: serverPageSize, + where, + orderBy, + signal: ctx.signal, + }) + gathered.push(...response.rows) + page = response.nextPage + } + + return gathered.slice(skip, limit === undefined ? undefined : skip + limit) + }, + }), +) + +// React example; the collection protocol is the same for Vue and Svelte. +const { data, fetchNextPage, hasNextPage } = useLiveInfiniteQuery( + (q) => q.from({ post: postsCollection }) + .orderBy(({ post }) => post.createdAt) + .orderBy(({ post }) => post.id), + { pageSize: 20 }, +) ``` +Reject failed requests instead of returning partial rows as success. An +unlimited request must drain until the endpoint reports exhaustion. If the +endpoint uses opaque cursors instead of page numbers, keep that cursor handling +inside `queryFn` or its adapter; honoring a new offset may require starting at +the beginning again. The hook does not maintain remote cursor history. + +Manually appending rows with `writeUpsert` is a separate, lower-level loading +strategy. It does not make an eager `queryFn` incremental: a later successful +refetch still replaces its complete state and can remove appended rows. +`staleTime: Infinity` does not prevent explicit refetch or invalidation. + ## Important Behaviors ### Full State Sync diff --git a/docs/collections/query-initial-placeholder-data-design.md b/docs/collections/query-initial-placeholder-data-design.md new file mode 100644 index 0000000000..9ee866f325 --- /dev/null +++ b/docs/collections/query-initial-placeholder-data-design.md @@ -0,0 +1,249 @@ +# Query Collection initial and placeholder data semantics + +## Status and scope + +This document defines the semantics for TanStack Query `initialData` and +`placeholderData` at the `@tanstack/query-db-collection` boundary. It is the +design follow-up for [RFC #1643](https://github.com/TanStack/db/issues/1643) +and [issue #346](https://github.com/TanStack/db/issues/346). The accompanying +implementation adds the approved eager `initialData` behavior without changing +the persistence format. + +The adapter connects two different models: + +- TanStack Query owns a document cache and remote-query lifecycle. +- TanStack DB owns normalized rows, local queries, and optimistic writes. +- Query Collection projects a Query response into rows and records which Query + key owns each materialized row. + +`initialData` and `placeholderData` must not be treated as equivalent ways to +provide an array. In Query Core 5.90.20, `initialData` initializes Query cache +state with `status: "success"` and a `dataUpdatedAt` timestamp. By contrast, +`placeholderData` is computed per observer only while Query state is pending; +it produces an observer result with `isPlaceholderData: true` but is not stored +in Query state. + +## Decision + +Support Query-owned `initialData` as an additive, eager-mode option. Materialize +it immediately through the existing row extraction and ownership pipeline. +Do not expose or materialize `placeholderData` in the first implementation. + +Query Collection can already materialize data that was seeded or hydrated into +the QueryClient before the observer is created, and QueryClient defaults can +already provide `initialData` indirectly. That is useful existing behavior, but +it does not close the configuration gap. Applications commonly share one +QueryClient across many Query Collections: a client-wide default is too broad, +while imperative `setQueryData` requires coordinating collection construction, +exact Query keys, and initialization elsewhere. The additive field supplies a +collection-local declaration while leaving Query as the cache authority. + +The minimal additive API is: + +```ts +initialData?: TQueryData | (() => TQueryData) +initialDataUpdatedAt?: number | (() => number | undefined) +``` + +These remain flat top-level fields. Their types describe the original Query +response, not the extracted row array. Consequently, wrapped responses use the +same adapter `select` as network responses: + +```ts +queryCollectionOptions({ + queryKey: ["todos"], + queryFn: fetchTodos, + initialData: { items: serverTodos, nextCursor: null }, + select: (response) => response.items, + // ... +}) +``` + +Query keys still define cache identity. If two Query Collections on the same +QueryClient use the same exact key, they observe one shared Query document; +`initialData` initializes that document only when it does not already exist. +Collection-local configuration does not create collection-local cache data, and +later observers must not replace an existing document with their initializer. +Collections that require independent initial documents must use distinct keys. + +This initial API is limited to `syncMode: "eager"`. A single configuration-level +value cannot lawfully initialize an open-ended family of on-demand subset keys, +and Query's `initialData` function receives no query key or subset context. +Applications that already know data for an exact on-demand key should seed or +hydrate that Query cache entry instead. A future subset-aware initializer would +need an explicit key/subset argument and a separate design. + +`placeholderData` remains Query UI vocabulary. A DB collection has no +observer-local result surface: materializing a placeholder would make it visible +to every DB query and mutation, assign it row ownership, and potentially persist +it. Callers should render placeholders in the consuming UI. A future opt-in +temporary-row feature, if needed, should be a DB feature with explicit provenance +and lifecycle rather than Query's `placeholderData` option. + +## Authority model + +"Authoritative" has two dimensions here. Query owns the authoritative document +for a Query key, while DB owns the current normalized row state. A server result +is the newest remote snapshot, but local optimistic transactions can temporarily +overlay its rows. + +| Phase | Query document authority | Materialized row authority | Consequence | +| --- | --- | --- | --- | +| No cached data | None | Existing DB/persisted rows, if any | The collection waits for Query; absence is not an empty result. | +| Initial data | Query cache `initialData` | Its projected rows, subject to normal local overlays | It is a real cached snapshot, not temporary presentation data. | +| Fetch/refetch in flight | Existing Query document | Existing rows | Loading does not clear rows or ownership. | +| Server success | Returned response replaces the Query document | Projected server rows reconcile the owning Query key | Missing rows lose this Query owner's lease; shared rows remain. | +| Fetch/refetch error | Last successful Query document | Existing rows | An error does not retract initial or previously fetched rows. | +| Placeholder presentation | No Query document | No rows | Placeholder data is never passed to DB. | + +`initialDataUpdatedAt` and `staleTime` remain Query-owned. They decide whether a +fetch starts; the adapter does not reproduce their freshness calculation. An +initial value is therefore a seed snapshot with ordinary Query authority, not a +weaker class of row waiting to be promoted. The first successful server result +reconciles it through the same path as any later refetch. + +## Behavior matrix + +| Concern | `initialData` | `placeholderData` | +| --- | --- | --- | +| Query cache | Stored as successful Query data | Not stored; observer-only | +| Existing cache entry | Existing cached/hydrated data wins; `initialData` is not reapplied | Not applicable | +| DB materialization | Immediate in eager mode | Never | +| Wrapped response | Adapter `select(initialData)` extracts rows; original envelope stays cached | Unsupported | +| Function value | Evaluated by Query once when the Query is created | Not forwarded or evaluated by the adapter | +| Ownership | The Query key owns projected rows exactly like a server success | No ownership | +| Overlapping subsets | Not applicable to the initial eager-only API | Not applicable | +| Ready state | Initial successful result can make the collection ready synchronously | Cannot make the collection ready | +| Refetch success | Reconciles additions, updates, and removals normally | N/A | +| Refetch error | Initial rows and ownership remain; error state is reported | N/A | +| Cancellation/cleanup | Existing ownership cleanup rules apply; a cancelled fetch does not retract the cached seed | No rows to clean up | +| Query cache GC/unload | Existing Query-to-row ownership and persisted-retention rules apply | No effect | +| Query dehydration | Query owns persistence of the initial response | Never persisted | +| DB persistence/hydration | Rows and owner metadata use the existing format; no provenance tag is added | Never persisted or hydrated | +| Direct writes before server success | Allowed under the existing write rules below | No target rows exist | +| QueryClient defaults | Supported for eager `initialData`; see compatibility guard below | Must be suppressed at this adapter boundary | + +## Select and writes + +Adapter `select` remains a one-way row extractor. It is applied identically to +initial and network responses, and TanStack Query retains the original response +shape. Query observer-level `select` remains unsupported. + +Direct writes before the first server result follow the current authority rule: +they update DB immediately and may patch Query cache only when the reverse update +is lawful. A raw array can be replaced. For a wrapped response, the existing +best-effort patch is lawful only when `select` returns an array property of the +cached object by reference, allowing the wrapper to be preserved. A derived +projection such as `response.edges.map(...)` has no general reverse projection; +the adapter must leave that Query document unchanged and rely on invalidate or +refetch. It must never fabricate an envelope around rows. + +The next successful remote response remains authoritative for that Query key and +may overwrite a direct cache patch or normalized row value. Mutation handlers and +optimistic transaction barriers retain their existing semantics. + +## Ownership, persistence, and transitions + +Initial rows use the existing `queryToRows` and `rowToQueries` relationship. No +`seed`, `temporary`, or `placeholder` bit is added to a row. This keeps these +invariants intact: + +1. A successful result, whether initial or fetched, is a complete snapshot for + its Query key. +2. A row is deleted when a snapshot omits it only if no other Query key owns it. +3. Unload, cache GC, and collection cleanup remove only the relevant ownership. +4. A failed or cancelled fetch cannot turn the last successful snapshot into an + empty snapshot. +5. Persistence records only Query data and the existing row ownership metadata; + observer-only presentation state is never persisted. + +The expected transitions are: + +- **Initial, fresh:** materialize and become ready; do not fetch until Query's + normal freshness triggers say to do so. +- **Initial, stale:** materialize and become ready while Query fetches; success + reconciles the same ownership, and error retains the seed. +- **Loading without data:** keep the collection's prior independently owned or + hydrated rows; do not infer an empty result. +- **Placeholder to loading/success/error:** the placeholder is UI-only. DB sees + no transition until success; error leaves DB unchanged. +- **Cleanup before network completion:** existing cancellation and readiness + listener cleanup apply. A late result must not mutate a cleaned-up collection. + +## Defaults and compatibility guard + +Query Collection currently constructs a `QueryObserver`, so QueryClient defaults +can contain semantic fields even when Query Collection does not expose them. +Implementation must explicitly enforce this design after Query defaults are +resolved: + +- reject an explicitly configured `initialData` in on-demand mode; +- prevent default `initialData` from initializing on-demand subset observers; +- prevent explicit or default `placeholderData` from reaching all Query + Collection observers; +- continue to let omitted eager `initialData` and `initialDataUpdatedAt` inherit + QueryClient defaults; +- never copy a function-valued initializer into adapter metadata or persisted + ownership metadata. Query Core may own it as an option and stores only its + evaluated data in Query state. + +Silently materializing default placeholder data would violate the public +compatibility table even before a top-level field is added. The guard is therefore +a correctness fix, not support for placeholder semantics. + +Other Query-owned options keep their current classification. Query key creation, +adapter `select`, subscriptions, `notifyOnChangeProps`, and structural sharing +remain adapter-owned or reinterpreted. No nested `queryOptions`, runtime binding +API, subset deduplication, or lease manager is introduced by this design. + +## Rejected designs + +- **Forward both options mechanically.** `result.isSuccess` is true for a + placeholder observer result, so the current success handler would normalize + presentation-only data and give it durable-looking ownership. +- **Tag placeholder rows and later promote or delete them.** Tags would have to + survive collisions with real rows, overlapping observers, local writes, + unload, GC, persistence, and hydration. Query's observer-local placeholder has + no collection-wide lifetime that can drive those transitions safely. +- **Treat initial rows as unowned temporary rows.** Query considers initial data + real cached data. Bypassing normal ownership would leak rows or allow cleanup + of one Query to remove rows still represented by another. +- **Seed every on-demand key from one value.** A collection-level initializer + cannot prove membership in arbitrary predicate/order/limit subsets. +- **Create a wrapped response from selected rows.** A read projection is not an + inverse. Fabricating metadata, cursors, or edges corrupts Query cache meaning. +- **Persist initializer functions.** Functions are not structured-clone safe and + are runtime configuration, not data. + +## Implementation and test sequence + +Each behavior PR should begin with the named failing or characterization tests. + +1. **Guard the existing boundary.** Add focused tests proving that explicit and + QueryClient-default `placeholderData` never materialize, never mark the + collection ready, and never survive dehydration as data. Characterize current + default `initialData` behavior in eager and on-demand modes. Then suppress + placeholder and on-demand initialization when constructing observers. +2. **Add eager initial data.** Add the two flat typed fields and forward only + defined values so QueryClient defaults remain intact. Test static and function + values, fresh versus stale timestamps, synchronous readiness, fetch error, + cancellation, cleanup, an explicit on-demand configuration error, multiple + collections on one QueryClient, and same-key first-initializer-wins behavior. +3. **Lock projection and writes.** Test raw arrays, direct-property wrapped + responses, and derived projections. Verify the full initial envelope remains + in Query cache, lawful direct writes preserve it, unlawful reverse projection + does not fabricate one, and server success reconciles rows. +4. **Lock ownership.** Test initial-to-server row removal, overlapping ownership + with externally hydrated/persisted rows, GC/unload, remount within `gcTime`, + and late notification after cleanup. Reuse the existing ownership machinery; + do not add a second seed ownership map. +5. **Lock persistence.** Dehydrate and `structuredClone` initial raw and wrapped + data; hydrate Query and DB state; verify ownership reconciliation and that no + function appears in Query metadata or adapter persistence metadata. +6. **Document the shipped API.** Move the settled behavior into the Query Options, + row extraction, direct writes, on-demand, and persistence sections of the user + guide. Update RFC #1643 and close or narrow #346 only after these contracts ship. + +Placeholder materialization, subset-aware initialization, and a lawful general +reverse projection are separate future proposals. None is a prerequisite for the +minimal eager `initialData` API. diff --git a/docs/collections/rxdb-collection.md b/docs/collections/rxdb-collection.md index d139ddc7f4..a1ba038c89 100644 --- a/docs/collections/rxdb-collection.md +++ b/docs/collections/rxdb-collection.md @@ -2,8 +2,6 @@ title: RxDB Collection --- -# RxDB Collection - RxDB collections provide seamless integration between TanStack DB and [RxDB](https://rxdb.info), enabling automatic synchronization between your in-memory TanStack DB collections and RxDB's local-first database. Giving you offline-ready persistence, and powerful sync capabilities with a wide range of backends. diff --git a/docs/collections/trailbase-collection.md b/docs/collections/trailbase-collection.md index 938e714a52..ea1c5ead36 100644 --- a/docs/collections/trailbase-collection.md +++ b/docs/collections/trailbase-collection.md @@ -2,8 +2,6 @@ title: TrailBase Collection --- -# TrailBase Collection - TrailBase collections provide seamless integration between TanStack DB and [TrailBase](https://trailbase.io), enabling real-time data synchronization with TrailBase's self-hosted application backend. ## Overview @@ -194,11 +192,13 @@ export const todosCollection = createCollection( // Use in component function TodoList() { - const { data: todos } = useLiveQuery((q) => - q.from({ todo: todosCollection }) - .where(({ todo }) => not(todo.completed)) - .orderBy(({ todo }) => todo.created_at, 'desc') - ) + const { data: todos } = useLiveQuery({ + query: (q) => + q + .from({ todo: todosCollection }) + .where(({ todo }) => not(todo.completed)) + .orderBy(({ todo }) => todo.created_at, 'desc'), + }) const addTodo = (text: string) => { todosCollection.insert({ diff --git a/docs/community/resources.md b/docs/community/resources.md index 781c11963e..86ed35d53d 100644 --- a/docs/community/resources.md +++ b/docs/community/resources.md @@ -3,8 +3,6 @@ title: Community Resources id: community-resources --- -# Community Resources - This page contains a curated list of community-created packages, tools, and resources that extend or complement TanStack DB. ## Community Packages diff --git a/docs/config.json b/docs/config.json index 09ca7c62ba..94a08ef0fd 100644 --- a/docs/config.json +++ b/docs/config.json @@ -30,6 +30,10 @@ "label": "Live Queries", "to": "guides/live-queries" }, + { + "label": "SSR and Hydration", + "to": "guides/ssr" + }, { "label": "Mutations", "to": "guides/mutations" diff --git a/docs/framework/angular/reference/functions/injectLiveQuery.md b/docs/framework/angular/reference/functions/injectLiveQuery.md index 5eb790a543..efe20157a4 100644 --- a/docs/framework/angular/reference/functions/injectLiveQuery.md +++ b/docs/framework/angular/reference/functions/injectLiveQuery.md @@ -11,7 +11,7 @@ title: injectLiveQuery function injectLiveQuery(options): InjectLiveQueryResult; ``` -Defined in: [index.ts:89](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L89) +Defined in: [index.ts:93](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L93) ### Type Parameters @@ -45,7 +45,7 @@ Defined in: [index.ts:89](https://github.com/TanStack/db/blob/main/packages/angu function injectLiveQuery(options): InjectLiveQueryResult; ``` -Defined in: [index.ts:99](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L99) +Defined in: [index.ts:103](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L103) ### Type Parameters @@ -79,7 +79,7 @@ Defined in: [index.ts:99](https://github.com/TanStack/db/blob/main/packages/angu function injectLiveQuery(queryFn): InjectLiveQueryResult; ``` -Defined in: [index.ts:109](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L109) +Defined in: [index.ts:113](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L113) ### Type Parameters @@ -103,7 +103,7 @@ Defined in: [index.ts:109](https://github.com/TanStack/db/blob/main/packages/ang function injectLiveQuery(queryFn): InjectLiveQueryResult; ``` -Defined in: [index.ts:112](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L112) +Defined in: [index.ts:116](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L116) ### Type Parameters @@ -127,7 +127,7 @@ Defined in: [index.ts:112](https://github.com/TanStack/db/blob/main/packages/ang function injectLiveQuery(config): InjectLiveQueryResult; ``` -Defined in: [index.ts:117](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L117) +Defined in: [index.ts:121](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L121) ### Type Parameters @@ -151,7 +151,7 @@ Defined in: [index.ts:117](https://github.com/TanStack/db/blob/main/packages/ang function injectLiveQuery(liveQueryCollection): InjectLiveQueryResultWithCollection; ``` -Defined in: [index.ts:121](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L121) +Defined in: [index.ts:125](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L125) ### Type Parameters @@ -183,7 +183,7 @@ Defined in: [index.ts:121](https://github.com/TanStack/db/blob/main/packages/ang function injectLiveQuery(liveQueryCollection): InjectLiveQueryResultWithSingleResultCollection; ``` -Defined in: [index.ts:129](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L129) +Defined in: [index.ts:133](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L133) ### Type Parameters diff --git a/docs/framework/angular/reference/interfaces/InjectLiveQueryResult.md b/docs/framework/angular/reference/interfaces/InjectLiveQueryResult.md index 066d6b5bab..cf4d8f3cee 100644 --- a/docs/framework/angular/reference/interfaces/InjectLiveQueryResult.md +++ b/docs/framework/angular/reference/interfaces/InjectLiveQueryResult.md @@ -5,7 +5,7 @@ title: InjectLiveQueryResult # Interface: InjectLiveQueryResult\ -Defined in: [index.ts:32](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L32) +Defined in: [index.ts:36](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L36) The result of calling `injectLiveQuery`. Contains reactive signals for the query state and data. @@ -27,7 +27,7 @@ collection: Signal< | null>; ``` -Defined in: [index.ts:38](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L38) +Defined in: [index.ts:42](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L42) A signal containing the underlying collection instance (null for disabled queries) @@ -39,7 +39,7 @@ A signal containing the underlying collection instance (null for disabled querie data: Signal>; ``` -Defined in: [index.ts:36](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L36) +Defined in: [index.ts:40](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L40) A signal containing the results as an array, or single result for findOne queries @@ -51,7 +51,7 @@ A signal containing the results as an array, or single result for findOne querie isCleanedUp: Signal; ``` -Defined in: [index.ts:54](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L54) +Defined in: [index.ts:58](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L58) A signal indicating whether the collection has been cleaned up @@ -63,7 +63,7 @@ A signal indicating whether the collection has been cleaned up isError: Signal; ``` -Defined in: [index.ts:52](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L52) +Defined in: [index.ts:56](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L56) A signal indicating whether the collection has an error @@ -75,7 +75,7 @@ A signal indicating whether the collection has an error isIdle: Signal; ``` -Defined in: [index.ts:50](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L50) +Defined in: [index.ts:54](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L54) A signal indicating whether the collection is idle @@ -87,7 +87,7 @@ A signal indicating whether the collection is idle isLoading: Signal; ``` -Defined in: [index.ts:46](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L46) +Defined in: [index.ts:50](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L50) A signal indicating whether the collection is currently loading @@ -99,7 +99,7 @@ A signal indicating whether the collection is currently loading isReady: Signal; ``` -Defined in: [index.ts:48](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L48) +Defined in: [index.ts:52](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L52) A signal indicating whether the collection is ready @@ -111,7 +111,7 @@ A signal indicating whether the collection is ready state: Signal[K] }>>; ``` -Defined in: [index.ts:34](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L34) +Defined in: [index.ts:38](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L38) A signal containing the complete state map of results keyed by their ID @@ -123,6 +123,6 @@ A signal containing the complete state map of results keyed by their ID status: Signal; ``` -Defined in: [index.ts:44](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L44) +Defined in: [index.ts:48](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L48) A signal containing the current status of the collection diff --git a/docs/framework/angular/reference/interfaces/InjectLiveQueryResultWithCollection.md b/docs/framework/angular/reference/interfaces/InjectLiveQueryResultWithCollection.md index c6f1c6c3bc..06cefc8bc6 100644 --- a/docs/framework/angular/reference/interfaces/InjectLiveQueryResultWithCollection.md +++ b/docs/framework/angular/reference/interfaces/InjectLiveQueryResultWithCollection.md @@ -5,7 +5,7 @@ title: InjectLiveQueryResultWithCollection # Interface: InjectLiveQueryResultWithCollection\ -Defined in: [index.ts:57](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L57) +Defined in: [index.ts:61](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L61) ## Type Parameters @@ -32,7 +32,7 @@ collection: Signal< | null>; ``` -Defined in: [index.ts:64](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L64) +Defined in: [index.ts:68](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L68) *** @@ -42,7 +42,7 @@ Defined in: [index.ts:64](https://github.com/TanStack/db/blob/main/packages/angu data: Signal; ``` -Defined in: [index.ts:63](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L63) +Defined in: [index.ts:67](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L67) *** @@ -52,7 +52,7 @@ Defined in: [index.ts:63](https://github.com/TanStack/db/blob/main/packages/angu isCleanedUp: Signal; ``` -Defined in: [index.ts:70](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L70) +Defined in: [index.ts:74](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L74) *** @@ -62,7 +62,7 @@ Defined in: [index.ts:70](https://github.com/TanStack/db/blob/main/packages/angu isError: Signal; ``` -Defined in: [index.ts:69](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L69) +Defined in: [index.ts:73](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L73) *** @@ -72,7 +72,7 @@ Defined in: [index.ts:69](https://github.com/TanStack/db/blob/main/packages/angu isIdle: Signal; ``` -Defined in: [index.ts:68](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L68) +Defined in: [index.ts:72](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L72) *** @@ -82,7 +82,7 @@ Defined in: [index.ts:68](https://github.com/TanStack/db/blob/main/packages/angu isLoading: Signal; ``` -Defined in: [index.ts:66](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L66) +Defined in: [index.ts:70](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L70) *** @@ -92,7 +92,7 @@ Defined in: [index.ts:66](https://github.com/TanStack/db/blob/main/packages/angu isReady: Signal; ``` -Defined in: [index.ts:67](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L67) +Defined in: [index.ts:71](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L71) *** @@ -102,7 +102,7 @@ Defined in: [index.ts:67](https://github.com/TanStack/db/blob/main/packages/angu state: Signal>; ``` -Defined in: [index.ts:62](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L62) +Defined in: [index.ts:66](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L66) *** @@ -112,4 +112,4 @@ Defined in: [index.ts:62](https://github.com/TanStack/db/blob/main/packages/angu status: Signal; ``` -Defined in: [index.ts:65](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L65) +Defined in: [index.ts:69](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L69) diff --git a/docs/framework/angular/reference/interfaces/InjectLiveQueryResultWithSingleResultCollection.md b/docs/framework/angular/reference/interfaces/InjectLiveQueryResultWithSingleResultCollection.md index 2cf88f2856..29daff2673 100644 --- a/docs/framework/angular/reference/interfaces/InjectLiveQueryResultWithSingleResultCollection.md +++ b/docs/framework/angular/reference/interfaces/InjectLiveQueryResultWithSingleResultCollection.md @@ -5,7 +5,7 @@ title: InjectLiveQueryResultWithSingleResultCollection # Interface: InjectLiveQueryResultWithSingleResultCollection\ -Defined in: [index.ts:73](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L73) +Defined in: [index.ts:77](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L77) ## Type Parameters @@ -32,7 +32,7 @@ collection: Signal< | null>; ``` -Defined in: [index.ts:80](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L80) +Defined in: [index.ts:84](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L84) *** @@ -42,7 +42,7 @@ Defined in: [index.ts:80](https://github.com/TanStack/db/blob/main/packages/angu data: Signal; ``` -Defined in: [index.ts:79](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L79) +Defined in: [index.ts:83](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L83) *** @@ -52,7 +52,7 @@ Defined in: [index.ts:79](https://github.com/TanStack/db/blob/main/packages/angu isCleanedUp: Signal; ``` -Defined in: [index.ts:86](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L86) +Defined in: [index.ts:90](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L90) *** @@ -62,7 +62,7 @@ Defined in: [index.ts:86](https://github.com/TanStack/db/blob/main/packages/angu isError: Signal; ``` -Defined in: [index.ts:85](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L85) +Defined in: [index.ts:89](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L89) *** @@ -72,7 +72,7 @@ Defined in: [index.ts:85](https://github.com/TanStack/db/blob/main/packages/angu isIdle: Signal; ``` -Defined in: [index.ts:84](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L84) +Defined in: [index.ts:88](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L88) *** @@ -82,7 +82,7 @@ Defined in: [index.ts:84](https://github.com/TanStack/db/blob/main/packages/angu isLoading: Signal; ``` -Defined in: [index.ts:82](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L82) +Defined in: [index.ts:86](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L86) *** @@ -92,7 +92,7 @@ Defined in: [index.ts:82](https://github.com/TanStack/db/blob/main/packages/angu isReady: Signal; ``` -Defined in: [index.ts:83](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L83) +Defined in: [index.ts:87](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L87) *** @@ -102,7 +102,7 @@ Defined in: [index.ts:83](https://github.com/TanStack/db/blob/main/packages/angu state: Signal>; ``` -Defined in: [index.ts:78](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L78) +Defined in: [index.ts:82](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L82) *** @@ -112,4 +112,4 @@ Defined in: [index.ts:78](https://github.com/TanStack/db/blob/main/packages/angu status: Signal; ``` -Defined in: [index.ts:81](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L81) +Defined in: [index.ts:85](https://github.com/TanStack/db/blob/main/packages/angular-db/src/index.ts#L85) diff --git a/docs/framework/react/overview.md b/docs/framework/react/overview.md index 1c10d644c6..dfb451f3da 100644 --- a/docs/framework/react/overview.md +++ b/docs/framework/react/overview.md @@ -17,19 +17,34 @@ For comprehensive documentation on writing queries (filtering, joins, aggregatio ## Basic Usage +Create a `DbClient` and provide it to your React tree: + +```tsx +import { DbClient, DbProvider } from '@tanstack/react-db' + +const dbClient = new DbClient() + +root.render( + + + +) +``` + ### useLiveQuery The `useLiveQuery` hook creates a live query that automatically updates your component when data changes: ```tsx -import { useLiveQuery, eq } from '@tanstack/react-db' +import { and, eq, gt, useDbClient, useLiveQuery } from '@tanstack/react-db' function TodoList() { - const { data, isLoading } = useLiveQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) - ) + const { data, isLoading } = useLiveQuery({ + query: (q) => + q.from({ todos: todoCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })), + }) if (isLoading) return
Loading...
@@ -41,29 +56,61 @@ function TodoList() { } ``` -### Dependency Arrays - -All query hooks (`useLiveQuery`, `useLiveInfiniteQuery`, `useLiveSuspenseQuery`) accept an optional dependency array as their last parameter. This array works similarly to React's `useEffect` dependencies - when any value in the array changes, the query is recreated and re-executed. +### Query Identity -#### When to Use Dependency Arrays +React live query hooks derive the live query identity from structured query IR by default. The hook runs the query builder, normalizes the resulting IR, and uses that as the identity. When the derived identity changes, the old live query collection is cleaned up and a new one is created. -Use dependency arrays when your query depends on external reactive values (props, state, or other hooks): +That means normal structured queries do not need a separate `queryKey`. Collection descriptors provide stable collection IDs, and captured values inside structured expressions become part of the derived identity: ```tsx function FilteredTodos({ minPriority }: { minPriority: number }) { - const { data } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) + const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }) .where(({ todos }) => gt(todos.priority, minPriority)), - [minPriority] // Re-run when minPriority changes - ) + }) return
{data.length} high-priority todos
} ``` -#### What Happens When Dependencies Change +#### Collection Hooks + +`useLiveQuery` resolves collection descriptors from `DbProvider` automatically. Create small collection hooks when components need imperative collection methods like `insert`, `update`, `delete`, or `preload`: + +```tsx +function useTodoCollection() { + return useDbClient().collection(todoCollection) +} +``` + +#### When to Use Query Keys + +Use `queryKey` only when DB cannot derive identity from structured IR, or when you intentionally want to avoid deriving identity on a hot render path. The common case is a functional query variant such as `.fn.where`, `.fn.select`, or `.fn.having`: -When a dependency value changes: +```tsx +function SearchTodos({ search }: { search: string }) { + const { data } = useLiveQuery({ + queryKey: [todoCollection.id, 'search', search], + query: (q) => q.from({ todos: todoCollection }) + .fn.where(({ todos }) => + todos.text.toLowerCase().includes(search.toLowerCase()) + ), + }) + + return
{data.length} matching todos
+} +``` + +Before 1.0, an unhashable query warns in development and keeps its legacy +mount-stable identity. The query still runs, but captured values inside opaque +logic only become reactive when they are represented in `queryKey`. In 1.0, an +unhashable query without `queryKey` will throw. If deriving identity becomes +expensive across renders, the hook warns once and suggests adding a `queryKey` +as a performance escape hatch. + +#### What Happens When Identity Changes + +When the derived identity or explicit query key changes: 1. The previous live query collection is cleaned up 2. A new query is created with the updated values 3. The component re-renders with the new data @@ -71,46 +118,41 @@ When a dependency value changes: #### Best Practices -**Include all external values used in the query:** +**Use structured expressions when possible:** ```tsx -// Good - all external values in deps -const { data } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +// Good - DB can derive identity from this structured IR +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }) .where(({ todos }) => and( eq(todos.userId, userId), eq(todos.status, status) )), - [userId, status] -) - -// Bad - missing dependencies -const { data } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.userId, userId)), - [] // Missing userId! -) +}) ``` -**Empty array for static queries:** +**Add a query key for opaque runtime logic:** ```tsx -// No external dependencies - query never changes -const { data } = useLiveQuery( - (q) => q.from({ todos: todosCollection }), - [] -) +const { data } = useLiveQuery({ + queryKey: [todoCollection.id, 'by-user-fn', userId], + query: (q) => q.from({ todos: todoCollection }) + .fn.where(({ todos }) => todos.userId === userId), +}) ``` -**Omit the array for queries with no external dependencies:** +**Omit query keys for static structured queries:** ```tsx -// Same as above - no deps needed -const { data } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) -) +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }), +}) ``` +Dependency arrays are still accepted for backwards compatibility, but they warn in development and will be removed in 1.0. + +For SSR setup, collection hydration, and migration details, see the [SSR and Hydration guide](../../guides/ssr.md). + ### useLiveInfiniteQuery For paginated data with live updates, use `useLiveInfiniteQuery`: @@ -123,14 +165,20 @@ const { data, pages, fetchNextPage, hasNextPage } = useLiveInfiniteQuery( .orderBy(({ posts }) => posts.createdAt, 'desc'), { pageSize: 20, - getNextPageParam: (lastPage, allPages) => - lastPage.length === 20 ? allPages.length : undefined - }, - [category] // Re-run when category changes + } ) ``` -**Note:** The dependency array is only available when using the query function variant, not when passing a pre-created collection. +`fetchNextPage()` returns a promise that resolves after the page request settles. Failures are exposed through the returned `error` value and do not reject the promise. + +This hook widens an ordered live query; it is not TanStack Query's +`useInfiniteQuery`. `getNextPageParam` was previously ignored and is now rejected. +`initialPageParam` only labels the returned pages; it does not set a server cursor +or skip remote rows. For server pagination, use an on-demand Query Collection +whose `queryFn` fulfills `meta.loadSubsetOptions`. See the +[server pagination guide](../../collections/query-collection.md#server-pagination-with-live-queries). + +The deprecated dependency array is only available when using the query function variant, not when passing a pre-created collection. ### useLiveSuspenseQuery @@ -138,11 +186,10 @@ For React Suspense integration, use `useLiveSuspenseQuery`: ```tsx function TodoList({ filter }: { filter: string }) { - const { data } = useLiveSuspenseQuery( - (q) => q.from({ todos: todosCollection }) + const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ todos: todoCollection }) .where(({ todos }) => eq(todos.filter, filter)), - [filter] // Re-suspends when filter changes - ) + }) return (
    @@ -160,4 +207,4 @@ function App() { } ``` -When dependencies change, `useLiveSuspenseQuery` will re-suspend, showing your Suspense fallback until the new data is ready. +When the derived identity or explicit `queryKey` changes, `useLiveSuspenseQuery` will re-suspend, showing your Suspense fallback until the new data is ready. diff --git a/docs/framework/react/reference/functions/DbProvider.md b/docs/framework/react/reference/functions/DbProvider.md new file mode 100644 index 0000000000..706f05eb33 --- /dev/null +++ b/docs/framework/react/reference/functions/DbProvider.md @@ -0,0 +1,22 @@ +--- +id: DbProvider +title: DbProvider +--- + +# Function: DbProvider() + +```ts +function DbProvider(props): Element; +``` + +Defined in: [DbProvider.tsx:14](https://github.com/TanStack/db/blob/main/packages/react-db/src/DbProvider.tsx#L14) + +## Parameters + +### props + +[`DbProviderProps`](../type-aliases/DbProviderProps.md) + +## Returns + +`Element` diff --git a/docs/framework/react/reference/functions/HydrationBoundary.md b/docs/framework/react/reference/functions/HydrationBoundary.md new file mode 100644 index 0000000000..4b478bc93a --- /dev/null +++ b/docs/framework/react/reference/functions/HydrationBoundary.md @@ -0,0 +1,22 @@ +--- +id: HydrationBoundary +title: HydrationBoundary +--- + +# Function: HydrationBoundary() + +```ts +function HydrationBoundary(__namedParameters): ReactNode; +``` + +Defined in: [HydrationBoundary.tsx:13](https://github.com/TanStack/db/blob/main/packages/react-db/src/HydrationBoundary.tsx#L13) + +## Parameters + +### \_\_namedParameters + +[`HydrationBoundaryProps`](../type-aliases/HydrationBoundaryProps.md) + +## Returns + +`ReactNode` diff --git a/docs/framework/react/reference/functions/useDbClient.md b/docs/framework/react/reference/functions/useDbClient.md new file mode 100644 index 0000000000..9caf24e490 --- /dev/null +++ b/docs/framework/react/reference/functions/useDbClient.md @@ -0,0 +1,16 @@ +--- +id: useDbClient +title: useDbClient +--- + +# Function: useDbClient() + +```ts +function useDbClient(): DbClient; +``` + +Defined in: [DbProvider.tsx:22](https://github.com/TanStack/db/blob/main/packages/react-db/src/DbProvider.tsx#L22) + +## Returns + +`DbClient` diff --git a/docs/framework/react/reference/functions/useLiveInfiniteQuery.md b/docs/framework/react/reference/functions/useLiveInfiniteQuery.md index 20377885e5..7de6d2a944 100644 --- a/docs/framework/react/reference/functions/useLiveInfiniteQuery.md +++ b/docs/framework/react/reference/functions/useLiveInfiniteQuery.md @@ -11,9 +11,9 @@ title: useLiveInfiniteQuery function useLiveInfiniteQuery(liveQueryCollection, config): UseLiveInfiniteQueryReturn; ``` -Defined in: [useLiveInfiniteQuery.ts:118](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L118) +Defined in: [useLiveInfiniteQuery.ts:116](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L116) -Create an infinite query using a query function with live updates +Create an infinite query using a query function with live updates. Uses `utils.setWindow()` to dynamically adjust the limit/offset window without recreating the live query collection on each page change. @@ -50,65 +50,6 @@ Configuration including pageSize and getNextPageParam Object with pages, data, and pagination controls -### Examples - -```ts -// Basic infinite query -const { data, pages, fetchNextPage, hasNextPage } = useLiveInfiniteQuery( - (q) => q - .from({ posts: postsCollection }) - .orderBy(({ posts }) => posts.createdAt, 'desc') - .select(({ posts }) => ({ - id: posts.id, - title: posts.title - })), - { - pageSize: 20, - getNextPageParam: (lastPage, allPages) => - lastPage.length === 20 ? allPages.length : undefined - } -) -``` - -```ts -// With dependencies -const { pages, fetchNextPage } = useLiveInfiniteQuery( - (q) => q - .from({ posts: postsCollection }) - .where(({ posts }) => eq(posts.category, category)) - .orderBy(({ posts }) => posts.createdAt, 'desc'), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined - }, - [category] -) -``` - -```ts -// Router loader pattern with pre-created collection -// In loader: -const postsQuery = createLiveQueryCollection({ - query: (q) => q - .from({ posts: postsCollection }) - .orderBy(({ posts }) => posts.createdAt, 'desc') - .limit(20) -}) -await postsQuery.preload() -return { postsQuery } - -// In component: -const { postsQuery } = useLoaderData() -const { data, fetchNextPage, hasNextPage } = useLiveInfiniteQuery( - postsQuery, - { - pageSize: 20, - getNextPageParam: (lastPage) => lastPage.length === 20 ? lastPage.length : undefined - } -) -``` - ## Call Signature ```ts @@ -118,9 +59,9 @@ function useLiveInfiniteQuery( deps?): UseLiveInfiniteQueryReturn; ``` -Defined in: [useLiveInfiniteQuery.ts:128](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L128) +Defined in: [useLiveInfiniteQuery.ts:126](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L126) -Create an infinite query using a query function with live updates +Create an infinite query using a query function with live updates. Uses `utils.setWindow()` to dynamically adjust the limit/offset window without recreating the live query collection on each page change. @@ -149,69 +90,10 @@ Configuration including pageSize and getNextPageParam `unknown`[] -Array of dependencies that trigger query re-execution when changed +Deprecated array of dependencies that trigger query re-execution when changed ### Returns [`UseLiveInfiniteQueryReturn`](../type-aliases/UseLiveInfiniteQueryReturn.md)\<`TContext`\> Object with pages, data, and pagination controls - -### Examples - -```ts -// Basic infinite query -const { data, pages, fetchNextPage, hasNextPage } = useLiveInfiniteQuery( - (q) => q - .from({ posts: postsCollection }) - .orderBy(({ posts }) => posts.createdAt, 'desc') - .select(({ posts }) => ({ - id: posts.id, - title: posts.title - })), - { - pageSize: 20, - getNextPageParam: (lastPage, allPages) => - lastPage.length === 20 ? allPages.length : undefined - } -) -``` - -```ts -// With dependencies -const { pages, fetchNextPage } = useLiveInfiniteQuery( - (q) => q - .from({ posts: postsCollection }) - .where(({ posts }) => eq(posts.category, category)) - .orderBy(({ posts }) => posts.createdAt, 'desc'), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined - }, - [category] -) -``` - -```ts -// Router loader pattern with pre-created collection -// In loader: -const postsQuery = createLiveQueryCollection({ - query: (q) => q - .from({ posts: postsCollection }) - .orderBy(({ posts }) => posts.createdAt, 'desc') - .limit(20) -}) -await postsQuery.preload() -return { postsQuery } - -// In component: -const { postsQuery } = useLoaderData() -const { data, fetchNextPage, hasNextPage } = useLiveInfiniteQuery( - postsQuery, - { - pageSize: 20, - getNextPageParam: (lastPage) => lastPage.length === 20 ? lastPage.length : undefined - } -) -``` diff --git a/docs/framework/react/reference/functions/useLiveQuery.md b/docs/framework/react/reference/functions/useLiveQuery.md index accc95e0fe..e7a899af7f 100644 --- a/docs/framework/react/reference/functions/useLiveQuery.md +++ b/docs/framework/react/reference/functions/useLiveQuery.md @@ -11,9 +11,9 @@ title: useLiveQuery function useLiveQuery(queryFn, deps?): object; ``` -Defined in: [useLiveQuery.ts:84](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L84) +Defined in: [useLiveQuery.ts:360](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L360) -Create a live query using a query function +Create a live query using a query function. ### Type Parameters @@ -33,7 +33,7 @@ Query function that defines what data to fetch `unknown`[] -Array of dependencies that trigger query re-execution when changed +Deprecated array of dependencies that trigger query re-execution when changed ### Returns @@ -105,52 +105,64 @@ status: CollectionStatus; ### Examples ```ts -// Basic query with object syntax -const { data, isLoading } = useLiveQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) -) +// Prefer config object syntax +const { data, isLoading } = useLiveQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) +}) ``` ```ts // Single result query -const { data } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => eq(todos.id, 1)) .findOne() -) +}) ``` ```ts -// With dependencies that trigger re-execution -const { data, state } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +// Structured captured values are included in derived query identity +const { data, state } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => gt(todos.priority, minPriority)), - [minPriority] // Re-run when minPriority changes -) +}) +``` + +```ts +// Return undefined or null to disable a query +const { data, isEnabled } = useLiveQuery({ + query: (q) => { + if (!userId) return undefined + return q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.userId, userId)) + }, +}) ``` ```ts // Join pattern -const { data } = useLiveQuery((q) => - q.from({ issues: issueCollection }) - .join({ persons: personCollection }, ({ issues, persons }) => - eq(issues.userId, persons.id) - ) - .select(({ issues, persons }) => ({ - id: issues.id, - title: issues.title, - userName: persons.name - })) -) +const { data } = useLiveQuery({ + query: (q) => + q.from({ issues: issueCollection }) + .join({ persons: personCollection }, ({ issues, persons }) => + eq(issues.userId, persons.id) + ) + .select(({ issues, persons }) => ({ + id: issues.id, + title: issues.title, + userName: persons.name + })) +}) ``` ```ts // Handle loading and error states -const { data, isLoading, isError, status } = useLiveQuery((q) => - q.from({ todos: todoCollection }) -) +const { data, isLoading, isError, status } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }) +}) if (isLoading) return
    Loading...
    if (isError) return
    Error: {status}
    @@ -168,9 +180,9 @@ return ( function useLiveQuery(queryFn, deps?): object; ``` -Defined in: [useLiveQuery.ts:101](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L101) +Defined in: [useLiveQuery.ts:377](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L377) -Create a live query using a query function +Create a live query using a query function. ### Type Parameters @@ -190,7 +202,7 @@ Query function that defines what data to fetch `unknown`[] -Array of dependencies that trigger query re-execution when changed +Deprecated array of dependencies that trigger query re-execution when changed ### Returns @@ -266,52 +278,64 @@ status: UseLiveQueryStatus; ### Examples ```ts -// Basic query with object syntax -const { data, isLoading } = useLiveQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) -) +// Prefer config object syntax +const { data, isLoading } = useLiveQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) +}) ``` ```ts // Single result query -const { data } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => eq(todos.id, 1)) .findOne() -) +}) ``` ```ts -// With dependencies that trigger re-execution -const { data, state } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +// Structured captured values are included in derived query identity +const { data, state } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => gt(todos.priority, minPriority)), - [minPriority] // Re-run when minPriority changes -) +}) +``` + +```ts +// Return undefined or null to disable a query +const { data, isEnabled } = useLiveQuery({ + query: (q) => { + if (!userId) return undefined + return q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.userId, userId)) + }, +}) ``` ```ts // Join pattern -const { data } = useLiveQuery((q) => - q.from({ issues: issueCollection }) - .join({ persons: personCollection }, ({ issues, persons }) => - eq(issues.userId, persons.id) - ) - .select(({ issues, persons }) => ({ - id: issues.id, - title: issues.title, - userName: persons.name - })) -) +const { data } = useLiveQuery({ + query: (q) => + q.from({ issues: issueCollection }) + .join({ persons: personCollection }, ({ issues, persons }) => + eq(issues.userId, persons.id) + ) + .select(({ issues, persons }) => ({ + id: issues.id, + title: issues.title, + userName: persons.name + })) +}) ``` ```ts // Handle loading and error states -const { data, isLoading, isError, status } = useLiveQuery((q) => - q.from({ todos: todoCollection }) -) +const { data, isLoading, isError, status } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }) +}) if (isLoading) return
    Loading...
    if (isError) return
    Error: {status}
    @@ -329,9 +353,9 @@ return ( function useLiveQuery(queryFn, deps?): object; ``` -Defined in: [useLiveQuery.ts:120](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L120) +Defined in: [useLiveQuery.ts:396](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L396) -Create a live query using a query function +Create a live query using a query function. ### Type Parameters @@ -354,7 +378,7 @@ Query function that defines what data to fetch `unknown`[] -Array of dependencies that trigger query re-execution when changed +Deprecated array of dependencies that trigger query re-execution when changed ### Returns @@ -430,52 +454,64 @@ status: UseLiveQueryStatus; ### Examples ```ts -// Basic query with object syntax -const { data, isLoading } = useLiveQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) -) +// Prefer config object syntax +const { data, isLoading } = useLiveQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) +}) ``` ```ts // Single result query -const { data } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => eq(todos.id, 1)) .findOne() -) +}) ``` ```ts -// With dependencies that trigger re-execution -const { data, state } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +// Structured captured values are included in derived query identity +const { data, state } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => gt(todos.priority, minPriority)), - [minPriority] // Re-run when minPriority changes -) +}) +``` + +```ts +// Return undefined or null to disable a query +const { data, isEnabled } = useLiveQuery({ + query: (q) => { + if (!userId) return undefined + return q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.userId, userId)) + }, +}) ``` ```ts // Join pattern -const { data } = useLiveQuery((q) => - q.from({ issues: issueCollection }) - .join({ persons: personCollection }, ({ issues, persons }) => - eq(issues.userId, persons.id) - ) - .select(({ issues, persons }) => ({ - id: issues.id, - title: issues.title, - userName: persons.name - })) -) +const { data } = useLiveQuery({ + query: (q) => + q.from({ issues: issueCollection }) + .join({ persons: personCollection }, ({ issues, persons }) => + eq(issues.userId, persons.id) + ) + .select(({ issues, persons }) => ({ + id: issues.id, + title: issues.title, + userName: persons.name + })) +}) ``` ```ts // Handle loading and error states -const { data, isLoading, isError, status } = useLiveQuery((q) => - q.from({ todos: todoCollection }) -) +const { data, isLoading, isError, status } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }) +}) if (isLoading) return
    Loading...
    if (isError) return
    Error: {status}
    @@ -493,9 +529,9 @@ return ( function useLiveQuery(queryFn, deps?): object; ``` -Defined in: [useLiveQuery.ts:139](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L139) +Defined in: [useLiveQuery.ts:415](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L415) -Create a live query using a query function +Create a live query using a query function. ### Type Parameters @@ -526,7 +562,7 @@ Query function that defines what data to fetch `unknown`[] -Array of dependencies that trigger query re-execution when changed +Deprecated array of dependencies that trigger query re-execution when changed ### Returns @@ -599,52 +635,64 @@ status: UseLiveQueryStatus; ### Examples ```ts -// Basic query with object syntax -const { data, isLoading } = useLiveQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) -) +// Prefer config object syntax +const { data, isLoading } = useLiveQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) +}) ``` ```ts // Single result query -const { data } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => eq(todos.id, 1)) .findOne() -) +}) ``` ```ts -// With dependencies that trigger re-execution -const { data, state } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +// Structured captured values are included in derived query identity +const { data, state } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => gt(todos.priority, minPriority)), - [minPriority] // Re-run when minPriority changes -) +}) +``` + +```ts +// Return undefined or null to disable a query +const { data, isEnabled } = useLiveQuery({ + query: (q) => { + if (!userId) return undefined + return q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.userId, userId)) + }, +}) ``` ```ts // Join pattern -const { data } = useLiveQuery((q) => - q.from({ issues: issueCollection }) - .join({ persons: personCollection }, ({ issues, persons }) => - eq(issues.userId, persons.id) - ) - .select(({ issues, persons }) => ({ - id: issues.id, - title: issues.title, - userName: persons.name - })) -) +const { data } = useLiveQuery({ + query: (q) => + q.from({ issues: issueCollection }) + .join({ persons: personCollection }, ({ issues, persons }) => + eq(issues.userId, persons.id) + ) + .select(({ issues, persons }) => ({ + id: issues.id, + title: issues.title, + userName: persons.name + })) +}) ``` ```ts // Handle loading and error states -const { data, isLoading, isError, status } = useLiveQuery((q) => - q.from({ todos: todoCollection }) -) +const { data, isLoading, isError, status } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }) +}) if (isLoading) return
    Loading...
    if (isError) return
    Error: {status}
    @@ -662,9 +710,9 @@ return ( function useLiveQuery(queryFn, deps?): object; ``` -Defined in: [useLiveQuery.ts:162](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L162) +Defined in: [useLiveQuery.ts:438](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L438) -Create a live query using a query function +Create a live query using a query function. ### Type Parameters @@ -701,7 +749,7 @@ Query function that defines what data to fetch `unknown`[] -Array of dependencies that trigger query re-execution when changed +Deprecated array of dependencies that trigger query re-execution when changed ### Returns @@ -779,52 +827,64 @@ status: UseLiveQueryStatus; ### Examples ```ts -// Basic query with object syntax -const { data, isLoading } = useLiveQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) -) +// Prefer config object syntax +const { data, isLoading } = useLiveQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) +}) ``` ```ts // Single result query -const { data } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => eq(todos.id, 1)) .findOne() -) +}) ``` ```ts -// With dependencies that trigger re-execution -const { data, state } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +// Structured captured values are included in derived query identity +const { data, state } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => gt(todos.priority, minPriority)), - [minPriority] // Re-run when minPriority changes -) +}) +``` + +```ts +// Return undefined or null to disable a query +const { data, isEnabled } = useLiveQuery({ + query: (q) => { + if (!userId) return undefined + return q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.userId, userId)) + }, +}) ``` ```ts // Join pattern -const { data } = useLiveQuery((q) => - q.from({ issues: issueCollection }) - .join({ persons: personCollection }, ({ issues, persons }) => - eq(issues.userId, persons.id) - ) - .select(({ issues, persons }) => ({ - id: issues.id, - title: issues.title, - userName: persons.name - })) -) +const { data } = useLiveQuery({ + query: (q) => + q.from({ issues: issueCollection }) + .join({ persons: personCollection }, ({ issues, persons }) => + eq(issues.userId, persons.id) + ) + .select(({ issues, persons }) => ({ + id: issues.id, + title: issues.title, + userName: persons.name + })) +}) ``` ```ts // Handle loading and error states -const { data, isLoading, isError, status } = useLiveQuery((q) => - q.from({ todos: todoCollection }) -) +const { data, isLoading, isError, status } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }) +}) if (isLoading) return
    Loading...
    if (isError) return
    Error: {status}
    @@ -839,10 +899,10 @@ return ( ## Call Signature ```ts -function useLiveQuery(config, deps?): object; +function useLiveQuery(config): object; ``` -Defined in: [useLiveQuery.ts:230](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L230) +Defined in: [useLiveQuery.ts:508](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L508) Create a live query using configuration object @@ -856,16 +916,10 @@ Create a live query using configuration object #### config -`LiveQueryCollectionConfig`\<`TContext`\> +[`UseLiveQueryConfig`](../type-aliases/UseLiveQueryConfig.md)\<`TContext`\> Configuration object with query and options -#### deps? - -`unknown`[] - -Array of dependencies that trigger query re-execution when changed - ### Returns `object` @@ -950,7 +1004,9 @@ const queryBuilder = new Query() .where(({ persons }) => gt(persons.age, 30)) .select(({ persons }) => ({ id: persons.id, name: persons.name })) -const { data, isReady } = useLiveQuery({ query: queryBuilder }) +const { data, isReady } = useLiveQuery({ + query: queryBuilder, +}) ``` ```ts @@ -968,11 +1024,514 @@ return
    {data.length} items loaded
    ## Call Signature +```ts +function useLiveQuery(config): object; +``` + +Defined in: [useLiveQuery.ts:524](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L524) + +Create a live query using a query function. + +### Type Parameters + +#### TContext + +`TContext` *extends* `Context` + +### Parameters + +#### config + +[`ConditionalUseLiveQueryConfig`](../type-aliases/ConditionalUseLiveQueryConfig.md)\<`TContext`\> + +### Returns + +`object` + +Object with reactive data, state, and status information + +#### collection + +```ts +collection: + | Collection<{ [K in string | number | symbol]: ResultValue[K] }, string | number, { +}, StandardSchemaV1, { [K in string | number | symbol]: ResultValue[K] }> + | undefined; +``` + +#### data + +```ts +data: InferResultType | undefined; +``` + +#### isCleanedUp + +```ts +isCleanedUp: boolean; +``` + +#### isEnabled + +```ts +isEnabled: boolean; +``` + +#### isError + +```ts +isError: boolean; +``` + +#### isIdle + +```ts +isIdle: boolean; +``` + +#### isLoading + +```ts +isLoading: boolean; +``` + +#### isReady + +```ts +isReady: boolean; +``` + +#### state + +```ts +state: + | Map[K] }> + | undefined; +``` + +#### status + +```ts +status: UseLiveQueryStatus; +``` + +### Examples + +```ts +// Prefer config object syntax +const { data, isLoading } = useLiveQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) +}) +``` + +```ts +// Single result query +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.id, 1)) + .findOne() +}) +``` + +```ts +// Structured captured values are included in derived query identity +const { data, state } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) + .where(({ todos }) => gt(todos.priority, minPriority)), +}) +``` + +```ts +// Return undefined or null to disable a query +const { data, isEnabled } = useLiveQuery({ + query: (q) => { + if (!userId) return undefined + return q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.userId, userId)) + }, +}) +``` + +```ts +// Join pattern +const { data } = useLiveQuery({ + query: (q) => + q.from({ issues: issueCollection }) + .join({ persons: personCollection }, ({ issues, persons }) => + eq(issues.userId, persons.id) + ) + .select(({ issues, persons }) => ({ + id: issues.id, + title: issues.title, + userName: persons.name + })) +}) +``` + +```ts +// Handle loading and error states +const { data, isLoading, isError, status } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }) +}) + +if (isLoading) return
    Loading...
    +if (isError) return
    Error: {status}
    + +return ( +
      + {data.map(todo =>
    • {todo.text}
      • )} +
    +) +``` + +## Call Signature + +```ts +function useLiveQuery(config, deps?): object; +``` + +Defined in: [useLiveQuery.ts:540](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L540) + +Create a live query using a query function. + +### Type Parameters + +#### TContext + +`TContext` *extends* `Context` + +### Parameters + +#### config + +`LiveQueryCollectionConfig`\<`TContext`\> + +#### deps? + +`unknown`[] + +Deprecated array of dependencies that trigger query re-execution when changed + +### Returns + +`object` + +Object with reactive data, state, and status information + +#### collection + +```ts +collection: Collection<{ [K in string | number | symbol]: ResultValue[K] }, string | number, { +}>; +``` + +#### data + +```ts +data: InferResultType; +``` + +#### isCleanedUp + +```ts +isCleanedUp: boolean; +``` + +#### isEnabled + +```ts +isEnabled: true; +``` + +#### isError + +```ts +isError: boolean; +``` + +#### isIdle + +```ts +isIdle: boolean; +``` + +#### isLoading + +```ts +isLoading: boolean; +``` + +#### isReady + +```ts +isReady: boolean; +``` + +#### state + +```ts +state: Map[K] }>; +``` + +#### status + +```ts +status: CollectionStatus; +``` + +### Examples + +```ts +// Prefer config object syntax +const { data, isLoading } = useLiveQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) +}) +``` + +```ts +// Single result query +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.id, 1)) + .findOne() +}) +``` + +```ts +// Structured captured values are included in derived query identity +const { data, state } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) + .where(({ todos }) => gt(todos.priority, minPriority)), +}) +``` + +```ts +// Return undefined or null to disable a query +const { data, isEnabled } = useLiveQuery({ + query: (q) => { + if (!userId) return undefined + return q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.userId, userId)) + }, +}) +``` + +```ts +// Join pattern +const { data } = useLiveQuery({ + query: (q) => + q.from({ issues: issueCollection }) + .join({ persons: personCollection }, ({ issues, persons }) => + eq(issues.userId, persons.id) + ) + .select(({ issues, persons }) => ({ + id: issues.id, + title: issues.title, + userName: persons.name + })) +}) +``` + +```ts +// Handle loading and error states +const { data, isLoading, isError, status } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }) +}) + +if (isLoading) return
    Loading...
    +if (isError) return
    Error: {status}
    + +return ( +
      + {data.map(todo =>
    • {todo.text}
      • )} +
    +) +``` + +## Call Signature + +```ts +function useLiveQuery(config, deps): object; +``` + +Defined in: [useLiveQuery.ts:557](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L557) + +Create a live query using a query function. + +### Type Parameters + +#### TContext + +`TContext` *extends* `Context` + +### Parameters + +#### config + +[`ConditionalUseLiveQueryConfig`](../type-aliases/ConditionalUseLiveQueryConfig.md)\<`TContext`\> + +#### deps + +`unknown`[] + +Deprecated array of dependencies that trigger query re-execution when changed + +### Returns + +`object` + +Object with reactive data, state, and status information + +#### collection + +```ts +collection: + | Collection<{ [K in string | number | symbol]: ResultValue[K] }, string | number, { +}, StandardSchemaV1, { [K in string | number | symbol]: ResultValue[K] }> + | undefined; +``` + +#### data + +```ts +data: InferResultType | undefined; +``` + +#### isCleanedUp + +```ts +isCleanedUp: boolean; +``` + +#### isEnabled + +```ts +isEnabled: boolean; +``` + +#### isError + +```ts +isError: boolean; +``` + +#### isIdle + +```ts +isIdle: boolean; +``` + +#### isLoading + +```ts +isLoading: boolean; +``` + +#### isReady + +```ts +isReady: boolean; +``` + +#### state + +```ts +state: + | Map[K] }> + | undefined; +``` + +#### status + +```ts +status: UseLiveQueryStatus; +``` + +### Examples + +```ts +// Prefer config object syntax +const { data, isLoading } = useLiveQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) +}) +``` + +```ts +// Single result query +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.id, 1)) + .findOne() +}) +``` + +```ts +// Structured captured values are included in derived query identity +const { data, state } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) + .where(({ todos }) => gt(todos.priority, minPriority)), +}) +``` + +```ts +// Return undefined or null to disable a query +const { data, isEnabled } = useLiveQuery({ + query: (q) => { + if (!userId) return undefined + return q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.userId, userId)) + }, +}) +``` + +```ts +// Join pattern +const { data } = useLiveQuery({ + query: (q) => + q.from({ issues: issueCollection }) + .join({ persons: personCollection }, ({ issues, persons }) => + eq(issues.userId, persons.id) + ) + .select(({ issues, persons }) => ({ + id: issues.id, + title: issues.title, + userName: persons.name + })) +}) +``` + +```ts +// Handle loading and error states +const { data, isLoading, isError, status } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }) +}) + +if (isLoading) return
    Loading...
    +if (isError) return
    Error: {status}
    + +return ( +
      + {data.map(todo =>
    • {todo.text}
      • )} +
    +) +``` + +## Call Signature + ```ts function useLiveQuery(liveQueryCollection): object; ``` -Defined in: [useLiveQuery.ts:276](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L276) +Defined in: [useLiveQuery.ts:603](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L603) Subscribe to an existing live query collection @@ -1100,9 +1659,9 @@ return
    {data.map(item => )}
    function useLiveQuery(liveQueryCollection): object; ``` -Defined in: [useLiveQuery.ts:296](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L296) +Defined in: [useLiveQuery.ts:623](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L623) -Create a live query using a query function +Create a live query using a query function. ### Type Parameters @@ -1193,52 +1752,64 @@ status: CollectionStatus; ### Examples ```ts -// Basic query with object syntax -const { data, isLoading } = useLiveQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) -) +// Prefer config object syntax +const { data, isLoading } = useLiveQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) +}) ``` ```ts // Single result query -const { data } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => eq(todos.id, 1)) .findOne() -) +}) ``` ```ts -// With dependencies that trigger re-execution -const { data, state } = useLiveQuery( - (q) => q.from({ todos: todosCollection }) +// Structured captured values are included in derived query identity +const { data, state } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => gt(todos.priority, minPriority)), - [minPriority] // Re-run when minPriority changes -) +}) +``` + +```ts +// Return undefined or null to disable a query +const { data, isEnabled } = useLiveQuery({ + query: (q) => { + if (!userId) return undefined + return q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.userId, userId)) + }, +}) ``` ```ts // Join pattern -const { data } = useLiveQuery((q) => - q.from({ issues: issueCollection }) - .join({ persons: personCollection }, ({ issues, persons }) => - eq(issues.userId, persons.id) - ) - .select(({ issues, persons }) => ({ - id: issues.id, - title: issues.title, - userName: persons.name - })) -) +const { data } = useLiveQuery({ + query: (q) => + q.from({ issues: issueCollection }) + .join({ persons: personCollection }, ({ issues, persons }) => + eq(issues.userId, persons.id) + ) + .select(({ issues, persons }) => ({ + id: issues.id, + title: issues.title, + userName: persons.name + })) +}) ``` ```ts // Handle loading and error states -const { data, isLoading, isError, status } = useLiveQuery((q) => - q.from({ todos: todoCollection }) -) +const { data, isLoading, isError, status } = useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }) +}) if (isLoading) return
    Loading...
    if (isError) return
    Error: {status}
    diff --git a/docs/framework/react/reference/functions/useLiveSuspenseQuery.md b/docs/framework/react/reference/functions/useLiveSuspenseQuery.md index 83ae3a93ee..1101f9406e 100644 --- a/docs/framework/react/reference/functions/useLiveSuspenseQuery.md +++ b/docs/framework/react/reference/functions/useLiveSuspenseQuery.md @@ -11,7 +11,7 @@ title: useLiveSuspenseQuery function useLiveSuspenseQuery(queryFn, deps?): object; ``` -Defined in: [useLiveSuspenseQuery.ts:109](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveSuspenseQuery.ts#L109) +Defined in: [useLiveSuspenseQuery.ts:110](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveSuspenseQuery.ts#L110) Create a live query with React Suspense support @@ -33,7 +33,7 @@ Query function that defines what data to fetch `unknown`[] -Array of dependencies that trigger query re-execution when changed +Deprecated array of dependencies that trigger query re-execution when changed ### Returns @@ -73,11 +73,12 @@ Error when collection fails (caught by Error boundary) ```ts // Basic usage with Suspense function TodoList() { - const { data } = useLiveSuspenseQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) + }) return (
      @@ -106,12 +107,156 @@ const { data } = useLiveSuspenseQuery( ``` ```ts -// With dependencies that trigger re-suspension +// Structured captured values are included in derived query identity and trigger re-suspension +const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ todos: todosCollection }) + .where(({ todos }) => gt(todos.priority, minPriority)), +}) +``` + +```ts +// With Error boundary +function App() { + return ( + Error loading data}> + Loading...}> + + + + ) +} +``` + +### Remarks + +**Important:** This hook does NOT support disabled queries (returning undefined/null). +Following TanStack Query's useSuspenseQuery design, the query callback must always +return a valid query, collection, or config object. + +❌ **This will cause a type error:** +```ts +useLiveSuspenseQuery( + (q) => userId ? q.from({ users }) : undefined // ❌ Error! +) +``` + +✅ **Use conditional rendering instead:** +```ts +function Profile({ userId }: { userId: string }) { + const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ users }).where(({ users }) => eq(users.id, userId)), + }) + return
      {data.name}
      +} + +// In parent component: +{userId ? :
      No user
      } +``` + +✅ **For optional inputs, conditionally render a component with complete query inputs:** +```ts +{userId ? :
      No user
      } +``` + +## Call Signature + +```ts +function useLiveSuspenseQuery(config): object; +``` + +Defined in: [useLiveSuspenseQuery.ts:120](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveSuspenseQuery.ts#L120) + +Create a live query with React Suspense support + +### Type Parameters + +#### TContext + +`TContext` *extends* `Context` + +### Parameters + +#### config + +[`UseLiveQueryConfig`](../type-aliases/UseLiveQueryConfig.md)\<`TContext`\> + +### Returns + +`object` + +Object with reactive data and state - data is guaranteed to be defined + +#### collection + +```ts +collection: Collection<{ [K in string | number | symbol]: ResultValue[K] }, string | number, { +}>; +``` + +#### data + +```ts +data: InferResultType; +``` + +#### state + +```ts +state: Map[K] }>; +``` + +### Throws + +Promise when data is loading (caught by Suspense boundary) + +### Throws + +Error when collection fails (caught by Error boundary) + +### Examples + +```ts +// Basic usage with Suspense +function TodoList() { + const { data } = useLiveSuspenseQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) + }) + + return ( +
        + {data.map(todo =>
      • {todo.text}
        • )} +
      + ) +} + +function App() { + return ( + Loading...}> + + + ) +} +``` + +```ts +// Single result query const { data } = useLiveSuspenseQuery( (q) => q.from({ todos: todosCollection }) - .where(({ todos }) => gt(todos.priority, minPriority)), - [minPriority] // Re-suspends when minPriority changes + .where(({ todos }) => eq(todos.id, 1)) + .findOne() ) +// data is guaranteed to be the single item (or undefined if not found) +``` + +```ts +// Structured captured values are included in derived query identity and trigger re-suspension +const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ todos: todosCollection }) + .where(({ todos }) => gt(todos.priority, minPriority)), +}) ``` ```ts @@ -143,9 +288,9 @@ useLiveSuspenseQuery( ✅ **Use conditional rendering instead:** ```ts function Profile({ userId }: { userId: string }) { - const { data } = useLiveSuspenseQuery( - (q) => q.from({ users }).where(({ users }) => eq(users.id, userId)) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ users }).where(({ users }) => eq(users.id, userId)), + }) return
      {data.name}
      } @@ -153,12 +298,9 @@ function Profile({ userId }: { userId: string }) { {userId ? :
      No user
      } ``` -✅ **Or use useLiveQuery for conditional queries:** +✅ **For optional inputs, conditionally render a component with complete query inputs:** ```ts -const { data, isEnabled } = useLiveQuery( - (q) => userId ? q.from({ users }) : undefined, // ✅ Supported! - [userId] -) +{userId ? :
      No user
      } ``` ## Call Signature @@ -167,7 +309,7 @@ const { data, isEnabled } = useLiveQuery( function useLiveSuspenseQuery(config, deps?): object; ``` -Defined in: [useLiveSuspenseQuery.ts:119](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveSuspenseQuery.ts#L119) +Defined in: [useLiveSuspenseQuery.ts:129](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveSuspenseQuery.ts#L129) Create a live query with React Suspense support @@ -187,7 +329,7 @@ Create a live query with React Suspense support `unknown`[] -Array of dependencies that trigger query re-execution when changed +Deprecated array of dependencies that trigger query re-execution when changed ### Returns @@ -227,11 +369,12 @@ Error when collection fails (caught by Error boundary) ```ts // Basic usage with Suspense function TodoList() { - const { data } = useLiveSuspenseQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) + }) return (
        @@ -260,12 +403,11 @@ const { data } = useLiveSuspenseQuery( ``` ```ts -// With dependencies that trigger re-suspension -const { data } = useLiveSuspenseQuery( - (q) => q.from({ todos: todosCollection }) +// Structured captured values are included in derived query identity and trigger re-suspension +const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => gt(todos.priority, minPriority)), - [minPriority] // Re-suspends when minPriority changes -) +}) ``` ```ts @@ -297,9 +439,9 @@ useLiveSuspenseQuery( ✅ **Use conditional rendering instead:** ```ts function Profile({ userId }: { userId: string }) { - const { data } = useLiveSuspenseQuery( - (q) => q.from({ users }).where(({ users }) => eq(users.id, userId)) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ users }).where(({ users }) => eq(users.id, userId)), + }) return
        {data.name}
        } @@ -307,12 +449,9 @@ function Profile({ userId }: { userId: string }) { {userId ? :
        No user
        } ``` -✅ **Or use useLiveQuery for conditional queries:** +✅ **For optional inputs, conditionally render a component with complete query inputs:** ```ts -const { data, isEnabled } = useLiveQuery( - (q) => userId ? q.from({ users }) : undefined, // ✅ Supported! - [userId] -) +{userId ? :
        No user
        } ``` ## Call Signature @@ -321,7 +460,7 @@ const { data, isEnabled } = useLiveQuery( function useLiveSuspenseQuery(liveQueryCollection): object; ``` -Defined in: [useLiveSuspenseQuery.ts:129](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveSuspenseQuery.ts#L129) +Defined in: [useLiveSuspenseQuery.ts:139](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveSuspenseQuery.ts#L139) Create a live query with React Suspense support @@ -382,11 +521,12 @@ Error when collection fails (caught by Error boundary) ```ts // Basic usage with Suspense function TodoList() { - const { data } = useLiveSuspenseQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) + }) return (
          @@ -415,12 +555,11 @@ const { data } = useLiveSuspenseQuery( ``` ```ts -// With dependencies that trigger re-suspension -const { data } = useLiveSuspenseQuery( - (q) => q.from({ todos: todosCollection }) +// Structured captured values are included in derived query identity and trigger re-suspension +const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => gt(todos.priority, minPriority)), - [minPriority] // Re-suspends when minPriority changes -) +}) ``` ```ts @@ -452,9 +591,9 @@ useLiveSuspenseQuery( ✅ **Use conditional rendering instead:** ```ts function Profile({ userId }: { userId: string }) { - const { data } = useLiveSuspenseQuery( - (q) => q.from({ users }).where(({ users }) => eq(users.id, userId)) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ users }).where(({ users }) => eq(users.id, userId)), + }) return
          {data.name}
          } @@ -462,12 +601,9 @@ function Profile({ userId }: { userId: string }) { {userId ? :
          No user
          } ``` -✅ **Or use useLiveQuery for conditional queries:** +✅ **For optional inputs, conditionally render a component with complete query inputs:** ```ts -const { data, isEnabled } = useLiveQuery( - (q) => userId ? q.from({ users }) : undefined, // ✅ Supported! - [userId] -) +{userId ? :
          No user
          } ``` ## Call Signature @@ -476,7 +612,7 @@ const { data, isEnabled } = useLiveQuery( function useLiveSuspenseQuery(liveQueryCollection): object; ``` -Defined in: [useLiveSuspenseQuery.ts:142](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveSuspenseQuery.ts#L142) +Defined in: [useLiveSuspenseQuery.ts:152](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveSuspenseQuery.ts#L152) Create a live query with React Suspense support @@ -537,11 +673,12 @@ Error when collection fails (caught by Error boundary) ```ts // Basic usage with Suspense function TodoList() { - const { data } = useLiveSuspenseQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) + }) return (
            @@ -570,12 +707,11 @@ const { data } = useLiveSuspenseQuery( ``` ```ts -// With dependencies that trigger re-suspension -const { data } = useLiveSuspenseQuery( - (q) => q.from({ todos: todosCollection }) +// Structured captured values are included in derived query identity and trigger re-suspension +const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ todos: todosCollection }) .where(({ todos }) => gt(todos.priority, minPriority)), - [minPriority] // Re-suspends when minPriority changes -) +}) ``` ```ts @@ -607,9 +743,9 @@ useLiveSuspenseQuery( ✅ **Use conditional rendering instead:** ```ts function Profile({ userId }: { userId: string }) { - const { data } = useLiveSuspenseQuery( - (q) => q.from({ users }).where(({ users }) => eq(users.id, userId)) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ users }).where(({ users }) => eq(users.id, userId)), + }) return
            {data.name}
            } @@ -617,10 +753,7 @@ function Profile({ userId }: { userId: string }) { {userId ? :
            No user
            } ``` -✅ **Or use useLiveQuery for conditional queries:** +✅ **For optional inputs, conditionally render a component with complete query inputs:** ```ts -const { data, isEnabled } = useLiveQuery( - (q) => userId ? q.from({ users }) : undefined, // ✅ Supported! - [userId] -) +{userId ? :
            No user
            } ``` diff --git a/docs/framework/react/reference/functions/useOptionalDbClient.md b/docs/framework/react/reference/functions/useOptionalDbClient.md new file mode 100644 index 0000000000..77361bf284 --- /dev/null +++ b/docs/framework/react/reference/functions/useOptionalDbClient.md @@ -0,0 +1,16 @@ +--- +id: useOptionalDbClient +title: useOptionalDbClient +--- + +# Function: useOptionalDbClient() + +```ts +function useOptionalDbClient(): DbClient | undefined; +``` + +Defined in: [DbProvider.tsx:30](https://github.com/TanStack/db/blob/main/packages/react-db/src/DbProvider.tsx#L30) + +## Returns + +`DbClient` \| `undefined` diff --git a/docs/framework/react/reference/index.md b/docs/framework/react/reference/index.md index 07d892ce4a..5c2d783be2 100644 --- a/docs/framework/react/reference/index.md +++ b/docs/framework/react/reference/index.md @@ -7,14 +7,23 @@ title: "@tanstack/react-db" ## Type Aliases +- [ConditionalUseLiveQueryConfig](type-aliases/ConditionalUseLiveQueryConfig.md) +- [DbProviderProps](type-aliases/DbProviderProps.md) +- [HydrationBoundaryProps](type-aliases/HydrationBoundaryProps.md) +- [LiveQueryKey](type-aliases/LiveQueryKey.md) - [UseLiveInfiniteQueryConfig](type-aliases/UseLiveInfiniteQueryConfig.md) - [UseLiveInfiniteQueryReturn](type-aliases/UseLiveInfiniteQueryReturn.md) +- [UseLiveQueryConfig](type-aliases/UseLiveQueryConfig.md) - [UseLiveQueryStatus](type-aliases/UseLiveQueryStatus.md) ## Functions +- [DbProvider](functions/DbProvider.md) +- [HydrationBoundary](functions/HydrationBoundary.md) +- [useDbClient](functions/useDbClient.md) - [useLiveInfiniteQuery](functions/useLiveInfiniteQuery.md) - [useLiveQuery](functions/useLiveQuery.md) - [useLiveQueryEffect](functions/useLiveQueryEffect.md) - [useLiveSuspenseQuery](functions/useLiveSuspenseQuery.md) +- [useOptionalDbClient](functions/useOptionalDbClient.md) - [usePacedMutations](functions/usePacedMutations.md) diff --git a/docs/framework/react/reference/type-aliases/ConditionalUseLiveQueryConfig.md b/docs/framework/react/reference/type-aliases/ConditionalUseLiveQueryConfig.md new file mode 100644 index 0000000000..3a6054c9ce --- /dev/null +++ b/docs/framework/react/reference/type-aliases/ConditionalUseLiveQueryConfig.md @@ -0,0 +1,28 @@ +--- +id: ConditionalUseLiveQueryConfig +title: ConditionalUseLiveQueryConfig +--- + +# Type Alias: ConditionalUseLiveQueryConfig\ + +```ts +type ConditionalUseLiveQueryConfig = UseLiveQueryConfigOptions & object; +``` + +Defined in: [useLiveQuery.ts:74](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L74) + +## Type Declaration + +### query + +```ts +query: + | ConfiguredQueryBuilder + | (q) => ConfiguredQueryBuilder | undefined | null; +``` + +## Type Parameters + +### TContext + +`TContext` *extends* `Context` diff --git a/docs/framework/react/reference/type-aliases/DbProviderProps.md b/docs/framework/react/reference/type-aliases/DbProviderProps.md new file mode 100644 index 0000000000..b79d882972 --- /dev/null +++ b/docs/framework/react/reference/type-aliases/DbProviderProps.md @@ -0,0 +1,32 @@ +--- +id: DbProviderProps +title: DbProviderProps +--- + +# Type Alias: DbProviderProps + +```ts +type DbProviderProps = object; +``` + +Defined in: [DbProvider.tsx:9](https://github.com/TanStack/db/blob/main/packages/react-db/src/DbProvider.tsx#L9) + +## Properties + +### children? + +```ts +optional children: ReactNode; +``` + +Defined in: [DbProvider.tsx:11](https://github.com/TanStack/db/blob/main/packages/react-db/src/DbProvider.tsx#L11) + +*** + +### client + +```ts +client: DbClient; +``` + +Defined in: [DbProvider.tsx:10](https://github.com/TanStack/db/blob/main/packages/react-db/src/DbProvider.tsx#L10) diff --git a/docs/framework/react/reference/type-aliases/HydrationBoundaryProps.md b/docs/framework/react/reference/type-aliases/HydrationBoundaryProps.md new file mode 100644 index 0000000000..cc99be8716 --- /dev/null +++ b/docs/framework/react/reference/type-aliases/HydrationBoundaryProps.md @@ -0,0 +1,32 @@ +--- +id: HydrationBoundaryProps +title: HydrationBoundaryProps +--- + +# Type Alias: HydrationBoundaryProps + +```ts +type HydrationBoundaryProps = object; +``` + +Defined in: [HydrationBoundary.tsx:8](https://github.com/TanStack/db/blob/main/packages/react-db/src/HydrationBoundary.tsx#L8) + +## Properties + +### children? + +```ts +optional children: ReactNode; +``` + +Defined in: [HydrationBoundary.tsx:10](https://github.com/TanStack/db/blob/main/packages/react-db/src/HydrationBoundary.tsx#L10) + +*** + +### state + +```ts +state: DehydratedDbState; +``` + +Defined in: [HydrationBoundary.tsx:9](https://github.com/TanStack/db/blob/main/packages/react-db/src/HydrationBoundary.tsx#L9) diff --git a/docs/framework/react/reference/type-aliases/LiveQueryKey.md b/docs/framework/react/reference/type-aliases/LiveQueryKey.md new file mode 100644 index 0000000000..5f435998ad --- /dev/null +++ b/docs/framework/react/reference/type-aliases/LiveQueryKey.md @@ -0,0 +1,12 @@ +--- +id: LiveQueryKey +title: LiveQueryKey +--- + +# Type Alias: LiveQueryKey + +```ts +type LiveQueryKey = ReadonlyArray; +``` + +Defined in: [useLiveQuery.ts:50](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L50) diff --git a/docs/framework/react/reference/type-aliases/UseLiveInfiniteQueryConfig.md b/docs/framework/react/reference/type-aliases/UseLiveInfiniteQueryConfig.md index 8ec4417d3b..c4e89182f8 100644 --- a/docs/framework/react/reference/type-aliases/UseLiveInfiniteQueryConfig.md +++ b/docs/framework/react/reference/type-aliases/UseLiveInfiniteQueryConfig.md @@ -9,7 +9,7 @@ title: UseLiveInfiniteQueryConfig type UseLiveInfiniteQueryConfig = object; ``` -Defined in: [useLiveInfiniteQuery.ts:23](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L23) +Defined in: [useLiveInfiniteQuery.ts:44](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L44) ## Type Parameters @@ -19,13 +19,25 @@ Defined in: [useLiveInfiniteQuery.ts:23](https://github.com/TanStack/db/blob/mai ## Properties +### client? + +```ts +optional client: DbClient; +``` + +Defined in: [useLiveInfiniteQuery.ts:52](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L52) + +Override the nearest DbProvider for this query. + +*** + ### ~~getNextPageParam()?~~ ```ts optional getNextPageParam: (lastPage, allPages, lastPageParam, allPageParams) => number | undefined; ``` -Defined in: [useLiveInfiniteQuery.ts:31](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L31) +Defined in: [useLiveInfiniteQuery.ts:60](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L60) #### Parameters @@ -63,7 +75,7 @@ Provided for API compatibility with TanStack Query conventions. optional initialPageParam: number; ``` -Defined in: [useLiveInfiniteQuery.ts:25](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L25) +Defined in: [useLiveInfiniteQuery.ts:54](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L54) *** @@ -73,4 +85,18 @@ Defined in: [useLiveInfiniteQuery.ts:25](https://github.com/TanStack/db/blob/mai optional pageSize: number; ``` -Defined in: [useLiveInfiniteQuery.ts:24](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L24) +Defined in: [useLiveInfiniteQuery.ts:53](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L53) + +*** + +### queryKey? + +```ts +optional queryKey: LiveQueryKey; +``` + +Defined in: [useLiveInfiniteQuery.ts:50](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L50) + +Explicit identity for queries that contain opaque functional variants or +are hot enough that deriving identity from structured IR is too expensive. +Structured queries should omit this so DB can derive identity directly. diff --git a/docs/framework/react/reference/type-aliases/UseLiveInfiniteQueryReturn.md b/docs/framework/react/reference/type-aliases/UseLiveInfiniteQueryReturn.md index e1f0cfc904..7448311172 100644 --- a/docs/framework/react/reference/type-aliases/UseLiveInfiniteQueryReturn.md +++ b/docs/framework/react/reference/type-aliases/UseLiveInfiniteQueryReturn.md @@ -9,7 +9,7 @@ title: UseLiveInfiniteQueryReturn type UseLiveInfiniteQueryReturn = Omit, "data"> & object; ``` -Defined in: [useLiveInfiniteQuery.ts:39](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L39) +Defined in: [useLiveInfiniteQuery.ts:68](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveInfiniteQuery.ts#L68) ## Type Declaration @@ -19,15 +19,21 @@ Defined in: [useLiveInfiniteQuery.ts:39](https://github.com/TanStack/db/blob/mai data: InferResultType; ``` +### error + +```ts +error: unknown; +``` + ### fetchNextPage() ```ts -fetchNextPage: () => void; +fetchNextPage: () => Promise; ``` #### Returns -`void` +`Promise`\<`void`\> ### hasNextPage diff --git a/docs/framework/react/reference/type-aliases/UseLiveQueryConfig.md b/docs/framework/react/reference/type-aliases/UseLiveQueryConfig.md new file mode 100644 index 0000000000..c02c436f61 --- /dev/null +++ b/docs/framework/react/reference/type-aliases/UseLiveQueryConfig.md @@ -0,0 +1,18 @@ +--- +id: UseLiveQueryConfig +title: UseLiveQueryConfig +--- + +# Type Alias: UseLiveQueryConfig\ + +```ts +type UseLiveQueryConfig = UseLiveQueryConfigOptions & Pick, "query">; +``` + +Defined in: [useLiveQuery.ts:70](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L70) + +## Type Parameters + +### TContext + +`TContext` *extends* `Context` diff --git a/docs/framework/react/reference/type-aliases/UseLiveQueryStatus.md b/docs/framework/react/reference/type-aliases/UseLiveQueryStatus.md index 93bbcd634b..dc726fcf53 100644 --- a/docs/framework/react/reference/type-aliases/UseLiveQueryStatus.md +++ b/docs/framework/react/reference/type-aliases/UseLiveQueryStatus.md @@ -9,4 +9,4 @@ title: UseLiveQueryStatus type UseLiveQueryStatus = CollectionStatus | "disabled"; ``` -Defined in: [useLiveQuery.ts:23](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L23) +Defined in: [useLiveQuery.ts:49](https://github.com/TanStack/db/blob/main/packages/react-db/src/useLiveQuery.ts#L49) diff --git a/docs/framework/solid/reference/functions/useLiveQuery.md b/docs/framework/solid/reference/functions/useLiveQuery.md index a4e51b7927..7df1a9de16 100644 --- a/docs/framework/solid/reference/functions/useLiveQuery.md +++ b/docs/framework/solid/reference/functions/useLiveQuery.md @@ -11,7 +11,7 @@ title: useLiveQuery function useLiveQuery(queryFn): Accessor> & object; ``` -Defined in: [useLiveQuery.ts:102](https://github.com/TanStack/db/blob/main/packages/solid-db/src/useLiveQuery.ts#L102) +Defined in: [useLiveQuery.ts:103](https://github.com/TanStack/db/blob/main/packages/solid-db/src/useLiveQuery.ts#L103) Create a live query using a query function @@ -111,7 +111,7 @@ return ( function useLiveQuery(queryFn): Accessor> & object; ``` -Defined in: [useLiveQuery.ts:121](https://github.com/TanStack/db/blob/main/packages/solid-db/src/useLiveQuery.ts#L121) +Defined in: [useLiveQuery.ts:122](https://github.com/TanStack/db/blob/main/packages/solid-db/src/useLiveQuery.ts#L122) Create a live query using a query function @@ -211,7 +211,7 @@ return ( function useLiveQuery(config): Accessor> & object; ``` -Defined in: [useLiveQuery.ts:182](https://github.com/TanStack/db/blob/main/packages/solid-db/src/useLiveQuery.ts#L182) +Defined in: [useLiveQuery.ts:183](https://github.com/TanStack/db/blob/main/packages/solid-db/src/useLiveQuery.ts#L183) Create a live query using configuration object @@ -280,7 +280,7 @@ return ( function useLiveQuery(liveQueryCollection): Accessor & object; ``` -Defined in: [useLiveQuery.ts:236](https://github.com/TanStack/db/blob/main/packages/solid-db/src/useLiveQuery.ts#L236) +Defined in: [useLiveQuery.ts:237](https://github.com/TanStack/db/blob/main/packages/solid-db/src/useLiveQuery.ts#L237) Subscribe to an existing live query collection @@ -352,7 +352,7 @@ return ( function useLiveQuery(liveQueryCollection): Accessor & object; ``` -Defined in: [useLiveQuery.ts:261](https://github.com/TanStack/db/blob/main/packages/solid-db/src/useLiveQuery.ts#L261) +Defined in: [useLiveQuery.ts:262](https://github.com/TanStack/db/blob/main/packages/solid-db/src/useLiveQuery.ts#L262) Create a live query using a query function diff --git a/docs/framework/svelte/overview.md b/docs/framework/svelte/overview.md index f7497d0e7e..5eeea14c81 100644 --- a/docs/framework/svelte/overview.md +++ b/docs/framework/svelte/overview.md @@ -17,6 +17,27 @@ For comprehensive documentation on writing queries (filtering, joins, aggregatio ## Basic Usage +### DbProvider + +Use one `DbClient` for each browser app and one per server request. `DbProvider` +lets queries resolve collection descriptors against that client: + +```svelte + + + + + +``` + +See [SSR and Hydration](../../guides/ssr.md) for server preloading, +dehydration, and snapshot handoff. + ### useLiveQuery The `useLiveQuery` utility creates a live query that automatically updates your component when data changes. It returns reactive values powered by Svelte 5 runes: @@ -26,11 +47,12 @@ The `useLiveQuery` utility creates a live query that automatically updates your import { useLiveQuery } from '@tanstack/svelte-db' import { eq } from '@tanstack/db' - const query = useLiveQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) - .select(({ todos }) => ({ id: todos.id, text: todos.text })) - ) + const query = useLiveQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)) + .select(({ todos }) => ({ id: todos.id, text: todos.text })) + }) {#if query.isLoading} @@ -46,105 +68,100 @@ The `useLiveQuery` utility creates a live query that automatically updates your **Note:** With Svelte 5, `useLiveQuery` returns reactive values through getters. Access `query.data` and `query.isLoading` directly (no `$` prefix needed). -### Dependency Arrays +### useLiveInfiniteQuery -The `useLiveQuery` utility accepts an optional dependency array as its last parameter. When any value in the array changes, the query is recreated and re-executed. - -#### When to Use Dependency Arrays - -Use dependency arrays when your query depends on external reactive values (props or state): +For ordered, paginated data with live updates, use `useLiveInfiniteQuery`: ```svelte -
            {query.data.length} high-priority todos
            +{#each query.data as post (post.id)} +
            {post.title}
            +{/each} + +{#if query.hasNextPage} + +{/if} ``` -**Note:** When using props or reactive state in the query, wrap them in a function for the dependency array. +`fetchNextPage()` returns a promise that resolves after the page request settles. Failures are exposed through `query.error` and do not reject the promise. -#### What Happens When Dependencies Change +The query must include `orderBy`. The dependency array is available only with +the query-function form. You can also pass an ordered, pre-created live query +collection directly. -When a dependency value changes: -1. The previous live query collection is cleaned up -2. A new query is created with the updated values -3. The component re-renders with the new data -4. The utility shows loading state again +### Query Identity -#### Best Practices +`useLiveQuery` derives identity from structured query IR. Svelte also tracks +reactive values read while building the query, so normal builder queries do not +need a dependency array or `queryKey`. -**Include all external values used in the query:** +Captured props and state become part of the derived identity: ```svelte - -
            {query.data.length} todos
            -``` - -**Empty array for static queries:** + import { gt } from '@tanstack/db' -```svelte - -
            {query.data.length} todos
            +
            {query.data.length} high-priority todos
            ``` -**Omit the array for queries with no external dependencies:** +When the derived identity changes: +1. The previous live query collection is cleaned up +2. A new query is created with the updated values +3. The component re-renders with the new data +4. The utility shows loading state again + +Use `queryKey` for opaque functional variants such as `.fn.where`, because DB +cannot inspect their closed-over values. Pass reactive key values through a +getter: ```svelte + let search = $state('ship') -
            {query.data.length} todos
            + const query = useLiveQuery({ + queryKey: () => [todosCollection.id, 'search', search], + query: (q) => + q.from({ todos: todosCollection }) + .fn.where(({ todos }) => todos.title.includes(search)) + }) + ``` +The legacy dependency array remains supported. Prefer derived identity for +structured queries and `queryKey` for opaque ones. + ### Accessing Multiple Properties You can access all status properties directly on the query result: diff --git a/docs/framework/svelte/reference/functions/useDbClient.md b/docs/framework/svelte/reference/functions/useDbClient.md new file mode 100644 index 0000000000..867132aeba --- /dev/null +++ b/docs/framework/svelte/reference/functions/useDbClient.md @@ -0,0 +1,16 @@ +--- +id: useDbClient +title: useDbClient +--- + +# Function: useDbClient() + +```ts +function useDbClient(): DbClient; +``` + +Defined in: [packages/svelte-db/src/db-context.ts:11](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/db-context.ts#L11) + +## Returns + +`DbClient` diff --git a/docs/framework/svelte/reference/functions/useLiveInfiniteQuery.md b/docs/framework/svelte/reference/functions/useLiveInfiniteQuery.md new file mode 100644 index 0000000000..91e746fa30 --- /dev/null +++ b/docs/framework/svelte/reference/functions/useLiveInfiniteQuery.md @@ -0,0 +1,83 @@ +--- +id: useLiveInfiniteQuery +title: useLiveInfiniteQuery +--- + +# Function: useLiveInfiniteQuery() + +## Call Signature + +```ts +function useLiveInfiniteQuery(liveQueryCollection, config): UseLiveInfiniteQueryReturnWithCollection; +``` + +Defined in: [packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts:97](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts#L97) + +Create a Svelte-native reactive view over the shared live-query window +controller. The query must include an `orderBy` clause. + +### Type Parameters + +#### TResult + +`TResult` *extends* `object` + +#### TKey + +`TKey` *extends* `string` \| `number` + +#### TUtils + +`TUtils` *extends* `Record`\<`string`, `any`\> + +### Parameters + +#### liveQueryCollection + +`MaybeGetter`\<`Collection`\<`TResult`, `TKey`, `TUtils`, `StandardSchemaV1`\<`unknown`, `unknown`\>, `TResult`\> & `NonSingleResult`\> + +#### config + +[`LiveInfiniteQueryConfig`](../type-aliases/LiveInfiniteQueryConfig.md)\<`TResult`\> + +### Returns + +[`UseLiveInfiniteQueryReturnWithCollection`](../type-aliases/UseLiveInfiniteQueryReturnWithCollection.md)\<`TResult`, `TKey`, `TUtils`\> + +## Call Signature + +```ts +function useLiveInfiniteQuery( + queryFn, + config, +deps?): UseLiveInfiniteQueryReturn; +``` + +Defined in: [packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts:108](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts#L108) + +Create a Svelte-native reactive view over the shared live-query window +controller. The query must include an `orderBy` clause. + +### Type Parameters + +#### TContext + +`TContext` *extends* `Context` + +### Parameters + +#### queryFn + +(`q`) => `QueryBuilder`\<`TContext`\> + +#### config + +[`UseLiveInfiniteQueryConfig`](../type-aliases/UseLiveInfiniteQueryConfig.md)\<`TContext`\> + +#### deps? + +() => `unknown`[] + +### Returns + +[`UseLiveInfiniteQueryReturn`](../type-aliases/UseLiveInfiniteQueryReturn.md)\<`TContext`\> diff --git a/docs/framework/svelte/reference/functions/useLiveQuery.md b/docs/framework/svelte/reference/functions/useLiveQuery.md index 1ad0b46d1b..ed5f24c067 100644 --- a/docs/framework/svelte/reference/functions/useLiveQuery.md +++ b/docs/framework/svelte/reference/functions/useLiveQuery.md @@ -11,7 +11,7 @@ title: useLiveQuery function useLiveQuery(queryFn, deps?): UseLiveQueryReturn<{ [K in string | number | symbol]: ResultValue[K] }, InferResultType>; ``` -Defined in: [useLiveQuery.svelte.ts:160](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L160) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:180](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L180) Create a live query using a query function @@ -137,7 +137,7 @@ const todosQuery = useLiveQuery((q) => function useLiveQuery(queryFn, deps?): UseLiveQueryReturn<{ [K in string | number | symbol]: ResultValue[K] }, InferResultType | undefined>; ``` -Defined in: [useLiveQuery.svelte.ts:166](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L166) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:186](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L186) Create a live query using a query function @@ -263,7 +263,7 @@ const todosQuery = useLiveQuery((q) => function useLiveQuery(config, deps?): UseLiveQueryReturn<{ [K in string | number | symbol]: ResultValue[K] }, InferResultType>; ``` -Defined in: [useLiveQuery.svelte.ts:214](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L214) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:234](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L234) Create a live query using configuration object @@ -277,7 +277,7 @@ Create a live query using configuration object #### config -`LiveQueryCollectionConfig`\<`TContext`\> +[`UseLiveQueryConfig`](../type-aliases/UseLiveQueryConfig.md)\<`TContext`\> Configuration object with query and options @@ -336,7 +336,7 @@ const itemsQuery = useLiveQuery({ function useLiveQuery(liveQueryCollection): UseLiveQueryReturnWithCollection; ``` -Defined in: [useLiveQuery.svelte.ts:263](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L263) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:283](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L283) Subscribe to an existing query collection (can be reactive) @@ -419,7 +419,7 @@ const queryResult = useLiveQuery(sharedQuery) function useLiveQuery(liveQueryCollection): UseLiveQueryReturnWithCollection; ``` -Defined in: [useLiveQuery.svelte.ts:274](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L274) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:294](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L294) Create a live query using a query function diff --git a/docs/framework/svelte/reference/functions/useOptionalDbClient.md b/docs/framework/svelte/reference/functions/useOptionalDbClient.md new file mode 100644 index 0000000000..c6120c31c8 --- /dev/null +++ b/docs/framework/svelte/reference/functions/useOptionalDbClient.md @@ -0,0 +1,16 @@ +--- +id: useOptionalDbClient +title: useOptionalDbClient +--- + +# Function: useOptionalDbClient() + +```ts +function useOptionalDbClient(): DbClient | undefined; +``` + +Defined in: [packages/svelte-db/src/db-context.ts:19](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/db-context.ts#L19) + +## Returns + +`DbClient` \| `undefined` diff --git a/docs/framework/svelte/reference/index.md b/docs/framework/svelte/reference/index.md index 152862d59a..70f111007b 100644 --- a/docs/framework/svelte/reference/index.md +++ b/docs/framework/svelte/reference/index.md @@ -10,6 +10,22 @@ title: "@tanstack/svelte-db" - [UseLiveQueryReturn](interfaces/UseLiveQueryReturn.md) - [UseLiveQueryReturnWithCollection](interfaces/UseLiveQueryReturnWithCollection.md) +## Type Aliases + +- [DbProvider](type-aliases/DbProvider.md) +- [LiveInfiniteQueryConfig](type-aliases/LiveInfiniteQueryConfig.md) +- [UseLiveInfiniteQueryConfig](type-aliases/UseLiveInfiniteQueryConfig.md) +- [UseLiveInfiniteQueryReturn](type-aliases/UseLiveInfiniteQueryReturn.md) +- [UseLiveInfiniteQueryReturnWithCollection](type-aliases/UseLiveInfiniteQueryReturnWithCollection.md) +- [UseLiveQueryConfig](type-aliases/UseLiveQueryConfig.md) + +## Variables + +- [DbProvider](variables/DbProvider.md) + ## Functions +- [useDbClient](functions/useDbClient.md) +- [useLiveInfiniteQuery](functions/useLiveInfiniteQuery.md) - [useLiveQuery](functions/useLiveQuery.md) +- [useOptionalDbClient](functions/useOptionalDbClient.md) diff --git a/docs/framework/svelte/reference/interfaces/UseLiveQueryReturn.md b/docs/framework/svelte/reference/interfaces/UseLiveQueryReturn.md index 714736f4bf..3a087f00d0 100644 --- a/docs/framework/svelte/reference/interfaces/UseLiveQueryReturn.md +++ b/docs/framework/svelte/reference/interfaces/UseLiveQueryReturn.md @@ -5,7 +5,7 @@ title: UseLiveQueryReturn # Interface: UseLiveQueryReturn\ -Defined in: [useLiveQuery.svelte.ts:33](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L33) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:47](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L47) Return type for useLiveQuery hook @@ -28,7 +28,7 @@ collection: Collection; ``` -Defined in: [useLiveQuery.svelte.ts:36](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L36) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:50](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L50) The underlying query collection instance @@ -40,7 +40,7 @@ The underlying query collection instance data: TData; ``` -Defined in: [useLiveQuery.svelte.ts:35](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L35) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:49](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L49) Reactive array of query results in order, or single item when using findOne() @@ -52,7 +52,7 @@ Reactive array of query results in order, or single item when using findOne() isCleanedUp: boolean; ``` -Defined in: [useLiveQuery.svelte.ts:42](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L42) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:56](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L56) True when query has been cleaned up @@ -64,7 +64,7 @@ True when query has been cleaned up isError: boolean; ``` -Defined in: [useLiveQuery.svelte.ts:41](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L41) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:55](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L55) True when query encountered an error @@ -76,7 +76,7 @@ True when query encountered an error isIdle: boolean; ``` -Defined in: [useLiveQuery.svelte.ts:40](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L40) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:54](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L54) True when query hasn't started yet @@ -88,7 +88,7 @@ True when query hasn't started yet isLoading: boolean; ``` -Defined in: [useLiveQuery.svelte.ts:38](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L38) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:52](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L52) True while initial query data is loading @@ -100,7 +100,7 @@ True while initial query data is loading isReady: boolean; ``` -Defined in: [useLiveQuery.svelte.ts:39](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L39) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:53](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L53) True when query has received first data and is ready @@ -112,7 +112,7 @@ True when query has received first data and is ready state: Map; ``` -Defined in: [useLiveQuery.svelte.ts:34](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L34) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:48](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L48) Reactive Map of query results (key → item) @@ -124,6 +124,6 @@ Reactive Map of query results (key → item) status: CollectionStatus; ``` -Defined in: [useLiveQuery.svelte.ts:37](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L37) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:51](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L51) Current query status diff --git a/docs/framework/svelte/reference/interfaces/UseLiveQueryReturnWithCollection.md b/docs/framework/svelte/reference/interfaces/UseLiveQueryReturnWithCollection.md index 0d62c7ae82..af99083a24 100644 --- a/docs/framework/svelte/reference/interfaces/UseLiveQueryReturnWithCollection.md +++ b/docs/framework/svelte/reference/interfaces/UseLiveQueryReturnWithCollection.md @@ -5,7 +5,7 @@ title: UseLiveQueryReturnWithCollection # Interface: UseLiveQueryReturnWithCollection\ -Defined in: [useLiveQuery.svelte.ts:45](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L45) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:59](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L59) ## Type Parameters @@ -33,7 +33,7 @@ Defined in: [useLiveQuery.svelte.ts:45](https://github.com/TanStack/db/blob/main collection: Collection; ``` -Defined in: [useLiveQuery.svelte.ts:53](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L53) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:67](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L67) *** @@ -43,7 +43,7 @@ Defined in: [useLiveQuery.svelte.ts:53](https://github.com/TanStack/db/blob/main data: TData; ``` -Defined in: [useLiveQuery.svelte.ts:52](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L52) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:66](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L66) *** @@ -53,7 +53,7 @@ Defined in: [useLiveQuery.svelte.ts:52](https://github.com/TanStack/db/blob/main isCleanedUp: boolean; ``` -Defined in: [useLiveQuery.svelte.ts:59](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L59) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:73](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L73) *** @@ -63,7 +63,7 @@ Defined in: [useLiveQuery.svelte.ts:59](https://github.com/TanStack/db/blob/main isError: boolean; ``` -Defined in: [useLiveQuery.svelte.ts:58](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L58) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:72](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L72) *** @@ -73,7 +73,7 @@ Defined in: [useLiveQuery.svelte.ts:58](https://github.com/TanStack/db/blob/main isIdle: boolean; ``` -Defined in: [useLiveQuery.svelte.ts:57](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L57) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:71](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L71) *** @@ -83,7 +83,7 @@ Defined in: [useLiveQuery.svelte.ts:57](https://github.com/TanStack/db/blob/main isLoading: boolean; ``` -Defined in: [useLiveQuery.svelte.ts:55](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L55) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:69](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L69) *** @@ -93,7 +93,7 @@ Defined in: [useLiveQuery.svelte.ts:55](https://github.com/TanStack/db/blob/main isReady: boolean; ``` -Defined in: [useLiveQuery.svelte.ts:56](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L56) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:70](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L70) *** @@ -103,7 +103,7 @@ Defined in: [useLiveQuery.svelte.ts:56](https://github.com/TanStack/db/blob/main state: Map; ``` -Defined in: [useLiveQuery.svelte.ts:51](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L51) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:65](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L65) *** @@ -113,4 +113,4 @@ Defined in: [useLiveQuery.svelte.ts:51](https://github.com/TanStack/db/blob/main status: CollectionStatus; ``` -Defined in: [useLiveQuery.svelte.ts:54](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L54) +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:68](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L68) diff --git a/docs/framework/svelte/reference/type-aliases/DbProvider.md b/docs/framework/svelte/reference/type-aliases/DbProvider.md new file mode 100644 index 0000000000..181ffc22bc --- /dev/null +++ b/docs/framework/svelte/reference/type-aliases/DbProvider.md @@ -0,0 +1,12 @@ +--- +id: DbProvider +title: DbProvider +--- + +# Type Alias: DbProvider + +```ts +type DbProvider = SvelteComponent; +``` + +Defined in: node\_modules/.pnpm/svelte@5.50.0/node\_modules/svelte/types/index.d.ts:3178 diff --git a/docs/framework/svelte/reference/type-aliases/LiveInfiniteQueryConfig.md b/docs/framework/svelte/reference/type-aliases/LiveInfiniteQueryConfig.md new file mode 100644 index 0000000000..4a08b4d397 --- /dev/null +++ b/docs/framework/svelte/reference/type-aliases/LiveInfiniteQueryConfig.md @@ -0,0 +1,53 @@ +--- +id: LiveInfiniteQueryConfig +title: LiveInfiniteQueryConfig +--- + +# Type Alias: LiveInfiniteQueryConfig\ + +```ts +type LiveInfiniteQueryConfig = InfiniteQueryOptions & object; +``` + +Defined in: [packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts:43](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts#L43) + +## Type Declaration + +### ~~getNextPageParam()?~~ + +```ts +optional getNextPageParam: (lastPage, allPages, lastPageParam, allPageParams) => number | undefined; +``` + +#### Parameters + +##### lastPage + +`TRow`[] + +##### allPages + +`TRow`[][] + +##### lastPageParam + +`number` + +##### allPageParams + +`number`[] + +#### Returns + +`number` \| `undefined` + +#### Deprecated + +Pagination uses the shared controller's peek-ahead strategy. +This remains for compatibility with TanStack Query conventions. + +## Type Parameters + +### TRow + +`TRow` diff --git a/docs/framework/svelte/reference/type-aliases/UseLiveInfiniteQueryConfig.md b/docs/framework/svelte/reference/type-aliases/UseLiveInfiniteQueryConfig.md new file mode 100644 index 0000000000..38650d4ee4 --- /dev/null +++ b/docs/framework/svelte/reference/type-aliases/UseLiveInfiniteQueryConfig.md @@ -0,0 +1,18 @@ +--- +id: UseLiveInfiniteQueryConfig +title: UseLiveInfiniteQueryConfig +--- + +# Type Alias: UseLiveInfiniteQueryConfig\ + +```ts +type UseLiveInfiniteQueryConfig = LiveInfiniteQueryConfig[number]>; +``` + +Defined in: [packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts:56](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts#L56) + +## Type Parameters + +### TContext + +`TContext` *extends* `Context` diff --git a/docs/framework/svelte/reference/type-aliases/UseLiveInfiniteQueryReturn.md b/docs/framework/svelte/reference/type-aliases/UseLiveInfiniteQueryReturn.md new file mode 100644 index 0000000000..108de73c9d --- /dev/null +++ b/docs/framework/svelte/reference/type-aliases/UseLiveInfiniteQueryReturn.md @@ -0,0 +1,66 @@ +--- +id: UseLiveInfiniteQueryReturn +title: UseLiveInfiniteQueryReturn +--- + +# Type Alias: UseLiveInfiniteQueryReturn\ + +```ts +type UseLiveInfiniteQueryReturn = Omit, "data"> & object; +``` + +Defined in: [packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts:59](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts#L59) + +## Type Declaration + +### data + +```ts +data: InferResultType; +``` + +### error + +```ts +error: unknown; +``` + +### fetchNextPage() + +```ts +fetchNextPage: () => Promise; +``` + +#### Returns + +`Promise`\<`void`\> + +### hasNextPage + +```ts +hasNextPage: boolean; +``` + +### isFetchingNextPage + +```ts +isFetchingNextPage: boolean; +``` + +### pageParams + +```ts +pageParams: number[]; +``` + +### pages + +```ts +pages: InferResultType[number][][]; +``` + +## Type Parameters + +### TContext + +`TContext` *extends* `Context` diff --git a/docs/framework/svelte/reference/type-aliases/UseLiveInfiniteQueryReturnWithCollection.md b/docs/framework/svelte/reference/type-aliases/UseLiveInfiniteQueryReturnWithCollection.md new file mode 100644 index 0000000000..52fff511f7 --- /dev/null +++ b/docs/framework/svelte/reference/type-aliases/UseLiveInfiniteQueryReturnWithCollection.md @@ -0,0 +1,74 @@ +--- +id: UseLiveInfiniteQueryReturnWithCollection +title: UseLiveInfiniteQueryReturnWithCollection +--- + +# Type Alias: UseLiveInfiniteQueryReturnWithCollection\ + +```ts +type UseLiveInfiniteQueryReturnWithCollection = Omit, "data"> & object; +``` + +Defined in: [packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts:72](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts#L72) + +## Type Declaration + +### data + +```ts +data: TResult[]; +``` + +### error + +```ts +error: unknown; +``` + +### fetchNextPage() + +```ts +fetchNextPage: () => Promise; +``` + +#### Returns + +`Promise`\<`void`\> + +### hasNextPage + +```ts +hasNextPage: boolean; +``` + +### isFetchingNextPage + +```ts +isFetchingNextPage: boolean; +``` + +### pageParams + +```ts +pageParams: number[]; +``` + +### pages + +```ts +pages: TResult[][]; +``` + +## Type Parameters + +### TResult + +`TResult` *extends* `object` + +### TKey + +`TKey` *extends* `string` \| `number` + +### TUtils + +`TUtils` *extends* `Record`\<`string`, `any`\> diff --git a/docs/framework/svelte/reference/type-aliases/UseLiveQueryConfig.md b/docs/framework/svelte/reference/type-aliases/UseLiveQueryConfig.md new file mode 100644 index 0000000000..14ea334738 --- /dev/null +++ b/docs/framework/svelte/reference/type-aliases/UseLiveQueryConfig.md @@ -0,0 +1,32 @@ +--- +id: UseLiveQueryConfig +title: UseLiveQueryConfig +--- + +# Type Alias: UseLiveQueryConfig\ + +```ts +type UseLiveQueryConfig = LiveQueryCollectionConfig & object; +``` + +Defined in: [packages/svelte-db/src/useLiveQuery.svelte.ts:78](https://github.com/TanStack/db/blob/main/packages/svelte-db/src/useLiveQuery.svelte.ts#L78) + +## Type Declaration + +### client? + +```ts +optional client: DbClient; +``` + +### queryKey? + +```ts +optional queryKey: MaybeGetter; +``` + +## Type Parameters + +### TContext + +`TContext` *extends* `Context` diff --git a/docs/framework/svelte/reference/variables/DbProvider.md b/docs/framework/svelte/reference/variables/DbProvider.md new file mode 100644 index 0000000000..b5f0d4541e --- /dev/null +++ b/docs/framework/svelte/reference/variables/DbProvider.md @@ -0,0 +1,12 @@ +--- +id: DbProvider +title: DbProvider +--- + +# Variable: DbProvider + +```ts +const DbProvider: LegacyComponentType; +``` + +Defined in: node\_modules/.pnpm/svelte@5.50.0/node\_modules/svelte/types/index.d.ts:3178 diff --git a/docs/framework/vue/overview.md b/docs/framework/vue/overview.md index eb78d370f4..db8cddcbb9 100644 --- a/docs/framework/vue/overview.md +++ b/docs/framework/vue/overview.md @@ -43,6 +43,49 @@ const { data, isLoading } = useLiveQuery((q) => **Note:** All return values (`data`, `isLoading`, `status`, etc.) are computed refs, so access them with `.value` in ` + + +``` + +`fetchNextPage()` returns a promise that resolves after the page request settles. Failures are exposed through the returned `error` ref and do not reject the promise. + +The query must include `orderBy`. The dependency array is available only with +the query-function form. You can also pass an ordered, pre-created live query +collection directly. + ### Dependency Arrays The `useLiveQuery` composable accepts an optional dependency array as its last parameter. When any reactive value in the array changes, the query is recreated and re-executed. diff --git a/docs/framework/vue/reference/functions/useLiveInfiniteQuery.md b/docs/framework/vue/reference/functions/useLiveInfiniteQuery.md new file mode 100644 index 0000000000..ace644e59e --- /dev/null +++ b/docs/framework/vue/reference/functions/useLiveInfiniteQuery.md @@ -0,0 +1,83 @@ +--- +id: useLiveInfiniteQuery +title: useLiveInfiniteQuery +--- + +# Function: useLiveInfiniteQuery() + +## Call Signature + +```ts +function useLiveInfiniteQuery(liveQueryCollection, config): UseLiveInfiniteQueryReturnWithCollection; +``` + +Defined in: [useLiveInfiniteQuery.ts:104](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L104) + +Create a Vue-native reactive view over the shared live-query window +controller. The query must include an `orderBy` clause. + +### Type Parameters + +#### TResult + +`TResult` *extends* `object` + +#### TKey + +`TKey` *extends* `string` \| `number` + +#### TUtils + +`TUtils` *extends* `UtilsRecord` + +### Parameters + +#### liveQueryCollection + +`MaybeRefOrGetter`\<`Collection`\<`TResult`, `TKey`, `TUtils`, `StandardSchemaV1`\<`unknown`, `unknown`\>, `TResult`\> & `NonSingleResult`\> + +#### config + +[`LiveInfiniteQueryConfig`](../type-aliases/LiveInfiniteQueryConfig.md)\<`TResult`\> + +### Returns + +[`UseLiveInfiniteQueryReturnWithCollection`](../interfaces/UseLiveInfiniteQueryReturnWithCollection.md)\<`TResult`, `TKey`, `TUtils`\> + +## Call Signature + +```ts +function useLiveInfiniteQuery( + queryFn, + config, +deps?): UseLiveInfiniteQueryReturn; +``` + +Defined in: [useLiveInfiniteQuery.ts:115](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L115) + +Create a Vue-native reactive view over the shared live-query window +controller. The query must include an `orderBy` clause. + +### Type Parameters + +#### TContext + +`TContext` *extends* `Context` & `NonSingleResult` + +### Parameters + +#### queryFn + +(`q`) => `QueryBuilder`\<`TContext`\> + +#### config + +[`UseLiveInfiniteQueryConfig`](../type-aliases/UseLiveInfiniteQueryConfig.md)\<`TContext`\> + +#### deps? + +`unknown`[] + +### Returns + +[`UseLiveInfiniteQueryReturn`](../interfaces/UseLiveInfiniteQueryReturn.md)\<`TContext`\> diff --git a/docs/framework/vue/reference/functions/useLiveQuery.md b/docs/framework/vue/reference/functions/useLiveQuery.md index 71c717e542..5b8126e1aa 100644 --- a/docs/framework/vue/reference/functions/useLiveQuery.md +++ b/docs/framework/vue/reference/functions/useLiveQuery.md @@ -11,7 +11,7 @@ title: useLiveQuery function useLiveQuery(queryFn, deps?): UseLiveQueryReturn; ``` -Defined in: [useLiveQuery.ts:134](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L134) +Defined in: [useLiveQuery.ts:138](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L138) Create a live query using a query function @@ -97,7 +97,7 @@ const { data, isLoading, isError, status } = useLiveQuery((q) => function useLiveQuery(queryFn, deps?): UseLiveQueryReturn; ``` -Defined in: [useLiveQuery.ts:140](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L140) +Defined in: [useLiveQuery.ts:144](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L144) Create a live query using a query function @@ -183,7 +183,7 @@ const { data, isLoading, isError, status } = useLiveQuery((q) => function useLiveQuery(config, deps?): UseLiveQueryReturn; ``` -Defined in: [useLiveQuery.ts:180](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L180) +Defined in: [useLiveQuery.ts:184](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L184) Create a live query using configuration object @@ -251,7 +251,7 @@ const { data, isLoading, isReady, isError } = useLiveQuery({ function useLiveQuery(liveQueryCollection): UseLiveQueryReturnWithCollection; ``` -Defined in: [useLiveQuery.ts:225](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L225) +Defined in: [useLiveQuery.ts:229](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L229) Subscribe to an existing query collection (can be reactive) @@ -330,7 +330,7 @@ const { data, isLoading, isError } = useLiveQuery(sharedQuery) function useLiveQuery(liveQueryCollection): UseLiveQueryReturnWithSingleResultCollection; ``` -Defined in: [useLiveQuery.ts:236](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L236) +Defined in: [useLiveQuery.ts:240](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L240) Create a live query using a query function diff --git a/docs/framework/vue/reference/index.md b/docs/framework/vue/reference/index.md index fb22a159e6..a03e38e1bc 100644 --- a/docs/framework/vue/reference/index.md +++ b/docs/framework/vue/reference/index.md @@ -7,10 +7,18 @@ title: "@tanstack/vue-db" ## Interfaces +- [UseLiveInfiniteQueryReturn](interfaces/UseLiveInfiniteQueryReturn.md) +- [UseLiveInfiniteQueryReturnWithCollection](interfaces/UseLiveInfiniteQueryReturnWithCollection.md) - [UseLiveQueryReturn](interfaces/UseLiveQueryReturn.md) - [UseLiveQueryReturnWithCollection](interfaces/UseLiveQueryReturnWithCollection.md) - [UseLiveQueryReturnWithSingleResultCollection](interfaces/UseLiveQueryReturnWithSingleResultCollection.md) +## Type Aliases + +- [LiveInfiniteQueryConfig](type-aliases/LiveInfiniteQueryConfig.md) +- [UseLiveInfiniteQueryConfig](type-aliases/UseLiveInfiniteQueryConfig.md) + ## Functions +- [useLiveInfiniteQuery](functions/useLiveInfiniteQuery.md) - [useLiveQuery](functions/useLiveQuery.md) diff --git a/docs/framework/vue/reference/interfaces/UseLiveInfiniteQueryReturn.md b/docs/framework/vue/reference/interfaces/UseLiveInfiniteQueryReturn.md new file mode 100644 index 0000000000..8124be822d --- /dev/null +++ b/docs/framework/vue/reference/interfaces/UseLiveInfiniteQueryReturn.md @@ -0,0 +1,168 @@ +--- +id: UseLiveInfiniteQueryReturn +title: UseLiveInfiniteQueryReturn +--- + +# Interface: UseLiveInfiniteQueryReturn\ + +Defined in: [useLiveInfiniteQuery.ts:56](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L56) + +## Type Parameters + +### TContext + +`TContext` *extends* `Context` & `NonSingleResult` + +## Properties + +### collection + +```ts +collection: ComputedRef[K] }, string | number, UtilsRecord, StandardSchemaV1, { [K in string | number | symbol]: ResultValue[K] }>>; +``` + +Defined in: [useLiveInfiniteQuery.ts:61](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L61) + +*** + +### data + +```ts +data: ComputedRef>; +``` + +Defined in: [useLiveInfiniteQuery.ts:60](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L60) + +*** + +### error + +```ts +error: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:75](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L75) + +*** + +### fetchNextPage() + +```ts +fetchNextPage: () => Promise; +``` + +Defined in: [useLiveInfiniteQuery.ts:72](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L72) + +#### Returns + +`Promise`\<`void`\> + +*** + +### hasNextPage + +```ts +hasNextPage: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:73](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L73) + +*** + +### isCleanedUp + +```ts +isCleanedUp: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:69](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L69) + +*** + +### isError + +```ts +isError: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:68](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L68) + +*** + +### isFetchingNextPage + +```ts +isFetchingNextPage: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:74](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L74) + +*** + +### isIdle + +```ts +isIdle: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:67](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L67) + +*** + +### isLoading + +```ts +isLoading: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:65](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L65) + +*** + +### isReady + +```ts +isReady: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:66](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L66) + +*** + +### pageParams + +```ts +pageParams: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:71](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L71) + +*** + +### pages + +```ts +pages: ComputedRef[number][][]>; +``` + +Defined in: [useLiveInfiniteQuery.ts:70](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L70) + +*** + +### state + +```ts +state: ComputedRef[K] }>>; +``` + +Defined in: [useLiveInfiniteQuery.ts:59](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L59) + +*** + +### status + +```ts +status: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:64](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L64) diff --git a/docs/framework/vue/reference/interfaces/UseLiveInfiniteQueryReturnWithCollection.md b/docs/framework/vue/reference/interfaces/UseLiveInfiniteQueryReturnWithCollection.md new file mode 100644 index 0000000000..53fb1342b1 --- /dev/null +++ b/docs/framework/vue/reference/interfaces/UseLiveInfiniteQueryReturnWithCollection.md @@ -0,0 +1,176 @@ +--- +id: UseLiveInfiniteQueryReturnWithCollection +title: UseLiveInfiniteQueryReturnWithCollection +--- + +# Interface: UseLiveInfiniteQueryReturnWithCollection\ + +Defined in: [useLiveInfiniteQuery.ts:78](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L78) + +## Type Parameters + +### TResult + +`TResult` *extends* `object` + +### TKey + +`TKey` *extends* `string` \| `number` + +### TUtils + +`TUtils` *extends* `UtilsRecord` + +## Properties + +### collection + +```ts +collection: ComputedRef, TResult>>; +``` + +Defined in: [useLiveInfiniteQuery.ts:85](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L85) + +*** + +### data + +```ts +data: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:84](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L84) + +*** + +### error + +```ts +error: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:97](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L97) + +*** + +### fetchNextPage() + +```ts +fetchNextPage: () => Promise; +``` + +Defined in: [useLiveInfiniteQuery.ts:94](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L94) + +#### Returns + +`Promise`\<`void`\> + +*** + +### hasNextPage + +```ts +hasNextPage: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:95](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L95) + +*** + +### isCleanedUp + +```ts +isCleanedUp: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:91](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L91) + +*** + +### isError + +```ts +isError: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:90](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L90) + +*** + +### isFetchingNextPage + +```ts +isFetchingNextPage: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:96](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L96) + +*** + +### isIdle + +```ts +isIdle: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:89](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L89) + +*** + +### isLoading + +```ts +isLoading: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:87](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L87) + +*** + +### isReady + +```ts +isReady: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:88](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L88) + +*** + +### pageParams + +```ts +pageParams: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:93](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L93) + +*** + +### pages + +```ts +pages: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:92](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L92) + +*** + +### state + +```ts +state: ComputedRef>; +``` + +Defined in: [useLiveInfiniteQuery.ts:83](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L83) + +*** + +### status + +```ts +status: ComputedRef; +``` + +Defined in: [useLiveInfiniteQuery.ts:86](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L86) diff --git a/docs/framework/vue/reference/interfaces/UseLiveQueryReturn.md b/docs/framework/vue/reference/interfaces/UseLiveQueryReturn.md index d434f46cd7..c2c10e752f 100644 --- a/docs/framework/vue/reference/interfaces/UseLiveQueryReturn.md +++ b/docs/framework/vue/reference/interfaces/UseLiveQueryReturn.md @@ -5,7 +5,7 @@ title: UseLiveQueryReturn # Interface: UseLiveQueryReturn\ -Defined in: [useLiveQuery.ts:40](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L40) +Defined in: [useLiveQuery.ts:44](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L44) Return type for useLiveQuery hook @@ -24,7 +24,7 @@ collection: ComputedRef, { [K in string | number | symbol]: ResultValue[K] }>>; ``` -Defined in: [useLiveQuery.ts:43](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L43) +Defined in: [useLiveQuery.ts:47](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L47) The underlying query collection instance @@ -36,7 +36,7 @@ The underlying query collection instance data: ComputedRef>; ``` -Defined in: [useLiveQuery.ts:42](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L42) +Defined in: [useLiveQuery.ts:46](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L46) Reactive array of query results in order, or single result for findOne queries @@ -48,7 +48,7 @@ Reactive array of query results in order, or single result for findOne queries isCleanedUp: ComputedRef; ``` -Defined in: [useLiveQuery.ts:49](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L49) +Defined in: [useLiveQuery.ts:53](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L53) True when query has been cleaned up @@ -60,7 +60,7 @@ True when query has been cleaned up isError: ComputedRef; ``` -Defined in: [useLiveQuery.ts:48](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L48) +Defined in: [useLiveQuery.ts:52](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L52) True when query encountered an error @@ -72,7 +72,7 @@ True when query encountered an error isIdle: ComputedRef; ``` -Defined in: [useLiveQuery.ts:47](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L47) +Defined in: [useLiveQuery.ts:51](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L51) True when query hasn't started yet @@ -84,7 +84,7 @@ True when query hasn't started yet isLoading: ComputedRef; ``` -Defined in: [useLiveQuery.ts:45](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L45) +Defined in: [useLiveQuery.ts:49](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L49) True while initial query data is loading @@ -96,7 +96,7 @@ True while initial query data is loading isReady: ComputedRef; ``` -Defined in: [useLiveQuery.ts:46](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L46) +Defined in: [useLiveQuery.ts:50](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L50) True when query has received first data and is ready @@ -108,7 +108,7 @@ True when query has received first data and is ready state: ComputedRef[K] }>>; ``` -Defined in: [useLiveQuery.ts:41](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L41) +Defined in: [useLiveQuery.ts:45](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L45) Reactive Map of query results (key → item) @@ -120,6 +120,6 @@ Reactive Map of query results (key → item) status: ComputedRef; ``` -Defined in: [useLiveQuery.ts:44](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L44) +Defined in: [useLiveQuery.ts:48](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L48) Current query status diff --git a/docs/framework/vue/reference/interfaces/UseLiveQueryReturnWithCollection.md b/docs/framework/vue/reference/interfaces/UseLiveQueryReturnWithCollection.md index 8f63d93141..085c5514fb 100644 --- a/docs/framework/vue/reference/interfaces/UseLiveQueryReturnWithCollection.md +++ b/docs/framework/vue/reference/interfaces/UseLiveQueryReturnWithCollection.md @@ -5,7 +5,7 @@ title: UseLiveQueryReturnWithCollection # Interface: UseLiveQueryReturnWithCollection\ -Defined in: [useLiveQuery.ts:52](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L52) +Defined in: [useLiveQuery.ts:56](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L56) ## Type Parameters @@ -29,7 +29,7 @@ Defined in: [useLiveQuery.ts:52](https://github.com/TanStack/db/blob/main/packag collection: ComputedRef, T>>; ``` -Defined in: [useLiveQuery.ts:59](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L59) +Defined in: [useLiveQuery.ts:63](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L63) *** @@ -39,7 +39,7 @@ Defined in: [useLiveQuery.ts:59](https://github.com/TanStack/db/blob/main/packag data: ComputedRef; ``` -Defined in: [useLiveQuery.ts:58](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L58) +Defined in: [useLiveQuery.ts:62](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L62) *** @@ -49,7 +49,7 @@ Defined in: [useLiveQuery.ts:58](https://github.com/TanStack/db/blob/main/packag isCleanedUp: ComputedRef; ``` -Defined in: [useLiveQuery.ts:65](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L65) +Defined in: [useLiveQuery.ts:69](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L69) *** @@ -59,7 +59,7 @@ Defined in: [useLiveQuery.ts:65](https://github.com/TanStack/db/blob/main/packag isError: ComputedRef; ``` -Defined in: [useLiveQuery.ts:64](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L64) +Defined in: [useLiveQuery.ts:68](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L68) *** @@ -69,7 +69,7 @@ Defined in: [useLiveQuery.ts:64](https://github.com/TanStack/db/blob/main/packag isIdle: ComputedRef; ``` -Defined in: [useLiveQuery.ts:63](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L63) +Defined in: [useLiveQuery.ts:67](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L67) *** @@ -79,7 +79,7 @@ Defined in: [useLiveQuery.ts:63](https://github.com/TanStack/db/blob/main/packag isLoading: ComputedRef; ``` -Defined in: [useLiveQuery.ts:61](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L61) +Defined in: [useLiveQuery.ts:65](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L65) *** @@ -89,7 +89,7 @@ Defined in: [useLiveQuery.ts:61](https://github.com/TanStack/db/blob/main/packag isReady: ComputedRef; ``` -Defined in: [useLiveQuery.ts:62](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L62) +Defined in: [useLiveQuery.ts:66](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L66) *** @@ -99,7 +99,7 @@ Defined in: [useLiveQuery.ts:62](https://github.com/TanStack/db/blob/main/packag state: ComputedRef>; ``` -Defined in: [useLiveQuery.ts:57](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L57) +Defined in: [useLiveQuery.ts:61](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L61) *** @@ -109,4 +109,4 @@ Defined in: [useLiveQuery.ts:57](https://github.com/TanStack/db/blob/main/packag status: ComputedRef; ``` -Defined in: [useLiveQuery.ts:60](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L60) +Defined in: [useLiveQuery.ts:64](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L64) diff --git a/docs/framework/vue/reference/interfaces/UseLiveQueryReturnWithSingleResultCollection.md b/docs/framework/vue/reference/interfaces/UseLiveQueryReturnWithSingleResultCollection.md index 5b9502a979..09eb8467f9 100644 --- a/docs/framework/vue/reference/interfaces/UseLiveQueryReturnWithSingleResultCollection.md +++ b/docs/framework/vue/reference/interfaces/UseLiveQueryReturnWithSingleResultCollection.md @@ -5,7 +5,7 @@ title: UseLiveQueryReturnWithSingleResultCollection # Interface: UseLiveQueryReturnWithSingleResultCollection\ -Defined in: [useLiveQuery.ts:68](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L68) +Defined in: [useLiveQuery.ts:72](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L72) ## Type Parameters @@ -29,7 +29,7 @@ Defined in: [useLiveQuery.ts:68](https://github.com/TanStack/db/blob/main/packag collection: ComputedRef, T> & SingleResult>; ``` -Defined in: [useLiveQuery.ts:75](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L75) +Defined in: [useLiveQuery.ts:79](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L79) *** @@ -39,7 +39,7 @@ Defined in: [useLiveQuery.ts:75](https://github.com/TanStack/db/blob/main/packag data: ComputedRef; ``` -Defined in: [useLiveQuery.ts:74](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L74) +Defined in: [useLiveQuery.ts:78](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L78) *** @@ -49,7 +49,7 @@ Defined in: [useLiveQuery.ts:74](https://github.com/TanStack/db/blob/main/packag isCleanedUp: ComputedRef; ``` -Defined in: [useLiveQuery.ts:81](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L81) +Defined in: [useLiveQuery.ts:85](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L85) *** @@ -59,7 +59,7 @@ Defined in: [useLiveQuery.ts:81](https://github.com/TanStack/db/blob/main/packag isError: ComputedRef; ``` -Defined in: [useLiveQuery.ts:80](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L80) +Defined in: [useLiveQuery.ts:84](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L84) *** @@ -69,7 +69,7 @@ Defined in: [useLiveQuery.ts:80](https://github.com/TanStack/db/blob/main/packag isIdle: ComputedRef; ``` -Defined in: [useLiveQuery.ts:79](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L79) +Defined in: [useLiveQuery.ts:83](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L83) *** @@ -79,7 +79,7 @@ Defined in: [useLiveQuery.ts:79](https://github.com/TanStack/db/blob/main/packag isLoading: ComputedRef; ``` -Defined in: [useLiveQuery.ts:77](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L77) +Defined in: [useLiveQuery.ts:81](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L81) *** @@ -89,7 +89,7 @@ Defined in: [useLiveQuery.ts:77](https://github.com/TanStack/db/blob/main/packag isReady: ComputedRef; ``` -Defined in: [useLiveQuery.ts:78](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L78) +Defined in: [useLiveQuery.ts:82](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L82) *** @@ -99,7 +99,7 @@ Defined in: [useLiveQuery.ts:78](https://github.com/TanStack/db/blob/main/packag state: ComputedRef>; ``` -Defined in: [useLiveQuery.ts:73](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L73) +Defined in: [useLiveQuery.ts:77](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L77) *** @@ -109,4 +109,4 @@ Defined in: [useLiveQuery.ts:73](https://github.com/TanStack/db/blob/main/packag status: ComputedRef; ``` -Defined in: [useLiveQuery.ts:76](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L76) +Defined in: [useLiveQuery.ts:80](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveQuery.ts#L80) diff --git a/docs/framework/vue/reference/type-aliases/LiveInfiniteQueryConfig.md b/docs/framework/vue/reference/type-aliases/LiveInfiniteQueryConfig.md new file mode 100644 index 0000000000..ae4bbd683e --- /dev/null +++ b/docs/framework/vue/reference/type-aliases/LiveInfiniteQueryConfig.md @@ -0,0 +1,53 @@ +--- +id: LiveInfiniteQueryConfig +title: LiveInfiniteQueryConfig +--- + +# Type Alias: LiveInfiniteQueryConfig\ + +```ts +type LiveInfiniteQueryConfig = InfiniteQueryOptions & object; +``` + +Defined in: [useLiveInfiniteQuery.ts:39](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L39) + +## Type Declaration + +### ~~getNextPageParam()?~~ + +```ts +optional getNextPageParam: (lastPage, allPages, lastPageParam, allPageParams) => number | undefined; +``` + +#### Parameters + +##### lastPage + +`TRow`[] + +##### allPages + +`TRow`[][] + +##### lastPageParam + +`number` + +##### allPageParams + +`number`[] + +#### Returns + +`number` \| `undefined` + +#### Deprecated + +Pagination uses the shared controller's peek-ahead strategy. +This remains for compatibility with TanStack Query conventions. + +## Type Parameters + +### TRow + +`TRow` diff --git a/docs/framework/vue/reference/type-aliases/UseLiveInfiniteQueryConfig.md b/docs/framework/vue/reference/type-aliases/UseLiveInfiniteQueryConfig.md new file mode 100644 index 0000000000..8d1625bf01 --- /dev/null +++ b/docs/framework/vue/reference/type-aliases/UseLiveInfiniteQueryConfig.md @@ -0,0 +1,18 @@ +--- +id: UseLiveInfiniteQueryConfig +title: UseLiveInfiniteQueryConfig +--- + +# Type Alias: UseLiveInfiniteQueryConfig\ + +```ts +type UseLiveInfiniteQueryConfig = LiveInfiniteQueryConfig[number]>; +``` + +Defined in: [useLiveInfiniteQuery.ts:52](https://github.com/TanStack/db/blob/main/packages/vue-db/src/useLiveInfiniteQuery.ts#L52) + +## Type Parameters + +### TContext + +`TContext` *extends* `Context` & `NonSingleResult` diff --git a/docs/guides/collection-options-creator.md b/docs/guides/collection-options-creator.md index d1f4d55c4a..d0e69d3a98 100644 --- a/docs/guides/collection-options-creator.md +++ b/docs/guides/collection-options-creator.md @@ -2,8 +2,6 @@ title: Creating a Collection Options Creator id: guide/collection-options-creator --- -# Creating a Collection Options Creator - A collection options creator is a factory function that generates configuration options for TanStack DB collections. It provides a standardized way to integrate different sync engines and data sources with TanStack DB's reactive sync-first architecture. ## Overview @@ -73,10 +71,11 @@ The sync function must return a cleanup function for proper garbage collection: ```typescript const sync: SyncConfig['sync'] = (params) => { - const { begin, write, commit, markReady, collection } = params + const { begin, write, commit, markReady, markError, collection } = params // 1. Initialize connection to your sync engine const connection = initializeConnection(config) + const initialSyncAbort = new AbortController() // 2. Set up real-time subscription FIRST (prevents race conditions) const eventBuffer: Array = [] @@ -110,7 +109,7 @@ const sync: SyncConfig['sync'] = (params) => { // 3. Perform initial data fetch async function initialSync() { try { - const data = await fetchInitialData() + const data = await fetchInitialData({ signal: initialSyncAbort.signal }) begin() // Start a transaction @@ -134,13 +133,16 @@ const sync: SyncConfig['sync'] = (params) => { commit() eventBuffer.splice(0) } - + + // A complete initial snapshot is now available. + markReady() } catch (error) { + if (initialSyncAbort.signal.aborted) return console.error('Initial sync failed:', error) - throw error - } finally { - // ALWAYS call markReady, even on error - markReady() + // No usable initial snapshot exists. + // Only initial startup owns collection readiness. A later refetch + // failure must keep the last ready snapshot usable. + if (collection.status === 'loading') markError(error) } } @@ -148,6 +150,7 @@ const sync: SyncConfig['sync'] = (params) => { // 4. Return cleanup function return () => { + initialSyncAbort.abort() connection.close() // Clean up any timers, intervals, or other resources } @@ -163,7 +166,27 @@ The sync process follows this lifecycle: 1. **begin()** - Start collecting changes 2. **write()** - Add changes to the pending transaction (buffered until commit) 3. **commit()** - Apply all changes atomically to the collection state -4. **markReady()** - Signal that initial sync is complete +4. **markReady()** - Signal that a usable initial or recovered snapshot exists +5. **markError(error?)** - Signal that initial sync failed before producing a usable snapshot; pass the cause so readiness waits reject with it + +`commit()` returns `true` if its writes and events are already visible, or a +promise that resolves when they become visible. A commit can wait behind a +pending optimistic transaction; receiving a server response is not the same as +applying its rows. A successful `loadSubset` must await or return every commit +receipt that establishes its result. Do not use `begin({ immediate: true })` to +bypass that ordering just to settle a load. + +For request-scoped writes, pass the request's abort signal to `commit(signal)`. +Cancellation before application rejects the receipt with `AbortError`; aborting +after application does not undo published rows. Do not attach one request's +signal to a shared stream transaction. + +If an adapter supplies `unloadSubset`, release only the acquisition belonging to +the supplied options. Release must be idempotent and non-throwing; the adapter +owns any remote unsubscribe retry. A synchronous `loadSubset` throw must clean +up resources acquired before it throws. Returning a promise transfers ownership +even if that promise later rejects, so failed acquisitions must remain safe to +release without affecting peers. **Race Condition Prevention:** Many sync engines start real-time subscriptions before the initial sync completes. Your implementation MUST deduplicate events that arrive via subscription that represent the same data as the initial sync. Consider: @@ -709,18 +732,23 @@ export function webSocketCollectionOptions( ## Usage Example ```typescript -import { createCollection } from '@tanstack/react-db' +import { DbClient, collectionOptions } from '@tanstack/react-db' import { webSocketCollectionOptions } from './websocket-collection' -const todos = createCollection( +const db = new DbClient() + +const todosCollection = collectionOptions('todos', () => webSocketCollectionOptions({ + id: 'todos', url: 'ws://localhost:8080/todos', getKey: (todo) => todo.id, - schema: todoSchema + schema: todoSchema, // Note: No onInsert/onUpdate/onDelete - handled by WebSocket automatically }) ) +const todos = db.collection(todosCollection) + // Use the collection todos.insert({ id: '1', text: 'Buy milk', completed: false }) @@ -895,8 +923,8 @@ const wrappedOnInsert = async (params) => { ## Best Practices -1. **Always call markReady()** - This signals that the collection has initial data and is ready for use -2. **Handle errors gracefully** - Call markReady() even on error to avoid blocking the app +1. **Report initial sync status** - Call `markReady()` after a usable snapshot, or `markError(error)` if initial sync fails +2. **Recover explicitly** - After an error, call `markReady()` only when a later sync has produced a usable snapshot 3. **Clean up resources** - Return a cleanup function from sync to prevent memory leaks 4. **Batch operations** - Use begin/commit to batch multiple changes for better performance 5. **Race Conditions** - Start listeners before initial fetch and buffer events diff --git a/docs/guides/error-handling.md b/docs/guides/error-handling.md index dfd4fa0b80..0beb5fc8ce 100644 --- a/docs/guides/error-handling.md +++ b/docs/guides/error-handling.md @@ -3,8 +3,6 @@ title: Error Handling id: error-handling --- -# Error Handling - TanStack DB provides comprehensive error handling capabilities to ensure robust data synchronization and state management. This guide covers the built-in error handling mechanisms and how to work with them effectively. ## Error Types @@ -91,7 +89,9 @@ const syncedCollection = createCollection( // Component can check error state function DataList() { - const { data } = useLiveQuery((q) => q.from({ item: syncedCollection })) + const { data } = useLiveQuery({ + query: (q) => q.from({ item: syncedCollection }), + }) const isError = syncedCollection.utils.isError const errorCount = syncedCollection.utils.errorCount @@ -117,6 +117,64 @@ Error tracking methods: - **`errorCount`**: Returns the number of consecutive sync failures. This counter is incremented only when queries fail completely (not per retry attempt) and is reset on successful queries: - **`clearError()`**: Clears the error state and triggers a refetch of the query. This method resets both `lastError` and `errorCount`: +## Incremental Subset Load Errors + +An incremental `loadSubset` failure does not discard rows that are already +available or put the shared source collection into `error`. The failure belongs +to the subscription that requested that subset: + +```ts +const subscription = todoCollection.subscribeChanges(handleChanges, { + includeInitialState: false, +}) + +subscription.on('loadSubset:error', ({ error, options }) => { + console.error('Subset failed', options, error) +}) + +subscription.requestSnapshot() + +// The most recent failure remains available for diagnostics. +console.log(subscription.lastError) +``` + +For ordered live queries, `utils.setWindow()` rejects with the same error. The +last failure is also available as `utils.lastSubsetError`, while the last +successful snapshot remains readable: + +```ts +try { + await liveTodos.utils.setWindow({ offset: 0, limit: 100 }) +} catch (error) { + console.error(liveTodos.utils.lastSubsetError) +} +``` + +Effects report subset failures through `onSourceError` and dispose because +their incremental result can no longer be kept complete. + +When a source change invalidates an ordered window, automatic full-source +repair keeps the last complete snapshot visible. A failed repair exposes +`utils.lastSubsetError` and retries at most twice, after 250 ms and 500 ms. +Exhausting those retries does not put an already-ready query into a terminal +error state or clear its rows. The app can show the error and explicitly retry +with `setWindow()`. Cleanup or truncate cancels the old repair timer. Failed +imperative window moves and initial loads do not use this background retry. + +For SQLite-persisted on-demand collections, a failed upstream `loadSubset` +rejects even when hydration succeeded. Cached rows remain readable; their +availability does not mean the remote request succeeded. Background coordinator +retry, where supported, does not change the failed caller's outcome. + +When a must-refetch truncate cannot reload every active subset, a subscription +keeps its last successful snapshot and reports the subset error. It discards +the incomplete replay batch and keeps later source changes private because they +cannot prove a complete replacement. The next truncate retries every active +subset. Overlapping truncates form one atomic replay: all in-flight requests +settle, the newest attempt decides the result, and subscribers receive the +replacement only when that attempt succeeds. Cleanup rejects window moves that +are waiting for replay with `AbortError`. + ## Collection Status and Error States Collections track their status and transition between states: @@ -276,6 +334,18 @@ try { } ``` +Explicit cancellation is different from a mutation failure. If you call +`tx.rollback()` while `mutationFn` is pending, the rollback settles +`tx.isPersisted.promise` as rejected. A later result or rejection from that +mutation function is ignored: the outstanding `commit()` call resolves and +`tx.error` is not populated by that late rejection. Observe `isPersisted.promise` +when you need the transaction's outcome, including explicit cancellation. + +After the mutation function succeeds, a publication listener can still throw +while the completed transaction updates its collections. In that case +`commit()` rejects with the listener error, but `isPersisted.promise` resolves +and the transaction remains completed. This is not a persistence failure. + ## Collection Operation Errors ### Invalid Collection State @@ -418,7 +488,7 @@ try { ### Query Collection Sync Errors -Query collections handle sync errors gracefully and mark the collection as ready even on error to avoid blocking applications: +Query collections distinguish an initial load failure from a later refetch failure: ```ts import { queryCollectionOptions } from "@tanstack/query-db-collection" @@ -445,9 +515,11 @@ const todoCollection = createCollection( When sync errors occur: - Error is logged to console: `[QueryCollection] Error observing query...` -- Collection is marked as ready to prevent blocking the application -- Cached data remains available +- An initial failure marks the collection as `error` because no usable snapshot exists +- Readiness waits such as `preload()` and `toArrayWhenReady()` reject with the cause passed to `markError(error)` while the collection is in that initial error state +- A later refetch failure keeps the collection `ready` and preserves its cached data - Error tracking counters are updated (`lastError`, `errorCount`) +- A later successful refetch recovers an initial `error` collection to `ready`; a new readiness wait then resolves normally ### Sync Write Errors diff --git a/docs/guides/live-queries.md b/docs/guides/live-queries.md index 0780871a3b..afeef894b2 100644 --- a/docs/guides/live-queries.md +++ b/docs/guides/live-queries.md @@ -3,8 +3,6 @@ title: Live Queries id: live-queries --- -# TanStack DB Live Queries - TanStack DB provides a powerful, type-safe query system that allows you to fetch, filter, transform, and aggregate data from collections using a SQL-like fluent API. All queries are **live** by default, meaning they automatically update when the underlying data changes. The query system is built around an API similar to SQL query builders like Kysely or Drizzle where you chain methods together to compose your query. The query builder doesn't perform operations in the order of method calls - instead, it composes your query into an optimal incremental pipeline that gets compiled and executed efficiently. Each method returns a new query builder, allowing you to chain operations together. @@ -164,14 +162,15 @@ bindings and reactive updates, use live queries instead. In React, you can use the `useLiveQuery` hook: ```tsx -import { useLiveQuery } from '@tanstack/react-db' +import { eq, useLiveQuery } from '@tanstack/react-db' function UserList() { - const activeUsers = useLiveQuery((q) => - q - .from({ user: usersCollection }) - .where(({ user }) => eq(user.active, true)) - ) + const { data: activeUsers } = useLiveQuery({ + query: (q) => + q + .from({ user: usersCollection }) + .where(({ user }) => eq(user.active, true)), + }) return (
              @@ -206,7 +205,44 @@ export class UserListComponent { } ``` -> **Note:** React hooks (`useLiveQuery`, `useLiveInfiniteQuery`, `useLiveSuspenseQuery`) accept an optional dependency array parameter to re-execute queries when values change, similar to React's `useEffect`. See the [React Adapter documentation](../framework/react/overview#dependency-arrays) for details on when and how to use dependency arrays. +> **Note:** React hooks derive query identity from structured query IR by +> default. Dependency arrays are still accepted for backwards compatibility, +> but warn in development and will be removed in 1.0. Unhashable queries also +> warn and keep legacy mount-stable identity until 1.0; add `queryKey` to make +> captured opaque values reactive. See the [React Adapter +> documentation](../framework/react/overview#query-identity) for details. + +For server rendering and hydration, live query preloading transports the ordered +query result without implicitly serializing its source collections. Explicit +collection preloading still transports normalized collection rows. See the +[SSR and Hydration guide](./ssr.md). + +#### When React Needs a Query Key + +Use `queryKey` when the query contains opaque runtime logic that cannot be represented in structured IR, such as `.fn.where`, `.fn.select`, or `.fn.having`. The key becomes the explicit identity for that query: + +```tsx +function UserSearch({ search }: { search: string }) { + const { data } = useLiveQuery({ + queryKey: [usersCollection.id, 'search', search], + query: (q) => + q + .from({ user: usersCollection }) + .fn.where(({ user }) => + user.name.toLowerCase().includes(search.toLowerCase()) + ), + }) + + return
              {data.length} users
              +} +``` + +You can also provide a `queryKey` as a performance escape hatch for a very hot render path, but normal structured queries should omit it. + +React development builds detect both cases. Before 1.0, opaque, unhashable IR +warns and keeps legacy mount-stable identity; repeated expensive identity +derivation also warns once. Both warnings point to the same `queryKey` escape +hatch. For more details on framework integration, see the [React](../framework/react/overview), [Vue](../framework/vue/overview), and [Angular](../framework/angular/overview) adapter documentation. @@ -220,11 +256,12 @@ import { Suspense } from 'react' function UserList() { // This will suspend until data is ready - const { data } = useLiveSuspenseQuery((q) => - q - .from({ user: usersCollection }) - .where(({ user }) => eq(user.active, true)) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => + q + .from({ user: usersCollection }) + .where(({ user }) => eq(user.active, true)), + }) // data is always defined - no need for optional chaining return ( @@ -251,9 +288,9 @@ The key difference from `useLiveQuery` is that `data` is always defined (never ` ```tsx function UserStats() { - const { data } = useLiveSuspenseQuery((q) => - q.from({ user: usersCollection }) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ user: usersCollection }), + }) // TypeScript knows data is Array, not Array | undefined return
              Total users: {data.length}
              @@ -284,9 +321,9 @@ After the initial load, data updates stream in without re-suspending: ```tsx function UserList() { - const { data } = useLiveSuspenseQuery((q) => - q.from({ user: usersCollection }) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ user: usersCollection }), + }) // Suspends once during initial load // After that, data updates automatically when users change @@ -301,19 +338,18 @@ function UserList() { } ``` -#### Re-suspending on Dependency Changes +#### Re-suspending on Query Identity Changes -When dependencies change, the hook re-suspends to load new data: +When the derived query identity changes, the hook re-suspends to load new data: ```tsx function FilteredUsers({ minAge }: { minAge: number }) { - const { data } = useLiveSuspenseQuery( - (q) => + const { data } = useLiveSuspenseQuery({ + query: (q) => q .from({ user: usersCollection }) .where(({ user }) => gt(user.age, minAge)), - [minAge] // Re-suspend when minAge changes - ) + }) return (
                @@ -334,7 +370,7 @@ function FilteredUsers({ minAge }: { minAge: number }) { - The query always needs to run (not conditional) - **Use `useLiveQuery`** when: - - You need conditional/disabled queries + - You prefer conditional rendering for optional query inputs - You prefer handling loading/error states within your component - You want to show loading states inline without Suspense - You need access to `status` and `isLoading` flags @@ -343,9 +379,9 @@ function FilteredUsers({ minAge }: { minAge: number }) { ```tsx // useLiveQuery - handle states in component function UserList() { - const { data, status, isLoading } = useLiveQuery((q) => - q.from({ user: usersCollection }) - ) + const { data, status, isLoading } = useLiveQuery({ + query: (q) => q.from({ user: usersCollection }), + }) if (isLoading) return
                Loading...
                if (status === 'error') return
                Error loading users
                @@ -355,9 +391,9 @@ function UserList() { // useLiveSuspenseQuery - handle states with Suspense/ErrorBoundary function UserList() { - const { data } = useLiveSuspenseQuery((q) => - q.from({ user: usersCollection }) - ) + const { data } = useLiveSuspenseQuery({ + query: (q) => q.from({ user: usersCollection }), + }) return
                  {data.map(user =>
                • {user.name}
                  • )}
                } @@ -377,9 +413,9 @@ const route = { // In your component: function UserList() { // Collection is already loaded, so data is immediately available - const { data } = useLiveQuery((q) => - q.from({ user: usersCollection }) - ) + const { data } = useLiveQuery({ + query: (q) => q.from({ user: usersCollection }), + }) return
                  {data?.map(user =>
                • {user.name}
                  • )}
                } @@ -387,28 +423,28 @@ function UserList() { ### Conditional Queries -In React, you can conditionally disable a query by returning `undefined` or `null` from the `useLiveQuery` callback. When disabled, the hook returns a special state indicating the query is not active. +For optional inputs, prefer rendering the query component only after the inputs exist. That avoids creating a live query before all required values exist. ```tsx import { useLiveQuery } from '@tanstack/react-db' -function TodoList({ userId }: { userId?: string }) { - const { data, isEnabled, status } = useLiveQuery((q) => { - // Disable the query when userId is not available - if (!userId) return undefined +function TodosPanel({ userId }: { userId?: string }) { + if (!userId) return
                Please select a user
                - return q - .from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.userId, userId)) - }, [userId]) + return +} - if (!isEnabled) { - return
                Please select a user
                - } +function TodoList({ userId }: { userId: string }) { + const { data } = useLiveQuery({ + query: (q) => + q + .from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.userId, userId)), + }) return (
                  - {data?.map(todo => ( + {data.map(todo => (
                • {todo.text}
                  • ))}
                @@ -416,32 +452,44 @@ function TodoList({ userId }: { userId?: string }) { } ``` -When the query is disabled (callback returns `undefined` or `null`): +The `query` callback can return `undefined` or `null` to disable a query. This still uses derived identity, so captured structured values do not need a dependency array: + +```tsx +const { data, isEnabled, status } = useLiveQuery({ + query: (q) => { + if (!userId) return undefined + + return q + .from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.userId, userId)) + }, +}) +``` + +The top-level callback form supports the same behavior. When the query is disabled: - `status` is `'disabled'` - `data`, `state`, and `collection` are `undefined` - `isEnabled` is `false` -- `isLoading`, `isReady`, `isIdle`, and `isError` are all `false` - -This pattern is useful for "wait until inputs exist" flows without needing to conditionally render the hook itself or manage an external enabled flag. +- `isReady` is `true` +- `isLoading`, `isIdle`, `isError`, and `isCleanedUp` are all `false` -### Alternative Callback Return Types - -The `useLiveQuery` callback can return different types depending on your use case: +### Alternative Input Forms #### Returning a Query Builder (Standard) -The most common pattern is to return a query builder: +The standard React pattern is an object with a query builder. For structured queries, React derives the identity from the query IR: ```tsx -const { data } = useLiveQuery((q) => - q.from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)) -) +const { data } = useLiveQuery({ + query: (q) => + q.from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)), +}) ``` #### Returning a Pre-created Collection -You can return an existing collection directly: +You can also subscribe to an existing collection directly: ```tsx const activeUsersCollection = createLiveQueryCollection((q) => @@ -449,15 +497,10 @@ const activeUsersCollection = createLiveQueryCollection((q) => .where(({ users }) => eq(users.active, true)) ) -function UserList({ usePrebuilt }: { usePrebuilt: boolean }) { - const { data } = useLiveQuery((q) => { - // Toggle between pre-created collection and ad-hoc query - if (usePrebuilt) return activeUsersCollection - - return q.from({ users: usersCollection }) - }, [usePrebuilt]) +function UserList() { + const { data } = useLiveQuery(activeUsersCollection) - return
                  {data?.map(user =>
                • {user.name}
                  • )}
                + return
                  {data.map(user =>
                • {user.name}
                  • )}
                } ``` @@ -466,13 +509,12 @@ function UserList({ usePrebuilt }: { usePrebuilt: boolean }) { You can return a configuration object to specify additional options like a custom ID: ```tsx -const { data } = useLiveQuery((q) => { - return { - query: q.from({ items: itemsCollection }) - .select(({ items }) => ({ id: items.id })), - id: 'items-view', // Custom ID for debugging - gcTime: 10000 // Custom garbage collection time - } +const { data } = useLiveQuery({ + query: (q) => + q.from({ items: itemsCollection }) + .select(({ items }) => ({ id: items.id })), + id: 'items-view', // Custom ID for debugging + gcTime: 10000 // Custom garbage collection time }) ``` @@ -728,6 +770,13 @@ not(condition) For a complete reference of all available functions, see the [Expression Functions Reference](#expression-functions-reference) section. +### Comparison semantics + +Comparisons follow SQL/PostgreSQL conventions rather than raw JavaScript: + +- **`null` / `undefined` use three-valued logic.** Any comparison involving `null` or `undefined` evaluates to `UNKNOWN`, so the row is not matched. For example `eq(user.score, null)` matches nothing — use a dedicated null check (e.g. `isUndefined`) to match missing values. +- **`NaN` follows PostgreSQL float semantics.** `NaN` is treated as equal to itself and greater than every other (non-null) value. So `eq(row.value, NaN)` matches `NaN` rows, `gt(row.value, x)` includes `NaN`, and ordering by such a field places `NaN` last. (Invalid `Date` values, whose timestamp is `NaN`, behave the same way.) This differs from JavaScript, where `NaN === NaN` is `false`, and matches how PostgreSQL orders and indexes floating-point values. + ## Select Use `select` to specify which fields to include in your results and transform your data. Without `select`, you get the full schema. @@ -1252,6 +1301,50 @@ const projectsWithIssues = createLiveQueryCollection((q) => With `toArray()`, the project row is re-emitted whenever its issues change. Without it, the child `Collection` updates independently. +### materialize + +`materialize()` is a single helper that covers both multi-row and single-row includes: + +- When the wrapped subquery returns multiple rows, the parent receives `Array` — same shape as `toArray()`. +- When the wrapped subquery ends in `.findOne()`, the parent receives `T | undefined` — a single object, or `undefined` when no child matches. + +This spares callers from unwrapping a singleton array whenever they know the child query yields at most one row. Reactive semantics match `toArray()`: the parent row is re-emitted whenever the underlying children change, including insert / update / delete transitions and rows moving in or out of a match. + +```ts +import { createLiveQueryCollection, eq, materialize } from '@tanstack/db' + +// Multi-row → issues: Array +const projectsWithIssues = createLiveQueryCollection((q) => + q.from({ p: projectsCollection }).select(({ p }) => ({ + ...p, + issues: materialize( + q + .from({ i: issuesCollection }) + .where(({ i }) => eq(i.projectId, p.id)), + ), + })), +) + +// Singleton → project: Project | undefined +const issuesWithProject = createLiveQueryCollection((q) => + q.from({ i: issuesCollection }).select(({ i }) => ({ + ...i, + project: materialize( + q + .from({ p: projectsCollection }) + .where(({ p }) => eq(p.id, i.projectId)) + .findOne(), + ), + })), +) +``` + +The singleton vs. array result type is inferred from whether the wrapped query ends in `.findOne()` — no extra type annotation is required. + +Like `toArray()`, `materialize()` is only valid as a top-level value in `.select()` — it cannot be nested inside expression helpers such as `coalesce()` or `eq()`. + +Do not return child queries, `toArray()`, `materialize()`, or query expressions such as `eq()` and `caseWhen()` from `.fn.select()`. Functional select callbacks run after the compiler builds the query graph, so they cannot add query operations to it. + ### Aggregates You can use aggregate functions in child queries. Aggregates are computed per parent: @@ -1311,19 +1404,20 @@ import { useLiveQuery } from '@tanstack/react-db' import { eq } from '@tanstack/db' function ProjectList() { - const { data: projects } = useLiveQuery((q) => - q.from({ p: projectsCollection }).select(({ p }) => ({ - id: p.id, - name: p.name, - issues: q - .from({ i: issuesCollection }) - .where(({ i }) => eq(i.projectId, p.id)) - .select(({ i }) => ({ - id: i.id, - title: i.title, - })), - })), - ) + const { data: projects } = useLiveQuery({ + query: (q) => + q.from({ p: projectsCollection }).select(({ p }) => ({ + id: p.id, + name: p.name, + issues: q + .from({ i: issuesCollection }) + .where(({ i }) => eq(i.projectId, p.id)) + .select(({ i }) => ({ + id: i.id, + title: i.title, + })), + })), + }) return (
                  @@ -1564,12 +1658,13 @@ import { useLiveQuery } from '@tanstack/react-db' import { eq } from '@tanstack/db' function UserProfile({ userId }: { userId: string }) { - const { data: user, isLoading } = useLiveQuery((q) => - q - .from({ users: usersCollection }) - .where(({ users }) => eq(users.id, userId)) - .findOne() - , [userId]) + const { data: user, isLoading } = useLiveQuery({ + query: (q) => + q + .from({ users: usersCollection }) + .where(({ users }) => eq(users.id, userId)) + .findOne(), + }) if (isLoading) return
                  Loading...
                  if (!user) return
                  User not found
                  @@ -1965,14 +2060,15 @@ You can chain multiple reusable filters: ```tsx import { useLiveQuery } from '@tanstack/react-db' -const { data } = useLiveQuery((q) => { - return q - .from({ item: itemsCollection }) - .where(({ item }) => eq(item.id, 1)) - .where(activeItemFilter) // Reusable filter 1 - .where(verifiedItemFilter) // Reusable filter 2 - .select(({ item }) => ({ ...item })) -}, []) +const { data } = useLiveQuery({ + query: (q) => + q + .from({ item: itemsCollection }) + .where(({ item }) => eq(item.id, 1)) + .where(activeItemFilter) // Reusable filter 1 + .where(verifiedItemFilter) // Reusable filter 2 + .select(({ item }) => ({ ...item })), +}) ``` #### Using with Different Aliases @@ -2350,7 +2446,9 @@ createEffect({ ### Using with React -The `useLiveQueryEffect` hook manages the effect lifecycle automatically — creating on mount, disposing on unmount, and recreating when dependencies change: +The `useLiveQueryEffect` hook manages the effect lifecycle automatically — +creating on mount, disposing on unmount, and recreating when effect dependencies +change: ```tsx import { useLiveQueryEffect } from '@tanstack/react-db' @@ -2375,7 +2473,10 @@ function ChatComponent({ channelId }: { channelId: string }) { } ``` -The second argument is a dependency array (like `useEffect`). When dependencies change, the old effect is disposed and a new one is created with the updated config. +The second argument is still a React-style dependency array for the effect +lifecycle. This is separate from `useLiveQuery` identity: React live query hooks +derive identity from structured IR by default and use `queryKey` only for opaque +or hot-path queries. ### Complete Example @@ -2551,6 +2652,54 @@ Add two numbers: add(user.salary, user.bonus) ``` +#### `subtract(left, right)` +Subtract two numbers: +```ts +subtract(user.salary, user.deductions) +``` + +#### `multiply(left, right)` +Multiply two numbers: +```ts +multiply(item.price, item.quantity) +``` + +#### `divide(left, right)` +Divide two numbers (returns `null` on divide-by-zero): +```ts +divide(order.total, order.itemCount) +``` + +#### Computed Columns in orderBy + +You can use math functions directly in `orderBy` to sort by computed values. This is useful for ranking algorithms that combine multiple factors: + +```ts +import { subtract, multiply, divide } from '@tanstack/db' + +// HN-style ranking: balance rating with recency +// Date.now() is captured when this query is created. Recreate the query if +// you need the recency score to advance as time passes. +const rankedRecipes = createLiveQueryCollection((q) => + q + .from({ r: recipesCollection }) + .orderBy( + ({ r }) => + subtract( + multiply(r.rating, r.timesMade), // weighted rating + divide( + subtract(Date.now(), r.lastMadeAt), // time since last made + 3600000 * 24 // convert ms to days + ) + ), + 'desc' + ) + .limit(20) +) +``` + +> **Note:** When using computed expressions in `orderBy` with `limit()`, lazy loading optimization is skipped (all matching data is loaded first, then sorted). For large collections where this matters, consider pre-computing the ranking score as a stored field. + ### Utility Functions #### `coalesce(...values)` @@ -2670,6 +2819,11 @@ The functional variant API provides an alternative to the standard API, offering ### Functional Select +> [!WARNING] +> `fn.select()` cannot consume Collection-valued includes, even when the callback ignores or passes through that field. This also applies to nested Collection-valued includes. Use `toArray()` or `materialize()` in the upstream `.select()` to provide inline child values. Keep these helpers outside the functional callback. + +Inline child updates rerun the functional projection. Arrays support JavaScript calculations, but do not expose Collection methods such as `get()`, `createIndex()`, or `subscribeChanges()`. To keep live child Collections, use standard `.select()`, or perform parent-only `.fn.select()` work before adding the child include. + > [!WARNING] > `fn.select()` cannot be used with `groupBy()`. The `groupBy` operator needs to statically analyze the `select` clause to discover which aggregate functions to compute, which is not possible with an opaque JavaScript function. Use the standard `.select()` API for grouped queries. diff --git a/docs/guides/mutations.md b/docs/guides/mutations.md index 9a60c47dc5..ddf6f17492 100644 --- a/docs/guides/mutations.md +++ b/docs/guides/mutations.md @@ -3,8 +3,6 @@ title: Mutations id: mutations --- -# TanStack DB Mutations - TanStack DB provides a powerful mutation system that enables optimistic updates with automatic state management. This system is built around a pattern of **optimistic mutation → backend persistence → sync back → confirmed state**. This creates a highly responsive user experience while maintaining data consistency and being easy to reason about. Local changes are applied immediately as optimistic state, then persisted to your backend, and finally the optimistic state is replaced by the confirmed server state once it syncs back. @@ -344,6 +342,33 @@ todoCollection.update( > [!IMPORTANT] > The `updater` function uses an Immer-like pattern to capture changes as immutable updates. You must not reassign the draft parameter itself—only mutate its properties. +Existing row values stay isolated from draft edits. New objects you assign or +add to a draft keep normal shared references during the synchronous callback: + +```ts +const tag = { label: 'new' } +todoCollection.update(todoId, (draft) => { + draft.tags.add(tag) // tags is a Set + tag.label = 'edited' // included in the update + for (const value of draft.tags) value.label = 'final' + // tag.label is now 'final' too +}) +tag.label = 'later' // does not change the stored row +``` + +The completed changes are copied when the callback returns. This applies to +new Map values, Set members, and objects assigned to draft properties. If you +need to keep a new caller-owned object unchanged during the callback, insert +your own copy. A thrown callback does not roll back edits to that caller-owned +object; it leaves existing collection data unchanged. + +Arbitrary class instances are an exception: newly assigned instances stay by +reference so their methods, prototypes, and private fields remain intact. +Later changes to such an instance can therefore affect stored data without a +new update or notification. Treat those instances as immutable, or convert them +to plain data before assignment when you need isolation. Supported native values +such as `URL`, `Date`, `RegExp`, and typed arrays are copied instead. + ### Delete Remove items from a collection: @@ -432,6 +457,12 @@ const todoCollection = createCollection({ > [!IMPORTANT] > Operation handlers must not resolve until the server changes have synced back to the collection. Different collection types provide different patterns to ensure this happens correctly. +> +> Do not call or await `collection.preload()`, live-query `preload()`, or a +> direct `loadSubset()` inside a mutation handler. The optimistic mutation is +> already applied when the handler starts. A preload may need a sync commit +> that is queued behind that same handler, which creates a deadlock. Use the +> collection adapter's documented mutation acknowledgement pattern instead. ### Collection-Specific Handler Patterns @@ -1653,9 +1684,9 @@ todoCollection.insert({ // Use view key for rendering const TodoList = () => { - const { data: todos } = useLiveQuery((q) => - q.from({ todo: todoCollection }) - ) + const { data: todos } = useLiveQuery({ + query: (q) => q.from({ todo: todoCollection }), + }) return (
                    diff --git a/docs/guides/schemas.md b/docs/guides/schemas.md index 4f453d45f0..9bcf22103d 100644 --- a/docs/guides/schemas.md +++ b/docs/guides/schemas.md @@ -3,8 +3,6 @@ title: Schemas id: schemas --- -# Schema Validation and Type Transformations - TanStack DB uses schemas to ensure your data is valid and type-safe throughout your application. ## What You'll Learn diff --git a/docs/guides/ssr.md b/docs/guides/ssr.md new file mode 100644 index 0000000000..4fb4a46a8c --- /dev/null +++ b/docs/guides/ssr.md @@ -0,0 +1,828 @@ +--- +title: SSR and Hydration +id: ssr +--- + +TanStack DB SSR transports the smallest useful snapshot for the work the server +performed: + +- Explicitly preloaded collections dehydrate as normalized collection rows. +- Preloaded or render-discovered live queries dehydrate as ordered query-result + snapshots, without serializing all of their source collections. + +The browser renders either snapshot immediately, starts its normal collection +sync and live-query pipeline, then atomically replaces a live-query snapshot +when the browser result becomes authoritative. + +## High-level Summary + +The SSR-friendly API adds six concepts: + +- `DbClient` owns materialized collection instances for one request, browser app, + test, or script. +- `collectionOptions(...)` creates a stable collection descriptor. Reusable + descriptors create fresh adapter config for each `DbClient`. +- `dbClient.dehydrate()`, `dbClient.hydrate(state)`, and + `dbClient.applyCollectionChunk(chunk)` move explicit collection state across + the server/client boundary. +- `dbClient.preloadLiveQuery(options)` captures only the ordered result of a + live query for hydration or streaming. +- React and Svelte apps use `DbProvider` so hooks can resolve collection + descriptors against the current client. +- `@tanstack/react-router-with-db` streams live queries discovered by Suspense + during a TanStack Start server render. + +Existing apps continue to work. `createCollection(...)` and direct collection +instances still exist. The migration is required when you want SSR-safe request +isolation, hydration, incremental chunks, Suspense streaming, or the 1.0-ready +React hook shape. + +The old dependency-array form now warns: + +```tsx +useLiveQuery((q) => q.from({ todos }).where(...), [status]) +``` + +It still works, but warns in development and will be removed in 1.0. Prefer: + +```tsx +useLiveQuery({ + query: (q) => q.from({ todos: todoCollection }).where(...), +}) +``` + +React derives live query identity from structured query IR by default. Add +`queryKey` only for opaque functional query logic or for a hot render path where +you want to skip derived identity work. + +## Cheat Sheet + +| Task | Before | SSR-friendly | +| --- | --- | --- | +| Define a collection | `createCollection(options)` | `collectionOptions(id, factory)` | +| Materialize a collection | module-level singleton | `dbClient.collection(todoCollection)` | +| Scope collection state | module lifetime | `new DbClient()` per request/browser/test | +| Provide React context | none | `` | +| Query from React | direct collection instance | descriptor in `from`, resolved by `DbProvider` | +| Mutate from React | import singleton collection | `useDbClient().collection(todoCollection)` | +| Server preload | ad hoc collection preload | `collection.preload()` or `dbClient.preloadLiveQuery(...)` | +| Serialize SSR state | none | `const state = dbClient.dehydrate()` | +| Hydrate in browser | none | `dbClient.hydrate(state)` before hooks read it | +| Apply rows incrementally | custom app state | `dbClient.applyCollectionChunk(chunk)` | +| Stream render-time results | none | `routerWithDbClient(router, dbClient)` | +| React query identity | dependency array | derived IR, or `queryKey` when needed | + +### Minimal React Pattern + +```tsx +import { + DbClient, + DbProvider, + collectionOptions, + eq, + useDbClient, + useLiveQuery, +} from '@tanstack/react-db' + +const todoCollection = collectionOptions('todos', () => ({ + id: 'todos', + getKey: (todo: Todo) => todo.id, + sync: { + sync: ({ markReady }) => { + markReady() + }, + }, +})) + +function useTodoCollection() { + return useDbClient().collection(todoCollection) +} + +function Todos({ status }: { status: string }) { + const todos = useTodoCollection() + + const { data } = useLiveQuery({ + query: (q) => + q + .from({ todo: todoCollection }) + .where(({ todo }) => eq(todo.status, status)), + }) + + return ( +
                      + {data.map((todo) => ( +
                    • todos.update(todo.id, (draft) => { + draft.done = true + })} + > + {todo.title} +
                      • + ))} +
                    + ) +} + +const dbClient = new DbClient() + +root.render( + + + +) +``` + +The factory matters when config contains mutable adapter state or closures. +Every `DbClient` gets a fresh config and collection instance. First-party +adapter option creators already attach an equivalent factory, so this is also +safe: + +```tsx +const todoCollection = collectionOptions( + localOnlyCollectionOptions({ + id: 'todos', + getKey: (todo) => todo.id, + }) +) +``` + +Within one `DbClient`, descriptors with the same `id` resolve to the same +collection. Only the first descriptor is materialized. Include every dynamic +parameter that changes the collection in its `id`. + +A descriptor created from an arbitrary concrete config can be materialized by +one `DbClient` only. Use the explicit factory form for custom adapters and +request-scoped dependencies. + +## SSR Flow + +The server and browser use the same descriptors, but different `DbClient` +instances. + +```txt +server request + -> new DbClient() + -> preload an explicit collection or live-query result + -> dbClient.dehydrate() + -> send state through framework loader + +browser + -> new DbClient() + -> dbClient.hydrate(loaderState) + -> + -> useLiveQuery({ query }) + -> start source sync + -> atomically replace any query snapshot with the live result +``` + +During React hydration, descriptor-backed queries read either hydrated +collection rows or their matching query-result snapshot for the first browser +render. Adapter sync and queued on-demand loads start when React commits the +external-store subscription, so the initial markup still matches the server. +The snapshot remains visible while the source is loading. Once the browser live +query is ready, DB publishes one handoff from the snapshot to the live result. + +### Server + +Create a fresh `DbClient` for each request. Materialize descriptors through that +client, preload the data needed for the route, and dehydrate the client. + +```tsx +import { DbClient, collectionOptions, eq } from '@tanstack/db' + +export const todoCollection = collectionOptions('todos', () => ({ + id: 'todos', + getKey: (todo: Todo) => todo.id, + syncMode: 'on-demand', + sync: { + sync: ({ markReady, begin, write, commit }) => { + markReady() + + return { + loadSubset: async () => { + const todos = await api.todos.list() + begin({ immediate: true }) + for (const todo of todos) { + write({ type: 'insert', value: todo }) + } + commit() + return true + }, + } + }, + }, +})) + +export async function loadTodosForSsr() { + const dbClient = new DbClient() + const todos = dbClient.collection(todoCollection) + await todos.preload() + + return dbClient.dehydrate() +} +``` + +This explicit collection preload dehydrates normalized source rows. Use it when +multiple browser queries need the same source data. + +If the source is much larger than the rendered result, preload the query instead: + +```tsx +const dbClient = new DbClient() + +await dbClient.preloadLiveQuery({ + query: (q) => + q + .from({ todo: todoCollection }) + .where(({ todo }) => eq(todo.status, 'open')) + .select(({ todo }) => ({ id: todo.id, title: todo.title })), +}) + +const state = dbClient.dehydrate() +``` + +This payload contains the projected query result and no source collection rows +unless that collection was also materialized explicitly. + +### Browser + +Hydrate the browser client before rendering components that read from DB. + +```tsx +import { + DbClient, + DbProvider, + HydrationBoundary, +} from '@tanstack/react-db' + +function App({ dehydratedDbState }: { dehydratedDbState: DehydratedDbState }) { + const [dbClient] = React.useState(() => new DbClient()) + + return ( + + + + + + ) +} +``` + +Frameworks differ in how loader data reaches the client, but the DB handoff is +the same: `DbClient` on the server, `dehydrate()`, then `hydrate()` into the +browser client. + +### Svelte + +Svelte resolves descriptors from its own `DbProvider` and reads hydrated query +snapshots synchronously during server rendering: + +```svelte + + + + + +``` + +Inside `Todos.svelte`, `useLiveQuery({ query })` can use collection descriptors +directly. The browser subscription starts source sync and performs the same +snapshot-to-live-result handoff as React. + +Live demo: https://tanstack-db-ssr-demo.netlify.app/ssr-db + +## Suspense Streaming with TanStack Start + +`@tanstack/react-router-with-db` follows the same integration pattern as +`@tanstack/react-router-with-query`: + +```tsx +import { DbClient } from '@tanstack/react-db' +import { createRouter } from '@tanstack/react-router' +import { routerWithDbClient } from '@tanstack/react-router-with-db' + +export type RouterContext = { + dbClient: DbClient +} + +export function getRouter() { + const dbClient = new DbClient() + const router = createRouter({ + routeTree, + context: { dbClient }, + }) + + return routerWithDbClient(router, dbClient) +} +``` + +The adapter adds `dbClient` to router context, wraps the app in `DbProvider`, +dehydrates critical state, and opens a stream for query results discovered later +during rendering. + +```tsx +function RouteComponent() { + return ( + Loading todos

                    }> + +                     + ) +} + +function TodoList() { + const { data } = useLiveSuspenseQuery({ + query: (q) => + q + .from({ todo: todoCollection }) + .where(({ todo }) => eq(todo.status, 'open')), + }) + + return data.map((todo) => ) +} +``` + +When `TodoList` suspends on the server, the adapter streams the pending query +promise. That promise resolves to the ordered live-query result snapshot inside +the streamed `DehydratedDbState`. The source collections and D2 graph do not +cross the wire. The browser shows the snapshot, starts the source collections +and live query normally, then replaces the snapshot when the browser result is +ready. + +The server and browser must derive the same live-query identity. Structured +queries do this automatically. An opaque query must provide a serializable +`queryKey`; render-time streaming throws if it cannot derive an identity. + +## Suspense Streaming with Next.js + +Next.js App Router can transport the same pending query promise through React +Server Components. Start the preload without awaiting it, dehydrate the pending +result, and pass that state to a client hydration boundary: + +```tsx +export default function Page() { + const dbClient = new DbClient() + void dbClient.preloadLiveQuery(openTodosQuery) + + const state = dbClient.dehydrate({ + shouldDehydrateCollection: () => false, + shouldDehydrateLiveQuery: () => true, + }) + + return ( + + Loading todos

                    }> + +                     +                     + ) +} +``` + +`DbHydration` is a client component that creates one browser `DbClient`, wraps +children in `DbProvider`, and passes `state` to `HydrationBoundary`. React streams +the promise result into that boundary. The full working integration is in +`examples/react/next-ssr-e2e`. + +## Incremental Collection Hydration + +Applications can also apply collection rows received through their own stream. +Incremental hydration uses the same collection chunk shape as holistic +dehydration: + +```ts +dbClient.applyCollectionChunk({ + collectionId: 'todos', + rows: [ + { + key: 'todo-1', + value: { + id: 'todo-1', + title: 'Streamed row', + status: 'open', + }, + metadata: { source: 'stream' }, + }, + ], + syncMeta: { version: 1, cursor: 'abc' }, +}) +``` + +If the target collection is already materialized, the rows apply immediately and +existing live queries react from collection state. If the collection is not +materialized yet, the chunk is stored and applied when that `collectionId` +materializes. + +## What Gets Serialized + +`dbClient.dehydrate()` can emit two independent snapshot types. + +Serialized: + +- explicit collection snapshots: collection id, synced row keys and values, row + metadata, and adapter sync metadata from `exportSyncMeta` +- live-query snapshots: query hash and ordered result rows; completed explicit + preloads are included by default, while framework integrations opt pending + promises into streaming + +Not serialized: + +- mutation handlers +- pending optimistic mutations +- pending subscriptions +- D2 graphs or compiled pipelines +- transaction stacks +- module-level runtime state +- source collection rows for a query-result snapshot, unless that collection was + also explicitly materialized for dehydration + +Choose the payload unit according to what the browser needs. Explicit collection +preloading preserves normalized rows for reuse across queries. Live-query +preloading avoids shipping a 50-100x larger source when the rendered projection +is small. Neither mode serializes executable query state. + +## Sync Metadata + +Adapters can participate in resumable sync with three optional hooks: + +```ts +type SyncConfig = { + exportSyncMeta?: () => unknown + importSyncMeta?: (meta: unknown) => void + mergeSyncMeta?: (current: unknown, incoming: unknown) => unknown +} +``` + +The metadata shape is adapter-owned. Version it inside the adapter payload. If an +adapter cannot understand incoming metadata, it should ignore it and restart +sync from a safe point. + +During hydration, DB imports `syncMeta` into the materialized collection. If the +collection already has current metadata, DB calls `mergeSyncMeta(current, +incoming)` when provided and imports the merged result. + +If an adapter does not implement sync metadata hooks, row snapshots still hydrate +and the adapter can restart sync normally. + +## Initial Data + +`initialData` is a startup seed, not a sync-ready signal. + +Before adapter sync starts, current `DbClient` precedence from lowest to highest +is: + +1. per-materialization `initialData` +2. persisted rows +3. hydrated rows + +Fresh adapter sync is authoritative over all three. Hydrated and initial rows +are provisional base state, so the adapter's first insert for the same key is +reconciled as an update instead of raising a duplicate-key error. + +Hydrated rows and `initialData` never mark adapter sync as ready by themselves. +The adapter still owns readiness through its sync lifecycle. + +## React Query Identity + +React hooks derive live query identity from structured query IR by default: + +```tsx +function Todos({ status }: { status: string }) { + return useLiveQuery({ + query: (q) => + q + .from({ todo: todoCollection }) + .where(({ todo }) => eq(todo.status, status)), + }) +} +``` + +The captured `status` value is represented in the structured IR, so no +dependency array or `queryKey` is required. + +Use `queryKey` when the query contains opaque runtime logic that DB cannot +stably represent: + +```tsx +function SearchTodos({ search }: { search: string }) { + return useLiveQuery({ + queryKey: [todoCollection.id, 'search', search], + query: (q) => + q + .from({ todo: todoCollection }) + .fn.where(({ todo }) => + todo.title.toLowerCase().includes(search.toLowerCase()) + ), + }) +} +``` + +Common reasons to add `queryKey`: + +- `.fn.where(...)` +- `.fn.select(...)` +- `.fn.having(...)` +- function values, symbols, class instances, or circular objects captured inside + the structured query +- a render path where derived identity becomes measurably expensive + +Before 1.0, DB warns when structured IR cannot be hashed and preserves the +legacy mount-stable identity. The query still works, but captured values inside +opaque logic are not reactive unless they are represented in `queryKey`. In 1.0, +an unhashable query without `queryKey` will throw. + +DB also warns once in development if deriving identity becomes expensive enough +that an explicit `queryKey` would be better. + +Dependency arrays are accepted for backwards compatibility: + +```tsx +useLiveQuery((q) => q.from({ todo: todoCollection }), [status]) +``` + +They warn in development and will be removed in 1.0. Migrate to the config +object form: + +```tsx +useLiveQuery({ + query: (q) => q.from({ todo: todoCollection }), +}) +``` + +Add `queryKey` only if the query uses opaque logic or trips the performance +warning. + +## Migration Guide + +### 1. Create descriptors instead of SSR singletons + +For collections that need SSR, replace module-level `createCollection(...)` +with a reusable `collectionOptions(...)` descriptor. + +```tsx +// Before +export const todoCollection = createCollection({ + id: 'todos', + getKey: (todo) => todo.id, + sync: todoSync, +}) + +// After +export const todoCollection = collectionOptions('todos', () => ({ + id: 'todos', + getKey: (todo: Todo) => todo.id, + sync: createTodoSync(), +})) +``` + +Put mutable state and closures inside the factory. First-party adapter option +creators can also be passed directly because they provide a fresh config +factory. Collections that never participate in SSR can keep using +`createCollection`. + +### 2. Add a `DbClient` + +Use a new client for every server request and a stable client for each browser +app instance. + +```tsx +const dbClient = new DbClient() +``` + +In tests, create a new client per test unless the test is explicitly covering +shared state. + +### 3. Wrap React with `DbProvider` + +```tsx +root.render( + + + +) +``` + +Hooks that resolve collection descriptors need this provider. Without it, DB +throws instead of falling back to hidden global state. + +### 4. Use collection hooks for imperative operations + +Use descriptors directly in live query sources, and materialize only when you +need collection methods: + +```tsx +function useTodoCollection() { + return useDbClient().collection(todoCollection) +} + +function TodoActions({ id }: { id: string }) { + const todos = useTodoCollection() + + return ( + + ) +} +``` + +This keeps request/client scoping in one place and avoids reintroducing +module-level collections. + +### 5. Replace dependency arrays + +Most queries can drop the dependency array entirely: + +```tsx +// Before +useLiveQuery( + (q) => + q + .from({ todo: todoCollection }) + .where(({ todo }) => eq(todo.status, status)), + [status], +) + +// After +useLiveQuery({ + query: (q) => + q + .from({ todo: todoCollection }) + .where(({ todo }) => eq(todo.status, status)), +}) +``` + +If the query uses opaque functional variants, add `queryKey`: + +```tsx +useLiveQuery({ + queryKey: [todoCollection.id, 'status-fn', status], + query: (q) => + q + .from({ todo: todoCollection }) + .fn.where(({ todo }) => todo.status === status), +}) +``` + +### 6. Preload and dehydrate on the server + +Preload a collection when the browser should receive normalized source rows: + +```tsx +const dbClient = new DbClient() +const todos = dbClient.collection(todoCollection) +await todos.preload() + +return { + dbState: dbClient.dehydrate(), +} +``` + +Preload a live query when the browser only needs the rendered result: + +```tsx +const dbClient = new DbClient() +await dbClient.preloadLiveQuery(openTodosQuery) + +return { + dbState: dbClient.dehydrate(), +} +``` + +### 7. Hydrate before client hooks read DB + +```tsx + + + + + +``` + +Imperative integrations can call `client.hydrate(loaderData.dbState)` before +rendering instead. + +## Compatibility + +No existing public API is removed by this change. + +Still supported: + +- `createCollection(...)` +- passing collection instances to `useLiveQuery(...)` +- `useLiveQuery(queryFn, deps)` +- `useLiveSuspenseQuery(queryFn, deps)` +- mutation APIs such as `insert`, `update`, `delete`, `subscribe`, and + optimistic mutation helpers + +Warnings: + +- React dependency arrays warn in development and will be removed in 1.0. +- Opaque query IR without `queryKey` warns in development and keeps legacy + mount-stable identity until 1.0. In 1.0 it will throw. +- Expensive derived identity warns in development and suggests `queryKey`. + +Required for SSR: + +- stable explicit collection ids +- request-scoped server `DbClient` +- browser-scoped client `DbClient` +- `DbProvider` for descriptor resolution in React +- `dehydrate()` on the server and `hydrate()` in the browser + +Required for render-time Suspense streaming: + +- `routerWithDbClient(router, dbClient)` +- `useLiveSuspenseQuery(...)` inside a Suspense boundary +- a stable derived query identity or explicit serializable `queryKey` + +## Detailed Changelog + +### Added + +- `DbClient` +- `collectionOptions(...)` +- `CollectionOptions` descriptor type +- `CollectionMaterializeOptions` +- `DehydratedDbState` +- `DehydratedCollectionChunk` +- `DehydratedCollectionRow` +- `dbClient.collection(descriptor, options?)` +- `dbClient.dehydrate()` +- `dbClient.hydrate(state)` +- `dbClient.applyCollectionChunk(chunk)` +- `dbClient.subscribe(listener)` +- `dbClient.createTransaction(config)` +- `dbClient.cleanup()` +- React `DbProvider` +- React `useDbClient()` +- React `useOptionalDbClient()` +- React `HydrationBoundary` +- React descriptor resolution inside live query builders +- React derived structured query identity +- React `queryKey` escape hatch for opaque or hot-path queries +- React per-query `client` override +- SSR-capable `useSyncExternalStore` server snapshot support +- `dbClient.preloadLiveQuery(...)` +- Svelte `DbProvider`, `useDbClient()`, descriptor resolution, and synchronous + server snapshot support +- TanStack Start and Next.js Playwright SSR E2E coverage +- `@tanstack/react-router-with-db` +- render-time `useLiveSuspenseQuery` promise streaming + +### Changed + +- React `useLiveQuery({ query })` can use collection descriptors directly in + `from`, `join`, `leftJoin`, and `unionAll` sources when a `DbProvider` is + present. +- React live query identity is derived from normalized structured IR when no + explicit `queryKey` or legacy dependency array is supplied. +- Explicit collection preloading serializes normalized collection rows. +- Live-query preloading and render-time discovery serialize ordered result + snapshots without implicitly serializing source collections. +- Browser observers keep the hydrated result visible while normal source sync + starts, then publish one authoritative handoff. +- Hydration applies rows as committed synced state without invoking mutation + handlers or creating optimistic state. +- Hydration and adapter sync begin in a deterministic order: pending rows and + sync metadata are imported before sync starts. +- `DbClient` owns collection instances and ambient transaction scope; cleanup + releases both. +- Incremental chunks use the same collection payload shape as full dehydration. +- Streamed live-query promises resolve to live-query result snapshots. + +### Deprecated + +- React dependency arrays for `useLiveQuery` and wrappers that delegate to it. + They still work and warn in development. They are planned for removal in 1.0. + +### Not Changed + +- `createCollection(...)` remains available. +- Direct collection runtime APIs remain available. +- Vue, Solid, and Angular keep their existing dependency/reactivity model until + they get their own SSR/client-provider work. Svelte is covered by this change. +- Query collection `queryKey` is still TanStack Query's cache key. It is + separate from React live query identity. + +## Validation + +The SSR strategy is covered by: + +- core `DbClient` tests for hydration, streaming chunks, sync metadata, + initial data precedence, explicit ids, and no optimistic serialization +- React tests for `DbProvider`, descriptor resolution, derived query identity, + `queryKey`, deprecation warnings, SSR result snapshots, and atomic handoff +- Svelte tests for provider ownership, server snapshot rendering, and browser + handoff +- query adapter tests to ensure Query cache behavior still holds +- persistence core tests to ensure persisted row behavior remains intact +- TanStack Start and Next.js Playwright E2Es that verify a Suspense fallback, + streamed query result, omitted source-only data, clean hydration, and atomic + replacement by browser sync diff --git a/docs/overview.md b/docs/overview.md index 63f0965d96..e5c5738795 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -3,8 +3,6 @@ title: Overview id: overview --- -# TanStack DB - Documentation - Welcome to the TanStack DB documentation. TanStack DB is the reactive client store for your API. It solves the problems of building fast, modern apps, helping you: @@ -35,6 +33,7 @@ It extends TanStack Query with collections, live queries and optimistic mutation ## Contents - [How it works](#how-it-works) — understand the TanStack DB development model and how the pieces fit together +- [SSR and hydration](./guides/ssr.md) — use `DbClient` to transport explicit collection rows or live-query result snapshots - [API reference](#api-reference) — for the primitives and function interfaces - [Usage examples](#usage-examples) — examples of common usage patterns - [More info](#more-info) — where to find support and more information @@ -48,21 +47,38 @@ TanStack DB works by: - [making optimistic mutations](#making-optimistic-mutations) using transactional mutators ```tsx -// Define collections to load data into -const todoCollection = createCollection({ +import { + DbClient, + DbProvider, + collectionOptions, + not, + useDbClient, + useLiveQuery, +} from '@tanstack/react-db' + +// Define stable collection descriptors to load data into +const todoCollection = collectionOptions('todos', () => ({ + id: 'todos', // ...your config onUpdate: updateMutationFn, -}) +})) + +function useTodoCollection() { + return useDbClient().collection(todoCollection) +} const Todos = () => { + const todosCollection = useTodoCollection() + // Bind data using live queries - const { data: todos } = useLiveQuery((q) => - q.from({ todo: todoCollection }).where(({ todo }) => not(todo.completed)) - ) + const { data: todos } = useLiveQuery({ + query: (q) => + q.from({ todo: todoCollection }).where(({ todo }) => not(todo.completed)), + }) const complete = (todo) => { // Instantly applies optimistic state - todoCollection.update(todo.id, (draft) => { + todosCollection.update(todo.id, (draft) => { draft.completed = true }) } @@ -77,6 +93,14 @@ const Todos = () => {
                  ) } + +const dbClient = new DbClient() + +const App = () => ( + + + +) ``` ### Defining collections @@ -103,14 +127,17 @@ Collections support three sync modes to optimize data loading: With on-demand mode, your component's query becomes the API call: ```tsx -const productsCollection = createCollection( +const productsCollection = collectionOptions('products', (client) => queryCollectionOptions({ + id: 'products', queryKey: ['products'], + queryClient: client.requireDependency('queryClient'), queryFn: async (ctx) => { // Query predicates passed automatically in ctx.meta const params = parseLoadSubsetOptions(ctx.meta?.loadSubsetOptions) return api.getProducts(params) // e.g., GET /api/products?category=electronics&price_lt=100 }, + getKey: (product) => product.id, syncMode: 'on-demand', // ← Enable query-driven sync }) ) @@ -143,17 +170,18 @@ Collections support `insert`, `update` and `delete` operations. When called, by ```ts // Define collection with persistence handlers -const todoCollection = createCollection({ +const todoCollection = collectionOptions('todos', () => ({ id: "todos", // ... other config onUpdate: async ({ transaction }) => { const { original, changes } = transaction.mutations[0] await api.todos.update(original.id, changes) }, -}) +})) +const todosCollection = dbClient.collection(todoCollection) // Immediately applies optimistic state -todoCollection.update(todo.id, (draft) => { +todosCollection.update(todo.id, (draft) => { draft.completed = true }) ``` @@ -227,12 +255,14 @@ const todoSchema = z.object({ priority: z.number().default(0) }) -const collection = createCollection( +const todoCollection = collectionOptions( queryCollectionOptions({ + id: "todos", schema: todoSchema, // ... }) ) +const collection = dbClient.collection(todoCollection) // Users provide simple inputs collection.insert({ @@ -269,16 +299,17 @@ import { useLiveQuery } from '@tanstack/react-db' import { eq } from '@tanstack/db' const Todos = () => { - const { data: todos } = useLiveQuery((q) => - q - .from({ todo: todoCollection }) - .where(({ todo }) => eq(todo.completed, false)) - .orderBy(({ todo }) => todo.created_at, 'asc') - .select(({ todo }) => ({ - id: todo.id, - text: todo.text - })) - ) + const { data: todos } = useLiveQuery({ + query: (q) => + q + .from({ todo: todoCollection }) + .where(({ todo }) => eq(todo.completed, false)) + .orderBy(({ todo }) => todo.created_at, 'asc') + .select(({ todo }) => ({ + id: todo.id, + text: todo.text + })), + }) return } @@ -291,21 +322,22 @@ import { useLiveQuery } from '@tanstack/react-db' import { eq } from '@tanstack/db' const Todos = () => { - const { data: todos } = useLiveQuery((q) => - q - .from({ todos: todoCollection }) - .join( - { lists: listCollection }, - ({ todos, lists }) => eq(lists.id, todos.listId), - 'inner' - ) - .where(({ lists }) => eq(lists.active, true)) - .select(({ todos, lists }) => ({ - id: todos.id, - title: todos.title, - listName: lists.name - })) - ) + const { data: todos } = useLiveQuery({ + query: (q) => + q + .from({ todos: todoCollection }) + .join( + { lists: listCollection }, + ({ todos, lists }) => eq(lists.id, todos.listId), + 'inner' + ) + .where(({ lists }) => eq(lists.active, true)) + .select(({ todos, lists }) => ({ + id: todos.id, + title: todos.title, + listName: lists.name + })), + }) return } @@ -321,11 +353,12 @@ import { Suspense } from 'react' const Todos = () => { // data is always defined - no need for optional chaining - const { data: todos } = useLiveSuspenseQuery((q) => - q - .from({ todo: todoCollection }) - .where(({ todo }) => eq(todo.completed, false)) - ) + const { data: todos } = useLiveSuspenseQuery({ + query: (q) => + q + .from({ todo: todoCollection }) + .where(({ todo }) => eq(todo.completed, false)), + }) return } @@ -397,15 +430,26 @@ The steps are to: 2. implement mutation handlers that handle mutations by posting them to your API endpoints ```tsx -import { useLiveQuery, createCollection } from "@tanstack/react-db" +import { + DbClient, + DbProvider, + collectionOptions, + useLiveQuery, +} from "@tanstack/react-db" import { queryCollectionOptions } from "@tanstack/query-db-collection" +import { QueryClient } from "@tanstack/query-core" + +const queryClient = new QueryClient() +const dbClient = new DbClient({ queryClient }) // Load data into collections using TanStack Query. // It's common to define these in a `collections` module. -const todoCollection = createCollection( +const todoCollection = collectionOptions("todos", (client) => queryCollectionOptions({ + id: "todos", queryKey: ["todos"], - queryFn: async () => fetch("/api/todos"), + queryClient: client.requireDependency("queryClient"), + queryFn: async () => fetch("/api/todos").then((response) => response.json()), getKey: (item) => item.id, schema: todoSchema, // any standard schema onInsert: async ({ transaction }) => { @@ -417,10 +461,13 @@ const todoCollection = createCollection( // also add onUpdate, onDelete as needed. }) ) -const listCollection = createCollection( +const listCollection = collectionOptions("todo-lists", (client) => queryCollectionOptions({ + id: "todo-lists", queryKey: ["todo-lists"], - queryFn: async () => fetch("/api/todo-lists"), + queryClient: client.requireDependency("queryClient"), + queryFn: async () => + fetch("/api/todo-lists").then((response) => response.json()), getKey: (item) => item.id, schema: todoListSchema, onInsert: async ({ transaction }) => { @@ -436,25 +483,32 @@ const listCollection = createCollection( const Todos = () => { // Read the data using live queries. Here we show a live // query that joins across two collections. - const { data: todos } = useLiveQuery((q) => - q - .from({ todo: todoCollection }) - .join( - { list: listCollection }, - ({ todo, list }) => eq(list.id, todo.list_id), - "inner" - ) - .where(({ list }) => eq(list.active, true)) - .select(({ todo, list }) => ({ - id: todo.id, - text: todo.text, - status: todo.status, - listName: list.name, - })) - ) + const { data: todos } = useLiveQuery({ + query: (q) => + q + .from({ todo: todoCollection }) + .join( + { list: listCollection }, + ({ todo, list }) => eq(list.id, todo.list_id), + "inner" + ) + .where(({ list }) => eq(list.active, true)) + .select(({ todo, list }) => ({ + id: todo.id, + text: todo.text, + status: todo.status, + listName: list.name, + })), + }) // ... } + +const App = () => ( + + + +) ``` This pattern allows you to extend an existing TanStack Query application, or any application built on a REST API, with blazing fast, cross-collection live queries and local optimistic mutations with automatically managed optimistic state. @@ -476,15 +530,14 @@ This pattern enables the "load everything once" approach that makes apps like Li Here, we illustrate this pattern using [ElectricSQL](https://electric-sql.com) as the sync engine, but this pattern also works with other sync engines like [PowerSync](https://www.powersync.com/?utm_source=tanstack&utm_campaign=tanstack_partner), [RxDB](https://rxdb.info/), and [TrailBase](https://trailbase.io/). ```tsx -import type { Collection } from "@tanstack/db" import type { MutationFn, PendingMutation, - createCollection, } from "@tanstack/react-db" +import { collectionOptions, useDbClient } from "@tanstack/react-db" import { electricCollectionOptions } from "@tanstack/electric-db-collection" -export const todoCollection = createCollection( +export const todoCollection = collectionOptions( electricCollectionOptions({ id: "todos", schema: todoSchema, @@ -497,7 +550,6 @@ export const todoCollection = createCollection( }, }, getKey: (item) => item.id, - schema: todoSchema, onInsert: async ({ transaction }) => { const response = await api.todos.create(transaction.mutations[0].modified) @@ -508,9 +560,11 @@ export const todoCollection = createCollection( ) const AddTodo = () => { + const todosCollection = useDbClient().collection(todoCollection) + return (
                ) } + +function App() { + return ( + + + + ) +} ``` You now have collections, live queries, and optimistic mutations! Let's break this down further. +If you are building with SSR, see the [SSR and Hydration guide](./guides/ssr.md) +after this quick start. The short version is that SSR apps use stable +`collectionOptions(...)` descriptors, materialize them through a request-scoped +`DbClient` on the server, then hydrate a browser `DbClient` with explicit +collection rows or a preloaded live-query result before React hooks read from +DB. + ## Installation ```bash -npm install @tanstack/react-db @tanstack/query-db-collection +npm install @tanstack/react-db @tanstack/query-db-collection @tanstack/query-core ``` ## 1. Create a Collection @@ -72,9 +107,11 @@ npm install @tanstack/react-db @tanstack/query-db-collection Collections store your data and handle persistence. The `queryCollectionOptions` loads data using TanStack Query and defines mutation handlers for server sync: ```tsx -const todoCollection = createCollection( +const todoCollection = collectionOptions('todos', (client) => queryCollectionOptions({ + id: 'todos', queryKey: ['todos'], + queryClient: client.requireDependency('queryClient'), queryFn: async () => { const response = await fetch('/api/todos') return response.json() @@ -103,41 +140,58 @@ const todoCollection = createCollection( ) ``` -## 2. Query with Live Queries +The `queryKey` above is TanStack Query's cache key for loading the collection. +React live queries below derive their own identity from structured query IR. + +## 2. Materialize the Collection -Live queries reactively update when data changes. They support filtering, sorting, joins, and transformations: +Use the `DbClient` from context to materialize the descriptor. A tiny collection hook keeps components from repeating the client lookup: + +```tsx +function useTodoCollection() { + return useDbClient().collection(todoCollection) +} +``` + +## 3. Query with Live Queries + +Live queries reactively update when data changes. They support filtering, sorting, joins, and transformations. React hooks derive query identity from the structured query by default, so normal builder queries do not need a separate `queryKey`: ```tsx function TodoList() { // Basic filtering and sorting - const { data: incompleteTodos } = useLiveQuery((q) => - q.from({ todo: todoCollection }) - .where(({ todo }) => eq(todo.completed, false)) - .orderBy(({ todo }) => todo.createdAt, 'desc') - ) + const { data: incompleteTodos } = useLiveQuery({ + query: (q) => + q.from({ todo: todoCollection }) + .where(({ todo }) => eq(todo.completed, false)) + .orderBy(({ todo }) => todo.createdAt, 'desc'), + }) // Transform the data - const { data: todoSummary } = useLiveQuery((q) => - q.from({ todo: todoCollection }) - .select(({ todo }) => ({ - id: todo.id, - summary: `${todo.text} (${todo.completed ? 'done' : 'pending'})`, - priority: todo.priority || 'normal' - })) - ) + const { data: todoSummary } = useLiveQuery({ + query: (q) => + q.from({ todo: todoCollection }) + .select(({ todo }) => ({ + id: todo.id, + summary: `${todo.text} (${todo.completed ? 'done' : 'pending'})`, + priority: todo.priority || 'normal' + })), + }) return
                {/* Render todos */}
                } ``` -## 3. Optimistic Mutations +## 4. Optimistic Mutations Mutations apply instantly and sync to your server. If the server request fails, changes automatically roll back: ```tsx function TodoActions({ todo }) { + const todosCollection = useTodoCollection() + const addTodo = () => { - todoCollection.insert({ + todosCollection.insert({ id: crypto.randomUUID(), text: 'New todo', completed: false, @@ -146,19 +200,19 @@ function TodoActions({ todo }) { } const toggleComplete = () => { - todoCollection.update(todo.id, (draft) => { + todosCollection.update(todo.id, (draft) => { draft.completed = !draft.completed }) } const updateText = (newText) => { - todoCollection.update(todo.id, (draft) => { + todosCollection.update(todo.id, (draft) => { draft.text = newText }) } const deleteTodo = () => { - todoCollection.delete(todo.id) + todosCollection.delete(todo.id) } return ( diff --git a/docs/reference/@tanstack/namespaces/IR/classes/Aggregate.md b/docs/reference/@tanstack/namespaces/IR/classes/Aggregate.md index 19066baac0..8bbb5e1827 100644 --- a/docs/reference/@tanstack/namespaces/IR/classes/Aggregate.md +++ b/docs/reference/@tanstack/namespaces/IR/classes/Aggregate.md @@ -5,7 +5,7 @@ title: Aggregate # Class: Aggregate\ -Defined in: [packages/db/src/query/ir.ts:129](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L129) +Defined in: [packages/db/src/query/ir.ts:174](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L174) ## Extends @@ -25,7 +25,7 @@ Defined in: [packages/db/src/query/ir.ts:129](https://github.com/TanStack/db/blo new Aggregate(name, args): Aggregate; ``` -Defined in: [packages/db/src/query/ir.ts:131](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L131) +Defined in: [packages/db/src/query/ir.ts:176](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L176) #### Parameters @@ -55,7 +55,7 @@ BaseExpression.constructor readonly __returnType: T; ``` -Defined in: [packages/db/src/query/ir.ts:73](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L73) +Defined in: [packages/db/src/query/ir.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L84) **`Internal`** @@ -75,7 +75,7 @@ BaseExpression.__returnType args: BasicExpression[]; ``` -Defined in: [packages/db/src/query/ir.ts:133](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L133) +Defined in: [packages/db/src/query/ir.ts:178](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L178) *** @@ -85,7 +85,7 @@ Defined in: [packages/db/src/query/ir.ts:133](https://github.com/TanStack/db/blo name: string; ``` -Defined in: [packages/db/src/query/ir.ts:132](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L132) +Defined in: [packages/db/src/query/ir.ts:177](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L177) *** @@ -95,7 +95,7 @@ Defined in: [packages/db/src/query/ir.ts:132](https://github.com/TanStack/db/blo type: "agg"; ``` -Defined in: [packages/db/src/query/ir.ts:130](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L130) +Defined in: [packages/db/src/query/ir.ts:175](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L175) #### Overrides diff --git a/docs/reference/@tanstack/namespaces/IR/classes/CollectionRef.md b/docs/reference/@tanstack/namespaces/IR/classes/CollectionRef.md index 416a1c83e7..82b1fcb3ee 100644 --- a/docs/reference/@tanstack/namespaces/IR/classes/CollectionRef.md +++ b/docs/reference/@tanstack/namespaces/IR/classes/CollectionRef.md @@ -5,7 +5,7 @@ title: CollectionRef # Class: CollectionRef -Defined in: [packages/db/src/query/ir.ts:76](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L76) +Defined in: [packages/db/src/query/ir.ts:87](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L87) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/query/ir.ts:76](https://github.com/TanStack/db/blob new CollectionRef(collection, alias): CollectionRef; ``` -Defined in: [packages/db/src/query/ir.ts:78](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L78) +Defined in: [packages/db/src/query/ir.ts:91](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L91) #### Parameters @@ -49,7 +49,7 @@ BaseExpression.constructor readonly __returnType: any; ``` -Defined in: [packages/db/src/query/ir.ts:73](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L73) +Defined in: [packages/db/src/query/ir.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L84) **`Internal`** @@ -69,7 +69,7 @@ BaseExpression.__returnType alias: string; ``` -Defined in: [packages/db/src/query/ir.ts:80](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L80) +Defined in: [packages/db/src/query/ir.ts:93](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L93) *** @@ -79,7 +79,19 @@ Defined in: [packages/db/src/query/ir.ts:80](https://github.com/TanStack/db/blob collection: CollectionImpl; ``` -Defined in: [packages/db/src/query/ir.ts:79](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L79) +Defined in: [packages/db/src/query/ir.ts:92](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L92) + +*** + +### sourceId + +```ts +readonly sourceId: string; +``` + +Defined in: [packages/db/src/query/ir.ts:90](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L90) + +Opaque runtime identity; aliases are lexical names only. *** @@ -89,7 +101,7 @@ Defined in: [packages/db/src/query/ir.ts:79](https://github.com/TanStack/db/blob type: "collectionRef"; ``` -Defined in: [packages/db/src/query/ir.ts:77](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L77) +Defined in: [packages/db/src/query/ir.ts:88](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L88) #### Overrides diff --git a/docs/reference/@tanstack/namespaces/IR/classes/ConditionalSelect.md b/docs/reference/@tanstack/namespaces/IR/classes/ConditionalSelect.md new file mode 100644 index 0000000000..ddf556f662 --- /dev/null +++ b/docs/reference/@tanstack/namespaces/IR/classes/ConditionalSelect.md @@ -0,0 +1,98 @@ +--- +id: ConditionalSelect +title: ConditionalSelect +--- + +# Class: ConditionalSelect + +Defined in: [packages/db/src/query/ir.ts:212](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L212) + +## Extends + +- `BaseExpression` + +## Constructors + +### Constructor + +```ts +new ConditionalSelect(branches, defaultValue?): ConditionalSelect; +``` + +Defined in: [packages/db/src/query/ir.ts:214](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L214) + +#### Parameters + +##### branches + +[`ConditionalSelectBranch`](../type-aliases/ConditionalSelectBranch.md)[] + +##### defaultValue? + +[`SelectValueExpression`](../type-aliases/SelectValueExpression.md) + +#### Returns + +`ConditionalSelect` + +#### Overrides + +```ts +BaseExpression.constructor +``` + +## Properties + +### \_\_returnType + +```ts +readonly __returnType: any; +``` + +Defined in: [packages/db/src/query/ir.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L84) + +**`Internal`** + +- Type brand for TypeScript inference + +#### Inherited from + +```ts +BaseExpression.__returnType +``` + +*** + +### branches + +```ts +branches: ConditionalSelectBranch[]; +``` + +Defined in: [packages/db/src/query/ir.ts:215](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L215) + +*** + +### defaultValue? + +```ts +optional defaultValue: SelectValueExpression; +``` + +Defined in: [packages/db/src/query/ir.ts:216](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L216) + +*** + +### type + +```ts +type: "conditionalSelect"; +``` + +Defined in: [packages/db/src/query/ir.ts:213](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L213) + +#### Overrides + +```ts +BaseExpression.type +``` diff --git a/docs/reference/@tanstack/namespaces/IR/classes/Func.md b/docs/reference/@tanstack/namespaces/IR/classes/Func.md index 4d6fb79848..143b486b8b 100644 --- a/docs/reference/@tanstack/namespaces/IR/classes/Func.md +++ b/docs/reference/@tanstack/namespaces/IR/classes/Func.md @@ -5,7 +5,7 @@ title: Func # Class: Func\ -Defined in: [packages/db/src/query/ir.ts:114](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L114) +Defined in: [packages/db/src/query/ir.ts:159](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L159) ## Extends @@ -25,7 +25,7 @@ Defined in: [packages/db/src/query/ir.ts:114](https://github.com/TanStack/db/blo new Func(name, args): Func; ``` -Defined in: [packages/db/src/query/ir.ts:116](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L116) +Defined in: [packages/db/src/query/ir.ts:161](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L161) #### Parameters @@ -55,7 +55,7 @@ BaseExpression.constructor readonly __returnType: T; ``` -Defined in: [packages/db/src/query/ir.ts:73](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L73) +Defined in: [packages/db/src/query/ir.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L84) **`Internal`** @@ -75,7 +75,7 @@ BaseExpression.__returnType args: BasicExpression[]; ``` -Defined in: [packages/db/src/query/ir.ts:118](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L118) +Defined in: [packages/db/src/query/ir.ts:163](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L163) *** @@ -85,7 +85,7 @@ Defined in: [packages/db/src/query/ir.ts:118](https://github.com/TanStack/db/blo name: string; ``` -Defined in: [packages/db/src/query/ir.ts:117](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L117) +Defined in: [packages/db/src/query/ir.ts:162](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L162) *** @@ -95,7 +95,7 @@ Defined in: [packages/db/src/query/ir.ts:117](https://github.com/TanStack/db/blo type: "func"; ``` -Defined in: [packages/db/src/query/ir.ts:115](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L115) +Defined in: [packages/db/src/query/ir.ts:160](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L160) #### Overrides diff --git a/docs/reference/@tanstack/namespaces/IR/classes/IncludesSubquery.md b/docs/reference/@tanstack/namespaces/IR/classes/IncludesSubquery.md index 8d932c9ead..47d02489d4 100644 --- a/docs/reference/@tanstack/namespaces/IR/classes/IncludesSubquery.md +++ b/docs/reference/@tanstack/namespaces/IR/classes/IncludesSubquery.md @@ -5,7 +5,7 @@ title: IncludesSubquery # Class: IncludesSubquery -Defined in: [packages/db/src/query/ir.ts:139](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L139) +Defined in: [packages/db/src/query/ir.ts:184](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L184) ## Extends @@ -27,7 +27,7 @@ new IncludesSubquery( scalarField?): IncludesSubquery; ``` -Defined in: [packages/db/src/query/ir.ts:141](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L141) +Defined in: [packages/db/src/query/ir.ts:186](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L186) #### Parameters @@ -81,7 +81,7 @@ BaseExpression.constructor readonly __returnType: any; ``` -Defined in: [packages/db/src/query/ir.ts:73](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L73) +Defined in: [packages/db/src/query/ir.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L84) **`Internal`** @@ -101,7 +101,7 @@ BaseExpression.__returnType childCorrelationField: PropRef; ``` -Defined in: [packages/db/src/query/ir.ts:144](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L144) +Defined in: [packages/db/src/query/ir.ts:189](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L189) *** @@ -111,7 +111,7 @@ Defined in: [packages/db/src/query/ir.ts:144](https://github.com/TanStack/db/blo correlationField: PropRef; ``` -Defined in: [packages/db/src/query/ir.ts:143](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L143) +Defined in: [packages/db/src/query/ir.ts:188](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L188) *** @@ -121,7 +121,7 @@ Defined in: [packages/db/src/query/ir.ts:143](https://github.com/TanStack/db/blo fieldName: string; ``` -Defined in: [packages/db/src/query/ir.ts:145](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L145) +Defined in: [packages/db/src/query/ir.ts:190](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L190) *** @@ -131,7 +131,7 @@ Defined in: [packages/db/src/query/ir.ts:145](https://github.com/TanStack/db/blo materialization: IncludesMaterialization; ``` -Defined in: [packages/db/src/query/ir.ts:148](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L148) +Defined in: [packages/db/src/query/ir.ts:193](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L193) *** @@ -141,7 +141,7 @@ Defined in: [packages/db/src/query/ir.ts:148](https://github.com/TanStack/db/blo optional parentFilters: Where[]; ``` -Defined in: [packages/db/src/query/ir.ts:146](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L146) +Defined in: [packages/db/src/query/ir.ts:191](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L191) *** @@ -151,7 +151,7 @@ Defined in: [packages/db/src/query/ir.ts:146](https://github.com/TanStack/db/blo optional parentProjection: PropRef[]; ``` -Defined in: [packages/db/src/query/ir.ts:147](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L147) +Defined in: [packages/db/src/query/ir.ts:192](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L192) *** @@ -161,7 +161,7 @@ Defined in: [packages/db/src/query/ir.ts:147](https://github.com/TanStack/db/blo query: QueryIR; ``` -Defined in: [packages/db/src/query/ir.ts:142](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L142) +Defined in: [packages/db/src/query/ir.ts:187](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L187) *** @@ -171,7 +171,7 @@ Defined in: [packages/db/src/query/ir.ts:142](https://github.com/TanStack/db/blo optional scalarField: string; ``` -Defined in: [packages/db/src/query/ir.ts:149](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L149) +Defined in: [packages/db/src/query/ir.ts:194](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L194) *** @@ -181,7 +181,7 @@ Defined in: [packages/db/src/query/ir.ts:149](https://github.com/TanStack/db/blo type: "includesSubquery"; ``` -Defined in: [packages/db/src/query/ir.ts:140](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L140) +Defined in: [packages/db/src/query/ir.ts:185](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L185) #### Overrides diff --git a/docs/reference/@tanstack/namespaces/IR/classes/PropRef.md b/docs/reference/@tanstack/namespaces/IR/classes/PropRef.md index f9dd66cf12..f96b2714dc 100644 --- a/docs/reference/@tanstack/namespaces/IR/classes/PropRef.md +++ b/docs/reference/@tanstack/namespaces/IR/classes/PropRef.md @@ -5,7 +5,7 @@ title: PropRef # Class: PropRef\ -Defined in: [packages/db/src/query/ir.ts:96](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L96) +Defined in: [packages/db/src/query/ir.ts:141](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L141) ## Extends @@ -25,7 +25,7 @@ Defined in: [packages/db/src/query/ir.ts:96](https://github.com/TanStack/db/blob new PropRef(path): PropRef; ``` -Defined in: [packages/db/src/query/ir.ts:98](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L98) +Defined in: [packages/db/src/query/ir.ts:143](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L143) #### Parameters @@ -51,7 +51,7 @@ BaseExpression.constructor readonly __returnType: T; ``` -Defined in: [packages/db/src/query/ir.ts:73](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L73) +Defined in: [packages/db/src/query/ir.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L84) **`Internal`** @@ -71,7 +71,7 @@ BaseExpression.__returnType path: string[]; ``` -Defined in: [packages/db/src/query/ir.ts:99](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L99) +Defined in: [packages/db/src/query/ir.ts:144](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L144) *** @@ -81,7 +81,7 @@ Defined in: [packages/db/src/query/ir.ts:99](https://github.com/TanStack/db/blob type: "ref"; ``` -Defined in: [packages/db/src/query/ir.ts:97](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L97) +Defined in: [packages/db/src/query/ir.ts:142](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L142) #### Overrides diff --git a/docs/reference/@tanstack/namespaces/IR/classes/QueryRef.md b/docs/reference/@tanstack/namespaces/IR/classes/QueryRef.md index 1dbc0357e9..11eaa1f87b 100644 --- a/docs/reference/@tanstack/namespaces/IR/classes/QueryRef.md +++ b/docs/reference/@tanstack/namespaces/IR/classes/QueryRef.md @@ -5,7 +5,7 @@ title: QueryRef # Class: QueryRef -Defined in: [packages/db/src/query/ir.ts:86](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L86) +Defined in: [packages/db/src/query/ir.ts:103](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L103) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/query/ir.ts:86](https://github.com/TanStack/db/blob new QueryRef(query, alias): QueryRef; ``` -Defined in: [packages/db/src/query/ir.ts:88](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L88) +Defined in: [packages/db/src/query/ir.ts:105](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L105) #### Parameters @@ -49,7 +49,7 @@ BaseExpression.constructor readonly __returnType: any; ``` -Defined in: [packages/db/src/query/ir.ts:73](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L73) +Defined in: [packages/db/src/query/ir.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L84) **`Internal`** @@ -69,7 +69,7 @@ BaseExpression.__returnType alias: string; ``` -Defined in: [packages/db/src/query/ir.ts:90](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L90) +Defined in: [packages/db/src/query/ir.ts:107](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L107) *** @@ -79,7 +79,7 @@ Defined in: [packages/db/src/query/ir.ts:90](https://github.com/TanStack/db/blob query: QueryIR; ``` -Defined in: [packages/db/src/query/ir.ts:89](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L89) +Defined in: [packages/db/src/query/ir.ts:106](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L106) *** @@ -89,7 +89,7 @@ Defined in: [packages/db/src/query/ir.ts:89](https://github.com/TanStack/db/blob type: "queryRef"; ``` -Defined in: [packages/db/src/query/ir.ts:87](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L87) +Defined in: [packages/db/src/query/ir.ts:104](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L104) #### Overrides diff --git a/docs/reference/@tanstack/namespaces/IR/classes/UnionAll.md b/docs/reference/@tanstack/namespaces/IR/classes/UnionAll.md new file mode 100644 index 0000000000..c6683168a9 --- /dev/null +++ b/docs/reference/@tanstack/namespaces/IR/classes/UnionAll.md @@ -0,0 +1,105 @@ +--- +id: UnionAll +title: UnionAll +--- + +# Class: UnionAll + +Defined in: [packages/db/src/query/ir.ts:124](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L124) + +## Extends + +- `BaseExpression` + +## Constructors + +### Constructor + +```ts +new UnionAll(queries): UnionAll; +``` + +Defined in: [packages/db/src/query/ir.ts:132](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L132) + +Result-level UNION ALL. Downstream query clauses see the union result row +shape, not the branch source aliases. Optimizers may push safe operations +into branches, but compiler phases should treat this as a derived relation +unless they are explicitly handling branch lowering. + +#### Parameters + +##### queries + +[`QueryIR`](../interfaces/QueryIR.md)[] + +#### Returns + +`UnionAll` + +#### Overrides + +```ts +BaseExpression.constructor +``` + +## Properties + +### \_\_returnType + +```ts +readonly __returnType: any; +``` + +Defined in: [packages/db/src/query/ir.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L84) + +**`Internal`** + +- Type brand for TypeScript inference + +#### Inherited from + +```ts +BaseExpression.__returnType +``` + +*** + +### queries + +```ts +queries: QueryIR[]; +``` + +Defined in: [packages/db/src/query/ir.ts:132](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L132) + +*** + +### type + +```ts +type: "unionAll"; +``` + +Defined in: [packages/db/src/query/ir.ts:125](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L125) + +#### Overrides + +```ts +BaseExpression.type +``` + +## Accessors + +### alias + +#### Get Signature + +```ts +get alias(): string; +``` + +Defined in: [packages/db/src/query/ir.ts:136](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L136) + +##### Returns + +`string` diff --git a/docs/reference/@tanstack/namespaces/IR/classes/UnionFrom.md b/docs/reference/@tanstack/namespaces/IR/classes/UnionFrom.md new file mode 100644 index 0000000000..288c9925fa --- /dev/null +++ b/docs/reference/@tanstack/namespaces/IR/classes/UnionFrom.md @@ -0,0 +1,100 @@ +--- +id: UnionFrom +title: UnionFrom +--- + +# Class: UnionFrom + +Defined in: [packages/db/src/query/ir.ts:113](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L113) + +## Extends + +- `BaseExpression` + +## Constructors + +### Constructor + +```ts +new UnionFrom(sources): UnionFrom; +``` + +Defined in: [packages/db/src/query/ir.ts:115](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L115) + +#### Parameters + +##### sources + +([`CollectionRef`](CollectionRef.md) \| [`QueryRef`](QueryRef.md))[] + +#### Returns + +`UnionFrom` + +#### Overrides + +```ts +BaseExpression.constructor +``` + +## Properties + +### \_\_returnType + +```ts +readonly __returnType: any; +``` + +Defined in: [packages/db/src/query/ir.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L84) + +**`Internal`** + +- Type brand for TypeScript inference + +#### Inherited from + +```ts +BaseExpression.__returnType +``` + +*** + +### sources + +```ts +sources: (CollectionRef | QueryRef)[]; +``` + +Defined in: [packages/db/src/query/ir.ts:115](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L115) + +*** + +### type + +```ts +type: "unionFrom"; +``` + +Defined in: [packages/db/src/query/ir.ts:114](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L114) + +#### Overrides + +```ts +BaseExpression.type +``` + +## Accessors + +### alias + +#### Get Signature + +```ts +get alias(): string; +``` + +Defined in: [packages/db/src/query/ir.ts:119](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L119) + +##### Returns + +`string` diff --git a/docs/reference/@tanstack/namespaces/IR/classes/Value.md b/docs/reference/@tanstack/namespaces/IR/classes/Value.md index fd1970ef7a..ce8a9f480e 100644 --- a/docs/reference/@tanstack/namespaces/IR/classes/Value.md +++ b/docs/reference/@tanstack/namespaces/IR/classes/Value.md @@ -5,7 +5,7 @@ title: Value # Class: Value\ -Defined in: [packages/db/src/query/ir.ts:105](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L105) +Defined in: [packages/db/src/query/ir.ts:150](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L150) ## Extends @@ -25,7 +25,7 @@ Defined in: [packages/db/src/query/ir.ts:105](https://github.com/TanStack/db/blo new Value(value): Value; ``` -Defined in: [packages/db/src/query/ir.ts:107](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L107) +Defined in: [packages/db/src/query/ir.ts:152](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L152) #### Parameters @@ -51,7 +51,7 @@ BaseExpression.constructor readonly __returnType: T; ``` -Defined in: [packages/db/src/query/ir.ts:73](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L73) +Defined in: [packages/db/src/query/ir.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L84) **`Internal`** @@ -71,7 +71,7 @@ BaseExpression.__returnType type: "val"; ``` -Defined in: [packages/db/src/query/ir.ts:106](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L106) +Defined in: [packages/db/src/query/ir.ts:151](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L151) #### Overrides @@ -87,4 +87,4 @@ BaseExpression.type value: T; ``` -Defined in: [packages/db/src/query/ir.ts:108](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L108) +Defined in: [packages/db/src/query/ir.ts:153](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L153) diff --git a/docs/reference/@tanstack/namespaces/IR/functions/collectCollectionSources.md b/docs/reference/@tanstack/namespaces/IR/functions/collectCollectionSources.md new file mode 100644 index 0000000000..87659358f1 --- /dev/null +++ b/docs/reference/@tanstack/namespaces/IR/functions/collectCollectionSources.md @@ -0,0 +1,24 @@ +--- +id: collectCollectionSources +title: collectCollectionSources +--- + +# Function: collectCollectionSources() + +```ts +function collectCollectionSources(query): CollectionRef[]; +``` + +Defined in: [packages/db/src/query/ir.ts:266](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L266) + +Returns each lexical Collection source in a query tree once. + +## Parameters + +### query + +[`QueryIR`](../interfaces/QueryIR.md) + +## Returns + +[`CollectionRef`](../classes/CollectionRef.md)[] diff --git a/docs/reference/@tanstack/namespaces/IR/functions/createResidualWhere.md b/docs/reference/@tanstack/namespaces/IR/functions/createResidualWhere.md index e4c71d86f1..ff4a9394f2 100644 --- a/docs/reference/@tanstack/namespaces/IR/functions/createResidualWhere.md +++ b/docs/reference/@tanstack/namespaces/IR/functions/createResidualWhere.md @@ -9,7 +9,7 @@ title: createResidualWhere function createResidualWhere(expression): Where; ``` -Defined in: [packages/db/src/query/ir.ts:208](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L208) +Defined in: [packages/db/src/query/ir.ts:353](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L353) Create a residual Where clause from an expression diff --git a/docs/reference/@tanstack/namespaces/IR/functions/followRef.md b/docs/reference/@tanstack/namespaces/IR/functions/followRef.md index 5d2636f8c3..66092c7805 100644 --- a/docs/reference/@tanstack/namespaces/IR/functions/followRef.md +++ b/docs/reference/@tanstack/namespaces/IR/functions/followRef.md @@ -12,12 +12,14 @@ function followRef( collection): | void | { + alias?: string; collection: Collection; path: string[]; + sourceId?: string; }; ``` -Defined in: [packages/db/src/query/ir.ts:234](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L234) +Defined in: [packages/db/src/query/ir.ts:392](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L392) Follows the given reference in a query until its finds the root field the reference points to. @@ -40,8 +42,14 @@ until its finds the root field the reference points to. \| `void` \| \{ + `alias?`: `string`; `collection`: [`Collection`](../../../../interfaces/Collection.md); `path`: `string`[]; + `sourceId?`: `string`; \} -The collection, its alias, and the path to the root field in this collection +The collection, its alias, and the path to the root field in this collection. +`alias` is the alias under which the resolved collection is referenced in the +query it was reached from (when the ref crosses into a joined source). It is +left undefined when the ref simply resolves to a field on the passed-in +`collection`, in which case the caller already knows the alias. diff --git a/docs/reference/@tanstack/namespaces/IR/functions/getFromSources.md b/docs/reference/@tanstack/namespaces/IR/functions/getFromSources.md new file mode 100644 index 0000000000..e424c3bfcf --- /dev/null +++ b/docs/reference/@tanstack/namespaces/IR/functions/getFromSources.md @@ -0,0 +1,28 @@ +--- +id: getFromSources +title: getFromSources +--- + +# Function: getFromSources() + +```ts +function getFromSources(from): ( + | CollectionRef + | QueryRef)[]; +``` + +Defined in: [packages/db/src/query/ir.ts:360](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L360) + +Sources declared by a FROM clause. UnionAll branches own their sources. + +## Parameters + +### from + +[`From`](../type-aliases/From.md) + +## Returns + +( + \| [`CollectionRef`](../classes/CollectionRef.md) + \| [`QueryRef`](../classes/QueryRef.md))[] diff --git a/docs/reference/@tanstack/namespaces/IR/functions/getHavingExpression.md b/docs/reference/@tanstack/namespaces/IR/functions/getHavingExpression.md index 1ce31a66a9..03f683b02a 100644 --- a/docs/reference/@tanstack/namespaces/IR/functions/getHavingExpression.md +++ b/docs/reference/@tanstack/namespaces/IR/functions/getHavingExpression.md @@ -11,7 +11,7 @@ function getHavingExpression(having): | Aggregate; ``` -Defined in: [packages/db/src/query/ir.ts:186](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L186) +Defined in: [packages/db/src/query/ir.ts:331](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L331) Extract the expression from a HAVING clause HAVING clauses can contain aggregates, unlike regular WHERE clauses diff --git a/docs/reference/@tanstack/namespaces/IR/functions/getWhereExpression.md b/docs/reference/@tanstack/namespaces/IR/functions/getWhereExpression.md index fbf8d49943..52ab731ea5 100644 --- a/docs/reference/@tanstack/namespaces/IR/functions/getWhereExpression.md +++ b/docs/reference/@tanstack/namespaces/IR/functions/getWhereExpression.md @@ -9,7 +9,7 @@ title: getWhereExpression function getWhereExpression(where): BasicExpression; ``` -Defined in: [packages/db/src/query/ir.ts:176](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L176) +Defined in: [packages/db/src/query/ir.ts:321](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L321) Extract the expression from a Where clause diff --git a/docs/reference/@tanstack/namespaces/IR/functions/isExpressionLike.md b/docs/reference/@tanstack/namespaces/IR/functions/isExpressionLike.md index 07961d0d13..e2230d1f1c 100644 --- a/docs/reference/@tanstack/namespaces/IR/functions/isExpressionLike.md +++ b/docs/reference/@tanstack/namespaces/IR/functions/isExpressionLike.md @@ -9,7 +9,7 @@ title: isExpressionLike function isExpressionLike(value): boolean; ``` -Defined in: [packages/db/src/query/ir.ts:159](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L159) +Defined in: [packages/db/src/query/ir.ts:226](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L226) Runtime helper to detect IR expression-like objects. Prefer this over ad-hoc local implementations to keep behavior consistent. diff --git a/docs/reference/@tanstack/namespaces/IR/functions/isResidualWhere.md b/docs/reference/@tanstack/namespaces/IR/functions/isResidualWhere.md index ff3b6b22f8..71a24e3f1f 100644 --- a/docs/reference/@tanstack/namespaces/IR/functions/isResidualWhere.md +++ b/docs/reference/@tanstack/namespaces/IR/functions/isResidualWhere.md @@ -9,7 +9,7 @@ title: isResidualWhere function isResidualWhere(where): boolean; ``` -Defined in: [packages/db/src/query/ir.ts:197](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L197) +Defined in: [packages/db/src/query/ir.ts:342](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L342) Check if a Where clause is marked as residual diff --git a/docs/reference/@tanstack/namespaces/IR/index.md b/docs/reference/@tanstack/namespaces/IR/index.md index 415a365542..65b969139d 100644 --- a/docs/reference/@tanstack/namespaces/IR/index.md +++ b/docs/reference/@tanstack/namespaces/IR/index.md @@ -9,10 +9,13 @@ title: IR - [Aggregate](classes/Aggregate.md) - [CollectionRef](classes/CollectionRef.md) +- [ConditionalSelect](classes/ConditionalSelect.md) - [Func](classes/Func.md) - [IncludesSubquery](classes/IncludesSubquery.md) - [PropRef](classes/PropRef.md) - [QueryRef](classes/QueryRef.md) +- [UnionAll](classes/UnionAll.md) +- [UnionFrom](classes/UnionFrom.md) - [Value](classes/Value.md) ## Interfaces @@ -23,6 +26,7 @@ title: IR ## Type Aliases - [BasicExpression](type-aliases/BasicExpression.md) +- [ConditionalSelectBranch](type-aliases/ConditionalSelectBranch.md) - [From](type-aliases/From.md) - [GroupBy](type-aliases/GroupBy.md) - [Having](type-aliases/Having.md) @@ -34,6 +38,7 @@ title: IR - [OrderByClause](type-aliases/OrderByClause.md) - [OrderByDirection](type-aliases/OrderByDirection.md) - [Select](type-aliases/Select.md) +- [SelectValueExpression](type-aliases/SelectValueExpression.md) - [Where](type-aliases/Where.md) ## Variables @@ -42,8 +47,10 @@ title: IR ## Functions +- [collectCollectionSources](functions/collectCollectionSources.md) - [createResidualWhere](functions/createResidualWhere.md) - [followRef](functions/followRef.md) +- [getFromSources](functions/getFromSources.md) - [getHavingExpression](functions/getHavingExpression.md) - [getWhereExpression](functions/getWhereExpression.md) - [isExpressionLike](functions/isExpressionLike.md) diff --git a/docs/reference/@tanstack/namespaces/IR/interfaces/JoinClause.md b/docs/reference/@tanstack/namespaces/IR/interfaces/JoinClause.md index e41a4e7153..d41b72cace 100644 --- a/docs/reference/@tanstack/namespaces/IR/interfaces/JoinClause.md +++ b/docs/reference/@tanstack/namespaces/IR/interfaces/JoinClause.md @@ -5,7 +5,7 @@ title: JoinClause # Interface: JoinClause -Defined in: [packages/db/src/query/ir.ts:40](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L40) +Defined in: [packages/db/src/query/ir.ts:49](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L49) ## Properties @@ -17,7 +17,7 @@ from: | QueryRef; ``` -Defined in: [packages/db/src/query/ir.ts:41](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L41) +Defined in: [packages/db/src/query/ir.ts:50](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L50) *** @@ -27,7 +27,7 @@ Defined in: [packages/db/src/query/ir.ts:41](https://github.com/TanStack/db/blob left: BasicExpression; ``` -Defined in: [packages/db/src/query/ir.ts:43](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L43) +Defined in: [packages/db/src/query/ir.ts:52](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L52) *** @@ -37,7 +37,7 @@ Defined in: [packages/db/src/query/ir.ts:43](https://github.com/TanStack/db/blob right: BasicExpression; ``` -Defined in: [packages/db/src/query/ir.ts:44](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L44) +Defined in: [packages/db/src/query/ir.ts:53](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L53) *** @@ -47,4 +47,4 @@ Defined in: [packages/db/src/query/ir.ts:44](https://github.com/TanStack/db/blob type: "inner" | "left" | "right" | "full" | "outer" | "cross"; ``` -Defined in: [packages/db/src/query/ir.ts:42](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L42) +Defined in: [packages/db/src/query/ir.ts:51](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L51) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/BasicExpression.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/BasicExpression.md index de4676e5b7..f4f56c185b 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/BasicExpression.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/BasicExpression.md @@ -12,7 +12,7 @@ type BasicExpression = | Func; ``` -Defined in: [packages/db/src/query/ir.ts:127](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L127) +Defined in: [packages/db/src/query/ir.ts:172](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L172) ## Type Parameters diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/ConditionalSelectBranch.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/ConditionalSelectBranch.md new file mode 100644 index 0000000000..db4dc5ddf3 --- /dev/null +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/ConditionalSelectBranch.md @@ -0,0 +1,32 @@ +--- +id: ConditionalSelectBranch +title: ConditionalSelectBranch +--- + +# Type Alias: ConditionalSelectBranch + +```ts +type ConditionalSelectBranch = object; +``` + +Defined in: [packages/db/src/query/ir.ts:200](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L200) + +## Properties + +### condition + +```ts +condition: BasicExpression; +``` + +Defined in: [packages/db/src/query/ir.ts:201](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L201) + +*** + +### value + +```ts +value: SelectValueExpression; +``` + +Defined in: [packages/db/src/query/ir.ts:202](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L202) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/From.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/From.md index f5dc25e149..4b863b7a95 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/From.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/From.md @@ -8,7 +8,9 @@ title: From ```ts type From = | CollectionRef - | QueryRef; + | QueryRef + | UnionFrom + | UnionAll; ``` -Defined in: [packages/db/src/query/ir.ts:32](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L32) +Defined in: [packages/db/src/query/ir.ts:36](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L36) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/GroupBy.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/GroupBy.md index 1ecd399968..e17b6a1c8d 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/GroupBy.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/GroupBy.md @@ -9,4 +9,4 @@ title: GroupBy type GroupBy = BasicExpression[]; ``` -Defined in: [packages/db/src/query/ir.ts:51](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L51) +Defined in: [packages/db/src/query/ir.ts:60](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L60) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/Having.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/Having.md index 7654798e3a..2f042ce4dd 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/Having.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/Having.md @@ -9,4 +9,4 @@ title: Having type Having = Where; ``` -Defined in: [packages/db/src/query/ir.ts:53](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L53) +Defined in: [packages/db/src/query/ir.ts:62](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L62) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/IncludesMaterialization.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/IncludesMaterialization.md index 02afe1663a..578786d243 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/IncludesMaterialization.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/IncludesMaterialization.md @@ -6,7 +6,7 @@ title: IncludesMaterialization # Type Alias: IncludesMaterialization ```ts -type IncludesMaterialization = "collection" | "array" | "concat"; +type IncludesMaterialization = "collection" | "array" | "singleton" | "concat"; ``` Defined in: [packages/db/src/query/ir.ts:28](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L28) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/Join.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/Join.md index 53b84b8b49..0e6c1d1b90 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/Join.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/Join.md @@ -9,4 +9,4 @@ title: Join type Join = JoinClause[]; ``` -Defined in: [packages/db/src/query/ir.ts:38](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L38) +Defined in: [packages/db/src/query/ir.ts:47](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L47) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/Limit.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/Limit.md index 38be20e012..b571ea0440 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/Limit.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/Limit.md @@ -9,4 +9,4 @@ title: Limit type Limit = number; ``` -Defined in: [packages/db/src/query/ir.ts:64](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L64) +Defined in: [packages/db/src/query/ir.ts:73](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L73) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/Offset.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/Offset.md index 771c53d41a..a95aa02514 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/Offset.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/Offset.md @@ -9,4 +9,4 @@ title: Offset type Offset = number; ``` -Defined in: [packages/db/src/query/ir.ts:66](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L66) +Defined in: [packages/db/src/query/ir.ts:75](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L75) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderBy.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderBy.md index 2d5a450cdf..14ce76dc14 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderBy.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderBy.md @@ -9,4 +9,4 @@ title: OrderBy type OrderBy = OrderByClause[]; ``` -Defined in: [packages/db/src/query/ir.ts:55](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L55) +Defined in: [packages/db/src/query/ir.ts:64](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L64) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderByClause.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderByClause.md index 2770aed38b..987cbf522b 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderByClause.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderByClause.md @@ -9,7 +9,7 @@ title: OrderByClause type OrderByClause = object; ``` -Defined in: [packages/db/src/query/ir.ts:57](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L57) +Defined in: [packages/db/src/query/ir.ts:66](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L66) ## Properties @@ -19,7 +19,7 @@ Defined in: [packages/db/src/query/ir.ts:57](https://github.com/TanStack/db/blob compareOptions: CompareOptions; ``` -Defined in: [packages/db/src/query/ir.ts:59](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L59) +Defined in: [packages/db/src/query/ir.ts:68](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L68) *** @@ -29,4 +29,4 @@ Defined in: [packages/db/src/query/ir.ts:59](https://github.com/TanStack/db/blob expression: BasicExpression; ``` -Defined in: [packages/db/src/query/ir.ts:58](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L58) +Defined in: [packages/db/src/query/ir.ts:67](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L67) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderByDirection.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderByDirection.md index 58c632a106..7466f4b263 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderByDirection.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/OrderByDirection.md @@ -9,4 +9,4 @@ title: OrderByDirection type OrderByDirection = "asc" | "desc"; ``` -Defined in: [packages/db/src/query/ir.ts:62](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L62) +Defined in: [packages/db/src/query/ir.ts:71](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L71) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/Select.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/Select.md index 4016ef6f4d..928566fefc 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/Select.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/Select.md @@ -9,14 +9,15 @@ title: Select type Select = object; ``` -Defined in: [packages/db/src/query/ir.ts:34](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L34) +Defined in: [packages/db/src/query/ir.ts:38](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L38) ## Index Signature ```ts [alias: string]: - | BasicExpression | Select + | BasicExpression | Aggregate | IncludesSubquery + | ConditionalSelect ``` diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/SelectValueExpression.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/SelectValueExpression.md new file mode 100644 index 0000000000..b4b0342801 --- /dev/null +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/SelectValueExpression.md @@ -0,0 +1,17 @@ +--- +id: SelectValueExpression +title: SelectValueExpression +--- + +# Type Alias: SelectValueExpression + +```ts +type SelectValueExpression = + | BasicExpression + | Aggregate + | Select + | IncludesSubquery + | ConditionalSelect; +``` + +Defined in: [packages/db/src/query/ir.ts:205](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L205) diff --git a/docs/reference/@tanstack/namespaces/IR/type-aliases/Where.md b/docs/reference/@tanstack/namespaces/IR/type-aliases/Where.md index fdbb878874..f80a99c4e7 100644 --- a/docs/reference/@tanstack/namespaces/IR/type-aliases/Where.md +++ b/docs/reference/@tanstack/namespaces/IR/type-aliases/Where.md @@ -14,4 +14,4 @@ type Where = }; ``` -Defined in: [packages/db/src/query/ir.ts:47](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L47) +Defined in: [packages/db/src/query/ir.ts:56](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L56) diff --git a/docs/reference/@tanstack/namespaces/IR/variables/INCLUDES_SCALAR_FIELD.md b/docs/reference/@tanstack/namespaces/IR/variables/INCLUDES_SCALAR_FIELD.md index c68802f241..ea124c83e6 100644 --- a/docs/reference/@tanstack/namespaces/IR/variables/INCLUDES_SCALAR_FIELD.md +++ b/docs/reference/@tanstack/namespaces/IR/variables/INCLUDES_SCALAR_FIELD.md @@ -9,4 +9,4 @@ title: INCLUDES_SCALAR_FIELD const INCLUDES_SCALAR_FIELD: "__includes_scalar__"; ``` -Defined in: [packages/db/src/query/ir.ts:30](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L30) +Defined in: [packages/db/src/query/ir.ts:34](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir.ts#L34) diff --git a/docs/reference/classes/AggregateFunctionNotInSelectError.md b/docs/reference/classes/AggregateFunctionNotInSelectError.md index 19e08fb682..29ffa13f63 100644 --- a/docs/reference/classes/AggregateFunctionNotInSelectError.md +++ b/docs/reference/classes/AggregateFunctionNotInSelectError.md @@ -5,7 +5,7 @@ title: AggregateFunctionNotInSelectError # Class: AggregateFunctionNotInSelectError -Defined in: [packages/db/src/errors.ts:614](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L614) +Defined in: [packages/db/src/errors.ts:661](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L661) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:614](https://github.com/TanStack/db/blob/ new AggregateFunctionNotInSelectError(functionName): AggregateFunctionNotInSelectError; ``` -Defined in: [packages/db/src/errors.ts:615](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L615) +Defined in: [packages/db/src/errors.ts:662](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L662) #### Parameters diff --git a/docs/reference/classes/BTreeIndex.md b/docs/reference/classes/BTreeIndex.md index 3072cef868..83be1797c4 100644 --- a/docs/reference/classes/BTreeIndex.md +++ b/docs/reference/classes/BTreeIndex.md @@ -5,7 +5,7 @@ title: BTreeIndex # Class: BTreeIndex\ -Defined in: [packages/db/src/indexes/btree-index.ts:35](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L35) +Defined in: [packages/db/src/indexes/btree-index.ts:44](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L44) B+Tree index for sorted data with range queries This maintains items in sorted order and provides efficient range operations @@ -32,7 +32,7 @@ new BTreeIndex( options?): BTreeIndex; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:55](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L55) +Defined in: [packages/db/src/indexes/btree-index.ts:67](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L67) #### Parameters @@ -68,7 +68,7 @@ Defined in: [packages/db/src/indexes/btree-index.ts:55](https://github.com/TanSt protected compareOptions: CompareOptions; ``` -Defined in: [packages/db/src/indexes/base-index.ts:92](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L92) +Defined in: [packages/db/src/indexes/base-index.ts:122](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L122) #### Inherited from @@ -82,7 +82,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:92](https://github.com/TanSta readonly expression: BasicExpression; ``` -Defined in: [packages/db/src/indexes/base-index.ts:86](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L86) +Defined in: [packages/db/src/indexes/base-index.ts:120](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L120) #### Inherited from @@ -90,45 +90,34 @@ Defined in: [packages/db/src/indexes/base-index.ts:86](https://github.com/TanSta *** -### id +### hasCustomComparator ```ts -readonly id: number; +protected hasCustomComparator: boolean = false; ``` -Defined in: [packages/db/src/indexes/base-index.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L84) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`id`](BaseIndex.md#id) - -*** - -### lastUpdated - -```ts -protected lastUpdated: Date; -``` +Defined in: [packages/db/src/indexes/base-index.ts:128](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L128) -Defined in: [packages/db/src/indexes/base-index.ts:91](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L91) +Set by subclasses when constructed with a user-supplied comparator, whose +ordering may not match the WHERE evaluator's relational operators. #### Inherited from -[`BaseIndex`](BaseIndex.md).[`lastUpdated`](BaseIndex.md#lastupdated) +[`BaseIndex`](BaseIndex.md).[`hasCustomComparator`](BaseIndex.md#hascustomcomparator) *** -### lookupCount +### id ```ts -protected lookupCount: number = 0; +readonly id: number; ``` -Defined in: [packages/db/src/indexes/base-index.ts:89](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L89) +Defined in: [packages/db/src/indexes/base-index.ts:118](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L118) #### Inherited from -[`BaseIndex`](BaseIndex.md).[`lookupCount`](BaseIndex.md#lookupcount) +[`BaseIndex`](BaseIndex.md).[`id`](BaseIndex.md#id) *** @@ -138,7 +127,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:89](https://github.com/TanSta readonly optional name: string; ``` -Defined in: [packages/db/src/indexes/base-index.ts:85](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L85) +Defined in: [packages/db/src/indexes/base-index.ts:119](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L119) #### Inherited from @@ -152,48 +141,14 @@ Defined in: [packages/db/src/indexes/base-index.ts:85](https://github.com/TanSta readonly supportedOperations: Set<"eq" | "gt" | "gte" | "lt" | "lte" | "in" | "like" | "ilike">; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:38](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L38) +Defined in: [packages/db/src/indexes/btree-index.ts:47](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L47) #### Overrides [`BaseIndex`](BaseIndex.md).[`supportedOperations`](BaseIndex.md#supportedoperations) -*** - -### totalLookupTime - -```ts -protected totalLookupTime: number = 0; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:90](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L90) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`totalLookupTime`](BaseIndex.md#totallookuptime) - ## Accessors -### indexedKeysSet - -#### Get Signature - -```ts -get indexedKeysSet(): Set; -``` - -Defined in: [packages/db/src/indexes/btree-index.ts:400](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L400) - -##### Returns - -`Set`\<`TKey`\> - -#### Overrides - -[`BaseIndex`](BaseIndex.md).[`indexedKeysSet`](BaseIndex.md#indexedkeysset) - -*** - ### keyCount #### Get Signature @@ -202,7 +157,7 @@ Defined in: [packages/db/src/indexes/btree-index.ts:400](https://github.com/TanS get keyCount(): number; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:213](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L213) +Defined in: [packages/db/src/indexes/btree-index.ts:277](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L277) Gets the number of indexed keys @@ -216,93 +171,83 @@ Gets the number of indexed keys *** -### orderedEntriesArray +### supportsRangeOptimization #### Get Signature ```ts -get orderedEntriesArray(): [any, Set][]; +get supportsRangeOptimization(): boolean; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:404](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L404) +Defined in: [packages/db/src/indexes/base-index.ts:193](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L193) -##### Returns +Whether range lookups (gt/gte/lt/lte) on this index can be trusted to +return every matching key. Range traversal relies on the index ordering, so +it is unsafe when the index uses a custom comparator, whose order may not +match the WHERE evaluator's relational operators. Callers must fall back to +a full scan when this is `false`. -\[`any`, `Set`\<`TKey`\>\][] +##### Returns -#### Overrides +`boolean` -[`BaseIndex`](BaseIndex.md).[`orderedEntriesArray`](BaseIndex.md#orderedentriesarray) +#### Inherited from -*** +[`BaseIndex`](BaseIndex.md).[`supportsRangeOptimization`](BaseIndex.md#supportsrangeoptimization) -### orderedEntriesArrayReversed +## Methods -#### Get Signature +### add() ```ts -get orderedEntriesArrayReversed(): [any, Set][]; +add(key, item): void; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:413](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L413) - -##### Returns - -\[`any`, `Set`\<`TKey`\>\][] - -#### Overrides +Defined in: [packages/db/src/indexes/btree-index.ts:98](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L98) -[`BaseIndex`](BaseIndex.md).[`orderedEntriesArrayReversed`](BaseIndex.md#orderedentriesarrayreversed) +Adds a value to the index -*** +#### Parameters -### valueMapData +##### key -#### Get Signature +`TKey` -```ts -get valueMapData(): Map>; -``` +##### item -Defined in: [packages/db/src/indexes/btree-index.ts:420](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L420) +`any` -##### Returns +#### Returns -`Map`\<`any`, `Set`\<`TKey`\>\> +`void` #### Overrides -[`BaseIndex`](BaseIndex.md).[`valueMapData`](BaseIndex.md#valuemapdata) +[`BaseIndex`](BaseIndex.md).[`add`](BaseIndex.md#add) -## Methods +*** -### add() +### addRangeValue() ```ts -add(key, item): void; +protected addRangeValue(value): void; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:83](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L83) - -Adds a value to the index +Defined in: [packages/db/src/indexes/base-index.ts:197](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L197) #### Parameters -##### key - -`TKey` - -##### item +##### value -`any` +`unknown` #### Returns `void` -#### Overrides +#### Inherited from -[`BaseIndex`](BaseIndex.md).[`add`](BaseIndex.md#add) +[`BaseIndex`](BaseIndex.md).[`addRangeValue`](BaseIndex.md#addrangevalue) *** @@ -312,7 +257,7 @@ Adds a value to the index build(entries): void; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:157](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L157) +Defined in: [packages/db/src/indexes/btree-index.ts:225](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L225) Builds the index from a collection of entries @@ -332,13 +277,41 @@ Builds the index from a collection of entries *** +### canOptimizeRangeFor() + +```ts +canOptimizeRangeFor(value): boolean; +``` + +Defined in: [packages/db/src/indexes/base-index.ts:219](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L219) + +Whether the live values in this index share the predicate operand's +relational domain. Mixed domains can sort differently in the index and +WHERE evaluator, which can make a range lookup omit matching rows. + +#### Parameters + +##### value + +`unknown` + +#### Returns + +`boolean` + +#### Inherited from + +[`BaseIndex`](BaseIndex.md).[`canOptimizeRangeFor`](BaseIndex.md#canoptimizerangefor) + +*** + ### clear() ```ts clear(): void; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:168](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L168) +Defined in: [packages/db/src/indexes/btree-index.ts:236](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L236) Clears all data from the index @@ -352,13 +325,31 @@ Clears all data from the index *** +### clearRangeValues() + +```ts +protected clearRangeValues(): void; +``` + +Defined in: [packages/db/src/indexes/base-index.ts:215](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L215) + +#### Returns + +`void` + +#### Inherited from + +[`BaseIndex`](BaseIndex.md).[`clearRangeValues`](BaseIndex.md#clearrangevalues) + +*** + ### equalityLookup() ```ts equalityLookup(value): Set; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:222](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L222) +Defined in: [packages/db/src/indexes/btree-index.ts:286](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L286) Performs an equality lookup @@ -384,7 +375,7 @@ Performs an equality lookup protected evaluateIndexExpression(item): any; ``` -Defined in: [packages/db/src/indexes/base-index.ts:194](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L194) +Defined in: [packages/db/src/indexes/base-index.ts:276](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L276) #### Parameters @@ -402,31 +393,13 @@ Defined in: [packages/db/src/indexes/base-index.ts:194](https://github.com/TanSt *** -### getStats() - -```ts -getStats(): IndexStats; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:182](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L182) - -#### Returns - -[`IndexStats`](../interfaces/IndexStats.md) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`getStats`](BaseIndex.md#getstats) - -*** - ### inArrayLookup() ```ts inArrayLookup(values): Set; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:385](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L385) +Defined in: [packages/db/src/indexes/btree-index.ts:431](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L431) Performs an IN array lookup @@ -452,7 +425,7 @@ Performs an IN array lookup protected initialize(_options?): void; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:78](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L78) +Defined in: [packages/db/src/indexes/btree-index.ts:93](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L93) #### Parameters @@ -476,7 +449,7 @@ Defined in: [packages/db/src/indexes/btree-index.ts:78](https://github.com/TanSt lookup(operation, value): Set; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:178](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L178) +Defined in: [packages/db/src/indexes/btree-index.ts:246](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L246) Performs a lookup operation @@ -506,7 +479,7 @@ Performs a lookup operation matchesCompareOptions(compareOptions): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:159](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L159) +Defined in: [packages/db/src/indexes/base-index.ts:241](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L241) Checks if the compare options match the index's compare options. The direction is ignored because the index can be reversed if the direction is different. @@ -533,7 +506,7 @@ The direction is ignored because the index can be reversed if the direction is d matchesDirection(direction): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:178](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L178) +Defined in: [packages/db/src/indexes/base-index.ts:270](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L270) Checks if the index matches the provided direction. @@ -559,7 +532,7 @@ Checks if the index matches the provided direction. matchesField(fieldPath): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:147](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L147) +Defined in: [packages/db/src/indexes/base-index.ts:229](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L229) #### Parameters @@ -583,7 +556,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:147](https://github.com/TanSt rangeQuery(options): Set; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:231](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L231) +Defined in: [packages/db/src/indexes/btree-index.ts:295](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L295) Performs a range query with options This is more efficient for compound queries like "WHERE a > 5 AND a < 10" @@ -610,9 +583,7 @@ This is more efficient for compound queries like "WHERE a > 5 AND a < 10" rangeQueryReversed(options): Set; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:269](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L269) - -Performs a reversed range query +Defined in: [packages/db/src/indexes/base-index.ts:175](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L175) #### Parameters @@ -624,7 +595,7 @@ Performs a reversed range query `Set`\<`TKey`\> -#### Overrides +#### Inherited from [`BaseIndex`](BaseIndex.md).[`rangeQueryReversed`](BaseIndex.md#rangequeryreversed) @@ -636,7 +607,7 @@ Performs a reversed range query remove(key, item): void; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:114](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L114) +Defined in: [packages/db/src/indexes/btree-index.ts:146](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L146) Removes a value from the index @@ -660,13 +631,37 @@ Removes a value from the index *** +### removeRangeValue() + +```ts +protected removeRangeValue(value): void; +``` + +Defined in: [packages/db/src/indexes/base-index.ts:206](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L206) + +#### Parameters + +##### value + +`unknown` + +#### Returns + +`void` + +#### Inherited from + +[`BaseIndex`](BaseIndex.md).[`removeRangeValue`](BaseIndex.md#removerangevalue) + +*** + ### supports() ```ts supports(operation): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:143](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L143) +Defined in: [packages/db/src/indexes/base-index.ts:189](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L189) #### Parameters @@ -693,7 +688,7 @@ take( filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:331](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L331) +Defined in: [packages/db/src/indexes/btree-index.ts:377](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L377) Returns the next n items after the provided item. @@ -733,7 +728,7 @@ The next n items after the provided key. takeFromStart(n, filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:344](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L344) +Defined in: [packages/db/src/indexes/btree-index.ts:390](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L390) Returns the first n items from the beginning. @@ -772,7 +767,7 @@ takeReversed( filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:356](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L356) +Defined in: [packages/db/src/indexes/btree-index.ts:402](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L402) Returns the next n items **before** the provided item (in descending order). @@ -812,7 +807,7 @@ The next n items **before** the provided key. takeReversedFromEnd(n, filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:373](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L373) +Defined in: [packages/db/src/indexes/btree-index.ts:419](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L419) Returns the last n items from the end. @@ -842,30 +837,6 @@ The last n items *** -### trackLookup() - -```ts -protected trackLookup(startTime): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:199](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L199) - -#### Parameters - -##### startTime - -`number` - -#### Returns - -`void` - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`trackLookup`](BaseIndex.md#tracklookup) - -*** - ### update() ```ts @@ -875,7 +846,7 @@ update( newItem): void; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:149](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L149) +Defined in: [packages/db/src/indexes/btree-index.ts:192](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L192) Updates a value in the index @@ -900,21 +871,3 @@ Updates a value in the index #### Overrides [`BaseIndex`](BaseIndex.md).[`update`](BaseIndex.md#update) - -*** - -### updateTimestamp() - -```ts -protected updateTimestamp(): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:205](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L205) - -#### Returns - -`void` - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`updateTimestamp`](BaseIndex.md#updatetimestamp) diff --git a/docs/reference/classes/BaseIndex.md b/docs/reference/classes/BaseIndex.md index bd512fdbea..1a2d4a95aa 100644 --- a/docs/reference/classes/BaseIndex.md +++ b/docs/reference/classes/BaseIndex.md @@ -5,7 +5,7 @@ title: BaseIndex # Abstract Class: BaseIndex\ -Defined in: [packages/db/src/indexes/base-index.ts:81](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L81) +Defined in: [packages/db/src/indexes/base-index.ts:115](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L115) Base abstract class that all index types extend @@ -36,7 +36,7 @@ new BaseIndex( options?): BaseIndex; ``` -Defined in: [packages/db/src/indexes/base-index.ts:94](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L94) +Defined in: [packages/db/src/indexes/base-index.ts:131](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L131) #### Parameters @@ -68,7 +68,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:94](https://github.com/TanSta protected compareOptions: CompareOptions; ``` -Defined in: [packages/db/src/indexes/base-index.ts:92](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L92) +Defined in: [packages/db/src/indexes/base-index.ts:122](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L122) *** @@ -78,37 +78,30 @@ Defined in: [packages/db/src/indexes/base-index.ts:92](https://github.com/TanSta readonly expression: BasicExpression; ``` -Defined in: [packages/db/src/indexes/base-index.ts:86](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L86) +Defined in: [packages/db/src/indexes/base-index.ts:120](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L120) *** -### id +### hasCustomComparator ```ts -readonly id: number; +protected hasCustomComparator: boolean = false; ``` -Defined in: [packages/db/src/indexes/base-index.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L84) - -*** - -### lastUpdated - -```ts -protected lastUpdated: Date; -``` +Defined in: [packages/db/src/indexes/base-index.ts:128](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L128) -Defined in: [packages/db/src/indexes/base-index.ts:91](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L91) +Set by subclasses when constructed with a user-supplied comparator, whose +ordering may not match the WHERE evaluator's relational operators. *** -### lookupCount +### id ```ts -protected lookupCount: number = 0; +readonly id: number; ``` -Defined in: [packages/db/src/indexes/base-index.ts:89](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L89) +Defined in: [packages/db/src/indexes/base-index.ts:118](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L118) *** @@ -118,7 +111,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:89](https://github.com/TanSta readonly optional name: string; ``` -Defined in: [packages/db/src/indexes/base-index.ts:85](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L85) +Defined in: [packages/db/src/indexes/base-index.ts:119](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L119) *** @@ -128,137 +121,117 @@ Defined in: [packages/db/src/indexes/base-index.ts:85](https://github.com/TanSta abstract readonly supportedOperations: Set<"eq" | "gt" | "gte" | "lt" | "lte" | "in" | "like" | "ilike">; ``` -Defined in: [packages/db/src/indexes/base-index.ts:87](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L87) - -*** - -### totalLookupTime - -```ts -protected totalLookupTime: number = 0; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:90](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L90) +Defined in: [packages/db/src/indexes/base-index.ts:121](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L121) ## Accessors -### indexedKeysSet +### keyCount #### Get Signature ```ts -get abstract indexedKeysSet(): Set; +get abstract keyCount(): number; ``` -Defined in: [packages/db/src/indexes/base-index.ts:139](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L139) +Defined in: [packages/db/src/indexes/base-index.ts:169](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L169) ##### Returns -`Set`\<`TKey`\> +`number` #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`indexedKeysSet`](../interfaces/IndexInterface.md#indexedkeysset) +[`IndexInterface`](../interfaces/IndexInterface.md).[`keyCount`](../interfaces/IndexInterface.md#keycount) *** -### keyCount +### supportsRangeOptimization #### Get Signature ```ts -get abstract keyCount(): number; +get supportsRangeOptimization(): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:132](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L132) +Defined in: [packages/db/src/indexes/base-index.ts:193](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L193) + +Whether range lookups (gt/gte/lt/lte) on this index can be trusted to +return every matching key. Range traversal relies on the index ordering, so +it is unsafe when the index uses a custom comparator, whose order may not +match the WHERE evaluator's relational operators. Callers must fall back to +a full scan when this is `false`. ##### Returns -`number` +`boolean` #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`keyCount`](../interfaces/IndexInterface.md#keycount) - -*** +[`IndexInterface`](../interfaces/IndexInterface.md).[`supportsRangeOptimization`](../interfaces/IndexInterface.md#supportsrangeoptimization) -### orderedEntriesArray +## Methods -#### Get Signature +### add() ```ts -get abstract orderedEntriesArray(): [any, Set][]; +abstract add(key, item): void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:137](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L137) - -##### Returns - -\[`any`, `Set`\<`TKey`\>\][] - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`orderedEntriesArray`](../interfaces/IndexInterface.md#orderedentriesarray) +Defined in: [packages/db/src/indexes/base-index.ts:145](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L145) -*** +#### Parameters -### orderedEntriesArrayReversed +##### key -#### Get Signature +`TKey` -```ts -get abstract orderedEntriesArrayReversed(): [any, Set][]; -``` +##### item -Defined in: [packages/db/src/indexes/base-index.ts:138](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L138) +`any` -##### Returns +#### Returns -\[`any`, `Set`\<`TKey`\>\][] +`void` #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`orderedEntriesArrayReversed`](../interfaces/IndexInterface.md#orderedentriesarrayreversed) +[`IndexInterface`](../interfaces/IndexInterface.md).[`add`](../interfaces/IndexInterface.md#add) *** -### valueMapData - -#### Get Signature +### addRangeValue() ```ts -get abstract valueMapData(): Map>; +protected addRangeValue(value): void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:140](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L140) +Defined in: [packages/db/src/indexes/base-index.ts:197](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L197) -##### Returns +#### Parameters -`Map`\<`any`, `Set`\<`TKey`\>\> +##### value -#### Implementation of +`unknown` -[`IndexInterface`](../interfaces/IndexInterface.md).[`valueMapData`](../interfaces/IndexInterface.md#valuemapdata) +#### Returns -## Methods +`void` -### add() +*** + +### build() ```ts -abstract add(key, item): void; +abstract build(entries): void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:108](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L108) +Defined in: [packages/db/src/indexes/base-index.ts:148](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L148) #### Parameters -##### key - -`TKey` - -##### item +##### entries -`any` +`Iterable`\<\[`TKey`, `any`\]\> #### Returns @@ -266,31 +239,35 @@ Defined in: [packages/db/src/indexes/base-index.ts:108](https://github.com/TanSt #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`add`](../interfaces/IndexInterface.md#add) +[`IndexInterface`](../interfaces/IndexInterface.md).[`build`](../interfaces/IndexInterface.md#build) *** -### build() +### canOptimizeRangeFor() ```ts -abstract build(entries): void; +canOptimizeRangeFor(value): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:111](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L111) +Defined in: [packages/db/src/indexes/base-index.ts:219](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L219) + +Whether the live values in this index share the predicate operand's +relational domain. Mixed domains can sort differently in the index and +WHERE evaluator, which can make a range lookup omit matching rows. #### Parameters -##### entries +##### value -`Iterable`\<\[`TKey`, `any`\]\> +`unknown` #### Returns -`void` +`boolean` #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`build`](../interfaces/IndexInterface.md#build) +[`IndexInterface`](../interfaces/IndexInterface.md).[`canOptimizeRangeFor`](../interfaces/IndexInterface.md#canoptimizerangefor) *** @@ -300,7 +277,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:111](https://github.com/TanSt abstract clear(): void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:112](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L112) +Defined in: [packages/db/src/indexes/base-index.ts:149](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L149) #### Returns @@ -312,13 +289,27 @@ Defined in: [packages/db/src/indexes/base-index.ts:112](https://github.com/TanSt *** +### clearRangeValues() + +```ts +protected clearRangeValues(): void; +``` + +Defined in: [packages/db/src/indexes/base-index.ts:215](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L215) + +#### Returns + +`void` + +*** + ### equalityLookup() ```ts abstract equalityLookup(value): Set; ``` -Defined in: [packages/db/src/indexes/base-index.ts:133](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L133) +Defined in: [packages/db/src/indexes/base-index.ts:170](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L170) #### Parameters @@ -342,7 +333,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:133](https://github.com/TanSt protected evaluateIndexExpression(item): any; ``` -Defined in: [packages/db/src/indexes/base-index.ts:194](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L194) +Defined in: [packages/db/src/indexes/base-index.ts:276](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L276) #### Parameters @@ -356,31 +347,13 @@ Defined in: [packages/db/src/indexes/base-index.ts:194](https://github.com/TanSt *** -### getStats() - -```ts -getStats(): IndexStats; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:182](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L182) - -#### Returns - -[`IndexStats`](../interfaces/IndexStats.md) - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`getStats`](../interfaces/IndexInterface.md#getstats) - -*** - ### inArrayLookup() ```ts abstract inArrayLookup(values): Set; ``` -Defined in: [packages/db/src/indexes/base-index.ts:134](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L134) +Defined in: [packages/db/src/indexes/base-index.ts:171](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L171) #### Parameters @@ -404,7 +377,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:134](https://github.com/TanSt abstract protected initialize(options?): void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:192](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L192) +Defined in: [packages/db/src/indexes/base-index.ts:274](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L274) #### Parameters @@ -424,7 +397,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:192](https://github.com/TanSt abstract lookup(operation, value): Set; ``` -Defined in: [packages/db/src/indexes/base-index.ts:113](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L113) +Defined in: [packages/db/src/indexes/base-index.ts:150](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L150) #### Parameters @@ -452,7 +425,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:113](https://github.com/TanSt matchesCompareOptions(compareOptions): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:159](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L159) +Defined in: [packages/db/src/indexes/base-index.ts:241](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L241) Checks if the compare options match the index's compare options. The direction is ignored because the index can be reversed if the direction is different. @@ -479,7 +452,7 @@ The direction is ignored because the index can be reversed if the direction is d matchesDirection(direction): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:178](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L178) +Defined in: [packages/db/src/indexes/base-index.ts:270](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L270) Checks if the index matches the provided direction. @@ -505,7 +478,7 @@ Checks if the index matches the provided direction. matchesField(fieldPath): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:147](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L147) +Defined in: [packages/db/src/indexes/base-index.ts:229](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L229) #### Parameters @@ -529,7 +502,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:147](https://github.com/TanSt abstract rangeQuery(options): Set; ``` -Defined in: [packages/db/src/indexes/base-index.ts:135](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L135) +Defined in: [packages/db/src/indexes/base-index.ts:172](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L172) #### Parameters @@ -550,16 +523,16 @@ Defined in: [packages/db/src/indexes/base-index.ts:135](https://github.com/TanSt ### rangeQueryReversed() ```ts -abstract rangeQueryReversed(options): Set; +rangeQueryReversed(options): Set; ``` -Defined in: [packages/db/src/indexes/base-index.ts:136](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L136) +Defined in: [packages/db/src/indexes/base-index.ts:175](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L175) #### Parameters ##### options -[`BTreeRangeQueryOptions`](../interfaces/BTreeRangeQueryOptions.md) +[`BTreeRangeQueryOptions`](../interfaces/BTreeRangeQueryOptions.md) = `{}` #### Returns @@ -577,7 +550,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:136](https://github.com/TanSt abstract remove(key, item): void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:109](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L109) +Defined in: [packages/db/src/indexes/base-index.ts:146](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L146) #### Parameters @@ -599,13 +572,33 @@ Defined in: [packages/db/src/indexes/base-index.ts:109](https://github.com/TanSt *** +### removeRangeValue() + +```ts +protected removeRangeValue(value): void; +``` + +Defined in: [packages/db/src/indexes/base-index.ts:206](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L206) + +#### Parameters + +##### value + +`unknown` + +#### Returns + +`void` + +*** + ### supports() ```ts supports(operation): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:143](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L143) +Defined in: [packages/db/src/indexes/base-index.ts:189](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L189) #### Parameters @@ -632,7 +625,7 @@ abstract take( filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/base-index.ts:114](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L114) +Defined in: [packages/db/src/indexes/base-index.ts:151](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L151) #### Parameters @@ -642,7 +635,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:114](https://github.com/TanSt ##### from -`TKey` +`unknown` ##### filterFn? @@ -664,7 +657,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:114](https://github.com/TanSt abstract takeFromStart(n, filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/base-index.ts:119](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L119) +Defined in: [packages/db/src/indexes/base-index.ts:156](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L156) #### Parameters @@ -695,7 +688,7 @@ abstract takeReversed( filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/base-index.ts:123](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L123) +Defined in: [packages/db/src/indexes/base-index.ts:160](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L160) #### Parameters @@ -705,7 +698,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:123](https://github.com/TanSt ##### from -`TKey` +`unknown` ##### filterFn? @@ -727,7 +720,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:123](https://github.com/TanSt abstract takeReversedFromEnd(n, filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/base-index.ts:128](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L128) +Defined in: [packages/db/src/indexes/base-index.ts:165](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L165) #### Parameters @@ -749,26 +742,6 @@ Defined in: [packages/db/src/indexes/base-index.ts:128](https://github.com/TanSt *** -### trackLookup() - -```ts -protected trackLookup(startTime): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:199](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L199) - -#### Parameters - -##### startTime - -`number` - -#### Returns - -`void` - -*** - ### update() ```ts @@ -778,7 +751,7 @@ abstract update( newItem): void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:110](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L110) +Defined in: [packages/db/src/indexes/base-index.ts:147](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L147) #### Parameters @@ -801,17 +774,3 @@ Defined in: [packages/db/src/indexes/base-index.ts:110](https://github.com/TanSt #### Implementation of [`IndexInterface`](../interfaces/IndexInterface.md).[`update`](../interfaces/IndexInterface.md#update) - -*** - -### updateTimestamp() - -```ts -protected updateTimestamp(): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:205](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L205) - -#### Returns - -`void` diff --git a/docs/reference/classes/BaseQueryBuilder.md b/docs/reference/classes/BaseQueryBuilder.md index 224e68885d..940ff2a46e 100644 --- a/docs/reference/classes/BaseQueryBuilder.md +++ b/docs/reference/classes/BaseQueryBuilder.md @@ -5,7 +5,7 @@ title: BaseQueryBuilder # Class: BaseQueryBuilder\ -Defined in: [packages/db/src/query/builder/index.ts:63](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L63) +Defined in: [packages/db/src/query/builder/index.ts:136](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L136) ## Type Parameters @@ -18,10 +18,10 @@ Defined in: [packages/db/src/query/builder/index.ts:63](https://github.com/TanSt ### Constructor ```ts -new BaseQueryBuilder(query): BaseQueryBuilder; +new BaseQueryBuilder(query, resolveCollection?): BaseQueryBuilder; ``` -Defined in: [packages/db/src/query/builder/index.ts:66](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L66) +Defined in: [packages/db/src/query/builder/index.ts:139](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L139) #### Parameters @@ -29,6 +29,10 @@ Defined in: [packages/db/src/query/builder/index.ts:66](https://github.com/TanSt `Partial`\<[`QueryIR`](../@tanstack/namespaces/IR/interfaces/QueryIR.md)\> = `{}` +##### resolveCollection? + +`CollectionResolver` + #### Returns `BaseQueryBuilder`\<`TContext`\> @@ -43,7 +47,7 @@ Defined in: [packages/db/src/query/builder/index.ts:66](https://github.com/TanSt get fn(): object; ``` -Defined in: [packages/db/src/query/builder/index.ts:770](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L770) +Defined in: [packages/db/src/query/builder/index.ts:924](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L924) Functional variants of the query builder These are imperative function that are called for ery row. @@ -98,7 +102,7 @@ query ###### select() ```ts -select(callback): QueryBuilder>; +select(callback): FnSelectQueryResult; ``` Select fields using a function that operates on each row @@ -120,7 +124,7 @@ A function that receives a row and returns the selected value ###### Returns -[`QueryBuilder`](../type-aliases/QueryBuilder.md)\<[`WithResult`](../type-aliases/WithResult.md)\<`TContext`, `TFuncSelectResult`\>\> +`FnSelectQueryResult`\<`TContext`, `TFuncSelectResult`\> A QueryBuilder with functional selection applied @@ -136,6 +140,16 @@ query })) ``` +Child query builders, query expressions, and helpers such as eq(), +toArray(), and materialize() cannot be returned from fn.select(). Use +them as fields in select() so the compiler can add them to the query +graph. + +Compiled Collection-valued includes cannot be inputs to fn.select(), +including nested descendants. Use toArray() or materialize() in the +upstream select(), or do parent-only functional work before adding +live Collection includes with select(). + ###### where() ```ts @@ -176,7 +190,7 @@ query _getQuery(): QueryIR; ``` -Defined in: [packages/db/src/query/builder/index.ts:857](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L857) +Defined in: [packages/db/src/query/builder/index.ts:1021](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L1021) #### Returns @@ -190,7 +204,7 @@ Defined in: [packages/db/src/query/builder/index.ts:857](https://github.com/TanS distinct(): QueryBuilder; ``` -Defined in: [packages/db/src/query/builder/index.ts:709](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L709) +Defined in: [packages/db/src/query/builder/index.ts:857](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L857) Specify that the query should return distinct rows. Deduplicates rows based on the selected columns. @@ -219,7 +233,7 @@ query findOne(): QueryBuilder; ``` -Defined in: [packages/db/src/query/builder/index.ts:729](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L729) +Defined in: [packages/db/src/query/builder/index.ts:877](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L877) Specify that the query should return a single result @@ -244,15 +258,10 @@ query ### from() ```ts -from(source): QueryBuilder<{ - baseSchema: SchemaFromSource; - fromSourceName: keyof TSource & string; - hasJoins: false; - schema: SchemaFromSource; -}>; +from(source): QueryBuilder>; ``` -Defined in: [packages/db/src/query/builder/index.ts:145](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L145) +Defined in: [packages/db/src/query/builder/index.ts:249](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L249) Specify the source table or subquery for the query @@ -266,18 +275,13 @@ Specify the source table or subquery for the query ##### source -`TSource` +[`SingleSource`](../type-aliases/SingleSource.md)\<`TSource`\> An object with a single key-value pair where the key is the table alias and the value is a Collection or subquery #### Returns -[`QueryBuilder`](../type-aliases/QueryBuilder.md)\<\{ - `baseSchema`: [`SchemaFromSource`](../type-aliases/SchemaFromSource.md)\<`TSource`\>; - `fromSourceName`: keyof `TSource` & `string`; - `hasJoins`: `false`; - `schema`: [`SchemaFromSource`](../type-aliases/SchemaFromSource.md)\<`TSource`\>; -\}\> +[`QueryBuilder`](../type-aliases/QueryBuilder.md)\<[`ContextFromSource`](../type-aliases/ContextFromSource.md)\<`TSource`\>\> A QueryBuilder with the specified source @@ -300,7 +304,7 @@ query.from({ activeUsers }) fullJoin(source, onCallback): QueryBuilder, "full">>; ``` -Defined in: [packages/db/src/query/builder/index.ts:336](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L336) +Defined in: [packages/db/src/query/builder/index.ts:484](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L484) Perform a FULL JOIN with another table or subquery @@ -320,7 +324,7 @@ An object with a single key-value pair where the key is the table alias and the ##### onCallback -[`JoinOnCallback`](../type-aliases/JoinOnCallback.md)\<[`MergeContextForJoinCallback`](../type-aliases/MergeContextForJoinCallback.md)\<`TContext`, \{ \[K in string \| number \| symbol\]: \{ \[K in string \| number \| symbol\]: TSource\[K\] extends CollectionImpl\ ? InferCollectionType\ : TSource\[K\] extends QueryBuilder\ ? \{ \[K in string \| number \| symbol\]: ResultValue\\[K\] \} : never \}\[K\] \}\>\> +[`JoinOnCallback`](../type-aliases/JoinOnCallback.md)\<[`MergeContextForJoinCallback`](../type-aliases/MergeContextForJoinCallback.md)\<`TContext`, \{ \[K in string \| number \| symbol\]: \{ \[K in string \| number \| symbol\]: TSource\[K\] extends CollectionImpl\ ? InferCollectionType\ : TSource\[K\] extends CollectionOptionsIdentity\ ? InferCollectionType\ : TSource\[K\] extends QueryBuilder\ ? ResultValue\ : never \}\[K\] \}\>\> A function that receives table references and returns the join condition @@ -347,7 +351,7 @@ query groupBy(callback): QueryBuilder; ``` -Defined in: [packages/db/src/query/builder/index.ts:631](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L631) +Defined in: [packages/db/src/query/builder/index.ts:779](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L779) Group rows by one or more columns for aggregation @@ -396,7 +400,7 @@ query having(callback): QueryBuilder; ``` -Defined in: [packages/db/src/query/builder/index.ts:430](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L430) +Defined in: [packages/db/src/query/builder/index.ts:578](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L578) Filter grouped rows based on aggregate conditions @@ -445,7 +449,7 @@ query innerJoin(source, onCallback): QueryBuilder, "inner">>; ``` -Defined in: [packages/db/src/query/builder/index.ts:310](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L310) +Defined in: [packages/db/src/query/builder/index.ts:458](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L458) Perform an INNER JOIN with another table or subquery @@ -465,7 +469,7 @@ An object with a single key-value pair where the key is the table alias and the ##### onCallback -[`JoinOnCallback`](../type-aliases/JoinOnCallback.md)\<[`MergeContextForJoinCallback`](../type-aliases/MergeContextForJoinCallback.md)\<`TContext`, \{ \[K in string \| number \| symbol\]: \{ \[K in string \| number \| symbol\]: TSource\[K\] extends CollectionImpl\ ? InferCollectionType\ : TSource\[K\] extends QueryBuilder\ ? \{ \[K in string \| number \| symbol\]: ResultValue\\[K\] \} : never \}\[K\] \}\>\> +[`JoinOnCallback`](../type-aliases/JoinOnCallback.md)\<[`MergeContextForJoinCallback`](../type-aliases/MergeContextForJoinCallback.md)\<`TContext`, \{ \[K in string \| number \| symbol\]: \{ \[K in string \| number \| symbol\]: TSource\[K\] extends CollectionImpl\ ? InferCollectionType\ : TSource\[K\] extends CollectionOptionsIdentity\ ? InferCollectionType\ : TSource\[K\] extends QueryBuilder\ ? ResultValue\ : never \}\[K\] \}\>\> A function that receives table references and returns the join condition @@ -495,7 +499,7 @@ join( type): QueryBuilder, TJoinType>>; ``` -Defined in: [packages/db/src/query/builder/index.ts:188](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L188) +Defined in: [packages/db/src/query/builder/index.ts:336](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L336) Join another table or subquery to the current query @@ -519,7 +523,7 @@ An object with a single key-value pair where the key is the table alias and the ##### onCallback -[`JoinOnCallback`](../type-aliases/JoinOnCallback.md)\<[`MergeContextForJoinCallback`](../type-aliases/MergeContextForJoinCallback.md)\<`TContext`, \{ \[K in string \| number \| symbol\]: \{ \[K in string \| number \| symbol\]: TSource\[K\] extends CollectionImpl\ ? InferCollectionType\ : TSource\[K\] extends QueryBuilder\ ? \{ \[K in string \| number \| symbol\]: ResultValue\\[K\] \} : never \}\[K\] \}\>\> +[`JoinOnCallback`](../type-aliases/JoinOnCallback.md)\<[`MergeContextForJoinCallback`](../type-aliases/MergeContextForJoinCallback.md)\<`TContext`, \{ \[K in string \| number \| symbol\]: \{ \[K in string \| number \| symbol\]: TSource\[K\] extends CollectionImpl\ ? InferCollectionType\ : TSource\[K\] extends CollectionOptionsIdentity\ ? InferCollectionType\ : TSource\[K\] extends QueryBuilder\ ? ResultValue\ : never \}\[K\] \}\>\> A function that receives table references and returns the join condition @@ -563,7 +567,7 @@ query leftJoin(source, onCallback): QueryBuilder, "left">>; ``` -Defined in: [packages/db/src/query/builder/index.ts:258](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L258) +Defined in: [packages/db/src/query/builder/index.ts:406](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L406) Perform a LEFT JOIN with another table or subquery @@ -583,7 +587,7 @@ An object with a single key-value pair where the key is the table alias and the ##### onCallback -[`JoinOnCallback`](../type-aliases/JoinOnCallback.md)\<[`MergeContextForJoinCallback`](../type-aliases/MergeContextForJoinCallback.md)\<`TContext`, \{ \[K in string \| number \| symbol\]: \{ \[K in string \| number \| symbol\]: TSource\[K\] extends CollectionImpl\ ? InferCollectionType\ : TSource\[K\] extends QueryBuilder\ ? \{ \[K in string \| number \| symbol\]: ResultValue\\[K\] \} : never \}\[K\] \}\>\> +[`JoinOnCallback`](../type-aliases/JoinOnCallback.md)\<[`MergeContextForJoinCallback`](../type-aliases/MergeContextForJoinCallback.md)\<`TContext`, \{ \[K in string \| number \| symbol\]: \{ \[K in string \| number \| symbol\]: TSource\[K\] extends CollectionImpl\ ? InferCollectionType\ : TSource\[K\] extends CollectionOptionsIdentity\ ? InferCollectionType\ : TSource\[K\] extends QueryBuilder\ ? ResultValue\ : never \}\[K\] \}\>\> A function that receives table references and returns the join condition @@ -610,7 +614,7 @@ query limit(count): QueryBuilder; ``` -Defined in: [packages/db/src/query/builder/index.ts:664](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L664) +Defined in: [packages/db/src/query/builder/index.ts:812](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L812) Limit the number of rows returned by the query `orderBy` is required for `limit` @@ -647,7 +651,7 @@ query offset(count): QueryBuilder; ``` -Defined in: [packages/db/src/query/builder/index.ts:688](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L688) +Defined in: [packages/db/src/query/builder/index.ts:836](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L836) Skip a number of rows before returning results `orderBy` is required for `offset` @@ -685,7 +689,7 @@ query orderBy(callback, options): QueryBuilder; ``` -Defined in: [packages/db/src/query/builder/index.ts:555](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L555) +Defined in: [packages/db/src/query/builder/index.ts:703](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L703) Sort the query results by one or more columns @@ -735,7 +739,7 @@ query rightJoin(source, onCallback): QueryBuilder, "right">>; ``` -Defined in: [packages/db/src/query/builder/index.ts:284](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L284) +Defined in: [packages/db/src/query/builder/index.ts:432](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L432) Perform a RIGHT JOIN with another table or subquery @@ -755,7 +759,7 @@ An object with a single key-value pair where the key is the table alias and the ##### onCallback -[`JoinOnCallback`](../type-aliases/JoinOnCallback.md)\<[`MergeContextForJoinCallback`](../type-aliases/MergeContextForJoinCallback.md)\<`TContext`, \{ \[K in string \| number \| symbol\]: \{ \[K in string \| number \| symbol\]: TSource\[K\] extends CollectionImpl\ ? InferCollectionType\ : TSource\[K\] extends QueryBuilder\ ? \{ \[K in string \| number \| symbol\]: ResultValue\\[K\] \} : never \}\[K\] \}\>\> +[`JoinOnCallback`](../type-aliases/JoinOnCallback.md)\<[`MergeContextForJoinCallback`](../type-aliases/MergeContextForJoinCallback.md)\<`TContext`, \{ \[K in string \| number \| symbol\]: \{ \[K in string \| number \| symbol\]: TSource\[K\] extends CollectionImpl\ ? InferCollectionType\ : TSource\[K\] extends CollectionOptionsIdentity\ ? InferCollectionType\ : TSource\[K\] extends QueryBuilder\ ? ResultValue\ : never \}\[K\] \}\>\> A function that receives table references and returns the join condition @@ -784,7 +788,7 @@ query select(callback): QueryBuilder>>; ``` -Defined in: [packages/db/src/query/builder/index.ts:496](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L496) +Defined in: [packages/db/src/query/builder/index.ts:644](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L644) Select specific columns or computed values from the query @@ -843,7 +847,7 @@ query select(callback): QueryBuilder>>; ``` -Defined in: [packages/db/src/query/builder/index.ts:501](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L501) +Defined in: [packages/db/src/query/builder/index.ts:649](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L649) Select specific columns or computed values from the query @@ -898,13 +902,95 @@ query *** +### unionAll() + +#### Call Signature + +```ts +unionAll(source): QueryBuilder>; +``` + +Defined in: [packages/db/src/query/builder/index.ts:275](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L275) + +Union multiple independent source streams in one query. + +##### Type Parameters + +###### TSource + +`TSource` *extends* [`Source`](../type-aliases/Source.md) + +##### Parameters + +###### source + +`TSource` + +An object with one or more aliases mapped to collections or subqueries + +##### Returns + +[`QueryBuilder`](../type-aliases/QueryBuilder.md)\<[`ContextFromUnionSource`](../type-aliases/ContextFromUnionSource.md)\<`TSource`\>\> + +A QueryBuilder with the unioned sources available + +##### Example + +```ts +query + .unionAll({ message: messagesCollection, toolCall: toolCallsCollection }) + .orderBy(({ message, toolCall }) => + coalesce(message.timestamp, toolCall.timestamp) + ) +``` + +#### Call Signature + +```ts +unionAll(...branches): QueryBuilder>; +``` + +Defined in: [packages/db/src/query/builder/index.ts:278](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L278) + +Union multiple independent source streams in one query. + +##### Type Parameters + +###### TBranches + +`TBranches` *extends* readonly \[[`QueryBuilder`](../type-aliases/QueryBuilder.md)\<`any`\>, [`QueryBuilder`](../type-aliases/QueryBuilder.md)\<`any`\>\] + +##### Parameters + +###### branches + +...`TBranches` + +##### Returns + +[`QueryBuilder`](../type-aliases/QueryBuilder.md)\<[`ContextFromUnionBranches`](../type-aliases/ContextFromUnionBranches.md)\<`TBranches`\>\> + +A QueryBuilder with the unioned sources available + +##### Example + +```ts +query + .unionAll({ message: messagesCollection, toolCall: toolCallsCollection }) + .orderBy(({ message, toolCall }) => + coalesce(message.timestamp, toolCall.timestamp) + ) +``` + +*** + ### where() ```ts where(callback): QueryBuilder; ``` -Defined in: [packages/db/src/query/builder/index.ts:375](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L375) +Defined in: [packages/db/src/query/builder/index.ts:523](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L523) Filter rows based on a condition diff --git a/docs/reference/classes/BasicIndex.md b/docs/reference/classes/BasicIndex.md index 35df050713..87820392ea 100644 --- a/docs/reference/classes/BasicIndex.md +++ b/docs/reference/classes/BasicIndex.md @@ -5,7 +5,7 @@ title: BasicIndex # Class: BasicIndex\ -Defined in: [packages/db/src/indexes/basic-index.ts:39](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L39) +Defined in: [packages/db/src/indexes/basic-index.ts:45](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L45) Basic index using Map + sorted Array. @@ -38,7 +38,7 @@ new BasicIndex( options?): BasicIndex; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:60](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L60) +Defined in: [packages/db/src/indexes/basic-index.ts:66](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L66) #### Parameters @@ -74,7 +74,7 @@ Defined in: [packages/db/src/indexes/basic-index.ts:60](https://github.com/TanSt protected compareOptions: CompareOptions; ``` -Defined in: [packages/db/src/indexes/base-index.ts:92](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L92) +Defined in: [packages/db/src/indexes/base-index.ts:122](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L122) #### Inherited from @@ -88,7 +88,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:92](https://github.com/TanSta readonly expression: BasicExpression; ``` -Defined in: [packages/db/src/indexes/base-index.ts:86](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L86) +Defined in: [packages/db/src/indexes/base-index.ts:120](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L120) #### Inherited from @@ -96,45 +96,34 @@ Defined in: [packages/db/src/indexes/base-index.ts:86](https://github.com/TanSta *** -### id +### hasCustomComparator ```ts -readonly id: number; +protected hasCustomComparator: boolean = false; ``` -Defined in: [packages/db/src/indexes/base-index.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L84) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`id`](BaseIndex.md#id) - -*** - -### lastUpdated - -```ts -protected lastUpdated: Date; -``` +Defined in: [packages/db/src/indexes/base-index.ts:128](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L128) -Defined in: [packages/db/src/indexes/base-index.ts:91](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L91) +Set by subclasses when constructed with a user-supplied comparator, whose +ordering may not match the WHERE evaluator's relational operators. #### Inherited from -[`BaseIndex`](BaseIndex.md).[`lastUpdated`](BaseIndex.md#lastupdated) +[`BaseIndex`](BaseIndex.md).[`hasCustomComparator`](BaseIndex.md#hascustomcomparator) *** -### lookupCount +### id ```ts -protected lookupCount: number = 0; +readonly id: number; ``` -Defined in: [packages/db/src/indexes/base-index.ts:89](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L89) +Defined in: [packages/db/src/indexes/base-index.ts:118](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L118) #### Inherited from -[`BaseIndex`](BaseIndex.md).[`lookupCount`](BaseIndex.md#lookupcount) +[`BaseIndex`](BaseIndex.md).[`id`](BaseIndex.md#id) *** @@ -144,7 +133,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:89](https://github.com/TanSta readonly optional name: string; ``` -Defined in: [packages/db/src/indexes/base-index.ts:85](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L85) +Defined in: [packages/db/src/indexes/base-index.ts:119](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L119) #### Inherited from @@ -158,48 +147,14 @@ Defined in: [packages/db/src/indexes/base-index.ts:85](https://github.com/TanSta readonly supportedOperations: Set<"eq" | "gt" | "gte" | "lt" | "lte" | "in" | "like" | "ilike">; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:42](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L42) +Defined in: [packages/db/src/indexes/basic-index.ts:48](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L48) #### Overrides [`BaseIndex`](BaseIndex.md).[`supportedOperations`](BaseIndex.md#supportedoperations) -*** - -### totalLookupTime - -```ts -protected totalLookupTime: number = 0; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:90](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L90) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`totalLookupTime`](BaseIndex.md#totallookuptime) - ## Accessors -### indexedKeysSet - -#### Get Signature - -```ts -get indexedKeysSet(): Set; -``` - -Defined in: [packages/db/src/indexes/basic-index.ts:484](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L484) - -##### Returns - -`Set`\<`TKey`\> - -#### Overrides - -[`BaseIndex`](BaseIndex.md).[`indexedKeysSet`](BaseIndex.md#indexedkeysset) - -*** - ### keyCount #### Get Signature @@ -208,7 +163,7 @@ Defined in: [packages/db/src/indexes/basic-index.ts:484](https://github.com/TanS get keyCount(): number; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:238](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L238) +Defined in: [packages/db/src/indexes/basic-index.ts:294](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L294) Gets the number of indexed keys @@ -222,93 +177,83 @@ Gets the number of indexed keys *** -### orderedEntriesArray +### supportsRangeOptimization #### Get Signature ```ts -get orderedEntriesArray(): [any, Set][]; +get supportsRangeOptimization(): boolean; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:488](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L488) +Defined in: [packages/db/src/indexes/base-index.ts:193](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L193) -##### Returns +Whether range lookups (gt/gte/lt/lte) on this index can be trusted to +return every matching key. Range traversal relies on the index ordering, so +it is unsafe when the index uses a custom comparator, whose order may not +match the WHERE evaluator's relational operators. Callers must fall back to +a full scan when this is `false`. -\[`any`, `Set`\<`TKey`\>\][] +##### Returns -#### Overrides +`boolean` -[`BaseIndex`](BaseIndex.md).[`orderedEntriesArray`](BaseIndex.md#orderedentriesarray) +#### Inherited from -*** +[`BaseIndex`](BaseIndex.md).[`supportsRangeOptimization`](BaseIndex.md#supportsrangeoptimization) -### orderedEntriesArrayReversed +## Methods -#### Get Signature +### add() ```ts -get orderedEntriesArrayReversed(): [any, Set][]; +add(key, item): void; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:495](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L495) - -##### Returns - -\[`any`, `Set`\<`TKey`\>\][] +Defined in: [packages/db/src/indexes/basic-index.ts:85](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L85) -#### Overrides - -[`BaseIndex`](BaseIndex.md).[`orderedEntriesArrayReversed`](BaseIndex.md#orderedentriesarrayreversed) +Adds a value to the index -*** +#### Parameters -### valueMapData +##### key -#### Get Signature +`TKey` -```ts -get valueMapData(): Map>; -``` +##### item -Defined in: [packages/db/src/indexes/basic-index.ts:504](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L504) +`any` -##### Returns +#### Returns -`Map`\<`any`, `Set`\<`TKey`\>\> +`void` #### Overrides -[`BaseIndex`](BaseIndex.md).[`valueMapData`](BaseIndex.md#valuemapdata) +[`BaseIndex`](BaseIndex.md).[`add`](BaseIndex.md#add) -## Methods +*** -### add() +### addRangeValue() ```ts -add(key, item): void; +protected addRangeValue(value): void; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:78](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L78) - -Adds a value to the index +Defined in: [packages/db/src/indexes/base-index.ts:197](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L197) #### Parameters -##### key - -`TKey` - -##### item +##### value -`any` +`unknown` #### Returns `void` -#### Overrides +#### Inherited from -[`BaseIndex`](BaseIndex.md).[`add`](BaseIndex.md#add) +[`BaseIndex`](BaseIndex.md).[`addRangeValue`](BaseIndex.md#addrangevalue) *** @@ -318,7 +263,7 @@ Adds a value to the index build(entries): void; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:156](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L156) +Defined in: [packages/db/src/indexes/basic-index.ts:217](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L217) Builds the index from a collection of entries @@ -338,13 +283,41 @@ Builds the index from a collection of entries *** +### canOptimizeRangeFor() + +```ts +canOptimizeRangeFor(value): boolean; +``` + +Defined in: [packages/db/src/indexes/base-index.ts:219](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L219) + +Whether the live values in this index share the predicate operand's +relational domain. Mixed domains can sort differently in the index and +WHERE evaluator, which can make a range lookup omit matching rows. + +#### Parameters + +##### value + +`unknown` + +#### Returns + +`boolean` + +#### Inherited from + +[`BaseIndex`](BaseIndex.md).[`canOptimizeRangeFor`](BaseIndex.md#canoptimizerangefor) + +*** + ### clear() ```ts clear(): void; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:193](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L193) +Defined in: [packages/db/src/indexes/basic-index.ts:253](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L253) Clears all data from the index @@ -358,13 +331,31 @@ Clears all data from the index *** +### clearRangeValues() + +```ts +protected clearRangeValues(): void; +``` + +Defined in: [packages/db/src/indexes/base-index.ts:215](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L215) + +#### Returns + +`void` + +#### Inherited from + +[`BaseIndex`](BaseIndex.md).[`clearRangeValues`](BaseIndex.md#clearrangevalues) + +*** + ### equalityLookup() ```ts equalityLookup(value): Set; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:245](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L245) +Defined in: [packages/db/src/indexes/basic-index.ts:301](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L301) Performs an equality lookup - O(1) @@ -390,7 +381,7 @@ Performs an equality lookup - O(1) protected evaluateIndexExpression(item): any; ``` -Defined in: [packages/db/src/indexes/base-index.ts:194](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L194) +Defined in: [packages/db/src/indexes/base-index.ts:276](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L276) #### Parameters @@ -408,31 +399,13 @@ Defined in: [packages/db/src/indexes/base-index.ts:194](https://github.com/TanSt *** -### getStats() - -```ts -getStats(): IndexStats; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:182](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L182) - -#### Returns - -[`IndexStats`](../interfaces/IndexStats.md) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`getStats`](BaseIndex.md#getstats) - -*** - ### inArrayLookup() ```ts inArrayLookup(values): Set; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:469](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L469) +Defined in: [packages/db/src/indexes/basic-index.ts:471](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L471) Performs an IN array lookup - O(k) where k is values.length @@ -458,7 +431,7 @@ Performs an IN array lookup - O(k) where k is values.length protected initialize(_options?): void; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:73](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L73) +Defined in: [packages/db/src/indexes/basic-index.ts:80](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L80) #### Parameters @@ -482,7 +455,7 @@ Defined in: [packages/db/src/indexes/basic-index.ts:73](https://github.com/TanSt lookup(operation, value): Set; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:203](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L203) +Defined in: [packages/db/src/indexes/basic-index.ts:263](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L263) Performs a lookup operation @@ -512,7 +485,7 @@ Performs a lookup operation matchesCompareOptions(compareOptions): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:159](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L159) +Defined in: [packages/db/src/indexes/base-index.ts:241](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L241) Checks if the compare options match the index's compare options. The direction is ignored because the index can be reversed if the direction is different. @@ -539,7 +512,7 @@ The direction is ignored because the index can be reversed if the direction is d matchesDirection(direction): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:178](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L178) +Defined in: [packages/db/src/indexes/base-index.ts:270](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L270) Checks if the index matches the provided direction. @@ -565,7 +538,7 @@ Checks if the index matches the provided direction. matchesField(fieldPath): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:147](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L147) +Defined in: [packages/db/src/indexes/base-index.ts:229](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L229) #### Parameters @@ -589,7 +562,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:147](https://github.com/TanSt rangeQuery(options): Set; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:253](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L253) +Defined in: [packages/db/src/indexes/basic-index.ts:309](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L309) Performs a range query using binary search - O(log n + m) @@ -615,21 +588,19 @@ Performs a range query using binary search - O(log n + m) rangeQueryReversed(options): Set; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:314](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L314) - -Performs a reversed range query +Defined in: [packages/db/src/indexes/base-index.ts:175](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L175) #### Parameters ##### options -[`RangeQueryOptions`](../interfaces/RangeQueryOptions.md) = `{}` +[`BTreeRangeQueryOptions`](../interfaces/BTreeRangeQueryOptions.md) = `{}` #### Returns `Set`\<`TKey`\> -#### Overrides +#### Inherited from [`BaseIndex`](BaseIndex.md).[`rangeQueryReversed`](BaseIndex.md#rangequeryreversed) @@ -641,7 +612,7 @@ Performs a reversed range query remove(key, item): void; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:114](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L114) +Defined in: [packages/db/src/indexes/basic-index.ts:126](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L126) Removes a value from the index @@ -665,13 +636,37 @@ Removes a value from the index *** +### removeRangeValue() + +```ts +protected removeRangeValue(value): void; +``` + +Defined in: [packages/db/src/indexes/base-index.ts:206](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L206) + +#### Parameters + +##### value + +`unknown` + +#### Returns + +`void` + +#### Inherited from + +[`BaseIndex`](BaseIndex.md).[`removeRangeValue`](BaseIndex.md#removerangevalue) + +*** + ### supports() ```ts supports(operation): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:143](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L143) +Defined in: [packages/db/src/indexes/base-index.ts:189](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L189) #### Parameters @@ -694,11 +689,11 @@ Defined in: [packages/db/src/indexes/base-index.ts:143](https://github.com/TanSt ```ts take( n, - from?, + from, filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:339](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L339) +Defined in: [packages/db/src/indexes/basic-index.ts:373](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L373) Returns the next n items in sorted order @@ -708,7 +703,7 @@ Returns the next n items in sorted order `number` -##### from? +##### from `any` @@ -732,7 +727,7 @@ Returns the next n items in sorted order takeFromStart(n, filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:424](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L424) +Defined in: [packages/db/src/indexes/basic-index.ts:420](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L420) Returns the first n items in sorted order (from the start) @@ -761,11 +756,11 @@ Returns the first n items in sorted order (from the start) ```ts takeReversed( n, - from?, + from, filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:381](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L381) +Defined in: [packages/db/src/indexes/basic-index.ts:394](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L394) Returns the next n items in reverse sorted order @@ -775,7 +770,7 @@ Returns the next n items in reverse sorted order `number` -##### from? +##### from `any` @@ -799,7 +794,7 @@ Returns the next n items in reverse sorted order takeReversedFromEnd(n, filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:443](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L443) +Defined in: [packages/db/src/indexes/basic-index.ts:427](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L427) Returns the first n items in reverse sorted order (from the end) @@ -823,30 +818,6 @@ Returns the first n items in reverse sorted order (from the end) *** -### trackLookup() - -```ts -protected trackLookup(startTime): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:199](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L199) - -#### Parameters - -##### startTime - -`number` - -#### Returns - -`void` - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`trackLookup`](BaseIndex.md#tracklookup) - -*** - ### update() ```ts @@ -856,7 +827,7 @@ update( newItem): void; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:148](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L148) +Defined in: [packages/db/src/indexes/basic-index.ts:183](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L183) Updates a value in the index @@ -881,21 +852,3 @@ Updates a value in the index #### Overrides [`BaseIndex`](BaseIndex.md).[`update`](BaseIndex.md#update) - -*** - -### updateTimestamp() - -```ts -protected updateTimestamp(): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:205](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L205) - -#### Returns - -`void` - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`updateTimestamp`](BaseIndex.md#updatetimestamp) diff --git a/docs/reference/classes/CannotCombineEmptyExpressionListError.md b/docs/reference/classes/CannotCombineEmptyExpressionListError.md index 121f7481d0..8938c625d8 100644 --- a/docs/reference/classes/CannotCombineEmptyExpressionListError.md +++ b/docs/reference/classes/CannotCombineEmptyExpressionListError.md @@ -5,7 +5,7 @@ title: CannotCombineEmptyExpressionListError # Class: CannotCombineEmptyExpressionListError -Defined in: [packages/db/src/errors.ts:693](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L693) +Defined in: [packages/db/src/errors.ts:764](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L764) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:693](https://github.com/TanStack/db/blob/ new CannotCombineEmptyExpressionListError(): CannotCombineEmptyExpressionListError; ``` -Defined in: [packages/db/src/errors.ts:694](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L694) +Defined in: [packages/db/src/errors.ts:765](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L765) #### Returns diff --git a/docs/reference/classes/CollectionImpl.md b/docs/reference/classes/CollectionImpl.md index 18c8c74d69..2fc3af6907 100644 --- a/docs/reference/classes/CollectionImpl.md +++ b/docs/reference/classes/CollectionImpl.md @@ -5,7 +5,7 @@ title: CollectionImpl # Class: CollectionImpl\ -Defined in: [packages/db/src/collection/index.ts:272](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L272) +Defined in: [packages/db/src/collection/index.ts:276](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L276) ## Extended by @@ -42,7 +42,7 @@ Defined in: [packages/db/src/collection/index.ts:272](https://github.com/TanStac new CollectionImpl(config): CollectionImpl; ``` -Defined in: [packages/db/src/collection/index.ts:318](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L318) +Defined in: [packages/db/src/collection/index.ts:322](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L322) Creates a new Collection instance @@ -70,7 +70,7 @@ Error if sync config is missing _lifecycle: CollectionLifecycleManager; ``` -Defined in: [packages/db/src/collection/index.ts:289](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L289) +Defined in: [packages/db/src/collection/index.ts:293](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L293) *** @@ -80,7 +80,7 @@ Defined in: [packages/db/src/collection/index.ts:289](https://github.com/TanStac _state: CollectionStateManager; ``` -Defined in: [packages/db/src/collection/index.ts:301](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L301) +Defined in: [packages/db/src/collection/index.ts:305](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L305) *** @@ -90,7 +90,7 @@ Defined in: [packages/db/src/collection/index.ts:301](https://github.com/TanStac _sync: CollectionSyncManager; ``` -Defined in: [packages/db/src/collection/index.ts:290](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L290) +Defined in: [packages/db/src/collection/index.ts:294](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L294) *** @@ -100,7 +100,7 @@ Defined in: [packages/db/src/collection/index.ts:290](https://github.com/TanStac config: CollectionConfig; ``` -Defined in: [packages/db/src/collection/index.ts:280](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L280) +Defined in: [packages/db/src/collection/index.ts:284](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L284) *** @@ -110,7 +110,7 @@ Defined in: [packages/db/src/collection/index.ts:280](https://github.com/TanStac deferDataRefresh: Promise | null = null; ``` -Defined in: [packages/db/src/collection/index.ts:308](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L308) +Defined in: [packages/db/src/collection/index.ts:312](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L312) When set, collection consumers should defer processing incoming data refreshes until this promise resolves. This prevents stale data from @@ -124,7 +124,7 @@ overwriting optimistic state while pending writes are being applied. id: string; ``` -Defined in: [packages/db/src/collection/index.ts:279](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L279) +Defined in: [packages/db/src/collection/index.ts:283](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L283) *** @@ -134,10 +134,49 @@ Defined in: [packages/db/src/collection/index.ts:279](https://github.com/TanStac utils: Record = {}; ``` -Defined in: [packages/db/src/collection/index.ts:284](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L284) +Defined in: [packages/db/src/collection/index.ts:288](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L288) ## Accessors +### \_layoutRevision + +#### Get Signature + +```ts +get _layoutRevision(): number; +``` + +Defined in: [packages/db/src/collection/index.ts:442](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L442) + +Monotonic revision of explicit layout-only publications. +Internal — used to distinguish them from empty ready events. + +##### Returns + +`number` + +*** + +### \_stateRevision + +#### Get Signature + +```ts +get _stateRevision(): number; +``` + +Defined in: [packages/db/src/collection/index.ts:434](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L434) + +Monotonic revision of the collection's visible state; advances once per +committed batch of changes, even while nothing is subscribed. +Internal — used by the live-query observer's snapshot cache. + +##### Returns + +`number` + +*** + ### compareOptions #### Get Signature @@ -146,7 +185,7 @@ Defined in: [packages/db/src/collection/index.ts:284](https://github.com/TanStac get compareOptions(): StringCollationConfig; ``` -Defined in: [packages/db/src/collection/index.ts:643](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L643) +Defined in: [packages/db/src/collection/index.ts:708](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L708) ##### Returns @@ -162,7 +201,7 @@ Defined in: [packages/db/src/collection/index.ts:643](https://github.com/TanStac get indexes(): Map>; ``` -Defined in: [packages/db/src/collection/index.ts:628](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L628) +Defined in: [packages/db/src/collection/index.ts:693](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L693) Get resolved indexes for query optimization @@ -180,7 +219,7 @@ Get resolved indexes for query optimization get isLoadingSubset(): boolean; ``` -Defined in: [packages/db/src/collection/index.ts:456](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L456) +Defined in: [packages/db/src/collection/index.ts:500](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L500) Check if the collection is currently loading more data @@ -200,7 +239,7 @@ true if the collection has pending load more operations, false otherwise get size(): number; ``` -Defined in: [packages/db/src/collection/index.ts:493](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L493) +Defined in: [packages/db/src/collection/index.ts:558](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L558) Get the current size of the collection (cached) @@ -218,7 +257,7 @@ Get the current size of the collection (cached) get state(): Map>; ``` -Defined in: [packages/db/src/collection/index.ts:820](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L820) +Defined in: [packages/db/src/collection/index.ts:885](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L885) Gets the current state of the collection as a Map @@ -254,7 +293,7 @@ Map containing all items in the collection, with keys as identifiers get status(): CollectionStatus; ``` -Defined in: [packages/db/src/collection/index.ts:411](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L411) +Defined in: [packages/db/src/collection/index.ts:418](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L418) Gets the current status of the collection @@ -272,7 +311,7 @@ Gets the current status of the collection get subscriberCount(): number; ``` -Defined in: [packages/db/src/collection/index.ts:418](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L418) +Defined in: [packages/db/src/collection/index.ts:425](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L425) Get the number of subscribers to the collection @@ -290,7 +329,7 @@ Get the number of subscribers to the collection get toArray(): WithVirtualProps[]; ``` -Defined in: [packages/db/src/collection/index.ts:849](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L849) +Defined in: [packages/db/src/collection/index.ts:914](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L914) Gets the current state of the collection as an Array @@ -302,13 +341,149 @@ An Array containing all items in the collection ## Methods +### \_deferPublication() + +```ts +_deferPublication(): PublicationDeferral; +``` + +Defined in: [packages/db/src/collection/index.ts:457](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L457) + +Defer subscriber events until a coherent multi-Collection commit ends. + +#### Returns + +`PublicationDeferral` + +*** + +### \_deferSyncStart() + +```ts +_deferSyncStart(): boolean; +``` + +Defined in: [packages/db/src/collection/index.ts:524](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L524) + +**`Internal`** + +#### Returns + +`boolean` + +*** + +### \_hasHydratedKey() + +```ts +_hasHydratedKey(key): boolean; +``` + +Defined in: [packages/db/src/collection/index.ts:519](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L519) + +**`Internal`** + +#### Parameters + +##### key + +`TKey` + +#### Returns + +`boolean` + +*** + +### \_markLayoutChange() + +```ts +_markLayoutChange(): void; +``` + +Defined in: [packages/db/src/collection/index.ts:452](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L452) + +Mark the active sync transaction as layout-changing. Internal. + +#### Returns + +`void` + +*** + +### \_resumeSyncStart() + +```ts +_resumeSyncStart(): void; +``` + +Defined in: [packages/db/src/collection/index.ts:529](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L529) + +**`Internal`** + +#### Returns + +`void` + +*** + +### \_setTransactionScope() + +```ts +_setTransactionScope(transactionScope): void; +``` + +Defined in: [packages/db/src/collection/index.ts:514](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L514) + +**`Internal`** + +#### Parameters + +##### transactionScope + +[`TransactionScope`](TransactionScope.md) + +#### Returns + +`void` + +*** + +### \_subscribeLayoutChanges() + +```ts +_subscribeLayoutChanges(listener): () => void; +``` + +Defined in: [packages/db/src/collection/index.ts:447](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L447) + +Subscribe to layout-only publications. Internal observer channel. + +#### Parameters + +##### listener + +() => `void` + +#### Returns + +```ts +(): void; +``` + +##### Returns + +`void` + +*** + ### \[iterator\]() ```ts iterator: IterableIterator<[TKey, WithVirtualProps]>; ``` -Defined in: [packages/db/src/collection/index.ts:531](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L531) +Defined in: [packages/db/src/collection/index.ts:596](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L596) Get all entries (virtual derived state) @@ -324,10 +499,12 @@ Get all entries (virtual derived state) cleanup(): Promise; ``` -Defined in: [packages/db/src/collection/index.ts:988](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L988) +Defined in: [packages/db/src/collection/index.ts:1055](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1055) Clean up the collection by stopping sync and clearing data This can be called manually or automatically by garbage collection +Cleanup callbacks must not restart this collection or call its preload(). +Wait until cleanup completes before starting a new sync session. #### Returns @@ -341,7 +518,7 @@ This can be called manually or automatically by garbage collection createIndex(indexCallback, config): BaseIndex; ``` -Defined in: [packages/db/src/collection/index.ts:597](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L597) +Defined in: [packages/db/src/collection/index.ts:662](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L662) Creates an index on a collection for faster queries. Indexes significantly improve query performance by allowing constant time lookups @@ -397,7 +574,7 @@ currentStateAsChanges(options): | ChangeMessage, string | number>[]; ``` -Defined in: [packages/db/src/collection/index.ts:887](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L887) +Defined in: [packages/db/src/collection/index.ts:952](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L952) Returns the current state of the collection as an array of changes @@ -441,7 +618,7 @@ const activeChanges = collection.currentStateAsChanges({ delete(keys, config?): Transaction; ``` -Defined in: [packages/db/src/collection/index.ts:797](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L797) +Defined in: [packages/db/src/collection/index.ts:862](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L862) Deletes one or more items from the collection @@ -504,7 +681,7 @@ try { entries(): IterableIterator<[TKey, WithVirtualProps]>; ``` -Defined in: [packages/db/src/collection/index.ts:519](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L519) +Defined in: [packages/db/src/collection/index.ts:584](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L584) Get all entries (virtual derived state) @@ -520,7 +697,7 @@ Get all entries (virtual derived state) forEach(callbackfn): void; ``` -Defined in: [packages/db/src/collection/index.ts:540](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L540) +Defined in: [packages/db/src/collection/index.ts:605](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L605) Execute a callback for each entry in the collection @@ -544,7 +721,7 @@ get(key): | undefined; ``` -Defined in: [packages/db/src/collection/index.ts:479](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L479) +Defined in: [packages/db/src/collection/index.ts:544](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L544) Get the current value for a key (virtual derived state) @@ -567,7 +744,7 @@ Get the current value for a key (virtual derived state) getIndexMetadata(): CollectionIndexMetadata[]; ``` -Defined in: [packages/db/src/collection/index.ts:621](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L621) +Defined in: [packages/db/src/collection/index.ts:686](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L686) Returns a snapshot of current index metadata sorted by indexId. Persistence wrappers can use this to bootstrap index state if indexes were @@ -585,7 +762,7 @@ created before event listeners were attached. getKeyFromItem(item): TKey; ``` -Defined in: [packages/db/src/collection/index.ts:571](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L571) +Defined in: [packages/db/src/collection/index.ts:636](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L636) #### Parameters @@ -605,7 +782,7 @@ Defined in: [packages/db/src/collection/index.ts:571](https://github.com/TanStac has(key): boolean; ``` -Defined in: [packages/db/src/collection/index.ts:486](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L486) +Defined in: [packages/db/src/collection/index.ts:551](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L551) Check if a key exists in the collection (virtual derived state) @@ -627,7 +804,7 @@ Check if a key exists in the collection (virtual derived state) insert(data, config?): Transaction>; ``` -Defined in: [packages/db/src/collection/index.ts:684](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L684) +Defined in: [packages/db/src/collection/index.ts:749](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L749) Inserts one or more items into the collection @@ -697,7 +874,7 @@ try { isReady(): boolean; ``` -Defined in: [packages/db/src/collection/index.ts:448](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L448) +Defined in: [packages/db/src/collection/index.ts:492](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L492) Check if the collection is ready for use Returns true if the collection has been marked as ready by its sync implementation @@ -727,7 +904,7 @@ if (collection.isReady()) { keys(): IterableIterator; ``` -Defined in: [packages/db/src/collection/index.ts:500](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L500) +Defined in: [packages/db/src/collection/index.ts:565](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L565) Get all keys (virtual derived state) @@ -743,7 +920,7 @@ Get all keys (virtual derived state) map(callbackfn): U[]; ``` -Defined in: [packages/db/src/collection/index.ts:556](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L556) +Defined in: [packages/db/src/collection/index.ts:621](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L621) Create a new array with the results of calling a function for each entry in the collection @@ -771,7 +948,7 @@ Create a new array with the results of calling a function for each entry in the off(event, callback): void; ``` -Defined in: [packages/db/src/collection/index.ts:967](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L967) +Defined in: [packages/db/src/collection/index.ts:1032](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1032) Unsubscribe from a collection event @@ -814,7 +991,7 @@ Unsubscribe from a collection event on(event, callback): () => void; ``` -Defined in: [packages/db/src/collection/index.ts:947](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L947) +Defined in: [packages/db/src/collection/index.ts:1012](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1012) Subscribe to a collection event @@ -863,7 +1040,7 @@ Subscribe to a collection event once(event, callback): () => void; ``` -Defined in: [packages/db/src/collection/index.ts:957](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L957) +Defined in: [packages/db/src/collection/index.ts:1022](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1022) Subscribe to a collection event once @@ -909,13 +1086,18 @@ Subscribe to a collection event once ### onFirstReady() ```ts -onFirstReady(callback): void; +onFirstReady(callback): () => void; ``` -Defined in: [packages/db/src/collection/index.ts:432](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L432) +Defined in: [packages/db/src/collection/index.ts:476](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L476) Register a callback to be executed when the collection first becomes ready Useful for preloading collections +Every callback queued before the transition runs. Because ready state is +established first, callbacks registered during or after delivery run +immediately. If one throws, the collection remains ready. Direct sync +startup rethrows the first failure; preload resolves from ready state. +Cleanup discards pending callbacks without invoking them. #### Parameters @@ -927,6 +1109,12 @@ Function to call when the collection first becomes ready #### Returns +```ts +(): void; +``` + +##### Returns + `void` #### Example @@ -946,7 +1134,7 @@ collection.onFirstReady(() => { preload(): Promise; ``` -Defined in: [packages/db/src/collection/index.ts:472](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L472) +Defined in: [packages/db/src/collection/index.ts:537](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L537) Preload the collection data by starting sync if not already started Multiple concurrent calls will share the same promise @@ -963,7 +1151,7 @@ Multiple concurrent calls will share the same promise removeIndex(indexOrId): boolean; ``` -Defined in: [packages/db/src/collection/index.ts:612](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L612) +Defined in: [packages/db/src/collection/index.ts:677](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L677) Removes an index created with createIndex. Returns true when an index existed and was removed. @@ -990,10 +1178,11 @@ as invalid after removal. startSyncImmediate(): void; ``` -Defined in: [packages/db/src/collection/index.ts:464](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L464) +Defined in: [packages/db/src/collection/index.ts:509](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L509) Start sync immediately - internal method for compiled queries This bypasses lazy loading for special cases like live query results +Throws during active cleanup; restart after cleanup completes instead. #### Returns @@ -1007,7 +1196,7 @@ This bypasses lazy loading for special cases like live query results stateWhenReady(): Promise>>; ``` -Defined in: [packages/db/src/collection/index.ts:834](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L834) +Defined in: [packages/db/src/collection/index.ts:899](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L899) Gets the current state of the collection as a Map, but only resolves when data is available Waits for the first sync commit to complete before resolving @@ -1026,7 +1215,7 @@ Promise that resolves to a Map containing all items in the collection subscribeChanges(callback, options): CollectionSubscription; ``` -Defined in: [packages/db/src/collection/index.ts:935](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L935) +Defined in: [packages/db/src/collection/index.ts:1000](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1000) Subscribe to changes in the collection @@ -1101,7 +1290,7 @@ const subscription = collection.subscribeChanges((changes) => { toArrayWhenReady(): Promise[]>; ``` -Defined in: [packages/db/src/collection/index.ts:859](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L859) +Defined in: [packages/db/src/collection/index.ts:924](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L924) Gets the current state of the collection as an Array, but only resolves when data is available Waits for the first sync commit to complete before resolving @@ -1122,7 +1311,7 @@ Promise that resolves to an Array containing all items in the collection update(key, callback): Transaction; ``` -Defined in: [packages/db/src/collection/index.ts:729](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L729) +Defined in: [packages/db/src/collection/index.ts:794](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L794) Updates one or more items in the collection using a callback function @@ -1193,7 +1382,7 @@ update( callback): Transaction; ``` -Defined in: [packages/db/src/collection/index.ts:735](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L735) +Defined in: [packages/db/src/collection/index.ts:800](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L800) Updates one or more items in the collection using a callback function @@ -1267,7 +1456,7 @@ try { update(id, callback): Transaction; ``` -Defined in: [packages/db/src/collection/index.ts:742](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L742) +Defined in: [packages/db/src/collection/index.ts:807](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L807) Updates one or more items in the collection using a callback function @@ -1338,7 +1527,7 @@ update( callback): Transaction; ``` -Defined in: [packages/db/src/collection/index.ts:748](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L748) +Defined in: [packages/db/src/collection/index.ts:813](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L813) Updates one or more items in the collection using a callback function @@ -1415,7 +1604,7 @@ validateData( key?): TOutput; ``` -Defined in: [packages/db/src/collection/index.ts:635](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L635) +Defined in: [packages/db/src/collection/index.ts:700](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L700) Validates the data against the schema @@ -1445,7 +1634,7 @@ Validates the data against the schema values(): IterableIterator>; ``` -Defined in: [packages/db/src/collection/index.ts:507](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L507) +Defined in: [packages/db/src/collection/index.ts:572](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L572) Get all values (virtual derived state) @@ -1461,7 +1650,7 @@ Get all values (virtual derived state) waitFor(event, timeout?): Promise; ``` -Defined in: [packages/db/src/collection/index.ts:977](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L977) +Defined in: [packages/db/src/collection/index.ts:1042](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1042) Wait for a collection event diff --git a/docs/reference/classes/CollectionInputNotFoundError.md b/docs/reference/classes/CollectionInputNotFoundError.md index b2d75b3f58..5ea78b239b 100644 --- a/docs/reference/classes/CollectionInputNotFoundError.md +++ b/docs/reference/classes/CollectionInputNotFoundError.md @@ -5,7 +5,7 @@ title: CollectionInputNotFoundError # Class: CollectionInputNotFoundError -Defined in: [packages/db/src/errors.ts:474](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L474) +Defined in: [packages/db/src/errors.ts:521](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L521) Error thrown when a collection input stream is not found during query compilation. In self-joins, each alias (e.g., 'employee', 'manager') requires its own input stream. @@ -25,7 +25,7 @@ new CollectionInputNotFoundError( availableKeys?): CollectionInputNotFoundError; ``` -Defined in: [packages/db/src/errors.ts:475](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L475) +Defined in: [packages/db/src/errors.ts:522](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L522) #### Parameters diff --git a/docs/reference/classes/CollectionOperationError.md b/docs/reference/classes/CollectionOperationError.md index 3de9c81444..b7a1ea6be0 100644 --- a/docs/reference/classes/CollectionOperationError.md +++ b/docs/reference/classes/CollectionOperationError.md @@ -5,7 +5,7 @@ title: CollectionOperationError # Class: CollectionOperationError -Defined in: [packages/db/src/errors.ts:139](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L139) +Defined in: [packages/db/src/errors.ts:151](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L151) ## Extends @@ -32,7 +32,7 @@ Defined in: [packages/db/src/errors.ts:139](https://github.com/TanStack/db/blob/ new CollectionOperationError(message): CollectionOperationError; ``` -Defined in: [packages/db/src/errors.ts:140](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L140) +Defined in: [packages/db/src/errors.ts:152](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L152) #### Parameters diff --git a/docs/reference/classes/CollectionPreloadAbortedError.md b/docs/reference/classes/CollectionPreloadAbortedError.md new file mode 100644 index 0000000000..5807998628 --- /dev/null +++ b/docs/reference/classes/CollectionPreloadAbortedError.md @@ -0,0 +1,232 @@ +--- +id: CollectionPreloadAbortedError +title: CollectionPreloadAbortedError +--- + +# Class: CollectionPreloadAbortedError + +Defined in: [packages/db/src/errors.ts:741](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L741) + +A collection was cleaned up before its initial preload became ready. + +## Extends + +- `Error` + +## Constructors + +### Constructor + +```ts +new CollectionPreloadAbortedError(): CollectionPreloadAbortedError; +``` + +Defined in: [packages/db/src/errors.ts:742](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L742) + +#### Returns + +`CollectionPreloadAbortedError` + +#### Overrides + +```ts +Error.constructor +``` + +## Properties + +### cause? + +```ts +optional cause: unknown; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es2022.error.d.ts:26 + +#### Inherited from + +```ts +Error.cause +``` + +*** + +### message + +```ts +message: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1077 + +#### Inherited from + +```ts +Error.message +``` + +*** + +### name + +```ts +name: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1076 + +#### Inherited from + +```ts +Error.name +``` + +*** + +### stack? + +```ts +optional stack: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1078 + +#### Inherited from + +```ts +Error.stack +``` + +*** + +### stackTraceLimit + +```ts +static stackTraceLimit: number; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:67 + +The `Error.stackTraceLimit` property specifies the number of stack frames +collected by a stack trace (whether generated by `new Error().stack` or +`Error.captureStackTrace(obj)`). + +The default value is `10` but may be set to any valid JavaScript number. Changes +will affect any stack trace captured _after_ the value has been changed. + +If set to a non-number value, or set to a negative number, stack traces will +not capture any frames. + +#### Inherited from + +```ts +Error.stackTraceLimit +``` + +## Methods + +### captureStackTrace() + +```ts +static captureStackTrace(targetObject, constructorOpt?): void; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:51 + +Creates a `.stack` property on `targetObject`, which when accessed returns +a string representing the location in the code at which +`Error.captureStackTrace()` was called. + +```js +const myObject = {}; +Error.captureStackTrace(myObject); +myObject.stack; // Similar to `new Error().stack` +``` + +The first line of the trace will be prefixed with +`${myObject.name}: ${myObject.message}`. + +The optional `constructorOpt` argument accepts a function. If given, all frames +above `constructorOpt`, including `constructorOpt`, will be omitted from the +generated stack trace. + +The `constructorOpt` argument is useful for hiding implementation +details of error generation from the user. For instance: + +```js +function a() { + b(); +} + +function b() { + c(); +} + +function c() { + // Create an error without stack trace to avoid calculating the stack trace twice. + const { stackTraceLimit } = Error; + Error.stackTraceLimit = 0; + const error = new Error(); + Error.stackTraceLimit = stackTraceLimit; + + // Capture the stack trace above function b + Error.captureStackTrace(error, b); // Neither function c, nor b is included in the stack trace + throw error; +} + +a(); +``` + +#### Parameters + +##### targetObject + +`object` + +##### constructorOpt? + +`Function` + +#### Returns + +`void` + +#### Inherited from + +```ts +Error.captureStackTrace +``` + +*** + +### prepareStackTrace() + +```ts +static prepareStackTrace(err, stackTraces): any; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:55 + +#### Parameters + +##### err + +`Error` + +##### stackTraces + +`CallSite`[] + +#### Returns + +`any` + +#### See + +https://v8.dev/docs/stack-trace-api#customizing-stack-traces + +#### Inherited from + +```ts +Error.prepareStackTrace +``` diff --git a/docs/reference/classes/CollectionStateError.md b/docs/reference/classes/CollectionStateError.md index f35a176c1a..c514ff8838 100644 --- a/docs/reference/classes/CollectionStateError.md +++ b/docs/reference/classes/CollectionStateError.md @@ -17,6 +17,8 @@ Defined in: [packages/db/src/errors.ts:103](https://github.com/TanStack/db/blob/ - [`InvalidCollectionStatusTransitionError`](InvalidCollectionStatusTransitionError.md) - [`CollectionIsInErrorStateError`](CollectionIsInErrorStateError.md) - [`NegativeActiveSubscribersError`](NegativeActiveSubscribersError.md) +- [`LiveQueryObserverDisposedError`](LiveQueryObserverDisposedError.md) +- [`LiveQueryWindowControllerDisposedError`](LiveQueryWindowControllerDisposedError.md) ## Constructors diff --git a/docs/reference/classes/DbClient.md b/docs/reference/classes/DbClient.md new file mode 100644 index 0000000000..091112601a --- /dev/null +++ b/docs/reference/classes/DbClient.md @@ -0,0 +1,637 @@ +--- +id: DbClient +title: DbClient +--- + +# Class: DbClient + +Defined in: [packages/db/src/client.ts:308](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L308) + +## Constructors + +### Constructor + +```ts +new DbClient(options): DbClient; +``` + +Defined in: [packages/db/src/client.ts:326](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L326) + +#### Parameters + +##### options + +[`DbClientOptions`](../type-aliases/DbClientOptions.md) = `{}` + +#### Returns + +`DbClient` + +## Accessors + +### activeTransaction + +#### Get Signature + +```ts +get activeTransaction(): + | Transaction> + | undefined; +``` + +Defined in: [packages/db/src/client.ts:342](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L342) + +##### Returns + + \| [`Transaction`](../interfaces/Transaction.md)\<`Record`\<`string`, `unknown`\>\> + \| `undefined` + +## Methods + +### \_consumeLiveQueryResult() + +```ts +_consumeLiveQueryResult(queryHash, dehydratedAt): void; +``` + +Defined in: [packages/db/src/client.ts:635](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L635) + +**`Internal`** + +#### Parameters + +##### queryHash + +`string` + +##### dehydratedAt + +`number` + +#### Returns + +`void` + +*** + +### \_failPendingLiveQueries() + +```ts +_failPendingLiveQueries(error): void; +``` + +Defined in: [packages/db/src/client.ts:678](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L678) + +**`Internal`** + +#### Parameters + +##### error + +`unknown` + +#### Returns + +`void` + +*** + +### \_getLiveQuery() + +```ts +_getLiveQuery(queryHash): DbClientLiveQuery | undefined; +``` + +Defined in: [packages/db/src/client.ts:630](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L630) + +**`Internal`** + +#### Parameters + +##### queryHash + +`string` + +#### Returns + +[`DbClientLiveQuery`](../type-aliases/DbClientLiveQuery.md) \| `undefined` + +*** + +### \_isSsrServerCleanupEnabled() + +```ts +_isSsrServerCleanupEnabled(): boolean; +``` + +Defined in: [packages/db/src/client.ts:625](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L625) + +**`Internal`** + +#### Returns + +`boolean` + +*** + +### \_isSsrStreamingEnabled() + +```ts +_isSsrStreamingEnabled(): boolean; +``` + +Defined in: [packages/db/src/client.ts:615](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L615) + +**`Internal`** + +#### Returns + +`boolean` + +*** + +### \_materializeCollectionForRender() + +```ts +_materializeCollectionForRender(options): Collection>; +``` + +Defined in: [packages/db/src/client.ts:432](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L432) + +**`Internal`** + +#### Type Parameters + +##### T + +`T` *extends* `object` + +##### TKey + +`TKey` *extends* `string` \| `number` + +##### TSchema + +`TSchema` *extends* `StandardSchemaV1`\<`unknown`, `unknown`\> + +##### TUtils + +`TUtils` *extends* [`UtilsRecord`](../type-aliases/UtilsRecord.md) + +#### Parameters + +##### options + +[`CollectionOptions`](../type-aliases/CollectionOptions.md)\<`T`, `TKey`, `TSchema`, `TUtils`\> + +#### Returns + +[`Collection`](../interfaces/Collection.md)\<`T`, `TKey`, `TUtils`, `TSchema`, \[`TSchema`\] *extends* \[`never`\] ? `T` : [`InferSchemaInput`](../type-aliases/InferSchemaInput.md)\<`TSchema`\>\> + +*** + +### \_registerLiveQuery() + +```ts +_registerLiveQuery(queryHash, promise): Promise; +``` + +Defined in: [packages/db/src/client.ts:643](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L643) + +**`Internal`** + +#### Parameters + +##### queryHash + +`string` + +##### promise + +`Promise`\<[`DehydratedLiveQueryResult`](../type-aliases/DehydratedLiveQueryResult.md)\<`object`, `string` \| `number`\>\> + +#### Returns + +`Promise`\<`void`\> + +*** + +### \_registerLiveQueryResource() + +```ts +_registerLiveQueryResource(owner, cleanup): () => void; +``` + +Defined in: [packages/db/src/client.ts:665](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L665) + +**`Internal`** + +#### Parameters + +##### owner + +`object` + +##### cleanup + +() => `Promise`\<`void`\> + +#### Returns + +```ts +(): void; +``` + +##### Returns + +`void` + +*** + +### \_setSsrServerCleanupEnabled() + +```ts +_setSsrServerCleanupEnabled(enabled): void; +``` + +Defined in: [packages/db/src/client.ts:620](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L620) + +**`Internal`** + +#### Parameters + +##### enabled + +`boolean` + +#### Returns + +`void` + +*** + +### \_setSsrStreamingEnabled() + +```ts +_setSsrStreamingEnabled(enabled): void; +``` + +Defined in: [packages/db/src/client.ts:610](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L610) + +**`Internal`** + +#### Parameters + +##### enabled + +`boolean` + +#### Returns + +`void` + +*** + +### applyCollectionChunk() + +```ts +applyCollectionChunk(chunk): void; +``` + +Defined in: [packages/db/src/client.ts:600](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L600) + +#### Parameters + +##### chunk + +[`DehydratedCollectionChunk`](../type-aliases/DehydratedCollectionChunk.md) + +#### Returns + +`void` + +*** + +### cleanup() + +```ts +cleanup(): Promise; +``` + +Defined in: [packages/db/src/client.ts:685](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L685) + +#### Returns + +`Promise`\<`void`\> + +*** + +### collection() + +#### Call Signature + +```ts +collection(options, materializeOptions?): Collection, TKey, TUtils, T, InferSchemaInput> & NonSingleResult; +``` + +Defined in: [packages/db/src/client.ts:388](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L388) + +##### Type Parameters + +###### T + +`T` *extends* `StandardSchemaV1`\<`unknown`, `unknown`\> + +###### TKey + +`TKey` *extends* `string` \| `number` + +###### TUtils + +`TUtils` *extends* [`UtilsRecord`](../type-aliases/UtilsRecord.md) + +##### Parameters + +###### options + +[`CollectionOptions`](../type-aliases/CollectionOptions.md)\<[`InferSchemaOutput`](../type-aliases/InferSchemaOutput.md)\<`T`\>, `TKey`, `T`, `TUtils`\> & [`NonSingleResult`](../type-aliases/NonSingleResult.md) + +###### materializeOptions? + +[`CollectionMaterializeOptions`](../type-aliases/CollectionMaterializeOptions.md)\<[`InferSchemaInput`](../type-aliases/InferSchemaInput.md)\<`T`\>\> + +##### Returns + +[`Collection`](../interfaces/Collection.md)\<[`InferSchemaOutput`](../type-aliases/InferSchemaOutput.md)\<`T`\>, `TKey`, `TUtils`, `T`, [`InferSchemaInput`](../type-aliases/InferSchemaInput.md)\<`T`\>\> & [`NonSingleResult`](../type-aliases/NonSingleResult.md) + +#### Call Signature + +```ts +collection(options, materializeOptions?): Collection, TKey, TUtils, T, InferSchemaInput> & SingleResult; +``` + +Defined in: [packages/db/src/client.ts:398](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L398) + +##### Type Parameters + +###### T + +`T` *extends* `StandardSchemaV1`\<`unknown`, `unknown`\> + +###### TKey + +`TKey` *extends* `string` \| `number` + +###### TUtils + +`TUtils` *extends* [`UtilsRecord`](../type-aliases/UtilsRecord.md) + +##### Parameters + +###### options + +[`CollectionOptions`](../type-aliases/CollectionOptions.md)\<[`InferSchemaOutput`](../type-aliases/InferSchemaOutput.md)\<`T`\>, `TKey`, `T`, `TUtils`\> & [`SingleResult`](../type-aliases/SingleResult.md) + +###### materializeOptions? + +[`CollectionMaterializeOptions`](../type-aliases/CollectionMaterializeOptions.md)\<[`InferSchemaInput`](../type-aliases/InferSchemaInput.md)\<`T`\>\> + +##### Returns + +[`Collection`](../interfaces/Collection.md)\<[`InferSchemaOutput`](../type-aliases/InferSchemaOutput.md)\<`T`\>, `TKey`, `TUtils`, `T`, [`InferSchemaInput`](../type-aliases/InferSchemaInput.md)\<`T`\>\> & [`SingleResult`](../type-aliases/SingleResult.md) + +#### Call Signature + +```ts +collection(options, materializeOptions?): Collection & NonSingleResult; +``` + +Defined in: [packages/db/src/client.ts:408](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L408) + +##### Type Parameters + +###### T + +`T` *extends* `object` + +###### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +###### TUtils + +`TUtils` *extends* [`UtilsRecord`](../type-aliases/UtilsRecord.md) = [`UtilsRecord`](../type-aliases/UtilsRecord.md) + +##### Parameters + +###### options + +[`CollectionOptions`](../type-aliases/CollectionOptions.md)\<`T`, `TKey`, `never`, `TUtils`\> & [`NonSingleResult`](../type-aliases/NonSingleResult.md) + +###### materializeOptions? + +[`CollectionMaterializeOptions`](../type-aliases/CollectionMaterializeOptions.md)\<`T`\> + +##### Returns + +[`Collection`](../interfaces/Collection.md)\<`T`, `TKey`, `TUtils`, `never`, `T`\> & [`NonSingleResult`](../type-aliases/NonSingleResult.md) + +#### Call Signature + +```ts +collection(options, materializeOptions?): Collection & SingleResult; +``` + +Defined in: [packages/db/src/client.ts:416](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L416) + +##### Type Parameters + +###### T + +`T` *extends* `object` + +###### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +###### TUtils + +`TUtils` *extends* [`UtilsRecord`](../type-aliases/UtilsRecord.md) = [`UtilsRecord`](../type-aliases/UtilsRecord.md) + +##### Parameters + +###### options + +[`CollectionOptions`](../type-aliases/CollectionOptions.md)\<`T`, `TKey`, `never`, `TUtils`\> & [`SingleResult`](../type-aliases/SingleResult.md) + +###### materializeOptions? + +[`CollectionMaterializeOptions`](../type-aliases/CollectionMaterializeOptions.md)\<`T`\> + +##### Returns + +[`Collection`](../interfaces/Collection.md)\<`T`, `TKey`, `TUtils`, `never`, `T`\> & [`SingleResult`](../type-aliases/SingleResult.md) + +*** + +### createTransaction() + +```ts +createTransaction(config): Transaction; +``` + +Defined in: [packages/db/src/client.ts:346](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L346) + +#### Type Parameters + +##### T + +`T` *extends* `object` = `Record`\<`string`, `unknown`\> + +#### Parameters + +##### config + +[`TransactionConfig`](../interfaces/TransactionConfig.md)\<`T`\> + +#### Returns + +[`Transaction`](../interfaces/Transaction.md)\<`T`\> + +*** + +### dehydrate() + +```ts +dehydrate(options): DehydratedDbState; +``` + +Defined in: [packages/db/src/client.ts:521](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L521) + +#### Parameters + +##### options + +[`DehydrateDbClientOptions`](../type-aliases/DehydrateDbClientOptions.md) = `{}` + +#### Returns + +[`DehydratedDbState`](../type-aliases/DehydratedDbState.md) + +*** + +### getDependency() + +```ts +getDependency(key): T | undefined; +``` + +Defined in: [packages/db/src/client.ts:328](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L328) + +#### Type Parameters + +##### T + +`T` + +#### Parameters + +##### key + +`string` + +#### Returns + +`T` \| `undefined` + +*** + +### hydrate() + +```ts +hydrate(state): void; +``` + +Defined in: [packages/db/src/client.ts:582](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L582) + +#### Parameters + +##### state + +[`DehydratedDbState`](../type-aliases/DehydratedDbState.md) + +#### Returns + +`void` + +*** + +### preloadLiveQuery() + +```ts +preloadLiveQuery(options): Promise; +``` + +Defined in: [packages/db/src/client.ts:352](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L352) + +#### Parameters + +##### options + +[`LiveQueryOptions`](../type-aliases/LiveQueryOptions.md) + +#### Returns + +`Promise`\<`void`\> + +*** + +### requireDependency() + +```ts +requireDependency(key): T; +``` + +Defined in: [packages/db/src/client.ts:332](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L332) + +#### Type Parameters + +##### T + +`T` + +#### Parameters + +##### key + +`string` + +#### Returns + +`T` + +*** + +### subscribe() + +```ts +subscribe(listener): () => void; +``` + +Defined in: [packages/db/src/client.ts:604](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L604) + +#### Parameters + +##### listener + +(`event`) => `void` + +#### Returns + +```ts +(): void; +``` + +##### Returns + +`void` diff --git a/docs/reference/classes/DeduplicatedLoadSubset.md b/docs/reference/classes/DeduplicatedLoadSubset.md index eb0a80f794..fb0742ad1a 100644 --- a/docs/reference/classes/DeduplicatedLoadSubset.md +++ b/docs/reference/classes/DeduplicatedLoadSubset.md @@ -5,60 +5,28 @@ title: DeduplicatedLoadSubset # Class: DeduplicatedLoadSubset -Defined in: [packages/db/src/query/subset-dedupe.ts:34](https://github.com/TanStack/db/blob/main/packages/db/src/query/subset-dedupe.ts#L34) +Defined in: [packages/db/src/query/subset-dedupe.ts:8](https://github.com/TanStack/db/blob/main/packages/db/src/query/subset-dedupe.ts#L8) -Deduplicated wrapper for a loadSubset function. -Tracks what data has been loaded and avoids redundant calls by applying -subset logic to predicates. - -## Param - -The options for the DeduplicatedLoadSubset - -## Param - -The underlying loadSubset function to wrap - -## Param - -An optional callback function that is invoked when a loadSubset call is deduplicated. - If the call is deduplicated because the requested data is being loaded by an inflight request, - then this callback is invoked when the inflight request completes successfully and the data is fully loaded. - This callback is useful if you need to track rows per query, in which case you can't ignore deduplicated calls - because you need to know which rows were loaded for each query. - -## Example - -```ts -const dedupe = new DeduplicatedLoadSubset({ loadSubset: myLoadSubset, onDeduplicate: (opts) => console.log(`Call was deduplicated:`, opts) }) - -// First call - fetches data -await dedupe.loadSubset({ where: gt(ref('age'), val(10)) }) - -// Second call - subset of first, returns true immediately -await dedupe.loadSubset({ where: gt(ref('age'), val(20)) }) - -// Clear state to start fresh -dedupe.reset() -``` +Deduplicates exact canonical demands without inferring broader coverage. +Requests follow the immutable LoadSubsetOptions contract; no copies are made. ## Constructors ### Constructor ```ts -new DeduplicatedLoadSubset(opts): DeduplicatedLoadSubset; +new DeduplicatedLoadSubset(options): DeduplicatedLoadSubset; ``` -Defined in: [packages/db/src/query/subset-dedupe.ts:67](https://github.com/TanStack/db/blob/main/packages/db/src/query/subset-dedupe.ts#L67) +Defined in: [packages/db/src/query/subset-dedupe.ts:13](https://github.com/TanStack/db/blob/main/packages/db/src/query/subset-dedupe.ts#L13) #### Parameters -##### opts +##### options ###### loadSubset -(`options`) => `true` \| `Promise`\<`void`\> +[`LoadSubsetFn`](../type-aliases/LoadSubsetFn.md) ###### onDeduplicate? @@ -76,13 +44,7 @@ Defined in: [packages/db/src/query/subset-dedupe.ts:67](https://github.com/TanSt loadSubset(options): true | Promise; ``` -Defined in: [packages/db/src/query/subset-dedupe.ts:85](https://github.com/TanStack/db/blob/main/packages/db/src/query/subset-dedupe.ts#L85) - -Load a subset of data, with automatic deduplication based on previously -loaded predicates and in-flight requests. - -This method is auto-bound, so it can be safely passed as a callback without -losing its `this` context (e.g., `loadSubset: dedupe.loadSubset` in a sync config). +Defined in: [packages/db/src/query/subset-dedupe.ts:20](https://github.com/TanStack/db/blob/main/packages/db/src/query/subset-dedupe.ts#L20) #### Parameters @@ -90,14 +52,10 @@ losing its `this` context (e.g., `loadSubset: dedupe.loadSubset` in a sync confi [`LoadSubsetOptions`](../type-aliases/LoadSubsetOptions.md) -The predicate options (where, orderBy, limit) - #### Returns `true` \| `Promise`\<`void`\> -true if data is already loaded, or a Promise that resolves when data is loaded - *** ### reset() @@ -106,14 +64,7 @@ true if data is already loaded, or a Promise that resolves when data is loaded reset(): void; ``` -Defined in: [packages/db/src/query/subset-dedupe.ts:194](https://github.com/TanStack/db/blob/main/packages/db/src/query/subset-dedupe.ts#L194) - -Reset all tracking state. -Clears the history of loaded predicates and in-flight calls. -Use this when you want to start fresh, for example after clearing the underlying data store. - -Note: Any in-flight requests will still complete, but they will not update the tracking -state after the reset. This prevents old requests from repopulating cleared state. +Defined in: [packages/db/src/query/subset-dedupe.ts:64](https://github.com/TanStack/db/blob/main/packages/db/src/query/subset-dedupe.ts#L64) #### Returns diff --git a/docs/reference/classes/DeleteKeyNotFoundError.md b/docs/reference/classes/DeleteKeyNotFoundError.md index 3bfcce5ba0..9756c56ac1 100644 --- a/docs/reference/classes/DeleteKeyNotFoundError.md +++ b/docs/reference/classes/DeleteKeyNotFoundError.md @@ -5,7 +5,7 @@ title: DeleteKeyNotFoundError # Class: DeleteKeyNotFoundError -Defined in: [packages/db/src/errors.ts:245](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L245) +Defined in: [packages/db/src/errors.ts:257](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L257) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:245](https://github.com/TanStack/db/blob/ new DeleteKeyNotFoundError(key): DeleteKeyNotFoundError; ``` -Defined in: [packages/db/src/errors.ts:246](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L246) +Defined in: [packages/db/src/errors.ts:258](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L258) #### Parameters diff --git a/docs/reference/classes/DistinctRequiresSelectError.md b/docs/reference/classes/DistinctRequiresSelectError.md index c72c004535..ce65f1e7e3 100644 --- a/docs/reference/classes/DistinctRequiresSelectError.md +++ b/docs/reference/classes/DistinctRequiresSelectError.md @@ -5,7 +5,7 @@ title: DistinctRequiresSelectError # Class: DistinctRequiresSelectError -Defined in: [packages/db/src/errors.ts:430](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L430) +Defined in: [packages/db/src/errors.ts:467](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L467) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:430](https://github.com/TanStack/db/blob/ new DistinctRequiresSelectError(): DistinctRequiresSelectError; ``` -Defined in: [packages/db/src/errors.ts:431](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L431) +Defined in: [packages/db/src/errors.ts:468](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L468) #### Returns diff --git a/docs/reference/classes/DuplicateAliasInSubqueryError.md b/docs/reference/classes/DuplicateAliasInSubqueryError.md index d32ef272f9..78a9a75d3f 100644 --- a/docs/reference/classes/DuplicateAliasInSubqueryError.md +++ b/docs/reference/classes/DuplicateAliasInSubqueryError.md @@ -5,7 +5,7 @@ title: DuplicateAliasInSubqueryError # Class: DuplicateAliasInSubqueryError -Defined in: [packages/db/src/errors.ts:495](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L495) +Defined in: [packages/db/src/errors.ts:542](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L542) Error thrown when a subquery uses the same alias as its parent query. This causes issues because parent and subquery would share the same input streams, @@ -23,7 +23,7 @@ leading to empty results or incorrect data (aggregation cross-leaking). new DuplicateAliasInSubqueryError(alias, parentAliases): DuplicateAliasInSubqueryError; ``` -Defined in: [packages/db/src/errors.ts:496](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L496) +Defined in: [packages/db/src/errors.ts:543](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L543) #### Parameters diff --git a/docs/reference/classes/DuplicateKeyError.md b/docs/reference/classes/DuplicateKeyError.md index 424d3daffa..c240ec2733 100644 --- a/docs/reference/classes/DuplicateKeyError.md +++ b/docs/reference/classes/DuplicateKeyError.md @@ -5,7 +5,7 @@ title: DuplicateKeyError # Class: DuplicateKeyError -Defined in: [packages/db/src/errors.ts:163](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L163) +Defined in: [packages/db/src/errors.ts:175](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L175) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:163](https://github.com/TanStack/db/blob/ new DuplicateKeyError(key): DuplicateKeyError; ``` -Defined in: [packages/db/src/errors.ts:164](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L164) +Defined in: [packages/db/src/errors.ts:176](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L176) #### Parameters diff --git a/docs/reference/classes/DuplicateKeySyncError.md b/docs/reference/classes/DuplicateKeySyncError.md index 8f9cd398e9..bad5c84e0d 100644 --- a/docs/reference/classes/DuplicateKeySyncError.md +++ b/docs/reference/classes/DuplicateKeySyncError.md @@ -5,7 +5,7 @@ title: DuplicateKeySyncError # Class: DuplicateKeySyncError -Defined in: [packages/db/src/errors.ts:171](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L171) +Defined in: [packages/db/src/errors.ts:183](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L183) ## Extends @@ -22,7 +22,7 @@ new DuplicateKeySyncError( options?): DuplicateKeySyncError; ``` -Defined in: [packages/db/src/errors.ts:172](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L172) +Defined in: [packages/db/src/errors.ts:184](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L184) #### Parameters diff --git a/docs/reference/classes/EmptyReferencePathError.md b/docs/reference/classes/EmptyReferencePathError.md index 2b5583daa0..dde90cc06b 100644 --- a/docs/reference/classes/EmptyReferencePathError.md +++ b/docs/reference/classes/EmptyReferencePathError.md @@ -5,7 +5,7 @@ title: EmptyReferencePathError # Class: EmptyReferencePathError -Defined in: [packages/db/src/errors.ts:518](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L518) +Defined in: [packages/db/src/errors.ts:565](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L565) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:518](https://github.com/TanStack/db/blob/ new EmptyReferencePathError(): EmptyReferencePathError; ``` -Defined in: [packages/db/src/errors.ts:519](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L519) +Defined in: [packages/db/src/errors.ts:566](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L566) #### Returns diff --git a/docs/reference/classes/FnSelectWithGroupByError.md b/docs/reference/classes/FnSelectWithGroupByError.md index 4886f6e568..377a262527 100644 --- a/docs/reference/classes/FnSelectWithGroupByError.md +++ b/docs/reference/classes/FnSelectWithGroupByError.md @@ -5,7 +5,7 @@ title: FnSelectWithGroupByError # Class: FnSelectWithGroupByError -Defined in: [packages/db/src/errors.ts:436](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L436) +Defined in: [packages/db/src/errors.ts:473](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L473) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:436](https://github.com/TanStack/db/blob/ new FnSelectWithGroupByError(): FnSelectWithGroupByError; ``` -Defined in: [packages/db/src/errors.ts:437](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L437) +Defined in: [packages/db/src/errors.ts:474](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L474) #### Returns diff --git a/docs/reference/classes/GroupByError.md b/docs/reference/classes/GroupByError.md index c010300fba..3cdd9d5638 100644 --- a/docs/reference/classes/GroupByError.md +++ b/docs/reference/classes/GroupByError.md @@ -5,7 +5,7 @@ title: GroupByError # Class: GroupByError -Defined in: [packages/db/src/errors.ts:593](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L593) +Defined in: [packages/db/src/errors.ts:640](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L640) ## Extends @@ -26,7 +26,7 @@ Defined in: [packages/db/src/errors.ts:593](https://github.com/TanStack/db/blob/ new GroupByError(message): GroupByError; ``` -Defined in: [packages/db/src/errors.ts:594](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L594) +Defined in: [packages/db/src/errors.ts:641](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L641) #### Parameters diff --git a/docs/reference/classes/HavingRequiresGroupByError.md b/docs/reference/classes/HavingRequiresGroupByError.md index bbf7cdb8fe..b441b7febc 100644 --- a/docs/reference/classes/HavingRequiresGroupByError.md +++ b/docs/reference/classes/HavingRequiresGroupByError.md @@ -5,7 +5,7 @@ title: HavingRequiresGroupByError # Class: HavingRequiresGroupByError -Defined in: [packages/db/src/errors.ts:456](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L456) +Defined in: [packages/db/src/errors.ts:503](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L503) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:456](https://github.com/TanStack/db/blob/ new HavingRequiresGroupByError(): HavingRequiresGroupByError; ``` -Defined in: [packages/db/src/errors.ts:457](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L457) +Defined in: [packages/db/src/errors.ts:504](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L504) #### Returns diff --git a/docs/reference/classes/InvalidJoinCondition.md b/docs/reference/classes/InvalidJoinCondition.md index 777521bdc4..ed1ed73688 100644 --- a/docs/reference/classes/InvalidJoinCondition.md +++ b/docs/reference/classes/InvalidJoinCondition.md @@ -5,7 +5,7 @@ title: InvalidJoinCondition # Class: InvalidJoinCondition -Defined in: [packages/db/src/errors.ts:580](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L580) +Defined in: [packages/db/src/errors.ts:627](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L627) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:580](https://github.com/TanStack/db/blob/ new InvalidJoinCondition(): InvalidJoinCondition; ``` -Defined in: [packages/db/src/errors.ts:581](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L581) +Defined in: [packages/db/src/errors.ts:628](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L628) #### Returns diff --git a/docs/reference/classes/InvalidJoinConditionLeftSourceError.md b/docs/reference/classes/InvalidJoinConditionLeftSourceError.md index 2c6c6257df..2d312af4cc 100644 --- a/docs/reference/classes/InvalidJoinConditionLeftSourceError.md +++ b/docs/reference/classes/InvalidJoinConditionLeftSourceError.md @@ -5,7 +5,7 @@ title: InvalidJoinConditionLeftSourceError # Class: InvalidJoinConditionLeftSourceError -Defined in: [packages/db/src/errors.ts:564](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L564) +Defined in: [packages/db/src/errors.ts:611](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L611) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:564](https://github.com/TanStack/db/blob/ new InvalidJoinConditionLeftSourceError(sourceAlias): InvalidJoinConditionLeftSourceError; ``` -Defined in: [packages/db/src/errors.ts:565](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L565) +Defined in: [packages/db/src/errors.ts:612](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L612) #### Parameters diff --git a/docs/reference/classes/InvalidJoinConditionRightSourceError.md b/docs/reference/classes/InvalidJoinConditionRightSourceError.md index 92f28a9565..a94ce893af 100644 --- a/docs/reference/classes/InvalidJoinConditionRightSourceError.md +++ b/docs/reference/classes/InvalidJoinConditionRightSourceError.md @@ -5,7 +5,7 @@ title: InvalidJoinConditionRightSourceError # Class: InvalidJoinConditionRightSourceError -Defined in: [packages/db/src/errors.ts:572](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L572) +Defined in: [packages/db/src/errors.ts:619](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L619) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:572](https://github.com/TanStack/db/blob/ new InvalidJoinConditionRightSourceError(sourceAlias): InvalidJoinConditionRightSourceError; ``` -Defined in: [packages/db/src/errors.ts:573](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L573) +Defined in: [packages/db/src/errors.ts:620](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L620) #### Parameters diff --git a/docs/reference/classes/InvalidJoinConditionSameSourceError.md b/docs/reference/classes/InvalidJoinConditionSameSourceError.md index 8b20d62b36..bdb8d6c6f7 100644 --- a/docs/reference/classes/InvalidJoinConditionSameSourceError.md +++ b/docs/reference/classes/InvalidJoinConditionSameSourceError.md @@ -5,7 +5,7 @@ title: InvalidJoinConditionSameSourceError # Class: InvalidJoinConditionSameSourceError -Defined in: [packages/db/src/errors.ts:550](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L550) +Defined in: [packages/db/src/errors.ts:597](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L597) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:550](https://github.com/TanStack/db/blob/ new InvalidJoinConditionSameSourceError(sourceAlias): InvalidJoinConditionSameSourceError; ``` -Defined in: [packages/db/src/errors.ts:551](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L551) +Defined in: [packages/db/src/errors.ts:598](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L598) #### Parameters diff --git a/docs/reference/classes/InvalidJoinConditionSourceMismatchError.md b/docs/reference/classes/InvalidJoinConditionSourceMismatchError.md index 3978012ae8..66c8cd87dd 100644 --- a/docs/reference/classes/InvalidJoinConditionSourceMismatchError.md +++ b/docs/reference/classes/InvalidJoinConditionSourceMismatchError.md @@ -5,7 +5,7 @@ title: InvalidJoinConditionSourceMismatchError # Class: InvalidJoinConditionSourceMismatchError -Defined in: [packages/db/src/errors.ts:558](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L558) +Defined in: [packages/db/src/errors.ts:605](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L605) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:558](https://github.com/TanStack/db/blob/ new InvalidJoinConditionSourceMismatchError(): InvalidJoinConditionSourceMismatchError; ``` -Defined in: [packages/db/src/errors.ts:559](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L559) +Defined in: [packages/db/src/errors.ts:606](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L606) #### Returns diff --git a/docs/reference/classes/InvalidKeyError.md b/docs/reference/classes/InvalidKeyError.md index b9f40eaccc..a0ada3323d 100644 --- a/docs/reference/classes/InvalidKeyError.md +++ b/docs/reference/classes/InvalidKeyError.md @@ -5,7 +5,7 @@ title: InvalidKeyError # Class: InvalidKeyError -Defined in: [packages/db/src/errors.ts:154](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L154) +Defined in: [packages/db/src/errors.ts:166](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L166) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:154](https://github.com/TanStack/db/blob/ new InvalidKeyError(key, item): InvalidKeyError; ``` -Defined in: [packages/db/src/errors.ts:155](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L155) +Defined in: [packages/db/src/errors.ts:167](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L167) #### Parameters diff --git a/docs/reference/classes/InvalidSourceError.md b/docs/reference/classes/InvalidSourceError.md index e0dc75a16d..ecb52702af 100644 --- a/docs/reference/classes/InvalidSourceError.md +++ b/docs/reference/classes/InvalidSourceError.md @@ -5,7 +5,7 @@ title: InvalidSourceError # Class: InvalidSourceError -Defined in: [packages/db/src/errors.ts:380](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L380) +Defined in: [packages/db/src/errors.ts:392](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L392) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:380](https://github.com/TanStack/db/blob/ new InvalidSourceError(alias): InvalidSourceError; ``` -Defined in: [packages/db/src/errors.ts:381](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L381) +Defined in: [packages/db/src/errors.ts:393](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L393) #### Parameters diff --git a/docs/reference/classes/InvalidSourceTypeError.md b/docs/reference/classes/InvalidSourceTypeError.md index 7128f7249f..67afe8db6b 100644 --- a/docs/reference/classes/InvalidSourceTypeError.md +++ b/docs/reference/classes/InvalidSourceTypeError.md @@ -5,7 +5,7 @@ title: InvalidSourceTypeError # Class: InvalidSourceTypeError -Defined in: [packages/db/src/errors.ts:388](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L388) +Defined in: [packages/db/src/errors.ts:405](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L405) ## Extends @@ -19,13 +19,13 @@ Defined in: [packages/db/src/errors.ts:388](https://github.com/TanStack/db/blob/ new InvalidSourceTypeError(context, type): InvalidSourceTypeError; ``` -Defined in: [packages/db/src/errors.ts:389](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L389) +Defined in: [packages/db/src/errors.ts:406](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L406) #### Parameters ##### context -`string` +[`SourceClauseContext`](../type-aliases/SourceClauseContext.md) ##### type diff --git a/docs/reference/classes/InvalidStorageDataFormatError.md b/docs/reference/classes/InvalidStorageDataFormatError.md index c7ec40856c..311464203a 100644 --- a/docs/reference/classes/InvalidStorageDataFormatError.md +++ b/docs/reference/classes/InvalidStorageDataFormatError.md @@ -5,7 +5,7 @@ title: InvalidStorageDataFormatError # Class: InvalidStorageDataFormatError -Defined in: [packages/db/src/errors.ts:658](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L658) +Defined in: [packages/db/src/errors.ts:705](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L705) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:658](https://github.com/TanStack/db/blob/ new InvalidStorageDataFormatError(storageKey, key): InvalidStorageDataFormatError; ``` -Defined in: [packages/db/src/errors.ts:659](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L659) +Defined in: [packages/db/src/errors.ts:706](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L706) #### Parameters diff --git a/docs/reference/classes/InvalidStorageObjectFormatError.md b/docs/reference/classes/InvalidStorageObjectFormatError.md index d2d3cec6e8..fe69512b61 100644 --- a/docs/reference/classes/InvalidStorageObjectFormatError.md +++ b/docs/reference/classes/InvalidStorageObjectFormatError.md @@ -5,7 +5,7 @@ title: InvalidStorageObjectFormatError # Class: InvalidStorageObjectFormatError -Defined in: [packages/db/src/errors.ts:666](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L666) +Defined in: [packages/db/src/errors.ts:713](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L713) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:666](https://github.com/TanStack/db/blob/ new InvalidStorageObjectFormatError(storageKey): InvalidStorageObjectFormatError; ``` -Defined in: [packages/db/src/errors.ts:667](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L667) +Defined in: [packages/db/src/errors.ts:714](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L714) #### Parameters diff --git a/docs/reference/classes/InvalidWhereExpressionError.md b/docs/reference/classes/InvalidWhereExpressionError.md index 26649e0ce7..8a6fe28987 100644 --- a/docs/reference/classes/InvalidWhereExpressionError.md +++ b/docs/reference/classes/InvalidWhereExpressionError.md @@ -5,7 +5,7 @@ title: InvalidWhereExpressionError # Class: InvalidWhereExpressionError -Defined in: [packages/db/src/errors.ts:409](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L409) +Defined in: [packages/db/src/errors.ts:436](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L436) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:409](https://github.com/TanStack/db/blob/ new InvalidWhereExpressionError(valueType): InvalidWhereExpressionError; ``` -Defined in: [packages/db/src/errors.ts:410](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L410) +Defined in: [packages/db/src/errors.ts:437](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L437) #### Parameters diff --git a/docs/reference/classes/JoinCollectionNotFoundError.md b/docs/reference/classes/JoinCollectionNotFoundError.md index 1268a7f58e..aa4a6e65a0 100644 --- a/docs/reference/classes/JoinCollectionNotFoundError.md +++ b/docs/reference/classes/JoinCollectionNotFoundError.md @@ -5,7 +5,7 @@ title: JoinCollectionNotFoundError # Class: JoinCollectionNotFoundError -Defined in: [packages/db/src/errors.ts:530](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L530) +Defined in: [packages/db/src/errors.ts:577](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L577) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:530](https://github.com/TanStack/db/blob/ new JoinCollectionNotFoundError(collectionId): JoinCollectionNotFoundError; ``` -Defined in: [packages/db/src/errors.ts:531](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L531) +Defined in: [packages/db/src/errors.ts:578](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L578) #### Parameters diff --git a/docs/reference/classes/JoinConditionMustBeEqualityError.md b/docs/reference/classes/JoinConditionMustBeEqualityError.md index fea19bb6b5..dc140de700 100644 --- a/docs/reference/classes/JoinConditionMustBeEqualityError.md +++ b/docs/reference/classes/JoinConditionMustBeEqualityError.md @@ -5,7 +5,7 @@ title: JoinConditionMustBeEqualityError # Class: JoinConditionMustBeEqualityError -Defined in: [packages/db/src/errors.ts:397](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L397) +Defined in: [packages/db/src/errors.ts:424](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L424) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:397](https://github.com/TanStack/db/blob/ new JoinConditionMustBeEqualityError(): JoinConditionMustBeEqualityError; ``` -Defined in: [packages/db/src/errors.ts:398](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L398) +Defined in: [packages/db/src/errors.ts:425](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L425) #### Returns diff --git a/docs/reference/classes/JoinError.md b/docs/reference/classes/JoinError.md index c1154a4667..10a48543fd 100644 --- a/docs/reference/classes/JoinError.md +++ b/docs/reference/classes/JoinError.md @@ -5,7 +5,7 @@ title: JoinError # Class: JoinError -Defined in: [packages/db/src/errors.ts:537](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L537) +Defined in: [packages/db/src/errors.ts:584](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L584) ## Extends @@ -29,7 +29,7 @@ Defined in: [packages/db/src/errors.ts:537](https://github.com/TanStack/db/blob/ new JoinError(message): JoinError; ``` -Defined in: [packages/db/src/errors.ts:538](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L538) +Defined in: [packages/db/src/errors.ts:585](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L585) #### Parameters diff --git a/docs/reference/classes/KeyUpdateNotAllowedError.md b/docs/reference/classes/KeyUpdateNotAllowedError.md index 315a85a7c8..19102a1fee 100644 --- a/docs/reference/classes/KeyUpdateNotAllowedError.md +++ b/docs/reference/classes/KeyUpdateNotAllowedError.md @@ -5,7 +5,7 @@ title: KeyUpdateNotAllowedError # Class: KeyUpdateNotAllowedError -Defined in: [packages/db/src/errors.ts:231](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L231) +Defined in: [packages/db/src/errors.ts:243](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L243) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:231](https://github.com/TanStack/db/blob/ new KeyUpdateNotAllowedError(originalKey, newKey): KeyUpdateNotAllowedError; ``` -Defined in: [packages/db/src/errors.ts:232](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L232) +Defined in: [packages/db/src/errors.ts:244](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L244) #### Parameters diff --git a/docs/reference/classes/LimitOffsetRequireOrderByError.md b/docs/reference/classes/LimitOffsetRequireOrderByError.md index 6a41edba30..a5af137a59 100644 --- a/docs/reference/classes/LimitOffsetRequireOrderByError.md +++ b/docs/reference/classes/LimitOffsetRequireOrderByError.md @@ -5,7 +5,7 @@ title: LimitOffsetRequireOrderByError # Class: LimitOffsetRequireOrderByError -Defined in: [packages/db/src/errors.ts:462](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L462) +Defined in: [packages/db/src/errors.ts:509](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L509) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:462](https://github.com/TanStack/db/blob/ new LimitOffsetRequireOrderByError(): LimitOffsetRequireOrderByError; ``` -Defined in: [packages/db/src/errors.ts:463](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L463) +Defined in: [packages/db/src/errors.ts:510](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L510) #### Returns diff --git a/docs/reference/classes/LiveQueryObserverDisposedError.md b/docs/reference/classes/LiveQueryObserverDisposedError.md new file mode 100644 index 0000000000..2c594783db --- /dev/null +++ b/docs/reference/classes/LiveQueryObserverDisposedError.md @@ -0,0 +1,214 @@ +--- +id: LiveQueryObserverDisposedError +title: LiveQueryObserverDisposedError +--- + +# Class: LiveQueryObserverDisposedError + +Defined in: [packages/db/src/errors.ts:138](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L138) + +## Extends + +- [`CollectionStateError`](CollectionStateError.md) + +## Constructors + +### Constructor + +```ts +new LiveQueryObserverDisposedError(): LiveQueryObserverDisposedError; +``` + +Defined in: [packages/db/src/errors.ts:139](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L139) + +#### Returns + +`LiveQueryObserverDisposedError` + +#### Overrides + +[`CollectionStateError`](CollectionStateError.md).[`constructor`](CollectionStateError.md#constructor) + +## Properties + +### cause? + +```ts +optional cause: unknown; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es2022.error.d.ts:26 + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`cause`](CollectionStateError.md#cause) + +*** + +### message + +```ts +message: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1077 + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`message`](CollectionStateError.md#message) + +*** + +### name + +```ts +name: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1076 + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`name`](CollectionStateError.md#name) + +*** + +### stack? + +```ts +optional stack: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1078 + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`stack`](CollectionStateError.md#stack) + +*** + +### stackTraceLimit + +```ts +static stackTraceLimit: number; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:67 + +The `Error.stackTraceLimit` property specifies the number of stack frames +collected by a stack trace (whether generated by `new Error().stack` or +`Error.captureStackTrace(obj)`). + +The default value is `10` but may be set to any valid JavaScript number. Changes +will affect any stack trace captured _after_ the value has been changed. + +If set to a non-number value, or set to a negative number, stack traces will +not capture any frames. + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`stackTraceLimit`](CollectionStateError.md#stacktracelimit) + +## Methods + +### captureStackTrace() + +```ts +static captureStackTrace(targetObject, constructorOpt?): void; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:51 + +Creates a `.stack` property on `targetObject`, which when accessed returns +a string representing the location in the code at which +`Error.captureStackTrace()` was called. + +```js +const myObject = {}; +Error.captureStackTrace(myObject); +myObject.stack; // Similar to `new Error().stack` +``` + +The first line of the trace will be prefixed with +`${myObject.name}: ${myObject.message}`. + +The optional `constructorOpt` argument accepts a function. If given, all frames +above `constructorOpt`, including `constructorOpt`, will be omitted from the +generated stack trace. + +The `constructorOpt` argument is useful for hiding implementation +details of error generation from the user. For instance: + +```js +function a() { + b(); +} + +function b() { + c(); +} + +function c() { + // Create an error without stack trace to avoid calculating the stack trace twice. + const { stackTraceLimit } = Error; + Error.stackTraceLimit = 0; + const error = new Error(); + Error.stackTraceLimit = stackTraceLimit; + + // Capture the stack trace above function b + Error.captureStackTrace(error, b); // Neither function c, nor b is included in the stack trace + throw error; +} + +a(); +``` + +#### Parameters + +##### targetObject + +`object` + +##### constructorOpt? + +`Function` + +#### Returns + +`void` + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`captureStackTrace`](CollectionStateError.md#capturestacktrace) + +*** + +### prepareStackTrace() + +```ts +static prepareStackTrace(err, stackTraces): any; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:55 + +#### Parameters + +##### err + +`Error` + +##### stackTraces + +`CallSite`[] + +#### Returns + +`any` + +#### See + +https://v8.dev/docs/stack-trace-api#customizing-stack-traces + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`prepareStackTrace`](CollectionStateError.md#preparestacktrace) diff --git a/docs/reference/classes/LiveQueryWindowControllerDisposedError.md b/docs/reference/classes/LiveQueryWindowControllerDisposedError.md new file mode 100644 index 0000000000..44dbfc096e --- /dev/null +++ b/docs/reference/classes/LiveQueryWindowControllerDisposedError.md @@ -0,0 +1,214 @@ +--- +id: LiveQueryWindowControllerDisposedError +title: LiveQueryWindowControllerDisposedError +--- + +# Class: LiveQueryWindowControllerDisposedError + +Defined in: [packages/db/src/errors.ts:144](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L144) + +## Extends + +- [`CollectionStateError`](CollectionStateError.md) + +## Constructors + +### Constructor + +```ts +new LiveQueryWindowControllerDisposedError(): LiveQueryWindowControllerDisposedError; +``` + +Defined in: [packages/db/src/errors.ts:145](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L145) + +#### Returns + +`LiveQueryWindowControllerDisposedError` + +#### Overrides + +[`CollectionStateError`](CollectionStateError.md).[`constructor`](CollectionStateError.md#constructor) + +## Properties + +### cause? + +```ts +optional cause: unknown; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es2022.error.d.ts:26 + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`cause`](CollectionStateError.md#cause) + +*** + +### message + +```ts +message: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1077 + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`message`](CollectionStateError.md#message) + +*** + +### name + +```ts +name: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1076 + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`name`](CollectionStateError.md#name) + +*** + +### stack? + +```ts +optional stack: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1078 + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`stack`](CollectionStateError.md#stack) + +*** + +### stackTraceLimit + +```ts +static stackTraceLimit: number; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:67 + +The `Error.stackTraceLimit` property specifies the number of stack frames +collected by a stack trace (whether generated by `new Error().stack` or +`Error.captureStackTrace(obj)`). + +The default value is `10` but may be set to any valid JavaScript number. Changes +will affect any stack trace captured _after_ the value has been changed. + +If set to a non-number value, or set to a negative number, stack traces will +not capture any frames. + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`stackTraceLimit`](CollectionStateError.md#stacktracelimit) + +## Methods + +### captureStackTrace() + +```ts +static captureStackTrace(targetObject, constructorOpt?): void; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:51 + +Creates a `.stack` property on `targetObject`, which when accessed returns +a string representing the location in the code at which +`Error.captureStackTrace()` was called. + +```js +const myObject = {}; +Error.captureStackTrace(myObject); +myObject.stack; // Similar to `new Error().stack` +``` + +The first line of the trace will be prefixed with +`${myObject.name}: ${myObject.message}`. + +The optional `constructorOpt` argument accepts a function. If given, all frames +above `constructorOpt`, including `constructorOpt`, will be omitted from the +generated stack trace. + +The `constructorOpt` argument is useful for hiding implementation +details of error generation from the user. For instance: + +```js +function a() { + b(); +} + +function b() { + c(); +} + +function c() { + // Create an error without stack trace to avoid calculating the stack trace twice. + const { stackTraceLimit } = Error; + Error.stackTraceLimit = 0; + const error = new Error(); + Error.stackTraceLimit = stackTraceLimit; + + // Capture the stack trace above function b + Error.captureStackTrace(error, b); // Neither function c, nor b is included in the stack trace + throw error; +} + +a(); +``` + +#### Parameters + +##### targetObject + +`object` + +##### constructorOpt? + +`Function` + +#### Returns + +`void` + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`captureStackTrace`](CollectionStateError.md#capturestacktrace) + +*** + +### prepareStackTrace() + +```ts +static prepareStackTrace(err, stackTraces): any; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:55 + +#### Parameters + +##### err + +`Error` + +##### stackTraces + +`CallSite`[] + +#### Returns + +`any` + +#### See + +https://v8.dev/docs/stack-trace-api#customizing-stack-traces + +#### Inherited from + +[`CollectionStateError`](CollectionStateError.md).[`prepareStackTrace`](CollectionStateError.md#preparestacktrace) diff --git a/docs/reference/classes/LoadSubsetOperationAbortedError.md b/docs/reference/classes/LoadSubsetOperationAbortedError.md new file mode 100644 index 0000000000..215598b5cb --- /dev/null +++ b/docs/reference/classes/LoadSubsetOperationAbortedError.md @@ -0,0 +1,232 @@ +--- +id: LoadSubsetOperationAbortedError +title: LoadSubsetOperationAbortedError +--- + +# Class: LoadSubsetOperationAbortedError + +Defined in: [packages/db/src/errors.ts:749](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L749) + +A subset operation was canceled before its result became visible. + +## Extends + +- `Error` + +## Constructors + +### Constructor + +```ts +new LoadSubsetOperationAbortedError(): LoadSubsetOperationAbortedError; +``` + +Defined in: [packages/db/src/errors.ts:750](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L750) + +#### Returns + +`LoadSubsetOperationAbortedError` + +#### Overrides + +```ts +Error.constructor +``` + +## Properties + +### cause? + +```ts +optional cause: unknown; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es2022.error.d.ts:26 + +#### Inherited from + +```ts +Error.cause +``` + +*** + +### message + +```ts +message: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1077 + +#### Inherited from + +```ts +Error.message +``` + +*** + +### name + +```ts +name: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1076 + +#### Inherited from + +```ts +Error.name +``` + +*** + +### stack? + +```ts +optional stack: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1078 + +#### Inherited from + +```ts +Error.stack +``` + +*** + +### stackTraceLimit + +```ts +static stackTraceLimit: number; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:67 + +The `Error.stackTraceLimit` property specifies the number of stack frames +collected by a stack trace (whether generated by `new Error().stack` or +`Error.captureStackTrace(obj)`). + +The default value is `10` but may be set to any valid JavaScript number. Changes +will affect any stack trace captured _after_ the value has been changed. + +If set to a non-number value, or set to a negative number, stack traces will +not capture any frames. + +#### Inherited from + +```ts +Error.stackTraceLimit +``` + +## Methods + +### captureStackTrace() + +```ts +static captureStackTrace(targetObject, constructorOpt?): void; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:51 + +Creates a `.stack` property on `targetObject`, which when accessed returns +a string representing the location in the code at which +`Error.captureStackTrace()` was called. + +```js +const myObject = {}; +Error.captureStackTrace(myObject); +myObject.stack; // Similar to `new Error().stack` +``` + +The first line of the trace will be prefixed with +`${myObject.name}: ${myObject.message}`. + +The optional `constructorOpt` argument accepts a function. If given, all frames +above `constructorOpt`, including `constructorOpt`, will be omitted from the +generated stack trace. + +The `constructorOpt` argument is useful for hiding implementation +details of error generation from the user. For instance: + +```js +function a() { + b(); +} + +function b() { + c(); +} + +function c() { + // Create an error without stack trace to avoid calculating the stack trace twice. + const { stackTraceLimit } = Error; + Error.stackTraceLimit = 0; + const error = new Error(); + Error.stackTraceLimit = stackTraceLimit; + + // Capture the stack trace above function b + Error.captureStackTrace(error, b); // Neither function c, nor b is included in the stack trace + throw error; +} + +a(); +``` + +#### Parameters + +##### targetObject + +`object` + +##### constructorOpt? + +`Function` + +#### Returns + +`void` + +#### Inherited from + +```ts +Error.captureStackTrace +``` + +*** + +### prepareStackTrace() + +```ts +static prepareStackTrace(err, stackTraces): any; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:55 + +#### Parameters + +##### err + +`Error` + +##### stackTraces + +`CallSite`[] + +#### Returns + +`any` + +#### See + +https://v8.dev/docs/stack-trace-api#customizing-stack-traces + +#### Inherited from + +```ts +Error.prepareStackTrace +``` diff --git a/docs/reference/classes/LocalStorageCollectionError.md b/docs/reference/classes/LocalStorageCollectionError.md index 78725fca5a..80d67748c8 100644 --- a/docs/reference/classes/LocalStorageCollectionError.md +++ b/docs/reference/classes/LocalStorageCollectionError.md @@ -5,7 +5,7 @@ title: LocalStorageCollectionError # Class: LocalStorageCollectionError -Defined in: [packages/db/src/errors.ts:645](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L645) +Defined in: [packages/db/src/errors.ts:692](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L692) ## Extends @@ -25,7 +25,7 @@ Defined in: [packages/db/src/errors.ts:645](https://github.com/TanStack/db/blob/ new LocalStorageCollectionError(message): LocalStorageCollectionError; ``` -Defined in: [packages/db/src/errors.ts:646](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L646) +Defined in: [packages/db/src/errors.ts:693](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L693) #### Parameters diff --git a/docs/reference/classes/MissingAliasInputsError.md b/docs/reference/classes/MissingAliasInputsError.md index 8e13aedf05..e689d702b3 100644 --- a/docs/reference/classes/MissingAliasInputsError.md +++ b/docs/reference/classes/MissingAliasInputsError.md @@ -5,7 +5,7 @@ title: MissingAliasInputsError # Class: MissingAliasInputsError -Defined in: [packages/db/src/errors.ts:742](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L742) +Defined in: [packages/db/src/errors.ts:774](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L774) Internal error when the compiler returns aliases that don't have corresponding input streams. This should never happen since all aliases come from user declarations. @@ -22,7 +22,7 @@ This should never happen since all aliases come from user declarations. new MissingAliasInputsError(missingAliases): MissingAliasInputsError; ``` -Defined in: [packages/db/src/errors.ts:743](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L743) +Defined in: [packages/db/src/errors.ts:775](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L775) #### Parameters diff --git a/docs/reference/classes/MissingDeleteHandlerError.md b/docs/reference/classes/MissingDeleteHandlerError.md index 5707e277fa..3687ffcb66 100644 --- a/docs/reference/classes/MissingDeleteHandlerError.md +++ b/docs/reference/classes/MissingDeleteHandlerError.md @@ -5,7 +5,7 @@ title: MissingDeleteHandlerError # Class: MissingDeleteHandlerError -Defined in: [packages/db/src/errors.ts:277](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L277) +Defined in: [packages/db/src/errors.ts:289](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L289) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:277](https://github.com/TanStack/db/blob/ new MissingDeleteHandlerError(): MissingDeleteHandlerError; ``` -Defined in: [packages/db/src/errors.ts:278](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L278) +Defined in: [packages/db/src/errors.ts:290](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L290) #### Returns diff --git a/docs/reference/classes/MissingHandlerError.md b/docs/reference/classes/MissingHandlerError.md index d7959c0eeb..1e93d31d53 100644 --- a/docs/reference/classes/MissingHandlerError.md +++ b/docs/reference/classes/MissingHandlerError.md @@ -5,7 +5,7 @@ title: MissingHandlerError # Class: MissingHandlerError -Defined in: [packages/db/src/errors.ts:254](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L254) +Defined in: [packages/db/src/errors.ts:266](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L266) ## Extends @@ -25,7 +25,7 @@ Defined in: [packages/db/src/errors.ts:254](https://github.com/TanStack/db/blob/ new MissingHandlerError(message): MissingHandlerError; ``` -Defined in: [packages/db/src/errors.ts:255](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L255) +Defined in: [packages/db/src/errors.ts:267](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L267) #### Parameters diff --git a/docs/reference/classes/MissingInsertHandlerError.md b/docs/reference/classes/MissingInsertHandlerError.md index b4fd9bf849..cc00696599 100644 --- a/docs/reference/classes/MissingInsertHandlerError.md +++ b/docs/reference/classes/MissingInsertHandlerError.md @@ -5,7 +5,7 @@ title: MissingInsertHandlerError # Class: MissingInsertHandlerError -Defined in: [packages/db/src/errors.ts:261](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L261) +Defined in: [packages/db/src/errors.ts:273](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L273) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:261](https://github.com/TanStack/db/blob/ new MissingInsertHandlerError(): MissingInsertHandlerError; ``` -Defined in: [packages/db/src/errors.ts:262](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L262) +Defined in: [packages/db/src/errors.ts:274](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L274) #### Returns diff --git a/docs/reference/classes/MissingMutationFunctionError.md b/docs/reference/classes/MissingMutationFunctionError.md index fff948a872..1c8d49a694 100644 --- a/docs/reference/classes/MissingMutationFunctionError.md +++ b/docs/reference/classes/MissingMutationFunctionError.md @@ -5,7 +5,7 @@ title: MissingMutationFunctionError # Class: MissingMutationFunctionError -Defined in: [packages/db/src/errors.ts:293](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L293) +Defined in: [packages/db/src/errors.ts:305](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L305) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:293](https://github.com/TanStack/db/blob/ new MissingMutationFunctionError(): MissingMutationFunctionError; ``` -Defined in: [packages/db/src/errors.ts:294](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L294) +Defined in: [packages/db/src/errors.ts:306](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L306) #### Returns diff --git a/docs/reference/classes/MissingUpdateArgumentError.md b/docs/reference/classes/MissingUpdateArgumentError.md index 4cc2d21830..67b408e364 100644 --- a/docs/reference/classes/MissingUpdateArgumentError.md +++ b/docs/reference/classes/MissingUpdateArgumentError.md @@ -5,7 +5,7 @@ title: MissingUpdateArgumentError # Class: MissingUpdateArgumentError -Defined in: [packages/db/src/errors.ts:211](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L211) +Defined in: [packages/db/src/errors.ts:223](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L223) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:211](https://github.com/TanStack/db/blob/ new MissingUpdateArgumentError(): MissingUpdateArgumentError; ``` -Defined in: [packages/db/src/errors.ts:212](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L212) +Defined in: [packages/db/src/errors.ts:224](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L224) #### Returns diff --git a/docs/reference/classes/MissingUpdateHandlerError.md b/docs/reference/classes/MissingUpdateHandlerError.md index 81ae37bdc4..37eb7d3a13 100644 --- a/docs/reference/classes/MissingUpdateHandlerError.md +++ b/docs/reference/classes/MissingUpdateHandlerError.md @@ -5,7 +5,7 @@ title: MissingUpdateHandlerError # Class: MissingUpdateHandlerError -Defined in: [packages/db/src/errors.ts:269](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L269) +Defined in: [packages/db/src/errors.ts:281](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L281) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:269](https://github.com/TanStack/db/blob/ new MissingUpdateHandlerError(): MissingUpdateHandlerError; ``` -Defined in: [packages/db/src/errors.ts:270](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L270) +Defined in: [packages/db/src/errors.ts:282](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L282) #### Returns diff --git a/docs/reference/classes/NoKeysPassedToDeleteError.md b/docs/reference/classes/NoKeysPassedToDeleteError.md index fd5ea18f1a..c67e9c01ba 100644 --- a/docs/reference/classes/NoKeysPassedToDeleteError.md +++ b/docs/reference/classes/NoKeysPassedToDeleteError.md @@ -5,7 +5,7 @@ title: NoKeysPassedToDeleteError # Class: NoKeysPassedToDeleteError -Defined in: [packages/db/src/errors.ts:239](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L239) +Defined in: [packages/db/src/errors.ts:251](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L251) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:239](https://github.com/TanStack/db/blob/ new NoKeysPassedToDeleteError(): NoKeysPassedToDeleteError; ``` -Defined in: [packages/db/src/errors.ts:240](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L240) +Defined in: [packages/db/src/errors.ts:252](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L252) #### Returns diff --git a/docs/reference/classes/NoKeysPassedToUpdateError.md b/docs/reference/classes/NoKeysPassedToUpdateError.md index 0803fd240a..0209ce7992 100644 --- a/docs/reference/classes/NoKeysPassedToUpdateError.md +++ b/docs/reference/classes/NoKeysPassedToUpdateError.md @@ -5,7 +5,7 @@ title: NoKeysPassedToUpdateError # Class: NoKeysPassedToUpdateError -Defined in: [packages/db/src/errors.ts:217](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L217) +Defined in: [packages/db/src/errors.ts:229](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L229) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:217](https://github.com/TanStack/db/blob/ new NoKeysPassedToUpdateError(): NoKeysPassedToUpdateError; ``` -Defined in: [packages/db/src/errors.ts:218](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L218) +Defined in: [packages/db/src/errors.ts:230](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L230) #### Returns diff --git a/docs/reference/classes/NoPendingSyncTransactionCommitError.md b/docs/reference/classes/NoPendingSyncTransactionCommitError.md index 61b1f21af2..9e5a78f8f5 100644 --- a/docs/reference/classes/NoPendingSyncTransactionCommitError.md +++ b/docs/reference/classes/NoPendingSyncTransactionCommitError.md @@ -5,7 +5,7 @@ title: NoPendingSyncTransactionCommitError # Class: NoPendingSyncTransactionCommitError -Defined in: [packages/db/src/errors.ts:346](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L346) +Defined in: [packages/db/src/errors.ts:358](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L358) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:346](https://github.com/TanStack/db/blob/ new NoPendingSyncTransactionCommitError(): NoPendingSyncTransactionCommitError; ``` -Defined in: [packages/db/src/errors.ts:347](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L347) +Defined in: [packages/db/src/errors.ts:359](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L359) #### Returns diff --git a/docs/reference/classes/NoPendingSyncTransactionWriteError.md b/docs/reference/classes/NoPendingSyncTransactionWriteError.md index dbb78c9828..5573ac8db3 100644 --- a/docs/reference/classes/NoPendingSyncTransactionWriteError.md +++ b/docs/reference/classes/NoPendingSyncTransactionWriteError.md @@ -5,7 +5,7 @@ title: NoPendingSyncTransactionWriteError # Class: NoPendingSyncTransactionWriteError -Defined in: [packages/db/src/errors.ts:332](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L332) +Defined in: [packages/db/src/errors.ts:344](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L344) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:332](https://github.com/TanStack/db/blob/ new NoPendingSyncTransactionWriteError(): NoPendingSyncTransactionWriteError; ``` -Defined in: [packages/db/src/errors.ts:333](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L333) +Defined in: [packages/db/src/errors.ts:345](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L345) #### Returns diff --git a/docs/reference/classes/NonAggregateExpressionNotInGroupByError.md b/docs/reference/classes/NonAggregateExpressionNotInGroupByError.md index b34fd77838..973bb552a4 100644 --- a/docs/reference/classes/NonAggregateExpressionNotInGroupByError.md +++ b/docs/reference/classes/NonAggregateExpressionNotInGroupByError.md @@ -5,7 +5,7 @@ title: NonAggregateExpressionNotInGroupByError # Class: NonAggregateExpressionNotInGroupByError -Defined in: [packages/db/src/errors.ts:600](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L600) +Defined in: [packages/db/src/errors.ts:647](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L647) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:600](https://github.com/TanStack/db/blob/ new NonAggregateExpressionNotInGroupByError(alias): NonAggregateExpressionNotInGroupByError; ``` -Defined in: [packages/db/src/errors.ts:601](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L601) +Defined in: [packages/db/src/errors.ts:648](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L648) #### Parameters diff --git a/docs/reference/classes/OnMutateMustBeSynchronousError.md b/docs/reference/classes/OnMutateMustBeSynchronousError.md index da84ef8cf9..b8695376e0 100644 --- a/docs/reference/classes/OnMutateMustBeSynchronousError.md +++ b/docs/reference/classes/OnMutateMustBeSynchronousError.md @@ -5,7 +5,7 @@ title: OnMutateMustBeSynchronousError # Class: OnMutateMustBeSynchronousError -Defined in: [packages/db/src/errors.ts:299](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L299) +Defined in: [packages/db/src/errors.ts:311](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L311) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:299](https://github.com/TanStack/db/blob/ new OnMutateMustBeSynchronousError(): OnMutateMustBeSynchronousError; ``` -Defined in: [packages/db/src/errors.ts:300](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L300) +Defined in: [packages/db/src/errors.ts:312](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L312) #### Returns diff --git a/docs/reference/classes/OnlyOneSourceAllowedError.md b/docs/reference/classes/OnlyOneSourceAllowedError.md index c2567bbcab..81d1c7b186 100644 --- a/docs/reference/classes/OnlyOneSourceAllowedError.md +++ b/docs/reference/classes/OnlyOneSourceAllowedError.md @@ -5,7 +5,7 @@ title: OnlyOneSourceAllowedError # Class: OnlyOneSourceAllowedError -Defined in: [packages/db/src/errors.ts:368](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L368) +Defined in: [packages/db/src/errors.ts:380](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L380) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:368](https://github.com/TanStack/db/blob/ new OnlyOneSourceAllowedError(context): OnlyOneSourceAllowedError; ``` -Defined in: [packages/db/src/errors.ts:369](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L369) +Defined in: [packages/db/src/errors.ts:381](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L381) #### Parameters diff --git a/docs/reference/classes/QueryBuilderError.md b/docs/reference/classes/QueryBuilderError.md index beef2bb093..02e10ee01d 100644 --- a/docs/reference/classes/QueryBuilderError.md +++ b/docs/reference/classes/QueryBuilderError.md @@ -5,7 +5,7 @@ title: QueryBuilderError # Class: QueryBuilderError -Defined in: [packages/db/src/errors.ts:361](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L361) +Defined in: [packages/db/src/errors.ts:373](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L373) ## Extends @@ -29,7 +29,7 @@ Defined in: [packages/db/src/errors.ts:361](https://github.com/TanStack/db/blob/ new QueryBuilderError(message): QueryBuilderError; ``` -Defined in: [packages/db/src/errors.ts:362](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L362) +Defined in: [packages/db/src/errors.ts:374](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L374) #### Parameters diff --git a/docs/reference/classes/QueryCompilationError.md b/docs/reference/classes/QueryCompilationError.md index abd2054dea..2a5a9987de 100644 --- a/docs/reference/classes/QueryCompilationError.md +++ b/docs/reference/classes/QueryCompilationError.md @@ -5,7 +5,7 @@ title: QueryCompilationError # Class: QueryCompilationError -Defined in: [packages/db/src/errors.ts:423](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L423) +Defined in: [packages/db/src/errors.ts:450](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L450) ## Extends @@ -13,8 +13,10 @@ Defined in: [packages/db/src/errors.ts:423](https://github.com/TanStack/db/blob/ ## Extended by +- [`UnsafeAliasPathError`](UnsafeAliasPathError.md) - [`DistinctRequiresSelectError`](DistinctRequiresSelectError.md) - [`FnSelectWithGroupByError`](FnSelectWithGroupByError.md) +- [`UnsupportedFnSelectResultError`](UnsupportedFnSelectResultError.md) - [`UnsupportedRootScalarSelectError`](UnsupportedRootScalarSelectError.md) - [`HavingRequiresGroupByError`](HavingRequiresGroupByError.md) - [`LimitOffsetRequireOrderByError`](LimitOffsetRequireOrderByError.md) @@ -25,8 +27,6 @@ Defined in: [packages/db/src/errors.ts:423](https://github.com/TanStack/db/blob/ - [`EmptyReferencePathError`](EmptyReferencePathError.md) - [`UnknownFunctionError`](UnknownFunctionError.md) - [`JoinCollectionNotFoundError`](JoinCollectionNotFoundError.md) -- [`SubscriptionNotFoundError`](SubscriptionNotFoundError.md) -- [`AggregateNotSupportedError`](AggregateNotSupportedError.md) - [`MissingAliasInputsError`](MissingAliasInputsError.md) - [`SetWindowRequiresOrderByError`](SetWindowRequiresOrderByError.md) @@ -38,7 +38,7 @@ Defined in: [packages/db/src/errors.ts:423](https://github.com/TanStack/db/blob/ new QueryCompilationError(message): QueryCompilationError; ``` -Defined in: [packages/db/src/errors.ts:424](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L424) +Defined in: [packages/db/src/errors.ts:451](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L451) #### Parameters diff --git a/docs/reference/classes/QueryMustHaveFromClauseError.md b/docs/reference/classes/QueryMustHaveFromClauseError.md index de0baea892..7821a1c566 100644 --- a/docs/reference/classes/QueryMustHaveFromClauseError.md +++ b/docs/reference/classes/QueryMustHaveFromClauseError.md @@ -5,7 +5,7 @@ title: QueryMustHaveFromClauseError # Class: QueryMustHaveFromClauseError -Defined in: [packages/db/src/errors.ts:403](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L403) +Defined in: [packages/db/src/errors.ts:430](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L430) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:403](https://github.com/TanStack/db/blob/ new QueryMustHaveFromClauseError(): QueryMustHaveFromClauseError; ``` -Defined in: [packages/db/src/errors.ts:404](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L404) +Defined in: [packages/db/src/errors.ts:431](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L431) #### Returns diff --git a/docs/reference/classes/QueryOptimizerError.md b/docs/reference/classes/QueryOptimizerError.md index f519315f23..5adc09b1ff 100644 --- a/docs/reference/classes/QueryOptimizerError.md +++ b/docs/reference/classes/QueryOptimizerError.md @@ -5,7 +5,7 @@ title: QueryOptimizerError # Class: QueryOptimizerError -Defined in: [packages/db/src/errors.ts:686](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L686) +Defined in: [packages/db/src/errors.ts:757](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L757) ## Extends @@ -14,7 +14,6 @@ Defined in: [packages/db/src/errors.ts:686](https://github.com/TanStack/db/blob/ ## Extended by - [`CannotCombineEmptyExpressionListError`](CannotCombineEmptyExpressionListError.md) -- [`WhereClauseConversionError`](WhereClauseConversionError.md) ## Constructors @@ -24,7 +23,7 @@ Defined in: [packages/db/src/errors.ts:686](https://github.com/TanStack/db/blob/ new QueryOptimizerError(message): QueryOptimizerError; ``` -Defined in: [packages/db/src/errors.ts:687](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L687) +Defined in: [packages/db/src/errors.ts:758](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L758) #### Parameters diff --git a/docs/reference/classes/ReverseIndex.md b/docs/reference/classes/ReverseIndex.md index cf4860b028..acaec1b570 100644 --- a/docs/reference/classes/ReverseIndex.md +++ b/docs/reference/classes/ReverseIndex.md @@ -5,7 +5,7 @@ title: ReverseIndex # Class: ReverseIndex\ -Defined in: [packages/db/src/indexes/reverse-index.ts:6](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L6) +Defined in: [packages/db/src/indexes/reverse-index.ts:4](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L4) ## Type Parameters @@ -15,7 +15,7 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:6](https://github.com/TanS ## Implements -- [`IndexInterface`](../interfaces/IndexInterface.md)\<`TKey`\> +- [`IndexReader`](../type-aliases/IndexReader.md)\<`TKey`\> ## Constructors @@ -25,7 +25,7 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:6](https://github.com/TanS new ReverseIndex(index): ReverseIndex; ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:11](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L11) +Defined in: [packages/db/src/indexes/reverse-index.ts:9](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L9) #### Parameters @@ -39,26 +39,6 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:11](https://github.com/Tan ## Accessors -### indexedKeysSet - -#### Get Signature - -```ts -get indexedKeysSet(): Set; -``` - -Defined in: [packages/db/src/indexes/reverse-index.ts:124](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L124) - -##### Returns - -`Set`\<`TKey`\> - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`indexedKeysSet`](../interfaces/IndexInterface.md#indexedkeysset) - -*** - ### keyCount #### Get Signature @@ -67,7 +47,7 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:124](https://github.com/Ta get keyCount(): number; ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:112](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L112) +Defined in: [packages/db/src/indexes/reverse-index.ts:55](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L55) ##### Returns @@ -75,204 +55,68 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:112](https://github.com/Ta #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`keyCount`](../interfaces/IndexInterface.md#keycount) - -*** - -### orderedEntriesArray - -#### Get Signature - ```ts -get orderedEntriesArray(): [any, Set][]; +IndexReader.keyCount ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:62](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L62) - -##### Returns - -\[`any`, `Set`\<`TKey`\>\][] - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`orderedEntriesArray`](../interfaces/IndexInterface.md#orderedentriesarray) - *** -### orderedEntriesArrayReversed +### supportsRangeOptimization #### Get Signature ```ts -get orderedEntriesArrayReversed(): [any, Set][]; +get supportsRangeOptimization(): boolean; ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:66](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L66) - -##### Returns - -\[`any`, `Set`\<`TKey`\>\][] - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`orderedEntriesArrayReversed`](../interfaces/IndexInterface.md#orderedentriesarrayreversed) - -*** - -### valueMapData - -#### Get Signature - -```ts -get valueMapData(): Map>; -``` +Defined in: [packages/db/src/indexes/reverse-index.ts:47](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L47) -Defined in: [packages/db/src/indexes/reverse-index.ts:128](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L128) +Whether range lookups (gt/gte/lt/lte) on this index can be trusted to +return every matching key. Range traversal relies on the index ordering, so +it is unsafe when the index uses a custom comparator, whose order may not +match the WHERE evaluator's relational operators. Callers must fall back to +a full scan when this is `false`. ##### Returns -`Map`\<`any`, `Set`\<`TKey`\>\> - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`valueMapData`](../interfaces/IndexInterface.md#valuemapdata) - -## Methods - -### add() - -```ts -add(key, item): void; -``` - -Defined in: [packages/db/src/indexes/reverse-index.ts:92](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L92) - -#### Parameters - -##### key - -`TKey` - -##### item - -`any` - -#### Returns - -`void` +`boolean` #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`add`](../interfaces/IndexInterface.md#add) - -*** - -### build() - ```ts -build(entries): void; +IndexReader.supportsRangeOptimization ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:104](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L104) - -#### Parameters - -##### entries - -`Iterable`\<\[`TKey`, `any`\]\> - -#### Returns - -`void` - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`build`](../interfaces/IndexInterface.md#build) - -*** +## Methods -### clear() +### canOptimizeRangeFor() ```ts -clear(): void; +canOptimizeRangeFor(value): boolean; ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:108](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L108) - -#### Returns - -`void` - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`clear`](../interfaces/IndexInterface.md#clear) - -*** - -### equalityLookup() - -```ts -equalityLookup(value): Set; -``` +Defined in: [packages/db/src/indexes/reverse-index.ts:51](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L51) -Defined in: [packages/db/src/indexes/reverse-index.ts:116](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L116) +Whether the live values in this index share the predicate operand's +relational domain. Mixed domains can sort differently in the index and +WHERE evaluator, which can make a range lookup omit matching rows. #### Parameters ##### value -`any` +`unknown` #### Returns -`Set`\<`TKey`\> +`boolean` #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`equalityLookup`](../interfaces/IndexInterface.md#equalitylookup) - -*** - -### getStats() - ```ts -getStats(): IndexStats; +IndexReader.canOptimizeRangeFor ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:88](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L88) - -#### Returns - -[`IndexStats`](../interfaces/IndexStats.md) - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`getStats`](../interfaces/IndexInterface.md#getstats) - -*** - -### inArrayLookup() - -```ts -inArrayLookup(values): Set; -``` - -Defined in: [packages/db/src/indexes/reverse-index.ts:120](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L120) - -#### Parameters - -##### values - -`any`[] - -#### Returns - -`Set`\<`TKey`\> - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`inArrayLookup`](../interfaces/IndexInterface.md#inarraylookup) - *** ### lookup() @@ -281,7 +125,7 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:120](https://github.com/Ta lookup(operation, value): Set; ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:17](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L17) +Defined in: [packages/db/src/indexes/reverse-index.ts:15](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L15) #### Parameters @@ -299,80 +143,10 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:17](https://github.com/Tan #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`lookup`](../interfaces/IndexInterface.md#lookup) - -*** - -### matchesCompareOptions() - ```ts -matchesCompareOptions(compareOptions): boolean; +IndexReader.lookup ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:80](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L80) - -#### Parameters - -##### compareOptions - -`CompareOptions` - -#### Returns - -`boolean` - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`matchesCompareOptions`](../interfaces/IndexInterface.md#matchescompareoptions) - -*** - -### matchesDirection() - -```ts -matchesDirection(direction): boolean; -``` - -Defined in: [packages/db/src/indexes/reverse-index.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L84) - -#### Parameters - -##### direction - -[`OrderByDirection`](../@tanstack/namespaces/IR/type-aliases/OrderByDirection.md) - -#### Returns - -`boolean` - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`matchesDirection`](../interfaces/IndexInterface.md#matchesdirection) - -*** - -### matchesField() - -```ts -matchesField(fieldPath): boolean; -``` - -Defined in: [packages/db/src/indexes/reverse-index.ts:76](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L76) - -#### Parameters - -##### fieldPath - -`string`[] - -#### Returns - -`boolean` - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`matchesField`](../interfaces/IndexInterface.md#matchesfield) - *** ### rangeQuery() @@ -381,7 +155,7 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:76](https://github.com/Tan rangeQuery(options): Set; ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:31](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L31) +Defined in: [packages/db/src/indexes/reverse-index.ts:29](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L29) #### Parameters @@ -395,60 +169,10 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:31](https://github.com/Tan #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`rangeQuery`](../interfaces/IndexInterface.md#rangequery) - -*** - -### rangeQueryReversed() - ```ts -rangeQueryReversed(options): Set; +IndexReader.rangeQuery ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:35](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L35) - -#### Parameters - -##### options - -[`BTreeRangeQueryOptions`](../interfaces/BTreeRangeQueryOptions.md) = `{}` - -#### Returns - -`Set`\<`TKey`\> - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`rangeQueryReversed`](../interfaces/IndexInterface.md#rangequeryreversed) - -*** - -### remove() - -```ts -remove(key, item): void; -``` - -Defined in: [packages/db/src/indexes/reverse-index.ts:96](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L96) - -#### Parameters - -##### key - -`TKey` - -##### item - -`any` - -#### Returns - -`void` - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`remove`](../interfaces/IndexInterface.md#remove) - *** ### supports() @@ -457,7 +181,7 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:96](https://github.com/Tan supports(operation): boolean; ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:72](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L72) +Defined in: [packages/db/src/indexes/reverse-index.ts:43](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L43) #### Parameters @@ -471,7 +195,9 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:72](https://github.com/Tan #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`supports`](../interfaces/IndexInterface.md#supports) +```ts +IndexReader.supports +``` *** @@ -484,7 +210,7 @@ take( filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:39](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L39) +Defined in: [packages/db/src/indexes/reverse-index.ts:33](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L33) #### Parameters @@ -506,80 +232,19 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:39](https://github.com/Tan #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`take`](../interfaces/IndexInterface.md#take) - -*** - -### takeFromStart() - -```ts -takeFromStart(n, filterFn?): TKey[]; -``` - -Defined in: [packages/db/src/indexes/reverse-index.ts:43](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L43) - -#### Parameters - -##### n - -`number` - -##### filterFn? - -(`key`) => `boolean` - -#### Returns - -`TKey`[] - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`takeFromStart`](../interfaces/IndexInterface.md#takefromstart) - -*** - -### takeReversed() - ```ts -takeReversed( - n, - from, - filterFn?): TKey[]; +IndexReader.take ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:47](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L47) - -#### Parameters - -##### n - -`number` - -##### from - -`any` - -##### filterFn? - -(`key`) => `boolean` - -#### Returns - -`TKey`[] - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`takeReversed`](../interfaces/IndexInterface.md#takereversed) - *** -### takeReversedFromEnd() +### takeFromStart() ```ts -takeReversedFromEnd(n, filterFn?): TKey[]; +takeFromStart(n, filterFn?): TKey[]; ``` -Defined in: [packages/db/src/indexes/reverse-index.ts:55](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L55) +Defined in: [packages/db/src/indexes/reverse-index.ts:37](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L37) #### Parameters @@ -597,39 +262,6 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:55](https://github.com/Tan #### Implementation of -[`IndexInterface`](../interfaces/IndexInterface.md).[`takeReversedFromEnd`](../interfaces/IndexInterface.md#takereversedfromend) - -*** - -### update() - ```ts -update( - key, - oldItem, - newItem): void; +IndexReader.takeFromStart ``` - -Defined in: [packages/db/src/indexes/reverse-index.ts:100](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L100) - -#### Parameters - -##### key - -`TKey` - -##### oldItem - -`any` - -##### newItem - -`any` - -#### Returns - -`void` - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`update`](../interfaces/IndexInterface.md#update) diff --git a/docs/reference/classes/SerializationError.md b/docs/reference/classes/SerializationError.md index 57d07991cd..fc8c27fbf9 100644 --- a/docs/reference/classes/SerializationError.md +++ b/docs/reference/classes/SerializationError.md @@ -5,7 +5,7 @@ title: SerializationError # Class: SerializationError -Defined in: [packages/db/src/errors.ts:636](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L636) +Defined in: [packages/db/src/errors.ts:683](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L683) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:636](https://github.com/TanStack/db/blob/ new SerializationError(operation, originalError): SerializationError; ``` -Defined in: [packages/db/src/errors.ts:637](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L637) +Defined in: [packages/db/src/errors.ts:684](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L684) #### Parameters diff --git a/docs/reference/classes/SetWindowReentrancyError.md b/docs/reference/classes/SetWindowReentrancyError.md new file mode 100644 index 0000000000..0eebc6f6b2 --- /dev/null +++ b/docs/reference/classes/SetWindowReentrancyError.md @@ -0,0 +1,216 @@ +--- +id: SetWindowReentrancyError +title: SetWindowReentrancyError +--- + +# Class: SetWindowReentrancyError + +Defined in: [packages/db/src/errors.ts:796](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L796) + +Error thrown when setWindow is called from inside another setWindow call. + +## Extends + +- [`TanStackDBError`](TanStackDBError.md) + +## Constructors + +### Constructor + +```ts +new SetWindowReentrancyError(): SetWindowReentrancyError; +``` + +Defined in: [packages/db/src/errors.ts:797](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L797) + +#### Returns + +`SetWindowReentrancyError` + +#### Overrides + +[`TanStackDBError`](TanStackDBError.md).[`constructor`](TanStackDBError.md#constructor) + +## Properties + +### cause? + +```ts +optional cause: unknown; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es2022.error.d.ts:26 + +#### Inherited from + +[`TanStackDBError`](TanStackDBError.md).[`cause`](TanStackDBError.md#cause) + +*** + +### message + +```ts +message: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1077 + +#### Inherited from + +[`TanStackDBError`](TanStackDBError.md).[`message`](TanStackDBError.md#message) + +*** + +### name + +```ts +name: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1076 + +#### Inherited from + +[`TanStackDBError`](TanStackDBError.md).[`name`](TanStackDBError.md#name) + +*** + +### stack? + +```ts +optional stack: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1078 + +#### Inherited from + +[`TanStackDBError`](TanStackDBError.md).[`stack`](TanStackDBError.md#stack) + +*** + +### stackTraceLimit + +```ts +static stackTraceLimit: number; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:67 + +The `Error.stackTraceLimit` property specifies the number of stack frames +collected by a stack trace (whether generated by `new Error().stack` or +`Error.captureStackTrace(obj)`). + +The default value is `10` but may be set to any valid JavaScript number. Changes +will affect any stack trace captured _after_ the value has been changed. + +If set to a non-number value, or set to a negative number, stack traces will +not capture any frames. + +#### Inherited from + +[`TanStackDBError`](TanStackDBError.md).[`stackTraceLimit`](TanStackDBError.md#stacktracelimit) + +## Methods + +### captureStackTrace() + +```ts +static captureStackTrace(targetObject, constructorOpt?): void; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:51 + +Creates a `.stack` property on `targetObject`, which when accessed returns +a string representing the location in the code at which +`Error.captureStackTrace()` was called. + +```js +const myObject = {}; +Error.captureStackTrace(myObject); +myObject.stack; // Similar to `new Error().stack` +``` + +The first line of the trace will be prefixed with +`${myObject.name}: ${myObject.message}`. + +The optional `constructorOpt` argument accepts a function. If given, all frames +above `constructorOpt`, including `constructorOpt`, will be omitted from the +generated stack trace. + +The `constructorOpt` argument is useful for hiding implementation +details of error generation from the user. For instance: + +```js +function a() { + b(); +} + +function b() { + c(); +} + +function c() { + // Create an error without stack trace to avoid calculating the stack trace twice. + const { stackTraceLimit } = Error; + Error.stackTraceLimit = 0; + const error = new Error(); + Error.stackTraceLimit = stackTraceLimit; + + // Capture the stack trace above function b + Error.captureStackTrace(error, b); // Neither function c, nor b is included in the stack trace + throw error; +} + +a(); +``` + +#### Parameters + +##### targetObject + +`object` + +##### constructorOpt? + +`Function` + +#### Returns + +`void` + +#### Inherited from + +[`TanStackDBError`](TanStackDBError.md).[`captureStackTrace`](TanStackDBError.md#capturestacktrace) + +*** + +### prepareStackTrace() + +```ts +static prepareStackTrace(err, stackTraces): any; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:55 + +#### Parameters + +##### err + +`Error` + +##### stackTraces + +`CallSite`[] + +#### Returns + +`any` + +#### See + +https://v8.dev/docs/stack-trace-api#customizing-stack-traces + +#### Inherited from + +[`TanStackDBError`](TanStackDBError.md).[`prepareStackTrace`](TanStackDBError.md#preparestacktrace) diff --git a/docs/reference/classes/SetWindowRequiresOrderByError.md b/docs/reference/classes/SetWindowRequiresOrderByError.md index 54e14e7ca9..aa6f1e8889 100644 --- a/docs/reference/classes/SetWindowRequiresOrderByError.md +++ b/docs/reference/classes/SetWindowRequiresOrderByError.md @@ -5,7 +5,7 @@ title: SetWindowRequiresOrderByError # Class: SetWindowRequiresOrderByError -Defined in: [packages/db/src/errors.ts:754](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L754) +Defined in: [packages/db/src/errors.ts:786](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L786) Error thrown when setWindow is called on a collection without an ORDER BY clause. @@ -21,7 +21,7 @@ Error thrown when setWindow is called on a collection without an ORDER BY clause new SetWindowRequiresOrderByError(): SetWindowRequiresOrderByError; ``` -Defined in: [packages/db/src/errors.ts:755](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L755) +Defined in: [packages/db/src/errors.ts:787](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L787) #### Returns diff --git a/docs/reference/classes/StorageError.md b/docs/reference/classes/StorageError.md index 503edc0846..46c7c793b8 100644 --- a/docs/reference/classes/StorageError.md +++ b/docs/reference/classes/StorageError.md @@ -5,7 +5,7 @@ title: StorageError # Class: StorageError -Defined in: [packages/db/src/errors.ts:629](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L629) +Defined in: [packages/db/src/errors.ts:676](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L676) ## Extends @@ -24,7 +24,7 @@ Defined in: [packages/db/src/errors.ts:629](https://github.com/TanStack/db/blob/ new StorageError(message): StorageError; ``` -Defined in: [packages/db/src/errors.ts:630](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L630) +Defined in: [packages/db/src/errors.ts:677](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L677) #### Parameters diff --git a/docs/reference/classes/StorageKeyRequiredError.md b/docs/reference/classes/StorageKeyRequiredError.md index 43c7b92c1b..9afea2005b 100644 --- a/docs/reference/classes/StorageKeyRequiredError.md +++ b/docs/reference/classes/StorageKeyRequiredError.md @@ -5,7 +5,7 @@ title: StorageKeyRequiredError # Class: StorageKeyRequiredError -Defined in: [packages/db/src/errors.ts:652](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L652) +Defined in: [packages/db/src/errors.ts:699](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L699) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:652](https://github.com/TanStack/db/blob/ new StorageKeyRequiredError(): StorageKeyRequiredError; ``` -Defined in: [packages/db/src/errors.ts:653](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L653) +Defined in: [packages/db/src/errors.ts:700](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L700) #### Returns diff --git a/docs/reference/classes/SubQueryMustHaveFromClauseError.md b/docs/reference/classes/SubQueryMustHaveFromClauseError.md index 495a880f2e..905bff0def 100644 --- a/docs/reference/classes/SubQueryMustHaveFromClauseError.md +++ b/docs/reference/classes/SubQueryMustHaveFromClauseError.md @@ -5,7 +5,7 @@ title: SubQueryMustHaveFromClauseError # Class: SubQueryMustHaveFromClauseError -Defined in: [packages/db/src/errors.ts:374](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L374) +Defined in: [packages/db/src/errors.ts:386](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L386) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:374](https://github.com/TanStack/db/blob/ new SubQueryMustHaveFromClauseError(context): SubQueryMustHaveFromClauseError; ``` -Defined in: [packages/db/src/errors.ts:375](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L375) +Defined in: [packages/db/src/errors.ts:387](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L387) #### Parameters diff --git a/docs/reference/classes/SyncCleanupError.md b/docs/reference/classes/SyncCleanupError.md index f36d0e1d45..e083d7e158 100644 --- a/docs/reference/classes/SyncCleanupError.md +++ b/docs/reference/classes/SyncCleanupError.md @@ -5,7 +5,7 @@ title: SyncCleanupError # Class: SyncCleanupError -Defined in: [packages/db/src/errors.ts:675](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L675) +Defined in: [packages/db/src/errors.ts:722](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L722) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:675](https://github.com/TanStack/db/blob/ new SyncCleanupError(collectionId, error): SyncCleanupError; ``` -Defined in: [packages/db/src/errors.ts:676](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L676) +Defined in: [packages/db/src/errors.ts:723](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L723) #### Parameters diff --git a/docs/reference/classes/SyncTransactionAbortedError.md b/docs/reference/classes/SyncTransactionAbortedError.md new file mode 100644 index 0000000000..b2593447c9 --- /dev/null +++ b/docs/reference/classes/SyncTransactionAbortedError.md @@ -0,0 +1,232 @@ +--- +id: SyncTransactionAbortedError +title: SyncTransactionAbortedError +--- + +# Class: SyncTransactionAbortedError + +Defined in: [packages/db/src/errors.ts:733](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L733) + +A sync transaction was canceled before its writes became visible. + +## Extends + +- `Error` + +## Constructors + +### Constructor + +```ts +new SyncTransactionAbortedError(): SyncTransactionAbortedError; +``` + +Defined in: [packages/db/src/errors.ts:734](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L734) + +#### Returns + +`SyncTransactionAbortedError` + +#### Overrides + +```ts +Error.constructor +``` + +## Properties + +### cause? + +```ts +optional cause: unknown; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es2022.error.d.ts:26 + +#### Inherited from + +```ts +Error.cause +``` + +*** + +### message + +```ts +message: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1077 + +#### Inherited from + +```ts +Error.message +``` + +*** + +### name + +```ts +name: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1076 + +#### Inherited from + +```ts +Error.name +``` + +*** + +### stack? + +```ts +optional stack: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1078 + +#### Inherited from + +```ts +Error.stack +``` + +*** + +### stackTraceLimit + +```ts +static stackTraceLimit: number; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:67 + +The `Error.stackTraceLimit` property specifies the number of stack frames +collected by a stack trace (whether generated by `new Error().stack` or +`Error.captureStackTrace(obj)`). + +The default value is `10` but may be set to any valid JavaScript number. Changes +will affect any stack trace captured _after_ the value has been changed. + +If set to a non-number value, or set to a negative number, stack traces will +not capture any frames. + +#### Inherited from + +```ts +Error.stackTraceLimit +``` + +## Methods + +### captureStackTrace() + +```ts +static captureStackTrace(targetObject, constructorOpt?): void; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:51 + +Creates a `.stack` property on `targetObject`, which when accessed returns +a string representing the location in the code at which +`Error.captureStackTrace()` was called. + +```js +const myObject = {}; +Error.captureStackTrace(myObject); +myObject.stack; // Similar to `new Error().stack` +``` + +The first line of the trace will be prefixed with +`${myObject.name}: ${myObject.message}`. + +The optional `constructorOpt` argument accepts a function. If given, all frames +above `constructorOpt`, including `constructorOpt`, will be omitted from the +generated stack trace. + +The `constructorOpt` argument is useful for hiding implementation +details of error generation from the user. For instance: + +```js +function a() { + b(); +} + +function b() { + c(); +} + +function c() { + // Create an error without stack trace to avoid calculating the stack trace twice. + const { stackTraceLimit } = Error; + Error.stackTraceLimit = 0; + const error = new Error(); + Error.stackTraceLimit = stackTraceLimit; + + // Capture the stack trace above function b + Error.captureStackTrace(error, b); // Neither function c, nor b is included in the stack trace + throw error; +} + +a(); +``` + +#### Parameters + +##### targetObject + +`object` + +##### constructorOpt? + +`Function` + +#### Returns + +`void` + +#### Inherited from + +```ts +Error.captureStackTrace +``` + +*** + +### prepareStackTrace() + +```ts +static prepareStackTrace(err, stackTraces): any; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:55 + +#### Parameters + +##### err + +`Error` + +##### stackTraces + +`CallSite`[] + +#### Returns + +`any` + +#### See + +https://v8.dev/docs/stack-trace-api#customizing-stack-traces + +#### Inherited from + +```ts +Error.prepareStackTrace +``` diff --git a/docs/reference/classes/SyncTransactionAlreadyCommittedError.md b/docs/reference/classes/SyncTransactionAlreadyCommittedError.md index ea20a83c9b..3d9223459a 100644 --- a/docs/reference/classes/SyncTransactionAlreadyCommittedError.md +++ b/docs/reference/classes/SyncTransactionAlreadyCommittedError.md @@ -5,7 +5,7 @@ title: SyncTransactionAlreadyCommittedError # Class: SyncTransactionAlreadyCommittedError -Defined in: [packages/db/src/errors.ts:352](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L352) +Defined in: [packages/db/src/errors.ts:364](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L364) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:352](https://github.com/TanStack/db/blob/ new SyncTransactionAlreadyCommittedError(): SyncTransactionAlreadyCommittedError; ``` -Defined in: [packages/db/src/errors.ts:353](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L353) +Defined in: [packages/db/src/errors.ts:365](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L365) #### Returns diff --git a/docs/reference/classes/SyncTransactionAlreadyCommittedWriteError.md b/docs/reference/classes/SyncTransactionAlreadyCommittedWriteError.md index cb22938812..b5ef1ef96b 100644 --- a/docs/reference/classes/SyncTransactionAlreadyCommittedWriteError.md +++ b/docs/reference/classes/SyncTransactionAlreadyCommittedWriteError.md @@ -5,7 +5,7 @@ title: SyncTransactionAlreadyCommittedWriteError # Class: SyncTransactionAlreadyCommittedWriteError -Defined in: [packages/db/src/errors.ts:338](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L338) +Defined in: [packages/db/src/errors.ts:350](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L350) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:338](https://github.com/TanStack/db/blob/ new SyncTransactionAlreadyCommittedWriteError(): SyncTransactionAlreadyCommittedWriteError; ``` -Defined in: [packages/db/src/errors.ts:339](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L339) +Defined in: [packages/db/src/errors.ts:351](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L351) #### Returns diff --git a/docs/reference/classes/TanStackDBError.md b/docs/reference/classes/TanStackDBError.md index 1f1f711b47..62f279ee45 100644 --- a/docs/reference/classes/TanStackDBError.md +++ b/docs/reference/classes/TanStackDBError.md @@ -28,6 +28,7 @@ Defined in: [packages/db/src/errors.ts:2](https://github.com/TanStack/db/blob/ma - [`StorageError`](StorageError.md) - [`SyncCleanupError`](SyncCleanupError.md) - [`QueryOptimizerError`](QueryOptimizerError.md) +- [`SetWindowReentrancyError`](SetWindowReentrancyError.md) ## Constructors diff --git a/docs/reference/classes/TransactionAlreadyCompletedRollbackError.md b/docs/reference/classes/TransactionAlreadyCompletedRollbackError.md index 4331ed7c56..1b4994bfe4 100644 --- a/docs/reference/classes/TransactionAlreadyCompletedRollbackError.md +++ b/docs/reference/classes/TransactionAlreadyCompletedRollbackError.md @@ -5,7 +5,7 @@ title: TransactionAlreadyCompletedRollbackError # Class: TransactionAlreadyCompletedRollbackError -Defined in: [packages/db/src/errors.ts:316](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L316) +Defined in: [packages/db/src/errors.ts:328](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L328) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:316](https://github.com/TanStack/db/blob/ new TransactionAlreadyCompletedRollbackError(): TransactionAlreadyCompletedRollbackError; ``` -Defined in: [packages/db/src/errors.ts:317](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L317) +Defined in: [packages/db/src/errors.ts:329](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L329) #### Returns diff --git a/docs/reference/classes/TransactionError.md b/docs/reference/classes/TransactionError.md index 4b658c8772..87ddfd8b22 100644 --- a/docs/reference/classes/TransactionError.md +++ b/docs/reference/classes/TransactionError.md @@ -5,7 +5,7 @@ title: TransactionError # Class: TransactionError -Defined in: [packages/db/src/errors.ts:286](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L286) +Defined in: [packages/db/src/errors.ts:298](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L298) ## Extends @@ -31,7 +31,7 @@ Defined in: [packages/db/src/errors.ts:286](https://github.com/TanStack/db/blob/ new TransactionError(message): TransactionError; ``` -Defined in: [packages/db/src/errors.ts:287](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L287) +Defined in: [packages/db/src/errors.ts:299](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L299) #### Parameters diff --git a/docs/reference/classes/TransactionNotPendingCommitError.md b/docs/reference/classes/TransactionNotPendingCommitError.md index bcaa2e6e2f..2642e5b407 100644 --- a/docs/reference/classes/TransactionNotPendingCommitError.md +++ b/docs/reference/classes/TransactionNotPendingCommitError.md @@ -5,7 +5,7 @@ title: TransactionNotPendingCommitError # Class: TransactionNotPendingCommitError -Defined in: [packages/db/src/errors.ts:324](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L324) +Defined in: [packages/db/src/errors.ts:336](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L336) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:324](https://github.com/TanStack/db/blob/ new TransactionNotPendingCommitError(): TransactionNotPendingCommitError; ``` -Defined in: [packages/db/src/errors.ts:325](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L325) +Defined in: [packages/db/src/errors.ts:337](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L337) #### Returns diff --git a/docs/reference/classes/TransactionNotPendingMutateError.md b/docs/reference/classes/TransactionNotPendingMutateError.md index 72819dacde..632fbeba3d 100644 --- a/docs/reference/classes/TransactionNotPendingMutateError.md +++ b/docs/reference/classes/TransactionNotPendingMutateError.md @@ -5,7 +5,7 @@ title: TransactionNotPendingMutateError # Class: TransactionNotPendingMutateError -Defined in: [packages/db/src/errors.ts:308](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L308) +Defined in: [packages/db/src/errors.ts:320](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L320) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:308](https://github.com/TanStack/db/blob/ new TransactionNotPendingMutateError(): TransactionNotPendingMutateError; ``` -Defined in: [packages/db/src/errors.ts:309](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L309) +Defined in: [packages/db/src/errors.ts:321](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L321) #### Returns diff --git a/docs/reference/classes/TransactionScope.md b/docs/reference/classes/TransactionScope.md new file mode 100644 index 0000000000..cc5be2d42c --- /dev/null +++ b/docs/reference/classes/TransactionScope.md @@ -0,0 +1,178 @@ +--- +id: TransactionScope +title: TransactionScope +--- + +# Class: TransactionScope + +Defined in: [packages/db/src/transactions.ts:21](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L21) + +## Constructors + +### Constructor + +```ts +new TransactionScope(): TransactionScope; +``` + +#### Returns + +`TransactionScope` + +## Methods + +### clear() + +```ts +clear(): void; +``` + +Defined in: [packages/db/src/transactions.ts:119](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L119) + +#### Returns + +`void` + +*** + +### createTransaction() + +```ts +createTransaction(config): Transaction; +``` + +Defined in: [packages/db/src/transactions.ts:26](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L26) + +#### Type Parameters + +##### T + +`T` *extends* `object` = `Record`\<`string`, `unknown`\> + +#### Parameters + +##### config + +[`TransactionConfig`](../interfaces/TransactionConfig.md)\<`T`\> + +#### Returns + +[`Transaction`](../interfaces/Transaction.md)\<`T`\> + +*** + +### getActiveTransaction() + +```ts +getActiveTransaction(): + | Transaction> + | undefined; +``` + +Defined in: [packages/db/src/transactions.ts:34](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L34) + +#### Returns + + \| [`Transaction`](../interfaces/Transaction.md)\<`Record`\<`string`, `unknown`\>\> + \| `undefined` + +*** + +### getActiveTransactionForCollection() + +```ts +getActiveTransactionForCollection(): + | Transaction> + | undefined; +``` + +Defined in: [packages/db/src/transactions.ts:38](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L38) + +#### Returns + + \| [`Transaction`](../interfaces/Transaction.md)\<`Record`\<`string`, `unknown`\>\> + \| `undefined` + +*** + +### registerTransaction() + +```ts +registerTransaction(transaction): void; +``` + +Defined in: [packages/db/src/transactions.ts:77](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L77) + +#### Parameters + +##### transaction + +[`Transaction`](../interfaces/Transaction.md)\<`any`\> + +#### Returns + +`void` + +*** + +### removeTransaction() + +```ts +removeTransaction(transaction): void; +``` + +Defined in: [packages/db/src/transactions.ts:93](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L93) + +#### Parameters + +##### transaction + +[`Transaction`](../interfaces/Transaction.md)\<`any`\> + +#### Returns + +`void` + +*** + +### rollbackConflictingTransactions() + +```ts +rollbackConflictingTransactions(transaction, mutationIds): void; +``` + +Defined in: [packages/db/src/transactions.ts:102](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L102) + +#### Parameters + +##### transaction + +[`Transaction`](../interfaces/Transaction.md)\<`any`\> + +##### mutationIds + +`Set`\<`string`\> + +#### Returns + +`void` + +*** + +### unregisterTransaction() + +```ts +unregisterTransaction(transaction): void; +``` + +Defined in: [packages/db/src/transactions.ts:83](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L83) + +#### Parameters + +##### transaction + +[`Transaction`](../interfaces/Transaction.md)\<`any`\> + +#### Returns + +`void` diff --git a/docs/reference/classes/UndefinedKeyError.md b/docs/reference/classes/UndefinedKeyError.md index f4b0dfdb4a..a517af00b1 100644 --- a/docs/reference/classes/UndefinedKeyError.md +++ b/docs/reference/classes/UndefinedKeyError.md @@ -5,7 +5,7 @@ title: UndefinedKeyError # Class: UndefinedKeyError -Defined in: [packages/db/src/errors.ts:146](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L146) +Defined in: [packages/db/src/errors.ts:158](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L158) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:146](https://github.com/TanStack/db/blob/ new UndefinedKeyError(item): UndefinedKeyError; ``` -Defined in: [packages/db/src/errors.ts:147](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L147) +Defined in: [packages/db/src/errors.ts:159](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L159) #### Parameters diff --git a/docs/reference/classes/WhereClauseConversionError.md b/docs/reference/classes/UnhashableQueryIRError.md similarity index 70% rename from docs/reference/classes/WhereClauseConversionError.md rename to docs/reference/classes/UnhashableQueryIRError.md index 21d94f4594..93be93d752 100644 --- a/docs/reference/classes/WhereClauseConversionError.md +++ b/docs/reference/classes/UnhashableQueryIRError.md @@ -1,45 +1,45 @@ --- -id: WhereClauseConversionError -title: WhereClauseConversionError +id: UnhashableQueryIRError +title: UnhashableQueryIRError --- -# Class: WhereClauseConversionError +# Class: UnhashableQueryIRError -Defined in: [packages/db/src/errors.ts:702](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L702) - -Internal error when the query optimizer fails to convert a WHERE clause to a collection filter. +Defined in: [packages/db/src/query/ir-stable-identity.ts:54](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L54) ## Extends -- [`QueryOptimizerError`](QueryOptimizerError.md) +- `Error` ## Constructors ### Constructor ```ts -new WhereClauseConversionError(collectionId, alias): WhereClauseConversionError; +new UnhashableQueryIRError(path, reason): UnhashableQueryIRError; ``` -Defined in: [packages/db/src/errors.ts:703](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L703) +Defined in: [packages/db/src/query/ir-stable-identity.ts:55](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L55) #### Parameters -##### collectionId +##### path `string` -##### alias +##### reason `string` #### Returns -`WhereClauseConversionError` +`UnhashableQueryIRError` #### Overrides -[`QueryOptimizerError`](QueryOptimizerError.md).[`constructor`](QueryOptimizerError.md#constructor) +```ts +Error.constructor +``` ## Properties @@ -53,7 +53,9 @@ Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/li #### Inherited from -[`QueryOptimizerError`](QueryOptimizerError.md).[`cause`](QueryOptimizerError.md#cause) +```ts +Error.cause +``` *** @@ -67,7 +69,9 @@ Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/li #### Inherited from -[`QueryOptimizerError`](QueryOptimizerError.md).[`message`](QueryOptimizerError.md#message) +```ts +Error.message +``` *** @@ -81,7 +85,29 @@ Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/li #### Inherited from -[`QueryOptimizerError`](QueryOptimizerError.md).[`name`](QueryOptimizerError.md#name) +```ts +Error.name +``` + +*** + +### path + +```ts +readonly path: string; +``` + +Defined in: [packages/db/src/query/ir-stable-identity.ts:56](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L56) + +*** + +### reason + +```ts +readonly reason: string; +``` + +Defined in: [packages/db/src/query/ir-stable-identity.ts:57](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L57) *** @@ -95,7 +121,9 @@ Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/li #### Inherited from -[`QueryOptimizerError`](QueryOptimizerError.md).[`stack`](QueryOptimizerError.md#stack) +```ts +Error.stack +``` *** @@ -119,7 +147,9 @@ not capture any frames. #### Inherited from -[`QueryOptimizerError`](QueryOptimizerError.md).[`stackTraceLimit`](QueryOptimizerError.md#stacktracelimit) +```ts +Error.stackTraceLimit +``` ## Methods @@ -191,7 +221,9 @@ a(); #### Inherited from -[`QueryOptimizerError`](QueryOptimizerError.md).[`captureStackTrace`](QueryOptimizerError.md#capturestacktrace) +```ts +Error.captureStackTrace +``` *** @@ -223,4 +255,6 @@ https://v8.dev/docs/stack-trace-api#customizing-stack-traces #### Inherited from -[`QueryOptimizerError`](QueryOptimizerError.md).[`prepareStackTrace`](QueryOptimizerError.md#preparestacktrace) +```ts +Error.prepareStackTrace +``` diff --git a/docs/reference/classes/UnknownExpressionTypeError.md b/docs/reference/classes/UnknownExpressionTypeError.md index 0aadc0abbc..4f98a0db71 100644 --- a/docs/reference/classes/UnknownExpressionTypeError.md +++ b/docs/reference/classes/UnknownExpressionTypeError.md @@ -5,7 +5,7 @@ title: UnknownExpressionTypeError # Class: UnknownExpressionTypeError -Defined in: [packages/db/src/errors.ts:512](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L512) +Defined in: [packages/db/src/errors.ts:559](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L559) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:512](https://github.com/TanStack/db/blob/ new UnknownExpressionTypeError(type): UnknownExpressionTypeError; ``` -Defined in: [packages/db/src/errors.ts:513](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L513) +Defined in: [packages/db/src/errors.ts:560](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L560) #### Parameters diff --git a/docs/reference/classes/UnknownFunctionError.md b/docs/reference/classes/UnknownFunctionError.md index fda586a8fb..072bdf0a3f 100644 --- a/docs/reference/classes/UnknownFunctionError.md +++ b/docs/reference/classes/UnknownFunctionError.md @@ -5,7 +5,7 @@ title: UnknownFunctionError # Class: UnknownFunctionError -Defined in: [packages/db/src/errors.ts:524](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L524) +Defined in: [packages/db/src/errors.ts:571](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L571) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:524](https://github.com/TanStack/db/blob/ new UnknownFunctionError(functionName): UnknownFunctionError; ``` -Defined in: [packages/db/src/errors.ts:525](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L525) +Defined in: [packages/db/src/errors.ts:572](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L572) #### Parameters diff --git a/docs/reference/classes/UnknownHavingExpressionTypeError.md b/docs/reference/classes/UnknownHavingExpressionTypeError.md index 6f4806558a..4853043506 100644 --- a/docs/reference/classes/UnknownHavingExpressionTypeError.md +++ b/docs/reference/classes/UnknownHavingExpressionTypeError.md @@ -5,7 +5,7 @@ title: UnknownHavingExpressionTypeError # Class: UnknownHavingExpressionTypeError -Defined in: [packages/db/src/errors.ts:622](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L622) +Defined in: [packages/db/src/errors.ts:669](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L669) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:622](https://github.com/TanStack/db/blob/ new UnknownHavingExpressionTypeError(type): UnknownHavingExpressionTypeError; ``` -Defined in: [packages/db/src/errors.ts:623](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L623) +Defined in: [packages/db/src/errors.ts:670](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L670) #### Parameters diff --git a/docs/reference/classes/AggregateNotSupportedError.md b/docs/reference/classes/UnsafeAliasPathError.md similarity index 89% rename from docs/reference/classes/AggregateNotSupportedError.md rename to docs/reference/classes/UnsafeAliasPathError.md index 7cdcf25e56..5171ccc871 100644 --- a/docs/reference/classes/AggregateNotSupportedError.md +++ b/docs/reference/classes/UnsafeAliasPathError.md @@ -1,13 +1,11 @@ --- -id: AggregateNotSupportedError -title: AggregateNotSupportedError +id: UnsafeAliasPathError +title: UnsafeAliasPathError --- -# Class: AggregateNotSupportedError +# Class: UnsafeAliasPathError -Defined in: [packages/db/src/errors.ts:730](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L730) - -Error thrown when aggregate expressions are used outside of a GROUP BY context. +Defined in: [packages/db/src/errors.ts:457](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L457) ## Extends @@ -18,14 +16,20 @@ Error thrown when aggregate expressions are used outside of a GROUP BY context. ### Constructor ```ts -new AggregateNotSupportedError(): AggregateNotSupportedError; +new UnsafeAliasPathError(segment): UnsafeAliasPathError; ``` -Defined in: [packages/db/src/errors.ts:731](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L731) +Defined in: [packages/db/src/errors.ts:458](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L458) + +#### Parameters + +##### segment + +`string` #### Returns -`AggregateNotSupportedError` +`UnsafeAliasPathError` #### Overrides diff --git a/docs/reference/classes/UnsupportedAggregateFunctionError.md b/docs/reference/classes/UnsupportedAggregateFunctionError.md index b2eaefe29b..f045825c11 100644 --- a/docs/reference/classes/UnsupportedAggregateFunctionError.md +++ b/docs/reference/classes/UnsupportedAggregateFunctionError.md @@ -5,7 +5,7 @@ title: UnsupportedAggregateFunctionError # Class: UnsupportedAggregateFunctionError -Defined in: [packages/db/src/errors.ts:608](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L608) +Defined in: [packages/db/src/errors.ts:655](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L655) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:608](https://github.com/TanStack/db/blob/ new UnsupportedAggregateFunctionError(functionName): UnsupportedAggregateFunctionError; ``` -Defined in: [packages/db/src/errors.ts:609](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L609) +Defined in: [packages/db/src/errors.ts:656](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L656) #### Parameters diff --git a/docs/reference/classes/SubscriptionNotFoundError.md b/docs/reference/classes/UnsupportedFnSelectResultError.md similarity index 84% rename from docs/reference/classes/SubscriptionNotFoundError.md rename to docs/reference/classes/UnsupportedFnSelectResultError.md index 29f745156e..7d4d1b5c7b 100644 --- a/docs/reference/classes/SubscriptionNotFoundError.md +++ b/docs/reference/classes/UnsupportedFnSelectResultError.md @@ -1,14 +1,11 @@ --- -id: SubscriptionNotFoundError -title: SubscriptionNotFoundError +id: UnsupportedFnSelectResultError +title: UnsupportedFnSelectResultError --- -# Class: SubscriptionNotFoundError +# Class: UnsupportedFnSelectResultError -Defined in: [packages/db/src/errors.ts:714](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L714) - -Error when a subscription cannot be found during lazy join processing. -For subqueries, aliases may be remapped (e.g., 'activeUser' → 'user'). +Defined in: [packages/db/src/errors.ts:484](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L484) ## Extends @@ -19,36 +16,20 @@ For subqueries, aliases may be remapped (e.g., 'activeUser' → 'user'). ### Constructor ```ts -new SubscriptionNotFoundError( - resolvedAlias, - originalAlias, - collectionId, - availableAliases): SubscriptionNotFoundError; +new UnsupportedFnSelectResultError(valueDescription): UnsupportedFnSelectResultError; ``` -Defined in: [packages/db/src/errors.ts:715](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L715) +Defined in: [packages/db/src/errors.ts:485](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L485) #### Parameters -##### resolvedAlias +##### valueDescription `string` -##### originalAlias - -`string` - -##### collectionId - -`string` - -##### availableAliases - -`string`[] - #### Returns -`SubscriptionNotFoundError` +`UnsupportedFnSelectResultError` #### Overrides diff --git a/docs/reference/classes/UnsupportedFromTypeError.md b/docs/reference/classes/UnsupportedFromTypeError.md index a2a98b89da..e961dd23ed 100644 --- a/docs/reference/classes/UnsupportedFromTypeError.md +++ b/docs/reference/classes/UnsupportedFromTypeError.md @@ -5,7 +5,7 @@ title: UnsupportedFromTypeError # Class: UnsupportedFromTypeError -Defined in: [packages/db/src/errors.ts:506](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L506) +Defined in: [packages/db/src/errors.ts:553](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L553) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:506](https://github.com/TanStack/db/blob/ new UnsupportedFromTypeError(type): UnsupportedFromTypeError; ``` -Defined in: [packages/db/src/errors.ts:507](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L507) +Defined in: [packages/db/src/errors.ts:554](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L554) #### Parameters diff --git a/docs/reference/classes/UnsupportedJoinSourceTypeError.md b/docs/reference/classes/UnsupportedJoinSourceTypeError.md index f355eee52c..5780342950 100644 --- a/docs/reference/classes/UnsupportedJoinSourceTypeError.md +++ b/docs/reference/classes/UnsupportedJoinSourceTypeError.md @@ -5,7 +5,7 @@ title: UnsupportedJoinSourceTypeError # Class: UnsupportedJoinSourceTypeError -Defined in: [packages/db/src/errors.ts:586](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L586) +Defined in: [packages/db/src/errors.ts:633](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L633) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:586](https://github.com/TanStack/db/blob/ new UnsupportedJoinSourceTypeError(type): UnsupportedJoinSourceTypeError; ``` -Defined in: [packages/db/src/errors.ts:587](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L587) +Defined in: [packages/db/src/errors.ts:634](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L634) #### Parameters diff --git a/docs/reference/classes/UnsupportedJoinTypeError.md b/docs/reference/classes/UnsupportedJoinTypeError.md index 23ba03b1b1..d2ea74d4c8 100644 --- a/docs/reference/classes/UnsupportedJoinTypeError.md +++ b/docs/reference/classes/UnsupportedJoinTypeError.md @@ -5,7 +5,7 @@ title: UnsupportedJoinTypeError # Class: UnsupportedJoinTypeError -Defined in: [packages/db/src/errors.ts:544](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L544) +Defined in: [packages/db/src/errors.ts:591](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L591) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:544](https://github.com/TanStack/db/blob/ new UnsupportedJoinTypeError(joinType): UnsupportedJoinTypeError; ``` -Defined in: [packages/db/src/errors.ts:545](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L545) +Defined in: [packages/db/src/errors.ts:592](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L592) #### Parameters diff --git a/docs/reference/classes/UnsupportedRootScalarSelectError.md b/docs/reference/classes/UnsupportedRootScalarSelectError.md index a3996fdf17..ef8a52246d 100644 --- a/docs/reference/classes/UnsupportedRootScalarSelectError.md +++ b/docs/reference/classes/UnsupportedRootScalarSelectError.md @@ -5,7 +5,7 @@ title: UnsupportedRootScalarSelectError # Class: UnsupportedRootScalarSelectError -Defined in: [packages/db/src/errors.ts:447](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L447) +Defined in: [packages/db/src/errors.ts:494](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L494) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:447](https://github.com/TanStack/db/blob/ new UnsupportedRootScalarSelectError(): UnsupportedRootScalarSelectError; ``` -Defined in: [packages/db/src/errors.ts:448](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L448) +Defined in: [packages/db/src/errors.ts:495](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L495) #### Returns diff --git a/docs/reference/classes/UpdateKeyNotFoundError.md b/docs/reference/classes/UpdateKeyNotFoundError.md index d6178ee922..85a0dfdd5a 100644 --- a/docs/reference/classes/UpdateKeyNotFoundError.md +++ b/docs/reference/classes/UpdateKeyNotFoundError.md @@ -5,7 +5,7 @@ title: UpdateKeyNotFoundError # Class: UpdateKeyNotFoundError -Defined in: [packages/db/src/errors.ts:223](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L223) +Defined in: [packages/db/src/errors.ts:235](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L235) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/db/src/errors.ts:223](https://github.com/TanStack/db/blob/ new UpdateKeyNotFoundError(key): UpdateKeyNotFoundError; ``` -Defined in: [packages/db/src/errors.ts:224](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L224) +Defined in: [packages/db/src/errors.ts:236](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L236) #### Parameters diff --git a/docs/reference/electric-db-collection/functions/electricCollectionOptions.md b/docs/reference/electric-db-collection/functions/electricCollectionOptions.md index 8e80aeed8e..a087d9bdc9 100644 --- a/docs/reference/electric-db-collection/functions/electricCollectionOptions.md +++ b/docs/reference/electric-db-collection/functions/electricCollectionOptions.md @@ -11,7 +11,7 @@ title: electricCollectionOptions function electricCollectionOptions(config): Omit, string | number, T, UtilsRecord>, "utils" | "onInsert" | "onUpdate" | "onDelete"> & Pick, T>, "onInsert" | "onUpdate" | "onDelete"> & object; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:576](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L576) +Defined in: [packages/electric-db-collection/src/electric.ts:765](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L765) Creates Electric collection options for use with a standard Collection @@ -43,7 +43,7 @@ Collection options with utilities function electricCollectionOptions(config): Omit, "utils" | "onInsert" | "onUpdate" | "onDelete"> & Pick, "onInsert" | "onUpdate" | "onDelete"> & object; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:594](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L594) +Defined in: [packages/electric-db-collection/src/electric.ts:783](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L783) Creates Electric collection options for use with a standard Collection diff --git a/docs/reference/electric-db-collection/functions/isChangeMessage.md b/docs/reference/electric-db-collection/functions/isChangeMessage.md index 0e9732cd2a..81de9bff39 100644 --- a/docs/reference/electric-db-collection/functions/isChangeMessage.md +++ b/docs/reference/electric-db-collection/functions/isChangeMessage.md @@ -9,7 +9,7 @@ title: isChangeMessage function isChangeMessage(message): message is ChangeMessage; ``` -Defined in: node\_modules/.pnpm/@electric-sql+client@1.5.14/node\_modules/@electric-sql/client/dist/index.d.ts:889 +Defined in: node\_modules/.pnpm/@electric-sql+client@1.5.15/node\_modules/@electric-sql/client/dist/index.d.ts:922 Type guard for checking Message is ChangeMessage. diff --git a/docs/reference/electric-db-collection/functions/isControlMessage.md b/docs/reference/electric-db-collection/functions/isControlMessage.md index 66ea4b70f3..6cd6b1582a 100644 --- a/docs/reference/electric-db-collection/functions/isControlMessage.md +++ b/docs/reference/electric-db-collection/functions/isControlMessage.md @@ -9,7 +9,7 @@ title: isControlMessage function isControlMessage(message): message is ControlMessage; ``` -Defined in: node\_modules/.pnpm/@electric-sql+client@1.5.14/node\_modules/@electric-sql/client/dist/index.d.ts:907 +Defined in: node\_modules/.pnpm/@electric-sql+client@1.5.15/node\_modules/@electric-sql/client/dist/index.d.ts:940 Type guard for checking Message is ControlMessage. diff --git a/docs/reference/electric-db-collection/interfaces/ElectricCollectionConfig.md b/docs/reference/electric-db-collection/interfaces/ElectricCollectionConfig.md index 95c0ec9f3c..991e624126 100644 --- a/docs/reference/electric-db-collection/interfaces/ElectricCollectionConfig.md +++ b/docs/reference/electric-db-collection/interfaces/ElectricCollectionConfig.md @@ -5,7 +5,7 @@ title: ElectricCollectionConfig # Interface: ElectricCollectionConfig\ -Defined in: [packages/electric-db-collection/src/electric.ts:148](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L148) +Defined in: [packages/electric-db-collection/src/electric.ts:282](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L282) Configuration interface for Electric collection options @@ -35,7 +35,7 @@ The schema type for validation optional [ELECTRIC_TEST_HOOKS]: ElectricTestHooks; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:171](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L171) +Defined in: [packages/electric-db-collection/src/electric.ts:305](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L305) Internal test hooks (for testing only) Hidden via Symbol to prevent accidental usage in production @@ -48,7 +48,7 @@ Hidden via Symbol to prevent accidental usage in production optional onDelete: (params) => Promise; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:288](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L288) +Defined in: [packages/electric-db-collection/src/electric.ts:422](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L422) Optional asynchronous handler function called before a delete operation @@ -100,7 +100,7 @@ onDelete: async ({ transaction, collection }) => { optional onInsert: (params) => Promise; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:219](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L219) +Defined in: [packages/electric-db-collection/src/electric.ts:353](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L353) Optional asynchronous handler function called before an insert operation @@ -174,7 +174,7 @@ onInsert: async ({ transaction, collection }) => { optional onUpdate: (params) => Promise; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:254](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L254) +Defined in: [packages/electric-db-collection/src/electric.ts:388](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L388) Optional asynchronous handler function called before an update operation @@ -227,7 +227,7 @@ onUpdate: async ({ transaction, collection }) => { shapeOptions: ShapeStreamOptions>; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:164](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L164) +Defined in: [packages/electric-db-collection/src/electric.ts:298](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L298) Configuration options for the ElectricSQL ShapeStream @@ -239,4 +239,4 @@ Configuration options for the ElectricSQL ShapeStream optional syncMode: ElectricSyncMode; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:165](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L165) +Defined in: [packages/electric-db-collection/src/electric.ts:299](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L299) diff --git a/docs/reference/electric-db-collection/interfaces/ElectricCollectionUtils.md b/docs/reference/electric-db-collection/interfaces/ElectricCollectionUtils.md index b7ce5f0c9c..a909f70113 100644 --- a/docs/reference/electric-db-collection/interfaces/ElectricCollectionUtils.md +++ b/docs/reference/electric-db-collection/interfaces/ElectricCollectionUtils.md @@ -5,7 +5,7 @@ title: ElectricCollectionUtils # Interface: ElectricCollectionUtils\ -Defined in: [packages/electric-db-collection/src/electric.ts:558](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L558) +Defined in: [packages/electric-db-collection/src/electric.ts:747](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L747) Electric collection utilities type @@ -33,7 +33,7 @@ Electric collection utilities type awaitMatch: AwaitMatchFn; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:562](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L562) +Defined in: [packages/electric-db-collection/src/electric.ts:751](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L751) *** @@ -43,4 +43,4 @@ Defined in: [packages/electric-db-collection/src/electric.ts:562](https://github awaitTxId: AwaitTxIdFn; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:561](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L561) +Defined in: [packages/electric-db-collection/src/electric.ts:750](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L750) diff --git a/docs/reference/electric-db-collection/type-aliases/AwaitTxIdFn.md b/docs/reference/electric-db-collection/type-aliases/AwaitTxIdFn.md index b03f81a358..7902672452 100644 --- a/docs/reference/electric-db-collection/type-aliases/AwaitTxIdFn.md +++ b/docs/reference/electric-db-collection/type-aliases/AwaitTxIdFn.md @@ -9,7 +9,7 @@ title: AwaitTxIdFn type AwaitTxIdFn = (txId, timeout?) => Promise; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:545](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L545) +Defined in: [packages/electric-db-collection/src/electric.ts:734](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L734) Type for the awaitTxId utility function diff --git a/docs/reference/electric-db-collection/type-aliases/Txid.md b/docs/reference/electric-db-collection/type-aliases/Txid.md index f16e8e37ad..dabb4250d1 100644 --- a/docs/reference/electric-db-collection/type-aliases/Txid.md +++ b/docs/reference/electric-db-collection/type-aliases/Txid.md @@ -9,6 +9,6 @@ title: Txid type Txid = number; ``` -Defined in: [packages/electric-db-collection/src/electric.ts:86](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L86) +Defined in: [packages/electric-db-collection/src/electric.ts:94](https://github.com/TanStack/db/blob/main/packages/electric-db-collection/src/electric.ts#L94) Type representing a transaction ID in ElectricSQL diff --git a/docs/reference/functions/add.md b/docs/reference/functions/add.md index 1c3ceb9246..c7dddb77cf 100644 --- a/docs/reference/functions/add.md +++ b/docs/reference/functions/add.md @@ -6,10 +6,10 @@ title: add # Function: add() ```ts -function add(left, right): BinaryNumericReturnType; +function add(left, right): BinaryNumericReturnType; ``` -Defined in: [packages/db/src/query/builder/functions.ts:354](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L354) +Defined in: [packages/db/src/query/builder/functions.ts:601](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L601) ## Type Parameters @@ -33,4 +33,4 @@ Defined in: [packages/db/src/query/builder/functions.ts:354](https://github.com/ ## Returns -`BinaryNumericReturnType`\<`T1`, `T2`\> +`BinaryNumericReturnType` diff --git a/docs/reference/functions/and.md b/docs/reference/functions/and.md index b99e1c2326..b463b2ec2f 100644 --- a/docs/reference/functions/and.md +++ b/docs/reference/functions/and.md @@ -11,7 +11,7 @@ title: and function and(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:199](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L199) +Defined in: [packages/db/src/query/builder/functions.ts:203](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L203) ### Parameters @@ -36,7 +36,7 @@ function and( rest): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:203](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L203) +Defined in: [packages/db/src/query/builder/functions.ts:207](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L207) ### Parameters diff --git a/docs/reference/functions/assertLiveQueryWindowManyResult.md b/docs/reference/functions/assertLiveQueryWindowManyResult.md new file mode 100644 index 0000000000..09b0010b58 --- /dev/null +++ b/docs/reference/functions/assertLiveQueryWindowManyResult.md @@ -0,0 +1,26 @@ +--- +id: assertLiveQueryWindowManyResult +title: assertLiveQueryWindowManyResult +--- + +# Function: assertLiveQueryWindowManyResult() + +```ts +function assertLiveQueryWindowManyResult(collection): void; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:378](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L378) + +**`Internal`** + +Shared validation for infinite-query adapters. + +## Parameters + +### collection + +[`Collection`](../interfaces/Collection.md)\<`any`, `any`, `any`\> + +## Returns + +`void` diff --git a/docs/reference/functions/avg.md b/docs/reference/functions/avg.md index acf474d673..2613e64655 100644 --- a/docs/reference/functions/avg.md +++ b/docs/reference/functions/avg.md @@ -9,7 +9,7 @@ title: avg function avg(arg): AggregateReturnType; ``` -Defined in: [packages/db/src/query/builder/functions.ts:370](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L370) +Defined in: [packages/db/src/query/builder/functions.ts:647](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L647) ## Type Parameters diff --git a/docs/reference/functions/canonicalizeQueryIR.md b/docs/reference/functions/canonicalizeQueryIR.md new file mode 100644 index 0000000000..3519ef1185 --- /dev/null +++ b/docs/reference/functions/canonicalizeQueryIR.md @@ -0,0 +1,22 @@ +--- +id: canonicalizeQueryIR +title: canonicalizeQueryIR +--- + +# Function: canonicalizeQueryIR() + +```ts +function canonicalizeQueryIR(query): StableIdentityValue; +``` + +Defined in: [packages/db/src/query/ir-stable-identity.ts:165](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L165) + +## Parameters + +### query + +[`QueryIR`](../@tanstack/namespaces/IR/interfaces/QueryIR.md) + +## Returns + +`StableIdentityValue` diff --git a/docs/reference/functions/caseWhen.md b/docs/reference/functions/caseWhen.md new file mode 100644 index 0000000000..17559636a0 --- /dev/null +++ b/docs/reference/functions/caseWhen.md @@ -0,0 +1,1299 @@ +--- +id: caseWhen +title: caseWhen +--- + +# Function: caseWhen() + +## Call Signature + +```ts +function caseWhen(condition1, value1): CaseWhenResult<[V1], false>; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:399](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L399) + +Returns the value for the first matching condition, similar to SQL +`CASE WHEN`. + +Arguments are evaluated as condition/value pairs followed by an optional +default value. Scalar branch values return a query expression and can be used +in expression contexts like `select`, `where`, `orderBy`, `groupBy`, +`having`, and equality join operands. If no scalar branch matches and no +default is provided, the result is `null`. + +When a branch value is a projection object, `caseWhen` becomes a select-only +projection value. Projection branches can include nested fields, ref spreads, +and includes. If no projection branch matches and no default is provided, the +result is `undefined`. + +### Type Parameters + +#### C1 + +`C1` *extends* `ExpressionLike` + +#### V1 + +`V1` *extends* `CaseWhenValue` + +### Parameters + +#### condition1 + +`C1` + +#### value1 + +`V1` + +### Returns + +`CaseWhenResult`\<\[`V1`\], `false`\> + +### Examples + +```ts +caseWhen(gt(user.age, 18), `adult`, `minor`) +``` + +```ts +caseWhen( + gt(user.age, 65), + `senior`, + gt(user.age, 18), + `adult`, + `minor`, +) +``` + +```ts +caseWhen(gt(user.age, 18), { + ...user, + posts: q + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)), +}) +``` + +## Call Signature + +```ts +function caseWhen( + condition1, + value1, +defaultValue): CaseWhenResult<[V1, D], true>; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:403](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L403) + +Returns the value for the first matching condition, similar to SQL +`CASE WHEN`. + +Arguments are evaluated as condition/value pairs followed by an optional +default value. Scalar branch values return a query expression and can be used +in expression contexts like `select`, `where`, `orderBy`, `groupBy`, +`having`, and equality join operands. If no scalar branch matches and no +default is provided, the result is `null`. + +When a branch value is a projection object, `caseWhen` becomes a select-only +projection value. Projection branches can include nested fields, ref spreads, +and includes. If no projection branch matches and no default is provided, the +result is `undefined`. + +### Type Parameters + +#### C1 + +`C1` *extends* `ExpressionLike` + +#### V1 + +`V1` *extends* `CaseWhenValue` + +#### D + +`D` *extends* `CaseWhenValue` + +### Parameters + +#### condition1 + +`C1` + +#### value1 + +`V1` + +#### defaultValue + +`D` + +### Returns + +`CaseWhenResult`\<\[`V1`, `D`\], `true`\> + +### Examples + +```ts +caseWhen(gt(user.age, 18), `adult`, `minor`) +``` + +```ts +caseWhen( + gt(user.age, 65), + `senior`, + gt(user.age, 18), + `adult`, + `minor`, +) +``` + +```ts +caseWhen(gt(user.age, 18), { + ...user, + posts: q + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)), +}) +``` + +## Call Signature + +```ts +function caseWhen( + condition1, + value1, + condition2, +value2): CaseWhenResult<[V1, V2], false>; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:408](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L408) + +Returns the value for the first matching condition, similar to SQL +`CASE WHEN`. + +Arguments are evaluated as condition/value pairs followed by an optional +default value. Scalar branch values return a query expression and can be used +in expression contexts like `select`, `where`, `orderBy`, `groupBy`, +`having`, and equality join operands. If no scalar branch matches and no +default is provided, the result is `null`. + +When a branch value is a projection object, `caseWhen` becomes a select-only +projection value. Projection branches can include nested fields, ref spreads, +and includes. If no projection branch matches and no default is provided, the +result is `undefined`. + +### Type Parameters + +#### C1 + +`C1` *extends* `ExpressionLike` + +#### V1 + +`V1` *extends* `CaseWhenValue` + +#### C2 + +`C2` *extends* `ExpressionLike` + +#### V2 + +`V2` *extends* `CaseWhenValue` + +### Parameters + +#### condition1 + +`C1` + +#### value1 + +`V1` + +#### condition2 + +`C2` + +#### value2 + +`V2` + +### Returns + +`CaseWhenResult`\<\[`V1`, `V2`\], `false`\> + +### Examples + +```ts +caseWhen(gt(user.age, 18), `adult`, `minor`) +``` + +```ts +caseWhen( + gt(user.age, 65), + `senior`, + gt(user.age, 18), + `adult`, + `minor`, +) +``` + +```ts +caseWhen(gt(user.age, 18), { + ...user, + posts: q + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)), +}) +``` + +## Call Signature + +```ts +function caseWhen( + condition1, + value1, + condition2, + value2, +defaultValue): CaseWhenResult<[V1, V2, D], true>; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:419](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L419) + +Returns the value for the first matching condition, similar to SQL +`CASE WHEN`. + +Arguments are evaluated as condition/value pairs followed by an optional +default value. Scalar branch values return a query expression and can be used +in expression contexts like `select`, `where`, `orderBy`, `groupBy`, +`having`, and equality join operands. If no scalar branch matches and no +default is provided, the result is `null`. + +When a branch value is a projection object, `caseWhen` becomes a select-only +projection value. Projection branches can include nested fields, ref spreads, +and includes. If no projection branch matches and no default is provided, the +result is `undefined`. + +### Type Parameters + +#### C1 + +`C1` *extends* `ExpressionLike` + +#### V1 + +`V1` *extends* `CaseWhenValue` + +#### C2 + +`C2` *extends* `ExpressionLike` + +#### V2 + +`V2` *extends* `CaseWhenValue` + +#### D + +`D` *extends* `CaseWhenValue` + +### Parameters + +#### condition1 + +`C1` + +#### value1 + +`V1` + +#### condition2 + +`C2` + +#### value2 + +`V2` + +#### defaultValue + +`D` + +### Returns + +`CaseWhenResult`\<\[`V1`, `V2`, `D`\], `true`\> + +### Examples + +```ts +caseWhen(gt(user.age, 18), `adult`, `minor`) +``` + +```ts +caseWhen( + gt(user.age, 65), + `senior`, + gt(user.age, 18), + `adult`, + `minor`, +) +``` + +```ts +caseWhen(gt(user.age, 18), { + ...user, + posts: q + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)), +}) +``` + +## Call Signature + +```ts +function caseWhen( + condition1, + value1, + condition2, + value2, + condition3, +value3): CaseWhenResult<[V1, V2, V3], false>; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:432](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L432) + +Returns the value for the first matching condition, similar to SQL +`CASE WHEN`. + +Arguments are evaluated as condition/value pairs followed by an optional +default value. Scalar branch values return a query expression and can be used +in expression contexts like `select`, `where`, `orderBy`, `groupBy`, +`having`, and equality join operands. If no scalar branch matches and no +default is provided, the result is `null`. + +When a branch value is a projection object, `caseWhen` becomes a select-only +projection value. Projection branches can include nested fields, ref spreads, +and includes. If no projection branch matches and no default is provided, the +result is `undefined`. + +### Type Parameters + +#### C1 + +`C1` *extends* `ExpressionLike` + +#### V1 + +`V1` *extends* `CaseWhenValue` + +#### C2 + +`C2` *extends* `ExpressionLike` + +#### V2 + +`V2` *extends* `CaseWhenValue` + +#### C3 + +`C3` *extends* `ExpressionLike` + +#### V3 + +`V3` *extends* `CaseWhenValue` + +### Parameters + +#### condition1 + +`C1` + +#### value1 + +`V1` + +#### condition2 + +`C2` + +#### value2 + +`V2` + +#### condition3 + +`C3` + +#### value3 + +`V3` + +### Returns + +`CaseWhenResult`\<\[`V1`, `V2`, `V3`\], `false`\> + +### Examples + +```ts +caseWhen(gt(user.age, 18), `adult`, `minor`) +``` + +```ts +caseWhen( + gt(user.age, 65), + `senior`, + gt(user.age, 18), + `adult`, + `minor`, +) +``` + +```ts +caseWhen(gt(user.age, 18), { + ...user, + posts: q + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)), +}) +``` + +## Call Signature + +```ts +function caseWhen( + condition1, + value1, + condition2, + value2, + condition3, + value3, +defaultValue): CaseWhenResult<[V1, V2, V3, D], true>; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:447](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L447) + +Returns the value for the first matching condition, similar to SQL +`CASE WHEN`. + +Arguments are evaluated as condition/value pairs followed by an optional +default value. Scalar branch values return a query expression and can be used +in expression contexts like `select`, `where`, `orderBy`, `groupBy`, +`having`, and equality join operands. If no scalar branch matches and no +default is provided, the result is `null`. + +When a branch value is a projection object, `caseWhen` becomes a select-only +projection value. Projection branches can include nested fields, ref spreads, +and includes. If no projection branch matches and no default is provided, the +result is `undefined`. + +### Type Parameters + +#### C1 + +`C1` *extends* `ExpressionLike` + +#### V1 + +`V1` *extends* `CaseWhenValue` + +#### C2 + +`C2` *extends* `ExpressionLike` + +#### V2 + +`V2` *extends* `CaseWhenValue` + +#### C3 + +`C3` *extends* `ExpressionLike` + +#### V3 + +`V3` *extends* `CaseWhenValue` + +#### D + +`D` *extends* `CaseWhenValue` + +### Parameters + +#### condition1 + +`C1` + +#### value1 + +`V1` + +#### condition2 + +`C2` + +#### value2 + +`V2` + +#### condition3 + +`C3` + +#### value3 + +`V3` + +#### defaultValue + +`D` + +### Returns + +`CaseWhenResult`\<\[`V1`, `V2`, `V3`, `D`\], `true`\> + +### Examples + +```ts +caseWhen(gt(user.age, 18), `adult`, `minor`) +``` + +```ts +caseWhen( + gt(user.age, 65), + `senior`, + gt(user.age, 18), + `adult`, + `minor`, +) +``` + +```ts +caseWhen(gt(user.age, 18), { + ...user, + posts: q + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)), +}) +``` + +## Call Signature + +```ts +function caseWhen( + condition1, + value1, + condition2, + value2, + condition3, + value3, + condition4, +value4): CaseWhenResult<[V1, V2, V3, V4], false>; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:464](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L464) + +Returns the value for the first matching condition, similar to SQL +`CASE WHEN`. + +Arguments are evaluated as condition/value pairs followed by an optional +default value. Scalar branch values return a query expression and can be used +in expression contexts like `select`, `where`, `orderBy`, `groupBy`, +`having`, and equality join operands. If no scalar branch matches and no +default is provided, the result is `null`. + +When a branch value is a projection object, `caseWhen` becomes a select-only +projection value. Projection branches can include nested fields, ref spreads, +and includes. If no projection branch matches and no default is provided, the +result is `undefined`. + +### Type Parameters + +#### C1 + +`C1` *extends* `ExpressionLike` + +#### V1 + +`V1` *extends* `CaseWhenValue` + +#### C2 + +`C2` *extends* `ExpressionLike` + +#### V2 + +`V2` *extends* `CaseWhenValue` + +#### C3 + +`C3` *extends* `ExpressionLike` + +#### V3 + +`V3` *extends* `CaseWhenValue` + +#### C4 + +`C4` *extends* `ExpressionLike` + +#### V4 + +`V4` *extends* `CaseWhenValue` + +### Parameters + +#### condition1 + +`C1` + +#### value1 + +`V1` + +#### condition2 + +`C2` + +#### value2 + +`V2` + +#### condition3 + +`C3` + +#### value3 + +`V3` + +#### condition4 + +`C4` + +#### value4 + +`V4` + +### Returns + +`CaseWhenResult`\<\[`V1`, `V2`, `V3`, `V4`\], `false`\> + +### Examples + +```ts +caseWhen(gt(user.age, 18), `adult`, `minor`) +``` + +```ts +caseWhen( + gt(user.age, 65), + `senior`, + gt(user.age, 18), + `adult`, + `minor`, +) +``` + +```ts +caseWhen(gt(user.age, 18), { + ...user, + posts: q + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)), +}) +``` + +## Call Signature + +```ts +function caseWhen( + condition1, + value1, + condition2, + value2, + condition3, + value3, + condition4, + value4, +defaultValue): CaseWhenResult<[V1, V2, V3, V4, D], true>; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:483](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L483) + +Returns the value for the first matching condition, similar to SQL +`CASE WHEN`. + +Arguments are evaluated as condition/value pairs followed by an optional +default value. Scalar branch values return a query expression and can be used +in expression contexts like `select`, `where`, `orderBy`, `groupBy`, +`having`, and equality join operands. If no scalar branch matches and no +default is provided, the result is `null`. + +When a branch value is a projection object, `caseWhen` becomes a select-only +projection value. Projection branches can include nested fields, ref spreads, +and includes. If no projection branch matches and no default is provided, the +result is `undefined`. + +### Type Parameters + +#### C1 + +`C1` *extends* `ExpressionLike` + +#### V1 + +`V1` *extends* `CaseWhenValue` + +#### C2 + +`C2` *extends* `ExpressionLike` + +#### V2 + +`V2` *extends* `CaseWhenValue` + +#### C3 + +`C3` *extends* `ExpressionLike` + +#### V3 + +`V3` *extends* `CaseWhenValue` + +#### C4 + +`C4` *extends* `ExpressionLike` + +#### V4 + +`V4` *extends* `CaseWhenValue` + +#### D + +`D` *extends* `CaseWhenValue` + +### Parameters + +#### condition1 + +`C1` + +#### value1 + +`V1` + +#### condition2 + +`C2` + +#### value2 + +`V2` + +#### condition3 + +`C3` + +#### value3 + +`V3` + +#### condition4 + +`C4` + +#### value4 + +`V4` + +#### defaultValue + +`D` + +### Returns + +`CaseWhenResult`\<\[`V1`, `V2`, `V3`, `V4`, `D`\], `true`\> + +### Examples + +```ts +caseWhen(gt(user.age, 18), `adult`, `minor`) +``` + +```ts +caseWhen( + gt(user.age, 65), + `senior`, + gt(user.age, 18), + `adult`, + `minor`, +) +``` + +```ts +caseWhen(gt(user.age, 18), { + ...user, + posts: q + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)), +}) +``` + +## Call Signature + +```ts +function caseWhen( + condition1, + value1, + condition2, + value2, + condition3, + value3, + condition4, + value4, + condition5, +value5): CaseWhenResult<[V1, V2, V3, V4, V5], false>; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:504](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L504) + +Returns the value for the first matching condition, similar to SQL +`CASE WHEN`. + +Arguments are evaluated as condition/value pairs followed by an optional +default value. Scalar branch values return a query expression and can be used +in expression contexts like `select`, `where`, `orderBy`, `groupBy`, +`having`, and equality join operands. If no scalar branch matches and no +default is provided, the result is `null`. + +When a branch value is a projection object, `caseWhen` becomes a select-only +projection value. Projection branches can include nested fields, ref spreads, +and includes. If no projection branch matches and no default is provided, the +result is `undefined`. + +### Type Parameters + +#### C1 + +`C1` *extends* `ExpressionLike` + +#### V1 + +`V1` *extends* `CaseWhenValue` + +#### C2 + +`C2` *extends* `ExpressionLike` + +#### V2 + +`V2` *extends* `CaseWhenValue` + +#### C3 + +`C3` *extends* `ExpressionLike` + +#### V3 + +`V3` *extends* `CaseWhenValue` + +#### C4 + +`C4` *extends* `ExpressionLike` + +#### V4 + +`V4` *extends* `CaseWhenValue` + +#### C5 + +`C5` *extends* `ExpressionLike` + +#### V5 + +`V5` *extends* `CaseWhenValue` + +### Parameters + +#### condition1 + +`C1` + +#### value1 + +`V1` + +#### condition2 + +`C2` + +#### value2 + +`V2` + +#### condition3 + +`C3` + +#### value3 + +`V3` + +#### condition4 + +`C4` + +#### value4 + +`V4` + +#### condition5 + +`C5` + +#### value5 + +`V5` + +### Returns + +`CaseWhenResult`\<\[`V1`, `V2`, `V3`, `V4`, `V5`\], `false`\> + +### Examples + +```ts +caseWhen(gt(user.age, 18), `adult`, `minor`) +``` + +```ts +caseWhen( + gt(user.age, 65), + `senior`, + gt(user.age, 18), + `adult`, + `minor`, +) +``` + +```ts +caseWhen(gt(user.age, 18), { + ...user, + posts: q + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)), +}) +``` + +## Call Signature + +```ts +function caseWhen( + condition1, + value1, + condition2, + value2, + condition3, + value3, + condition4, + value4, + condition5, + value5, +defaultValue): CaseWhenResult<[V1, V2, V3, V4, V5, D], true>; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:527](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L527) + +Returns the value for the first matching condition, similar to SQL +`CASE WHEN`. + +Arguments are evaluated as condition/value pairs followed by an optional +default value. Scalar branch values return a query expression and can be used +in expression contexts like `select`, `where`, `orderBy`, `groupBy`, +`having`, and equality join operands. If no scalar branch matches and no +default is provided, the result is `null`. + +When a branch value is a projection object, `caseWhen` becomes a select-only +projection value. Projection branches can include nested fields, ref spreads, +and includes. If no projection branch matches and no default is provided, the +result is `undefined`. + +### Type Parameters + +#### C1 + +`C1` *extends* `ExpressionLike` + +#### V1 + +`V1` *extends* `CaseWhenValue` + +#### C2 + +`C2` *extends* `ExpressionLike` + +#### V2 + +`V2` *extends* `CaseWhenValue` + +#### C3 + +`C3` *extends* `ExpressionLike` + +#### V3 + +`V3` *extends* `CaseWhenValue` + +#### C4 + +`C4` *extends* `ExpressionLike` + +#### V4 + +`V4` *extends* `CaseWhenValue` + +#### C5 + +`C5` *extends* `ExpressionLike` + +#### V5 + +`V5` *extends* `CaseWhenValue` + +#### D + +`D` *extends* `CaseWhenValue` + +### Parameters + +#### condition1 + +`C1` + +#### value1 + +`V1` + +#### condition2 + +`C2` + +#### value2 + +`V2` + +#### condition3 + +`C3` + +#### value3 + +`V3` + +#### condition4 + +`C4` + +#### value4 + +`V4` + +#### condition5 + +`C5` + +#### value5 + +`V5` + +#### defaultValue + +`D` + +### Returns + +`CaseWhenResult`\<\[`V1`, `V2`, `V3`, `V4`, `V5`, `D`\], `true`\> + +### Examples + +```ts +caseWhen(gt(user.age, 18), `adult`, `minor`) +``` + +```ts +caseWhen( + gt(user.age, 65), + `senior`, + gt(user.age, 18), + `adult`, + `minor`, +) +``` + +```ts +caseWhen(gt(user.age, 18), { + ...user, + posts: q + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)), +}) +``` + +## Call Signature + +```ts +function caseWhen( + condition1, + value1, + condition2, + value2, + condition3, + value3, + condition4, + value4, + condition5, + value5, + condition6, + value6, ... + rest): any; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:552](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L552) + +Returns the value for the first matching condition, similar to SQL +`CASE WHEN`. + +Arguments are evaluated as condition/value pairs followed by an optional +default value. Scalar branch values return a query expression and can be used +in expression contexts like `select`, `where`, `orderBy`, `groupBy`, +`having`, and equality join operands. If no scalar branch matches and no +default is provided, the result is `null`. + +When a branch value is a projection object, `caseWhen` becomes a select-only +projection value. Projection branches can include nested fields, ref spreads, +and includes. If no projection branch matches and no default is provided, the +result is `undefined`. + +### Type Parameters + +#### C1 + +`C1` *extends* `ExpressionLike` + +#### V1 + +`V1` *extends* `CaseWhenValue` + +#### C2 + +`C2` *extends* `ExpressionLike` + +#### V2 + +`V2` *extends* `CaseWhenValue` + +#### C3 + +`C3` *extends* `ExpressionLike` + +#### V3 + +`V3` *extends* `CaseWhenValue` + +#### C4 + +`C4` *extends* `ExpressionLike` + +#### V4 + +`V4` *extends* `CaseWhenValue` + +#### C5 + +`C5` *extends* `ExpressionLike` + +#### V5 + +`V5` *extends* `CaseWhenValue` + +### Parameters + +#### condition1 + +`C1` + +#### value1 + +`V1` + +#### condition2 + +`C2` + +#### value2 + +`V2` + +#### condition3 + +`C3` + +#### value3 + +`V3` + +#### condition4 + +`C4` + +#### value4 + +`V4` + +#### condition5 + +`C5` + +#### value5 + +`V5` + +#### condition6 + +`ExpressionLike` + +#### value6 + +`CaseWhenValue` + +#### rest + +...`CaseWhenValue`[] + +### Returns + +`any` + +### Examples + +```ts +caseWhen(gt(user.age, 18), `adult`, `minor`) +``` + +```ts +caseWhen( + gt(user.age, 65), + `senior`, + gt(user.age, 18), + `adult`, + `minor`, +) +``` + +```ts +caseWhen(gt(user.age, 18), { + ...user, + posts: q + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)), +}) +``` diff --git a/docs/reference/functions/coalesce.md b/docs/reference/functions/coalesce.md index d78f0bbc8c..6d4e95ecb2 100644 --- a/docs/reference/functions/coalesce.md +++ b/docs/reference/functions/coalesce.md @@ -9,7 +9,7 @@ title: coalesce function coalesce(...args): CoalesceReturnType; ``` -Defined in: [packages/db/src/query/builder/functions.ts:345](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L345) +Defined in: [packages/db/src/query/builder/functions.ts:349](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L349) ## Type Parameters diff --git a/docs/reference/functions/collectionOptions.md b/docs/reference/functions/collectionOptions.md new file mode 100644 index 0000000000..bb0153b4d3 --- /dev/null +++ b/docs/reference/functions/collectionOptions.md @@ -0,0 +1,162 @@ +--- +id: collectionOptions +title: collectionOptions +--- + +# Function: collectionOptions() + +## Call Signature + +```ts +function collectionOptions(options): CollectionOptions, TKey, T, TUtils> & NonSingleResult; +``` + +Defined in: [packages/db/src/client.ts:180](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L180) + +### Type Parameters + +#### T + +`T` *extends* `StandardSchemaV1`\<`unknown`, `unknown`\> + +#### TKey + +`TKey` *extends* `string` \| `number` + +#### TUtils + +`TUtils` *extends* [`UtilsRecord`](../type-aliases/UtilsRecord.md) + +### Parameters + +#### options + +[`CollectionConfig`](../interfaces/CollectionConfig.md)\<[`InferSchemaOutput`](../type-aliases/InferSchemaOutput.md)\<`T`\>, `TKey`, `T`, `TUtils`\> & `object` & [`NonSingleResult`](../type-aliases/NonSingleResult.md) + +### Returns + +[`CollectionOptions`](../type-aliases/CollectionOptions.md)\<[`InferSchemaOutput`](../type-aliases/InferSchemaOutput.md)\<`T`\>, `TKey`, `T`, `TUtils`\> & [`NonSingleResult`](../type-aliases/NonSingleResult.md) + +## Call Signature + +```ts +function collectionOptions(options): CollectionOptions, TKey, T, TUtils> & SingleResult; +``` + +Defined in: [packages/db/src/client.ts:189](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L189) + +### Type Parameters + +#### T + +`T` *extends* `StandardSchemaV1`\<`unknown`, `unknown`\> + +#### TKey + +`TKey` *extends* `string` \| `number` + +#### TUtils + +`TUtils` *extends* [`UtilsRecord`](../type-aliases/UtilsRecord.md) + +### Parameters + +#### options + +[`CollectionConfig`](../interfaces/CollectionConfig.md)\<[`InferSchemaOutput`](../type-aliases/InferSchemaOutput.md)\<`T`\>, `TKey`, `T`, `TUtils`\> & `object` & [`SingleResult`](../type-aliases/SingleResult.md) + +### Returns + +[`CollectionOptions`](../type-aliases/CollectionOptions.md)\<[`InferSchemaOutput`](../type-aliases/InferSchemaOutput.md)\<`T`\>, `TKey`, `T`, `TUtils`\> & [`SingleResult`](../type-aliases/SingleResult.md) + +## Call Signature + +```ts +function collectionOptions(options): CollectionOptions & NonSingleResult; +``` + +Defined in: [packages/db/src/client.ts:198](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L198) + +### Type Parameters + +#### T + +`T` *extends* `object` + +#### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +#### TUtils + +`TUtils` *extends* [`UtilsRecord`](../type-aliases/UtilsRecord.md) = [`UtilsRecord`](../type-aliases/UtilsRecord.md) + +### Parameters + +#### options + +[`CollectionConfig`](../interfaces/CollectionConfig.md)\<`T`, `TKey`, `never`, `TUtils`\> & `object` & [`NonSingleResult`](../type-aliases/NonSingleResult.md) + +### Returns + +[`CollectionOptions`](../type-aliases/CollectionOptions.md)\<`T`, `TKey`, `never`, `TUtils`\> & [`NonSingleResult`](../type-aliases/NonSingleResult.md) + +## Call Signature + +```ts +function collectionOptions(options): CollectionOptions & SingleResult; +``` + +Defined in: [packages/db/src/client.ts:207](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L207) + +### Type Parameters + +#### T + +`T` *extends* `object` + +#### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +#### TUtils + +`TUtils` *extends* [`UtilsRecord`](../type-aliases/UtilsRecord.md) = [`UtilsRecord`](../type-aliases/UtilsRecord.md) + +### Parameters + +#### options + +[`CollectionConfig`](../interfaces/CollectionConfig.md)\<`T`, `TKey`, `never`, `TUtils`\> & `object` & [`SingleResult`](../type-aliases/SingleResult.md) + +### Returns + +[`CollectionOptions`](../type-aliases/CollectionOptions.md)\<`T`, `TKey`, `never`, `TUtils`\> & [`SingleResult`](../type-aliases/SingleResult.md) + +## Call Signature + +```ts +function collectionOptions(id, factory): DescriptorFromConfig; +``` + +Defined in: [packages/db/src/client.ts:216](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L216) + +### Type Parameters + +#### TConfig + +`TConfig` *extends* `AnyCollectionConfig` + +### Parameters + +#### id + +`string` + +#### factory + +(`client`) => `TConfig` + +### Returns + +`DescriptorFromConfig`\<`TConfig`\> diff --git a/docs/reference/functions/compareLiveQueryWindowDependencies.md b/docs/reference/functions/compareLiveQueryWindowDependencies.md new file mode 100644 index 0000000000..0514989215 --- /dev/null +++ b/docs/reference/functions/compareLiveQueryWindowDependencies.md @@ -0,0 +1,42 @@ +--- +id: compareLiveQueryWindowDependencies +title: compareLiveQueryWindowDependencies +--- + +# Function: compareLiveQueryWindowDependencies() + +```ts +function compareLiveQueryWindowDependencies(previous, current): object; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:432](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L432) + +**`Internal`** + +Compare adapter dependencies by identity and structure. + +## Parameters + +### previous + +readonly `unknown`[] | `null` | `undefined` + +### current + +readonly `unknown`[] + +## Returns + +`object` + +### changed + +```ts +changed: boolean; +``` + +### structurallyEqual + +```ts +structurallyEqual: boolean; +``` diff --git a/docs/reference/functions/compileExpression.md b/docs/reference/functions/compileExpression.md index 1908f7522a..5c6dc813d2 100644 --- a/docs/reference/functions/compileExpression.md +++ b/docs/reference/functions/compileExpression.md @@ -9,7 +9,7 @@ title: compileExpression function compileExpression(expr, isSingleRow): CompiledSingleRowExpression | CompiledExpression; ``` -Defined in: [packages/db/src/query/compiler/evaluators.ts:72](https://github.com/TanStack/db/blob/main/packages/db/src/query/compiler/evaluators.ts#L72) +Defined in: [packages/db/src/query/compiler/evaluators.ts:96](https://github.com/TanStack/db/blob/main/packages/db/src/query/compiler/evaluators.ts#L96) Compiles an expression into an optimized evaluator function. This eliminates branching during evaluation by pre-compiling the expression structure. diff --git a/docs/reference/functions/compileQuery.md b/docs/reference/functions/compileQuery.md index 4f6db2f575..d0895555a7 100644 --- a/docs/reference/functions/compileQuery.md +++ b/docs/reference/functions/compileQuery.md @@ -21,7 +21,7 @@ function compileQuery( childCorrelationField?): CompilationResult; ``` -Defined in: [packages/db/src/query/compiler/index.ts:130](https://github.com/TanStack/db/blob/main/packages/db/src/query/compiler/index.ts#L130) +Defined in: [packages/db/src/query/compiler/index.ts:364](https://github.com/TanStack/db/blob/main/packages/db/src/query/compiler/index.ts#L364) Compiles a query IR into a D2 pipeline @@ -61,13 +61,13 @@ Mapping of source aliases to lazy loading callbacks `Set`\<`string`\> -Set of source aliases that should load data lazily +Set of source identities that should load data lazily ### optimizableOrderByCollections `Record`\<`string`, `OrderByOptimizationInfo`\> -Map of collection IDs to order-by optimization info +Map of source IDs to order-by optimization info ### setWindowFn diff --git a/docs/reference/functions/compileSingleRowExpression.md b/docs/reference/functions/compileSingleRowExpression.md index 19ab88aa27..2b9265fc8f 100644 --- a/docs/reference/functions/compileSingleRowExpression.md +++ b/docs/reference/functions/compileSingleRowExpression.md @@ -9,7 +9,7 @@ title: compileSingleRowExpression function compileSingleRowExpression(expr): CompiledSingleRowExpression; ``` -Defined in: [packages/db/src/query/compiler/evaluators.ts:83](https://github.com/TanStack/db/blob/main/packages/db/src/query/compiler/evaluators.ts#L83) +Defined in: [packages/db/src/query/compiler/evaluators.ts:107](https://github.com/TanStack/db/blob/main/packages/db/src/query/compiler/evaluators.ts#L107) Compiles a single-row expression into an optimized evaluator function. diff --git a/docs/reference/functions/concat.md b/docs/reference/functions/concat.md index ddfaf560e5..8ac8228207 100644 --- a/docs/reference/functions/concat.md +++ b/docs/reference/functions/concat.md @@ -11,7 +11,7 @@ title: concat function concat(arg): ConcatToArrayWrapper; ``` -Defined in: [packages/db/src/query/builder/functions.ts:297](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L297) +Defined in: [packages/db/src/query/builder/functions.ts:301](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L301) ### Type Parameters @@ -35,7 +35,7 @@ Defined in: [packages/db/src/query/builder/functions.ts:297](https://github.com/ function concat(...args): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:300](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L300) +Defined in: [packages/db/src/query/builder/functions.ts:304](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L304) ### Parameters diff --git a/docs/reference/functions/count.md b/docs/reference/functions/count.md index c1a8436e97..f09ebb4134 100644 --- a/docs/reference/functions/count.md +++ b/docs/reference/functions/count.md @@ -9,7 +9,7 @@ title: count function count(arg): Aggregate; ``` -Defined in: [packages/db/src/query/builder/functions.ts:366](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L366) +Defined in: [packages/db/src/query/builder/functions.ts:643](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L643) ## Parameters diff --git a/docs/reference/functions/createArrayChangeProxy.md b/docs/reference/functions/createArrayChangeProxy.md index 79d80a31c1..7311b6b7d2 100644 --- a/docs/reference/functions/createArrayChangeProxy.md +++ b/docs/reference/functions/createArrayChangeProxy.md @@ -9,7 +9,7 @@ title: createArrayChangeProxy function createArrayChangeProxy(targets): object; ``` -Defined in: [packages/db/src/proxy.ts:1130](https://github.com/TanStack/db/blob/main/packages/db/src/proxy.ts#L1130) +Defined in: [packages/db/src/proxy.ts:932](https://github.com/TanStack/db/blob/main/packages/db/src/proxy.ts#L932) Creates proxies for an array of objects and tracks changes to each diff --git a/docs/reference/functions/createChangeProxy.md b/docs/reference/functions/createChangeProxy.md index efc5b50b31..13eff12c48 100644 --- a/docs/reference/functions/createChangeProxy.md +++ b/docs/reference/functions/createChangeProxy.md @@ -9,7 +9,7 @@ title: createChangeProxy function createChangeProxy(target, parent?): object; ``` -Defined in: [packages/db/src/proxy.ts:628](https://github.com/TanStack/db/blob/main/packages/db/src/proxy.ts#L628) +Defined in: [packages/db/src/proxy.ts:451](https://github.com/TanStack/db/blob/main/packages/db/src/proxy.ts#L451) Creates a proxy that tracks changes to the target object @@ -29,15 +29,9 @@ The object to proxy ### parent? -Optional parent information - -#### prop - -`string` \| `symbol` +`ChangeParent` -#### tracker - -`ChangeTracker`\<`Record`\<`string` \| `symbol`, `unknown`\>\> +Optional parent information ## Returns diff --git a/docs/reference/functions/createCollection.md b/docs/reference/functions/createCollection.md index c36372d477..52e541ab75 100644 --- a/docs/reference/functions/createCollection.md +++ b/docs/reference/functions/createCollection.md @@ -11,7 +11,7 @@ title: createCollection function createCollection(options): Collection, TKey, TUtils, T, InferSchemaInput> & NonSingleResult; ``` -Defined in: [packages/db/src/collection/index.ts:140](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L140) +Defined in: [packages/db/src/collection/index.ts:144](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L144) Creates a new Collection instance with the given configuration @@ -120,7 +120,7 @@ const todos = createCollection({ function createCollection(options): Collection, TKey, Exclude, T, InferSchemaInput> & NonSingleResult; ``` -Defined in: [packages/db/src/collection/index.ts:157](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L157) +Defined in: [packages/db/src/collection/index.ts:161](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L161) Creates a new Collection instance with the given configuration @@ -229,7 +229,7 @@ const todos = createCollection({ function createCollection(options): Collection, TKey, TUtils, T, InferSchemaInput> & SingleResult; ``` -Defined in: [packages/db/src/collection/index.ts:175](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L175) +Defined in: [packages/db/src/collection/index.ts:179](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L179) Creates a new Collection instance with the given configuration @@ -338,7 +338,7 @@ const todos = createCollection({ function createCollection(options): Collection, TKey, TUtils, T, InferSchemaInput> & SingleResult; ``` -Defined in: [packages/db/src/collection/index.ts:191](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L191) +Defined in: [packages/db/src/collection/index.ts:195](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L195) Creates a new Collection instance with the given configuration @@ -447,7 +447,7 @@ const todos = createCollection({ function createCollection(options): Collection & NonSingleResult; ``` -Defined in: [packages/db/src/collection/index.ts:204](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L204) +Defined in: [packages/db/src/collection/index.ts:208](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L208) Creates a new Collection instance with the given configuration @@ -556,7 +556,7 @@ const todos = createCollection({ function createCollection(options): Collection & NonSingleResult; ``` -Defined in: [packages/db/src/collection/index.ts:217](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L217) +Defined in: [packages/db/src/collection/index.ts:221](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L221) Creates a new Collection instance with the given configuration @@ -665,7 +665,7 @@ const todos = createCollection({ function createCollection(options): Collection & SingleResult; ``` -Defined in: [packages/db/src/collection/index.ts:229](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L229) +Defined in: [packages/db/src/collection/index.ts:233](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L233) Creates a new Collection instance with the given configuration @@ -774,7 +774,7 @@ const todos = createCollection({ function createCollection(options): Collection & SingleResult; ``` -Defined in: [packages/db/src/collection/index.ts:242](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L242) +Defined in: [packages/db/src/collection/index.ts:246](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L246) Creates a new Collection instance with the given configuration diff --git a/docs/reference/functions/createEffect.md b/docs/reference/functions/createEffect.md index 8ab2ead620..0496c6e663 100644 --- a/docs/reference/functions/createEffect.md +++ b/docs/reference/functions/createEffect.md @@ -9,7 +9,7 @@ title: createEffect function createEffect(config): Effect; ``` -Defined in: [packages/db/src/query/effect.ts:184](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L184) +Defined in: [packages/db/src/query/effect.ts:194](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L194) Creates a reactive effect that fires handlers when rows enter, exit, or update within a query result. Effects process deltas only — they do not diff --git a/docs/reference/functions/createLiveQueryObserver.md b/docs/reference/functions/createLiveQueryObserver.md new file mode 100644 index 0000000000..aef55b6bcd --- /dev/null +++ b/docs/reference/functions/createLiveQueryObserver.md @@ -0,0 +1,46 @@ +--- +id: createLiveQueryObserver +title: createLiveQueryObserver +--- + +# Function: createLiveQueryObserver() + +```ts +function createLiveQueryObserver(collection, options): LiveQueryObserver; +``` + +Defined in: [packages/db/src/live-query-observer.ts:884](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L884) + +**`Internal`** + +Create a [LiveQueryObserver](../interfaces/LiveQueryObserver.md) for a resolved live-query collection, or a +disabled observer when `collection` is `null`/`undefined`. + + This is an unstable contract shared by TanStack DB's official +framework adapters. It is exported so the adapter packages can use it, but +it is not a public extension point yet: its API may change in any release +without a semver major. + +## Type Parameters + +### T + +`T` *extends* `object` + +### TKey + +`TKey` *extends* `string` \| `number` + +## Parameters + +### collection + +[`Collection`](../interfaces/Collection.md)\<`T`, `TKey`, `any`, `StandardSchemaV1`\<`unknown`, `unknown`\>, `T`\> | `null` | `undefined` + +### options + +[`CreateLiveQueryObserverOptions`](../interfaces/CreateLiveQueryObserverOptions.md) = `{}` + +## Returns + +[`LiveQueryObserver`](../interfaces/LiveQueryObserver.md)\<`T`, `TKey`\> diff --git a/docs/reference/functions/createLiveQueryWindowController.md b/docs/reference/functions/createLiveQueryWindowController.md new file mode 100644 index 0000000000..d8d13bedae --- /dev/null +++ b/docs/reference/functions/createLiveQueryWindowController.md @@ -0,0 +1,42 @@ +--- +id: createLiveQueryWindowController +title: createLiveQueryWindowController +--- + +# Function: createLiveQueryWindowController() + +```ts +function createLiveQueryWindowController(collection, options): LiveQueryWindowController; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:1035](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L1035) + +**`Internal`** + +Create an internal forward-window controller for an ordered live query. + + This factory is unstable while RFC #1623 is being implemented. + +## Type Parameters + +### T + +`T` *extends* `object` + +### TKey + +`TKey` *extends* `string` \| `number` + +## Parameters + +### collection + +[`Collection`](../interfaces/Collection.md)\<`T`, `TKey`, `any`, `StandardSchemaV1`\<`unknown`, `unknown`\>, `T`\> | `null` | `undefined` + +### options + +[`CreateLiveQueryWindowControllerOptions`](../interfaces/CreateLiveQueryWindowControllerOptions.md) = `{}` + +## Returns + +[`LiveQueryWindowController`](../interfaces/LiveQueryWindowController.md)\<`T`, `TKey`\> diff --git a/docs/reference/functions/createTransaction.md b/docs/reference/functions/createTransaction.md index e1207af829..9521731769 100644 --- a/docs/reference/functions/createTransaction.md +++ b/docs/reference/functions/createTransaction.md @@ -9,7 +9,7 @@ title: createTransaction function createTransaction(config): Transaction; ``` -Defined in: [packages/db/src/transactions.ts:156](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L156) +Defined in: [packages/db/src/transactions.ts:284](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L284) Creates a new transaction for grouping multiple collection operations diff --git a/docs/reference/functions/divide.md b/docs/reference/functions/divide.md new file mode 100644 index 0000000000..b8424ec323 --- /dev/null +++ b/docs/reference/functions/divide.md @@ -0,0 +1,36 @@ +--- +id: divide +title: divide +--- + +# Function: divide() + +```ts +function divide(left, right): DivideReturnType; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:631](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L631) + +## Type Parameters + +### T1 + +`T1` *extends* `ExpressionLike` + +### T2 + +`T2` *extends* `ExpressionLike` + +## Parameters + +### left + +`T1` + +### right + +`T2` + +## Returns + +`DivideReturnType` diff --git a/docs/reference/functions/eq.md b/docs/reference/functions/eq.md index 794ec946c3..bda6e68399 100644 --- a/docs/reference/functions/eq.md +++ b/docs/reference/functions/eq.md @@ -11,7 +11,7 @@ title: eq function eq(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:133](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L133) +Defined in: [packages/db/src/query/builder/functions.ts:137](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L137) ### Type Parameters @@ -39,7 +39,7 @@ Defined in: [packages/db/src/query/builder/functions.ts:133](https://github.com/ function eq(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:137](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L137) +Defined in: [packages/db/src/query/builder/functions.ts:141](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L141) ### Type Parameters @@ -67,7 +67,7 @@ Defined in: [packages/db/src/query/builder/functions.ts:137](https://github.com/ function eq(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:141](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L141) +Defined in: [packages/db/src/query/builder/functions.ts:145](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L145) ### Type Parameters diff --git a/docs/reference/functions/fetchNextLiveQueryWindowPage.md b/docs/reference/functions/fetchNextLiveQueryWindowPage.md new file mode 100644 index 0000000000..502fbca21e --- /dev/null +++ b/docs/reference/functions/fetchNextLiveQueryWindowPage.md @@ -0,0 +1,30 @@ +--- +id: fetchNextLiveQueryWindowPage +title: fetchNextLiveQueryWindowPage +--- + +# Function: fetchNextLiveQueryWindowPage() + +```ts +function fetchNextLiveQueryWindowPage(controller): Promise; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:535](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L535) + +**`Internal`** + +Run an adapter-facing page fetch. The controller records failures in its +snapshot; consuming the rejection here keeps event handlers safe while the +returned promise still settles with the request. + + This contract is unstable while RFC #1623 is being implemented. + +## Parameters + +### controller + +`Pick`\<[`LiveQueryWindowController`](../interfaces/LiveQueryWindowController.md)\<`object`, `string` \| `number`\>, `"fetchNextPage"`\> + +## Returns + +`Promise`\<`void`\> diff --git a/docs/reference/functions/findIndexForField.md b/docs/reference/functions/findIndexForField.md index aaab37d8ef..0cc1a9e190 100644 --- a/docs/reference/functions/findIndexForField.md +++ b/docs/reference/functions/findIndexForField.md @@ -9,12 +9,10 @@ title: findIndexForField function findIndexForField( collection, fieldPath, - compareOptions?): - | IndexInterface - | undefined; + compareOptions?): IndexReader | undefined; ``` -Defined in: [packages/db/src/utils/index-optimization.ts:37](https://github.com/TanStack/db/blob/main/packages/db/src/utils/index-optimization.ts#L37) +Defined in: [packages/db/src/utils/index-optimization.ts:45](https://github.com/TanStack/db/blob/main/packages/db/src/utils/index-optimization.ts#L45) Finds an index that matches a given field path @@ -40,5 +38,4 @@ Finds an index that matches a given field path ## Returns - \| [`IndexInterface`](../interfaces/IndexInterface.md)\<`TKey`\> - \| `undefined` +[`IndexReader`](../type-aliases/IndexReader.md)\<`TKey`\> \| `undefined` diff --git a/docs/reference/functions/getActiveTransaction.md b/docs/reference/functions/getActiveTransaction.md index b1488c8d08..c1969b51d5 100644 --- a/docs/reference/functions/getActiveTransaction.md +++ b/docs/reference/functions/getActiveTransaction.md @@ -11,7 +11,7 @@ function getActiveTransaction(): | undefined; ``` -Defined in: [packages/db/src/transactions.ts:175](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L175) +Defined in: [packages/db/src/transactions.ts:301](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L301) Gets the currently active ambient transaction, if any Used internally by collection operations to join existing transactions diff --git a/docs/reference/functions/getLiveQueryHash.md b/docs/reference/functions/getLiveQueryHash.md new file mode 100644 index 0000000000..459c9a3d60 --- /dev/null +++ b/docs/reference/functions/getLiveQueryHash.md @@ -0,0 +1,26 @@ +--- +id: getLiveQueryHash +title: getLiveQueryHash +--- + +# Function: getLiveQueryHash() + +```ts +function getLiveQueryHash(preparedValue, queryKey?): string; +``` + +Defined in: [packages/db/src/live-query-options.ts:129](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-options.ts#L129) + +## Parameters + +### preparedValue + +`unknown` + +### queryKey? + +[`LiveQueryKey`](../type-aliases/LiveQueryKey.md) + +## Returns + +`string` diff --git a/docs/reference/functions/getLiveQueryStatusFlags.md b/docs/reference/functions/getLiveQueryStatusFlags.md new file mode 100644 index 0000000000..4dedb0d47f --- /dev/null +++ b/docs/reference/functions/getLiveQueryStatusFlags.md @@ -0,0 +1,26 @@ +--- +id: getLiveQueryStatusFlags +title: getLiveQueryStatusFlags +--- + +# Function: getLiveQueryStatusFlags() + +```ts +function getLiveQueryStatusFlags(status): LiveQueryStatusFlags; +``` + +Defined in: [packages/db/src/live-query-adapter.ts:58](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-adapter.ts#L58) + +Derive the boolean status flags from a collection status. Adapters represent +a disabled query separately (with `isReady: true`); this covers the real +`CollectionStatus` values. + +## Parameters + +### status + +[`CollectionStatus`](../type-aliases/CollectionStatus.md) + +## Returns + +[`LiveQueryStatusFlags`](../interfaces/LiveQueryStatusFlags.md) diff --git a/docs/reference/functions/getLiveQueryWindowCollectionWarning.md b/docs/reference/functions/getLiveQueryWindowCollectionWarning.md new file mode 100644 index 0000000000..09c229a306 --- /dev/null +++ b/docs/reference/functions/getLiveQueryWindowCollectionWarning.md @@ -0,0 +1,33 @@ +--- +id: getLiveQueryWindowCollectionWarning +title: getLiveQueryWindowCollectionWarning +--- + +# Function: getLiveQueryWindowCollectionWarning() + +```ts +function getLiveQueryWindowCollectionWarning(collection, expectedLimit): string | undefined; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:404](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L404) + +**`Internal`** + +Validate a pre-created infinite-query collection and describe any window +adjustment the adapter should warn about. + + Shared validation for infinite-query adapters. + +## Parameters + +### collection + +[`Collection`](../interfaces/Collection.md)\<`any`, `any`, `any`\> + +### expectedLimit + +`number` + +## Returns + +`string` \| `undefined` diff --git a/docs/reference/functions/getLiveQueryWindowInputKind.md b/docs/reference/functions/getLiveQueryWindowInputKind.md new file mode 100644 index 0000000000..9e12f7486d --- /dev/null +++ b/docs/reference/functions/getLiveQueryWindowInputKind.md @@ -0,0 +1,30 @@ +--- +id: getLiveQueryWindowInputKind +title: getLiveQueryWindowInputKind +--- + +# Function: getLiveQueryWindowInputKind() + +```ts +function getLiveQueryWindowInputKind(input): LiveQueryWindowInputKind; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:41](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L41) + +**`Internal`** + +Classify an infinite-query input without invoking its query callback. +Frameworks use this during lifecycle comparison so unchanged React renders +do not execute the callback again. + + This contract is unstable while RFC #1623 is being implemented. + +## Parameters + +### input + +`unknown` + +## Returns + +[`LiveQueryWindowInputKind`](../type-aliases/LiveQueryWindowInputKind.md) diff --git a/docs/reference/functions/getLoadSubsetDemandKey.md b/docs/reference/functions/getLoadSubsetDemandKey.md new file mode 100644 index 0000000000..5e934d72d8 --- /dev/null +++ b/docs/reference/functions/getLoadSubsetDemandKey.md @@ -0,0 +1,31 @@ +--- +id: getLoadSubsetDemandKey +title: getLoadSubsetDemandKey +--- + +# Function: getLoadSubsetDemandKey() + +```ts +function getLoadSubsetDemandKey(options): DemandKey | undefined; +``` + +Defined in: [packages/db/src/query/ir-stable-identity.ts:100](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L100) + +Returns the exact semantic identity of a loadSubset request. + +Abort signals and subscriptions are owners of a request, not part of the +requested data, and therefore do not affect the key. A demand generation +scopes one asynchronous attempt rather than the data it requests. Code that +rejects stale work compares this key alongside its generation; query-db uses +the key alone so equivalent data demands can reuse one cache entry across +generations. + +## Parameters + +### options + +[`LoadSubsetOptions`](../type-aliases/LoadSubsetOptions.md) + +## Returns + +[`DemandKey`](../type-aliases/DemandKey.md) \| `undefined` diff --git a/docs/reference/functions/getPreparedLiveQueryIdentity.md b/docs/reference/functions/getPreparedLiveQueryIdentity.md new file mode 100644 index 0000000000..eeb3116dac --- /dev/null +++ b/docs/reference/functions/getPreparedLiveQueryIdentity.md @@ -0,0 +1,22 @@ +--- +id: getPreparedLiveQueryIdentity +title: getPreparedLiveQueryIdentity +--- + +# Function: getPreparedLiveQueryIdentity() + +```ts +function getPreparedLiveQueryIdentity(value): unknown; +``` + +Defined in: [packages/db/src/live-query-options.ts:109](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-options.ts#L109) + +## Parameters + +### value + +`unknown` + +## Returns + +`unknown` diff --git a/docs/reference/functions/getQueryIdentity.md b/docs/reference/functions/getQueryIdentity.md new file mode 100644 index 0000000000..e9e724b8dd --- /dev/null +++ b/docs/reference/functions/getQueryIdentity.md @@ -0,0 +1,29 @@ +--- +id: getQueryIdentity +title: getQueryIdentity +--- + +# Function: getQueryIdentity() + +```ts +function getQueryIdentity(query): QueryIdentity; +``` + +Defined in: [packages/db/src/query/ir-stable-identity.ts:86](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L86) + +Returns the semantic identity of a structured query. + +Logical conjunctions and disjunctions are associative, commutative, and +idempotent. Equality operands are commutative, while reversed inequalities +are normalized by inverting their operator. Order-sensitive clauses and +function arguments retain their original order. + +## Parameters + +### query + +[`QueryIR`](../@tanstack/namespaces/IR/interfaces/QueryIR.md) + +## Returns + +[`QueryIdentity`](../type-aliases/QueryIdentity.md) diff --git a/docs/reference/functions/getStableQueryBuilderHash.md b/docs/reference/functions/getStableQueryBuilderHash.md new file mode 100644 index 0000000000..9c8fd23f96 --- /dev/null +++ b/docs/reference/functions/getStableQueryBuilderHash.md @@ -0,0 +1,22 @@ +--- +id: getStableQueryBuilderHash +title: getStableQueryBuilderHash +--- + +# Function: getStableQueryBuilderHash() + +```ts +function getStableQueryBuilderHash(query): string; +``` + +Defined in: [packages/db/src/query/ir-stable-identity.ts:68](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L68) + +## Parameters + +### query + +[`QueryBuilder`](../type-aliases/QueryBuilder.md)\<`any`\> | [`InitialQueryBuilder`](../type-aliases/InitialQueryBuilder.md) + +## Returns + +`string` diff --git a/docs/reference/functions/getStableQueryIRHash.md b/docs/reference/functions/getStableQueryIRHash.md new file mode 100644 index 0000000000..60a0d4489a --- /dev/null +++ b/docs/reference/functions/getStableQueryIRHash.md @@ -0,0 +1,22 @@ +--- +id: getStableQueryIRHash +title: getStableQueryIRHash +--- + +# Function: getStableQueryIRHash() + +```ts +function getStableQueryIRHash(query): string; +``` + +Defined in: [packages/db/src/query/ir-stable-identity.ts:64](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L64) + +## Parameters + +### query + +[`QueryIR`](../@tanstack/namespaces/IR/interfaces/QueryIR.md) + +## Returns + +`string` diff --git a/docs/reference/functions/getStableValueHash.md b/docs/reference/functions/getStableValueHash.md new file mode 100644 index 0000000000..ff48ce7250 --- /dev/null +++ b/docs/reference/functions/getStableValueHash.md @@ -0,0 +1,26 @@ +--- +id: getStableValueHash +title: getStableValueHash +--- + +# Function: getStableValueHash() + +```ts +function getStableValueHash(value, path): string; +``` + +Defined in: [packages/db/src/query/ir-stable-identity.ts:74](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L74) + +## Parameters + +### value + +`unknown` + +### path + +`string` = `...` + +## Returns + +`string` diff --git a/docs/reference/functions/gt.md b/docs/reference/functions/gt.md index 42f0ff1883..5e7dd5cc89 100644 --- a/docs/reference/functions/gt.md +++ b/docs/reference/functions/gt.md @@ -11,7 +11,7 @@ title: gt function gt(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:146](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L146) +Defined in: [packages/db/src/query/builder/functions.ts:150](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L150) ### Type Parameters @@ -39,7 +39,7 @@ Defined in: [packages/db/src/query/builder/functions.ts:146](https://github.com/ function gt(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:150](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L150) +Defined in: [packages/db/src/query/builder/functions.ts:154](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L154) ### Type Parameters @@ -67,7 +67,7 @@ Defined in: [packages/db/src/query/builder/functions.ts:150](https://github.com/ function gt(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:154](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L154) +Defined in: [packages/db/src/query/builder/functions.ts:158](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L158) ### Type Parameters diff --git a/docs/reference/functions/gte.md b/docs/reference/functions/gte.md index 18aecd8c67..efcba19155 100644 --- a/docs/reference/functions/gte.md +++ b/docs/reference/functions/gte.md @@ -11,7 +11,7 @@ title: gte function gte(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:159](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L159) +Defined in: [packages/db/src/query/builder/functions.ts:163](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L163) ### Type Parameters @@ -39,7 +39,7 @@ Defined in: [packages/db/src/query/builder/functions.ts:159](https://github.com/ function gte(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:163](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L163) +Defined in: [packages/db/src/query/builder/functions.ts:167](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L167) ### Type Parameters @@ -67,7 +67,7 @@ Defined in: [packages/db/src/query/builder/functions.ts:163](https://github.com/ function gte(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:167](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L167) +Defined in: [packages/db/src/query/builder/functions.ts:171](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L171) ### Type Parameters diff --git a/docs/reference/functions/hasLiveQueryWindowLeases.md b/docs/reference/functions/hasLiveQueryWindowLeases.md new file mode 100644 index 0000000000..da315ed117 --- /dev/null +++ b/docs/reference/functions/hasLiveQueryWindowLeases.md @@ -0,0 +1,26 @@ +--- +id: hasLiveQueryWindowLeases +title: hasLiveQueryWindowLeases +--- + +# Function: hasLiveQueryWindowLeases() + +```ts +function hasLiveQueryWindowLeases(target): boolean; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:373](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L373) + +**`Internal`** + +Whether an infinite-query controller currently owns this window. + +## Parameters + +### target + +`object` + +## Returns + +`boolean` diff --git a/docs/reference/functions/hasVirtualProps.md b/docs/reference/functions/hasVirtualProps.md index 57a2b9c614..19013aac4e 100644 --- a/docs/reference/functions/hasVirtualProps.md +++ b/docs/reference/functions/hasVirtualProps.md @@ -9,7 +9,7 @@ title: hasVirtualProps function hasVirtualProps(value): value is VirtualRowProps; ``` -Defined in: [packages/db/src/virtual-props.ts:145](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L145) +Defined in: [packages/db/src/virtual-props.ts:150](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L150) Checks if a value has virtual properties attached. diff --git a/docs/reference/functions/ilike.md b/docs/reference/functions/ilike.md index faf4f82371..4d51513275 100644 --- a/docs/reference/functions/ilike.md +++ b/docs/reference/functions/ilike.md @@ -9,7 +9,7 @@ title: ilike function ilike(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:270](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L270) +Defined in: [packages/db/src/query/builder/functions.ts:274](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L274) ## Parameters diff --git a/docs/reference/functions/inArray.md b/docs/reference/functions/inArray.md index 6a8d545296..c6ed1471b9 100644 --- a/docs/reference/functions/inArray.md +++ b/docs/reference/functions/inArray.md @@ -9,7 +9,7 @@ title: inArray function inArray(value, array): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:255](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L255) +Defined in: [packages/db/src/query/builder/functions.ts:259](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L259) ## Parameters diff --git a/docs/reference/functions/isCollection.md b/docs/reference/functions/isCollection.md new file mode 100644 index 0000000000..a846ecfa33 --- /dev/null +++ b/docs/reference/functions/isCollection.md @@ -0,0 +1,29 @@ +--- +id: isCollection +title: isCollection +--- + +# Function: isCollection() + +```ts +function isCollection(value): value is Collection, any>; +``` + +Defined in: [packages/db/src/live-query-adapter.ts:22](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-adapter.ts#L22) + +Structural check for a live-query/`Collection` instance. + +Uses duck typing rather than `instanceof CollectionImpl` on purpose: adapters +and core can resolve to different copies of `@tanstack/db` (dual-package / +multi-realm), where `instanceof` gives false negatives. The three methods +below uniquely identify a Collection. + +## Parameters + +### value + +`unknown` + +## Returns + +`value is Collection, any>` diff --git a/docs/reference/functions/isCollectionOptions.md b/docs/reference/functions/isCollectionOptions.md new file mode 100644 index 0000000000..4cd34a660c --- /dev/null +++ b/docs/reference/functions/isCollectionOptions.md @@ -0,0 +1,22 @@ +--- +id: isCollectionOptions +title: isCollectionOptions +--- + +# Function: isCollectionOptions() + +```ts +function isCollectionOptions(value): value is CollectionOptions; +``` + +Defined in: [packages/db/src/client.ts:302](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L302) + +## Parameters + +### value + +`unknown` + +## Returns + +value is CollectionOptions\ diff --git a/docs/reference/functions/isLimitSubset.md b/docs/reference/functions/isLimitSubset.md deleted file mode 100644 index eb2b6ab3f8..0000000000 --- a/docs/reference/functions/isLimitSubset.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -id: isLimitSubset -title: isLimitSubset ---- - -# Function: isLimitSubset() - -```ts -function isLimitSubset(subset, superset): boolean; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:773](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L773) - -Check if one limit is a subset of another. -Returns true if the subset limit requirements are satisfied by the superset limit. - -Note: This function does NOT consider offset. For offset-aware subset checking, -use `isOffsetLimitSubset` instead. - -## Parameters - -### subset - -The limit requirement to check - -`number` | `undefined` - -### superset - -The limit that might satisfy the requirement - -`number` | `undefined` - -## Returns - -`boolean` - -true if subset is satisfied by superset - -## Example - -```ts -isLimitSubset(10, 20) // true (requesting 10 items when 20 are available) -isLimitSubset(20, 10) // false (requesting 20 items when only 10 are available) -isLimitSubset(10, undefined) // true (requesting 10 items when unlimited are available) -``` diff --git a/docs/reference/functions/isLiveQueryWindowCollection.md b/docs/reference/functions/isLiveQueryWindowCollection.md new file mode 100644 index 0000000000..58d9614625 --- /dev/null +++ b/docs/reference/functions/isLiveQueryWindowCollection.md @@ -0,0 +1,26 @@ +--- +id: isLiveQueryWindowCollection +title: isLiveQueryWindowCollection +--- + +# Function: isLiveQueryWindowCollection() + +```ts +function isLiveQueryWindowCollection(collection): collection is LiveQueryWindowCollection; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:389](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L389) + +**`Internal`** + +Whether a collection exposes an active ordered window. + +## Parameters + +### collection + +[`Collection`](../interfaces/Collection.md)\<`any`, `any`, `any`\> + +## Returns + +`collection is LiveQueryWindowCollection` diff --git a/docs/reference/functions/isNull.md b/docs/reference/functions/isNull.md index cf303fc635..fbd75223ca 100644 --- a/docs/reference/functions/isNull.md +++ b/docs/reference/functions/isNull.md @@ -9,7 +9,7 @@ title: isNull function isNull(value): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:251](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L251) +Defined in: [packages/db/src/query/builder/functions.ts:255](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L255) ## Parameters diff --git a/docs/reference/functions/isOffsetLimitSubset.md b/docs/reference/functions/isOffsetLimitSubset.md deleted file mode 100644 index a3d63e97b5..0000000000 --- a/docs/reference/functions/isOffsetLimitSubset.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -id: isOffsetLimitSubset -title: isOffsetLimitSubset ---- - -# Function: isOffsetLimitSubset() - -```ts -function isOffsetLimitSubset(subset, superset): boolean; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:813](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L813) - -Check if one offset+limit range is a subset of another. -Returns true if the subset range is fully contained within the superset range. - -A query with `{limit: 10, offset: 0}` loads rows [0, 10). -A query with `{limit: 10, offset: 20}` loads rows [20, 30). - -For subset to be satisfied by superset: -- Superset must start at or before subset (superset.offset <= subset.offset) -- Superset must end at or after subset (superset.offset + superset.limit >= subset.offset + subset.limit) - -## Parameters - -### subset - -The offset+limit requirements to check - -#### limit? - -`number` - -#### offset? - -`number` - -### superset - -The offset+limit that might satisfy the requirements - -#### limit? - -`number` - -#### offset? - -`number` - -## Returns - -`boolean` - -true if subset range is fully contained within superset range - -## Example - -```ts -isOffsetLimitSubset({ offset: 0, limit: 5 }, { offset: 0, limit: 10 }) // true -isOffsetLimitSubset({ offset: 5, limit: 5 }, { offset: 0, limit: 10 }) // true (rows 5-9 within 0-9) -isOffsetLimitSubset({ offset: 5, limit: 10 }, { offset: 0, limit: 10 }) // false (rows 5-14 exceed 0-9) -isOffsetLimitSubset({ offset: 20, limit: 10 }, { offset: 0, limit: 10 }) // false (rows 20-29 outside 0-9) -``` diff --git a/docs/reference/functions/isOrderBySubset.md b/docs/reference/functions/isOrderBySubset.md deleted file mode 100644 index f6eb48bf8c..0000000000 --- a/docs/reference/functions/isOrderBySubset.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -id: isOrderBySubset -title: isOrderBySubset ---- - -# Function: isOrderBySubset() - -```ts -function isOrderBySubset(subset, superset): boolean; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:715](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L715) - -Check if one orderBy clause is a subset of another. -Returns true if the subset ordering requirements are satisfied by the superset ordering. - -## Parameters - -### subset - -The ordering requirements to check - -[`OrderBy`](../@tanstack/namespaces/IR/type-aliases/OrderBy.md) | `undefined` - -### superset - -The ordering that might satisfy the requirements - -[`OrderBy`](../@tanstack/namespaces/IR/type-aliases/OrderBy.md) | `undefined` - -## Returns - -`boolean` - -true if subset is satisfied by superset - -## Example - -```ts -// Subset is prefix of superset -isOrderBySubset([{expr: age, asc}], [{expr: age, asc}, {expr: name, desc}]) // true -``` diff --git a/docs/reference/functions/isPredicateSubset.md b/docs/reference/functions/isPredicateSubset.md deleted file mode 100644 index 9574ec9c25..0000000000 --- a/docs/reference/functions/isPredicateSubset.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -id: isPredicateSubset -title: isPredicateSubset ---- - -# Function: isPredicateSubset() - -```ts -function isPredicateSubset(subset, superset): boolean; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:856](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L856) - -Check if one predicate (where + orderBy + limit + offset) is a subset of another. -Returns true if all aspects of the subset predicate are satisfied by the superset. - -## Parameters - -### subset - -[`LoadSubsetOptions`](../type-aliases/LoadSubsetOptions.md) - -The predicate requirements to check - -### superset - -[`LoadSubsetOptions`](../type-aliases/LoadSubsetOptions.md) - -The predicate that might satisfy the requirements - -## Returns - -`boolean` - -true if subset is satisfied by superset - -## Example - -```ts -isPredicateSubset( - { where: gt(ref('age'), val(20)), limit: 10 }, - { where: gt(ref('age'), val(10)), limit: 20 } -) // true -``` diff --git a/docs/reference/functions/isSingleResultCollection.md b/docs/reference/functions/isSingleResultCollection.md new file mode 100644 index 0000000000..1ec948a6d7 --- /dev/null +++ b/docs/reference/functions/isSingleResultCollection.md @@ -0,0 +1,24 @@ +--- +id: isSingleResultCollection +title: isSingleResultCollection +--- + +# Function: isSingleResultCollection() + +```ts +function isSingleResultCollection(collection): boolean; +``` + +Defined in: [packages/db/src/live-query-adapter.ts:35](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-adapter.ts#L35) + +Whether a collection yields a single result (`findOne`) rather than an array. + +## Parameters + +### collection + +[`Collection`](../interfaces/Collection.md)\<`any`, `any`, `any`\> + +## Returns + +`boolean` diff --git a/docs/reference/functions/isUndefined.md b/docs/reference/functions/isUndefined.md index 49d99d535b..5444e21830 100644 --- a/docs/reference/functions/isUndefined.md +++ b/docs/reference/functions/isUndefined.md @@ -9,7 +9,7 @@ title: isUndefined function isUndefined(value): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:247](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L247) +Defined in: [packages/db/src/query/builder/functions.ts:251](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L251) ## Parameters diff --git a/docs/reference/functions/isWhereSubset.md b/docs/reference/functions/isWhereSubset.md deleted file mode 100644 index 4817c778e2..0000000000 --- a/docs/reference/functions/isWhereSubset.md +++ /dev/null @@ -1,47 +0,0 @@ ---- -id: isWhereSubset -title: isWhereSubset ---- - -# Function: isWhereSubset() - -```ts -function isWhereSubset(subset, superset): boolean; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:21](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L21) - -Check if one where clause is a logical subset of another. -Returns true if the subset predicate is more restrictive than (or equal to) the superset predicate. - -## Parameters - -### subset - -The potentially more restrictive predicate - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> | `undefined` - -### superset - -The potentially less restrictive predicate - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> | `undefined` - -## Returns - -`boolean` - -true if subset logically implies superset - -## Examples - -```ts -// age > 20 is subset of age > 10 (more restrictive) -isWhereSubset(gt(ref('age'), val(20)), gt(ref('age'), val(10))) // true -``` - -```ts -// age > 10 AND name = 'X' is subset of age > 10 (more conditions) -isWhereSubset(and(gt(ref('age'), val(10)), eq(ref('name'), val('X'))), gt(ref('age'), val(10))) // true -``` diff --git a/docs/reference/functions/length.md b/docs/reference/functions/length.md index d452d3a863..1586545934 100644 --- a/docs/reference/functions/length.md +++ b/docs/reference/functions/length.md @@ -9,7 +9,7 @@ title: length function length(arg): NumericFunctionReturnType; ``` -Defined in: [packages/db/src/query/builder/functions.ts:291](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L291) +Defined in: [packages/db/src/query/builder/functions.ts:295](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L295) ## Type Parameters diff --git a/docs/reference/functions/like.md b/docs/reference/functions/like.md index a05fa13226..ee6f72f03e 100644 --- a/docs/reference/functions/like.md +++ b/docs/reference/functions/like.md @@ -9,7 +9,7 @@ title: like function like(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:262](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L262) +Defined in: [packages/db/src/query/builder/functions.ts:266](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L266) ## Parameters diff --git a/docs/reference/functions/localOnlyCollectionOptions.md b/docs/reference/functions/localOnlyCollectionOptions.md index 41c22862b3..e98c60701c 100644 --- a/docs/reference/functions/localOnlyCollectionOptions.md +++ b/docs/reference/functions/localOnlyCollectionOptions.md @@ -11,7 +11,7 @@ title: localOnlyCollectionOptions function localOnlyCollectionOptions(config): CollectionConfig, TKey, T, LocalOnlyCollectionUtils> & object & object; ``` -Defined in: [packages/db/src/local-only.ts:149](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L149) +Defined in: [packages/db/src/local-only.ts:151](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L151) Creates Local-only collection options for use with a standard Collection @@ -123,7 +123,7 @@ await tx.commit() function localOnlyCollectionOptions(config): CollectionConfig & object & object; ``` -Defined in: [packages/db/src/local-only.ts:162](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L162) +Defined in: [packages/db/src/local-only.ts:164](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L164) Creates Local-only collection options for use with a standard Collection diff --git a/docs/reference/functions/localStorageCollectionOptions.md b/docs/reference/functions/localStorageCollectionOptions.md index 6ea5ecad08..379832aad0 100644 --- a/docs/reference/functions/localStorageCollectionOptions.md +++ b/docs/reference/functions/localStorageCollectionOptions.md @@ -11,7 +11,7 @@ title: localStorageCollectionOptions function localStorageCollectionOptions(config): CollectionConfig, TKey, T, LocalStorageCollectionUtils> & object; ``` -Defined in: [packages/db/src/local-storage.ts:316](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L316) +Defined in: [packages/db/src/local-storage.ts:318](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L318) Creates localStorage collection options for use with a standard Collection @@ -126,7 +126,7 @@ await tx.commit() function localStorageCollectionOptions(config): CollectionConfig & object; ``` -Defined in: [packages/db/src/local-storage.ts:336](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L336) +Defined in: [packages/db/src/local-storage.ts:338](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L338) Creates localStorage collection options for use with a standard Collection diff --git a/docs/reference/functions/lower.md b/docs/reference/functions/lower.md index fff4fc0830..f9739693e7 100644 --- a/docs/reference/functions/lower.md +++ b/docs/reference/functions/lower.md @@ -9,7 +9,7 @@ title: lower function lower(arg): StringFunctionReturnType; ``` -Defined in: [packages/db/src/query/builder/functions.ts:285](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L285) +Defined in: [packages/db/src/query/builder/functions.ts:289](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L289) ## Type Parameters diff --git a/docs/reference/functions/lt.md b/docs/reference/functions/lt.md index 7c204f4a93..e103058e7d 100644 --- a/docs/reference/functions/lt.md +++ b/docs/reference/functions/lt.md @@ -11,7 +11,7 @@ title: lt function lt(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:172](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L172) +Defined in: [packages/db/src/query/builder/functions.ts:176](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L176) ### Type Parameters @@ -39,7 +39,7 @@ Defined in: [packages/db/src/query/builder/functions.ts:172](https://github.com/ function lt(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:176](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L176) +Defined in: [packages/db/src/query/builder/functions.ts:180](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L180) ### Type Parameters @@ -67,7 +67,7 @@ Defined in: [packages/db/src/query/builder/functions.ts:176](https://github.com/ function lt(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:180](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L180) +Defined in: [packages/db/src/query/builder/functions.ts:184](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L184) ### Type Parameters diff --git a/docs/reference/functions/lte.md b/docs/reference/functions/lte.md index 2b2b5423cc..01fc872a0e 100644 --- a/docs/reference/functions/lte.md +++ b/docs/reference/functions/lte.md @@ -11,7 +11,7 @@ title: lte function lte(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:185](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L185) +Defined in: [packages/db/src/query/builder/functions.ts:189](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L189) ### Type Parameters @@ -39,7 +39,7 @@ Defined in: [packages/db/src/query/builder/functions.ts:185](https://github.com/ function lte(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:189](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L189) +Defined in: [packages/db/src/query/builder/functions.ts:193](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L193) ### Type Parameters @@ -67,7 +67,7 @@ Defined in: [packages/db/src/query/builder/functions.ts:189](https://github.com/ function lte(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:193](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L193) +Defined in: [packages/db/src/query/builder/functions.ts:197](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L197) ### Type Parameters diff --git a/docs/reference/functions/materialize.md b/docs/reference/functions/materialize.md new file mode 100644 index 0000000000..8f80a39eac --- /dev/null +++ b/docs/reference/functions/materialize.md @@ -0,0 +1,58 @@ +--- +id: materialize +title: materialize +--- + +# Function: materialize() + +```ts +function materialize(query): MaterializeWrapper, TContext extends SingleResult ? true : false>; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:848](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L848) + +Materialize an includes subquery into a plain value on the parent row. + +- For multi-row subqueries, the parent receives an `Array` snapshot + (equivalent to `toArray()`). +- For `findOne()` subqueries, the parent receives a single `T | undefined` + value — `undefined` when no child matches. + +The snapshot updates reactively: parent rows re-emit when the underlying +children change. + +## Type Parameters + +### TContext + +`TContext` *extends* [`Context`](../interfaces/Context.md) + +## Parameters + +### query + +[`QueryBuilder`](../type-aliases/QueryBuilder.md)\<`TContext`\> + +## Returns + +`MaterializeWrapper`\<`GetRawResult`\<`TContext`\>, `TContext` *extends* [`SingleResult`](../type-aliases/SingleResult.md) ? `true` : `false`\> + +## Example + +```ts +// Multi-row: produces Array on each project +select(({ p }) => ({ + ...p, + issues: materialize( + q.from({ i: issues }).where(({ i }) => eq(i.projectId, p.id)), + ), +})) + +// Singleton: produces Author | undefined on each post +select(({ p }) => ({ + ...p, + author: materialize( + q.from({ a: authors }).where(({ a }) => eq(a.id, p.authorId)).findOne(), + ), +})) +``` diff --git a/docs/reference/functions/max.md b/docs/reference/functions/max.md index 008439a7d1..6d04ccf084 100644 --- a/docs/reference/functions/max.md +++ b/docs/reference/functions/max.md @@ -9,7 +9,7 @@ title: max function max(arg): AggregateReturnType; ``` -Defined in: [packages/db/src/query/builder/functions.ts:382](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L382) +Defined in: [packages/db/src/query/builder/functions.ts:659](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L659) ## Type Parameters diff --git a/docs/reference/functions/min.md b/docs/reference/functions/min.md index 8c5788295b..f470eef662 100644 --- a/docs/reference/functions/min.md +++ b/docs/reference/functions/min.md @@ -9,7 +9,7 @@ title: min function min(arg): AggregateReturnType; ``` -Defined in: [packages/db/src/query/builder/functions.ts:378](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L378) +Defined in: [packages/db/src/query/builder/functions.ts:655](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L655) ## Type Parameters diff --git a/docs/reference/functions/minusWherePredicates.md b/docs/reference/functions/minusWherePredicates.md deleted file mode 100644 index 5326ec322b..0000000000 --- a/docs/reference/functions/minusWherePredicates.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -id: minusWherePredicates -title: minusWherePredicates ---- - -# Function: minusWherePredicates() - -```ts -function minusWherePredicates(fromPredicate, subtractPredicate): - | BasicExpression - | null; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:340](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L340) - -Compute the difference between two where predicates: `fromPredicate AND NOT(subtractPredicate)`. -Returns the simplified predicate, or null if the difference cannot be simplified -(in which case the caller should fetch the full fromPredicate). - -## Parameters - -### fromPredicate - -The predicate to subtract from - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> | `undefined` - -### subtractPredicate - -The predicate to subtract - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> | `undefined` - -## Returns - - \| [`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> - \| `null` - -The simplified difference, or null if cannot be simplified - -## Examples - -```ts -// Range difference -minusWherePredicates( - gt(ref('age'), val(10)), // age > 10 - gt(ref('age'), val(20)) // age > 20 -) // → age > 10 AND age <= 20 -``` - -```ts -// Set difference -minusWherePredicates( - inOp(ref('status'), ['A', 'B', 'C', 'D']), // status IN ['A','B','C','D'] - inOp(ref('status'), ['B', 'C']) // status IN ['B','C'] -) // → status IN ['A', 'D'] -``` - -```ts -// Common conditions -minusWherePredicates( - and(gt(ref('age'), val(10)), eq(ref('status'), val('active'))), // age > 10 AND status = 'active' - and(gt(ref('age'), val(20)), eq(ref('status'), val('active'))) // age > 20 AND status = 'active' -) // → age > 10 AND age <= 20 AND status = 'active' -``` - -```ts -// Complete overlap - empty result -minusWherePredicates( - gt(ref('age'), val(20)), // age > 20 - gt(ref('age'), val(10)) // age > 10 -) // → {type: 'val', value: false} (empty set) -``` diff --git a/docs/reference/functions/multiply.md b/docs/reference/functions/multiply.md new file mode 100644 index 0000000000..868c04b5da --- /dev/null +++ b/docs/reference/functions/multiply.md @@ -0,0 +1,36 @@ +--- +id: multiply +title: multiply +--- + +# Function: multiply() + +```ts +function multiply(left, right): BinaryNumericReturnType; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:621](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L621) + +## Type Parameters + +### T1 + +`T1` *extends* `ExpressionLike` + +### T2 + +`T2` *extends* `ExpressionLike` + +## Parameters + +### left + +`T1` + +### right + +`T2` + +## Returns + +`BinaryNumericReturnType` diff --git a/docs/reference/functions/normalizeLiveQueryWindowPageSize.md b/docs/reference/functions/normalizeLiveQueryWindowPageSize.md new file mode 100644 index 0000000000..436d8b8f09 --- /dev/null +++ b/docs/reference/functions/normalizeLiveQueryWindowPageSize.md @@ -0,0 +1,26 @@ +--- +id: normalizeLiveQueryWindowPageSize +title: normalizeLiveQueryWindowPageSize +--- + +# Function: normalizeLiveQueryWindowPageSize() + +```ts +function normalizeLiveQueryWindowPageSize(pageSize): number; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:90](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L90) + +**`Internal`** + +This contract is unstable while RFC #1623 is being implemented. + +## Parameters + +### pageSize + +`number` | `undefined` + +## Returns + +`number` diff --git a/docs/reference/functions/not.md b/docs/reference/functions/not.md index e6020ac2de..9b0c6b2164 100644 --- a/docs/reference/functions/not.md +++ b/docs/reference/functions/not.md @@ -9,7 +9,7 @@ title: not function not(value): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:242](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L242) +Defined in: [packages/db/src/query/builder/functions.ts:246](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L246) ## Parameters diff --git a/docs/reference/functions/optimizeExpressionWithIndexes.md b/docs/reference/functions/optimizeExpressionWithIndexes.md index 31d06210a8..4ae6c49544 100644 --- a/docs/reference/functions/optimizeExpressionWithIndexes.md +++ b/docs/reference/functions/optimizeExpressionWithIndexes.md @@ -9,7 +9,7 @@ title: optimizeExpressionWithIndexes function optimizeExpressionWithIndexes(expression, collection): OptimizationResult; ``` -Defined in: [packages/db/src/utils/index-optimization.ts:100](https://github.com/TanStack/db/blob/main/packages/db/src/utils/index-optimization.ts#L100) +Defined in: [packages/db/src/utils/index-optimization.ts:199](https://github.com/TanStack/db/blob/main/packages/db/src/utils/index-optimization.ts#L199) Optimizes a query expression using available indexes to find matching keys diff --git a/docs/reference/functions/or.md b/docs/reference/functions/or.md index 95adf2ff2e..73b058e122 100644 --- a/docs/reference/functions/or.md +++ b/docs/reference/functions/or.md @@ -11,7 +11,7 @@ title: or function or(left, right): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:221](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L221) +Defined in: [packages/db/src/query/builder/functions.ts:225](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L225) ### Parameters @@ -36,7 +36,7 @@ function or( rest): BasicExpression; ``` -Defined in: [packages/db/src/query/builder/functions.ts:225](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L225) +Defined in: [packages/db/src/query/builder/functions.ts:229](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L229) ### Parameters diff --git a/docs/reference/functions/prepareLiveQueryValue.md b/docs/reference/functions/prepareLiveQueryValue.md new file mode 100644 index 0000000000..fe658d79b0 --- /dev/null +++ b/docs/reference/functions/prepareLiveQueryValue.md @@ -0,0 +1,33 @@ +--- +id: prepareLiveQueryValue +title: prepareLiveQueryValue +--- + +# Function: prepareLiveQueryValue() + +```ts +function prepareLiveQueryValue( + value, + dbClient, + deferredCollections): unknown; +``` + +Defined in: [packages/db/src/live-query-options.ts:64](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-options.ts#L64) + +## Parameters + +### value + +`unknown` + +### dbClient + +[`DbClient`](../classes/DbClient.md) | `undefined` + +### deferredCollections + +[`DeferredLiveQueryCollections`](../type-aliases/DeferredLiveQueryCollections.md) + +## Returns + +`unknown` diff --git a/docs/reference/functions/resolveLiveQueryWindowInput.md b/docs/reference/functions/resolveLiveQueryWindowInput.md new file mode 100644 index 0000000000..6ad01eb8b2 --- /dev/null +++ b/docs/reference/functions/resolveLiveQueryWindowInput.md @@ -0,0 +1,36 @@ +--- +id: resolveLiveQueryWindowInput +title: resolveLiveQueryWindowInput +--- + +# Function: resolveLiveQueryWindowInput() + +```ts +function resolveLiveQueryWindowInput(input): ResolvedLiveQueryWindowInput; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:59](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L59) + +**`Internal`** + +Resolve a supported infinite-query input and invoke a query callback once. +A function may resolve to a collection for framework getter compatibility. +Nullable/disabled and config-object inputs are intentionally not supported. + + This contract is unstable while RFC #1623 is being implemented. + +## Type Parameters + +### TContext + +`TContext` *extends* [`Context`](../interfaces/Context.md) + +## Parameters + +### input + +`unknown` + +## Returns + +[`ResolvedLiveQueryWindowInput`](../type-aliases/ResolvedLiveQueryWindowInput.md)\<`TContext`\> diff --git a/docs/reference/functions/safeRandomUUID.md b/docs/reference/functions/safeRandomUUID.md new file mode 100644 index 0000000000..145a79caaf --- /dev/null +++ b/docs/reference/functions/safeRandomUUID.md @@ -0,0 +1,25 @@ +--- +id: safeRandomUUID +title: safeRandomUUID +--- + +# Function: safeRandomUUID() + +```ts +function safeRandomUUID(): string; +``` + +Defined in: [packages/db/src/utils/uuid.ts:11](https://github.com/TanStack/db/blob/main/packages/db/src/utils/uuid.ts#L11) + +Returns a RFC 4122 version 4 UUID. + +Prefers `crypto.randomUUID()` when available. In non-secure browser contexts +(e.g. a dev server accessed via a LAN IP over HTTP) `crypto.randomUUID` is +`undefined`, so this falls back to building a UUIDv4 from +`crypto.getRandomValues`. Throws if neither API is available. + +See https://github.com/TanStack/db/issues/1541. + +## Returns + +`string` diff --git a/docs/reference/functions/shouldPreserveLiveQueryWindowPageCount.md b/docs/reference/functions/shouldPreserveLiveQueryWindowPageCount.md new file mode 100644 index 0000000000..03fc36c0bb --- /dev/null +++ b/docs/reference/functions/shouldPreserveLiveQueryWindowPageCount.md @@ -0,0 +1,52 @@ +--- +id: shouldPreserveLiveQueryWindowPageCount +title: shouldPreserveLiveQueryWindowPageCount +--- + +# Function: shouldPreserveLiveQueryWindowPageCount() + +```ts +function shouldPreserveLiveQueryWindowPageCount(options): boolean; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:451](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L451) + +**`Internal`** + +Shared page-depth preservation policy for framework adapters. + +## Parameters + +### options + +#### dependenciesChanged + +`boolean` + +#### dependenciesStructurallyEqual + +`boolean` + +#### hasPreviousController + +`boolean` + +#### inputKind + +`"collection"` \| `"query"` + +#### pageShapeChanged + +`boolean` + +#### previousInputKind + +`"collection"` \| `"query"` \| `undefined` + +#### sameCollection + +`boolean` + +## Returns + +`boolean` diff --git a/docs/reference/functions/subtract.md b/docs/reference/functions/subtract.md new file mode 100644 index 0000000000..3ae204d17e --- /dev/null +++ b/docs/reference/functions/subtract.md @@ -0,0 +1,36 @@ +--- +id: subtract +title: subtract +--- + +# Function: subtract() + +```ts +function subtract(left, right): BinaryNumericReturnType; +``` + +Defined in: [packages/db/src/query/builder/functions.ts:611](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L611) + +## Type Parameters + +### T1 + +`T1` *extends* `ExpressionLike` + +### T2 + +`T2` *extends* `ExpressionLike` + +## Parameters + +### left + +`T1` + +### right + +`T2` + +## Returns + +`BinaryNumericReturnType` diff --git a/docs/reference/functions/sum.md b/docs/reference/functions/sum.md index 3dcf0b62d7..2b6850a87c 100644 --- a/docs/reference/functions/sum.md +++ b/docs/reference/functions/sum.md @@ -9,7 +9,7 @@ title: sum function sum(arg): AggregateReturnType; ``` -Defined in: [packages/db/src/query/builder/functions.ts:374](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L374) +Defined in: [packages/db/src/query/builder/functions.ts:651](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L651) ## Type Parameters diff --git a/docs/reference/functions/toArray.md b/docs/reference/functions/toArray.md index edcf651bf0..f4dbfcf302 100644 --- a/docs/reference/functions/toArray.md +++ b/docs/reference/functions/toArray.md @@ -9,7 +9,7 @@ title: toArray function toArray(query): ToArrayWrapper>; ``` -Defined in: [packages/db/src/query/builder/functions.ts:453](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L453) +Defined in: [packages/db/src/query/builder/functions.ts:752](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L752) ## Type Parameters diff --git a/docs/reference/functions/toBooleanPredicate.md b/docs/reference/functions/toBooleanPredicate.md index e1ed96757b..160c1a68b3 100644 --- a/docs/reference/functions/toBooleanPredicate.md +++ b/docs/reference/functions/toBooleanPredicate.md @@ -9,7 +9,7 @@ title: toBooleanPredicate function toBooleanPredicate(result): boolean; ``` -Defined in: [packages/db/src/query/compiler/evaluators.ts:54](https://github.com/TanStack/db/blob/main/packages/db/src/query/compiler/evaluators.ts#L54) +Defined in: [packages/db/src/query/compiler/evaluators.ts:78](https://github.com/TanStack/db/blob/main/packages/db/src/query/compiler/evaluators.ts#L78) Converts a 3-valued logic result to a boolean for use in WHERE/HAVING filters. In SQL, UNKNOWN (null) values in WHERE clauses exclude rows, matching false behavior. diff --git a/docs/reference/functions/unionWherePredicates.md b/docs/reference/functions/unionWherePredicates.md deleted file mode 100644 index 7792c606b5..0000000000 --- a/docs/reference/functions/unionWherePredicates.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -id: unionWherePredicates -title: unionWherePredicates ---- - -# Function: unionWherePredicates() - -```ts -function unionWherePredicates(predicates): BasicExpression; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:297](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L297) - -Combine multiple where predicates with OR logic (union). -Returns a predicate that is satisfied when any input predicate is satisfied. -Simplifies when possible (e.g., age > 10 OR age > 20 → age > 10). - -## Parameters - -### predicates - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\>[] - -Array of where predicates to union - -## Returns - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> - -Combined predicate representing the union - -## Examples - -```ts -// Take least restrictive -unionWherePredicates([gt(ref('age'), val(10)), gt(ref('age'), val(20))]) // age > 10 -``` - -```ts -// Combine equals into IN -unionWherePredicates([eq(ref('age'), val(5)), eq(ref('age'), val(10))]) // age IN [5, 10] -``` diff --git a/docs/reference/functions/upper.md b/docs/reference/functions/upper.md index 390fc9fb31..bb33cb7976 100644 --- a/docs/reference/functions/upper.md +++ b/docs/reference/functions/upper.md @@ -9,7 +9,7 @@ title: upper function upper(arg): StringFunctionReturnType; ``` -Defined in: [packages/db/src/query/builder/functions.ts:279](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L279) +Defined in: [packages/db/src/query/builder/functions.ts:283](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L283) ## Type Parameters diff --git a/docs/reference/functions/withArrayChangeTracking.md b/docs/reference/functions/withArrayChangeTracking.md index 363f47c420..02b80cbb1e 100644 --- a/docs/reference/functions/withArrayChangeTracking.md +++ b/docs/reference/functions/withArrayChangeTracking.md @@ -9,7 +9,7 @@ title: withArrayChangeTracking function withArrayChangeTracking(targets, callback): Record[]; ``` -Defined in: [packages/db/src/proxy.ts:1171](https://github.com/TanStack/db/blob/main/packages/db/src/proxy.ts#L1171) +Defined in: [packages/db/src/proxy.ts:973](https://github.com/TanStack/db/blob/main/packages/db/src/proxy.ts#L973) Creates proxies for an array of objects, passes them to a callback function, and returns the changes made by the callback for each object diff --git a/docs/reference/functions/withChangeTracking.md b/docs/reference/functions/withChangeTracking.md index 70ce033005..caccf6d0bc 100644 --- a/docs/reference/functions/withChangeTracking.md +++ b/docs/reference/functions/withChangeTracking.md @@ -9,7 +9,7 @@ title: withChangeTracking function withChangeTracking(target, callback): Record; ``` -Defined in: [packages/db/src/proxy.ts:1152](https://github.com/TanStack/db/blob/main/packages/db/src/proxy.ts#L1152) +Defined in: [packages/db/src/proxy.ts:954](https://github.com/TanStack/db/blob/main/packages/db/src/proxy.ts#L954) Creates a proxy for an object, passes it to a callback function, and returns the changes made by the callback diff --git a/docs/reference/functions/withCollectionConfigFactory.md b/docs/reference/functions/withCollectionConfigFactory.md new file mode 100644 index 0000000000..75c127e490 --- /dev/null +++ b/docs/reference/functions/withCollectionConfigFactory.md @@ -0,0 +1,37 @@ +--- +id: withCollectionConfigFactory +title: withCollectionConfigFactory +--- + +# Function: withCollectionConfigFactory() + +```ts +function withCollectionConfigFactory(config, factory): CollectionConfigWithFactory; +``` + +Defined in: [packages/db/src/client.ts:81](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L81) + +Adds a fresh-config materializer to an adapter options object. + +Adapter option creators should use this so a module-scoped descriptor can be +materialized safely by more than one DbClient. + +## Type Parameters + +### TConfig + +`TConfig` *extends* `AnyCollectionConfig` + +## Parameters + +### config + +`TConfig` + +### factory + +(`client`) => `TConfig` + +## Returns + +`CollectionConfigWithFactory`\<`TConfig`\> diff --git a/docs/reference/index.md b/docs/reference/index.md index ce09d71916..51c1d667c5 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -12,7 +12,6 @@ title: "@tanstack/db" ## Classes - [AggregateFunctionNotInSelectError](classes/AggregateFunctionNotInSelectError.md) -- [AggregateNotSupportedError](classes/AggregateNotSupportedError.md) - [BaseIndex](classes/BaseIndex.md) - [BaseQueryBuilder](classes/BaseQueryBuilder.md) - [BasicIndex](classes/BasicIndex.md) @@ -24,9 +23,11 @@ title: "@tanstack/db" - [CollectionInputNotFoundError](classes/CollectionInputNotFoundError.md) - [CollectionIsInErrorStateError](classes/CollectionIsInErrorStateError.md) - [CollectionOperationError](classes/CollectionOperationError.md) +- [CollectionPreloadAbortedError](classes/CollectionPreloadAbortedError.md) - [CollectionRequiresConfigError](classes/CollectionRequiresConfigError.md) - [CollectionRequiresSyncConfigError](classes/CollectionRequiresSyncConfigError.md) - [CollectionStateError](classes/CollectionStateError.md) +- [DbClient](classes/DbClient.md) - [DeduplicatedLoadSubset](classes/DeduplicatedLoadSubset.md) - [DeleteKeyNotFoundError](classes/DeleteKeyNotFoundError.md) - [DistinctRequiresSelectError](classes/DistinctRequiresSelectError.md) @@ -56,6 +57,9 @@ title: "@tanstack/db" - [JoinError](classes/JoinError.md) - [KeyUpdateNotAllowedError](classes/KeyUpdateNotAllowedError.md) - [LimitOffsetRequireOrderByError](classes/LimitOffsetRequireOrderByError.md) +- [LiveQueryObserverDisposedError](classes/LiveQueryObserverDisposedError.md) +- [LiveQueryWindowControllerDisposedError](classes/LiveQueryWindowControllerDisposedError.md) +- [LoadSubsetOperationAbortedError](classes/LoadSubsetOperationAbortedError.md) - [LocalStorageCollectionError](classes/LocalStorageCollectionError.md) - [MissingAliasInputsError](classes/MissingAliasInputsError.md) - [MissingDeleteHandlerError](classes/MissingDeleteHandlerError.md) @@ -81,13 +85,14 @@ title: "@tanstack/db" - [SchemaMustBeSynchronousError](classes/SchemaMustBeSynchronousError.md) - [SchemaValidationError](classes/SchemaValidationError.md) - [SerializationError](classes/SerializationError.md) +- [SetWindowReentrancyError](classes/SetWindowReentrancyError.md) - [SetWindowRequiresOrderByError](classes/SetWindowRequiresOrderByError.md) - [SortedMap](classes/SortedMap.md) - [StorageError](classes/StorageError.md) - [StorageKeyRequiredError](classes/StorageKeyRequiredError.md) - [SubQueryMustHaveFromClauseError](classes/SubQueryMustHaveFromClauseError.md) -- [SubscriptionNotFoundError](classes/SubscriptionNotFoundError.md) - [SyncCleanupError](classes/SyncCleanupError.md) +- [SyncTransactionAbortedError](classes/SyncTransactionAbortedError.md) - [SyncTransactionAlreadyCommittedError](classes/SyncTransactionAlreadyCommittedError.md) - [SyncTransactionAlreadyCommittedWriteError](classes/SyncTransactionAlreadyCommittedWriteError.md) - [TanStackDBError](classes/TanStackDBError.md) @@ -95,17 +100,20 @@ title: "@tanstack/db" - [TransactionError](classes/TransactionError.md) - [TransactionNotPendingCommitError](classes/TransactionNotPendingCommitError.md) - [TransactionNotPendingMutateError](classes/TransactionNotPendingMutateError.md) +- [TransactionScope](classes/TransactionScope.md) - [UndefinedKeyError](classes/UndefinedKeyError.md) +- [UnhashableQueryIRError](classes/UnhashableQueryIRError.md) - [UnknownExpressionTypeError](classes/UnknownExpressionTypeError.md) - [UnknownFunctionError](classes/UnknownFunctionError.md) - [UnknownHavingExpressionTypeError](classes/UnknownHavingExpressionTypeError.md) +- [UnsafeAliasPathError](classes/UnsafeAliasPathError.md) - [UnsupportedAggregateFunctionError](classes/UnsupportedAggregateFunctionError.md) +- [UnsupportedFnSelectResultError](classes/UnsupportedFnSelectResultError.md) - [UnsupportedFromTypeError](classes/UnsupportedFromTypeError.md) - [UnsupportedJoinSourceTypeError](classes/UnsupportedJoinSourceTypeError.md) - [UnsupportedJoinTypeError](classes/UnsupportedJoinTypeError.md) - [UnsupportedRootScalarSelectError](classes/UnsupportedRootScalarSelectError.md) - [UpdateKeyNotFoundError](classes/UpdateKeyNotFoundError.md) -- [WhereClauseConversionError](classes/WhereClauseConversionError.md) ## Interfaces @@ -119,6 +127,8 @@ title: "@tanstack/db" - [CollectionIndexMetadata](interfaces/CollectionIndexMetadata.md) - [CollectionLike](interfaces/CollectionLike.md) - [Context](interfaces/Context.md) +- [CreateLiveQueryObserverOptions](interfaces/CreateLiveQueryObserverOptions.md) +- [CreateLiveQueryWindowControllerOptions](interfaces/CreateLiveQueryWindowControllerOptions.md) - [CreateOptimisticActionsOptions](interfaces/CreateOptimisticActionsOptions.md) - [CurrentStateAsChangesOptions](interfaces/CurrentStateAsChangesOptions.md) - [DebounceStrategy](interfaces/DebounceStrategy.md) @@ -129,10 +139,14 @@ title: "@tanstack/db" - [IndexDevModeConfig](interfaces/IndexDevModeConfig.md) - [IndexInterface](interfaces/IndexInterface.md) - [IndexOptions](interfaces/IndexOptions.md) -- [IndexStats](interfaces/IndexStats.md) - [IndexSuggestion](interfaces/IndexSuggestion.md) - [InsertConfig](interfaces/InsertConfig.md) - [LiveQueryCollectionConfig](interfaces/LiveQueryCollectionConfig.md) +- [LiveQueryObserver](interfaces/LiveQueryObserver.md) +- [LiveQuerySnapshot](interfaces/LiveQuerySnapshot.md) +- [LiveQueryStatusFlags](interfaces/LiveQueryStatusFlags.md) +- [LiveQueryWindowController](interfaces/LiveQueryWindowController.md) +- [LiveQueryWindowSnapshot](interfaces/LiveQueryWindowSnapshot.md) - [LocalOnlyCollectionConfig](interfaces/LocalOnlyCollectionConfig.md) - [LocalOnlyCollectionUtils](interfaces/LocalOnlyCollectionUtils.md) - [LocalStorageCollectionConfig](interfaces/LocalStorageCollectionConfig.md) @@ -151,6 +165,7 @@ title: "@tanstack/db" - [SubscribeChangesOptions](interfaces/SubscribeChangesOptions.md) - [SubscribeChangesSnapshotOptions](interfaces/SubscribeChangesSnapshotOptions.md) - [Subscription](interfaces/Subscription.md) +- [SubscriptionLoadSubsetErrorEvent](interfaces/SubscriptionLoadSubsetErrorEvent.md) - [SubscriptionStatusChangeEvent](interfaces/SubscriptionStatusChangeEvent.md) - [SubscriptionStatusEvent](interfaces/SubscriptionStatusEvent.md) - [SubscriptionUnsubscribedEvent](interfaces/SubscriptionUnsubscribedEvent.md) @@ -171,14 +186,31 @@ title: "@tanstack/db" - [CleanupFn](type-aliases/CleanupFn.md) - [ClearStorageFn](type-aliases/ClearStorageFn.md) - [CollectionConfigSingleRowOption](type-aliases/CollectionConfigSingleRowOption.md) +- [CollectionMaterializeOptions](type-aliases/CollectionMaterializeOptions.md) +- [CollectionOptions](type-aliases/CollectionOptions.md) - [CollectionStatus](type-aliases/CollectionStatus.md) +- [ContextFromSource](type-aliases/ContextFromSource.md) +- [ContextFromUnionBranches](type-aliases/ContextFromUnionBranches.md) +- [ContextFromUnionSource](type-aliases/ContextFromUnionSource.md) - [ContextSchema](type-aliases/ContextSchema.md) - [CursorExpressions](type-aliases/CursorExpressions.md) +- [DbClientEvent](type-aliases/DbClientEvent.md) +- [DbClientLiveQuery](type-aliases/DbClientLiveQuery.md) +- [DbClientLiveQueryState](type-aliases/DbClientLiveQueryState.md) +- [DbClientOptions](type-aliases/DbClientOptions.md) +- [DeferredLiveQueryCollections](type-aliases/DeferredLiveQueryCollections.md) +- [DehydrateDbClientOptions](type-aliases/DehydrateDbClientOptions.md) +- [DehydratedCollectionChunk](type-aliases/DehydratedCollectionChunk.md) +- [DehydratedCollectionRow](type-aliases/DehydratedCollectionRow.md) +- [DehydratedDbState](type-aliases/DehydratedDbState.md) +- [DehydratedLiveQuery](type-aliases/DehydratedLiveQuery.md) +- [DehydratedLiveQueryResult](type-aliases/DehydratedLiveQueryResult.md) - [DeleteKeyMessage](type-aliases/DeleteKeyMessage.md) - [DeleteMutationFn](type-aliases/DeleteMutationFn.md) - [DeleteMutationFnParams](type-aliases/DeleteMutationFnParams.md) - [DeltaEvent](type-aliases/DeltaEvent.md) - [DeltaType](type-aliases/DeltaType.md) +- [DemandKey](type-aliases/DemandKey.md) - [EffectQueryInput](type-aliases/EffectQueryInput.md) - [ExtractContext](type-aliases/ExtractContext.md) - [FieldPath](type-aliases/FieldPath.md) @@ -190,6 +222,7 @@ title: "@tanstack/db" - [IndexConstructor](type-aliases/IndexConstructor.md) - [IndexOperation](type-aliases/IndexOperation.md) - [IndexOperation](type-aliases/IndexOperation-1.md) +- [IndexReader](type-aliases/IndexReader.md) - [InferCollectionType](type-aliases/InferCollectionType.md) - [InferResultType](type-aliases/InferResultType.md) - [InferSchemaInput](type-aliases/InferSchemaInput.md) @@ -202,8 +235,14 @@ title: "@tanstack/db" - [KeyedNamespacedRow](type-aliases/KeyedNamespacedRow.md) - [KeyedStream](type-aliases/KeyedStream.md) - [LiveQueryCollectionUtils](type-aliases/LiveQueryCollectionUtils.md) +- [LiveQueryKey](type-aliases/LiveQueryKey.md) +- [LiveQueryObserverListener](type-aliases/LiveQueryObserverListener.md) +- [LiveQueryOptions](type-aliases/LiveQueryOptions.md) +- [LiveQueryWindowCollection](type-aliases/LiveQueryWindowCollection.md) +- [LiveQueryWindowInputKind](type-aliases/LiveQueryWindowInputKind.md) - [LoadSubsetFn](type-aliases/LoadSubsetFn.md) - [LoadSubsetOptions](type-aliases/LoadSubsetOptions.md) +- [LoadSubsetRequestResult](type-aliases/LoadSubsetRequestResult.md) - [MakeOptional](type-aliases/MakeOptional.md) - [MaybeSingleResult](type-aliases/MaybeSingleResult.md) - [MergeContextForJoinCallback](type-aliases/MergeContextForJoinCallback.md) @@ -220,9 +259,11 @@ title: "@tanstack/db" - [OrderByCallback](type-aliases/OrderByCallback.md) - [Prettify](type-aliases/Prettify.md) - [QueryBuilder](type-aliases/QueryBuilder.md) +- [QueryIdentity](type-aliases/QueryIdentity.md) - [QueryResult](type-aliases/QueryResult.md) - [Ref](type-aliases/Ref.md) - [RefsForContext](type-aliases/RefsForContext.md) +- [ResolvedLiveQueryWindowInput](type-aliases/ResolvedLiveQueryWindowInput.md) - [ResolveTransactionChanges](type-aliases/ResolveTransactionChanges.md) - [ResultStream](type-aliases/ResultStream.md) - [ResultTypeFromSelect](type-aliases/ResultTypeFromSelect.md) @@ -230,7 +271,9 @@ title: "@tanstack/db" - [SchemaFromSource](type-aliases/SchemaFromSource.md) - [SelectObject](type-aliases/SelectObject.md) - [SingleResult](type-aliases/SingleResult.md) +- [SingleSource](type-aliases/SingleSource.md) - [Source](type-aliases/Source.md) +- [SourceClauseContext](type-aliases/SourceClauseContext.md) - [StandardSchema](type-aliases/StandardSchema.md) - [StandardSchemaAlias](type-aliases/StandardSchemaAlias.md) - [StorageApi](type-aliases/StorageApi.md) @@ -240,6 +283,7 @@ title: "@tanstack/db" - [StringCollationConfig](type-aliases/StringCollationConfig.md) - [SubscriptionEvents](type-aliases/SubscriptionEvents.md) - [SubscriptionStatus](type-aliases/SubscriptionStatus.md) +- [SyncAppliedReceipt](type-aliases/SyncAppliedReceipt.md) - [SyncConfigRes](type-aliases/SyncConfigRes.md) - [SyncMode](type-aliases/SyncMode.md) - [TransactionState](type-aliases/TransactionState.md) @@ -264,10 +308,14 @@ title: "@tanstack/db" - [add](functions/add.md) - [and](functions/and.md) +- [assertLiveQueryWindowManyResult](functions/assertLiveQueryWindowManyResult.md) - [avg](functions/avg.md) +- [canonicalizeQueryIR](functions/canonicalizeQueryIR.md) - [caseWhen](functions/caseWhen.md) - [clearQueryPatterns](functions/clearQueryPatterns.md) - [coalesce](functions/coalesce.md) +- [collectionOptions](functions/collectionOptions.md) +- [compareLiveQueryWindowDependencies](functions/compareLiveQueryWindowDependencies.md) - [compileExpression](functions/compileExpression.md) - [compileQuery](functions/compileQuery.md) - [compileSingleRowExpression](functions/compileSingleRowExpression.md) @@ -279,32 +327,46 @@ title: "@tanstack/db" - [createCollection](functions/createCollection.md) - [createEffect](functions/createEffect.md) - [createLiveQueryCollection](functions/createLiveQueryCollection.md) +- [createLiveQueryObserver](functions/createLiveQueryObserver.md) +- [createLiveQueryWindowController](functions/createLiveQueryWindowController.md) - [createOptimisticAction](functions/createOptimisticAction.md) - [createPacedMutations](functions/createPacedMutations.md) - [createTransaction](functions/createTransaction.md) - [debounceStrategy](functions/debounceStrategy.md) - [deepEquals](functions/deepEquals.md) +- [divide](functions/divide.md) - [eq](functions/eq.md) - [extractFieldPath](functions/extractFieldPath.md) - [extractSimpleComparisons](functions/extractSimpleComparisons.md) - [extractValue](functions/extractValue.md) +- [fetchNextLiveQueryWindowPage](functions/fetchNextLiveQueryWindowPage.md) - [findIndexForField](functions/findIndexForField.md) - [getActiveTransaction](functions/getActiveTransaction.md) - [getIndexDevModeConfig](functions/getIndexDevModeConfig.md) +- [getLiveQueryHash](functions/getLiveQueryHash.md) +- [getLiveQueryStatusFlags](functions/getLiveQueryStatusFlags.md) +- [getLiveQueryWindowCollectionWarning](functions/getLiveQueryWindowCollectionWarning.md) +- [getLiveQueryWindowInputKind](functions/getLiveQueryWindowInputKind.md) +- [getLoadSubsetDemandKey](functions/getLoadSubsetDemandKey.md) +- [getPreparedLiveQueryIdentity](functions/getPreparedLiveQueryIdentity.md) +- [getQueryIdentity](functions/getQueryIdentity.md) - [getQueryPatterns](functions/getQueryPatterns.md) +- [getStableQueryBuilderHash](functions/getStableQueryBuilderHash.md) +- [getStableQueryIRHash](functions/getStableQueryIRHash.md) +- [getStableValueHash](functions/getStableValueHash.md) - [gt](functions/gt.md) - [gte](functions/gte.md) +- [hasLiveQueryWindowLeases](functions/hasLiveQueryWindowLeases.md) - [hasVirtualProps](functions/hasVirtualProps.md) - [ilike](functions/ilike.md) - [inArray](functions/inArray.md) +- [isCollection](functions/isCollection.md) +- [isCollectionOptions](functions/isCollectionOptions.md) - [isDevModeEnabled](functions/isDevModeEnabled.md) -- [isLimitSubset](functions/isLimitSubset.md) +- [isLiveQueryWindowCollection](functions/isLiveQueryWindowCollection.md) - [isNull](functions/isNull.md) -- [isOffsetLimitSubset](functions/isOffsetLimitSubset.md) -- [isOrderBySubset](functions/isOrderBySubset.md) -- [isPredicateSubset](functions/isPredicateSubset.md) +- [isSingleResultCollection](functions/isSingleResultCollection.md) - [isUndefined](functions/isUndefined.md) -- [isWhereSubset](functions/isWhereSubset.md) - [length](functions/length.md) - [like](functions/like.md) - [liveQueryCollectionOptions](functions/liveQueryCollectionOptions.md) @@ -313,24 +375,31 @@ title: "@tanstack/db" - [lower](functions/lower.md) - [lt](functions/lt.md) - [lte](functions/lte.md) +- [materialize](functions/materialize.md) - [max](functions/max.md) - [min](functions/min.md) -- [minusWherePredicates](functions/minusWherePredicates.md) +- [multiply](functions/multiply.md) +- [normalizeLiveQueryWindowPageSize](functions/normalizeLiveQueryWindowPageSize.md) - [not](functions/not.md) - [optimizeExpressionWithIndexes](functions/optimizeExpressionWithIndexes.md) - [or](functions/or.md) - [parseLoadSubsetOptions](functions/parseLoadSubsetOptions.md) - [parseOrderByExpression](functions/parseOrderByExpression.md) - [parseWhereExpression](functions/parseWhereExpression.md) +- [prepareLiveQueryValue](functions/prepareLiveQueryValue.md) - [queryOnce](functions/queryOnce.md) - [queueStrategy](functions/queueStrategy.md) +- [resolveLiveQueryWindowInput](functions/resolveLiveQueryWindowInput.md) +- [safeRandomUUID](functions/safeRandomUUID.md) +- [shouldPreserveLiveQueryWindowPageCount](functions/shouldPreserveLiveQueryWindowPageCount.md) +- [subtract](functions/subtract.md) - [sum](functions/sum.md) - [throttleStrategy](functions/throttleStrategy.md) - [toArray](functions/toArray.md) - [toBooleanPredicate](functions/toBooleanPredicate.md) - [trackQuery](functions/trackQuery.md) -- [unionWherePredicates](functions/unionWherePredicates.md) - [upper](functions/upper.md) - [walkExpression](functions/walkExpression.md) - [withArrayChangeTracking](functions/withArrayChangeTracking.md) - [withChangeTracking](functions/withChangeTracking.md) +- [withCollectionConfigFactory](functions/withCollectionConfigFactory.md) diff --git a/docs/reference/interfaces/BTreeRangeQueryOptions.md b/docs/reference/interfaces/BTreeRangeQueryOptions.md index 080fcc45bf..609f8e735a 100644 --- a/docs/reference/interfaces/BTreeRangeQueryOptions.md +++ b/docs/reference/interfaces/BTreeRangeQueryOptions.md @@ -5,7 +5,7 @@ title: BTreeRangeQueryOptions # Interface: BTreeRangeQueryOptions -Defined in: [packages/db/src/indexes/btree-index.ts:24](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L24) +Defined in: [packages/db/src/indexes/btree-index.ts:27](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L27) Options for range queries @@ -17,7 +17,7 @@ Options for range queries optional from: any; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:25](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L25) +Defined in: [packages/db/src/indexes/btree-index.ts:28](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L28) *** @@ -27,7 +27,7 @@ Defined in: [packages/db/src/indexes/btree-index.ts:25](https://github.com/TanSt optional fromInclusive: boolean; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:27](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L27) +Defined in: [packages/db/src/indexes/btree-index.ts:30](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L30) *** @@ -37,7 +37,7 @@ Defined in: [packages/db/src/indexes/btree-index.ts:27](https://github.com/TanSt optional to: any; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:26](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L26) +Defined in: [packages/db/src/indexes/btree-index.ts:29](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L29) *** @@ -47,4 +47,4 @@ Defined in: [packages/db/src/indexes/btree-index.ts:26](https://github.com/TanSt optional toInclusive: boolean; ``` -Defined in: [packages/db/src/indexes/btree-index.ts:28](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L28) +Defined in: [packages/db/src/indexes/btree-index.ts:31](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/btree-index.ts#L31) diff --git a/docs/reference/interfaces/BaseCollectionConfig.md b/docs/reference/interfaces/BaseCollectionConfig.md index cf0e34c7d9..389d86e13b 100644 --- a/docs/reference/interfaces/BaseCollectionConfig.md +++ b/docs/reference/interfaces/BaseCollectionConfig.md @@ -5,7 +5,7 @@ title: BaseCollectionConfig # Interface: BaseCollectionConfig\ -Defined in: [packages/db/src/types.ts:521](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L521) +Defined in: [packages/db/src/types.ts:613](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L613) ## Extended by @@ -42,7 +42,7 @@ Defined in: [packages/db/src/types.ts:521](https://github.com/TanStack/db/blob/m optional autoIndex: "off" | "eager"; ``` -Defined in: [packages/db/src/types.ts:571](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L571) +Defined in: [packages/db/src/types.ts:663](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L663) Auto-indexing mode for the collection. When enabled, indexes will be automatically created for simple where expressions. @@ -67,7 +67,7 @@ When enabled, indexes will be automatically created for simple where expressions optional compare: (x, y) => number; ``` -Defined in: [packages/db/src/types.ts:596](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L596) +Defined in: [packages/db/src/types.ts:688](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L688) Optional function to compare two items. This is used to order the items in the collection. @@ -107,7 +107,7 @@ compare: (x, y) => x.createdAt.getTime() - y.createdAt.getTime() optional defaultIndexType: IndexConstructor; ``` -Defined in: [packages/db/src/types.ts:585](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L585) +Defined in: [packages/db/src/types.ts:677](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L677) Default index type to use when creating indexes without an explicit type. Required for auto-indexing. Import from '@tanstack/db'. @@ -131,7 +131,7 @@ const collection = createCollection({ optional defaultStringCollation: StringCollationConfig; ``` -Defined in: [packages/db/src/types.ts:742](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L742) +Defined in: [packages/db/src/types.ts:834](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L834) Specifies how to compare data in the collection. This should be configured to match data ordering on the backend. @@ -146,7 +146,7 @@ E.g., when using the Electric DB collection these options optional gcTime: number; ``` -Defined in: [packages/db/src/types.ts:550](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L550) +Defined in: [packages/db/src/types.ts:642](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L642) Time in milliseconds after which the collection will be garbage collected when it has no active subscribers. Defaults to 5 minutes (300000ms). @@ -159,7 +159,7 @@ when it has no active subscribers. Defaults to 5 minutes (300000ms). getKey: (item) => TKey; ``` -Defined in: [packages/db/src/types.ts:545](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L545) +Defined in: [packages/db/src/types.ts:637](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L637) Function to extract the ID from an object This is required for update/delete operations which now only accept IDs @@ -193,7 +193,7 @@ getKey: (item) => item.uuid optional id: string; ``` -Defined in: [packages/db/src/types.ts:534](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L534) +Defined in: [packages/db/src/types.ts:626](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L626) *** @@ -203,7 +203,7 @@ Defined in: [packages/db/src/types.ts:534](https://github.com/TanStack/db/blob/m optional onDelete: DeleteMutationFn; ``` -Defined in: [packages/db/src/types.ts:734](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L734) +Defined in: [packages/db/src/types.ts:826](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L826) Optional asynchronous handler function called before a delete operation @@ -267,7 +267,7 @@ onDelete: async ({ transaction, collection }) => { optional onInsert: InsertMutationFn; ``` -Defined in: [packages/db/src/types.ts:647](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L647) +Defined in: [packages/db/src/types.ts:739](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L739) Optional asynchronous handler function called before an insert operation @@ -330,7 +330,7 @@ onInsert: async ({ transaction, collection }) => { optional onUpdate: UpdateMutationFn; ``` -Defined in: [packages/db/src/types.ts:691](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L691) +Defined in: [packages/db/src/types.ts:783](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L783) Optional asynchronous handler function called before an update operation @@ -394,7 +394,7 @@ onUpdate: async ({ transaction, collection }) => { optional schema: TSchema; ``` -Defined in: [packages/db/src/types.ts:535](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L535) +Defined in: [packages/db/src/types.ts:627](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L627) *** @@ -404,7 +404,7 @@ Defined in: [packages/db/src/types.ts:535](https://github.com/TanStack/db/blob/m optional startSync: boolean; ``` -Defined in: [packages/db/src/types.ts:561](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L561) +Defined in: [packages/db/src/types.ts:653](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L653) Whether to eagerly start syncing on collection creation. When true, syncing begins immediately. When false, syncing starts when the first subscriber attaches. @@ -427,7 +427,7 @@ false optional syncMode: SyncMode; ``` -Defined in: [packages/db/src/types.ts:605](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L605) +Defined in: [packages/db/src/types.ts:697](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L697) The mode of sync to use for the collection. @@ -449,4 +449,4 @@ The exact implementation of the sync mode is up to the sync implementation. optional utils: TUtils; ``` -Defined in: [packages/db/src/types.ts:744](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L744) +Defined in: [packages/db/src/types.ts:836](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L836) diff --git a/docs/reference/interfaces/BasicIndexOptions.md b/docs/reference/interfaces/BasicIndexOptions.md index 9918e1e33b..94f168fc00 100644 --- a/docs/reference/interfaces/BasicIndexOptions.md +++ b/docs/reference/interfaces/BasicIndexOptions.md @@ -5,7 +5,7 @@ title: BasicIndexOptions # Interface: BasicIndexOptions -Defined in: [packages/db/src/indexes/basic-index.ts:24](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L24) +Defined in: [packages/db/src/indexes/basic-index.ts:30](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L30) Options for Basic index @@ -17,7 +17,7 @@ Options for Basic index optional compareFn: (a, b) => number; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:25](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L25) +Defined in: [packages/db/src/indexes/basic-index.ts:31](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L31) #### Parameters @@ -41,4 +41,4 @@ Defined in: [packages/db/src/indexes/basic-index.ts:25](https://github.com/TanSt optional compareOptions: CompareOptions; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:26](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L26) +Defined in: [packages/db/src/indexes/basic-index.ts:32](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L32) diff --git a/docs/reference/interfaces/ChangeMessage.md b/docs/reference/interfaces/ChangeMessage.md index 339b3c3505..e45c77c661 100644 --- a/docs/reference/interfaces/ChangeMessage.md +++ b/docs/reference/interfaces/ChangeMessage.md @@ -5,7 +5,7 @@ title: ChangeMessage # Interface: ChangeMessage\ -Defined in: [packages/db/src/types.ts:381](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L381) +Defined in: [packages/db/src/types.ts:472](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L472) ## Type Parameters @@ -25,7 +25,7 @@ Defined in: [packages/db/src/types.ts:381](https://github.com/TanStack/db/blob/m key: TKey; ``` -Defined in: [packages/db/src/types.ts:385](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L385) +Defined in: [packages/db/src/types.ts:476](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L476) *** @@ -35,7 +35,7 @@ Defined in: [packages/db/src/types.ts:385](https://github.com/TanStack/db/blob/m optional metadata: Record; ``` -Defined in: [packages/db/src/types.ts:389](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L389) +Defined in: [packages/db/src/types.ts:480](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L480) *** @@ -45,7 +45,7 @@ Defined in: [packages/db/src/types.ts:389](https://github.com/TanStack/db/blob/m optional previousValue: T; ``` -Defined in: [packages/db/src/types.ts:387](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L387) +Defined in: [packages/db/src/types.ts:478](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L478) *** @@ -55,7 +55,7 @@ Defined in: [packages/db/src/types.ts:387](https://github.com/TanStack/db/blob/m type: OperationType; ``` -Defined in: [packages/db/src/types.ts:388](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L388) +Defined in: [packages/db/src/types.ts:479](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L479) *** @@ -65,4 +65,4 @@ Defined in: [packages/db/src/types.ts:388](https://github.com/TanStack/db/blob/m value: T; ``` -Defined in: [packages/db/src/types.ts:386](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L386) +Defined in: [packages/db/src/types.ts:477](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L477) diff --git a/docs/reference/interfaces/Collection.md b/docs/reference/interfaces/Collection.md index dfc0e6aab8..c90a23a519 100644 --- a/docs/reference/interfaces/Collection.md +++ b/docs/reference/interfaces/Collection.md @@ -5,7 +5,7 @@ title: Collection # Interface: Collection\ -Defined in: [packages/db/src/collection/index.ts:54](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L54) +Defined in: [packages/db/src/collection/index.ts:58](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L58) Enhanced Collection interface that includes both data type T and utilities TUtils @@ -51,7 +51,7 @@ The type for insert operations (can be different from T for schemas with default _lifecycle: CollectionLifecycleManager; ``` -Defined in: [packages/db/src/collection/index.ts:289](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L289) +Defined in: [packages/db/src/collection/index.ts:293](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L293) #### Inherited from @@ -65,7 +65,7 @@ Defined in: [packages/db/src/collection/index.ts:289](https://github.com/TanStac _state: CollectionStateManager; ``` -Defined in: [packages/db/src/collection/index.ts:301](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L301) +Defined in: [packages/db/src/collection/index.ts:305](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L305) #### Inherited from @@ -79,7 +79,7 @@ Defined in: [packages/db/src/collection/index.ts:301](https://github.com/TanStac _sync: CollectionSyncManager; ``` -Defined in: [packages/db/src/collection/index.ts:290](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L290) +Defined in: [packages/db/src/collection/index.ts:294](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L294) #### Inherited from @@ -93,7 +93,7 @@ Defined in: [packages/db/src/collection/index.ts:290](https://github.com/TanStac config: CollectionConfig; ``` -Defined in: [packages/db/src/collection/index.ts:280](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L280) +Defined in: [packages/db/src/collection/index.ts:284](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L284) #### Inherited from @@ -107,7 +107,7 @@ Defined in: [packages/db/src/collection/index.ts:280](https://github.com/TanStac deferDataRefresh: Promise | null = null; ``` -Defined in: [packages/db/src/collection/index.ts:308](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L308) +Defined in: [packages/db/src/collection/index.ts:312](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L312) When set, collection consumers should defer processing incoming data refreshes until this promise resolves. This prevents stale data from @@ -125,7 +125,7 @@ overwriting optimistic state while pending writes are being applied. id: string; ``` -Defined in: [packages/db/src/collection/index.ts:279](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L279) +Defined in: [packages/db/src/collection/index.ts:283](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L283) #### Inherited from @@ -139,7 +139,7 @@ Defined in: [packages/db/src/collection/index.ts:279](https://github.com/TanStac readonly optional singleResult: true; ``` -Defined in: [packages/db/src/collection/index.ts:62](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L62) +Defined in: [packages/db/src/collection/index.ts:66](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L66) *** @@ -149,7 +149,7 @@ Defined in: [packages/db/src/collection/index.ts:62](https://github.com/TanStack readonly utils: TUtils; ``` -Defined in: [packages/db/src/collection/index.ts:61](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L61) +Defined in: [packages/db/src/collection/index.ts:65](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L65) #### Overrides @@ -157,6 +157,53 @@ Defined in: [packages/db/src/collection/index.ts:61](https://github.com/TanStack ## Accessors +### \_layoutRevision + +#### Get Signature + +```ts +get _layoutRevision(): number; +``` + +Defined in: [packages/db/src/collection/index.ts:442](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L442) + +Monotonic revision of explicit layout-only publications. +Internal — used to distinguish them from empty ready events. + +##### Returns + +`number` + +#### Inherited from + +[`CollectionImpl`](../classes/CollectionImpl.md).[`_layoutRevision`](../classes/CollectionImpl.md#_layoutrevision) + +*** + +### \_stateRevision + +#### Get Signature + +```ts +get _stateRevision(): number; +``` + +Defined in: [packages/db/src/collection/index.ts:434](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L434) + +Monotonic revision of the collection's visible state; advances once per +committed batch of changes, even while nothing is subscribed. +Internal — used by the live-query observer's snapshot cache. + +##### Returns + +`number` + +#### Inherited from + +[`CollectionImpl`](../classes/CollectionImpl.md).[`_stateRevision`](../classes/CollectionImpl.md#_staterevision) + +*** + ### compareOptions #### Get Signature @@ -165,7 +212,7 @@ Defined in: [packages/db/src/collection/index.ts:61](https://github.com/TanStack get compareOptions(): StringCollationConfig; ``` -Defined in: [packages/db/src/collection/index.ts:643](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L643) +Defined in: [packages/db/src/collection/index.ts:708](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L708) ##### Returns @@ -185,7 +232,7 @@ Defined in: [packages/db/src/collection/index.ts:643](https://github.com/TanStac get indexes(): Map>; ``` -Defined in: [packages/db/src/collection/index.ts:628](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L628) +Defined in: [packages/db/src/collection/index.ts:693](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L693) Get resolved indexes for query optimization @@ -207,7 +254,7 @@ Get resolved indexes for query optimization get isLoadingSubset(): boolean; ``` -Defined in: [packages/db/src/collection/index.ts:456](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L456) +Defined in: [packages/db/src/collection/index.ts:500](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L500) Check if the collection is currently loading more data @@ -231,7 +278,7 @@ true if the collection has pending load more operations, false otherwise get size(): number; ``` -Defined in: [packages/db/src/collection/index.ts:493](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L493) +Defined in: [packages/db/src/collection/index.ts:558](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L558) Get the current size of the collection (cached) @@ -253,7 +300,7 @@ Get the current size of the collection (cached) get state(): Map>; ``` -Defined in: [packages/db/src/collection/index.ts:820](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L820) +Defined in: [packages/db/src/collection/index.ts:885](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L885) Gets the current state of the collection as a Map @@ -293,7 +340,7 @@ Map containing all items in the collection, with keys as identifiers get status(): CollectionStatus; ``` -Defined in: [packages/db/src/collection/index.ts:411](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L411) +Defined in: [packages/db/src/collection/index.ts:418](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L418) Gets the current status of the collection @@ -315,7 +362,7 @@ Gets the current status of the collection get subscriberCount(): number; ``` -Defined in: [packages/db/src/collection/index.ts:418](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L418) +Defined in: [packages/db/src/collection/index.ts:425](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L425) Get the number of subscribers to the collection @@ -337,7 +384,7 @@ Get the number of subscribers to the collection get toArray(): WithVirtualProps[]; ``` -Defined in: [packages/db/src/collection/index.ts:849](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L849) +Defined in: [packages/db/src/collection/index.ts:914](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L914) Gets the current state of the collection as an Array @@ -353,13 +400,177 @@ An Array containing all items in the collection ## Methods +### \_deferPublication() + +```ts +_deferPublication(): PublicationDeferral; +``` + +Defined in: [packages/db/src/collection/index.ts:457](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L457) + +Defer subscriber events until a coherent multi-Collection commit ends. + +#### Returns + +`PublicationDeferral` + +#### Inherited from + +[`CollectionImpl`](../classes/CollectionImpl.md).[`_deferPublication`](../classes/CollectionImpl.md#_deferpublication) + +*** + +### \_deferSyncStart() + +```ts +_deferSyncStart(): boolean; +``` + +Defined in: [packages/db/src/collection/index.ts:524](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L524) + +**`Internal`** + +#### Returns + +`boolean` + +#### Inherited from + +[`CollectionImpl`](../classes/CollectionImpl.md).[`_deferSyncStart`](../classes/CollectionImpl.md#_defersyncstart) + +*** + +### \_hasHydratedKey() + +```ts +_hasHydratedKey(key): boolean; +``` + +Defined in: [packages/db/src/collection/index.ts:519](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L519) + +**`Internal`** + +#### Parameters + +##### key + +`TKey` + +#### Returns + +`boolean` + +#### Inherited from + +[`CollectionImpl`](../classes/CollectionImpl.md).[`_hasHydratedKey`](../classes/CollectionImpl.md#_hashydratedkey) + +*** + +### \_markLayoutChange() + +```ts +_markLayoutChange(): void; +``` + +Defined in: [packages/db/src/collection/index.ts:452](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L452) + +Mark the active sync transaction as layout-changing. Internal. + +#### Returns + +`void` + +#### Inherited from + +[`CollectionImpl`](../classes/CollectionImpl.md).[`_markLayoutChange`](../classes/CollectionImpl.md#_marklayoutchange) + +*** + +### \_resumeSyncStart() + +```ts +_resumeSyncStart(): void; +``` + +Defined in: [packages/db/src/collection/index.ts:529](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L529) + +**`Internal`** + +#### Returns + +`void` + +#### Inherited from + +[`CollectionImpl`](../classes/CollectionImpl.md).[`_resumeSyncStart`](../classes/CollectionImpl.md#_resumesyncstart) + +*** + +### \_setTransactionScope() + +```ts +_setTransactionScope(transactionScope): void; +``` + +Defined in: [packages/db/src/collection/index.ts:514](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L514) + +**`Internal`** + +#### Parameters + +##### transactionScope + +[`TransactionScope`](../classes/TransactionScope.md) + +#### Returns + +`void` + +#### Inherited from + +[`CollectionImpl`](../classes/CollectionImpl.md).[`_setTransactionScope`](../classes/CollectionImpl.md#_settransactionscope) + +*** + +### \_subscribeLayoutChanges() + +```ts +_subscribeLayoutChanges(listener): () => void; +``` + +Defined in: [packages/db/src/collection/index.ts:447](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L447) + +Subscribe to layout-only publications. Internal observer channel. + +#### Parameters + +##### listener + +() => `void` + +#### Returns + +```ts +(): void; +``` + +##### Returns + +`void` + +#### Inherited from + +[`CollectionImpl`](../classes/CollectionImpl.md).[`_subscribeLayoutChanges`](../classes/CollectionImpl.md#_subscribelayoutchanges) + +*** + ### \[iterator\]() ```ts iterator: IterableIterator<[TKey, WithVirtualProps]>; ``` -Defined in: [packages/db/src/collection/index.ts:531](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L531) +Defined in: [packages/db/src/collection/index.ts:596](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L596) Get all entries (virtual derived state) @@ -379,10 +590,12 @@ Get all entries (virtual derived state) cleanup(): Promise; ``` -Defined in: [packages/db/src/collection/index.ts:988](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L988) +Defined in: [packages/db/src/collection/index.ts:1055](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1055) Clean up the collection by stopping sync and clearing data This can be called manually or automatically by garbage collection +Cleanup callbacks must not restart this collection or call its preload(). +Wait until cleanup completes before starting a new sync session. #### Returns @@ -400,7 +613,7 @@ This can be called manually or automatically by garbage collection createIndex(indexCallback, config): BaseIndex; ``` -Defined in: [packages/db/src/collection/index.ts:597](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L597) +Defined in: [packages/db/src/collection/index.ts:662](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L662) Creates an index on a collection for faster queries. Indexes significantly improve query performance by allowing constant time lookups @@ -460,7 +673,7 @@ currentStateAsChanges(options): | ChangeMessage, string | number>[]; ``` -Defined in: [packages/db/src/collection/index.ts:887](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L887) +Defined in: [packages/db/src/collection/index.ts:952](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L952) Returns the current state of the collection as an array of changes @@ -508,7 +721,7 @@ const activeChanges = collection.currentStateAsChanges({ delete(keys, config?): Transaction; ``` -Defined in: [packages/db/src/collection/index.ts:797](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L797) +Defined in: [packages/db/src/collection/index.ts:862](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L862) Deletes one or more items from the collection @@ -575,7 +788,7 @@ try { entries(): IterableIterator<[TKey, WithVirtualProps]>; ``` -Defined in: [packages/db/src/collection/index.ts:519](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L519) +Defined in: [packages/db/src/collection/index.ts:584](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L584) Get all entries (virtual derived state) @@ -595,7 +808,7 @@ Get all entries (virtual derived state) forEach(callbackfn): void; ``` -Defined in: [packages/db/src/collection/index.ts:540](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L540) +Defined in: [packages/db/src/collection/index.ts:605](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L605) Execute a callback for each entry in the collection @@ -623,7 +836,7 @@ get(key): | undefined; ``` -Defined in: [packages/db/src/collection/index.ts:479](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L479) +Defined in: [packages/db/src/collection/index.ts:544](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L544) Get the current value for a key (virtual derived state) @@ -650,7 +863,7 @@ Get the current value for a key (virtual derived state) getIndexMetadata(): CollectionIndexMetadata[]; ``` -Defined in: [packages/db/src/collection/index.ts:621](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L621) +Defined in: [packages/db/src/collection/index.ts:686](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L686) Returns a snapshot of current index metadata sorted by indexId. Persistence wrappers can use this to bootstrap index state if indexes were @@ -672,7 +885,7 @@ created before event listeners were attached. getKeyFromItem(item): TKey; ``` -Defined in: [packages/db/src/collection/index.ts:571](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L571) +Defined in: [packages/db/src/collection/index.ts:636](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L636) #### Parameters @@ -696,7 +909,7 @@ Defined in: [packages/db/src/collection/index.ts:571](https://github.com/TanStac has(key): boolean; ``` -Defined in: [packages/db/src/collection/index.ts:486](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L486) +Defined in: [packages/db/src/collection/index.ts:551](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L551) Check if a key exists in the collection (virtual derived state) @@ -722,7 +935,7 @@ Check if a key exists in the collection (virtual derived state) insert(data, config?): Transaction>; ``` -Defined in: [packages/db/src/collection/index.ts:684](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L684) +Defined in: [packages/db/src/collection/index.ts:749](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L749) Inserts one or more items into the collection @@ -796,7 +1009,7 @@ try { isReady(): boolean; ``` -Defined in: [packages/db/src/collection/index.ts:448](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L448) +Defined in: [packages/db/src/collection/index.ts:492](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L492) Check if the collection is ready for use Returns true if the collection has been marked as ready by its sync implementation @@ -830,7 +1043,7 @@ if (collection.isReady()) { keys(): IterableIterator; ``` -Defined in: [packages/db/src/collection/index.ts:500](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L500) +Defined in: [packages/db/src/collection/index.ts:565](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L565) Get all keys (virtual derived state) @@ -850,7 +1063,7 @@ Get all keys (virtual derived state) map(callbackfn): U[]; ``` -Defined in: [packages/db/src/collection/index.ts:556](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L556) +Defined in: [packages/db/src/collection/index.ts:621](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L621) Create a new array with the results of calling a function for each entry in the collection @@ -882,7 +1095,7 @@ Create a new array with the results of calling a function for each entry in the off(event, callback): void; ``` -Defined in: [packages/db/src/collection/index.ts:967](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L967) +Defined in: [packages/db/src/collection/index.ts:1032](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1032) Unsubscribe from a collection event @@ -929,7 +1142,7 @@ Unsubscribe from a collection event on(event, callback): () => void; ``` -Defined in: [packages/db/src/collection/index.ts:947](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L947) +Defined in: [packages/db/src/collection/index.ts:1012](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1012) Subscribe to a collection event @@ -982,7 +1195,7 @@ Subscribe to a collection event once(event, callback): () => void; ``` -Defined in: [packages/db/src/collection/index.ts:957](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L957) +Defined in: [packages/db/src/collection/index.ts:1022](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1022) Subscribe to a collection event once @@ -1032,13 +1245,18 @@ Subscribe to a collection event once ### onFirstReady() ```ts -onFirstReady(callback): void; +onFirstReady(callback): () => void; ``` -Defined in: [packages/db/src/collection/index.ts:432](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L432) +Defined in: [packages/db/src/collection/index.ts:476](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L476) Register a callback to be executed when the collection first becomes ready Useful for preloading collections +Every callback queued before the transition runs. Because ready state is +established first, callbacks registered during or after delivery run +immediately. If one throws, the collection remains ready. Direct sync +startup rethrows the first failure; preload resolves from ready state. +Cleanup discards pending callbacks without invoking them. #### Parameters @@ -1050,6 +1268,12 @@ Function to call when the collection first becomes ready #### Returns +```ts +(): void; +``` + +##### Returns + `void` #### Example @@ -1073,7 +1297,7 @@ collection.onFirstReady(() => { preload(): Promise; ``` -Defined in: [packages/db/src/collection/index.ts:472](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L472) +Defined in: [packages/db/src/collection/index.ts:537](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L537) Preload the collection data by starting sync if not already started Multiple concurrent calls will share the same promise @@ -1094,7 +1318,7 @@ Multiple concurrent calls will share the same promise removeIndex(indexOrId): boolean; ``` -Defined in: [packages/db/src/collection/index.ts:612](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L612) +Defined in: [packages/db/src/collection/index.ts:677](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L677) Removes an index created with createIndex. Returns true when an index existed and was removed. @@ -1125,10 +1349,11 @@ as invalid after removal. startSyncImmediate(): void; ``` -Defined in: [packages/db/src/collection/index.ts:464](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L464) +Defined in: [packages/db/src/collection/index.ts:509](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L509) Start sync immediately - internal method for compiled queries This bypasses lazy loading for special cases like live query results +Throws during active cleanup; restart after cleanup completes instead. #### Returns @@ -1146,7 +1371,7 @@ This bypasses lazy loading for special cases like live query results stateWhenReady(): Promise>>; ``` -Defined in: [packages/db/src/collection/index.ts:834](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L834) +Defined in: [packages/db/src/collection/index.ts:899](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L899) Gets the current state of the collection as a Map, but only resolves when data is available Waits for the first sync commit to complete before resolving @@ -1169,7 +1394,7 @@ Promise that resolves to a Map containing all items in the collection subscribeChanges(callback, options): CollectionSubscription; ``` -Defined in: [packages/db/src/collection/index.ts:935](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L935) +Defined in: [packages/db/src/collection/index.ts:1000](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1000) Subscribe to changes in the collection @@ -1248,7 +1473,7 @@ const subscription = collection.subscribeChanges((changes) => { toArrayWhenReady(): Promise[]>; ``` -Defined in: [packages/db/src/collection/index.ts:859](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L859) +Defined in: [packages/db/src/collection/index.ts:924](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L924) Gets the current state of the collection as an Array, but only resolves when data is available Waits for the first sync commit to complete before resolving @@ -1273,7 +1498,7 @@ Promise that resolves to an Array containing all items in the collection update(key, callback): Transaction; ``` -Defined in: [packages/db/src/collection/index.ts:729](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L729) +Defined in: [packages/db/src/collection/index.ts:794](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L794) Updates one or more items in the collection using a callback function @@ -1348,7 +1573,7 @@ update( callback): Transaction; ``` -Defined in: [packages/db/src/collection/index.ts:735](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L735) +Defined in: [packages/db/src/collection/index.ts:800](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L800) Updates one or more items in the collection using a callback function @@ -1426,7 +1651,7 @@ try { update(id, callback): Transaction; ``` -Defined in: [packages/db/src/collection/index.ts:742](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L742) +Defined in: [packages/db/src/collection/index.ts:807](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L807) Updates one or more items in the collection using a callback function @@ -1501,7 +1726,7 @@ update( callback): Transaction; ``` -Defined in: [packages/db/src/collection/index.ts:748](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L748) +Defined in: [packages/db/src/collection/index.ts:813](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L813) Updates one or more items in the collection using a callback function @@ -1582,7 +1807,7 @@ validateData( key?): T; ``` -Defined in: [packages/db/src/collection/index.ts:635](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L635) +Defined in: [packages/db/src/collection/index.ts:700](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L700) Validates the data against the schema @@ -1616,7 +1841,7 @@ Validates the data against the schema values(): IterableIterator>; ``` -Defined in: [packages/db/src/collection/index.ts:507](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L507) +Defined in: [packages/db/src/collection/index.ts:572](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L572) Get all values (virtual derived state) @@ -1636,7 +1861,7 @@ Get all values (virtual derived state) waitFor(event, timeout?): Promise; ``` -Defined in: [packages/db/src/collection/index.ts:977](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L977) +Defined in: [packages/db/src/collection/index.ts:1042](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L1042) Wait for a collection event diff --git a/docs/reference/interfaces/CollectionConfig.md b/docs/reference/interfaces/CollectionConfig.md index 6d72de9c21..f5e506de5e 100644 --- a/docs/reference/interfaces/CollectionConfig.md +++ b/docs/reference/interfaces/CollectionConfig.md @@ -5,7 +5,7 @@ title: CollectionConfig # Interface: CollectionConfig\ -Defined in: [packages/db/src/types.ts:747](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L747) +Defined in: [packages/db/src/types.ts:839](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L839) ## Extends @@ -37,7 +37,7 @@ Defined in: [packages/db/src/types.ts:747](https://github.com/TanStack/db/blob/m optional autoIndex: "off" | "eager"; ``` -Defined in: [packages/db/src/types.ts:571](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L571) +Defined in: [packages/db/src/types.ts:663](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L663) Auto-indexing mode for the collection. When enabled, indexes will be automatically created for simple where expressions. @@ -66,7 +66,7 @@ When enabled, indexes will be automatically created for simple where expressions optional compare: (x, y) => number; ``` -Defined in: [packages/db/src/types.ts:596](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L596) +Defined in: [packages/db/src/types.ts:688](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L688) Optional function to compare two items. This is used to order the items in the collection. @@ -110,7 +110,7 @@ compare: (x, y) => x.createdAt.getTime() - y.createdAt.getTime() optional defaultIndexType: IndexConstructor; ``` -Defined in: [packages/db/src/types.ts:585](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L585) +Defined in: [packages/db/src/types.ts:677](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L677) Default index type to use when creating indexes without an explicit type. Required for auto-indexing. Import from '@tanstack/db'. @@ -138,7 +138,7 @@ const collection = createCollection({ optional defaultStringCollation: StringCollationConfig; ``` -Defined in: [packages/db/src/types.ts:742](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L742) +Defined in: [packages/db/src/types.ts:834](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L834) Specifies how to compare data in the collection. This should be configured to match data ordering on the backend. @@ -157,7 +157,7 @@ E.g., when using the Electric DB collection these options optional gcTime: number; ``` -Defined in: [packages/db/src/types.ts:550](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L550) +Defined in: [packages/db/src/types.ts:642](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L642) Time in milliseconds after which the collection will be garbage collected when it has no active subscribers. Defaults to 5 minutes (300000ms). @@ -174,7 +174,7 @@ when it has no active subscribers. Defaults to 5 minutes (300000ms). getKey: (item) => TKey; ``` -Defined in: [packages/db/src/types.ts:545](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L545) +Defined in: [packages/db/src/types.ts:637](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L637) Function to extract the ID from an object This is required for update/delete operations which now only accept IDs @@ -212,7 +212,7 @@ getKey: (item) => item.uuid optional id: string; ``` -Defined in: [packages/db/src/types.ts:534](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L534) +Defined in: [packages/db/src/types.ts:626](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L626) #### Inherited from @@ -226,7 +226,7 @@ Defined in: [packages/db/src/types.ts:534](https://github.com/TanStack/db/blob/m optional onDelete: DeleteMutationFn; ``` -Defined in: [packages/db/src/types.ts:734](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L734) +Defined in: [packages/db/src/types.ts:826](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L826) Optional asynchronous handler function called before a delete operation @@ -294,7 +294,7 @@ onDelete: async ({ transaction, collection }) => { optional onInsert: InsertMutationFn; ``` -Defined in: [packages/db/src/types.ts:647](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L647) +Defined in: [packages/db/src/types.ts:739](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L739) Optional asynchronous handler function called before an insert operation @@ -361,7 +361,7 @@ onInsert: async ({ transaction, collection }) => { optional onUpdate: UpdateMutationFn; ``` -Defined in: [packages/db/src/types.ts:691](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L691) +Defined in: [packages/db/src/types.ts:783](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L783) Optional asynchronous handler function called before an update operation @@ -429,7 +429,7 @@ onUpdate: async ({ transaction, collection }) => { optional schema: TSchema; ``` -Defined in: [packages/db/src/types.ts:535](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L535) +Defined in: [packages/db/src/types.ts:627](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L627) #### Inherited from @@ -443,7 +443,7 @@ Defined in: [packages/db/src/types.ts:535](https://github.com/TanStack/db/blob/m optional startSync: boolean; ``` -Defined in: [packages/db/src/types.ts:561](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L561) +Defined in: [packages/db/src/types.ts:653](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L653) Whether to eagerly start syncing on collection creation. When true, syncing begins immediately. When false, syncing starts when the first subscriber attaches. @@ -470,7 +470,7 @@ false sync: SyncConfig; ``` -Defined in: [packages/db/src/types.ts:753](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L753) +Defined in: [packages/db/src/types.ts:845](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L845) *** @@ -480,7 +480,7 @@ Defined in: [packages/db/src/types.ts:753](https://github.com/TanStack/db/blob/m optional syncMode: SyncMode; ``` -Defined in: [packages/db/src/types.ts:605](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L605) +Defined in: [packages/db/src/types.ts:697](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L697) The mode of sync to use for the collection. @@ -506,7 +506,7 @@ The exact implementation of the sync mode is up to the sync implementation. optional utils: TUtils; ``` -Defined in: [packages/db/src/types.ts:744](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L744) +Defined in: [packages/db/src/types.ts:836](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L836) #### Inherited from diff --git a/docs/reference/interfaces/CollectionLike.md b/docs/reference/interfaces/CollectionLike.md index cc5593c1d1..c79c8c0341 100644 --- a/docs/reference/interfaces/CollectionLike.md +++ b/docs/reference/interfaces/CollectionLike.md @@ -32,7 +32,7 @@ for the change events system to work compareOptions: StringCollationConfig; ``` -Defined in: [packages/db/src/collection/index.ts:643](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L643) +Defined in: [packages/db/src/collection/index.ts:708](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L708) #### Inherited from @@ -46,7 +46,7 @@ Defined in: [packages/db/src/collection/index.ts:643](https://github.com/TanStac id: string; ``` -Defined in: [packages/db/src/collection/index.ts:279](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L279) +Defined in: [packages/db/src/collection/index.ts:283](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L283) #### Inherited from @@ -60,7 +60,7 @@ Defined in: [packages/db/src/collection/index.ts:279](https://github.com/TanStac indexes: Map>; ``` -Defined in: [packages/db/src/collection/index.ts:628](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L628) +Defined in: [packages/db/src/collection/index.ts:693](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L693) #### Inherited from @@ -76,7 +76,7 @@ Pick.indexes entries(): IterableIterator<[TKey, WithVirtualProps]>; ``` -Defined in: [packages/db/src/collection/index.ts:519](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L519) +Defined in: [packages/db/src/collection/index.ts:584](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L584) Get all entries (virtual derived state) @@ -100,7 +100,7 @@ get(key): | undefined; ``` -Defined in: [packages/db/src/collection/index.ts:479](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L479) +Defined in: [packages/db/src/collection/index.ts:544](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L544) Get the current value for a key (virtual derived state) @@ -129,7 +129,7 @@ Pick.get has(key): boolean; ``` -Defined in: [packages/db/src/collection/index.ts:486](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L486) +Defined in: [packages/db/src/collection/index.ts:551](https://github.com/TanStack/db/blob/main/packages/db/src/collection/index.ts#L551) Check if a key exists in the collection (virtual derived state) diff --git a/docs/reference/interfaces/Context.md b/docs/reference/interfaces/Context.md index 2caa57dab1..0ca928a441 100644 --- a/docs/reference/interfaces/Context.md +++ b/docs/reference/interfaces/Context.md @@ -5,7 +5,7 @@ title: Context # Interface: Context -Defined in: [packages/db/src/query/builder/types.ts:37](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L37) +Defined in: [packages/db/src/query/builder/types.ts:44](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L44) Context - The central state container for query builder operations @@ -16,7 +16,8 @@ This interface tracks all the information needed to build and type-check queries - `schema`: Current available tables (expands with joins, contracts with subqueries) **Query State**: -- `fromSourceName`: Which table was used in `from()` - needed for optionality logic +- `fromSourceName`: Which table was used in `from()` or the first + `unionAll()` source - needed for optionality logic - `hasJoins`: Whether any joins have been added (affects result type inference) - `joinTypes`: Maps table aliases to their join types for optionality calculations @@ -36,7 +37,7 @@ The context evolves through the query builder chain: baseSchema: ContextSchema; ``` -Defined in: [packages/db/src/query/builder/types.ts:39](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L39) +Defined in: [packages/db/src/query/builder/types.ts:46](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L46) *** @@ -46,7 +47,17 @@ Defined in: [packages/db/src/query/builder/types.ts:39](https://github.com/TanSt fromSourceName: string; ``` -Defined in: [packages/db/src/query/builder/types.ts:43](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L43) +Defined in: [packages/db/src/query/builder/types.ts:52](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L52) + +*** + +### fromSourceNames? + +```ts +optional fromSourceNames: readonly string[]; +``` + +Defined in: [packages/db/src/query/builder/types.ts:54](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L54) *** @@ -56,7 +67,7 @@ Defined in: [packages/db/src/query/builder/types.ts:43](https://github.com/TanSt optional hasJoins: boolean; ``` -Defined in: [packages/db/src/query/builder/types.ts:45](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L45) +Defined in: [packages/db/src/query/builder/types.ts:58](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L58) *** @@ -66,7 +77,17 @@ Defined in: [packages/db/src/query/builder/types.ts:45](https://github.com/TanSt optional hasResult: true; ``` -Defined in: [packages/db/src/query/builder/types.ts:54](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L54) +Defined in: [packages/db/src/query/builder/types.ts:67](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L67) + +*** + +### hasUnionFrom? + +```ts +optional hasUnionFrom: true; +``` + +Defined in: [packages/db/src/query/builder/types.ts:56](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L56) *** @@ -76,7 +97,17 @@ Defined in: [packages/db/src/query/builder/types.ts:54](https://github.com/TanSt optional joinTypes: Record; ``` -Defined in: [packages/db/src/query/builder/types.ts:47](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L47) +Defined in: [packages/db/src/query/builder/types.ts:60](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L60) + +*** + +### refsSchema? + +```ts +optional refsSchema: ContextSchema; +``` + +Defined in: [packages/db/src/query/builder/types.ts:50](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L50) *** @@ -86,7 +117,7 @@ Defined in: [packages/db/src/query/builder/types.ts:47](https://github.com/TanSt optional result: any; ``` -Defined in: [packages/db/src/query/builder/types.ts:52](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L52) +Defined in: [packages/db/src/query/builder/types.ts:65](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L65) *** @@ -96,7 +127,7 @@ Defined in: [packages/db/src/query/builder/types.ts:52](https://github.com/TanSt schema: ContextSchema; ``` -Defined in: [packages/db/src/query/builder/types.ts:41](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L41) +Defined in: [packages/db/src/query/builder/types.ts:48](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L48) *** @@ -106,4 +137,4 @@ Defined in: [packages/db/src/query/builder/types.ts:41](https://github.com/TanSt optional singleResult: boolean; ``` -Defined in: [packages/db/src/query/builder/types.ts:56](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L56) +Defined in: [packages/db/src/query/builder/types.ts:69](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L69) diff --git a/docs/reference/interfaces/CreateLiveQueryObserverOptions.md b/docs/reference/interfaces/CreateLiveQueryObserverOptions.md new file mode 100644 index 0000000000..81515296f9 --- /dev/null +++ b/docs/reference/interfaces/CreateLiveQueryObserverOptions.md @@ -0,0 +1,71 @@ +--- +id: CreateLiveQueryObserverOptions +title: CreateLiveQueryObserverOptions +--- + +# Interface: CreateLiveQueryObserverOptions + +Defined in: [packages/db/src/live-query-observer.ts:851](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L851) + +## Properties + +### client? + +```ts +optional client: DbClient; +``` + +Defined in: [packages/db/src/live-query-observer.ts:868](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L868) + +DbClient cache that owns SSR snapshots for this query identity. + +*** + +### mode? + +```ts +optional mode: "granular" | "wholesale"; +``` + +Defined in: [packages/db/src/live-query-observer.ts:866](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L866) + +How subscribers consume the observer: + +- `granular` (default): subscribers apply the delivered `ChangeMessage[]` + deltas to their own keyed state (Vue/Svelte/Solid). The observer + subscribes with initial state and seeds late subscribers, so every + subscriber converges from deltas alone. +- `wholesale`: subscribers treat notifications as a wake-up and re-read + `getSnapshot()` (React/Angular). The observer subscribes WITHOUT initial + state, preserving those adapters' loading policy — no snapshot request, + so no unfiltered `loadSubset` against on-demand collections. Nothing is + delivered synchronously during `subscribe`, which keeps + `useSyncExternalStore`-style consumers safe by construction. + +*** + +### onPreload()? + +```ts +optional onPreload: () => void; +``` + +Defined in: [packages/db/src/live-query-observer.ts:872](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L872) + +Resume framework-deferred query sources before a server preload. + +#### Returns + +`void` + +*** + +### queryHash? + +```ts +optional queryHash: string; +``` + +Defined in: [packages/db/src/live-query-observer.ts:870](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L870) + +Stable live-query identity used for dehydration and hydration. diff --git a/docs/reference/interfaces/CreateLiveQueryWindowControllerOptions.md b/docs/reference/interfaces/CreateLiveQueryWindowControllerOptions.md new file mode 100644 index 0000000000..8768599dd6 --- /dev/null +++ b/docs/reference/interfaces/CreateLiveQueryWindowControllerOptions.md @@ -0,0 +1,48 @@ +--- +id: CreateLiveQueryWindowControllerOptions +title: CreateLiveQueryWindowControllerOptions +--- + +# Interface: CreateLiveQueryWindowControllerOptions + +Defined in: [packages/db/src/live-query-window-controller.ts:504](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L504) + +**`Internal`** + +This contract is unstable while RFC #1623 is being implemented. + +## Properties + +### initialPageCount? + +```ts +optional initialPageCount: number; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:510](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L510) + +Committed pages to preserve when a framework binding changes page shape. + +*** + +### initialPageParam? + +```ts +optional initialPageParam: number; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:508](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L508) + +Value of the first page's `pageParam` (default 0). + +*** + +### pageSize? + +```ts +optional pageSize: number; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:506](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L506) + +Rows per page (default 20). Invalid values use the default. diff --git a/docs/reference/interfaces/CreateOptimisticActionsOptions.md b/docs/reference/interfaces/CreateOptimisticActionsOptions.md index c76b7a4d0e..e9610d1d57 100644 --- a/docs/reference/interfaces/CreateOptimisticActionsOptions.md +++ b/docs/reference/interfaces/CreateOptimisticActionsOptions.md @@ -5,7 +5,7 @@ title: CreateOptimisticActionsOptions # Interface: CreateOptimisticActionsOptions\ -Defined in: [packages/db/src/types.ts:181](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L181) +Defined in: [packages/db/src/types.ts:187](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L187) Options for the createOptimisticAction helper @@ -31,7 +31,7 @@ Options for the createOptimisticAction helper optional autoCommit: boolean; ``` -Defined in: [packages/db/src/types.ts:172](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L172) +Defined in: [packages/db/src/types.ts:178](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L178) #### Inherited from @@ -45,7 +45,7 @@ Defined in: [packages/db/src/types.ts:172](https://github.com/TanStack/db/blob/m optional id: string; ``` -Defined in: [packages/db/src/types.ts:170](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L170) +Defined in: [packages/db/src/types.ts:176](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L176) Unique identifier for the transaction @@ -61,7 +61,7 @@ Unique identifier for the transaction optional metadata: Record; ``` -Defined in: [packages/db/src/types.ts:175](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L175) +Defined in: [packages/db/src/types.ts:181](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L181) Custom metadata to associate with the transaction @@ -77,7 +77,7 @@ Custom metadata to associate with the transaction mutationFn: (vars, params) => Promise; ``` -Defined in: [packages/db/src/types.ts:188](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L188) +Defined in: [packages/db/src/types.ts:194](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L194) Function to execute the mutation on the server @@ -103,7 +103,7 @@ Function to execute the mutation on the server onMutate: (vars) => void; ``` -Defined in: [packages/db/src/types.ts:186](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L186) +Defined in: [packages/db/src/types.ts:192](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L192) Function to apply optimistic updates locally before the mutation completes diff --git a/docs/reference/interfaces/CurrentStateAsChangesOptions.md b/docs/reference/interfaces/CurrentStateAsChangesOptions.md index accb752744..ec524ae281 100644 --- a/docs/reference/interfaces/CurrentStateAsChangesOptions.md +++ b/docs/reference/interfaces/CurrentStateAsChangesOptions.md @@ -5,7 +5,7 @@ title: CurrentStateAsChangesOptions # Interface: CurrentStateAsChangesOptions -Defined in: [packages/db/src/types.ts:880](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L880) +Defined in: [packages/db/src/types.ts:979](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L979) Options for getting current state as changes @@ -17,7 +17,7 @@ Options for getting current state as changes optional limit: number; ``` -Defined in: [packages/db/src/types.ts:884](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L884) +Defined in: [packages/db/src/types.ts:983](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L983) *** @@ -27,7 +27,7 @@ Defined in: [packages/db/src/types.ts:884](https://github.com/TanStack/db/blob/m optional optimizedOnly: boolean; ``` -Defined in: [packages/db/src/types.ts:885](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L885) +Defined in: [packages/db/src/types.ts:984](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L984) *** @@ -37,7 +37,7 @@ Defined in: [packages/db/src/types.ts:885](https://github.com/TanStack/db/blob/m optional orderBy: OrderBy; ``` -Defined in: [packages/db/src/types.ts:883](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L883) +Defined in: [packages/db/src/types.ts:982](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L982) *** @@ -47,6 +47,6 @@ Defined in: [packages/db/src/types.ts:883](https://github.com/TanStack/db/blob/m optional where: BasicExpression; ``` -Defined in: [packages/db/src/types.ts:882](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L882) +Defined in: [packages/db/src/types.ts:981](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L981) Pre-compiled expression for filtering the current state diff --git a/docs/reference/interfaces/Effect.md b/docs/reference/interfaces/Effect.md index a332987ee4..58d2ca7450 100644 --- a/docs/reference/interfaces/Effect.md +++ b/docs/reference/interfaces/Effect.md @@ -5,7 +5,7 @@ title: Effect # Interface: Effect -Defined in: [packages/db/src/query/effect.ts:134](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L134) +Defined in: [packages/db/src/query/effect.ts:141](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L141) Handle returned by createEffect @@ -17,9 +17,10 @@ Handle returned by createEffect dispose: () => Promise; ``` -Defined in: [packages/db/src/query/effect.ts:136](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L136) +Defined in: [packages/db/src/query/effect.ts:146](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L146) -Dispose the effect. Returns a promise that resolves when in-flight handlers complete. +Dispose the effect and await in-flight handlers. Calls during one cleanup +attempt, including calls from abort/release callbacks, share its outcome. #### Returns @@ -33,6 +34,6 @@ Dispose the effect. Returns a promise that resolves when in-flight handlers comp readonly disposed: boolean; ``` -Defined in: [packages/db/src/query/effect.ts:138](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L138) +Defined in: [packages/db/src/query/effect.ts:148](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L148) Whether this effect has been disposed diff --git a/docs/reference/interfaces/EffectConfig.md b/docs/reference/interfaces/EffectConfig.md index 7a8a915caf..03dd7cebb0 100644 --- a/docs/reference/interfaces/EffectConfig.md +++ b/docs/reference/interfaces/EffectConfig.md @@ -5,7 +5,7 @@ title: EffectConfig # Interface: EffectConfig\ -Defined in: [packages/db/src/query/effect.ts:93](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L93) +Defined in: [packages/db/src/query/effect.ts:100](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L100) Effect configuration @@ -27,7 +27,7 @@ Effect configuration optional id: string; ``` -Defined in: [packages/db/src/query/effect.ts:98](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L98) +Defined in: [packages/db/src/query/effect.ts:105](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L105) Optional ID for debugging/tracing @@ -39,7 +39,7 @@ Optional ID for debugging/tracing optional onBatch: EffectBatchHandler; ``` -Defined in: [packages/db/src/query/effect.ts:113](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L113) +Defined in: [packages/db/src/query/effect.ts:120](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L120) Called once per graph run with all delta events from that batch @@ -51,7 +51,7 @@ Called once per graph run with all delta events from that batch optional onEnter: EffectEventHandler; ``` -Defined in: [packages/db/src/query/effect.ts:104](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L104) +Defined in: [packages/db/src/query/effect.ts:111](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L111) Called once for each row entering the query result @@ -63,7 +63,7 @@ Called once for each row entering the query result optional onError: (error, event) => void; ``` -Defined in: [packages/db/src/query/effect.ts:116](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L116) +Defined in: [packages/db/src/query/effect.ts:123](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L123) Error handler for exceptions thrown by effect callbacks @@ -89,7 +89,7 @@ Error handler for exceptions thrown by effect callbacks optional onExit: EffectEventHandler; ``` -Defined in: [packages/db/src/query/effect.ts:110](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L110) +Defined in: [packages/db/src/query/effect.ts:117](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L117) Called once for each row exiting the query result @@ -101,7 +101,7 @@ Called once for each row exiting the query result optional onSourceError: (error) => void; ``` -Defined in: [packages/db/src/query/effect.ts:123](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L123) +Defined in: [packages/db/src/query/effect.ts:130](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L130) Called when a source collection enters an error or cleaned-up state. The effect is automatically disposed after this callback fires. @@ -125,7 +125,7 @@ If not provided, the error is logged to console.error. optional onUpdate: EffectEventHandler; ``` -Defined in: [packages/db/src/query/effect.ts:107](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L107) +Defined in: [packages/db/src/query/effect.ts:114](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L114) Called once for each row updating within the query result @@ -137,7 +137,7 @@ Called once for each row updating within the query result query: EffectQueryInput; ``` -Defined in: [packages/db/src/query/effect.ts:101](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L101) +Defined in: [packages/db/src/query/effect.ts:108](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L108) Query to watch for deltas @@ -149,7 +149,7 @@ Query to watch for deltas optional skipInitial: boolean; ``` -Defined in: [packages/db/src/query/effect.ts:130](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L130) +Defined in: [packages/db/src/query/effect.ts:137](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L137) Skip deltas during initial collection load. Defaults to false (process all deltas including initial sync). diff --git a/docs/reference/interfaces/EffectContext.md b/docs/reference/interfaces/EffectContext.md index db13c1ab3a..f1adda0836 100644 --- a/docs/reference/interfaces/EffectContext.md +++ b/docs/reference/interfaces/EffectContext.md @@ -5,7 +5,7 @@ title: EffectContext # Interface: EffectContext -Defined in: [packages/db/src/query/effect.ts:67](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L67) +Defined in: [packages/db/src/query/effect.ts:74](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L74) Context passed to effect handlers @@ -17,7 +17,7 @@ Context passed to effect handlers effectId: string; ``` -Defined in: [packages/db/src/query/effect.ts:69](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L69) +Defined in: [packages/db/src/query/effect.ts:76](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L76) ID of this effect (auto-generated if not provided) @@ -29,6 +29,6 @@ ID of this effect (auto-generated if not provided) signal: AbortSignal; ``` -Defined in: [packages/db/src/query/effect.ts:71](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L71) +Defined in: [packages/db/src/query/effect.ts:78](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L78) Aborted when effect.dispose() is called diff --git a/docs/reference/interfaces/IndexInterface.md b/docs/reference/interfaces/IndexInterface.md index 43400d877c..bc3e4afd71 100644 --- a/docs/reference/interfaces/IndexInterface.md +++ b/docs/reference/interfaces/IndexInterface.md @@ -5,7 +5,7 @@ title: IndexInterface # Interface: IndexInterface\ -Defined in: [packages/db/src/indexes/base-index.ts:28](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L28) +Defined in: [packages/db/src/indexes/base-index.ts:54](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L54) ## Type Parameters @@ -21,7 +21,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:28](https://github.com/TanSta add: (key, item) => void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:31](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L31) +Defined in: [packages/db/src/indexes/base-index.ts:57](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L57) #### Parameters @@ -45,7 +45,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:31](https://github.com/TanSta build: (entries) => void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:35](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L35) +Defined in: [packages/db/src/indexes/base-index.ts:61](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L61) #### Parameters @@ -59,13 +59,37 @@ Defined in: [packages/db/src/indexes/base-index.ts:35](https://github.com/TanSta *** +### canOptimizeRangeFor()? + +```ts +optional canOptimizeRangeFor: (value) => boolean; +``` + +Defined in: [packages/db/src/indexes/base-index.ts:105](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L105) + +Whether the live values in this index share the predicate operand's +relational domain. Mixed domains can sort differently in the index and +WHERE evaluator, which can make a range lookup omit matching rows. + +#### Parameters + +##### value + +`unknown` + +#### Returns + +`boolean` + +*** + ### clear() ```ts clear: () => void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:36](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L36) +Defined in: [packages/db/src/indexes/base-index.ts:62](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L62) #### Returns @@ -79,7 +103,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:36](https://github.com/TanSta equalityLookup: (value) => Set; ``` -Defined in: [packages/db/src/indexes/base-index.ts:40](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L40) +Defined in: [packages/db/src/indexes/base-index.ts:66](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L66) #### Parameters @@ -93,27 +117,13 @@ Defined in: [packages/db/src/indexes/base-index.ts:40](https://github.com/TanSta *** -### getStats() - -```ts -getStats: () => IndexStats; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:75](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L75) - -#### Returns - -[`IndexStats`](IndexStats.md) - -*** - ### inArrayLookup() ```ts inArrayLookup: (values) => Set; ``` -Defined in: [packages/db/src/indexes/base-index.ts:41](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L41) +Defined in: [packages/db/src/indexes/base-index.ts:67](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L67) #### Parameters @@ -133,7 +143,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:41](https://github.com/TanSta lookup: (operation, value) => Set; ``` -Defined in: [packages/db/src/indexes/base-index.ts:38](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L38) +Defined in: [packages/db/src/indexes/base-index.ts:64](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L64) #### Parameters @@ -157,7 +167,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:38](https://github.com/TanSta matchesCompareOptions: (compareOptions) => boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:72](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L72) +Defined in: [packages/db/src/indexes/base-index.ts:108](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L108) #### Parameters @@ -177,7 +187,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:72](https://github.com/TanSta matchesDirection: (direction) => boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:73](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L73) +Defined in: [packages/db/src/indexes/base-index.ts:109](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L109) #### Parameters @@ -197,7 +207,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:73](https://github.com/TanSta matchesField: (fieldPath) => boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:71](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L71) +Defined in: [packages/db/src/indexes/base-index.ts:107](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L107) #### Parameters @@ -217,7 +227,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:71](https://github.com/TanSta rangeQuery: (options) => Set; ``` -Defined in: [packages/db/src/indexes/base-index.ts:43](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L43) +Defined in: [packages/db/src/indexes/base-index.ts:69](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L69) #### Parameters @@ -237,7 +247,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:43](https://github.com/TanSta rangeQueryReversed: (options) => Set; ``` -Defined in: [packages/db/src/indexes/base-index.ts:44](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L44) +Defined in: [packages/db/src/indexes/base-index.ts:70](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L70) #### Parameters @@ -257,7 +267,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:44](https://github.com/TanSta remove: (key, item) => void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:32](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L32) +Defined in: [packages/db/src/indexes/base-index.ts:58](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L58) #### Parameters @@ -281,7 +291,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:32](https://github.com/TanSta supports: (operation) => boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:69](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L69) +Defined in: [packages/db/src/indexes/base-index.ts:89](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L89) #### Parameters @@ -301,7 +311,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:69](https://github.com/TanSta take: (n, from, filterFn?) => TKey[]; ``` -Defined in: [packages/db/src/indexes/base-index.ts:46](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L46) +Defined in: [packages/db/src/indexes/base-index.ts:72](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L72) #### Parameters @@ -311,7 +321,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:46](https://github.com/TanSta ##### from -`TKey` +`unknown` ##### filterFn? @@ -329,7 +339,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:46](https://github.com/TanSta takeFromStart: (n, filterFn?) => TKey[]; ``` -Defined in: [packages/db/src/indexes/base-index.ts:51](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L51) +Defined in: [packages/db/src/indexes/base-index.ts:77](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L77) #### Parameters @@ -353,7 +363,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:51](https://github.com/TanSta takeReversed: (n, from, filterFn?) => TKey[]; ``` -Defined in: [packages/db/src/indexes/base-index.ts:52](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L52) +Defined in: [packages/db/src/indexes/base-index.ts:78](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L78) #### Parameters @@ -363,7 +373,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:52](https://github.com/TanSta ##### from -`TKey` +`unknown` ##### filterFn? @@ -381,7 +391,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:52](https://github.com/TanSta takeReversedFromEnd: (n, filterFn?) => TKey[]; ``` -Defined in: [packages/db/src/indexes/base-index.ts:57](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L57) +Defined in: [packages/db/src/indexes/base-index.ts:83](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L83) #### Parameters @@ -405,7 +415,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:57](https://github.com/TanSta update: (key, oldItem, newItem) => void; ``` -Defined in: [packages/db/src/indexes/base-index.ts:33](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L33) +Defined in: [packages/db/src/indexes/base-index.ts:59](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L59) #### Parameters @@ -427,22 +437,6 @@ Defined in: [packages/db/src/indexes/base-index.ts:33](https://github.com/TanSta ## Accessors -### indexedKeysSet - -#### Get Signature - -```ts -get indexedKeysSet(): Set; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:66](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L66) - -##### Returns - -`Set`\<`TKey`\> - -*** - ### keyCount #### Get Signature @@ -451,7 +445,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:66](https://github.com/TanSta get keyCount(): number; ``` -Defined in: [packages/db/src/indexes/base-index.ts:62](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L62) +Defined in: [packages/db/src/indexes/base-index.ts:88](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L88) ##### Returns @@ -459,48 +453,22 @@ Defined in: [packages/db/src/indexes/base-index.ts:62](https://github.com/TanSta *** -### orderedEntriesArray - -#### Get Signature - -```ts -get orderedEntriesArray(): [any, Set][]; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:63](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L63) - -##### Returns - -\[`any`, `Set`\<`TKey`\>\][] - -*** - -### orderedEntriesArrayReversed +### supportsRangeOptimization #### Get Signature ```ts -get orderedEntriesArrayReversed(): [any, Set][]; +get supportsRangeOptimization(): boolean; ``` -Defined in: [packages/db/src/indexes/base-index.ts:64](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L64) - -##### Returns - -\[`any`, `Set`\<`TKey`\>\][] +Defined in: [packages/db/src/indexes/base-index.ts:98](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L98) -*** - -### valueMapData - -#### Get Signature - -```ts -get valueMapData(): Map>; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:67](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L67) +Whether range lookups (gt/gte/lt/lte) on this index can be trusted to +return every matching key. Range traversal relies on the index ordering, so +it is unsafe when the index uses a custom comparator, whose order may not +match the WHERE evaluator's relational operators. Callers must fall back to +a full scan when this is `false`. ##### Returns -`Map`\<`any`, `Set`\<`TKey`\>\> +`boolean` diff --git a/docs/reference/interfaces/IndexStats.md b/docs/reference/interfaces/IndexStats.md deleted file mode 100644 index ff3db9e4b2..0000000000 --- a/docs/reference/interfaces/IndexStats.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -id: IndexStats -title: IndexStats ---- - -# Interface: IndexStats - -Defined in: [packages/db/src/indexes/base-index.ts:21](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L21) - -Statistics about index usage and performance - -## Properties - -### averageLookupTime - -```ts -readonly averageLookupTime: number; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:24](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L24) - -*** - -### entryCount - -```ts -readonly entryCount: number; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:22](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L22) - -*** - -### lastUpdated - -```ts -readonly lastUpdated: Date; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:25](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L25) - -*** - -### lookupCount - -```ts -readonly lookupCount: number; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:23](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L23) diff --git a/docs/reference/interfaces/InsertConfig.md b/docs/reference/interfaces/InsertConfig.md index 7e4007630b..2c4ef0f9c7 100644 --- a/docs/reference/interfaces/InsertConfig.md +++ b/docs/reference/interfaces/InsertConfig.md @@ -5,7 +5,7 @@ title: InsertConfig # Interface: InsertConfig -Defined in: [packages/db/src/types.ts:439](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L439) +Defined in: [packages/db/src/types.ts:530](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L530) ## Properties @@ -15,7 +15,7 @@ Defined in: [packages/db/src/types.ts:439](https://github.com/TanStack/db/blob/m optional metadata: Record; ``` -Defined in: [packages/db/src/types.ts:440](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L440) +Defined in: [packages/db/src/types.ts:531](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L531) *** @@ -25,6 +25,6 @@ Defined in: [packages/db/src/types.ts:440](https://github.com/TanStack/db/blob/m optional optimistic: boolean; ``` -Defined in: [packages/db/src/types.ts:442](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L442) +Defined in: [packages/db/src/types.ts:533](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L533) Whether to apply optimistic updates immediately. Defaults to true. diff --git a/docs/reference/interfaces/LiveQueryCollectionConfig.md b/docs/reference/interfaces/LiveQueryCollectionConfig.md index 6958f4a382..5b782dc135 100644 --- a/docs/reference/interfaces/LiveQueryCollectionConfig.md +++ b/docs/reference/interfaces/LiveQueryCollectionConfig.md @@ -5,7 +5,7 @@ title: LiveQueryCollectionConfig # Interface: LiveQueryCollectionConfig\ -Defined in: [packages/db/src/query/live/types.ts:59](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L59) +Defined in: [packages/db/src/query/live/types.ts:65](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L65) Configuration interface for live query collection options @@ -49,7 +49,7 @@ const config: LiveQueryCollectionConfig = { optional defaultStringCollation: StringCollationConfig; ``` -Defined in: [packages/db/src/query/live/types.ts:115](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L115) +Defined in: [packages/db/src/query/live/types.ts:121](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L121) Optional compare options for string sorting. If provided, these will be used instead of inheriting from the FROM collection. @@ -62,7 +62,7 @@ If provided, these will be used instead of inheriting from the FROM collection. optional gcTime: number; ``` -Defined in: [packages/db/src/query/live/types.ts:104](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L104) +Defined in: [packages/db/src/query/live/types.ts:110](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L110) GC time for the collection @@ -74,7 +74,7 @@ GC time for the collection optional getKey: (item) => string | number; ``` -Defined in: [packages/db/src/query/live/types.ts:82](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L82) +Defined in: [packages/db/src/query/live/types.ts:88](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L88) Function to extract the key from result items If not provided, defaults to using the key from the D2 stream @@ -97,7 +97,7 @@ If not provided, defaults to using the key from the D2 stream optional id: string; ``` -Defined in: [packages/db/src/query/live/types.ts:67](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L67) +Defined in: [packages/db/src/query/live/types.ts:73](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L73) Unique identifier for the collection If not provided, defaults to `live-query-${number}` with auto-incrementing number @@ -110,7 +110,7 @@ If not provided, defaults to `live-query-${number}` with auto-incrementing numbe optional onDelete: DeleteMutationFn; ``` -Defined in: [packages/db/src/query/live/types.ts:94](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L94) +Defined in: [packages/db/src/query/live/types.ts:100](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L100) *** @@ -120,7 +120,7 @@ Defined in: [packages/db/src/query/live/types.ts:94](https://github.com/TanStack optional onInsert: InsertMutationFn; ``` -Defined in: [packages/db/src/query/live/types.ts:92](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L92) +Defined in: [packages/db/src/query/live/types.ts:98](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L98) Optional mutation handlers @@ -132,7 +132,7 @@ Optional mutation handlers optional onUpdate: UpdateMutationFn; ``` -Defined in: [packages/db/src/query/live/types.ts:93](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L93) +Defined in: [packages/db/src/query/live/types.ts:99](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L99) *** @@ -144,7 +144,7 @@ query: | QueryBuilder & RootObjectResultConstraint; ``` -Defined in: [packages/db/src/query/live/types.ts:72](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L72) +Defined in: [packages/db/src/query/live/types.ts:78](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L78) Query builder function that defines the live query @@ -156,7 +156,7 @@ Query builder function that defines the live query optional schema: undefined; ``` -Defined in: [packages/db/src/query/live/types.ts:87](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L87) +Defined in: [packages/db/src/query/live/types.ts:93](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L93) Optional schema for validation @@ -168,7 +168,7 @@ Optional schema for validation optional singleResult: true; ``` -Defined in: [packages/db/src/query/live/types.ts:109](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L109) +Defined in: [packages/db/src/query/live/types.ts:115](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L115) If enabled the collection will return a single object instead of an array @@ -180,6 +180,6 @@ If enabled the collection will return a single object instead of an array optional startSync: boolean; ``` -Defined in: [packages/db/src/query/live/types.ts:99](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L99) +Defined in: [packages/db/src/query/live/types.ts:105](https://github.com/TanStack/db/blob/main/packages/db/src/query/live/types.ts#L105) Start sync / the query immediately diff --git a/docs/reference/interfaces/LiveQueryObserver.md b/docs/reference/interfaces/LiveQueryObserver.md new file mode 100644 index 0000000000..718a7e7eb9 --- /dev/null +++ b/docs/reference/interfaces/LiveQueryObserver.md @@ -0,0 +1,160 @@ +--- +id: LiveQueryObserver +title: LiveQueryObserver +--- + +# Interface: LiveQueryObserver\ + +Defined in: [packages/db/src/live-query-observer.ts:72](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L72) + +**`Internal`** + +Wraps a resolved live-query `Collection` (or `null` for a disabled query) with +the shared lifecycle every framework adapter needs: start sync on first +subscribe, subscribe to changes and status transitions, expose a stable +snapshot for wholesale consumers, and deliver the raw change set for +granular consumers. + +Input resolution (query fn / config / collection / disabled) stays in the +adapter — it is framework-reactive. The observer owns everything after the +input is resolved to a concrete collection. + + Unstable contract for TanStack DB's official framework adapters — +not a public extension point yet; may change in any release. + +## Type Parameters + +### T + +`T` *extends* `object` + +### TKey + +`TKey` *extends* `string` \| `number` + +## Properties + +### dehydrate() + +```ts +dehydrate: () => DehydratedLiveQueryResult; +``` + +Defined in: [packages/db/src/live-query-observer.ts:92](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L92) + +Capture the ordered query result without serializing its source collections. + +#### Returns + +[`DehydratedLiveQueryResult`](../type-aliases/DehydratedLiveQueryResult.md)\<`T`, `TKey`\> + +*** + +### dispose() + +```ts +dispose: () => void; +``` + +Defined in: [packages/db/src/live-query-observer.ts:94](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L94) + +Idempotent teardown. + +#### Returns + +`void` + +*** + +### getError() + +```ts +getError: () => unknown; +``` + +Defined in: [packages/db/src/live-query-observer.ts:90](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L90) + +The transport or preload error for this query, if it has not produced data. + +#### Returns + +`unknown` + +*** + +### getServerSnapshot() + +```ts +getServerSnapshot: () => LiveQuerySnapshot; +``` + +Defined in: [packages/db/src/live-query-observer.ts:79](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L79) + +Stable server snapshot used by useSyncExternalStore-style adapters. + +#### Returns + +[`LiveQuerySnapshot`](LiveQuerySnapshot.md)\<`T`, `TKey`\> + +*** + +### getSnapshot() + +```ts +getSnapshot: () => LiveQuerySnapshot; +``` + +Defined in: [packages/db/src/live-query-observer.ts:77](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L77) + +Stable per-revision snapshot for wholesale materialization. + +#### Returns + +[`LiveQuerySnapshot`](LiveQuerySnapshot.md)\<`T`, `TKey`\> + +*** + +### preload() + +```ts +preload: () => Promise; +``` + +Defined in: [packages/db/src/live-query-observer.ts:88](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L88) + +Resolve once the collection has loaded its first data. + +#### Returns + +`Promise`\<`void`\> + +*** + +### subscribe() + +```ts +subscribe: (listener) => () => void; +``` + +Defined in: [packages/db/src/live-query-observer.ts:86](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L86) + +Subscribe to changes. The listener receives the change set (or `undefined` +for the synthetic notify a ready collection emits on attach). Granular +adapters apply the changes; wholesale adapters can ignore them and re-read +`getSnapshot()`. Returns an unsubscribe function. + +#### Parameters + +##### listener + +[`LiveQueryObserverListener`](../type-aliases/LiveQueryObserverListener.md)\<`T`, `TKey`\> + +#### Returns + +```ts +(): void; +``` + +##### Returns + +`void` diff --git a/docs/reference/interfaces/LiveQuerySnapshot.md b/docs/reference/interfaces/LiveQuerySnapshot.md new file mode 100644 index 0000000000..58d622bbc5 --- /dev/null +++ b/docs/reference/interfaces/LiveQuerySnapshot.md @@ -0,0 +1,152 @@ +--- +id: LiveQuerySnapshot +title: LiveQuerySnapshot +--- + +# Interface: LiveQuerySnapshot\ + +Defined in: [packages/db/src/live-query-observer.ts:19](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L19) + +The canonical, adapter-agnostic view of a live query at a point in time. + +`getSnapshot()` returns a stable object identity that only changes when the +query changes, so `useSyncExternalStore`-style consumers can compare by +reference. Each snapshot owns a captured view of `state`/`data`, so reading +an older snapshot cannot expose rows from a later revision. + +## Type Parameters + +### T + +`T` *extends* `object` + +### TKey + +`TKey` *extends* `string` \| `number` + +## Properties + +### collection + +```ts +collection: + | Collection, T> + | undefined; +``` + +Defined in: [packages/db/src/live-query-observer.ts:28](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L28) + +The underlying collection, or `undefined` when disabled. + +*** + +### data + +```ts +data: T | readonly T[] | undefined; +``` + +Defined in: [packages/db/src/live-query-observer.ts:26](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L26) + +Ordered results (single row for `findOne`), or `undefined` when disabled. + +*** + +### isCleanedUp + +```ts +isCleanedUp: boolean; +``` + +Defined in: [packages/db/src/live-query-observer.ts:45](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L45) + +*** + +### isEnabled + +```ts +isEnabled: boolean; +``` + +Defined in: [packages/db/src/live-query-observer.ts:46](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L46) + +*** + +### isError + +```ts +isError: boolean; +``` + +Defined in: [packages/db/src/live-query-observer.ts:44](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L44) + +*** + +### isIdle + +```ts +isIdle: boolean; +``` + +Defined in: [packages/db/src/live-query-observer.ts:43](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L43) + +*** + +### isLoading + +```ts +isLoading: boolean; +``` + +Defined in: [packages/db/src/live-query-observer.ts:41](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L41) + +*** + +### isReady + +```ts +isReady: boolean; +``` + +Defined in: [packages/db/src/live-query-observer.ts:42](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L42) + +*** + +### layoutRevision + +```ts +layoutRevision: number; +``` + +Defined in: [packages/db/src/live-query-observer.ts:39](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L39) + +Monotonic counter bumped whenever the visible layout (the ordered key +sequence) changes — membership, ordering, or an order-only move. Lets +consumers detect a reorder that changed no row value (which `data`/`state` +identity alone can't express once row values are structurally shared). + +It is NOT in lockstep with snapshot identity: a value-only update produces a +new snapshot while `layoutRevision` stays put. A `layoutRevision` change +always accompanies a new snapshot, but not vice versa. + +*** + +### state + +```ts +state: ReadonlyMap | undefined; +``` + +Defined in: [packages/db/src/live-query-observer.ts:24](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L24) + +Keyed results, or `undefined` for a disabled query. + +*** + +### status + +```ts +status: CollectionStatus | "disabled"; +``` + +Defined in: [packages/db/src/live-query-observer.ts:40](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L40) diff --git a/docs/reference/interfaces/LiveQueryStatusFlags.md b/docs/reference/interfaces/LiveQueryStatusFlags.md new file mode 100644 index 0000000000..e687c972b0 --- /dev/null +++ b/docs/reference/interfaces/LiveQueryStatusFlags.md @@ -0,0 +1,60 @@ +--- +id: LiveQueryStatusFlags +title: LiveQueryStatusFlags +--- + +# Interface: LiveQueryStatusFlags + +Defined in: [packages/db/src/live-query-adapter.ts:45](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-adapter.ts#L45) + +The derived boolean status flags every adapter exposes for a query. + +## Properties + +### isCleanedUp + +```ts +isCleanedUp: boolean; +``` + +Defined in: [packages/db/src/live-query-adapter.ts:50](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-adapter.ts#L50) + +*** + +### isError + +```ts +isError: boolean; +``` + +Defined in: [packages/db/src/live-query-adapter.ts:49](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-adapter.ts#L49) + +*** + +### isIdle + +```ts +isIdle: boolean; +``` + +Defined in: [packages/db/src/live-query-adapter.ts:48](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-adapter.ts#L48) + +*** + +### isLoading + +```ts +isLoading: boolean; +``` + +Defined in: [packages/db/src/live-query-adapter.ts:46](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-adapter.ts#L46) + +*** + +### isReady + +```ts +isReady: boolean; +``` + +Defined in: [packages/db/src/live-query-adapter.ts:47](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-adapter.ts#L47) diff --git a/docs/reference/interfaces/LiveQueryWindowController.md b/docs/reference/interfaces/LiveQueryWindowController.md new file mode 100644 index 0000000000..adaca969fd --- /dev/null +++ b/docs/reference/interfaces/LiveQueryWindowController.md @@ -0,0 +1,122 @@ +--- +id: LiveQueryWindowController +title: LiveQueryWindowController +--- + +# Interface: LiveQueryWindowController\ + +Defined in: [packages/db/src/live-query-window-controller.ts:514](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L514) + +**`Internal`** + +This contract is unstable while RFC #1623 is being implemented. + +## Type Parameters + +### T + +`T` *extends* `object` + +### TKey + +`TKey` *extends* `string` \| `number` + +## Properties + +### dispose() + +```ts +dispose: () => void; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:525](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L525) + +#### Returns + +`void` + +*** + +### fetchNextPage() + +```ts +fetchNextPage: () => Promise; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:521](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L521) + +Load one more page, resolving only after that page is committed. + +#### Returns + +`Promise`\<`void`\> + +*** + +### getSnapshot() + +```ts +getSnapshot: () => LiveQueryWindowSnapshot; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:518](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L518) + +#### Returns + +[`LiveQueryWindowSnapshot`](LiveQueryWindowSnapshot.md)\<`T`, `TKey`\> + +*** + +### preload() + +```ts +preload: () => Promise; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:524](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L524) + +#### Returns + +`Promise`\<`void`\> + +*** + +### reset() + +```ts +reset: () => Promise; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:523](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L523) + +Reset to the first page, resolving after the smaller window is accepted. + +#### Returns + +`Promise`\<`void`\> + +*** + +### subscribe() + +```ts +subscribe: (listener) => () => void; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:519](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L519) + +#### Parameters + +##### listener + +() => `void` + +#### Returns + +```ts +(): void; +``` + +##### Returns + +`void` diff --git a/docs/reference/interfaces/LiveQueryWindowSnapshot.md b/docs/reference/interfaces/LiveQueryWindowSnapshot.md new file mode 100644 index 0000000000..eb79c721bc --- /dev/null +++ b/docs/reference/interfaces/LiveQueryWindowSnapshot.md @@ -0,0 +1,186 @@ +--- +id: LiveQueryWindowSnapshot +title: LiveQueryWindowSnapshot +--- + +# Interface: LiveQueryWindowSnapshot\ + +Defined in: [packages/db/src/live-query-window-controller.ts:477](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L477) + +**`Internal`** + +A page-windowed view of a live query at a point in time. + + This contract is unstable while RFC #1623 is being implemented. + +## Type Parameters + +### T + +`T` *extends* `object` + +### TKey + +`TKey` *extends* `string` \| `number` + +## Properties + +### collection + +```ts +collection: + | Collection, T> + | undefined; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:493](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L493) + +*** + +### data + +```ts +data: readonly T[]; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:482](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L482) + +Rows across all committed pages, with the peek-ahead row removed. + +*** + +### error + +```ts +error: unknown; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:490](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L490) + +The last pagination failure, cleared when a retry begins. + +*** + +### hasNextPage + +```ts +hasNextPage: boolean; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:487](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L487) + +*** + +### isCleanedUp + +```ts +isCleanedUp: boolean; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:499](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L499) + +*** + +### isEnabled + +```ts +isEnabled: boolean; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:500](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L500) + +*** + +### isError + +```ts +isError: boolean; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:498](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L498) + +*** + +### isFetchingNextPage + +```ts +isFetchingNextPage: boolean; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:488](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L488) + +*** + +### isIdle + +```ts +isIdle: boolean; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:497](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L497) + +*** + +### isLoading + +```ts +isLoading: boolean; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:495](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L495) + +*** + +### isReady + +```ts +isReady: boolean; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:496](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L496) + +*** + +### pageParams + +```ts +pageParams: readonly number[]; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:486](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L486) + +`initialPageParam + i` for each committed page. + +*** + +### pages + +```ts +pages: readonly readonly T[][]; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:484](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L484) + +Rows grouped into committed pages of `pageSize`. + +*** + +### state + +```ts +state: ReadonlyMap | undefined; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:492](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L492) + +Keyed results for the physical window, or `undefined` when disabled. + +*** + +### status + +```ts +status: CollectionStatus | "disabled"; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:494](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L494) diff --git a/docs/reference/interfaces/LocalOnlyCollectionConfig.md b/docs/reference/interfaces/LocalOnlyCollectionConfig.md index e948a015e3..0e72a1aafd 100644 --- a/docs/reference/interfaces/LocalOnlyCollectionConfig.md +++ b/docs/reference/interfaces/LocalOnlyCollectionConfig.md @@ -5,7 +5,7 @@ title: LocalOnlyCollectionConfig # Interface: LocalOnlyCollectionConfig\ -Defined in: [packages/db/src/local-only.ts:22](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L22) +Defined in: [packages/db/src/local-only.ts:24](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L24) Configuration interface for Local-only collection options @@ -41,7 +41,7 @@ The type of the key returned by `getKey` optional autoIndex: "off" | "eager"; ``` -Defined in: [packages/db/src/types.ts:571](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L571) +Defined in: [packages/db/src/types.ts:663](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L663) Auto-indexing mode for the collection. When enabled, indexes will be automatically created for simple where expressions. @@ -70,7 +70,7 @@ When enabled, indexes will be automatically created for simple where expressions optional compare: (x, y) => number; ``` -Defined in: [packages/db/src/types.ts:596](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L596) +Defined in: [packages/db/src/types.ts:688](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L688) Optional function to compare two items. This is used to order the items in the collection. @@ -116,7 +116,7 @@ Omit.compare optional defaultIndexType: IndexConstructor; ``` -Defined in: [packages/db/src/types.ts:585](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L585) +Defined in: [packages/db/src/types.ts:677](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L677) Default index type to use when creating indexes without an explicit type. Required for auto-indexing. Import from '@tanstack/db'. @@ -146,7 +146,7 @@ Omit.defaultIndexType optional defaultStringCollation: StringCollationConfig; ``` -Defined in: [packages/db/src/types.ts:742](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L742) +Defined in: [packages/db/src/types.ts:834](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L834) Specifies how to compare data in the collection. This should be configured to match data ordering on the backend. @@ -167,7 +167,7 @@ Omit.defaultStringCollation getKey: (item) => TKey; ``` -Defined in: [packages/db/src/types.ts:545](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L545) +Defined in: [packages/db/src/types.ts:637](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L637) Function to extract the ID from an object This is required for update/delete operations which now only accept IDs @@ -207,7 +207,7 @@ Omit.getKey optional id: string; ``` -Defined in: [packages/db/src/types.ts:534](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L534) +Defined in: [packages/db/src/types.ts:626](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L626) #### Inherited from @@ -221,7 +221,7 @@ Defined in: [packages/db/src/types.ts:534](https://github.com/TanStack/db/blob/m optional initialData: T[]; ``` -Defined in: [packages/db/src/local-only.ts:34](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L34) +Defined in: [packages/db/src/local-only.ts:36](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L36) Optional initial data to populate the collection with on creation This data will be applied during the initial sync process @@ -234,7 +234,7 @@ This data will be applied during the initial sync process optional onDelete: DeleteMutationFn; ``` -Defined in: [packages/db/src/types.ts:734](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L734) +Defined in: [packages/db/src/types.ts:826](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L826) Optional asynchronous handler function called before a delete operation @@ -304,7 +304,7 @@ Omit.onDelete optional onInsert: InsertMutationFn; ``` -Defined in: [packages/db/src/types.ts:647](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L647) +Defined in: [packages/db/src/types.ts:739](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L739) Optional asynchronous handler function called before an insert operation @@ -373,7 +373,7 @@ Omit.onInsert optional onUpdate: UpdateMutationFn; ``` -Defined in: [packages/db/src/types.ts:691](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L691) +Defined in: [packages/db/src/types.ts:783](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L783) Optional asynchronous handler function called before an update operation @@ -443,7 +443,7 @@ Omit.onUpdate optional schema: TSchema; ``` -Defined in: [packages/db/src/types.ts:535](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L535) +Defined in: [packages/db/src/types.ts:627](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L627) #### Inherited from @@ -459,7 +459,7 @@ Omit.schema optional syncMode: SyncMode; ``` -Defined in: [packages/db/src/types.ts:605](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L605) +Defined in: [packages/db/src/types.ts:697](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L697) The mode of sync to use for the collection. @@ -485,7 +485,7 @@ The exact implementation of the sync mode is up to the sync implementation. optional utils: LocalOnlyCollectionUtils; ``` -Defined in: [packages/db/src/types.ts:744](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L744) +Defined in: [packages/db/src/types.ts:836](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L836) #### Inherited from diff --git a/docs/reference/interfaces/LocalOnlyCollectionUtils.md b/docs/reference/interfaces/LocalOnlyCollectionUtils.md index c86fbc07b0..0849582f14 100644 --- a/docs/reference/interfaces/LocalOnlyCollectionUtils.md +++ b/docs/reference/interfaces/LocalOnlyCollectionUtils.md @@ -5,7 +5,7 @@ title: LocalOnlyCollectionUtils # Interface: LocalOnlyCollectionUtils -Defined in: [packages/db/src/local-only.ts:40](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L40) +Defined in: [packages/db/src/local-only.ts:42](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L42) Local-only collection utilities type @@ -27,7 +27,7 @@ Local-only collection utilities type acceptMutations: (transaction) => void; ``` -Defined in: [packages/db/src/local-only.ts:58](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L58) +Defined in: [packages/db/src/local-only.ts:60](https://github.com/TanStack/db/blob/main/packages/db/src/local-only.ts#L60) Accepts mutations from a transaction that belong to this collection and persists them. This should be called in your transaction's mutationFn to persist local-only data. diff --git a/docs/reference/interfaces/LocalStorageCollectionConfig.md b/docs/reference/interfaces/LocalStorageCollectionConfig.md index f6884ecefb..ea71d17820 100644 --- a/docs/reference/interfaces/LocalStorageCollectionConfig.md +++ b/docs/reference/interfaces/LocalStorageCollectionConfig.md @@ -5,7 +5,7 @@ title: LocalStorageCollectionConfig # Interface: LocalStorageCollectionConfig\ -Defined in: [packages/db/src/local-storage.ts:58](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L58) +Defined in: [packages/db/src/local-storage.ts:60](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L60) Configuration interface for localStorage collection options @@ -41,7 +41,7 @@ The type of the key returned by `getKey` optional autoIndex: "off" | "eager"; ``` -Defined in: [packages/db/src/types.ts:571](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L571) +Defined in: [packages/db/src/types.ts:663](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L663) Auto-indexing mode for the collection. When enabled, indexes will be automatically created for simple where expressions. @@ -70,7 +70,7 @@ When enabled, indexes will be automatically created for simple where expressions optional compare: (x, y) => number; ``` -Defined in: [packages/db/src/types.ts:596](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L596) +Defined in: [packages/db/src/types.ts:688](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L688) Optional function to compare two items. This is used to order the items in the collection. @@ -114,7 +114,7 @@ compare: (x, y) => x.createdAt.getTime() - y.createdAt.getTime() optional defaultIndexType: IndexConstructor; ``` -Defined in: [packages/db/src/types.ts:585](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L585) +Defined in: [packages/db/src/types.ts:677](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L677) Default index type to use when creating indexes without an explicit type. Required for auto-indexing. Import from '@tanstack/db'. @@ -142,7 +142,7 @@ const collection = createCollection({ optional defaultStringCollation: StringCollationConfig; ``` -Defined in: [packages/db/src/types.ts:742](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L742) +Defined in: [packages/db/src/types.ts:834](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L834) Specifies how to compare data in the collection. This should be configured to match data ordering on the backend. @@ -161,7 +161,7 @@ E.g., when using the Electric DB collection these options optional gcTime: number; ``` -Defined in: [packages/db/src/types.ts:550](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L550) +Defined in: [packages/db/src/types.ts:642](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L642) Time in milliseconds after which the collection will be garbage collected when it has no active subscribers. Defaults to 5 minutes (300000ms). @@ -178,7 +178,7 @@ when it has no active subscribers. Defaults to 5 minutes (300000ms). getKey: (item) => TKey; ``` -Defined in: [packages/db/src/types.ts:545](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L545) +Defined in: [packages/db/src/types.ts:637](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L637) Function to extract the ID from an object This is required for update/delete operations which now only accept IDs @@ -216,7 +216,7 @@ getKey: (item) => item.uuid optional id: string; ``` -Defined in: [packages/db/src/types.ts:534](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L534) +Defined in: [packages/db/src/types.ts:626](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L626) #### Inherited from @@ -230,7 +230,7 @@ Defined in: [packages/db/src/types.ts:534](https://github.com/TanStack/db/blob/m optional onDelete: DeleteMutationFn; ``` -Defined in: [packages/db/src/types.ts:734](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L734) +Defined in: [packages/db/src/types.ts:826](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L826) Optional asynchronous handler function called before a delete operation @@ -298,7 +298,7 @@ onDelete: async ({ transaction, collection }) => { optional onInsert: InsertMutationFn; ``` -Defined in: [packages/db/src/types.ts:647](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L647) +Defined in: [packages/db/src/types.ts:739](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L739) Optional asynchronous handler function called before an insert operation @@ -365,7 +365,7 @@ onInsert: async ({ transaction, collection }) => { optional onUpdate: UpdateMutationFn; ``` -Defined in: [packages/db/src/types.ts:691](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L691) +Defined in: [packages/db/src/types.ts:783](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L783) Optional asynchronous handler function called before an update operation @@ -433,7 +433,7 @@ onUpdate: async ({ transaction, collection }) => { optional parser: Parser; ``` -Defined in: [packages/db/src/local-storage.ts:84](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L84) +Defined in: [packages/db/src/local-storage.ts:86](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L86) Parser to use for serializing and deserializing data to and from storage Defaults to JSON @@ -446,7 +446,7 @@ Defaults to JSON optional schema: TSchema; ``` -Defined in: [packages/db/src/types.ts:535](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L535) +Defined in: [packages/db/src/types.ts:627](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L627) #### Inherited from @@ -460,7 +460,7 @@ Defined in: [packages/db/src/types.ts:535](https://github.com/TanStack/db/blob/m optional startSync: boolean; ``` -Defined in: [packages/db/src/types.ts:561](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L561) +Defined in: [packages/db/src/types.ts:653](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L653) Whether to eagerly start syncing on collection creation. When true, syncing begins immediately. When false, syncing starts when the first subscriber attaches. @@ -487,7 +487,7 @@ false optional storage: StorageApi; ``` -Defined in: [packages/db/src/local-storage.ts:72](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L72) +Defined in: [packages/db/src/local-storage.ts:74](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L74) Storage API to use (defaults to window.localStorage) Can be any object that implements the Storage interface (e.g., sessionStorage) @@ -500,7 +500,7 @@ Can be any object that implements the Storage interface (e.g., sessionStorage) optional storageEventApi: StorageEventApi; ``` -Defined in: [packages/db/src/local-storage.ts:78](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L78) +Defined in: [packages/db/src/local-storage.ts:80](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L80) Storage event API to use for cross-tab synchronization (defaults to window) Can be any object that implements addEventListener/removeEventListener for storage events @@ -513,7 +513,7 @@ Can be any object that implements addEventListener/removeEventListener for stora storageKey: string; ``` -Defined in: [packages/db/src/local-storage.ts:66](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L66) +Defined in: [packages/db/src/local-storage.ts:68](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L68) The key to use for storing the collection data in localStorage/sessionStorage @@ -525,7 +525,7 @@ The key to use for storing the collection data in localStorage/sessionStorage optional syncMode: SyncMode; ``` -Defined in: [packages/db/src/types.ts:605](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L605) +Defined in: [packages/db/src/types.ts:697](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L697) The mode of sync to use for the collection. @@ -551,7 +551,7 @@ The exact implementation of the sync mode is up to the sync implementation. optional utils: UtilsRecord; ``` -Defined in: [packages/db/src/types.ts:744](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L744) +Defined in: [packages/db/src/types.ts:836](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L836) #### Inherited from diff --git a/docs/reference/interfaces/LocalStorageCollectionUtils.md b/docs/reference/interfaces/LocalStorageCollectionUtils.md index 9472f18f60..6850be07e9 100644 --- a/docs/reference/interfaces/LocalStorageCollectionUtils.md +++ b/docs/reference/interfaces/LocalStorageCollectionUtils.md @@ -5,7 +5,7 @@ title: LocalStorageCollectionUtils # Interface: LocalStorageCollectionUtils -Defined in: [packages/db/src/local-storage.ts:100](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L100) +Defined in: [packages/db/src/local-storage.ts:102](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L102) LocalStorage collection utilities type @@ -27,7 +27,7 @@ LocalStorage collection utilities type acceptMutations: (transaction) => void; ``` -Defined in: [packages/db/src/local-storage.ts:120](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L120) +Defined in: [packages/db/src/local-storage.ts:122](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L122) Accepts mutations from a transaction that belong to this collection and persists them to localStorage. This should be called in your transaction's mutationFn to persist local-storage data. @@ -69,7 +69,7 @@ const tx = createTransaction({ clearStorage: ClearStorageFn; ``` -Defined in: [packages/db/src/local-storage.ts:101](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L101) +Defined in: [packages/db/src/local-storage.ts:103](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L103) *** @@ -79,4 +79,4 @@ Defined in: [packages/db/src/local-storage.ts:101](https://github.com/TanStack/d getStorageSize: GetStorageSizeFn; ``` -Defined in: [packages/db/src/local-storage.ts:102](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L102) +Defined in: [packages/db/src/local-storage.ts:104](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L104) diff --git a/docs/reference/interfaces/OperationConfig.md b/docs/reference/interfaces/OperationConfig.md index 3526e49cc1..1365e85456 100644 --- a/docs/reference/interfaces/OperationConfig.md +++ b/docs/reference/interfaces/OperationConfig.md @@ -5,7 +5,7 @@ title: OperationConfig # Interface: OperationConfig -Defined in: [packages/db/src/types.ts:433](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L433) +Defined in: [packages/db/src/types.ts:524](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L524) ## Properties @@ -15,7 +15,7 @@ Defined in: [packages/db/src/types.ts:433](https://github.com/TanStack/db/blob/m optional metadata: Record; ``` -Defined in: [packages/db/src/types.ts:434](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L434) +Defined in: [packages/db/src/types.ts:525](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L525) *** @@ -25,6 +25,6 @@ Defined in: [packages/db/src/types.ts:434](https://github.com/TanStack/db/blob/m optional optimistic: boolean; ``` -Defined in: [packages/db/src/types.ts:436](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L436) +Defined in: [packages/db/src/types.ts:527](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L527) Whether to apply optimistic updates immediately. Defaults to true. diff --git a/docs/reference/interfaces/ParseWhereOptions.md b/docs/reference/interfaces/ParseWhereOptions.md index 8a90e90142..5fee3544ef 100644 --- a/docs/reference/interfaces/ParseWhereOptions.md +++ b/docs/reference/interfaces/ParseWhereOptions.md @@ -87,6 +87,22 @@ optional avg: (...args) => T; `T` +##### caseWhen()? + +```ts +optional caseWhen: (...args) => T; +``` + +###### Parameters + +###### args + +...`any`[] + +###### Returns + +`T` + ##### coalesce()? ```ts @@ -135,6 +151,22 @@ optional count: (...args) => T; `T` +##### divide()? + +```ts +optional divide: (...args) => T; +``` + +###### Parameters + +###### args + +...`any`[] + +###### Returns + +`T` + ##### eq()? ```ts @@ -359,6 +391,22 @@ optional min: (...args) => T; `T` +##### multiply()? + +```ts +optional multiply: (...args) => T; +``` + +###### Parameters + +###### args + +...`any`[] + +###### Returns + +`T` + ##### not()? ```ts @@ -391,6 +439,22 @@ optional or: (...args) => T; `T` +##### subtract()? + +```ts +optional subtract: (...args) => T; +``` + +###### Parameters + +###### args + +...`any`[] + +###### Returns + +`T` + ##### sum()? ```ts diff --git a/docs/reference/interfaces/Parser.md b/docs/reference/interfaces/Parser.md index d1eafc24c7..fee8fb04a7 100644 --- a/docs/reference/interfaces/Parser.md +++ b/docs/reference/interfaces/Parser.md @@ -5,7 +5,7 @@ title: Parser # Interface: Parser -Defined in: [packages/db/src/local-storage.ts:47](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L47) +Defined in: [packages/db/src/local-storage.ts:49](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L49) ## Properties @@ -15,7 +15,7 @@ Defined in: [packages/db/src/local-storage.ts:47](https://github.com/TanStack/db parse: (data) => unknown; ``` -Defined in: [packages/db/src/local-storage.ts:48](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L48) +Defined in: [packages/db/src/local-storage.ts:50](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L50) #### Parameters @@ -35,7 +35,7 @@ Defined in: [packages/db/src/local-storage.ts:48](https://github.com/TanStack/db stringify: (data) => string; ``` -Defined in: [packages/db/src/local-storage.ts:49](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L49) +Defined in: [packages/db/src/local-storage.ts:51](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L51) #### Parameters diff --git a/docs/reference/interfaces/RangeQueryOptions.md b/docs/reference/interfaces/RangeQueryOptions.md index 204e822ba8..58abee7fa9 100644 --- a/docs/reference/interfaces/RangeQueryOptions.md +++ b/docs/reference/interfaces/RangeQueryOptions.md @@ -5,7 +5,7 @@ title: RangeQueryOptions # Interface: RangeQueryOptions -Defined in: [packages/db/src/indexes/basic-index.ts:14](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L14) +Defined in: [packages/db/src/indexes/basic-index.ts:20](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L20) Options for range queries @@ -17,7 +17,7 @@ Options for range queries optional from: any; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:15](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L15) +Defined in: [packages/db/src/indexes/basic-index.ts:21](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L21) *** @@ -27,7 +27,7 @@ Defined in: [packages/db/src/indexes/basic-index.ts:15](https://github.com/TanSt optional fromInclusive: boolean; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:17](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L17) +Defined in: [packages/db/src/indexes/basic-index.ts:23](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L23) *** @@ -37,7 +37,7 @@ Defined in: [packages/db/src/indexes/basic-index.ts:17](https://github.com/TanSt optional to: any; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:16](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L16) +Defined in: [packages/db/src/indexes/basic-index.ts:22](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L22) *** @@ -47,4 +47,4 @@ Defined in: [packages/db/src/indexes/basic-index.ts:16](https://github.com/TanSt optional toInclusive: boolean; ``` -Defined in: [packages/db/src/indexes/basic-index.ts:18](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L18) +Defined in: [packages/db/src/indexes/basic-index.ts:24](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/basic-index.ts#L24) diff --git a/docs/reference/interfaces/SubscribeChangesOptions.md b/docs/reference/interfaces/SubscribeChangesOptions.md index a981f44602..a84136da60 100644 --- a/docs/reference/interfaces/SubscribeChangesOptions.md +++ b/docs/reference/interfaces/SubscribeChangesOptions.md @@ -5,7 +5,7 @@ title: SubscribeChangesOptions # Interface: SubscribeChangesOptions\ -Defined in: [packages/db/src/types.ts:822](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L822) +Defined in: [packages/db/src/types.ts:914](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L914) Options for subscribing to collection changes @@ -27,7 +27,7 @@ Options for subscribing to collection changes optional includeInitialState: boolean; ``` -Defined in: [packages/db/src/types.ts:827](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L827) +Defined in: [packages/db/src/types.ts:919](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L919) Whether to include the current state as initial changes @@ -39,7 +39,7 @@ Whether to include the current state as initial changes optional limit: number; ``` -Defined in: [packages/db/src/types.ts:860](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L860) +Defined in: [packages/db/src/types.ts:952](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L952) **`Internal`** @@ -47,13 +47,37 @@ Optional limit to include in loadSubset for query-specific cache keys. *** +### onLoadSubsetError()? + +```ts +optional onLoadSubsetError: (event) => void; +``` + +Defined in: [packages/db/src/types.ts:960](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L960) + +**`Internal`** + +Receives subset-load failures scoped to this subscription. + +#### Parameters + +##### event + +[`SubscriptionLoadSubsetErrorEvent`](SubscriptionLoadSubsetErrorEvent.md) + +#### Returns + +`void` + +*** + ### onLoadSubsetResult()? ```ts optional onLoadSubsetResult: (result) => void; ``` -Defined in: [packages/db/src/types.ts:866](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L866) +Defined in: [packages/db/src/types.ts:958](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L958) **`Internal`** @@ -64,7 +88,7 @@ Allows the caller to directly track the loading promise for isReady status. ##### result -`true` | `Promise`\<`void`\> +[`LoadSubsetRequestResult`](../type-aliases/LoadSubsetRequestResult.md) #### Returns @@ -78,7 +102,7 @@ Allows the caller to directly track the loading promise for isReady status. optional onStatusChange: (event) => void; ``` -Defined in: [packages/db/src/types.ts:850](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L850) +Defined in: [packages/db/src/types.ts:942](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L942) **`Internal`** @@ -103,7 +127,7 @@ Registered BEFORE any snapshot is requested, ensuring no status transitions are optional orderBy: OrderBy; ``` -Defined in: [packages/db/src/types.ts:855](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L855) +Defined in: [packages/db/src/types.ts:947](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L947) **`Internal`** @@ -111,13 +135,47 @@ Optional orderBy to include in loadSubset for query-specific cache keys. *** +### truncateReplayPublication? + +```ts +optional truncateReplayPublication: object; +``` + +Defined in: [packages/db/src/types.ts:962](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L962) + +**`Internal`** + +Lets a live-query graph retain its last publication during replay. + +#### start() + +```ts +readonly start: () => void; +``` + +##### Returns + +`void` + +#### succeed() + +```ts +readonly succeed: () => void; +``` + +##### Returns + +`void` + +*** + ### where()? ```ts optional where: (row) => any; ``` -Defined in: [packages/db/src/types.ts:842](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L842) +Defined in: [packages/db/src/types.ts:934](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L934) Callback function for filtering changes using a row proxy. The callback receives a proxy object that records property access, @@ -151,6 +209,6 @@ collection.subscribeChanges(callback, { optional whereExpression: BasicExpression; ``` -Defined in: [packages/db/src/types.ts:844](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L844) +Defined in: [packages/db/src/types.ts:936](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L936) Pre-compiled expression for filtering changes diff --git a/docs/reference/interfaces/SubscribeChangesSnapshotOptions.md b/docs/reference/interfaces/SubscribeChangesSnapshotOptions.md index f60351b338..63adc2a555 100644 --- a/docs/reference/interfaces/SubscribeChangesSnapshotOptions.md +++ b/docs/reference/interfaces/SubscribeChangesSnapshotOptions.md @@ -5,7 +5,7 @@ title: SubscribeChangesSnapshotOptions # Interface: SubscribeChangesSnapshotOptions\ -Defined in: [packages/db/src/types.ts:869](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L869) +Defined in: [packages/db/src/types.ts:968](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L968) ## Extends @@ -29,7 +29,7 @@ Defined in: [packages/db/src/types.ts:869](https://github.com/TanStack/db/blob/m optional limit: number; ``` -Defined in: [packages/db/src/types.ts:874](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L874) +Defined in: [packages/db/src/types.ts:973](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L973) **`Internal`** @@ -41,13 +41,43 @@ Optional limit to include in loadSubset for query-specific cache keys. *** +### onLoadSubsetError()? + +```ts +optional onLoadSubsetError: (event) => void; +``` + +Defined in: [packages/db/src/types.ts:960](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L960) + +**`Internal`** + +Receives subset-load failures scoped to this subscription. + +#### Parameters + +##### event + +[`SubscriptionLoadSubsetErrorEvent`](SubscriptionLoadSubsetErrorEvent.md) + +#### Returns + +`void` + +#### Inherited from + +```ts +Omit.onLoadSubsetError +``` + +*** + ### onLoadSubsetResult()? ```ts optional onLoadSubsetResult: (result) => void; ``` -Defined in: [packages/db/src/types.ts:866](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L866) +Defined in: [packages/db/src/types.ts:958](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L958) **`Internal`** @@ -58,7 +88,7 @@ Allows the caller to directly track the loading promise for isReady status. ##### result -`true` | `Promise`\<`void`\> +[`LoadSubsetRequestResult`](../type-aliases/LoadSubsetRequestResult.md) #### Returns @@ -78,7 +108,7 @@ Omit.onLoadSubsetResult optional onStatusChange: (event) => void; ``` -Defined in: [packages/db/src/types.ts:850](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L850) +Defined in: [packages/db/src/types.ts:942](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L942) **`Internal`** @@ -109,7 +139,7 @@ Omit.onStatusChange optional orderBy: OrderBy; ``` -Defined in: [packages/db/src/types.ts:873](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L873) +Defined in: [packages/db/src/types.ts:972](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L972) **`Internal`** @@ -121,13 +151,53 @@ Optional orderBy to include in loadSubset for query-specific cache keys. *** +### truncateReplayPublication? + +```ts +optional truncateReplayPublication: object; +``` + +Defined in: [packages/db/src/types.ts:962](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L962) + +**`Internal`** + +Lets a live-query graph retain its last publication during replay. + +#### start() + +```ts +readonly start: () => void; +``` + +##### Returns + +`void` + +#### succeed() + +```ts +readonly succeed: () => void; +``` + +##### Returns + +`void` + +#### Inherited from + +```ts +Omit.truncateReplayPublication +``` + +*** + ### where()? ```ts optional where: (row) => any; ``` -Defined in: [packages/db/src/types.ts:842](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L842) +Defined in: [packages/db/src/types.ts:934](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L934) Callback function for filtering changes using a row proxy. The callback receives a proxy object that records property access, @@ -167,7 +237,7 @@ Omit.where optional whereExpression: BasicExpression; ``` -Defined in: [packages/db/src/types.ts:844](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L844) +Defined in: [packages/db/src/types.ts:936](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L936) Pre-compiled expression for filtering changes diff --git a/docs/reference/interfaces/Subscription.md b/docs/reference/interfaces/Subscription.md index b3ff2b6671..9ee70d14b3 100644 --- a/docs/reference/interfaces/Subscription.md +++ b/docs/reference/interfaces/Subscription.md @@ -5,7 +5,7 @@ title: Subscription # Interface: Subscription -Defined in: [packages/db/src/types.ts:254](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L254) +Defined in: [packages/db/src/types.ts:269](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L269) Public interface for a collection subscription Used by sync implementations to track subscription lifecycle @@ -16,13 +16,25 @@ Used by sync implementations to track subscription lifecycle ## Properties +### lastError + +```ts +readonly lastError: unknown; +``` + +Defined in: [packages/db/src/types.ts:273](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L273) + +Most recent subset-load failure observed by this subscription. + +*** + ### status ```ts readonly status: SubscriptionStatus; ``` -Defined in: [packages/db/src/types.ts:256](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L256) +Defined in: [packages/db/src/types.ts:271](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L271) Current status of the subscription @@ -34,7 +46,7 @@ Current status of the subscription protected clearListeners(): void; ``` -Defined in: [packages/db/src/event-emitter.ts:115](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L115) +Defined in: [packages/db/src/event-emitter.ts:156](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L156) Clear all listeners @@ -56,7 +68,7 @@ EventEmitter.clearListeners protected emitInner(event, eventPayload): void; ``` -Defined in: [packages/db/src/event-emitter.ts:96](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L96) +Defined in: [packages/db/src/event-emitter.ts:124](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L124) **`Internal`** @@ -95,13 +107,58 @@ EventEmitter.emitInner *** +### emitInnerWhile() + +```ts +protected emitInnerWhile( + event, + eventPayload, + isCurrent): void; +``` + +Defined in: [packages/db/src/event-emitter.ts:132](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L132) + +Emit until a reentrant callback invalidates the event being delivered. + +#### Type Parameters + +##### T + +`T` *extends* keyof [`SubscriptionEvents`](../type-aliases/SubscriptionEvents.md) + +#### Parameters + +##### event + +`T` + +##### eventPayload + +[`SubscriptionEvents`](../type-aliases/SubscriptionEvents.md)\[`T`\] + +##### isCurrent + +() => `boolean` + +#### Returns + +`void` + +#### Inherited from + +```ts +EventEmitter.emitInnerWhile +``` + +*** + ### off() ```ts off(event, callback): void; ``` -Defined in: [packages/db/src/event-emitter.ts:53](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L53) +Defined in: [packages/db/src/event-emitter.ts:72](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L72) Unsubscribe from an event @@ -143,7 +200,7 @@ EventEmitter.off on(event, callback): () => void; ``` -Defined in: [packages/db/src/event-emitter.ts:17](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L17) +Defined in: [packages/db/src/event-emitter.ts:21](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L21) Subscribe to an event @@ -193,7 +250,7 @@ EventEmitter.on once(event, callback): () => void; ``` -Defined in: [packages/db/src/event-emitter.ts:37](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L37) +Defined in: [packages/db/src/event-emitter.ts:50](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L50) Subscribe to an event once (automatically unsubscribes after first emission) @@ -243,7 +300,7 @@ EventEmitter.once waitFor(event, timeout?): Promise; ``` -Defined in: [packages/db/src/event-emitter.ts:66](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L66) +Defined in: [packages/db/src/event-emitter.ts:94](https://github.com/TanStack/db/blob/main/packages/db/src/event-emitter.ts#L94) Wait for an event to be emitted diff --git a/docs/reference/interfaces/SubscriptionLoadSubsetErrorEvent.md b/docs/reference/interfaces/SubscriptionLoadSubsetErrorEvent.md new file mode 100644 index 0000000000..ef10d20a56 --- /dev/null +++ b/docs/reference/interfaces/SubscriptionLoadSubsetErrorEvent.md @@ -0,0 +1,50 @@ +--- +id: SubscriptionLoadSubsetErrorEvent +title: SubscriptionLoadSubsetErrorEvent +--- + +# Interface: SubscriptionLoadSubsetErrorEvent + +Defined in: [packages/db/src/types.ts:239](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L239) + +Event emitted when a subset requested by this subscription fails to load. + +## Properties + +### error + +```ts +error: unknown; +``` + +Defined in: [packages/db/src/types.ts:243](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L243) + +*** + +### options + +```ts +options: LoadSubsetOptions; +``` + +Defined in: [packages/db/src/types.ts:242](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L242) + +*** + +### subscription + +```ts +subscription: Subscription; +``` + +Defined in: [packages/db/src/types.ts:241](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L241) + +*** + +### type + +```ts +type: "loadSubset:error"; +``` + +Defined in: [packages/db/src/types.ts:240](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L240) diff --git a/docs/reference/interfaces/SubscriptionStatusChangeEvent.md b/docs/reference/interfaces/SubscriptionStatusChangeEvent.md index 87c7e584a3..baffa9cbe6 100644 --- a/docs/reference/interfaces/SubscriptionStatusChangeEvent.md +++ b/docs/reference/interfaces/SubscriptionStatusChangeEvent.md @@ -5,7 +5,7 @@ title: SubscriptionStatusChangeEvent # Interface: SubscriptionStatusChangeEvent -Defined in: [packages/db/src/types.ts:215](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L215) +Defined in: [packages/db/src/types.ts:221](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L221) Event emitted when subscription status changes @@ -17,7 +17,7 @@ Event emitted when subscription status changes previousStatus: SubscriptionStatus; ``` -Defined in: [packages/db/src/types.ts:218](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L218) +Defined in: [packages/db/src/types.ts:224](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L224) *** @@ -27,7 +27,7 @@ Defined in: [packages/db/src/types.ts:218](https://github.com/TanStack/db/blob/m status: SubscriptionStatus; ``` -Defined in: [packages/db/src/types.ts:219](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L219) +Defined in: [packages/db/src/types.ts:225](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L225) *** @@ -37,7 +37,7 @@ Defined in: [packages/db/src/types.ts:219](https://github.com/TanStack/db/blob/m subscription: Subscription; ``` -Defined in: [packages/db/src/types.ts:217](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L217) +Defined in: [packages/db/src/types.ts:223](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L223) *** @@ -47,4 +47,4 @@ Defined in: [packages/db/src/types.ts:217](https://github.com/TanStack/db/blob/m type: "status:change"; ``` -Defined in: [packages/db/src/types.ts:216](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L216) +Defined in: [packages/db/src/types.ts:222](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L222) diff --git a/docs/reference/interfaces/SubscriptionStatusEvent.md b/docs/reference/interfaces/SubscriptionStatusEvent.md index 5a3aee984d..a8db29fb42 100644 --- a/docs/reference/interfaces/SubscriptionStatusEvent.md +++ b/docs/reference/interfaces/SubscriptionStatusEvent.md @@ -5,7 +5,7 @@ title: SubscriptionStatusEvent # Interface: SubscriptionStatusEvent\ -Defined in: [packages/db/src/types.ts:225](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L225) +Defined in: [packages/db/src/types.ts:231](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L231) Event emitted when subscription status changes to a specific status @@ -23,7 +23,7 @@ Event emitted when subscription status changes to a specific status previousStatus: SubscriptionStatus; ``` -Defined in: [packages/db/src/types.ts:228](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L228) +Defined in: [packages/db/src/types.ts:234](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L234) *** @@ -33,7 +33,7 @@ Defined in: [packages/db/src/types.ts:228](https://github.com/TanStack/db/blob/m status: T; ``` -Defined in: [packages/db/src/types.ts:229](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L229) +Defined in: [packages/db/src/types.ts:235](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L235) *** @@ -43,7 +43,7 @@ Defined in: [packages/db/src/types.ts:229](https://github.com/TanStack/db/blob/m subscription: Subscription; ``` -Defined in: [packages/db/src/types.ts:227](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L227) +Defined in: [packages/db/src/types.ts:233](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L233) *** @@ -53,4 +53,4 @@ Defined in: [packages/db/src/types.ts:227](https://github.com/TanStack/db/blob/m type: `status:${T}`; ``` -Defined in: [packages/db/src/types.ts:226](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L226) +Defined in: [packages/db/src/types.ts:232](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L232) diff --git a/docs/reference/interfaces/SubscriptionUnsubscribedEvent.md b/docs/reference/interfaces/SubscriptionUnsubscribedEvent.md index a62ce55f61..00829571da 100644 --- a/docs/reference/interfaces/SubscriptionUnsubscribedEvent.md +++ b/docs/reference/interfaces/SubscriptionUnsubscribedEvent.md @@ -5,7 +5,7 @@ title: SubscriptionUnsubscribedEvent # Interface: SubscriptionUnsubscribedEvent -Defined in: [packages/db/src/types.ts:235](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L235) +Defined in: [packages/db/src/types.ts:249](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L249) Event emitted when subscription is unsubscribed @@ -17,7 +17,7 @@ Event emitted when subscription is unsubscribed subscription: Subscription; ``` -Defined in: [packages/db/src/types.ts:237](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L237) +Defined in: [packages/db/src/types.ts:251](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L251) *** @@ -27,4 +27,4 @@ Defined in: [packages/db/src/types.ts:237](https://github.com/TanStack/db/blob/m type: "unsubscribed"; ``` -Defined in: [packages/db/src/types.ts:236](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L236) +Defined in: [packages/db/src/types.ts:250](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L250) diff --git a/docs/reference/interfaces/SyncConfig.md b/docs/reference/interfaces/SyncConfig.md index adc0e4bb1c..c30ef766a9 100644 --- a/docs/reference/interfaces/SyncConfig.md +++ b/docs/reference/interfaces/SyncConfig.md @@ -5,7 +5,7 @@ title: SyncConfig # Interface: SyncConfig\ -Defined in: [packages/db/src/types.ts:327](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L327) +Defined in: [packages/db/src/types.ts:387](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L387) ## Type Parameters @@ -19,13 +19,30 @@ Defined in: [packages/db/src/types.ts:327](https://github.com/TanStack/db/blob/m ## Properties +### exportSyncMeta()? + +```ts +optional exportSyncMeta: () => unknown; +``` + +Defined in: [packages/db/src/types.ts:431](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L431) + +Export adapter-specific metadata that lets hydration/persistence resume sync. +The payload shape is owned by the adapter. + +#### Returns + +`unknown` + +*** + ### getSyncMetadata()? ```ts optional getSyncMetadata: () => Record; ``` -Defined in: [packages/db/src/types.ts:350](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L350) +Defined in: [packages/db/src/types.ts:425](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L425) Get the sync metadata for insert operations @@ -37,13 +54,61 @@ Record containing relation information *** +### importSyncMeta()? + +```ts +optional importSyncMeta: (meta) => void; +``` + +Defined in: [packages/db/src/types.ts:436](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L436) + +Import adapter-specific metadata produced by exportSyncMeta. + +#### Parameters + +##### meta + +`unknown` + +#### Returns + +`void` + +*** + +### mergeSyncMeta()? + +```ts +optional mergeSyncMeta: (current, incoming) => unknown; +``` + +Defined in: [packages/db/src/types.ts:441](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L441) + +Merge two adapter-specific metadata payloads during hydration. + +#### Parameters + +##### current + +`unknown` + +##### incoming + +`unknown` + +#### Returns + +`unknown` + +*** + ### rowUpdateMode? ```ts optional rowUpdateMode: "full" | "partial"; ``` -Defined in: [packages/db/src/types.ts:359](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L359) +Defined in: [packages/db/src/types.ts:450](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L450) The row update mode used to sync to the collection. @@ -67,7 +132,7 @@ sync: (params) => | SyncConfigRes; ``` -Defined in: [packages/db/src/types.ts:331](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L331) +Defined in: [packages/db/src/types.ts:391](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L391) #### Parameters @@ -85,12 +150,29 @@ Begin a new sync transaction. ###### commit -() => `void` +(`signal?`) => [`SyncAppliedReceipt`](../type-aliases/SyncAppliedReceipt.md) + +Commit the active sync transaction in FIFO order. +Returns `true` when the writes and events are already visible. Otherwise +returns a receipt that resolves after they become visible. If collection +cleanup or an optional request abort abandons the transaction first, the +receipt rejects with an error named `AbortError`. +Pass a signal only for request-scoped work that must not publish after +cancellation. Aborting after application has no effect. + +###### markError + +(`error?`) => `void` + +Signal that initial sync failed before producing a usable snapshot. +When supplied, `error` is preserved as the rejection reason from `preload()`. ###### markReady () => `void` +Signal that a usable initial or recovered snapshot is available. + ###### metadata? [`SyncMetadataApi`](SyncMetadataApi.md)\<`TKey`\> diff --git a/docs/reference/interfaces/SyncMetadataApi.md b/docs/reference/interfaces/SyncMetadataApi.md index 15d6787dcb..430729c9e4 100644 --- a/docs/reference/interfaces/SyncMetadataApi.md +++ b/docs/reference/interfaces/SyncMetadataApi.md @@ -5,7 +5,7 @@ title: SyncMetadataApi # Interface: SyncMetadataApi\ -Defined in: [packages/db/src/types.ts:362](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L362) +Defined in: [packages/db/src/types.ts:453](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L453) ## Type Parameters @@ -21,7 +21,7 @@ Defined in: [packages/db/src/types.ts:362](https://github.com/TanStack/db/blob/m collection: object; ``` -Defined in: [packages/db/src/types.ts:370](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L370) +Defined in: [packages/db/src/types.ts:461](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L461) #### delete() @@ -99,7 +99,7 @@ set: (key, value) => void; row: object; ``` -Defined in: [packages/db/src/types.ts:365](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L365) +Defined in: [packages/db/src/types.ts:456](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L456) #### delete() diff --git a/docs/reference/interfaces/Transaction.md b/docs/reference/interfaces/Transaction.md index cffa506996..a3b78164be 100644 --- a/docs/reference/interfaces/Transaction.md +++ b/docs/reference/interfaces/Transaction.md @@ -5,7 +5,7 @@ title: Transaction # Interface: Transaction\ -Defined in: [packages/db/src/transactions.ts:208](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L208) +Defined in: [packages/db/src/transactions.ts:305](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L305) ## Type Parameters @@ -21,7 +21,7 @@ Defined in: [packages/db/src/transactions.ts:208](https://github.com/TanStack/db autoCommit: boolean; ``` -Defined in: [packages/db/src/transactions.ts:214](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L214) +Defined in: [packages/db/src/transactions.ts:323](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L323) *** @@ -31,7 +31,7 @@ Defined in: [packages/db/src/transactions.ts:214](https://github.com/TanStack/db createdAt: Date; ``` -Defined in: [packages/db/src/transactions.ts:215](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L215) +Defined in: [packages/db/src/transactions.ts:324](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L324) *** @@ -41,7 +41,7 @@ Defined in: [packages/db/src/transactions.ts:215](https://github.com/TanStack/db optional error: object; ``` -Defined in: [packages/db/src/transactions.ts:218](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L218) +Defined in: [packages/db/src/transactions.ts:327](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L327) #### error @@ -63,7 +63,7 @@ message: string; id: string; ``` -Defined in: [packages/db/src/transactions.ts:209](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L209) +Defined in: [packages/db/src/transactions.ts:306](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L306) *** @@ -73,7 +73,18 @@ Defined in: [packages/db/src/transactions.ts:209](https://github.com/TanStack/db isPersisted: Deferred>; ``` -Defined in: [packages/db/src/transactions.ts:213](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L213) +Defined in: [packages/db/src/transactions.ts:322](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L322) + +Deferred that settles when this transaction settles. + +Await `isPersisted.promise`, not `isPersisted` itself. The promise resolves +when the transaction completes successfully and rejects if the transaction +fails or is rolled back. + +For non-empty commits, the mutation function is the normal settlement +boundary. This does not inherently prove that a backend has uploaded, +confirmed, or read back the write unless the mutation function waits for +that backend observation before returning. *** @@ -83,7 +94,7 @@ Defined in: [packages/db/src/transactions.ts:213](https://github.com/TanStack/db metadata: Record; ``` -Defined in: [packages/db/src/transactions.ts:217](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L217) +Defined in: [packages/db/src/transactions.ts:326](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L326) *** @@ -93,7 +104,7 @@ Defined in: [packages/db/src/transactions.ts:217](https://github.com/TanStack/db mutationFn: MutationFn; ``` -Defined in: [packages/db/src/transactions.ts:211](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L211) +Defined in: [packages/db/src/transactions.ts:308](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L308) *** @@ -103,7 +114,7 @@ Defined in: [packages/db/src/transactions.ts:211](https://github.com/TanStack/db mutations: PendingMutation>[]; ``` -Defined in: [packages/db/src/transactions.ts:212](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L212) +Defined in: [packages/db/src/transactions.ts:309](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L309) *** @@ -113,7 +124,7 @@ Defined in: [packages/db/src/transactions.ts:212](https://github.com/TanStack/db sequenceNumber: number; ``` -Defined in: [packages/db/src/transactions.ts:216](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L216) +Defined in: [packages/db/src/transactions.ts:325](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L325) *** @@ -123,7 +134,7 @@ Defined in: [packages/db/src/transactions.ts:216](https://github.com/TanStack/db state: TransactionState; ``` -Defined in: [packages/db/src/transactions.ts:210](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L210) +Defined in: [packages/db/src/transactions.ts:307](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L307) ## Methods @@ -133,7 +144,7 @@ Defined in: [packages/db/src/transactions.ts:210](https://github.com/TanStack/db applyMutations(mutations): void; ``` -Defined in: [packages/db/src/transactions.ts:327](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L327) +Defined in: [packages/db/src/transactions.ts:460](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L460) Apply new mutations to this transaction, intelligently merging with existing mutations @@ -169,7 +180,7 @@ Array of new mutations to apply commit(): Promise>; ``` -Defined in: [packages/db/src/transactions.ts:472](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L472) +Defined in: [packages/db/src/transactions.ts:618](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L618) Commit the transaction and execute the mutation function @@ -228,7 +239,7 @@ console.log(tx.state) // "completed" or "failed" compareCreatedAt(other): number; ``` -Defined in: [packages/db/src/transactions.ts:526](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L526) +Defined in: [packages/db/src/transactions.ts:675](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L675) Compare two transactions by their createdAt time and sequence number in order to sort them in the order they were created. @@ -255,7 +266,7 @@ The other transaction to compare to mutate(callback): Transaction; ``` -Defined in: [packages/db/src/transactions.ts:287](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L287) +Defined in: [packages/db/src/transactions.ts:410](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L410) Execute collection operations within this transaction @@ -265,9 +276,12 @@ Execute collection operations within this transaction () => `void` -Function containing collection operations to group together. If the -callback returns a Promise, the transaction context will remain active until the promise -settles, allowing optimistic writes after `await` boundaries. +Synchronous function containing collection operations to group together. +The transaction context is active only for the synchronous duration of this callback. +Async work should happen in `mutationFn`; collection operations after `await` boundaries +inside this callback will not be part of this transaction. For manual transactions, call +`mutate` multiple times before committing to add more synchronous operations to the same +transaction. #### Returns @@ -311,6 +325,11 @@ tx.mutate(() => { collection.insert({ id: "1", text: "Item" }) }) +// Add more synchronous mutations to the same transaction +tx.mutate(() => { + collection.update("1", draft => { draft.text = "Updated item" }) +}) + // Commit later when ready await tx.commit() ``` @@ -323,7 +342,7 @@ await tx.commit() rollback(config?): Transaction; ``` -Defined in: [packages/db/src/transactions.ts:389](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L389) +Defined in: [packages/db/src/transactions.ts:534](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L534) Rollback the transaction and any conflicting transactions @@ -390,7 +409,7 @@ try { setState(newState): void; ``` -Defined in: [packages/db/src/transactions.ts:238](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L238) +Defined in: [packages/db/src/transactions.ts:353](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L353) #### Parameters @@ -410,7 +429,7 @@ Defined in: [packages/db/src/transactions.ts:238](https://github.com/TanStack/db touchCollection(): void; ``` -Defined in: [packages/db/src/transactions.ts:417](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L417) +Defined in: [packages/db/src/transactions.ts:563](https://github.com/TanStack/db/blob/main/packages/db/src/transactions.ts#L563) #### Returns diff --git a/docs/reference/interfaces/TransactionConfig.md b/docs/reference/interfaces/TransactionConfig.md index c146f2f113..9c5d5bbe37 100644 --- a/docs/reference/interfaces/TransactionConfig.md +++ b/docs/reference/interfaces/TransactionConfig.md @@ -5,7 +5,7 @@ title: TransactionConfig # Interface: TransactionConfig\ -Defined in: [packages/db/src/types.ts:168](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L168) +Defined in: [packages/db/src/types.ts:174](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L174) ## Type Parameters @@ -21,7 +21,7 @@ Defined in: [packages/db/src/types.ts:168](https://github.com/TanStack/db/blob/m optional autoCommit: boolean; ``` -Defined in: [packages/db/src/types.ts:172](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L172) +Defined in: [packages/db/src/types.ts:178](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L178) *** @@ -31,7 +31,7 @@ Defined in: [packages/db/src/types.ts:172](https://github.com/TanStack/db/blob/m optional id: string; ``` -Defined in: [packages/db/src/types.ts:170](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L170) +Defined in: [packages/db/src/types.ts:176](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L176) Unique identifier for the transaction @@ -43,7 +43,7 @@ Unique identifier for the transaction optional metadata: Record; ``` -Defined in: [packages/db/src/types.ts:175](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L175) +Defined in: [packages/db/src/types.ts:181](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L181) Custom metadata to associate with the transaction @@ -55,4 +55,4 @@ Custom metadata to associate with the transaction mutationFn: MutationFn; ``` -Defined in: [packages/db/src/types.ts:173](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L173) +Defined in: [packages/db/src/types.ts:179](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L179) diff --git a/docs/reference/interfaces/VirtualRowProps.md b/docs/reference/interfaces/VirtualRowProps.md index c4719d188e..63d2ca95c5 100644 --- a/docs/reference/interfaces/VirtualRowProps.md +++ b/docs/reference/interfaces/VirtualRowProps.md @@ -21,7 +21,7 @@ These properties are: // Accessing virtual properties on a row const user = collection.get('user-1') if (user.$synced) { - console.log('Confirmed by backend') + console.log('No pending local optimistic writes for this row') } if (user.$origin === 'local') { console.log('Created/modified locally') @@ -30,7 +30,7 @@ if (user.$origin === 'local') { ```typescript // Using virtual properties in queries -const confirmedOrders = createLiveQueryCollection({ +const ordersWithoutLocalWrites = createLiveQueryCollection({ query: (q) => q .from({ order: orders }) .where(({ order }) => eq(order.$synced, true)) @@ -53,7 +53,7 @@ The type of the row's key (string or number) readonly $collectionId: string; ``` -Defined in: [packages/db/src/virtual-props.ts:96](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L96) +Defined in: [packages/db/src/virtual-props.ts:101](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L101) The ID of the source collection this row originated from. @@ -68,7 +68,7 @@ For live query collections, this is the ID of the upstream collection. readonly $key: TKey; ``` -Defined in: [packages/db/src/virtual-props.ts:88](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L88) +Defined in: [packages/db/src/virtual-props.ts:93](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L93) The row's key (primary identifier). @@ -83,7 +83,7 @@ Useful when you need the key in projections or computations. readonly $origin: VirtualOrigin; ``` -Defined in: [packages/db/src/virtual-props.ts:80](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L80) +Defined in: [packages/db/src/virtual-props.ts:85](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L85) Origin of the last confirmed change to this row, from the current client's perspective. @@ -101,12 +101,17 @@ For live query collections, this is passed through from the source collection. readonly $synced: boolean; ``` -Defined in: [packages/db/src/virtual-props.ts:69](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L69) +Defined in: [packages/db/src/virtual-props.ts:74](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L74) -Whether this row reflects confirmed state from the backend. +Whether this row currently has no pending local optimistic writes. -- `true`: Row is confirmed by the backend (no pending optimistic mutations) -- `false`: Row has pending optimistic mutations that haven't been confirmed +- `true`: No pending local optimistic mutation currently affects this row +- `false`: One or more pending local optimistic mutations currently affect this row + +This is local mutation status. It does not prove that a backend has uploaded, +confirmed, or read back the row. If you need backend-confirmed status, keep +your mutation function pending until that backend observation has happened, +or expose adapter-specific status. For local-only collections (no sync), this is always `true`. For live query collections, this is passed through from the source collection. diff --git a/docs/reference/powersync-db-collection/classes/PowerSyncTransactor.md b/docs/reference/powersync-db-collection/classes/PowerSyncTransactor.md index 10d3c8d3e8..9d2f7088e2 100644 --- a/docs/reference/powersync-db-collection/classes/PowerSyncTransactor.md +++ b/docs/reference/powersync-db-collection/classes/PowerSyncTransactor.md @@ -5,7 +5,7 @@ title: PowerSyncTransactor # Class: PowerSyncTransactor -Defined in: [PowerSyncTransactor.ts:54](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L54) +Defined in: [PowerSyncTransactor.ts:55](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L55) Applies mutations to the PowerSync database. This method is called automatically by the collection's insert, update, and delete operations. You typically don't need to call this directly unless you @@ -51,7 +51,7 @@ The transaction containing mutations to apply new PowerSyncTransactor(options): PowerSyncTransactor; ``` -Defined in: [PowerSyncTransactor.ts:58](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L58) +Defined in: [PowerSyncTransactor.ts:59](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L59) #### Parameters @@ -71,7 +71,7 @@ Defined in: [PowerSyncTransactor.ts:58](https://github.com/TanStack/db/blob/main database: AbstractPowerSyncDatabase; ``` -Defined in: [PowerSyncTransactor.ts:55](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L55) +Defined in: [PowerSyncTransactor.ts:56](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L56) *** @@ -81,7 +81,7 @@ Defined in: [PowerSyncTransactor.ts:55](https://github.com/TanStack/db/blob/main pendingOperationStore: PendingOperationStore; ``` -Defined in: [PowerSyncTransactor.ts:56](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L56) +Defined in: [PowerSyncTransactor.ts:57](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L57) ## Methods @@ -91,7 +91,7 @@ Defined in: [PowerSyncTransactor.ts:56](https://github.com/TanStack/db/blob/main applyTransaction(transaction): Promise; ``` -Defined in: [PowerSyncTransactor.ts:66](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L66) +Defined in: [PowerSyncTransactor.ts:67](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L67) Persists a Transaction to the PowerSync SQLite database. @@ -113,7 +113,7 @@ Persists a Transaction to the PowerSync SQLite database. protected getMutationCollectionMeta(mutation): PowerSyncCollectionMeta; ``` -Defined in: [PowerSyncTransactor.ts:297](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L297) +Defined in: [PowerSyncTransactor.ts:320](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L320) #### Parameters @@ -136,7 +136,7 @@ protected handleDelete( waitForCompletion): Promise; ``` -Defined in: [PowerSyncTransactor.ts:223](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L223) +Defined in: [PowerSyncTransactor.ts:246](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L246) #### Parameters @@ -167,7 +167,7 @@ protected handleInsert( waitForCompletion): Promise; ``` -Defined in: [PowerSyncTransactor.ts:152](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L152) +Defined in: [PowerSyncTransactor.ts:175](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L175) #### Parameters @@ -199,7 +199,7 @@ protected handleOperationWithCompletion( handler): Promise; ``` -Defined in: [PowerSyncTransactor.ts:266](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L266) +Defined in: [PowerSyncTransactor.ts:289](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L289) Helper function which wraps a persistence operation by: - Fetching the mutation's collection's SQLite table details @@ -239,7 +239,7 @@ protected handleUpdate( waitForCompletion): Promise; ``` -Defined in: [PowerSyncTransactor.ts:188](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L188) +Defined in: [PowerSyncTransactor.ts:211](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L211) #### Parameters @@ -267,7 +267,7 @@ Defined in: [PowerSyncTransactor.ts:188](https://github.com/TanStack/db/blob/mai protected processMutationMetadata(mutation): string | null; ``` -Defined in: [PowerSyncTransactor.ts:316](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L316) +Defined in: [PowerSyncTransactor.ts:339](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L339) Processes collection mutation metadata for persistence to the database. We only support storing string metadata. diff --git a/docs/reference/powersync-db-collection/functions/powerSyncCollectionOptions.md b/docs/reference/powersync-db-collection/functions/powerSyncCollectionOptions.md index ca73a4a4bc..9954738cc6 100644 --- a/docs/reference/powersync-db-collection/functions/powerSyncCollectionOptions.md +++ b/docs/reference/powersync-db-collection/functions/powerSyncCollectionOptions.md @@ -13,7 +13,7 @@ Implementation of powerSyncCollectionOptions that handles both schema and non-sc function powerSyncCollectionOptions(config): EnhancedPowerSyncCollectionConfig, never>; ``` -Defined in: [powersync.ts:78](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/powersync.ts#L78) +Defined in: [powersync.ts:79](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/powersync.ts#L79) Creates a PowerSync collection configuration with basic default validation. Input and Output types are the SQLite column types. @@ -66,7 +66,7 @@ const collection = createCollection( function powerSyncCollectionOptions(config): CollectionConfig, string, TSchema, PowerSyncCollectionUtils> & object & object; ``` -Defined in: [powersync.ts:135](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/powersync.ts#L135) +Defined in: [powersync.ts:136](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/powersync.ts#L136) Creates a PowerSync collection configuration with schema validation. @@ -141,7 +141,7 @@ const collection = createCollection( function powerSyncCollectionOptions(config): CollectionConfig, string, TSchema, PowerSyncCollectionUtils> & object & object; ``` -Defined in: [powersync.ts:203](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/powersync.ts#L203) +Defined in: [powersync.ts:204](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/powersync.ts#L204) Creates a PowerSync collection configuration with schema validation. diff --git a/docs/reference/powersync-db-collection/type-aliases/TransactorOptions.md b/docs/reference/powersync-db-collection/type-aliases/TransactorOptions.md index 7a9051b65c..273130a81b 100644 --- a/docs/reference/powersync-db-collection/type-aliases/TransactorOptions.md +++ b/docs/reference/powersync-db-collection/type-aliases/TransactorOptions.md @@ -9,7 +9,7 @@ title: TransactorOptions type TransactorOptions = object; ``` -Defined in: [PowerSyncTransactor.ts:15](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L15) +Defined in: [PowerSyncTransactor.ts:16](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L16) ## Properties @@ -19,4 +19,4 @@ Defined in: [PowerSyncTransactor.ts:15](https://github.com/TanStack/db/blob/main database: AbstractPowerSyncDatabase; ``` -Defined in: [PowerSyncTransactor.ts:16](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L16) +Defined in: [PowerSyncTransactor.ts:17](https://github.com/TanStack/db/blob/main/packages/powersync-db-collection/src/PowerSyncTransactor.ts#L17) diff --git a/docs/reference/query-db-collection/classes/DeleteOperationItemNotFoundError.md b/docs/reference/query-db-collection/classes/DeleteOperationItemNotFoundError.md index 925711239a..2655d95a22 100644 --- a/docs/reference/query-db-collection/classes/DeleteOperationItemNotFoundError.md +++ b/docs/reference/query-db-collection/classes/DeleteOperationItemNotFoundError.md @@ -5,7 +5,7 @@ title: DeleteOperationItemNotFoundError # Class: DeleteOperationItemNotFoundError -Defined in: [packages/query-db-collection/src/errors.ts:76](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L76) +Defined in: [packages/query-db-collection/src/errors.ts:85](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L85) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/query-db-collection/src/errors.ts:76](https://github.com/T new DeleteOperationItemNotFoundError(key): DeleteOperationItemNotFoundError; ``` -Defined in: [packages/query-db-collection/src/errors.ts:77](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L77) +Defined in: [packages/query-db-collection/src/errors.ts:86](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L86) #### Parameters diff --git a/docs/reference/query-db-collection/classes/DuplicateKeyInBatchError.md b/docs/reference/query-db-collection/classes/DuplicateKeyInBatchError.md index 7eeb9248fa..9d342a37a4 100644 --- a/docs/reference/query-db-collection/classes/DuplicateKeyInBatchError.md +++ b/docs/reference/query-db-collection/classes/DuplicateKeyInBatchError.md @@ -5,7 +5,7 @@ title: DuplicateKeyInBatchError # Class: DuplicateKeyInBatchError -Defined in: [packages/query-db-collection/src/errors.ts:62](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L62) +Defined in: [packages/query-db-collection/src/errors.ts:71](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L71) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/query-db-collection/src/errors.ts:62](https://github.com/T new DuplicateKeyInBatchError(key): DuplicateKeyInBatchError; ``` -Defined in: [packages/query-db-collection/src/errors.ts:63](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L63) +Defined in: [packages/query-db-collection/src/errors.ts:72](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L72) #### Parameters diff --git a/docs/reference/query-db-collection/classes/InitialDataInOnDemandModeError.md b/docs/reference/query-db-collection/classes/InitialDataInOnDemandModeError.md new file mode 100644 index 0000000000..3c1822a478 --- /dev/null +++ b/docs/reference/query-db-collection/classes/InitialDataInOnDemandModeError.md @@ -0,0 +1,214 @@ +--- +id: InitialDataInOnDemandModeError +title: InitialDataInOnDemandModeError +--- + +# Class: InitialDataInOnDemandModeError + +Defined in: [packages/query-db-collection/src/errors.ts:39](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L39) + +## Extends + +- [`QueryCollectionError`](QueryCollectionError.md) + +## Constructors + +### Constructor + +```ts +new InitialDataInOnDemandModeError(): InitialDataInOnDemandModeError; +``` + +Defined in: [packages/query-db-collection/src/errors.ts:40](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L40) + +#### Returns + +`InitialDataInOnDemandModeError` + +#### Overrides + +[`QueryCollectionError`](QueryCollectionError.md).[`constructor`](QueryCollectionError.md#constructor) + +## Properties + +### cause? + +```ts +optional cause: unknown; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es2022.error.d.ts:26 + +#### Inherited from + +[`QueryCollectionError`](QueryCollectionError.md).[`cause`](QueryCollectionError.md#cause) + +*** + +### message + +```ts +message: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1077 + +#### Inherited from + +[`QueryCollectionError`](QueryCollectionError.md).[`message`](QueryCollectionError.md#message) + +*** + +### name + +```ts +name: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1076 + +#### Inherited from + +[`QueryCollectionError`](QueryCollectionError.md).[`name`](QueryCollectionError.md#name) + +*** + +### stack? + +```ts +optional stack: string; +``` + +Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1078 + +#### Inherited from + +[`QueryCollectionError`](QueryCollectionError.md).[`stack`](QueryCollectionError.md#stack) + +*** + +### stackTraceLimit + +```ts +static stackTraceLimit: number; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:67 + +The `Error.stackTraceLimit` property specifies the number of stack frames +collected by a stack trace (whether generated by `new Error().stack` or +`Error.captureStackTrace(obj)`). + +The default value is `10` but may be set to any valid JavaScript number. Changes +will affect any stack trace captured _after_ the value has been changed. + +If set to a non-number value, or set to a negative number, stack traces will +not capture any frames. + +#### Inherited from + +[`QueryCollectionError`](QueryCollectionError.md).[`stackTraceLimit`](QueryCollectionError.md#stacktracelimit) + +## Methods + +### captureStackTrace() + +```ts +static captureStackTrace(targetObject, constructorOpt?): void; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:51 + +Creates a `.stack` property on `targetObject`, which when accessed returns +a string representing the location in the code at which +`Error.captureStackTrace()` was called. + +```js +const myObject = {}; +Error.captureStackTrace(myObject); +myObject.stack; // Similar to `new Error().stack` +``` + +The first line of the trace will be prefixed with +`${myObject.name}: ${myObject.message}`. + +The optional `constructorOpt` argument accepts a function. If given, all frames +above `constructorOpt`, including `constructorOpt`, will be omitted from the +generated stack trace. + +The `constructorOpt` argument is useful for hiding implementation +details of error generation from the user. For instance: + +```js +function a() { + b(); +} + +function b() { + c(); +} + +function c() { + // Create an error without stack trace to avoid calculating the stack trace twice. + const { stackTraceLimit } = Error; + Error.stackTraceLimit = 0; + const error = new Error(); + Error.stackTraceLimit = stackTraceLimit; + + // Capture the stack trace above function b + Error.captureStackTrace(error, b); // Neither function c, nor b is included in the stack trace + throw error; +} + +a(); +``` + +#### Parameters + +##### targetObject + +`object` + +##### constructorOpt? + +`Function` + +#### Returns + +`void` + +#### Inherited from + +[`QueryCollectionError`](QueryCollectionError.md).[`captureStackTrace`](QueryCollectionError.md#capturestacktrace) + +*** + +### prepareStackTrace() + +```ts +static prepareStackTrace(err, stackTraces): any; +``` + +Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:55 + +#### Parameters + +##### err + +`Error` + +##### stackTraces + +`CallSite`[] + +#### Returns + +`any` + +#### See + +https://v8.dev/docs/stack-trace-api#customizing-stack-traces + +#### Inherited from + +[`QueryCollectionError`](QueryCollectionError.md).[`prepareStackTrace`](QueryCollectionError.md#preparestacktrace) diff --git a/docs/reference/query-db-collection/classes/InvalidItemStructureError.md b/docs/reference/query-db-collection/classes/InvalidItemStructureError.md index 7b01648f54..99389fe3f7 100644 --- a/docs/reference/query-db-collection/classes/InvalidItemStructureError.md +++ b/docs/reference/query-db-collection/classes/InvalidItemStructureError.md @@ -5,7 +5,7 @@ title: InvalidItemStructureError # Class: InvalidItemStructureError -Defined in: [packages/query-db-collection/src/errors.ts:48](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L48) +Defined in: [packages/query-db-collection/src/errors.ts:57](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L57) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/query-db-collection/src/errors.ts:48](https://github.com/T new InvalidItemStructureError(message): InvalidItemStructureError; ``` -Defined in: [packages/query-db-collection/src/errors.ts:49](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L49) +Defined in: [packages/query-db-collection/src/errors.ts:58](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L58) #### Parameters diff --git a/docs/reference/query-db-collection/classes/InvalidSyncOperationError.md b/docs/reference/query-db-collection/classes/InvalidSyncOperationError.md index 2691b6f06d..797b887d87 100644 --- a/docs/reference/query-db-collection/classes/InvalidSyncOperationError.md +++ b/docs/reference/query-db-collection/classes/InvalidSyncOperationError.md @@ -5,7 +5,7 @@ title: InvalidSyncOperationError # Class: InvalidSyncOperationError -Defined in: [packages/query-db-collection/src/errors.ts:83](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L83) +Defined in: [packages/query-db-collection/src/errors.ts:92](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L92) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/query-db-collection/src/errors.ts:83](https://github.com/T new InvalidSyncOperationError(message): InvalidSyncOperationError; ``` -Defined in: [packages/query-db-collection/src/errors.ts:84](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L84) +Defined in: [packages/query-db-collection/src/errors.ts:93](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L93) #### Parameters diff --git a/docs/reference/query-db-collection/classes/ItemNotFoundError.md b/docs/reference/query-db-collection/classes/ItemNotFoundError.md index 85c1400f60..21326a6bd3 100644 --- a/docs/reference/query-db-collection/classes/ItemNotFoundError.md +++ b/docs/reference/query-db-collection/classes/ItemNotFoundError.md @@ -5,7 +5,7 @@ title: ItemNotFoundError # Class: ItemNotFoundError -Defined in: [packages/query-db-collection/src/errors.ts:55](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L55) +Defined in: [packages/query-db-collection/src/errors.ts:64](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L64) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/query-db-collection/src/errors.ts:55](https://github.com/T new ItemNotFoundError(key): ItemNotFoundError; ``` -Defined in: [packages/query-db-collection/src/errors.ts:56](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L56) +Defined in: [packages/query-db-collection/src/errors.ts:65](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L65) #### Parameters diff --git a/docs/reference/query-db-collection/classes/MissingKeyFieldError.md b/docs/reference/query-db-collection/classes/MissingKeyFieldError.md index 3c44dc2aca..3fc58996d5 100644 --- a/docs/reference/query-db-collection/classes/MissingKeyFieldError.md +++ b/docs/reference/query-db-collection/classes/MissingKeyFieldError.md @@ -5,7 +5,7 @@ title: MissingKeyFieldError # Class: MissingKeyFieldError -Defined in: [packages/query-db-collection/src/errors.ts:97](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L97) +Defined in: [packages/query-db-collection/src/errors.ts:106](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L106) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/query-db-collection/src/errors.ts:97](https://github.com/T new MissingKeyFieldError(operation, message): MissingKeyFieldError; ``` -Defined in: [packages/query-db-collection/src/errors.ts:98](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L98) +Defined in: [packages/query-db-collection/src/errors.ts:107](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L107) #### Parameters diff --git a/docs/reference/query-db-collection/classes/QueryCollectionError.md b/docs/reference/query-db-collection/classes/QueryCollectionError.md index 9e3c7fce57..01d33ccc15 100644 --- a/docs/reference/query-db-collection/classes/QueryCollectionError.md +++ b/docs/reference/query-db-collection/classes/QueryCollectionError.md @@ -17,6 +17,7 @@ Defined in: [packages/query-db-collection/src/errors.ts:4](https://github.com/Ta - [`QueryFnRequiredError`](QueryFnRequiredError.md) - [`QueryClientRequiredError`](QueryClientRequiredError.md) - [`GetKeyRequiredError`](GetKeyRequiredError.md) +- [`InitialDataInOnDemandModeError`](InitialDataInOnDemandModeError.md) - [`SyncNotInitializedError`](SyncNotInitializedError.md) - [`InvalidItemStructureError`](InvalidItemStructureError.md) - [`ItemNotFoundError`](ItemNotFoundError.md) diff --git a/docs/reference/query-db-collection/classes/SyncNotInitializedError.md b/docs/reference/query-db-collection/classes/SyncNotInitializedError.md index 8e5a132f5a..f8954b8eb1 100644 --- a/docs/reference/query-db-collection/classes/SyncNotInitializedError.md +++ b/docs/reference/query-db-collection/classes/SyncNotInitializedError.md @@ -5,7 +5,7 @@ title: SyncNotInitializedError # Class: SyncNotInitializedError -Defined in: [packages/query-db-collection/src/errors.ts:39](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L39) +Defined in: [packages/query-db-collection/src/errors.ts:48](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L48) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/query-db-collection/src/errors.ts:39](https://github.com/T new SyncNotInitializedError(): SyncNotInitializedError; ``` -Defined in: [packages/query-db-collection/src/errors.ts:40](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L40) +Defined in: [packages/query-db-collection/src/errors.ts:49](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L49) #### Returns diff --git a/docs/reference/query-db-collection/classes/UnknownOperationTypeError.md b/docs/reference/query-db-collection/classes/UnknownOperationTypeError.md index ab9b767152..f4392d794a 100644 --- a/docs/reference/query-db-collection/classes/UnknownOperationTypeError.md +++ b/docs/reference/query-db-collection/classes/UnknownOperationTypeError.md @@ -5,7 +5,7 @@ title: UnknownOperationTypeError # Class: UnknownOperationTypeError -Defined in: [packages/query-db-collection/src/errors.ts:90](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L90) +Defined in: [packages/query-db-collection/src/errors.ts:99](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L99) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/query-db-collection/src/errors.ts:90](https://github.com/T new UnknownOperationTypeError(type): UnknownOperationTypeError; ``` -Defined in: [packages/query-db-collection/src/errors.ts:91](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L91) +Defined in: [packages/query-db-collection/src/errors.ts:100](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L100) #### Parameters diff --git a/docs/reference/query-db-collection/classes/UpdateOperationItemNotFoundError.md b/docs/reference/query-db-collection/classes/UpdateOperationItemNotFoundError.md index 304555e780..d2ae0ed614 100644 --- a/docs/reference/query-db-collection/classes/UpdateOperationItemNotFoundError.md +++ b/docs/reference/query-db-collection/classes/UpdateOperationItemNotFoundError.md @@ -5,7 +5,7 @@ title: UpdateOperationItemNotFoundError # Class: UpdateOperationItemNotFoundError -Defined in: [packages/query-db-collection/src/errors.ts:69](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L69) +Defined in: [packages/query-db-collection/src/errors.ts:78](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L78) ## Extends @@ -19,7 +19,7 @@ Defined in: [packages/query-db-collection/src/errors.ts:69](https://github.com/T new UpdateOperationItemNotFoundError(key): UpdateOperationItemNotFoundError; ``` -Defined in: [packages/query-db-collection/src/errors.ts:70](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L70) +Defined in: [packages/query-db-collection/src/errors.ts:79](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/errors.ts#L79) #### Parameters diff --git a/docs/reference/query-db-collection/functions/queryCollectionOptions.md b/docs/reference/query-db-collection/functions/queryCollectionOptions.md index 360652b593..436bc1d575 100644 --- a/docs/reference/query-db-collection/functions/queryCollectionOptions.md +++ b/docs/reference/query-db-collection/functions/queryCollectionOptions.md @@ -11,7 +11,7 @@ title: queryCollectionOptions function queryCollectionOptions(config): CollectionConfig, TKey, T, QueryCollectionUtils, TKey, InferSchemaInput, TError>> & object; ``` -Defined in: [packages/query-db-collection/src/query.ts:431](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L431) +Defined in: [packages/query-db-collection/src/query.ts:536](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L536) Creates query collection options for use with a standard Collection. This integrates TanStack Query with TanStack DB for automatic synchronization. @@ -151,7 +151,7 @@ const todosCollection = createCollection( function queryCollectionOptions(config): CollectionConfig> & object; ``` -Defined in: [packages/query-db-collection/src/query.ts:466](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L466) +Defined in: [packages/query-db-collection/src/query.ts:571](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L571) Creates query collection options for use with a standard Collection. This integrates TanStack Query with TanStack DB for automatic synchronization. @@ -291,7 +291,7 @@ const todosCollection = createCollection( function queryCollectionOptions(config): CollectionConfig, TKey, T, QueryCollectionUtils, TKey, InferSchemaInput, TError>> & object; ``` -Defined in: [packages/query-db-collection/src/query.ts:499](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L499) +Defined in: [packages/query-db-collection/src/query.ts:604](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L604) Creates query collection options for use with a standard Collection. This integrates TanStack Query with TanStack DB for automatic synchronization. @@ -423,7 +423,7 @@ const todosCollection = createCollection( function queryCollectionOptions(config): CollectionConfig> & object; ``` -Defined in: [packages/query-db-collection/src/query.ts:533](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L533) +Defined in: [packages/query-db-collection/src/query.ts:638](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L638) Creates query collection options for use with a standard Collection. This integrates TanStack Query with TanStack DB for automatic synchronization. diff --git a/docs/reference/query-db-collection/index.md b/docs/reference/query-db-collection/index.md index 569ee31dde..a25e301173 100644 --- a/docs/reference/query-db-collection/index.md +++ b/docs/reference/query-db-collection/index.md @@ -10,6 +10,7 @@ title: "@tanstack/query-db-collection" - [DeleteOperationItemNotFoundError](classes/DeleteOperationItemNotFoundError.md) - [DuplicateKeyInBatchError](classes/DuplicateKeyInBatchError.md) - [GetKeyRequiredError](classes/GetKeyRequiredError.md) +- [InitialDataInOnDemandModeError](classes/InitialDataInOnDemandModeError.md) - [InvalidItemStructureError](classes/InvalidItemStructureError.md) - [InvalidSyncOperationError](classes/InvalidSyncOperationError.md) - [ItemNotFoundError](classes/ItemNotFoundError.md) diff --git a/docs/reference/query-db-collection/interfaces/QueryCollectionConfig.md b/docs/reference/query-db-collection/interfaces/QueryCollectionConfig.md index a53cae230d..1e2f38c78e 100644 --- a/docs/reference/query-db-collection/interfaces/QueryCollectionConfig.md +++ b/docs/reference/query-db-collection/interfaces/QueryCollectionConfig.md @@ -5,7 +5,7 @@ title: QueryCollectionConfig # Interface: QueryCollectionConfig\ -Defined in: [packages/query-db-collection/src/query.ts:61](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L61) +Defined in: [packages/query-db-collection/src/query.ts:101](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L101) Configuration options for creating a Query Collection @@ -63,19 +63,64 @@ The schema type for validation optional enabled: Enabled; ``` -Defined in: [packages/query-db-collection/src/query.ts:87](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L87) +Defined in: [packages/query-db-collection/src/query.ts:130](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L130) Whether the query should automatically run (default: true) *** +### gcTime? + +```ts +optional gcTime: number; +``` + +Defined in: [packages/query-db-collection/src/query.ts:165](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L165) + +Time in milliseconds after which the collection will be garbage collected +when it has no active subscribers. Defaults to 5 minutes (300000ms). + +#### Overrides + +```ts +BaseCollectionConfig.gcTime +``` + +*** + +### initialData? + +```ts +optional initialData: TQueryData | InitialDataFunction; +``` + +Defined in: [packages/query-db-collection/src/query.ts:205](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L205) + +Data used to initialize the TanStack Query cache for an eager collection. +The value has the original Query response shape and is projected through +the collection's select option before rows are materialized. + +*** + +### initialDataUpdatedAt? + +```ts +optional initialDataUpdatedAt: number | () => number | undefined; +``` + +Defined in: [packages/query-db-collection/src/query.ts:213](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L213) + +The timestamp TanStack Query uses to determine initialData freshness. + +*** + ### meta? ```ts optional meta: Record; ``` -Defined in: [packages/query-db-collection/src/query.ts:144](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L144) +Defined in: [packages/query-db-collection/src/query.ts:242](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L242) Metadata to pass to the query. Available in queryFn via context.meta @@ -101,13 +146,23 @@ meta: { *** +### networkMode? + +```ts +optional networkMode: NetworkMode; +``` + +Defined in: [packages/query-db-collection/src/query.ts:193](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L193) + +*** + ### persistedGcTime? ```ts optional persistedGcTime: number; ``` -Defined in: [packages/query-db-collection/src/query.ts:122](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L122) +Defined in: [packages/query-db-collection/src/query.ts:220](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L220) *** @@ -117,7 +172,7 @@ Defined in: [packages/query-db-collection/src/query.ts:122](https://github.com/T queryClient: QueryClient; ``` -Defined in: [packages/query-db-collection/src/query.ts:83](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L83) +Defined in: [packages/query-db-collection/src/query.ts:126](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L126) The TanStack Query client instance @@ -129,7 +184,7 @@ The TanStack Query client instance queryFn: TQueryFn extends (context) => any[] | Promise ? (context) => T[] | Promise : TQueryFn; ``` -Defined in: [packages/query-db-collection/src/query.ts:75](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L75) +Defined in: [packages/query-db-collection/src/query.ts:115](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L115) Function that fetches data from the server. Must return the complete collection state @@ -141,7 +196,7 @@ Function that fetches data from the server. Must return the complete collection queryKey: TQueryKey | TQueryKeyBuilder; ``` -Defined in: [packages/query-db-collection/src/query.ts:73](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L73) +Defined in: [packages/query-db-collection/src/query.ts:113](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L113) The query key used by TanStack Query to identify this query @@ -153,7 +208,37 @@ The query key used by TanStack Query to identify this query optional refetchInterval: number | false | (query) => number | false | undefined; ``` -Defined in: [packages/query-db-collection/src/query.ts:94](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L94) +Defined in: [packages/query-db-collection/src/query.ts:137](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L137) + +*** + +### refetchOnMount? + +```ts +optional refetchOnMount: boolean | "always" | (query) => boolean | "always"; +``` + +Defined in: [packages/query-db-collection/src/query.ts:186](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L186) + +*** + +### refetchOnReconnect? + +```ts +optional refetchOnReconnect: boolean | "always" | (query) => boolean | "always"; +``` + +Defined in: [packages/query-db-collection/src/query.ts:179](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L179) + +*** + +### refetchOnWindowFocus? + +```ts +optional refetchOnWindowFocus: boolean | "always" | (query) => boolean | "always"; +``` + +Defined in: [packages/query-db-collection/src/query.ts:172](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L172) *** @@ -163,7 +248,7 @@ Defined in: [packages/query-db-collection/src/query.ts:94](https://github.com/Ta optional retry: RetryValue; ``` -Defined in: [packages/query-db-collection/src/query.ts:101](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L101) +Defined in: [packages/query-db-collection/src/query.ts:144](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L144) *** @@ -173,7 +258,7 @@ Defined in: [packages/query-db-collection/src/query.ts:101](https://github.com/T optional retryDelay: RetryDelayValue; ``` -Defined in: [packages/query-db-collection/src/query.ts:108](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L108) +Defined in: [packages/query-db-collection/src/query.ts:151](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L151) *** @@ -183,7 +268,10 @@ Defined in: [packages/query-db-collection/src/query.ts:108](https://github.com/T optional select: (data) => T[]; ``` -Defined in: [packages/query-db-collection/src/query.ts:81](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L81) +Defined in: [packages/query-db-collection/src/query.ts:124](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L124) + +Extracts the row array TanStack DB materializes from the Query response. +The Query cache keeps the original response shape. #### Parameters @@ -203,4 +291,4 @@ Defined in: [packages/query-db-collection/src/query.ts:81](https://github.com/Ta optional staleTime: StaleTimeFunction; ``` -Defined in: [packages/query-db-collection/src/query.ts:115](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L115) +Defined in: [packages/query-db-collection/src/query.ts:158](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L158) diff --git a/docs/reference/query-db-collection/interfaces/QueryCollectionUtils.md b/docs/reference/query-db-collection/interfaces/QueryCollectionUtils.md index d30be795f5..2650ee3779 100644 --- a/docs/reference/query-db-collection/interfaces/QueryCollectionUtils.md +++ b/docs/reference/query-db-collection/interfaces/QueryCollectionUtils.md @@ -5,7 +5,7 @@ title: QueryCollectionUtils # Interface: QueryCollectionUtils\ -Defined in: [packages/query-db-collection/src/query.ts:163](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L163) +Defined in: [packages/query-db-collection/src/query.ts:261](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L261) Utility methods available on Query Collections for direct writes and manual operations. Direct writes bypass the normal query/mutation flow and write directly to the synced data store. @@ -54,7 +54,7 @@ The type of errors that can occur during queries clearError: () => Promise; ``` -Defined in: [packages/query-db-collection/src/query.ts:208](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L208) +Defined in: [packages/query-db-collection/src/query.ts:306](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L306) Clear the error state and trigger a refetch of the query @@ -76,7 +76,7 @@ Error if the refetch fails dataUpdatedAt: number; ``` -Defined in: [packages/query-db-collection/src/query.ts:199](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L199) +Defined in: [packages/query-db-collection/src/query.ts:297](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L297) Get timestamp of last successful data update (in milliseconds) @@ -88,7 +88,7 @@ Get timestamp of last successful data update (in milliseconds) errorCount: number; ``` -Defined in: [packages/query-db-collection/src/query.ts:191](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L191) +Defined in: [packages/query-db-collection/src/query.ts:289](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L289) Get the number of consecutive sync failures. Incremented only when query fails completely (not per retry attempt); reset on success. @@ -101,7 +101,7 @@ Incremented only when query fails completely (not per retry attempt); reset on s fetchStatus: "idle" | "fetching" | "paused"; ``` -Defined in: [packages/query-db-collection/src/query.ts:201](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L201) +Defined in: [packages/query-db-collection/src/query.ts:299](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L299) Get current fetch status @@ -113,7 +113,7 @@ Get current fetch status isError: boolean; ``` -Defined in: [packages/query-db-collection/src/query.ts:186](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L186) +Defined in: [packages/query-db-collection/src/query.ts:284](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L284) Check if the collection is in an error state @@ -125,7 +125,7 @@ Check if the collection is in an error state isFetching: boolean; ``` -Defined in: [packages/query-db-collection/src/query.ts:193](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L193) +Defined in: [packages/query-db-collection/src/query.ts:291](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L291) Check if query is currently fetching (initial or background) @@ -137,7 +137,7 @@ Check if query is currently fetching (initial or background) isLoading: boolean; ``` -Defined in: [packages/query-db-collection/src/query.ts:197](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L197) +Defined in: [packages/query-db-collection/src/query.ts:295](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L295) Check if query is loading for the first time (no data yet) @@ -149,7 +149,7 @@ Check if query is loading for the first time (no data yet) isRefetching: boolean; ``` -Defined in: [packages/query-db-collection/src/query.ts:195](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L195) +Defined in: [packages/query-db-collection/src/query.ts:293](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L293) Check if query is refetching in background (not initial fetch) @@ -161,7 +161,7 @@ Check if query is refetching in background (not initial fetch) lastError: TError | undefined; ``` -Defined in: [packages/query-db-collection/src/query.ts:184](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L184) +Defined in: [packages/query-db-collection/src/query.ts:282](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L282) Get the last error encountered by the query (if any); reset on success @@ -173,7 +173,7 @@ Get the last error encountered by the query (if any); reset on success refetch: RefetchFn; ``` -Defined in: [packages/query-db-collection/src/query.ts:170](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L170) +Defined in: [packages/query-db-collection/src/query.ts:268](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L268) Manually trigger a refetch of the query @@ -185,7 +185,7 @@ Manually trigger a refetch of the query writeBatch: (callback) => void; ``` -Defined in: [packages/query-db-collection/src/query.ts:180](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L180) +Defined in: [packages/query-db-collection/src/query.ts:278](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L278) Execute multiple write operations as a single atomic batch to the synced data store @@ -207,7 +207,7 @@ Execute multiple write operations as a single atomic batch to the synced data st writeDelete: (keys) => void; ``` -Defined in: [packages/query-db-collection/src/query.ts:176](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L176) +Defined in: [packages/query-db-collection/src/query.ts:274](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L274) Delete one or more items directly from the synced data store without triggering a query refetch or optimistic update @@ -229,7 +229,7 @@ Delete one or more items directly from the synced data store without triggering writeInsert: (data) => void; ``` -Defined in: [packages/query-db-collection/src/query.ts:172](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L172) +Defined in: [packages/query-db-collection/src/query.ts:270](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L270) Insert one or more items directly into the synced data store without triggering a query refetch or optimistic update @@ -251,7 +251,7 @@ Insert one or more items directly into the synced data store without triggering writeUpdate: (updates) => void; ``` -Defined in: [packages/query-db-collection/src/query.ts:174](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L174) +Defined in: [packages/query-db-collection/src/query.ts:272](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L272) Update one or more items directly in the synced data store without triggering a query refetch or optimistic update @@ -273,7 +273,7 @@ Update one or more items directly in the synced data store without triggering a writeUpsert: (data) => void; ``` -Defined in: [packages/query-db-collection/src/query.ts:178](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L178) +Defined in: [packages/query-db-collection/src/query.ts:276](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/query.ts#L276) Insert or update one or more items directly in the synced data store without triggering a query refetch or optimistic update diff --git a/docs/reference/query-db-collection/type-aliases/SyncOperation.md b/docs/reference/query-db-collection/type-aliases/SyncOperation.md index 9cb0b5eb9a..39261d7328 100644 --- a/docs/reference/query-db-collection/type-aliases/SyncOperation.md +++ b/docs/reference/query-db-collection/type-aliases/SyncOperation.md @@ -25,7 +25,7 @@ type SyncOperation = }; ``` -Defined in: [packages/query-db-collection/src/manual-sync.ts:20](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/manual-sync.ts#L20) +Defined in: [packages/query-db-collection/src/manual-sync.ts:24](https://github.com/TanStack/db/blob/main/packages/query-db-collection/src/manual-sync.ts#L24) ## Type Parameters diff --git a/docs/reference/rxdb-db-collection/functions/rxdbCollectionOptions.md b/docs/reference/rxdb-db-collection/functions/rxdbCollectionOptions.md index adf36b74af..84fd8f15f1 100644 --- a/docs/reference/rxdb-db-collection/functions/rxdbCollectionOptions.md +++ b/docs/reference/rxdb-db-collection/functions/rxdbCollectionOptions.md @@ -11,7 +11,7 @@ title: rxdbCollectionOptions function rxdbCollectionOptions(config): CollectionConfig, string, T, UtilsRecord> & object; ``` -Defined in: [rxdb.ts:89](https://github.com/TanStack/db/blob/main/packages/rxdb-db-collection/src/rxdb.ts#L89) +Defined in: [rxdb.ts:90](https://github.com/TanStack/db/blob/main/packages/rxdb-db-collection/src/rxdb.ts#L90) Creates RxDB collection options for use with a standard Collection @@ -41,7 +41,7 @@ Collection options with utilities function rxdbCollectionOptions(config): CollectionConfig & object; ``` -Defined in: [rxdb.ts:96](https://github.com/TanStack/db/blob/main/packages/rxdb-db-collection/src/rxdb.ts#L96) +Defined in: [rxdb.ts:97](https://github.com/TanStack/db/blob/main/packages/rxdb-db-collection/src/rxdb.ts#L97) Creates RxDB collection options for use with a standard Collection diff --git a/docs/reference/rxdb-db-collection/type-aliases/RxDBCollectionConfig.md b/docs/reference/rxdb-db-collection/type-aliases/RxDBCollectionConfig.md index 447463e906..006554157e 100644 --- a/docs/reference/rxdb-db-collection/type-aliases/RxDBCollectionConfig.md +++ b/docs/reference/rxdb-db-collection/type-aliases/RxDBCollectionConfig.md @@ -9,7 +9,7 @@ title: RxDBCollectionConfig type RxDBCollectionConfig = Omit, "onInsert" | "onUpdate" | "onDelete" | "getKey"> & object; ``` -Defined in: [rxdb.ts:49](https://github.com/TanStack/db/blob/main/packages/rxdb-db-collection/src/rxdb.ts#L49) +Defined in: [rxdb.ts:50](https://github.com/TanStack/db/blob/main/packages/rxdb-db-collection/src/rxdb.ts#L50) Configuration interface for RxDB collection options diff --git a/docs/reference/rxdb-db-collection/variables/OPEN_RXDB_SUBSCRIPTIONS.md b/docs/reference/rxdb-db-collection/variables/OPEN_RXDB_SUBSCRIPTIONS.md index 12525d564d..4dad6b2eb5 100644 --- a/docs/reference/rxdb-db-collection/variables/OPEN_RXDB_SUBSCRIPTIONS.md +++ b/docs/reference/rxdb-db-collection/variables/OPEN_RXDB_SUBSCRIPTIONS.md @@ -9,6 +9,6 @@ title: OPEN_RXDB_SUBSCRIPTIONS const OPEN_RXDB_SUBSCRIPTIONS: WeakMap>; ``` -Defined in: [rxdb.ts:31](https://github.com/TanStack/db/blob/main/packages/rxdb-db-collection/src/rxdb.ts#L31) +Defined in: [rxdb.ts:32](https://github.com/TanStack/db/blob/main/packages/rxdb-db-collection/src/rxdb.ts#L32) Used in tests to ensure proper cleanup diff --git a/docs/reference/trailbase-db-collection/functions/trailBaseCollectionOptions.md b/docs/reference/trailbase-db-collection/functions/trailBaseCollectionOptions.md index 5905d86c64..45964382a8 100644 --- a/docs/reference/trailbase-db-collection/functions/trailBaseCollectionOptions.md +++ b/docs/reference/trailbase-db-collection/functions/trailBaseCollectionOptions.md @@ -9,7 +9,7 @@ title: trailBaseCollectionOptions function trailBaseCollectionOptions(config): CollectionConfig & object; ``` -Defined in: [packages/trailbase-db-collection/src/trailbase.ts:121](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L121) +Defined in: [packages/trailbase-db-collection/src/trailbase.ts:122](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L122) ## Type Parameters diff --git a/docs/reference/trailbase-db-collection/interfaces/TrailBaseCollectionConfig.md b/docs/reference/trailbase-db-collection/interfaces/TrailBaseCollectionConfig.md index 40b63eb70c..2ed73d361d 100644 --- a/docs/reference/trailbase-db-collection/interfaces/TrailBaseCollectionConfig.md +++ b/docs/reference/trailbase-db-collection/interfaces/TrailBaseCollectionConfig.md @@ -5,7 +5,7 @@ title: TrailBaseCollectionConfig # Interface: TrailBaseCollectionConfig\ -Defined in: [packages/trailbase-db-collection/src/trailbase.ts:92](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L92) +Defined in: [packages/trailbase-db-collection/src/trailbase.ts:93](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L93) Configuration interface for Trailbase Collection @@ -35,7 +35,7 @@ Configuration interface for Trailbase Collection parse: Conversions; ``` -Defined in: [packages/trailbase-db-collection/src/trailbase.ts:111](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L111) +Defined in: [packages/trailbase-db-collection/src/trailbase.ts:112](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L112) *** @@ -45,7 +45,7 @@ Defined in: [packages/trailbase-db-collection/src/trailbase.ts:111](https://gith recordApi: RecordApi; ``` -Defined in: [packages/trailbase-db-collection/src/trailbase.ts:103](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L103) +Defined in: [packages/trailbase-db-collection/src/trailbase.ts:104](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L104) Record API name @@ -57,7 +57,7 @@ Record API name serialize: Conversions; ``` -Defined in: [packages/trailbase-db-collection/src/trailbase.ts:112](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L112) +Defined in: [packages/trailbase-db-collection/src/trailbase.ts:113](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L113) *** @@ -67,7 +67,7 @@ Defined in: [packages/trailbase-db-collection/src/trailbase.ts:112](https://gith optional syncMode: SyncMode; ``` -Defined in: [packages/trailbase-db-collection/src/trailbase.ts:109](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L109) +Defined in: [packages/trailbase-db-collection/src/trailbase.ts:110](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L110) The mode of sync to use for the collection. diff --git a/docs/reference/trailbase-db-collection/interfaces/TrailBaseCollectionUtils.md b/docs/reference/trailbase-db-collection/interfaces/TrailBaseCollectionUtils.md index ef951da0d7..6786815288 100644 --- a/docs/reference/trailbase-db-collection/interfaces/TrailBaseCollectionUtils.md +++ b/docs/reference/trailbase-db-collection/interfaces/TrailBaseCollectionUtils.md @@ -5,7 +5,7 @@ title: TrailBaseCollectionUtils # Interface: TrailBaseCollectionUtils -Defined in: [packages/trailbase-db-collection/src/trailbase.ts:117](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L117) +Defined in: [packages/trailbase-db-collection/src/trailbase.ts:118](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L118) ## Extends @@ -25,7 +25,7 @@ Defined in: [packages/trailbase-db-collection/src/trailbase.ts:117](https://gith cancel: () => void; ``` -Defined in: [packages/trailbase-db-collection/src/trailbase.ts:118](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L118) +Defined in: [packages/trailbase-db-collection/src/trailbase.ts:119](https://github.com/TanStack/db/blob/main/packages/trailbase-db-collection/src/trailbase.ts#L119) #### Returns diff --git a/docs/reference/type-aliases/ApplyJoinOptionalityToMergedSchema.md b/docs/reference/type-aliases/ApplyJoinOptionalityToMergedSchema.md index de18e0b6d1..f619d3ae04 100644 --- a/docs/reference/type-aliases/ApplyJoinOptionalityToMergedSchema.md +++ b/docs/reference/type-aliases/ApplyJoinOptionalityToMergedSchema.md @@ -3,13 +3,13 @@ id: ApplyJoinOptionalityToMergedSchema title: ApplyJoinOptionalityToMergedSchema --- -# Type Alias: ApplyJoinOptionalityToMergedSchema\ +# Type Alias: ApplyJoinOptionalityToMergedSchema\ ```ts -type ApplyJoinOptionalityToMergedSchema = { [K in keyof TExistingSchema]: K extends TFromSourceName ? TJoinType extends "right" | "full" ? TExistingSchema[K] | undefined : TExistingSchema[K] : TExistingSchema[K] } & { [K in keyof TNewSchema]: TJoinType extends "left" | "full" ? TNewSchema[K] | undefined : TNewSchema[K] }; +type ApplyJoinOptionalityToMergedSchema = { [K in keyof TExistingSchema]: K extends TFromSourceNames ? TJoinType extends "right" | "full" ? TExistingSchema[K] | undefined : TExistingSchema[K] : TExistingSchema[K] } & { [K in keyof TNewSchema]: TJoinType extends "left" | "full" ? TNewSchema[K] | undefined : TNewSchema[K] }; ``` -Defined in: [packages/db/src/query/builder/types.ts:779](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L779) +Defined in: [packages/db/src/query/builder/types.ts:996](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L996) ApplyJoinOptionalityToMergedSchema - Applies optionality rules when merging schemas @@ -49,6 +49,6 @@ into a single type while preserving all table references. `TJoinType` *extends* `"inner"` \| `"left"` \| `"right"` \| `"full"` \| `"outer"` \| `"cross"` -### TFromSourceName +### TFromSourceNames -`TFromSourceName` *extends* `string` +`TFromSourceNames` *extends* `string` diff --git a/docs/reference/type-aliases/ChangeListener.md b/docs/reference/type-aliases/ChangeListener.md index 0d82c8cdbd..3b4ebea53b 100644 --- a/docs/reference/type-aliases/ChangeListener.md +++ b/docs/reference/type-aliases/ChangeListener.md @@ -9,7 +9,7 @@ title: ChangeListener type ChangeListener = (changes) => void; ``` -Defined in: [packages/db/src/types.ts:919](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L919) +Defined in: [packages/db/src/types.ts:1018](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L1018) Function type for listening to collection changes diff --git a/docs/reference/type-aliases/ChangeMessageOrDeleteKeyMessage.md b/docs/reference/type-aliases/ChangeMessageOrDeleteKeyMessage.md index 59b97fa117..80fac28a6e 100644 --- a/docs/reference/type-aliases/ChangeMessageOrDeleteKeyMessage.md +++ b/docs/reference/type-aliases/ChangeMessageOrDeleteKeyMessage.md @@ -11,7 +11,7 @@ type ChangeMessageOrDeleteKeyMessage = | DeleteKeyMessage; ``` -Defined in: [packages/db/src/types.ts:397](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L397) +Defined in: [packages/db/src/types.ts:488](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L488) ## Type Parameters diff --git a/docs/reference/type-aliases/ChangesPayload.md b/docs/reference/type-aliases/ChangesPayload.md index 387c8fb909..0ead4ea5a2 100644 --- a/docs/reference/type-aliases/ChangesPayload.md +++ b/docs/reference/type-aliases/ChangesPayload.md @@ -9,7 +9,7 @@ title: ChangesPayload type ChangesPayload = ChangeMessage, TKey>[]; ``` -Defined in: [packages/db/src/types.ts:779](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L779) +Defined in: [packages/db/src/types.ts:871](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L871) ## Type Parameters diff --git a/docs/reference/type-aliases/CleanupFn.md b/docs/reference/type-aliases/CleanupFn.md index 83dc420aab..5a09880107 100644 --- a/docs/reference/type-aliases/CleanupFn.md +++ b/docs/reference/type-aliases/CleanupFn.md @@ -9,7 +9,7 @@ title: CleanupFn type CleanupFn = () => void; ``` -Defined in: [packages/db/src/types.ts:320](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L320) +Defined in: [packages/db/src/types.ts:380](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L380) ## Returns diff --git a/docs/reference/type-aliases/ClearStorageFn.md b/docs/reference/type-aliases/ClearStorageFn.md index 05751d0912..adaf881b6f 100644 --- a/docs/reference/type-aliases/ClearStorageFn.md +++ b/docs/reference/type-aliases/ClearStorageFn.md @@ -9,7 +9,7 @@ title: ClearStorageFn type ClearStorageFn = () => void; ``` -Defined in: [packages/db/src/local-storage.ts:90](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L90) +Defined in: [packages/db/src/local-storage.ts:92](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L92) Type for the clear utility function diff --git a/docs/reference/type-aliases/CollectionConfigSingleRowOption.md b/docs/reference/type-aliases/CollectionConfigSingleRowOption.md index b376bac57f..896b114db1 100644 --- a/docs/reference/type-aliases/CollectionConfigSingleRowOption.md +++ b/docs/reference/type-aliases/CollectionConfigSingleRowOption.md @@ -9,7 +9,7 @@ title: CollectionConfigSingleRowOption type CollectionConfigSingleRowOption = CollectionConfig & MaybeSingleResult; ``` -Defined in: [packages/db/src/types.ts:772](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L772) +Defined in: [packages/db/src/types.ts:864](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L864) ## Type Parameters diff --git a/docs/reference/type-aliases/CollectionMaterializeOptions.md b/docs/reference/type-aliases/CollectionMaterializeOptions.md new file mode 100644 index 0000000000..8e5331d31e --- /dev/null +++ b/docs/reference/type-aliases/CollectionMaterializeOptions.md @@ -0,0 +1,28 @@ +--- +id: CollectionMaterializeOptions +title: CollectionMaterializeOptions +--- + +# Type Alias: CollectionMaterializeOptions\ + +```ts +type CollectionMaterializeOptions = object; +``` + +Defined in: [packages/db/src/client.ts:94](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L94) + +## Type Parameters + +### T + +`T` *extends* `object` + +## Properties + +### initialData? + +```ts +optional initialData: T[]; +``` + +Defined in: [packages/db/src/client.ts:95](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L95) diff --git a/docs/reference/type-aliases/CollectionOptions.md b/docs/reference/type-aliases/CollectionOptions.md new file mode 100644 index 0000000000..f701d7d332 --- /dev/null +++ b/docs/reference/type-aliases/CollectionOptions.md @@ -0,0 +1,30 @@ +--- +id: CollectionOptions +title: CollectionOptions +--- + +# Type Alias: CollectionOptions\ + +```ts +type CollectionOptions = CollectionOptionsIdentity; +``` + +Defined in: [packages/db/src/client.ts:39](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L39) + +## Type Parameters + +### T + +`T` *extends* `object` = `Record`\<`string`, `unknown`\> + +### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +### TSchema + +`TSchema` *extends* `StandardSchemaV1` = `never` + +### TUtils + +`TUtils` *extends* [`UtilsRecord`](UtilsRecord.md) = [`UtilsRecord`](UtilsRecord.md) diff --git a/docs/reference/type-aliases/CollectionStatus.md b/docs/reference/type-aliases/CollectionStatus.md index 9138181949..1654a908d6 100644 --- a/docs/reference/type-aliases/CollectionStatus.md +++ b/docs/reference/type-aliases/CollectionStatus.md @@ -9,7 +9,7 @@ title: CollectionStatus type CollectionStatus = "idle" | "loading" | "ready" | "error" | "cleaned-up"; ``` -Defined in: [packages/db/src/types.ts:507](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L507) +Defined in: [packages/db/src/types.ts:599](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L599) Collection status values for lifecycle management @@ -27,5 +27,6 @@ if (collection.status === "loading") { ```ts // Status transitions // idle → loading → ready (when markReady() is called) -// Any status can transition to → error or cleaned-up +// Any active status can transition to → error or cleaned-up +// error → ready after a successful sync recovery ``` diff --git a/docs/reference/type-aliases/ContextFromSource.md b/docs/reference/type-aliases/ContextFromSource.md new file mode 100644 index 0000000000..860929df9e --- /dev/null +++ b/docs/reference/type-aliases/ContextFromSource.md @@ -0,0 +1,58 @@ +--- +id: ContextFromSource +title: ContextFromSource +--- + +# Type Alias: ContextFromSource\ + +```ts +type ContextFromSource = object; +``` + +Defined in: [packages/db/src/query/builder/types.ts:151](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L151) + +## Type Parameters + +### TSource + +`TSource` *extends* [`Source`](Source.md) + +## Properties + +### baseSchema + +```ts +baseSchema: SchemaFromSource; +``` + +Defined in: [packages/db/src/query/builder/types.ts:152](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L152) + +*** + +### fromSourceName + +```ts +fromSourceName: keyof TSource & string; +``` + +Defined in: [packages/db/src/query/builder/types.ts:154](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L154) + +*** + +### hasJoins + +```ts +hasJoins: false; +``` + +Defined in: [packages/db/src/query/builder/types.ts:155](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L155) + +*** + +### schema + +```ts +schema: SchemaFromSource; +``` + +Defined in: [packages/db/src/query/builder/types.ts:153](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L153) diff --git a/docs/reference/type-aliases/ContextFromUnionBranches.md b/docs/reference/type-aliases/ContextFromUnionBranches.md new file mode 100644 index 0000000000..95c0ebfc1f --- /dev/null +++ b/docs/reference/type-aliases/ContextFromUnionBranches.md @@ -0,0 +1,88 @@ +--- +id: ContextFromUnionBranches +title: ContextFromUnionBranches +--- + +# Type Alias: ContextFromUnionBranches\ + +```ts +type ContextFromUnionBranches = object; +``` + +Defined in: [packages/db/src/query/builder/types.ts:183](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L183) + +## Type Parameters + +### TBranches + +`TBranches` *extends* readonly \[[`QueryBuilder`](QueryBuilder.md)\<`any`\>, `...QueryBuilder[]`\] + +## Properties + +### baseSchema + +```ts +baseSchema: UnionBranchSchema & ContextSchema; +``` + +Defined in: [packages/db/src/query/builder/types.ts:186](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L186) + +*** + +### fromSourceName + +```ts +fromSourceName: keyof UnionBranchSchema & string; +``` + +Defined in: [packages/db/src/query/builder/types.ts:189](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L189) + +*** + +### hasJoins + +```ts +hasJoins: false; +``` + +Defined in: [packages/db/src/query/builder/types.ts:190](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L190) + +*** + +### hasResult + +```ts +hasResult: true; +``` + +Defined in: [packages/db/src/query/builder/types.ts:192](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L192) + +*** + +### refsSchema + +```ts +refsSchema: UnionBranchSchema; +``` + +Defined in: [packages/db/src/query/builder/types.ts:188](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L188) + +*** + +### result + +```ts +result: PrettifyIfPlainObject>; +``` + +Defined in: [packages/db/src/query/builder/types.ts:191](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L191) + +*** + +### schema + +```ts +schema: UnionBranchSchema & ContextSchema; +``` + +Defined in: [packages/db/src/query/builder/types.ts:187](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L187) diff --git a/docs/reference/type-aliases/ContextFromUnionSource.md b/docs/reference/type-aliases/ContextFromUnionSource.md new file mode 100644 index 0000000000..3f286ef90d --- /dev/null +++ b/docs/reference/type-aliases/ContextFromUnionSource.md @@ -0,0 +1,18 @@ +--- +id: ContextFromUnionSource +title: ContextFromUnionSource +--- + +# Type Alias: ContextFromUnionSource\ + +```ts +type ContextFromUnionSource = IsUnion extends true ? object : ContextFromSource; +``` + +Defined in: [packages/db/src/query/builder/types.ts:158](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L158) + +## Type Parameters + +### TSource + +`TSource` *extends* [`Source`](Source.md) diff --git a/docs/reference/type-aliases/ContextSchema.md b/docs/reference/type-aliases/ContextSchema.md index 09d96d6a38..33443849f8 100644 --- a/docs/reference/type-aliases/ContextSchema.md +++ b/docs/reference/type-aliases/ContextSchema.md @@ -9,7 +9,7 @@ title: ContextSchema type ContextSchema = Record; ``` -Defined in: [packages/db/src/query/builder/types.ts:68](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L68) +Defined in: [packages/db/src/query/builder/types.ts:81](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L81) ContextSchema - The shape of available tables/collections in a query context diff --git a/docs/reference/type-aliases/CursorExpressions.md b/docs/reference/type-aliases/CursorExpressions.md index 2f84898f30..126d9f5bcc 100644 --- a/docs/reference/type-aliases/CursorExpressions.md +++ b/docs/reference/type-aliases/CursorExpressions.md @@ -9,7 +9,7 @@ title: CursorExpressions type CursorExpressions = object; ``` -Defined in: [packages/db/src/types.ts:266](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L266) +Defined in: [packages/db/src/types.ts:283](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L283) Cursor expressions for pagination, passed separately from the main `where` clause. The sync layer can choose to use cursor-based pagination (combining these with the where) @@ -25,7 +25,7 @@ Neither expression includes the main `where` clause - they are cursor-specific o optional lastKey: string | number; ``` -Defined in: [packages/db/src/types.ts:284](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L284) +Defined in: [packages/db/src/types.ts:300](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L300) The key of the last item that was loaded. Can be used by sync layers for tracking or deduplication. @@ -38,7 +38,7 @@ Can be used by sync layers for tracking or deduplication. whereCurrent: BasicExpression; ``` -Defined in: [packages/db/src/types.ts:279](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L279) +Defined in: [packages/db/src/types.ts:295](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L295) Expression for rows equal to the current cursor value (first orderBy column only). Used to handle tie-breaking/duplicates at the boundary. @@ -52,9 +52,8 @@ Example: eq(col1, v1) or for Dates: and(gte(col1, v1), lt(col1, v1+1ms)) whereFrom: BasicExpression; ``` -Defined in: [packages/db/src/types.ts:273](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L273) +Defined in: [packages/db/src/types.ts:289](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L289) Expression for rows greater than (after) the cursor value. -For multi-column orderBy, this is a composite cursor using OR of conditions. -Example for [col1 ASC, col2 DESC] with values [v1, v2]: - or(gt(col1, v1), and(eq(col1, v1), lt(col2, v2))) +Core emits cursors for a single order column. Multi-column queries use +prefix-and-tie loading instead of constructing a composite cursor. diff --git a/docs/reference/type-aliases/DbClientEvent.md b/docs/reference/type-aliases/DbClientEvent.md new file mode 100644 index 0000000000..5481d5895d --- /dev/null +++ b/docs/reference/type-aliases/DbClientEvent.md @@ -0,0 +1,20 @@ +--- +id: DbClientEvent +title: DbClientEvent +--- + +# Type Alias: DbClientEvent + +```ts +type DbClientEvent = + | { + query: DbClientLiveQuery; + type: "liveQueryAdded" | "liveQueryUpdated"; +} + | { + error: unknown; + type: "liveQueryStreamError"; +}; +``` + +Defined in: [packages/db/src/client.ts:146](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L146) diff --git a/docs/reference/type-aliases/DbClientLiveQuery.md b/docs/reference/type-aliases/DbClientLiveQuery.md new file mode 100644 index 0000000000..a1bcf85c2a --- /dev/null +++ b/docs/reference/type-aliases/DbClientLiveQuery.md @@ -0,0 +1,72 @@ +--- +id: DbClientLiveQuery +title: DbClientLiveQuery +--- + +# Type Alias: DbClientLiveQuery + +```ts +type DbClientLiveQuery = object; +``` + +Defined in: [packages/db/src/client.ts:137](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L137) + +## Properties + +### dehydratedAt + +```ts +readonly dehydratedAt: number; +``` + +Defined in: [packages/db/src/client.ts:139](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L139) + +*** + +### error? + +```ts +readonly optional error: unknown; +``` + +Defined in: [packages/db/src/client.ts:143](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L143) + +*** + +### promise + +```ts +readonly promise: Promise; +``` + +Defined in: [packages/db/src/client.ts:141](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L141) + +*** + +### queryHash + +```ts +readonly queryHash: string; +``` + +Defined in: [packages/db/src/client.ts:138](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L138) + +*** + +### snapshot? + +```ts +readonly optional snapshot: DehydratedLiveQueryResult; +``` + +Defined in: [packages/db/src/client.ts:142](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L142) + +*** + +### status + +```ts +readonly status: DbClientLiveQueryState; +``` + +Defined in: [packages/db/src/client.ts:140](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L140) diff --git a/docs/reference/type-aliases/DbClientLiveQueryState.md b/docs/reference/type-aliases/DbClientLiveQueryState.md new file mode 100644 index 0000000000..44c35dddcf --- /dev/null +++ b/docs/reference/type-aliases/DbClientLiveQueryState.md @@ -0,0 +1,12 @@ +--- +id: DbClientLiveQueryState +title: DbClientLiveQueryState +--- + +# Type Alias: DbClientLiveQueryState + +```ts +type DbClientLiveQueryState = "pending" | "success" | "error"; +``` + +Defined in: [packages/db/src/client.ts:135](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L135) diff --git a/docs/reference/type-aliases/DbClientOptions.md b/docs/reference/type-aliases/DbClientOptions.md new file mode 100644 index 0000000000..5bf1450350 --- /dev/null +++ b/docs/reference/type-aliases/DbClientOptions.md @@ -0,0 +1,12 @@ +--- +id: DbClientOptions +title: DbClientOptions +--- + +# Type Alias: DbClientOptions + +```ts +type DbClientOptions = Record; +``` + +Defined in: [packages/db/src/client.ts:178](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L178) diff --git a/docs/reference/type-aliases/DeferredLiveQueryCollections.md b/docs/reference/type-aliases/DeferredLiveQueryCollections.md new file mode 100644 index 0000000000..c82a8f56c5 --- /dev/null +++ b/docs/reference/type-aliases/DeferredLiveQueryCollections.md @@ -0,0 +1,12 @@ +--- +id: DeferredLiveQueryCollections +title: DeferredLiveQueryCollections +--- + +# Type Alias: DeferredLiveQueryCollections + +```ts +type DeferredLiveQueryCollections = Set>; +``` + +Defined in: [packages/db/src/live-query-options.ts:23](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-options.ts#L23) diff --git a/docs/reference/type-aliases/DehydrateDbClientOptions.md b/docs/reference/type-aliases/DehydrateDbClientOptions.md new file mode 100644 index 0000000000..cd23499b1c --- /dev/null +++ b/docs/reference/type-aliases/DehydrateDbClientOptions.md @@ -0,0 +1,52 @@ +--- +id: DehydrateDbClientOptions +title: DehydrateDbClientOptions +--- + +# Type Alias: DehydrateDbClientOptions + +```ts +type DehydrateDbClientOptions = object; +``` + +Defined in: [packages/db/src/client.ts:156](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L156) + +## Properties + +### shouldDehydrateCollection()? + +```ts +optional shouldDehydrateCollection: (collection) => boolean; +``` + +Defined in: [packages/db/src/client.ts:157](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L157) + +#### Parameters + +##### collection + +[`Collection`](../interfaces/Collection.md) + +#### Returns + +`boolean` + +*** + +### shouldDehydrateLiveQuery()? + +```ts +optional shouldDehydrateLiveQuery: (query) => boolean; +``` + +Defined in: [packages/db/src/client.ts:158](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L158) + +#### Parameters + +##### query + +[`DbClientLiveQuery`](DbClientLiveQuery.md) + +#### Returns + +`boolean` diff --git a/docs/reference/type-aliases/DehydratedCollectionChunk.md b/docs/reference/type-aliases/DehydratedCollectionChunk.md new file mode 100644 index 0000000000..c6cbcb329a --- /dev/null +++ b/docs/reference/type-aliases/DehydratedCollectionChunk.md @@ -0,0 +1,52 @@ +--- +id: DehydratedCollectionChunk +title: DehydratedCollectionChunk +--- + +# Type Alias: DehydratedCollectionChunk\ + +```ts +type DehydratedCollectionChunk = object; +``` + +Defined in: [packages/db/src/client.ts:107](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L107) + +## Type Parameters + +### T + +`T` *extends* `object` = `Record`\<`string`, `unknown`\> + +### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +## Properties + +### collectionId + +```ts +collectionId: string; +``` + +Defined in: [packages/db/src/client.ts:111](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L111) + +*** + +### rows + +```ts +rows: DehydratedCollectionRow[]; +``` + +Defined in: [packages/db/src/client.ts:112](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L112) + +*** + +### syncMeta? + +```ts +optional syncMeta: unknown; +``` + +Defined in: [packages/db/src/client.ts:113](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L113) diff --git a/docs/reference/type-aliases/DehydratedCollectionRow.md b/docs/reference/type-aliases/DehydratedCollectionRow.md new file mode 100644 index 0000000000..143761ff30 --- /dev/null +++ b/docs/reference/type-aliases/DehydratedCollectionRow.md @@ -0,0 +1,52 @@ +--- +id: DehydratedCollectionRow +title: DehydratedCollectionRow +--- + +# Type Alias: DehydratedCollectionRow\ + +```ts +type DehydratedCollectionRow = object; +``` + +Defined in: [packages/db/src/client.ts:98](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L98) + +## Type Parameters + +### T + +`T` *extends* `object` = `Record`\<`string`, `unknown`\> + +### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +## Properties + +### key + +```ts +key: TKey; +``` + +Defined in: [packages/db/src/client.ts:102](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L102) + +*** + +### metadata? + +```ts +optional metadata: unknown; +``` + +Defined in: [packages/db/src/client.ts:104](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L104) + +*** + +### value + +```ts +value: T; +``` + +Defined in: [packages/db/src/client.ts:103](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L103) diff --git a/docs/reference/type-aliases/DehydratedDbState.md b/docs/reference/type-aliases/DehydratedDbState.md new file mode 100644 index 0000000000..97de8572d3 --- /dev/null +++ b/docs/reference/type-aliases/DehydratedDbState.md @@ -0,0 +1,32 @@ +--- +id: DehydratedDbState +title: DehydratedDbState +--- + +# Type Alias: DehydratedDbState + +```ts +type DehydratedDbState = object; +``` + +Defined in: [packages/db/src/client.ts:130](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L130) + +## Properties + +### collections + +```ts +collections: DehydratedCollectionChunk[]; +``` + +Defined in: [packages/db/src/client.ts:131](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L131) + +*** + +### liveQueries? + +```ts +optional liveQueries: DehydratedLiveQuery[]; +``` + +Defined in: [packages/db/src/client.ts:132](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L132) diff --git a/docs/reference/type-aliases/DehydratedLiveQuery.md b/docs/reference/type-aliases/DehydratedLiveQuery.md new file mode 100644 index 0000000000..c24db1a55d --- /dev/null +++ b/docs/reference/type-aliases/DehydratedLiveQuery.md @@ -0,0 +1,52 @@ +--- +id: DehydratedLiveQuery +title: DehydratedLiveQuery +--- + +# Type Alias: DehydratedLiveQuery + +```ts +type DehydratedLiveQuery = object; +``` + +Defined in: [packages/db/src/client.ts:116](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L116) + +## Properties + +### dehydratedAt + +```ts +dehydratedAt: number; +``` + +Defined in: [packages/db/src/client.ts:118](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L118) + +*** + +### promise? + +```ts +optional promise: Promise; +``` + +Defined in: [packages/db/src/client.ts:120](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L120) + +*** + +### queryHash + +```ts +queryHash: string; +``` + +Defined in: [packages/db/src/client.ts:117](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L117) + +*** + +### snapshot? + +```ts +optional snapshot: DehydratedLiveQueryResult; +``` + +Defined in: [packages/db/src/client.ts:119](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L119) diff --git a/docs/reference/type-aliases/DehydratedLiveQueryResult.md b/docs/reference/type-aliases/DehydratedLiveQueryResult.md new file mode 100644 index 0000000000..d989ecaca6 --- /dev/null +++ b/docs/reference/type-aliases/DehydratedLiveQueryResult.md @@ -0,0 +1,32 @@ +--- +id: DehydratedLiveQueryResult +title: DehydratedLiveQueryResult +--- + +# Type Alias: DehydratedLiveQueryResult\ + +```ts +type DehydratedLiveQueryResult = object; +``` + +Defined in: [packages/db/src/client.ts:123](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L123) + +## Type Parameters + +### T + +`T` *extends* `object` = `object` + +### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +## Properties + +### rows + +```ts +rows: DehydratedCollectionRow[]; +``` + +Defined in: [packages/db/src/client.ts:127](https://github.com/TanStack/db/blob/main/packages/db/src/client.ts#L127) diff --git a/docs/reference/type-aliases/DeleteKeyMessage.md b/docs/reference/type-aliases/DeleteKeyMessage.md index c28ab6b531..742cba48ef 100644 --- a/docs/reference/type-aliases/DeleteKeyMessage.md +++ b/docs/reference/type-aliases/DeleteKeyMessage.md @@ -9,7 +9,7 @@ title: DeleteKeyMessage type DeleteKeyMessage = Omit, "value" | "previousValue" | "type"> & object; ``` -Defined in: [packages/db/src/types.ts:392](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L392) +Defined in: [packages/db/src/types.ts:483](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L483) ## Type Declaration diff --git a/docs/reference/type-aliases/DeleteMutationFn.md b/docs/reference/type-aliases/DeleteMutationFn.md index ff5b3bd6f2..a62140919c 100644 --- a/docs/reference/type-aliases/DeleteMutationFn.md +++ b/docs/reference/type-aliases/DeleteMutationFn.md @@ -9,7 +9,7 @@ title: DeleteMutationFn type DeleteMutationFn = (params) => Promise; ``` -Defined in: [packages/db/src/types.ts:485](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L485) +Defined in: [packages/db/src/types.ts:576](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L576) ## Type Parameters diff --git a/docs/reference/type-aliases/DeleteMutationFnParams.md b/docs/reference/type-aliases/DeleteMutationFnParams.md index a0480fd0e2..0d627c9afa 100644 --- a/docs/reference/type-aliases/DeleteMutationFnParams.md +++ b/docs/reference/type-aliases/DeleteMutationFnParams.md @@ -9,7 +9,7 @@ title: DeleteMutationFnParams type DeleteMutationFnParams = object; ``` -Defined in: [packages/db/src/types.ts:462](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L462) +Defined in: [packages/db/src/types.ts:553](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L553) ## Type Parameters @@ -33,7 +33,7 @@ Defined in: [packages/db/src/types.ts:462](https://github.com/TanStack/db/blob/m collection: Collection; ``` -Defined in: [packages/db/src/types.ts:468](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L468) +Defined in: [packages/db/src/types.ts:559](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L559) *** @@ -43,4 +43,4 @@ Defined in: [packages/db/src/types.ts:468](https://github.com/TanStack/db/blob/m transaction: TransactionWithMutations; ``` -Defined in: [packages/db/src/types.ts:467](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L467) +Defined in: [packages/db/src/types.ts:558](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L558) diff --git a/docs/reference/type-aliases/DeltaEvent.md b/docs/reference/type-aliases/DeltaEvent.md index 9fd3b89093..3cebfb4e37 100644 --- a/docs/reference/type-aliases/DeltaEvent.md +++ b/docs/reference/type-aliases/DeltaEvent.md @@ -28,7 +28,7 @@ type DeltaEvent = }; ``` -Defined in: [packages/db/src/query/effect.ts:38](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L38) +Defined in: [packages/db/src/query/effect.ts:45](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L45) Delta event emitted when a row enters, exits, or updates within a query result diff --git a/docs/reference/type-aliases/DeltaType.md b/docs/reference/type-aliases/DeltaType.md index 7d0aeedff4..bb0a14a3cc 100644 --- a/docs/reference/type-aliases/DeltaType.md +++ b/docs/reference/type-aliases/DeltaType.md @@ -9,6 +9,6 @@ title: DeltaType type DeltaType = "enter" | "exit" | "update"; ``` -Defined in: [packages/db/src/query/effect.ts:35](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L35) +Defined in: [packages/db/src/query/effect.ts:42](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L42) Event types for query result deltas diff --git a/docs/reference/type-aliases/DemandKey.md b/docs/reference/type-aliases/DemandKey.md new file mode 100644 index 0000000000..905a1a69c2 --- /dev/null +++ b/docs/reference/type-aliases/DemandKey.md @@ -0,0 +1,22 @@ +--- +id: DemandKey +title: DemandKey +--- + +# Type Alias: DemandKey + +```ts +type DemandKey = string & object; +``` + +Defined in: [packages/db/src/query/ir-stable-identity.ts:50](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L50) + +Exact identity for one loadSubset demand, including its requested window. + +## Type Declaration + +### \[demandKeyBrand\] + +```ts +readonly [demandKeyBrand]: true; +``` diff --git a/docs/reference/type-aliases/EffectQueryInput.md b/docs/reference/type-aliases/EffectQueryInput.md index 6b171ac5f6..cdf4c3810d 100644 --- a/docs/reference/type-aliases/EffectQueryInput.md +++ b/docs/reference/type-aliases/EffectQueryInput.md @@ -11,7 +11,7 @@ type EffectQueryInput = | QueryBuilder; ``` -Defined in: [packages/db/src/query/effect.ts:75](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L75) +Defined in: [packages/db/src/query/effect.ts:82](https://github.com/TanStack/db/blob/main/packages/db/src/query/effect.ts#L82) Query input - can be a builder function or a prebuilt query diff --git a/docs/reference/type-aliases/ExtractContext.md b/docs/reference/type-aliases/ExtractContext.md index 4bbdc55025..b0ab947143 100644 --- a/docs/reference/type-aliases/ExtractContext.md +++ b/docs/reference/type-aliases/ExtractContext.md @@ -9,7 +9,7 @@ title: ExtractContext type ExtractContext = T extends BaseQueryBuilder ? TContext : T extends QueryBuilder ? TContext : never; ``` -Defined in: [packages/db/src/query/builder/index.ts:1235](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L1235) +Defined in: [packages/db/src/query/builder/index.ts:1643](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L1643) ## Type Parameters diff --git a/docs/reference/type-aliases/FunctionalHavingRow.md b/docs/reference/type-aliases/FunctionalHavingRow.md index 039184e3c1..ed8ddce62d 100644 --- a/docs/reference/type-aliases/FunctionalHavingRow.md +++ b/docs/reference/type-aliases/FunctionalHavingRow.md @@ -9,7 +9,7 @@ title: FunctionalHavingRow type FunctionalHavingRow = TContext["schema"] & TContext["hasResult"] extends true ? object : object; ``` -Defined in: [packages/db/src/query/builder/types.ts:483](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L483) +Defined in: [packages/db/src/query/builder/types.ts:649](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L649) FunctionalHavingRow - Type for the row parameter in functional having callbacks diff --git a/docs/reference/type-aliases/GetResult.md b/docs/reference/type-aliases/GetResult.md index 36f2043108..95ac01c25a 100644 --- a/docs/reference/type-aliases/GetResult.md +++ b/docs/reference/type-aliases/GetResult.md @@ -9,7 +9,7 @@ title: GetResult type GetResult = Prettify>; ``` -Defined in: [packages/db/src/query/builder/types.ts:848](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L848) +Defined in: [packages/db/src/query/builder/types.ts:1110](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L1110) ## Type Parameters diff --git a/docs/reference/type-aliases/GetStorageSizeFn.md b/docs/reference/type-aliases/GetStorageSizeFn.md index 2008fbbd6c..a385c9dfa3 100644 --- a/docs/reference/type-aliases/GetStorageSizeFn.md +++ b/docs/reference/type-aliases/GetStorageSizeFn.md @@ -9,7 +9,7 @@ title: GetStorageSizeFn type GetStorageSizeFn = () => number; ``` -Defined in: [packages/db/src/local-storage.ts:95](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L95) +Defined in: [packages/db/src/local-storage.ts:97](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L97) Type for the getStorageSize utility function diff --git a/docs/reference/type-aliases/GroupByCallback.md b/docs/reference/type-aliases/GroupByCallback.md index d5f6e0d5e9..c5f1781f7a 100644 --- a/docs/reference/type-aliases/GroupByCallback.md +++ b/docs/reference/type-aliases/GroupByCallback.md @@ -9,7 +9,7 @@ title: GroupByCallback type GroupByCallback = (refs) => any; ``` -Defined in: [packages/db/src/query/builder/types.ts:446](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L446) +Defined in: [packages/db/src/query/builder/types.ts:612](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L612) GroupByCallback - Type for groupBy clause callback functions diff --git a/docs/reference/type-aliases/IndexConstructor.md b/docs/reference/type-aliases/IndexConstructor.md index 0a64c46ab2..b5554c3b90 100644 --- a/docs/reference/type-aliases/IndexConstructor.md +++ b/docs/reference/type-aliases/IndexConstructor.md @@ -9,7 +9,7 @@ title: IndexConstructor type IndexConstructor = (id, expression, name?, options?) => BaseIndex; ``` -Defined in: [packages/db/src/indexes/base-index.ts:213](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L213) +Defined in: [packages/db/src/indexes/base-index.ts:302](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L302) Type for index constructor diff --git a/docs/reference/type-aliases/IndexOperation-1.md b/docs/reference/type-aliases/IndexOperation-1.md index 3dbce14d12..025823309a 100644 --- a/docs/reference/type-aliases/IndexOperation-1.md +++ b/docs/reference/type-aliases/IndexOperation-1.md @@ -9,6 +9,6 @@ title: IndexOperation type IndexOperation = typeof comparisonFunctions[number]; ``` -Defined in: [packages/db/src/indexes/base-index.ts:11](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L11) +Defined in: [packages/db/src/indexes/base-index.ts:34](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L34) Type for index operation values diff --git a/docs/reference/type-aliases/IndexOperation.md b/docs/reference/type-aliases/IndexOperation.md index 23f84e1c09..42c757f781 100644 --- a/docs/reference/type-aliases/IndexOperation.md +++ b/docs/reference/type-aliases/IndexOperation.md @@ -9,6 +9,6 @@ title: IndexOperation type IndexOperation = readonly ["eq", "gt", "gte", "lt", "lte", "in", "like", "ilike"]; ``` -Defined in: [packages/db/src/indexes/base-index.ts:11](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L11) +Defined in: [packages/db/src/indexes/base-index.ts:34](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L34) Operations that indexes can support, imported from available comparison functions diff --git a/docs/reference/type-aliases/IndexReader.md b/docs/reference/type-aliases/IndexReader.md new file mode 100644 index 0000000000..8926baa848 --- /dev/null +++ b/docs/reference/type-aliases/IndexReader.md @@ -0,0 +1,28 @@ +--- +id: IndexReader +title: IndexReader +--- + +# Type Alias: IndexReader\ + +```ts +type IndexReader = Pick, + | "lookup" + | "rangeQuery" + | "take" + | "takeFromStart" + | "keyCount" + | "supports" + | "supportsRangeOptimization" +| "canOptimizeRangeFor">; +``` + +Defined in: [packages/db/src/indexes/base-index.ts:42](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L42) + +The read-side surface consumers use on a resolved (possibly reversed) index. + +## Type Parameters + +### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` diff --git a/docs/reference/type-aliases/InferCollectionType.md b/docs/reference/type-aliases/InferCollectionType.md index 466fb9aeba..08e32121ad 100644 --- a/docs/reference/type-aliases/InferCollectionType.md +++ b/docs/reference/type-aliases/InferCollectionType.md @@ -6,10 +6,10 @@ title: InferCollectionType # Type Alias: InferCollectionType\ ```ts -type InferCollectionType = T extends CollectionImpl ? WithVirtualProps : never; +type InferCollectionType = T extends CollectionImpl ? WithVirtualProps : T extends CollectionOptionsIdentity ? WithVirtualProps : never; ``` -Defined in: [packages/db/src/query/builder/types.ts:89](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L89) +Defined in: [packages/db/src/query/builder/types.ts:105](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L105) InferCollectionType - Extracts the TypeScript type from a CollectionImpl diff --git a/docs/reference/type-aliases/InferResultType.md b/docs/reference/type-aliases/InferResultType.md index 15b1f23375..72e5762af7 100644 --- a/docs/reference/type-aliases/InferResultType.md +++ b/docs/reference/type-aliases/InferResultType.md @@ -9,7 +9,7 @@ title: InferResultType type InferResultType = TContext extends SingleResult ? GetResult | undefined : GetResult[]; ``` -Defined in: [packages/db/src/query/builder/types.ts:805](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L805) +Defined in: [packages/db/src/query/builder/types.ts:1022](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L1022) Utility type to infer the query result size (single row or an array) diff --git a/docs/reference/type-aliases/InitialQueryBuilder.md b/docs/reference/type-aliases/InitialQueryBuilder.md index 1308b07fc3..b9de80b192 100644 --- a/docs/reference/type-aliases/InitialQueryBuilder.md +++ b/docs/reference/type-aliases/InitialQueryBuilder.md @@ -6,7 +6,7 @@ title: InitialQueryBuilder # Type Alias: InitialQueryBuilder ```ts -type InitialQueryBuilder = Pick, "from">; +type InitialQueryBuilder = Pick, "from" | "unionAll">; ``` -Defined in: [packages/db/src/query/builder/index.ts:1221](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L1221) +Defined in: [packages/db/src/query/builder/index.ts:1626](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L1626) diff --git a/docs/reference/type-aliases/InputRow.md b/docs/reference/type-aliases/InputRow.md index 62bab028ff..a8960225a7 100644 --- a/docs/reference/type-aliases/InputRow.md +++ b/docs/reference/type-aliases/InputRow.md @@ -9,6 +9,6 @@ title: InputRow type InputRow = [unknown, Record]; ``` -Defined in: [packages/db/src/types.ts:787](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L787) +Defined in: [packages/db/src/types.ts:879](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L879) An input row from a collection diff --git a/docs/reference/type-aliases/InsertMutationFn.md b/docs/reference/type-aliases/InsertMutationFn.md index 219f71c9ad..ab7352cb1d 100644 --- a/docs/reference/type-aliases/InsertMutationFn.md +++ b/docs/reference/type-aliases/InsertMutationFn.md @@ -9,7 +9,7 @@ title: InsertMutationFn type InsertMutationFn = (params) => Promise; ``` -Defined in: [packages/db/src/types.ts:471](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L471) +Defined in: [packages/db/src/types.ts:562](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L562) ## Type Parameters diff --git a/docs/reference/type-aliases/InsertMutationFnParams.md b/docs/reference/type-aliases/InsertMutationFnParams.md index 8607d1dba1..220ebbf56c 100644 --- a/docs/reference/type-aliases/InsertMutationFnParams.md +++ b/docs/reference/type-aliases/InsertMutationFnParams.md @@ -9,7 +9,7 @@ title: InsertMutationFnParams type InsertMutationFnParams = object; ``` -Defined in: [packages/db/src/types.ts:454](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L454) +Defined in: [packages/db/src/types.ts:545](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L545) ## Type Parameters @@ -33,7 +33,7 @@ Defined in: [packages/db/src/types.ts:454](https://github.com/TanStack/db/blob/m collection: Collection; ``` -Defined in: [packages/db/src/types.ts:460](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L460) +Defined in: [packages/db/src/types.ts:551](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L551) *** @@ -43,4 +43,4 @@ Defined in: [packages/db/src/types.ts:460](https://github.com/TanStack/db/blob/m transaction: TransactionWithMutations; ``` -Defined in: [packages/db/src/types.ts:459](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L459) +Defined in: [packages/db/src/types.ts:550](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L550) diff --git a/docs/reference/type-aliases/JoinOnCallback.md b/docs/reference/type-aliases/JoinOnCallback.md index 27ba29f3ad..f1eab2e165 100644 --- a/docs/reference/type-aliases/JoinOnCallback.md +++ b/docs/reference/type-aliases/JoinOnCallback.md @@ -9,7 +9,7 @@ title: JoinOnCallback type JoinOnCallback = (refs) => any; ``` -Defined in: [packages/db/src/query/builder/types.ts:462](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L462) +Defined in: [packages/db/src/query/builder/types.ts:628](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L628) JoinOnCallback - Type for join condition callback functions diff --git a/docs/reference/type-aliases/KeyedNamespacedRow.md b/docs/reference/type-aliases/KeyedNamespacedRow.md index d5f1694b3f..227a793ab8 100644 --- a/docs/reference/type-aliases/KeyedNamespacedRow.md +++ b/docs/reference/type-aliases/KeyedNamespacedRow.md @@ -9,7 +9,7 @@ title: KeyedNamespacedRow type KeyedNamespacedRow = [unknown, NamespacedRow]; ``` -Defined in: [packages/db/src/types.ts:810](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L810) +Defined in: [packages/db/src/types.ts:902](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L902) A keyed namespaced row is a row with a key and a namespaced row This is the main representation of a row in a query pipeline diff --git a/docs/reference/type-aliases/KeyedStream.md b/docs/reference/type-aliases/KeyedStream.md index 20d1aac876..a47e3c904a 100644 --- a/docs/reference/type-aliases/KeyedStream.md +++ b/docs/reference/type-aliases/KeyedStream.md @@ -9,7 +9,7 @@ title: KeyedStream type KeyedStream = IStreamBuilder; ``` -Defined in: [packages/db/src/types.ts:793](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L793) +Defined in: [packages/db/src/types.ts:885](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L885) A keyed stream is a stream of rows This is used as the inputs from a collection to a query diff --git a/docs/reference/type-aliases/LiveQueryCollectionUtils.md b/docs/reference/type-aliases/LiveQueryCollectionUtils.md index ecb1be4892..1dfeb49d2a 100644 --- a/docs/reference/type-aliases/LiveQueryCollectionUtils.md +++ b/docs/reference/type-aliases/LiveQueryCollectionUtils.md @@ -19,16 +19,6 @@ Defined in: [packages/db/src/query/live/collection-config-builder.ts:54](https:/ [LIVE_QUERY_INTERNAL]: LiveQueryInternalUtils; ``` -### getRunCount() - -```ts -getRunCount: () => number; -``` - -#### Returns - -`number` - ### getWindow() ```ts @@ -52,6 +42,14 @@ Gets the current window (offset and limit) for an ordered query. The current window settings, or `undefined` if the query is not windowed +### lastSubsetError + +```ts +readonly lastSubsetError: unknown | undefined; +``` + +Most recent subset-load failure observed by this live query. + ### setWindow() ```ts diff --git a/docs/reference/type-aliases/LiveQueryKey.md b/docs/reference/type-aliases/LiveQueryKey.md new file mode 100644 index 0000000000..d45f6deb57 --- /dev/null +++ b/docs/reference/type-aliases/LiveQueryKey.md @@ -0,0 +1,12 @@ +--- +id: LiveQueryKey +title: LiveQueryKey +--- + +# Type Alias: LiveQueryKey + +```ts +type LiveQueryKey = ReadonlyArray; +``` + +Defined in: [packages/db/src/live-query-options.ts:17](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-options.ts#L17) diff --git a/docs/reference/type-aliases/LiveQueryObserverListener.md b/docs/reference/type-aliases/LiveQueryObserverListener.md new file mode 100644 index 0000000000..e903ab90e8 --- /dev/null +++ b/docs/reference/type-aliases/LiveQueryObserverListener.md @@ -0,0 +1,35 @@ +--- +id: LiveQueryObserverListener +title: LiveQueryObserverListener +--- + +# Type Alias: LiveQueryObserverListener()\ + +```ts +type LiveQueryObserverListener = (changes) => void; +``` + +Defined in: [packages/db/src/live-query-observer.ts:53](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-observer.ts#L53) + +Listener payload: changes, `[]` for an internal layout-only publication, or +`undefined` for a synthetic status/ready notification. + +## Type Parameters + +### T + +`T` *extends* `object` + +### TKey + +`TKey` *extends* `string` \| `number` + +## Parameters + +### changes + +[`ChangeMessage`](../interfaces/ChangeMessage.md)\<`T`, `TKey`\>[] | `undefined` + +## Returns + +`void` diff --git a/docs/reference/type-aliases/LiveQueryOptions.md b/docs/reference/type-aliases/LiveQueryOptions.md new file mode 100644 index 0000000000..127ca56d19 --- /dev/null +++ b/docs/reference/type-aliases/LiveQueryOptions.md @@ -0,0 +1,20 @@ +--- +id: LiveQueryOptions +title: LiveQueryOptions +--- + +# Type Alias: LiveQueryOptions + +```ts +type LiveQueryOptions = LiveQueryCollectionConfig & object; +``` + +Defined in: [packages/db/src/live-query-options.ts:19](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-options.ts#L19) + +## Type Declaration + +### queryKey? + +```ts +optional queryKey: LiveQueryKey; +``` diff --git a/docs/reference/type-aliases/LiveQueryWindowCollection.md b/docs/reference/type-aliases/LiveQueryWindowCollection.md new file mode 100644 index 0000000000..a63c1e833e --- /dev/null +++ b/docs/reference/type-aliases/LiveQueryWindowCollection.md @@ -0,0 +1,50 @@ +--- +id: LiveQueryWindowCollection +title: LiveQueryWindowCollection +--- + +# Type Alias: LiveQueryWindowCollection + +```ts +type LiveQueryWindowCollection = Collection & object; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:109](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L109) + +**`Internal`** + +Shared adapter view of a collection with an ordered window. + +## Type Declaration + +### utils + +```ts +utils: object; +``` + +#### utils.getWindow() + +```ts +getWindow: () => LiveQueryWindow | undefined; +``` + +##### Returns + +`LiveQueryWindow` \| `undefined` + +#### utils.setWindow() + +```ts +setWindow: (options) => WindowResult; +``` + +##### Parameters + +###### options + +`LiveQueryWindow` + +##### Returns + +`WindowResult` diff --git a/docs/reference/type-aliases/LiveQueryWindowInputKind.md b/docs/reference/type-aliases/LiveQueryWindowInputKind.md new file mode 100644 index 0000000000..42162e095c --- /dev/null +++ b/docs/reference/type-aliases/LiveQueryWindowInputKind.md @@ -0,0 +1,12 @@ +--- +id: LiveQueryWindowInputKind +title: LiveQueryWindowInputKind +--- + +# Type Alias: LiveQueryWindowInputKind + +```ts +type LiveQueryWindowInputKind = "collection" | "query"; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:27](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L27) diff --git a/docs/reference/type-aliases/LoadSubsetFn.md b/docs/reference/type-aliases/LoadSubsetFn.md index c7825337d7..53fb84ab69 100644 --- a/docs/reference/type-aliases/LoadSubsetFn.md +++ b/docs/reference/type-aliases/LoadSubsetFn.md @@ -9,7 +9,14 @@ title: LoadSubsetFn type LoadSubsetFn = (options) => true | Promise; ``` -Defined in: [packages/db/src/types.ts:316](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L316) +Defined in: [packages/db/src/types.ts:360](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L360) + +Loads one subset and transfers its ongoing resource ownership only after +returning `true` or a promise. An implementation that throws synchronously +must release any partially acquired resource before throwing. A successful +implementation must await or return every applied receipt from the sync +`commit()` calls that establish the loaded subset. A result describes only +the exact `options` passed to this call. ## Parameters diff --git a/docs/reference/type-aliases/LoadSubsetOptions.md b/docs/reference/type-aliases/LoadSubsetOptions.md index f1b155adda..ddee1c41d1 100644 --- a/docs/reference/type-aliases/LoadSubsetOptions.md +++ b/docs/reference/type-aliases/LoadSubsetOptions.md @@ -9,7 +9,15 @@ title: LoadSubsetOptions type LoadSubsetOptions = object; ``` -Defined in: [packages/db/src/types.ts:287](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L287) +Defined in: [packages/db/src/types.ts:312](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L312) + +Immutable request data. From submission onward, callers and adapters must +not mutate these options, their expression trees, comparison options, or +constant payloads (including Dates, byte arrays, and membership arrays). +Create new request data to change a demand; core does not clone or freeze it. +Use stable data properties, not stateful getters, for request data. +Signal and subscription references stay fixed, but their lifecycle remains +live: aborting the signal or releasing the subscription is supported. ## Properties @@ -19,7 +27,7 @@ Defined in: [packages/db/src/types.ts:287](https://github.com/TanStack/db/blob/m optional cursor: CursorExpressions; ``` -Defined in: [packages/db/src/types.ts:299](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L299) +Defined in: [packages/db/src/types.ts:324](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L324) Cursor expressions for cursor-based pagination. These are separate from `where` - the sync layer should combine them if using cursor-based pagination. @@ -33,7 +41,7 @@ Neither expression includes the main `where` clause. optional limit: number; ``` -Defined in: [packages/db/src/types.ts:293](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L293) +Defined in: [packages/db/src/types.ts:318](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L318) The limit of the data to load @@ -45,7 +53,7 @@ The limit of the data to load optional offset: number; ``` -Defined in: [packages/db/src/types.ts:304](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L304) +Defined in: [packages/db/src/types.ts:329](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L329) Row offset for offset-based pagination. The sync layer can use this instead of `cursor` if it prefers offset-based pagination. @@ -58,19 +66,35 @@ The sync layer can use this instead of `cursor` if it prefers offset-based pagin optional orderBy: OrderBy; ``` -Defined in: [packages/db/src/types.ts:291](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L291) +Defined in: [packages/db/src/types.ts:316](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L316) The order by clause to sort the data *** +### signal? + +```ts +optional signal: AbortSignal; +``` + +Defined in: [packages/db/src/types.ts:337](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L337) + +Aborted when this exact subset request is no longer current. Cancellation +is cooperative: async adapters should stop before installing more +request-scoped rows. If an in-flight baseline cannot be canceled, the +returned load promise must settle after those writes become visible so +core can keep overlapping replay private until then. + +*** + ### subscription? ```ts optional subscription: Subscription; ``` -Defined in: [packages/db/src/types.ts:313](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L313) +Defined in: [packages/db/src/types.ts:346](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L346) The subscription that triggered the load. Advanced sync implementations can use this for: @@ -90,6 +114,6 @@ Available when called from CollectionSubscription, may be undefined for direct c optional where: BasicExpression; ``` -Defined in: [packages/db/src/types.ts:289](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L289) +Defined in: [packages/db/src/types.ts:314](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L314) The where expression to filter the data (does NOT include cursor expressions) diff --git a/docs/reference/type-aliases/LoadSubsetRequestResult.md b/docs/reference/type-aliases/LoadSubsetRequestResult.md new file mode 100644 index 0000000000..07a9b3c17a --- /dev/null +++ b/docs/reference/type-aliases/LoadSubsetRequestResult.md @@ -0,0 +1,16 @@ +--- +id: LoadSubsetRequestResult +title: LoadSubsetRequestResult +--- + +# Type Alias: LoadSubsetRequestResult + +```ts +type LoadSubsetRequestResult = true | Promise; +``` + +Defined in: [packages/db/src/types.ts:350](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L350) + +**`Internal`** + +Result returned by the collection's normalized subset boundary. diff --git a/docs/reference/type-aliases/MakeOptional.md b/docs/reference/type-aliases/MakeOptional.md index 122f6f993d..6d3f7d9916 100644 --- a/docs/reference/type-aliases/MakeOptional.md +++ b/docs/reference/type-aliases/MakeOptional.md @@ -9,7 +9,7 @@ title: MakeOptional type MakeOptional = Omit & Partial>; ``` -Defined in: [packages/db/src/types.ts:998](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L998) +Defined in: [packages/db/src/types.ts:1097](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L1097) ## Type Parameters diff --git a/docs/reference/type-aliases/MaybeSingleResult.md b/docs/reference/type-aliases/MaybeSingleResult.md index 238da7bd4b..8187c7a33f 100644 --- a/docs/reference/type-aliases/MaybeSingleResult.md +++ b/docs/reference/type-aliases/MaybeSingleResult.md @@ -9,7 +9,7 @@ title: MaybeSingleResult type MaybeSingleResult = object; ``` -Defined in: [packages/db/src/types.ts:764](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L764) +Defined in: [packages/db/src/types.ts:856](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L856) ## Properties @@ -19,6 +19,6 @@ Defined in: [packages/db/src/types.ts:764](https://github.com/TanStack/db/blob/m optional singleResult: true; ``` -Defined in: [packages/db/src/types.ts:768](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L768) +Defined in: [packages/db/src/types.ts:860](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L860) If enabled the collection will return a single object instead of an array diff --git a/docs/reference/type-aliases/MergeContextForJoinCallback.md b/docs/reference/type-aliases/MergeContextForJoinCallback.md index b675c42d15..f3cb136704 100644 --- a/docs/reference/type-aliases/MergeContextForJoinCallback.md +++ b/docs/reference/type-aliases/MergeContextForJoinCallback.md @@ -6,10 +6,10 @@ title: MergeContextForJoinCallback # Type Alias: MergeContextForJoinCallback\ ```ts -type MergeContextForJoinCallback = object & PreserveHasResultFlag; +type MergeContextForJoinCallback = object & PreserveHasResultFlag & PreserveUnionFromFlag & PreserveFromSourceNames; ``` -Defined in: [packages/db/src/query/builder/types.ts:1005](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L1005) +Defined in: [packages/db/src/query/builder/types.ts:1267](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L1267) MergeContextForJoinCallback - Special context for join condition callbacks @@ -63,6 +63,12 @@ hasJoins: true; joinTypes: TContext["joinTypes"] extends Record ? TContext["joinTypes"] : object; ``` +### refsSchema + +```ts +refsSchema: RefsSchemaForContext & TNewSchema; +``` + ### result ```ts diff --git a/docs/reference/type-aliases/MergeContextWithJoinType.md b/docs/reference/type-aliases/MergeContextWithJoinType.md index 983e847180..4a38b7c078 100644 --- a/docs/reference/type-aliases/MergeContextWithJoinType.md +++ b/docs/reference/type-aliases/MergeContextWithJoinType.md @@ -6,10 +6,10 @@ title: MergeContextWithJoinType # Type Alias: MergeContextWithJoinType\ ```ts -type MergeContextWithJoinType = object & PreserveSingleResultFlag & PreserveHasResultFlag; +type MergeContextWithJoinType = object & PreserveSingleResultFlag & PreserveHasResultFlag & PreserveUnionFromFlag & PreserveFromSourceNames; ``` -Defined in: [packages/db/src/query/builder/types.ts:729](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L729) +Defined in: [packages/db/src/query/builder/types.ts:938](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L938) MergeContextWithJoinType - Creates a new context after a join operation @@ -59,6 +59,12 @@ hasJoins: true; joinTypes: TContext["joinTypes"] extends Record ? TContext["joinTypes"] : object & { [K in keyof TNewSchema & string]: TJoinType }; ``` +### refsSchema + +```ts +refsSchema: ApplyJoinOptionalityToMergedSchema, TNewSchema, TJoinType, FromSourceNamesForOptionality>; +``` + ### result ```ts @@ -68,7 +74,7 @@ result: TContext["result"]; ### schema ```ts -schema: ApplyJoinOptionalityToMergedSchema; +schema: ApplyJoinOptionalityToMergedSchema>; ``` ## Type Parameters diff --git a/docs/reference/type-aliases/MutationFn.md b/docs/reference/type-aliases/MutationFn.md index 647bb2cc48..6c129683d4 100644 --- a/docs/reference/type-aliases/MutationFn.md +++ b/docs/reference/type-aliases/MutationFn.md @@ -9,7 +9,12 @@ title: MutationFn type MutationFn = (params) => Promise; ``` -Defined in: [packages/db/src/types.ts:129](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L129) +Defined in: [packages/db/src/types.ts:135](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L135) + +Persists an optimistic transaction. Do not start or await collection or +live-query preloads here. Sync commits queue behind this function, so waiting +for preload work that needs one of those commits can deadlock the mutation. +Use the collection adapter's mutation acknowledgement helper instead. ## Type Parameters diff --git a/docs/reference/type-aliases/NamespacedAndKeyedStream.md b/docs/reference/type-aliases/NamespacedAndKeyedStream.md index c7d5479c95..280bc9923d 100644 --- a/docs/reference/type-aliases/NamespacedAndKeyedStream.md +++ b/docs/reference/type-aliases/NamespacedAndKeyedStream.md @@ -9,7 +9,7 @@ title: NamespacedAndKeyedStream type NamespacedAndKeyedStream = IStreamBuilder; ``` -Defined in: [packages/db/src/types.ts:817](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L817) +Defined in: [packages/db/src/types.ts:909](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L909) A namespaced and keyed stream is a stream of rows This is used throughout a query pipeline and as the output from a query without diff --git a/docs/reference/type-aliases/NamespacedRow.md b/docs/reference/type-aliases/NamespacedRow.md index 47cfc5e996..981e3a1d5c 100644 --- a/docs/reference/type-aliases/NamespacedRow.md +++ b/docs/reference/type-aliases/NamespacedRow.md @@ -9,6 +9,6 @@ title: NamespacedRow type NamespacedRow = Record>; ``` -Defined in: [packages/db/src/types.ts:804](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L804) +Defined in: [packages/db/src/types.ts:896](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L896) A namespaced row is a row withing a pipeline that had each table wrapped in its alias diff --git a/docs/reference/type-aliases/NonEmptyArray.md b/docs/reference/type-aliases/NonEmptyArray.md index 6b43145f5a..95138b1eb6 100644 --- a/docs/reference/type-aliases/NonEmptyArray.md +++ b/docs/reference/type-aliases/NonEmptyArray.md @@ -9,7 +9,7 @@ title: NonEmptyArray type NonEmptyArray = [T, ...T[]]; ``` -Defined in: [packages/db/src/types.ts:136](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L136) +Defined in: [packages/db/src/types.ts:142](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L142) Represents a non-empty array (at least one element) diff --git a/docs/reference/type-aliases/NonSingleResult.md b/docs/reference/type-aliases/NonSingleResult.md index b4ecf311f5..60fb551f3a 100644 --- a/docs/reference/type-aliases/NonSingleResult.md +++ b/docs/reference/type-aliases/NonSingleResult.md @@ -9,7 +9,7 @@ title: NonSingleResult type NonSingleResult = object; ``` -Defined in: [packages/db/src/types.ts:760](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L760) +Defined in: [packages/db/src/types.ts:852](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L852) ## Properties @@ -19,4 +19,4 @@ Defined in: [packages/db/src/types.ts:760](https://github.com/TanStack/db/blob/m optional singleResult: never; ``` -Defined in: [packages/db/src/types.ts:761](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L761) +Defined in: [packages/db/src/types.ts:853](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L853) diff --git a/docs/reference/type-aliases/OperationType.md b/docs/reference/type-aliases/OperationType.md index f6bdc31170..9b78ab07d4 100644 --- a/docs/reference/type-aliases/OperationType.md +++ b/docs/reference/type-aliases/OperationType.md @@ -9,4 +9,4 @@ title: OperationType type OperationType = "insert" | "update" | "delete"; ``` -Defined in: [packages/db/src/types.ts:205](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L205) +Defined in: [packages/db/src/types.ts:211](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L211) diff --git a/docs/reference/type-aliases/OperatorName.md b/docs/reference/type-aliases/OperatorName.md index 92c072d006..695ed994ab 100644 --- a/docs/reference/type-aliases/OperatorName.md +++ b/docs/reference/type-aliases/OperatorName.md @@ -9,4 +9,4 @@ title: OperatorName type OperatorName = typeof operators[number]; ``` -Defined in: [packages/db/src/query/builder/functions.ts:437](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L437) +Defined in: [packages/db/src/query/builder/functions.ts:718](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L718) diff --git a/docs/reference/type-aliases/OptimisticChangeMessage.md b/docs/reference/type-aliases/OptimisticChangeMessage.md index a08950003c..c0bcf4e640 100644 --- a/docs/reference/type-aliases/OptimisticChangeMessage.md +++ b/docs/reference/type-aliases/OptimisticChangeMessage.md @@ -11,7 +11,7 @@ type OptimisticChangeMessage = | DeleteKeyMessage & object; ``` -Defined in: [packages/db/src/types.ts:402](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L402) +Defined in: [packages/db/src/types.ts:493](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L493) ## Type Parameters diff --git a/docs/reference/type-aliases/OrderByCallback.md b/docs/reference/type-aliases/OrderByCallback.md index b58fde0d12..8f4059b59f 100644 --- a/docs/reference/type-aliases/OrderByCallback.md +++ b/docs/reference/type-aliases/OrderByCallback.md @@ -9,7 +9,7 @@ title: OrderByCallback type OrderByCallback = (refs) => any; ``` -Defined in: [packages/db/src/query/builder/types.ts:410](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L410) +Defined in: [packages/db/src/query/builder/types.ts:576](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L576) OrderByCallback - Type for orderBy clause callback functions diff --git a/docs/reference/type-aliases/Prettify.md b/docs/reference/type-aliases/Prettify.md index 673257fd9a..fefbda6623 100644 --- a/docs/reference/type-aliases/Prettify.md +++ b/docs/reference/type-aliases/Prettify.md @@ -9,7 +9,7 @@ title: Prettify type Prettify = { [K in keyof T]: T[K] } & object; ``` -Defined in: [packages/db/src/query/builder/types.ts:1044](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L1044) +Defined in: [packages/db/src/query/builder/types.ts:1309](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L1309) Prettify - Utility type for clean IDE display diff --git a/docs/reference/type-aliases/QueryBuilder.md b/docs/reference/type-aliases/QueryBuilder.md index 060999bb6c..8483e7bab8 100644 --- a/docs/reference/type-aliases/QueryBuilder.md +++ b/docs/reference/type-aliases/QueryBuilder.md @@ -6,10 +6,10 @@ title: QueryBuilder # Type Alias: QueryBuilder\ ```ts -type QueryBuilder = Omit, "from" | "_getQuery">; +type QueryBuilder = Omit, "from" | "unionAll" | "_getQuery">; ``` -Defined in: [packages/db/src/query/builder/index.ts:1225](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L1225) +Defined in: [packages/db/src/query/builder/index.ts:1633](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L1633) ## Type Parameters diff --git a/docs/reference/type-aliases/QueryIdentity.md b/docs/reference/type-aliases/QueryIdentity.md new file mode 100644 index 0000000000..117077201b --- /dev/null +++ b/docs/reference/type-aliases/QueryIdentity.md @@ -0,0 +1,22 @@ +--- +id: QueryIdentity +title: QueryIdentity +--- + +# Type Alias: QueryIdentity + +```ts +type QueryIdentity = string & object; +``` + +Defined in: [packages/db/src/query/ir-stable-identity.ts:45](https://github.com/TanStack/db/blob/main/packages/db/src/query/ir-stable-identity.ts#L45) + +Semantic identity for a query plan, independent of its runtime owners. + +## Type Declaration + +### \[queryIdentityBrand\] + +```ts +readonly [queryIdentityBrand]: true; +``` diff --git a/docs/reference/type-aliases/QueryResult.md b/docs/reference/type-aliases/QueryResult.md index 26cecaf634..0e007767da 100644 --- a/docs/reference/type-aliases/QueryResult.md +++ b/docs/reference/type-aliases/QueryResult.md @@ -9,7 +9,7 @@ title: QueryResult type QueryResult = GetResult>; ``` -Defined in: [packages/db/src/query/builder/index.ts:1243](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L1243) +Defined in: [packages/db/src/query/builder/index.ts:1651](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L1651) ## Type Parameters diff --git a/docs/reference/type-aliases/Ref.md b/docs/reference/type-aliases/Ref.md index fce38d02f4..345ef5ec71 100644 --- a/docs/reference/type-aliases/Ref.md +++ b/docs/reference/type-aliases/Ref.md @@ -6,10 +6,10 @@ title: Ref # Type Alias: Ref\ ```ts -type Ref = { [K in keyof T]: IsNonExactOptional extends true ? IsNonExactNullable extends true ? IsPlainObject> extends true ? Ref, Nullable> | undefined : RefLeaf, Nullable> | undefined : IsPlainObject> extends true ? Ref, Nullable> | undefined : RefLeaf, Nullable> | undefined : IsNonExactNullable extends true ? IsPlainObject> extends true ? Ref, Nullable> | null : RefLeaf, Nullable> | null : IsPlainObject extends true ? Ref : RefLeaf } & RefLeaf & VirtualPropsRef; +type Ref = T extends unknown ? RefBranch : never; ``` -Defined in: [packages/db/src/query/builder/types.ts:639](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L639) +Defined in: [packages/db/src/query/builder/types.ts:836](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L836) Ref - The user-facing ref interface for the query builder diff --git a/docs/reference/type-aliases/RefsForContext.md b/docs/reference/type-aliases/RefsForContext.md index 3476398e73..43f0b0cebe 100644 --- a/docs/reference/type-aliases/RefsForContext.md +++ b/docs/reference/type-aliases/RefsForContext.md @@ -6,25 +6,10 @@ title: RefsForContext # Type Alias: RefsForContext\ ```ts -type RefsForContext = { [K in keyof TContext["schema"]]: IsNonExactOptional extends true ? IsNonExactNullable extends true ? Ref, true> : Ref, true> : IsNonExactNullable extends true ? Ref, true> : Ref } & TContext["hasResult"] extends true ? object : object; +type RefsForContext = { [K in KeysOfUnion>]: IsNonExactOptional, K>> extends true ? IsNonExactNullable, K>> extends true ? RefForContextValue, K>>, true> : RefForContextValue, K>>, true> : IsNonExactNullable, K>> extends true ? RefForContextValue, K>>, true> : RefForContextValue, K>> } & TContext["hasResult"] extends true ? object : object; ``` -Defined in: [packages/db/src/query/builder/types.ts:502](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L502) - -RefsForContext - Creates ref proxies for all tables/collections in a query context - -This is the main entry point for creating ref objects in query builder callbacks. -For nullable join sides (left/right/full joins), it produces `Ref` instead -of `Ref | undefined`. This accurately reflects that the proxy object is always -present at build time (it's a truthy proxy that records property access paths), -while the `Nullable` flag ensures the result type correctly includes `| undefined`. - -Examples: -- Required field: `Ref` → user.name works, result is T -- Nullable join side: `Ref` → user.name works, result is T | undefined - -After `select()` is called, this type also includes `$selected` which provides access -to the SELECT result fields via `$selected.fieldName` syntax. +Defined in: [packages/db/src/query/builder/types.ts:686](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L686) ## Type Parameters diff --git a/docs/reference/type-aliases/ResolvedLiveQueryWindowInput.md b/docs/reference/type-aliases/ResolvedLiveQueryWindowInput.md new file mode 100644 index 0000000000..f01025d905 --- /dev/null +++ b/docs/reference/type-aliases/ResolvedLiveQueryWindowInput.md @@ -0,0 +1,30 @@ +--- +id: ResolvedLiveQueryWindowInput +title: ResolvedLiveQueryWindowInput +--- + +# Type Alias: ResolvedLiveQueryWindowInput\ + +```ts +type ResolvedLiveQueryWindowInput = + | { + collection: Collection; + kind: "collection"; +} + | { + kind: "query"; + query: QueryBuilder; +}; +``` + +Defined in: [packages/db/src/live-query-window-controller.ts:30](https://github.com/TanStack/db/blob/main/packages/db/src/live-query-window-controller.ts#L30) + +**`Internal`** + +The supported, enabled input forms for infinite-query adapters. + +## Type Parameters + +### TContext + +`TContext` *extends* [`Context`](../interfaces/Context.md) diff --git a/docs/reference/type-aliases/ResultStream.md b/docs/reference/type-aliases/ResultStream.md index dc50b1c698..dc3c780379 100644 --- a/docs/reference/type-aliases/ResultStream.md +++ b/docs/reference/type-aliases/ResultStream.md @@ -9,7 +9,7 @@ title: ResultStream type ResultStream = IStreamBuilder<[unknown, [any, string | undefined]]>; ``` -Defined in: [packages/db/src/types.ts:799](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L799) +Defined in: [packages/db/src/types.ts:891](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L891) Result stream type representing the output of compiled queries Always returns [key, [result, orderByIndex]] where orderByIndex is undefined for unordered queries diff --git a/docs/reference/type-aliases/ResultTypeFromSelect.md b/docs/reference/type-aliases/ResultTypeFromSelect.md index dd7e568aad..5777d9dc94 100644 --- a/docs/reference/type-aliases/ResultTypeFromSelect.md +++ b/docs/reference/type-aliases/ResultTypeFromSelect.md @@ -6,10 +6,10 @@ title: ResultTypeFromSelect # Type Alias: ResultTypeFromSelect\ ```ts -type ResultTypeFromSelect = IsAny extends true ? any : WithoutRefBrand extends true ? ExtractExpressionType : TSelectObject[K] extends ToArrayWrapper ? T[] : TSelectObject[K] extends ConcatToArrayWrapper ? string : TSelectObject[K] extends QueryBuilder ? Collection> : TSelectObject[K] extends Ref ? ExtractRef<(...)[(...)]> : (...)[(...)] extends RefLeaf<(...)> ? (...) extends (...) ? (...) : (...) : (...) extends (...) ? (...) : (...) }>>; +type ResultTypeFromSelect = IsAny extends true ? any : WithoutRefBrand extends true ? ExtractExpressionType : TSelectObject[K] extends ToArrayWrapper ? T[] : TSelectObject[K] extends ConcatToArrayWrapper ? string : TSelectObject[K] extends MaterializeWrapper ? IsSingle extends true ? T | undefined : T[] : TSelectObject[K] extends { __brand: "CaseWhenWrapper"; _result?: infer T } ? ResultTypeFromCaseWhen : (...)[(...)] extends QueryBuilder<(...)> ? Collection<(...)> : (...) extends (...) ? (...) : (...) }>>; ``` -Defined in: [packages/db/src/query/builder/types.ts:309](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L309) +Defined in: [packages/db/src/query/builder/types.ts:403](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L403) ResultTypeFromSelect - Infers the result type from a select object diff --git a/docs/reference/type-aliases/Row.md b/docs/reference/type-aliases/Row.md index c2039a5edb..b3a6be1d4d 100644 --- a/docs/reference/type-aliases/Row.md +++ b/docs/reference/type-aliases/Row.md @@ -9,7 +9,7 @@ title: Row type Row = Record>; ``` -Defined in: [packages/db/src/types.ts:203](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L203) +Defined in: [packages/db/src/types.ts:209](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L209) ## Type Parameters diff --git a/docs/reference/type-aliases/SchemaFromSource.md b/docs/reference/type-aliases/SchemaFromSource.md index 35569f09df..94cebd9d82 100644 --- a/docs/reference/type-aliases/SchemaFromSource.md +++ b/docs/reference/type-aliases/SchemaFromSource.md @@ -6,10 +6,10 @@ title: SchemaFromSource # Type Alias: SchemaFromSource\ ```ts -type SchemaFromSource = Prettify<{ [K in keyof T]: T[K] extends CollectionImpl ? InferCollectionType : T[K] extends QueryBuilder ? GetResult : never }>; +type SchemaFromSource = Prettify<{ [K in keyof T]: T[K] extends CollectionImpl ? InferCollectionType : T[K] extends CollectionOptionsIdentity ? InferCollectionType : T[K] extends QueryBuilder ? GetRawResult : never }>; ``` -Defined in: [packages/db/src/query/builder/types.ts:104](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L104) +Defined in: [packages/db/src/query/builder/types.ts:128](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L128) SchemaFromSource - Converts a Source definition into a ContextSchema diff --git a/docs/reference/type-aliases/SelectObject.md b/docs/reference/type-aliases/SelectObject.md index 61ebb31249..db918df1e6 100644 --- a/docs/reference/type-aliases/SelectObject.md +++ b/docs/reference/type-aliases/SelectObject.md @@ -9,7 +9,7 @@ title: SelectObject type SelectObject = T; ``` -Defined in: [packages/db/src/query/builder/types.ts:208](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L208) +Defined in: [packages/db/src/query/builder/types.ts:293](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L293) SelectObject - Wrapper type for select clause objects diff --git a/docs/reference/type-aliases/SingleResult.md b/docs/reference/type-aliases/SingleResult.md index c12442b35d..514ed281fb 100644 --- a/docs/reference/type-aliases/SingleResult.md +++ b/docs/reference/type-aliases/SingleResult.md @@ -9,7 +9,7 @@ title: SingleResult type SingleResult = object; ``` -Defined in: [packages/db/src/types.ts:756](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L756) +Defined in: [packages/db/src/types.ts:848](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L848) ## Properties @@ -19,4 +19,4 @@ Defined in: [packages/db/src/types.ts:756](https://github.com/TanStack/db/blob/m singleResult: true; ``` -Defined in: [packages/db/src/types.ts:757](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L757) +Defined in: [packages/db/src/types.ts:849](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L849) diff --git a/docs/reference/type-aliases/SingleSource.md b/docs/reference/type-aliases/SingleSource.md new file mode 100644 index 0000000000..af4021394f --- /dev/null +++ b/docs/reference/type-aliases/SingleSource.md @@ -0,0 +1,18 @@ +--- +id: SingleSource +title: SingleSource +--- + +# Type Alias: SingleSource\ + +```ts +type SingleSource = IsUnion extends true ? never : TSource; +``` + +Defined in: [packages/db/src/query/builder/types.ts:148](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L148) + +## Type Parameters + +### TSource + +`TSource` *extends* [`Source`](Source.md) diff --git a/docs/reference/type-aliases/Source.md b/docs/reference/type-aliases/Source.md index 9c797516a3..42970e6ea1 100644 --- a/docs/reference/type-aliases/Source.md +++ b/docs/reference/type-aliases/Source.md @@ -9,21 +9,22 @@ title: Source type Source = object; ``` -Defined in: [packages/db/src/query/builder/types.ts:79](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L79) +Defined in: [packages/db/src/query/builder/types.ts:92](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L92) -Source - Input definition for query builder `from()` clause +Source - Input definition for query builder `from()` and `unionAll()` clauses Maps table aliases to either: - `CollectionImpl`: A database collection/table - `QueryBuilder`: A subquery that can be used as a table -Example: `{ users: usersCollection, orders: ordersCollection }` +Example: `{ users: usersCollection }` ## Index Signature ```ts [alias: string]: + | QueryBuilder | CollectionImpl, any> -| QueryBuilder +| CollectionOptionsIdentity ``` diff --git a/docs/reference/type-aliases/SourceClauseContext.md b/docs/reference/type-aliases/SourceClauseContext.md new file mode 100644 index 0000000000..47e7ef34e1 --- /dev/null +++ b/docs/reference/type-aliases/SourceClauseContext.md @@ -0,0 +1,12 @@ +--- +id: SourceClauseContext +title: SourceClauseContext +--- + +# Type Alias: SourceClauseContext + +```ts +type SourceClauseContext = "from clause" | "unionAll clause" | "join clause"; +``` + +Defined in: [packages/db/src/errors.ts:400](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L400) diff --git a/docs/reference/type-aliases/StandardSchema.md b/docs/reference/type-aliases/StandardSchema.md index 552e457610..bd0a4610b0 100644 --- a/docs/reference/type-aliases/StandardSchema.md +++ b/docs/reference/type-aliases/StandardSchema.md @@ -9,7 +9,7 @@ title: StandardSchema type StandardSchema = StandardSchemaV1 & object; ``` -Defined in: [packages/db/src/types.ts:419](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L419) +Defined in: [packages/db/src/types.ts:510](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L510) The Standard Schema interface. This follows the standard-schema specification: https://github.com/standard-schema/standard-schema diff --git a/docs/reference/type-aliases/StandardSchemaAlias.md b/docs/reference/type-aliases/StandardSchemaAlias.md index b7efab5ee8..78442d4127 100644 --- a/docs/reference/type-aliases/StandardSchemaAlias.md +++ b/docs/reference/type-aliases/StandardSchemaAlias.md @@ -9,7 +9,7 @@ title: StandardSchemaAlias type StandardSchemaAlias = StandardSchema; ``` -Defined in: [packages/db/src/types.ts:431](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L431) +Defined in: [packages/db/src/types.ts:522](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L522) Type alias for StandardSchema diff --git a/docs/reference/type-aliases/StorageApi.md b/docs/reference/type-aliases/StorageApi.md index fde4dcad56..2a72dd281c 100644 --- a/docs/reference/type-aliases/StorageApi.md +++ b/docs/reference/type-aliases/StorageApi.md @@ -9,6 +9,6 @@ title: StorageApi type StorageApi = Pick; ``` -Defined in: [packages/db/src/local-storage.ts:23](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L23) +Defined in: [packages/db/src/local-storage.ts:25](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L25) Storage API interface - subset of DOM Storage that we need diff --git a/docs/reference/type-aliases/StorageEventApi.md b/docs/reference/type-aliases/StorageEventApi.md index 4e3d8f10ab..ef3d6a1051 100644 --- a/docs/reference/type-aliases/StorageEventApi.md +++ b/docs/reference/type-aliases/StorageEventApi.md @@ -9,7 +9,7 @@ title: StorageEventApi type StorageEventApi = object; ``` -Defined in: [packages/db/src/local-storage.ts:28](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L28) +Defined in: [packages/db/src/local-storage.ts:30](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L30) Storage event API - subset of Window for 'storage' events only @@ -21,7 +21,7 @@ Storage event API - subset of Window for 'storage' events only addEventListener: (type, listener) => void; ``` -Defined in: [packages/db/src/local-storage.ts:29](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L29) +Defined in: [packages/db/src/local-storage.ts:31](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L31) #### Parameters @@ -45,7 +45,7 @@ Defined in: [packages/db/src/local-storage.ts:29](https://github.com/TanStack/db removeEventListener: (type, listener) => void; ``` -Defined in: [packages/db/src/local-storage.ts:33](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L33) +Defined in: [packages/db/src/local-storage.ts:35](https://github.com/TanStack/db/blob/main/packages/db/src/local-storage.ts#L35) #### Parameters diff --git a/docs/reference/type-aliases/SubscriptionEvents.md b/docs/reference/type-aliases/SubscriptionEvents.md index 15b3c11814..b5723e749b 100644 --- a/docs/reference/type-aliases/SubscriptionEvents.md +++ b/docs/reference/type-aliases/SubscriptionEvents.md @@ -9,19 +9,29 @@ title: SubscriptionEvents type SubscriptionEvents = object; ``` -Defined in: [packages/db/src/types.ts:243](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L243) +Defined in: [packages/db/src/types.ts:257](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L257) All subscription events ## Properties +### loadSubset:error + +```ts +loadSubset:error: SubscriptionLoadSubsetErrorEvent; +``` + +Defined in: [packages/db/src/types.ts:261](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L261) + +*** + ### status:change ```ts status:change: SubscriptionStatusChangeEvent; ``` -Defined in: [packages/db/src/types.ts:244](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L244) +Defined in: [packages/db/src/types.ts:258](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L258) *** @@ -31,7 +41,7 @@ Defined in: [packages/db/src/types.ts:244](https://github.com/TanStack/db/blob/m status:loadingSubset: SubscriptionStatusEvent<"loadingSubset">; ``` -Defined in: [packages/db/src/types.ts:246](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L246) +Defined in: [packages/db/src/types.ts:260](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L260) *** @@ -41,7 +51,7 @@ Defined in: [packages/db/src/types.ts:246](https://github.com/TanStack/db/blob/m status:ready: SubscriptionStatusEvent<"ready">; ``` -Defined in: [packages/db/src/types.ts:245](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L245) +Defined in: [packages/db/src/types.ts:259](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L259) *** @@ -51,4 +61,4 @@ Defined in: [packages/db/src/types.ts:245](https://github.com/TanStack/db/blob/m unsubscribed: SubscriptionUnsubscribedEvent; ``` -Defined in: [packages/db/src/types.ts:247](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L247) +Defined in: [packages/db/src/types.ts:262](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L262) diff --git a/docs/reference/type-aliases/SubscriptionStatus.md b/docs/reference/type-aliases/SubscriptionStatus.md index 32218fdc9a..aeb093c67f 100644 --- a/docs/reference/type-aliases/SubscriptionStatus.md +++ b/docs/reference/type-aliases/SubscriptionStatus.md @@ -9,6 +9,6 @@ title: SubscriptionStatus type SubscriptionStatus = "ready" | "loadingSubset"; ``` -Defined in: [packages/db/src/types.ts:210](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L210) +Defined in: [packages/db/src/types.ts:216](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L216) Subscription status values diff --git a/docs/reference/type-aliases/SyncAppliedReceipt.md b/docs/reference/type-aliases/SyncAppliedReceipt.md new file mode 100644 index 0000000000..1fce9cce04 --- /dev/null +++ b/docs/reference/type-aliases/SyncAppliedReceipt.md @@ -0,0 +1,17 @@ +--- +id: SyncAppliedReceipt +title: SyncAppliedReceipt +--- + +# Type Alias: SyncAppliedReceipt + +```ts +type SyncAppliedReceipt = true | Promise; +``` + +Defined in: [packages/db/src/types.ts:368](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L368) + +Confirms whether a committed sync transaction is visible or is waiting for +its turn in the collection's causal queue. A pending receipt rejects with an +error named `AbortError` if cancellation wins before application. Once the +writes are visible, later cancellation has no effect. diff --git a/docs/reference/type-aliases/SyncConfigRes.md b/docs/reference/type-aliases/SyncConfigRes.md index f9e77f546a..5992ed6e70 100644 --- a/docs/reference/type-aliases/SyncConfigRes.md +++ b/docs/reference/type-aliases/SyncConfigRes.md @@ -9,7 +9,7 @@ title: SyncConfigRes type SyncConfigRes = object; ``` -Defined in: [packages/db/src/types.ts:322](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L322) +Defined in: [packages/db/src/types.ts:382](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L382) ## Properties @@ -19,7 +19,7 @@ Defined in: [packages/db/src/types.ts:322](https://github.com/TanStack/db/blob/m optional cleanup: CleanupFn; ``` -Defined in: [packages/db/src/types.ts:323](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L323) +Defined in: [packages/db/src/types.ts:383](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L383) *** @@ -29,7 +29,7 @@ Defined in: [packages/db/src/types.ts:323](https://github.com/TanStack/db/blob/m optional loadSubset: LoadSubsetFn; ``` -Defined in: [packages/db/src/types.ts:324](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L324) +Defined in: [packages/db/src/types.ts:384](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L384) *** @@ -39,4 +39,4 @@ Defined in: [packages/db/src/types.ts:324](https://github.com/TanStack/db/blob/m optional unloadSubset: UnloadSubsetFn; ``` -Defined in: [packages/db/src/types.ts:325](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L325) +Defined in: [packages/db/src/types.ts:385](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L385) diff --git a/docs/reference/type-aliases/SyncMode.md b/docs/reference/type-aliases/SyncMode.md index b1222c800c..aca546121f 100644 --- a/docs/reference/type-aliases/SyncMode.md +++ b/docs/reference/type-aliases/SyncMode.md @@ -9,4 +9,4 @@ title: SyncMode type SyncMode = "eager" | "on-demand"; ``` -Defined in: [packages/db/src/types.ts:519](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L519) +Defined in: [packages/db/src/types.ts:611](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L611) diff --git a/docs/reference/type-aliases/TransactionWithMutations.md b/docs/reference/type-aliases/TransactionWithMutations.md index cb00d35567..7ec410db3d 100644 --- a/docs/reference/type-aliases/TransactionWithMutations.md +++ b/docs/reference/type-aliases/TransactionWithMutations.md @@ -9,7 +9,7 @@ title: TransactionWithMutations type TransactionWithMutations = Omit, "mutations"> & object; ``` -Defined in: [packages/db/src/types.ts:142](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L142) +Defined in: [packages/db/src/types.ts:148](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L148) Utility type for a Transaction with at least one mutation This is used internally by the Transaction.commit method diff --git a/docs/reference/type-aliases/UnloadSubsetFn.md b/docs/reference/type-aliases/UnloadSubsetFn.md index bbb033adba..e2d5f0b3db 100644 --- a/docs/reference/type-aliases/UnloadSubsetFn.md +++ b/docs/reference/type-aliases/UnloadSubsetFn.md @@ -9,7 +9,14 @@ title: UnloadSubsetFn type UnloadSubsetFn = (options) => void; ``` -Defined in: [packages/db/src/types.ts:318](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L318) +Defined in: [packages/db/src/types.ts:378](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L378) + +Releases the exact acquisition created for `options`. + +Implementations must be idempotent and must not throw. An adapter owns any +remote unsubscribe retry needed to make release reliable. Core attempts +each acquisition's release once, reports failures, and continues retiring +other acquisitions. It does not retry a failed subset release. ## Parameters diff --git a/docs/reference/type-aliases/UpdateMutationFn.md b/docs/reference/type-aliases/UpdateMutationFn.md index 06d9837d02..cf50182597 100644 --- a/docs/reference/type-aliases/UpdateMutationFn.md +++ b/docs/reference/type-aliases/UpdateMutationFn.md @@ -9,7 +9,7 @@ title: UpdateMutationFn type UpdateMutationFn = (params) => Promise; ``` -Defined in: [packages/db/src/types.ts:478](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L478) +Defined in: [packages/db/src/types.ts:569](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L569) ## Type Parameters diff --git a/docs/reference/type-aliases/UpdateMutationFnParams.md b/docs/reference/type-aliases/UpdateMutationFnParams.md index ea80ff58dc..e3ecc707a6 100644 --- a/docs/reference/type-aliases/UpdateMutationFnParams.md +++ b/docs/reference/type-aliases/UpdateMutationFnParams.md @@ -9,7 +9,7 @@ title: UpdateMutationFnParams type UpdateMutationFnParams = object; ``` -Defined in: [packages/db/src/types.ts:445](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L445) +Defined in: [packages/db/src/types.ts:536](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L536) ## Type Parameters @@ -33,7 +33,7 @@ Defined in: [packages/db/src/types.ts:445](https://github.com/TanStack/db/blob/m collection: Collection; ``` -Defined in: [packages/db/src/types.ts:451](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L451) +Defined in: [packages/db/src/types.ts:542](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L542) *** @@ -43,4 +43,4 @@ Defined in: [packages/db/src/types.ts:451](https://github.com/TanStack/db/blob/m transaction: TransactionWithMutations; ``` -Defined in: [packages/db/src/types.ts:450](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L450) +Defined in: [packages/db/src/types.ts:541](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L541) diff --git a/docs/reference/type-aliases/WhereCallback.md b/docs/reference/type-aliases/WhereCallback.md index 2f556c7215..f2a5adfcc0 100644 --- a/docs/reference/type-aliases/WhereCallback.md +++ b/docs/reference/type-aliases/WhereCallback.md @@ -9,7 +9,7 @@ title: WhereCallback type WhereCallback = (refs) => any; ``` -Defined in: [packages/db/src/query/builder/types.ts:129](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L129) +Defined in: [packages/db/src/query/builder/types.ts:212](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L212) WhereCallback - Type for where/having clause callback functions diff --git a/docs/reference/type-aliases/WithResult.md b/docs/reference/type-aliases/WithResult.md index 862a84cf22..6343821f03 100644 --- a/docs/reference/type-aliases/WithResult.md +++ b/docs/reference/type-aliases/WithResult.md @@ -9,7 +9,7 @@ title: WithResult type WithResult = Prettify & object>; ``` -Defined in: [packages/db/src/query/builder/types.ts:1034](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L1034) +Defined in: [packages/db/src/query/builder/types.ts:1299](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/types.ts#L1299) WithResult - Updates a context with a new result type after select() diff --git a/docs/reference/type-aliases/WithVirtualProps.md b/docs/reference/type-aliases/WithVirtualProps.md index 7a8b21aeaf..d5cec665fe 100644 --- a/docs/reference/type-aliases/WithVirtualProps.md +++ b/docs/reference/type-aliases/WithVirtualProps.md @@ -9,7 +9,7 @@ title: WithVirtualProps type WithVirtualProps = T & VirtualRowProps; ``` -Defined in: [packages/db/src/virtual-props.ts:112](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L112) +Defined in: [packages/db/src/virtual-props.ts:117](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L117) Adds virtual properties to a row type. diff --git a/docs/reference/type-aliases/WithoutVirtualProps.md b/docs/reference/type-aliases/WithoutVirtualProps.md index f7c3d864e2..e314776ce2 100644 --- a/docs/reference/type-aliases/WithoutVirtualProps.md +++ b/docs/reference/type-aliases/WithoutVirtualProps.md @@ -9,7 +9,7 @@ title: WithoutVirtualProps type WithoutVirtualProps = Omit; ``` -Defined in: [packages/db/src/virtual-props.ts:130](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L130) +Defined in: [packages/db/src/virtual-props.ts:135](https://github.com/TanStack/db/blob/main/packages/db/src/virtual-props.ts#L135) Extracts the base type from a type that may have virtual properties. Useful when you need to work with the raw data without virtual properties. diff --git a/docs/reference/type-aliases/WritableDeep.md b/docs/reference/type-aliases/WritableDeep.md index 41c352f9db..7201548f36 100644 --- a/docs/reference/type-aliases/WritableDeep.md +++ b/docs/reference/type-aliases/WritableDeep.md @@ -9,7 +9,7 @@ title: WritableDeep type WritableDeep = T extends BuiltIns ? T : T extends (...arguments_) => unknown ? object extends WritableObjectDeep ? T : HasMultipleCallSignatures extends true ? T : (...arguments_) => ReturnType & WritableObjectDeep : T extends ReadonlyMap ? WritableMapDeep : T extends ReadonlySet ? WritableSetDeep : T extends ReadonlyArray ? WritableArrayDeep : T extends object ? WritableObjectDeep : unknown; ``` -Defined in: [packages/db/src/types.ts:979](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L979) +Defined in: [packages/db/src/types.ts:1078](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L1078) ## Type Parameters diff --git a/docs/reference/variables/Query.md b/docs/reference/variables/Query.md index 30e7d7ba37..15b042ab65 100644 --- a/docs/reference/variables/Query.md +++ b/docs/reference/variables/Query.md @@ -9,4 +9,4 @@ title: Query const Query: InitialQueryBuilderConstructor = BaseQueryBuilder; ``` -Defined in: [packages/db/src/query/builder/index.ts:1232](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L1232) +Defined in: [packages/db/src/query/builder/index.ts:1640](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/index.ts#L1640) diff --git a/docs/reference/variables/operators.md b/docs/reference/variables/operators.md index 84e9a00189..8338611985 100644 --- a/docs/reference/variables/operators.md +++ b/docs/reference/variables/operators.md @@ -6,9 +6,9 @@ title: operators # Variable: operators ```ts -const operators: readonly ["eq", "gt", "gte", "lt", "lte", "in", "like", "ilike", "and", "or", "not", "isNull", "isUndefined", "upper", "lower", "length", "concat", "add", "coalesce", "caseWhen", "count", "avg", "sum", "min", "max"]; +const operators: readonly ["eq", "gt", "gte", "lt", "lte", "in", "like", "ilike", "and", "or", "not", "isNull", "isUndefined", "upper", "lower", "length", "concat", "add", "subtract", "multiply", "divide", "coalesce", "caseWhen", "count", "avg", "sum", "min", "max"]; ``` -Defined in: [packages/db/src/query/builder/functions.ts](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts) +Defined in: [packages/db/src/query/builder/functions.ts:680](https://github.com/TanStack/db/blob/main/packages/db/src/query/builder/functions.ts#L680) All supported operator names in TanStack DB expressions diff --git a/examples/angular/todos/CHANGELOG.md b/examples/angular/todos/CHANGELOG.md index 95bcec8e9b..57d7c75652 100644 --- a/examples/angular/todos/CHANGELOG.md +++ b/examples/angular/todos/CHANGELOG.md @@ -1,5 +1,29 @@ # todos +## 0.0.21 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + - @tanstack/angular-db@0.1.89 + +## 0.0.20 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + - @tanstack/angular-db@0.1.81 + +## 0.0.19 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + - @tanstack/angular-db@0.1.78 + ## 0.0.18 ### Patch Changes diff --git a/examples/angular/todos/package.json b/examples/angular/todos/package.json index fd4d3ac679..a08b9aee4e 100644 --- a/examples/angular/todos/package.json +++ b/examples/angular/todos/package.json @@ -1,6 +1,6 @@ { "name": "todos", - "version": "0.0.18", + "version": "0.0.21", "scripts": { "ng": "ng", "start": "ng serve", @@ -28,8 +28,8 @@ "@angular/forms": "^20.3.16", "@angular/platform-browser": "^20.3.16", "@angular/router": "^20.3.16", - "@tanstack/angular-db": "^0.1.68", - "@tanstack/db": "^0.6.8", + "@tanstack/angular-db": "^0.1.89", + "@tanstack/db": "^0.9.0", "rxjs": "^7.8.2", "tslib": "^2.8.1", "zone.js": "~0.15.0" diff --git a/examples/electron/offline-first/CHANGELOG.md b/examples/electron/offline-first/CHANGELOG.md index 48d92620b1..9b2e957225 100644 --- a/examples/electron/offline-first/CHANGELOG.md +++ b/examples/electron/offline-first/CHANGELOG.md @@ -1,5 +1,27 @@ # offline-first-electron +## 1.0.4 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/react-db@0.3.0 + - @tanstack/query-db-collection@1.2.5 + - @tanstack/offline-transactions@1.0.46 + - @tanstack/electron-db-sqlite-persistence@0.1.25 + - @tanstack/node-db-sqlite-persistence@0.2.13 + +## 1.0.3 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/react-db@0.2.0 + - @tanstack/offline-transactions@1.0.44 + - @tanstack/query-db-collection@1.2.3 + - @tanstack/electron-db-sqlite-persistence@0.1.23 + - @tanstack/node-db-sqlite-persistence@0.2.11 + ## 1.0.2 ### Patch Changes diff --git a/examples/electron/offline-first/package.json b/examples/electron/offline-first/package.json index b15ad88932..5b54c7975c 100644 --- a/examples/electron/offline-first/package.json +++ b/examples/electron/offline-first/package.json @@ -1,6 +1,6 @@ { "name": "offline-first-electron", - "version": "1.0.2", + "version": "1.0.4", "private": true, "type": "module", "main": "electron/main.ts", @@ -13,11 +13,11 @@ "postinstall": "prebuild-install --runtime electron --target 40.2.1 --arch arm64 || echo 'prebuild-install failed, try: npx @electron/rebuild'" }, "dependencies": { - "@tanstack/electron-db-sqlite-persistence": "^0.1.12", - "@tanstack/node-db-sqlite-persistence": "^0.2.0", - "@tanstack/offline-transactions": "^1.0.33", - "@tanstack/query-db-collection": "^1.0.40", - "@tanstack/react-db": "^0.1.86", + "@tanstack/electron-db-sqlite-persistence": "^0.1.33", + "@tanstack/node-db-sqlite-persistence": "^0.2.21", + "@tanstack/offline-transactions": "^1.0.54", + "@tanstack/query-db-collection": "^1.2.13", + "@tanstack/react-db": "^0.3.8", "@tanstack/react-query": "^5.90.20", "better-sqlite3": "^12.6.2", "react": "^19.2.4", diff --git a/examples/react-native/offline-transactions/CHANGELOG.md b/examples/react-native/offline-transactions/CHANGELOG.md index e3669b49b6..3c1d4124a0 100644 --- a/examples/react-native/offline-transactions/CHANGELOG.md +++ b/examples/react-native/offline-transactions/CHANGELOG.md @@ -1,5 +1,49 @@ # offline-transactions-react-native +## 1.0.7 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + - @tanstack/query-db-collection@1.2.13 + - @tanstack/offline-transactions@1.0.54 + - @tanstack/react-db@0.3.8 + - @tanstack/react-native-db-sqlite-persistence@0.2.21 + +## 1.0.6 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + - @tanstack/react-db@0.3.0 + - @tanstack/query-db-collection@1.2.5 + - @tanstack/offline-transactions@1.0.46 + - @tanstack/react-native-db-sqlite-persistence@0.2.13 + +## 1.0.5 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/react-db@0.2.0 + - @tanstack/db@0.7.1 + - @tanstack/offline-transactions@1.0.44 + - @tanstack/query-db-collection@1.2.3 + - @tanstack/react-native-db-sqlite-persistence@0.2.11 + +## 1.0.4 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + - @tanstack/react-db@0.1.96 + - @tanstack/offline-transactions@1.0.43 + - @tanstack/query-db-collection@1.2.2 + - @tanstack/react-native-db-sqlite-persistence@0.2.10 + ## 1.0.3 ### Patch Changes diff --git a/examples/react-native/offline-transactions/package.json b/examples/react-native/offline-transactions/package.json index 6274e7353c..11de0ec5c9 100644 --- a/examples/react-native/offline-transactions/package.json +++ b/examples/react-native/offline-transactions/package.json @@ -1,6 +1,6 @@ { "name": "offline-transactions-react-native", - "version": "1.0.3", + "version": "1.0.7", "private": true, "main": "expo-router/entry", "scripts": { @@ -15,11 +15,11 @@ "@op-engineering/op-sqlite": "^15.2.5", "@react-native-async-storage/async-storage": "2.1.2", "@react-native-community/netinfo": "11.4.1", - "@tanstack/db": "^0.6.8", - "@tanstack/offline-transactions": "^1.0.33", - "@tanstack/query-db-collection": "^1.0.40", - "@tanstack/react-db": "^0.1.86", - "@tanstack/react-native-db-sqlite-persistence": "^0.2.0", + "@tanstack/db": "^0.9.0", + "@tanstack/offline-transactions": "^1.0.54", + "@tanstack/query-db-collection": "^1.2.13", + "@tanstack/react-db": "^0.3.8", + "@tanstack/react-native-db-sqlite-persistence": "^0.2.21", "@tanstack/react-query": "^5.90.20", "expo": "~53.0.26", "expo-constants": "~17.1.0", diff --git a/examples/react-native/offline-transactions/src/components/TodoList.tsx b/examples/react-native/offline-transactions/src/components/TodoList.tsx index f5aa666c2b..aaf3fe8d2c 100644 --- a/examples/react-native/offline-transactions/src/components/TodoList.tsx +++ b/examples/react-native/offline-transactions/src/components/TodoList.tsx @@ -30,9 +30,12 @@ export function TodoList({ collection, executor }: TodoListProps) { [executor, collection], ) - const { data: todoList = [], isLoading } = useLiveQuery((q) => - q.from({ todo: collection }).orderBy(({ todo }) => todo.createdAt, `desc`), - ) + const { data: todoList = [], isLoading } = useLiveQuery({ + query: (q) => + q + .from({ todo: collection }) + .orderBy(({ todo }) => todo.createdAt, `desc`), + }) // Monitor network status for UI display // (The executor's ReactNativeOnlineDetector handles sync retries internally) diff --git a/examples/react-native/shopping-list/CHANGELOG.md b/examples/react-native/shopping-list/CHANGELOG.md index 04428e0922..660a93ce4d 100644 --- a/examples/react-native/shopping-list/CHANGELOG.md +++ b/examples/react-native/shopping-list/CHANGELOG.md @@ -1,5 +1,49 @@ # shopping-list-react-native +## 1.0.7 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + - @tanstack/electric-db-collection@0.4.8 + - @tanstack/offline-transactions@1.0.54 + - @tanstack/react-db@0.3.8 + - @tanstack/react-native-db-sqlite-persistence@0.2.21 + +## 1.0.6 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + - @tanstack/react-db@0.3.0 + - @tanstack/electric-db-collection@0.4.0 + - @tanstack/offline-transactions@1.0.46 + - @tanstack/react-native-db-sqlite-persistence@0.2.13 + +## 1.0.5 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/react-db@0.2.0 + - @tanstack/db@0.7.1 + - @tanstack/electric-db-collection@0.3.17 + - @tanstack/offline-transactions@1.0.44 + - @tanstack/react-native-db-sqlite-persistence@0.2.11 + +## 1.0.4 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + - @tanstack/react-db@0.1.96 + - @tanstack/electric-db-collection@0.3.16 + - @tanstack/offline-transactions@1.0.43 + - @tanstack/react-native-db-sqlite-persistence@0.2.10 + ## 1.0.3 ### Patch Changes diff --git a/examples/react-native/shopping-list/app/list/[id].tsx b/examples/react-native/shopping-list/app/list/[id].tsx index 5e97589059..5d9d1f9c45 100644 --- a/examples/react-native/shopping-list/app/list/[id].tsx +++ b/examples/react-native/shopping-list/app/list/[id].tsx @@ -1,7 +1,6 @@ -import { useLocalSearchParams, Stack } from 'expo-router' +import { Stack, useLocalSearchParams } from 'expo-router' import { SafeAreaView } from 'react-native-safe-area-context' -import { useLiveQuery } from '@tanstack/react-db' -import { eq } from '@tanstack/react-db' +import { eq, useLiveQuery } from '@tanstack/react-db' import { listsCollection } from '../../src/db/collections' import { ListDetail } from '../../src/components/ListDetail' @@ -9,15 +8,14 @@ export default function ListScreen() { const { id } = useLocalSearchParams<{ id: string }>() as { id: string } // Get the list name for the header - const listResult = useLiveQuery((q) => - q - .from({ list: listsCollection }) - .where(({ list }) => eq(list.id, id)) - .select(({ list }) => ({ id: list.id, name: list.name })), - ) - const list = (listResult.data ?? [])[0] as - | { id: string; name: string } - | undefined + const listResult = useLiveQuery({ + query: (q) => + q + .from({ list: listsCollection }) + .where(({ list }) => eq(list.id, id)) + .select(({ list }) => ({ id: list.id, name: list.name })), + }) + const list = listResult.data[0] as { id: string; name: string } | undefined return ( <> diff --git a/examples/react-native/shopping-list/package.json b/examples/react-native/shopping-list/package.json index be14aed669..35a09d283c 100644 --- a/examples/react-native/shopping-list/package.json +++ b/examples/react-native/shopping-list/package.json @@ -1,6 +1,6 @@ { "name": "shopping-list-react-native", - "version": "1.0.3", + "version": "1.0.7", "private": true, "main": "expo-router/entry", "scripts": { @@ -18,11 +18,11 @@ "@op-engineering/op-sqlite": "^15.2.5", "@react-native-async-storage/async-storage": "2.1.2", "@react-native-community/netinfo": "11.4.1", - "@tanstack/db": "^0.6.8", - "@tanstack/electric-db-collection": "^0.3.6", - "@tanstack/offline-transactions": "^1.0.33", - "@tanstack/react-db": "^0.1.86", - "@tanstack/react-native-db-sqlite-persistence": "^0.2.0", + "@tanstack/db": "^0.9.0", + "@tanstack/electric-db-collection": "^0.4.8", + "@tanstack/offline-transactions": "^1.0.54", + "@tanstack/react-db": "^0.3.8", + "@tanstack/react-native-db-sqlite-persistence": "^0.2.21", "@tanstack/react-query": "^5.90.20", "expo": "~53.0.26", "expo-constants": "~17.1.0", diff --git a/examples/react-native/shopping-list/src/components/ListDetail.tsx b/examples/react-native/shopping-list/src/components/ListDetail.tsx index 6bcd31788b..2b3e300e3a 100644 --- a/examples/react-native/shopping-list/src/components/ListDetail.tsx +++ b/examples/react-native/shopping-list/src/components/ListDetail.tsx @@ -82,12 +82,13 @@ export function ListDetail({ listId }: ListDetailProps) { const { itemActions } = useShopping() // Get items for this list - const itemsResult = useLiveQuery((q) => - q - .from({ item: itemsCollection }) - .where(({ item }) => eq(item.listId, listId)) - .orderBy(({ item }) => item.createdAt, `asc`), - ) + const itemsResult = useLiveQuery({ + query: (q) => + q + .from({ item: itemsCollection }) + .where(({ item }) => eq(item.listId, listId)) + .orderBy(({ item }) => item.createdAt, `asc`), + }) const items = itemsResult.data as Array const handleAddItem = async () => { diff --git a/examples/react-native/shopping-list/src/components/ListsScreen.tsx b/examples/react-native/shopping-list/src/components/ListsScreen.tsx index 1a5b501c3d..d46b277644 100644 --- a/examples/react-native/shopping-list/src/components/ListsScreen.tsx +++ b/examples/react-native/shopping-list/src/components/ListsScreen.tsx @@ -107,37 +107,38 @@ export function ListsScreen() { // ★ Includes query with aggregate subqueries: each list gets child collections // with computed counts. ListCard subscribes to them via useLiveQuery. - const queryResult = useLiveQuery((q) => - q - .from({ list: listsCollection }) - .select(({ list }) => ({ - id: list.id, - name: list.name, - createdAt: list.createdAt, - $synced: list.$synced, - totalItems: q - .from({ item: itemsCollection }) - .where(({ item }) => eq(item.listId, list.id)) - .select(({ item }) => ({ n: count(item.id) })), - uncheckedPreview: q - .from({ item: itemsCollection }) - .where(({ item }) => eq(item.listId, list.id)) - .where(({ item }) => eq(item.checked, false)) - .select(({ item }) => ({ - id: item.id, - text: item.text, - createdAt: item.createdAt, - })) - .orderBy(({ item }) => item.createdAt, `asc`) - .limit(3), - checkedItems: q - .from({ item: itemsCollection }) - .where(({ item }) => eq(item.listId, list.id)) - .where(({ item }) => eq(item.checked, true)) - .select(({ item }) => ({ n: count(item.id) })), - })) - .orderBy(({ list }) => list.createdAt, `desc`), - ) + const queryResult = useLiveQuery({ + query: (q) => + q + .from({ list: listsCollection }) + .select(({ list }) => ({ + id: list.id, + name: list.name, + createdAt: list.createdAt, + $synced: list.$synced, + totalItems: q + .from({ item: itemsCollection }) + .where(({ item }) => eq(item.listId, list.id)) + .select(({ item }) => ({ n: count(item.id) })), + uncheckedPreview: q + .from({ item: itemsCollection }) + .where(({ item }) => eq(item.listId, list.id)) + .where(({ item }) => eq(item.checked, false)) + .select(({ item }) => ({ + id: item.id, + text: item.text, + createdAt: item.createdAt, + })) + .orderBy(({ item }) => item.createdAt, `asc`) + .limit(3), + checkedItems: q + .from({ item: itemsCollection }) + .where(({ item }) => eq(item.listId, list.id)) + .where(({ item }) => eq(item.checked, true)) + .select(({ item }) => ({ n: count(item.id) })), + })) + .orderBy(({ list }) => list.createdAt, `desc`), + }) const lists = queryResult.data as unknown as Array<{ id: string name: string diff --git a/examples/react/next-ssr-e2e/CHANGELOG.md b/examples/react/next-ssr-e2e/CHANGELOG.md new file mode 100644 index 0000000000..2799b35c19 --- /dev/null +++ b/examples/react/next-ssr-e2e/CHANGELOG.md @@ -0,0 +1,17 @@ +# @tanstack/db-example-react-next-ssr-e2e + +## 0.0.2 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + - @tanstack/react-db@0.3.8 + +## 0.0.1 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + - @tanstack/react-db@0.3.0 diff --git a/examples/react/next-ssr-e2e/app/db-hydration.tsx b/examples/react/next-ssr-e2e/app/db-hydration.tsx new file mode 100644 index 0000000000..2bbe8b42cf --- /dev/null +++ b/examples/react/next-ssr-e2e/app/db-hydration.tsx @@ -0,0 +1,23 @@ +'use client' + +import { useState } from 'react' +import { DbClient } from '@tanstack/db' +import { DbProvider, HydrationBoundary } from '@tanstack/react-db' +import type { DehydratedDbState } from '@tanstack/db' +import type { ReactNode } from 'react' + +export function DbHydration({ + state, + children, +}: { + state: DehydratedDbState + children: ReactNode +}) { + const [client] = useState(() => new DbClient({ runtime: `browser` })) + + return ( + + {children} + + ) +} diff --git a/examples/react/next-ssr-e2e/app/layout.tsx b/examples/react/next-ssr-e2e/app/layout.tsx new file mode 100644 index 0000000000..d7cbe9fc83 --- /dev/null +++ b/examples/react/next-ssr-e2e/app/layout.tsx @@ -0,0 +1,14 @@ +import type { Metadata } from 'next' +import type { ReactNode } from 'react' + +export const metadata: Metadata = { + title: `TanStack DB Next.js SSR E2E`, +} + +export default function RootLayout({ children }: { children: ReactNode }) { + return ( + + {children} + + ) +} diff --git a/examples/react/next-ssr-e2e/app/page.tsx b/examples/react/next-ssr-e2e/app/page.tsx new file mode 100644 index 0000000000..7f6f18bb3d --- /dev/null +++ b/examples/react/next-ssr-e2e/app/page.tsx @@ -0,0 +1,27 @@ +import { Suspense } from 'react' +import { DbClient } from '@tanstack/db' +import { DbHydration } from './db-hydration' +import { streamedTodoQuery } from './ssr-fixture' +import { StreamedTodos } from './streamed-todos' + +export const dynamic = `force-dynamic` + +export default function Page() { + const dbClient = new DbClient({ runtime: `server` }) + void dbClient.preloadLiveQuery(streamedTodoQuery) + const state = dbClient.dehydrate({ + shouldDehydrateCollection: () => false, + shouldDehydrateLiveQuery: () => true, + }) + + return ( +
                +

                TanStack DB Next.js SSR

                + + Loading todos

                }> + +                 +                 +
                + ) +} diff --git a/examples/react/next-ssr-e2e/app/ssr-fixture.ts b/examples/react/next-ssr-e2e/app/ssr-fixture.ts new file mode 100644 index 0000000000..c3001cc6f5 --- /dev/null +++ b/examples/react/next-ssr-e2e/app/ssr-fixture.ts @@ -0,0 +1,70 @@ +import { collectionOptions, eq } from '@tanstack/db' +import type { InitialQueryBuilder } from '@tanstack/db' + +export type StreamedTodo = { + id: string + text: string + status: `open` | `done` + sourcePayload: string +} + +const serverTodo: StreamedTodo = { + id: `next-server-1`, + text: `Streamed from Next.js`, + status: `open`, + sourcePayload: `NEXT_SOURCE_ONLY_DO_NOT_TRANSPORT`, +} + +const browserTodo: StreamedTodo = { + id: `next-browser-1`, + text: `Reconciled by Next.js browser sync`, + status: `open`, + sourcePayload: `NEXT_BROWSER_SOURCE_ONLY_DO_NOT_TRANSPORT`, +} + +export const streamedTodoCollection = collectionOptions( + `next-ssr-stream-todos`, + (client) => { + const runtime = client.requireDependency<`server` | `browser`>(`runtime`) + + return { + id: `next-ssr-stream-todos`, + getKey: (todo: StreamedTodo) => todo.id, + syncMode: `on-demand` as const, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: () => + new Promise((resolve) => { + setTimeout( + () => { + begin({ immediate: true }) + write({ + type: `insert`, + value: runtime === `server` ? serverTodo : browserTodo, + }) + commit() + resolve() + }, + runtime === `server` ? 1000 : 1500, + ) + }), + } + }, + }, + } + }, +) + +export const streamedTodoQuery = { + query: (q: InitialQueryBuilder) => + q + .from({ todo: streamedTodoCollection }) + .where(({ todo }) => eq(todo.status, `open`)) + .select(({ todo }) => ({ + id: todo.id, + text: todo.text, + status: todo.status, + })), +} diff --git a/examples/react/next-ssr-e2e/app/streamed-todos.tsx b/examples/react/next-ssr-e2e/app/streamed-todos.tsx new file mode 100644 index 0000000000..b893421cae --- /dev/null +++ b/examples/react/next-ssr-e2e/app/streamed-todos.tsx @@ -0,0 +1,18 @@ +'use client' + +import { useLiveSuspenseQuery } from '@tanstack/react-db' +import { streamedTodoQuery } from './ssr-fixture' + +export function StreamedTodos() { + const { data: todos } = useLiveSuspenseQuery(streamedTodoQuery) + + return ( +
                  + {todos.map((todo) => ( +
                • + {todo.text} +
                  • + ))} +
                + ) +} diff --git a/examples/react/next-ssr-e2e/e2e/ssr-db.spec.ts b/examples/react/next-ssr-e2e/e2e/ssr-db.spec.ts new file mode 100644 index 0000000000..02d046cc62 --- /dev/null +++ b/examples/react/next-ssr-e2e/e2e/ssr-db.spec.ts @@ -0,0 +1,35 @@ +import { expect, test } from '@playwright/test' + +test(`Next.js streams a DB result snapshot and hands off to browser sync`, async ({ + page, + request, +}) => { + const response = await request.get(`/`) + expect(response.ok()).toBe(true) + const html = await response.text() + expect(html).toContain(`Streamed from Next.js`) + expect(html).not.toContain(`NEXT_SOURCE_ONLY_DO_NOT_TRANSPORT`) + + const browserErrors: Array = [] + page.on(`console`, (message) => { + if (message.type() === `error`) browserErrors.push(message.text()) + }) + page.on(`pageerror`, (error) => { + browserErrors.push(error.message) + }) + + await page.goto(`/`, { waitUntil: `commit` }) + + await expect(page.getByTestId(`stream-fallback`)).toBeVisible() + await expect(page.getByTestId(`streamed-todo-next-server-1`)).toHaveText( + `Streamed from Next.js`, + ) + await expect(page.getByTestId(`stream-fallback`)).not.toBeVisible() + await expect( + page.getByTestId(`streamed-todo-next-server-1`), + ).not.toBeVisible() + await expect(page.getByTestId(`streamed-todo-next-browser-1`)).toHaveText( + `Reconciled by Next.js browser sync`, + ) + expect(browserErrors).toEqual([]) +}) diff --git a/examples/react/next-ssr-e2e/next.config.ts b/examples/react/next-ssr-e2e/next.config.ts new file mode 100644 index 0000000000..6491458f0a --- /dev/null +++ b/examples/react/next-ssr-e2e/next.config.ts @@ -0,0 +1,8 @@ +import type { NextConfig } from 'next' + +const nextConfig: NextConfig = { + reactStrictMode: true, + transpilePackages: [`@tanstack/db`, `@tanstack/react-db`], +} + +export default nextConfig diff --git a/examples/react/next-ssr-e2e/package.json b/examples/react/next-ssr-e2e/package.json new file mode 100644 index 0000000000..ca41543dbf --- /dev/null +++ b/examples/react/next-ssr-e2e/package.json @@ -0,0 +1,25 @@ +{ + "name": "@tanstack/db-example-react-next-ssr-e2e", + "private": true, + "version": "0.0.2", + "scripts": { + "build": "next build", + "dev": "next dev", + "start": "next start", + "test:e2e": "pnpm --filter @tanstack/db build && pnpm --filter @tanstack/react-db build && playwright test" + }, + "dependencies": { + "@tanstack/db": "^0.9.0", + "@tanstack/react-db": "^0.3.8", + "next": "^16.3.1", + "react": "^19.2.4", + "react-dom": "^19.2.4" + }, + "devDependencies": { + "@playwright/test": "^1.60.0", + "@types/node": "^25.2.2", + "@types/react": "^19.2.13", + "@types/react-dom": "^19.2.3", + "typescript": "^5.9.2" + } +} diff --git a/examples/react/next-ssr-e2e/playwright.config.ts b/examples/react/next-ssr-e2e/playwright.config.ts new file mode 100644 index 0000000000..7a6ae0db52 --- /dev/null +++ b/examples/react/next-ssr-e2e/playwright.config.ts @@ -0,0 +1,31 @@ +import { defineConfig, devices } from '@playwright/test' + +const baseURL = process.env.PLAYWRIGHT_BASE_URL ?? `http://127.0.0.1:4176` +const shouldStartWebServer = process.env.PLAYWRIGHT_BASE_URL === undefined + +export default defineConfig({ + testDir: `./e2e`, + timeout: 30000, + expect: { + timeout: 10000, + }, + fullyParallel: false, + use: { + baseURL, + trace: `on-first-retry`, + }, + webServer: shouldStartWebServer + ? { + command: `pnpm dev --hostname 127.0.0.1 --port 4176`, + reuseExistingServer: !process.env.CI, + timeout: 120000, + url: baseURL, + } + : undefined, + projects: [ + { + name: `chromium`, + use: { ...devices[`Desktop Chrome`] }, + }, + ], +}) diff --git a/examples/react/next-ssr-e2e/tsconfig.json b/examples/react/next-ssr-e2e/tsconfig.json new file mode 100644 index 0000000000..b134f6f799 --- /dev/null +++ b/examples/react/next-ssr-e2e/tsconfig.json @@ -0,0 +1,26 @@ +{ + "compilerOptions": { + "target": "ES2017", + "lib": ["dom", "dom.iterable", "esnext"], + "allowJs": false, + "skipLibCheck": true, + "strict": true, + "noEmit": true, + "esModuleInterop": true, + "module": "esnext", + "moduleResolution": "bundler", + "resolveJsonModule": true, + "isolatedModules": true, + "jsx": "react-jsx", + "incremental": true, + "plugins": [{ "name": "next" }] + }, + "include": [ + "next-env.d.ts", + ".next/types/**/*.ts", + ".next/dev/types/**/*.ts", + "**/*.ts", + "**/*.tsx" + ], + "exclude": ["node_modules"] +} diff --git a/examples/react/offline-transactions/package.json b/examples/react/offline-transactions/package.json index 05ce1caea1..66a133500d 100644 --- a/examples/react/offline-transactions/package.json +++ b/examples/react/offline-transactions/package.json @@ -8,11 +8,11 @@ "build": "vite build && tsc --noEmit" }, "dependencies": { - "@tanstack/browser-db-sqlite-persistence": "^0.2.0", - "@tanstack/db": "^0.6.8", - "@tanstack/offline-transactions": "^1.0.33", - "@tanstack/query-db-collection": "^1.0.40", - "@tanstack/react-db": "^0.1.86", + "@tanstack/browser-db-sqlite-persistence": "^0.2.21", + "@tanstack/db": "^0.9.0", + "@tanstack/offline-transactions": "^1.0.54", + "@tanstack/query-db-collection": "^1.2.13", + "@tanstack/react-db": "^0.3.8", "@tanstack/react-query": "^5.90.20", "@tanstack/react-router": "^1.159.5", "@tanstack/react-router-devtools": "^1.159.5", diff --git a/examples/react/offline-transactions/src/components/PersistedTodoDemo.tsx b/examples/react/offline-transactions/src/components/PersistedTodoDemo.tsx index e7252f5f4e..c7f25cd99c 100644 --- a/examples/react/offline-transactions/src/components/PersistedTodoDemo.tsx +++ b/examples/react/offline-transactions/src/components/PersistedTodoDemo.tsx @@ -11,9 +11,12 @@ export function PersistedTodoDemo({ collection }: PersistedTodoDemoProps) { const [newTodoText, setNewTodoText] = useState(``) const [error, setError] = useState(null) - const { data: todoList = [] } = useLiveQuery((q) => - q.from({ todo: collection }).orderBy(({ todo }) => todo.createdAt, `desc`), - ) + const { data: todoList = [] } = useLiveQuery({ + query: (q) => + q + .from({ todo: collection }) + .orderBy(({ todo }) => todo.createdAt, `desc`), + }) const handleAddTodo = () => { if (!newTodoText.trim()) return diff --git a/examples/react/offline-transactions/src/components/TodoDemo.tsx b/examples/react/offline-transactions/src/components/TodoDemo.tsx index fcdf088da1..4f95b79a92 100644 --- a/examples/react/offline-transactions/src/components/TodoDemo.tsx +++ b/examples/react/offline-transactions/src/components/TodoDemo.tsx @@ -25,11 +25,12 @@ export function TodoDemo({ console.log({ offline, actions }) // Use live query to get todos - const { data: todoList = [], isLoading } = useLiveQuery((q) => - q - .from({ todo: todoCollection }) - .orderBy(({ todo }) => todo.createdAt, `desc`), - ) + const { data: todoList = [], isLoading } = useLiveQuery({ + query: (q) => + q + .from({ todo: todoCollection }) + .orderBy(({ todo }) => todo.createdAt, `desc`), + }) // Monitor online status useEffect(() => { diff --git a/examples/react/paced-mutations-demo/CHANGELOG.md b/examples/react/paced-mutations-demo/CHANGELOG.md index bca3fcf4b4..5260d3a5a5 100644 --- a/examples/react/paced-mutations-demo/CHANGELOG.md +++ b/examples/react/paced-mutations-demo/CHANGELOG.md @@ -1,5 +1,37 @@ # @tanstack/db-example-paced-mutations-demo +## 0.0.12 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + - @tanstack/react-db@0.3.8 + +## 0.0.11 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + - @tanstack/react-db@0.3.0 + +## 0.0.10 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/react-db@0.2.0 + - @tanstack/db@0.7.1 + +## 0.0.9 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + - @tanstack/react-db@0.1.96 + ## 0.0.8 ### Patch Changes diff --git a/examples/react/paced-mutations-demo/package.json b/examples/react/paced-mutations-demo/package.json index 02a929269c..eb8f93ee53 100644 --- a/examples/react/paced-mutations-demo/package.json +++ b/examples/react/paced-mutations-demo/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/db-example-paced-mutations-demo", - "version": "0.0.8", + "version": "0.0.12", "private": true, "type": "module", "scripts": { @@ -9,8 +9,8 @@ "preview": "vite preview" }, "dependencies": { - "@tanstack/db": "^0.6.8", - "@tanstack/react-db": "^0.1.86", + "@tanstack/db": "^0.9.0", + "@tanstack/react-db": "^0.3.8", "mitt": "^3.0.1", "react": "^19.2.4", "react-dom": "^19.2.4" diff --git a/examples/react/projects/package.json b/examples/react/projects/package.json index 1f0178c52e..424e88605c 100644 --- a/examples/react/projects/package.json +++ b/examples/react/projects/package.json @@ -17,8 +17,8 @@ "dependencies": { "@tailwindcss/vite": "^4.1.18", "@tanstack/query-core": "^5.90.20", - "@tanstack/query-db-collection": "^1.0.40", - "@tanstack/react-db": "^0.1.86", + "@tanstack/query-db-collection": "^1.2.13", + "@tanstack/react-db": "^0.3.8", "@tanstack/react-router": "^1.159.5", "@tanstack/react-router-devtools": "^1.159.5", "@tanstack/react-router-with-query": "^1.130.17", diff --git a/examples/react/projects/src/routes/_authenticated.tsx b/examples/react/projects/src/routes/_authenticated.tsx index 17ed734276..43142ee5ca 100644 --- a/examples/react/projects/src/routes/_authenticated.tsx +++ b/examples/react/projects/src/routes/_authenticated.tsx @@ -20,7 +20,9 @@ function AuthenticatedLayout() { const [showNewProjectForm, setShowNewProjectForm] = useState(false) const [newProjectName, setNewProjectName] = useState(``) - const { data: projects } = useLiveQuery((q) => q.from({ projectCollection })) + const { data: projects } = useLiveQuery({ + query: (q) => q.from({ projectCollection }), + }) const handleLogout = async () => { await authClient.signOut() diff --git a/examples/react/projects/src/routes/_authenticated/project/$projectId.tsx b/examples/react/projects/src/routes/_authenticated/project/$projectId.tsx index 0ac0be409e..a60c4848d3 100644 --- a/examples/react/projects/src/routes/_authenticated/project/$projectId.tsx +++ b/examples/react/projects/src/routes/_authenticated/project/$projectId.tsx @@ -25,41 +25,40 @@ export const Route = createFileRoute(`/_authenticated/project/$projectId`)({ function ProjectPage() { const { projectId } = Route.useParams() + const projectIdNumber = parseInt(projectId, 10) const { data: session } = authClient.useSession() const [newTodoText, setNewTodoText] = useState(``) - const { data: todos } = useLiveQuery( - (q) => + const { data: todos } = useLiveQuery({ + query: (q) => q .from({ todo: todoCollection }) - .where(({ todo }) => eq(todo.project_id, parseInt(projectId, 10))) + .where(({ todo }) => eq(todo.project_id, projectIdNumber)) .orderBy(({ todo }) => todo.created_at), - [projectId] - ) - - const { data: users } = useLiveQuery((q) => - q.from({ users: usersCollection }) - ) - const { data: usersInProjects } = useLiveQuery( - (q) => + }) + + const { data: users } = useLiveQuery({ + query: (q) => q.from({ users: usersCollection }), + }) + const { data: usersInProjects } = useLiveQuery({ + queryKey: [projectCollection.id, `users-in-project`, projectIdNumber], + query: (q) => q .from({ projects: projectCollection }) - .where(({ projects }) => eq(projects.id, parseInt(projectId, 10))) + .where(({ projects }) => eq(projects.id, projectIdNumber)) .fn.select(({ projects }) => ({ users: projects.shared_user_ids.concat(projects.owner_id), owner: projects.owner_id, })), - [projectId] - ) + }) const usersInProject = usersInProjects[0] - const { data: projects } = useLiveQuery( - (q) => + const { data: projects } = useLiveQuery({ + query: (q) => q .from({ p: projectCollection }) - .where(({ p }) => eq(p.id, parseInt(projectId, 10))), - [projectId] - ) + .where(({ p }) => eq(p.id, projectIdNumber)), + }) const project = projects[0] const addTodo = () => { @@ -69,7 +68,7 @@ function ProjectPage() { id: Math.floor(Math.random() * 100000), text: newTodoText.trim(), completed: false, - project_id: parseInt(projectId), + project_id: projectIdNumber, user_ids: [], created_at: new Date(), }) diff --git a/examples/react/start-ssr-e2e/CHANGELOG.md b/examples/react/start-ssr-e2e/CHANGELOG.md new file mode 100644 index 0000000000..472e71614c --- /dev/null +++ b/examples/react/start-ssr-e2e/CHANGELOG.md @@ -0,0 +1,9 @@ +# @tanstack/db-example-react-start-ssr-e2e + +## 0.0.1 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/react-db@0.3.0 + - @tanstack/react-router-with-db@0.1.0 diff --git a/examples/react/start-ssr-e2e/README.md b/examples/react/start-ssr-e2e/README.md new file mode 100644 index 0000000000..3fbfd0d042 --- /dev/null +++ b/examples/react/start-ssr-e2e/README.md @@ -0,0 +1,55 @@ +# TanStack DB Start SSR Demo + +This example is a minimal TanStack Start app that demonstrates TanStack DB SSR +with collection-row hydration. + +It verifies five things: + +- server HTML contains rows loaded through a request-scoped `DbClient` +- the browser hydrates those rows into a client `DbClient` +- fresh adapter sync replaces a stale hydrated row with the same key +- an incremental collection chunk updates an existing live query +- critical collection rows hydrate while a query discovered later in the same + render streams through a Suspense boundary + +Live demo: https://tanstack-db-ssr-demo.netlify.app/ssr-db + +## Run Locally + +```sh +pnpm --filter @tanstack/db build +pnpm --filter @tanstack/react-db build +pnpm --filter @tanstack/db-example-react-start-ssr-e2e dev +``` + +Open `/ssr-db` for holistic and incremental hydration, or `/ssr-db-stream` for +render-time Suspense streaming. + +## Run E2E + +```sh +pnpm --filter @tanstack/db-example-react-start-ssr-e2e test:e2e +``` + +The Playwright tests cover raw SSR HTML, browser hydration, fresh-sync +reconciliation, incremental collection hydration, and critical hydration plus a +render-time query in the same request. The streaming route shows its Suspense +fallback before the streamed server rows arrive. + +## Deploy Demo + +The demo requires an SSR-capable host for TanStack Start. + +Netlify deployment is configured through `netlify.toml` and +`netlify/functions/server.mjs`. Deploy with: + +```sh +cd examples/react/start-ssr-e2e +netlify deploy --prod --site-name tanstack-db-ssr-demo --team tanstack +``` + +After deployment, verify the live URL with: + +```sh +PLAYWRIGHT_BASE_URL=https://your-demo-url pnpm --filter @tanstack/db-example-react-start-ssr-e2e test:e2e:hosted +``` diff --git a/examples/react/start-ssr-e2e/e2e/ssr-db.spec.ts b/examples/react/start-ssr-e2e/e2e/ssr-db.spec.ts new file mode 100644 index 0000000000..f9461d6fac --- /dev/null +++ b/examples/react/start-ssr-e2e/e2e/ssr-db.spec.ts @@ -0,0 +1,93 @@ +import { expect, test } from '@playwright/test' + +test(`TanStack Start hydrates, reconciles, and incrementally applies DB rows`, async ({ + page, + request, +}) => { + const response = await request.get(`/ssr-db`) + expect(response.ok()).toBe(true) + + const html = await response.text() + expect(html).toContain(`Pay invoices`) + expect(html).not.toContain(`Pay invoices (reconciled from sync)`) + expect(html).toContain(`Review pull requests`) + expect(html).toContain(`ssr`) + expect(html).not.toContain(`Streamed from collection chunk`) + + const browserErrors: Array = [] + page.on(`console`, (message) => { + if (message.type() === `error`) { + browserErrors.push(message.text()) + } + }) + page.on(`pageerror`, (error) => { + browserErrors.push(error.message) + }) + + await page.goto(`/ssr-db`) + + await expect(page.getByTestId(`hydration-state`)).toHaveText(`hydrated`) + await expect(page.getByTestId(`ready-state`)).toHaveText(`ready`) + await expect(page.getByTestId(`streamed-status`)).toHaveText(`waiting`) + await expect(page.getByTestId(`ssr-row-count`)).toHaveText(`2`) + await expect(page.getByTestId(`ssr-todo-list`)).toContainText( + `Pay invoices (reconciled from sync)`, + ) + await expect(page.getByTestId(`ssr-todo-server-1`)).toContainText(`(sync)`) + await expect(page.getByTestId(`ssr-todo-list`)).toContainText( + `Review pull requests`, + ) + await expect(page.getByTestId(`ssr-todo-list`)).not.toContainText( + `Archived roadmap`, + ) + + await page.getByTestId(`apply-stream-chunk`).click() + + await expect(page.getByTestId(`streamed-status`)).toHaveText(`streamed`) + await expect(page.getByTestId(`ssr-row-count`)).toHaveText(`3`) + await expect(page.getByTestId(`ssr-todo-streamed-1`)).toBeVisible() + await expect(page.getByTestId(`ssr-todo-streamed-1`)).toContainText( + `Streamed from collection chunk`, + ) + expect(browserErrors).toEqual([]) +}) + +test(`TanStack Start streams a DB result snapshot and hands off to browser sync`, async ({ + page, + request, +}) => { + const response = await request.get(`/ssr-db-stream`) + expect(response.ok()).toBe(true) + const html = await response.text() + expect(html).toContain(`Streamed while rendering`) + expect(html).not.toContain(`SOURCE_ONLY_DO_NOT_TRANSPORT`) + + const browserErrors: Array = [] + page.on(`console`, (message) => { + if (message.type() === `error`) { + browserErrors.push(message.text()) + } + }) + page.on(`pageerror`, (error) => { + browserErrors.push(error.message) + }) + + await page.goto(`/ssr-db-stream`, { waitUntil: `commit` }) + + await expect(page.getByTestId(`critical-todo-server-1`)).toContainText( + `Pay invoices`, + ) + await expect(page.getByTestId(`stream-fallback`)).toBeVisible() + await expect(page.getByTestId(`streamed-todo-list`)).not.toBeVisible() + await expect(page.getByTestId(`streamed-todo-streamed-server-1`)).toHaveText( + `Streamed while rendering`, + ) + await expect(page.getByTestId(`stream-fallback`)).not.toBeVisible() + await expect( + page.getByTestId(`streamed-todo-streamed-server-1`), + ).not.toBeVisible() + await expect(page.getByTestId(`streamed-todo-streamed-browser-1`)).toHaveText( + `Reconciled from browser sync`, + ) + expect(browserErrors).toEqual([]) +}) diff --git a/examples/react/start-ssr-e2e/netlify.toml b/examples/react/start-ssr-e2e/netlify.toml new file mode 100644 index 0000000000..8b7d900e4c --- /dev/null +++ b/examples/react/start-ssr-e2e/netlify.toml @@ -0,0 +1,8 @@ +[build] +command = "pnpm build" +publish = "dist/client" +functions = "netlify/functions" + +[functions] +node_bundler = "esbuild" +included_files = ["dist/server/**"] diff --git a/examples/react/start-ssr-e2e/netlify/functions/server.mjs b/examples/react/start-ssr-e2e/netlify/functions/server.mjs new file mode 100644 index 0000000000..ce2ee0d83d --- /dev/null +++ b/examples/react/start-ssr-e2e/netlify/functions/server.mjs @@ -0,0 +1,10 @@ +import server from '../../dist/server/server.js' + +export const config = { + path: '/*', + preferStatic: true, +} + +export default function handler(request) { + return server.fetch(request) +} diff --git a/examples/react/start-ssr-e2e/package.json b/examples/react/start-ssr-e2e/package.json new file mode 100644 index 0000000000..b9d06a2ecd --- /dev/null +++ b/examples/react/start-ssr-e2e/package.json @@ -0,0 +1,30 @@ +{ + "name": "@tanstack/db-example-react-start-ssr-e2e", + "private": true, + "version": "0.0.1", + "type": "module", + "scripts": { + "build": "vite build", + "dev": "vite dev", + "test:e2e": "pnpm --filter @tanstack/db build && pnpm --filter @tanstack/react-db build && pnpm --filter @tanstack/react-router-with-db build && playwright test", + "test:e2e:hosted": "playwright test" + }, + "dependencies": { + "@tanstack/react-db": "^0.3.8", + "@tanstack/react-router": "^1.159.5", + "@tanstack/react-router-with-db": "^0.1.0", + "@tanstack/react-start": "^1.159.5", + "react": "^19.2.4", + "react-dom": "^19.2.4", + "vite-tsconfig-paths": "^5.1.4" + }, + "devDependencies": { + "@playwright/test": "^1.60.0", + "@types/node": "^25.2.2", + "@types/react": "^19.2.13", + "@types/react-dom": "^19.2.3", + "@vitejs/plugin-react": "^5.1.3", + "typescript": "^5.9.2", + "vite": "^7.3.0" + } +} diff --git a/examples/react/start-ssr-e2e/playwright.config.ts b/examples/react/start-ssr-e2e/playwright.config.ts new file mode 100644 index 0000000000..ef3628ad29 --- /dev/null +++ b/examples/react/start-ssr-e2e/playwright.config.ts @@ -0,0 +1,31 @@ +import { defineConfig, devices } from '@playwright/test' + +const baseURL = process.env.PLAYWRIGHT_BASE_URL ?? `http://127.0.0.1:4175` +const shouldStartWebServer = process.env.PLAYWRIGHT_BASE_URL === undefined + +export default defineConfig({ + testDir: `./e2e`, + timeout: 30000, + expect: { + timeout: 10000, + }, + fullyParallel: false, + use: { + baseURL, + trace: `on-first-retry`, + }, + webServer: shouldStartWebServer + ? { + command: `pnpm dev --host 127.0.0.1 --port 4175`, + reuseExistingServer: !process.env.CI, + timeout: 120000, + url: baseURL, + } + : undefined, + projects: [ + { + name: `chromium`, + use: { ...devices[`Desktop Chrome`] }, + }, + ], +}) diff --git a/examples/react/start-ssr-e2e/src/lib/ssr-fixture.ts b/examples/react/start-ssr-e2e/src/lib/ssr-fixture.ts new file mode 100644 index 0000000000..affaeda9bd --- /dev/null +++ b/examples/react/start-ssr-e2e/src/lib/ssr-fixture.ts @@ -0,0 +1,108 @@ +import { DbClient, collectionOptions, eq } from '@tanstack/react-db' +import type { DehydratedDbState } from '@tanstack/react-db' + +export type SsrTodo = { + id: string + text: string + status: `open` | `done` + source: `server` | `sync` | `stream` +} + +export const ssrTodoCollectionId = `ssr-e2e-todos` + +const serverTodos: Array = [ + { + id: `server-1`, + text: `Pay invoices`, + status: `open`, + source: `server`, + }, + { + id: `server-2`, + text: `Review pull requests`, + status: `open`, + source: `server`, + }, + { + id: `server-3`, + text: `Archived roadmap`, + status: `done`, + source: `server`, + }, +] + +const browserTodos: Array = serverTodos.map((todo) => + todo.id === `server-1` + ? { + ...todo, + text: `Pay invoices (reconciled from sync)`, + source: `sync`, + } + : { ...todo, source: `sync` }, +) + +export const streamedTodo: SsrTodo = { + id: `streamed-1`, + text: `Streamed from collection chunk`, + status: `open`, + source: `stream`, +} + +export const ssrTodoCollection = collectionOptions(ssrTodoCollectionId, () => ({ + id: ssrTodoCollectionId, + getKey: (todo: SsrTodo) => todo.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + + return { + loadSubset: () => { + const todos = + typeof window === `undefined` ? serverTodos : browserTodos + + begin({ immediate: true }) + for (const todo of todos) { + write({ + type: `insert`, + value: todo, + }) + } + commit() + return true + }, + } + }, + }, +})) + +export async function preloadSsrTodos(dbClient: DbClient): Promise { + await dbClient.preloadLiveQuery({ + query: (q) => + q + .from({ todo: ssrTodoCollection }) + .where(({ todo }) => eq(todo.status, `open`)), + }) +} + +export async function createDehydratedSsrTodoState(): Promise { + const dbClient = new DbClient() + try { + await preloadSsrTodos(dbClient) + return dbClient.dehydrate() + } finally { + await dbClient.cleanup() + } +} + +export function applyStreamedTodo(dbClient: DbClient): void { + dbClient.applyCollectionChunk({ + collectionId: ssrTodoCollectionId, + rows: [ + { + key: streamedTodo.id, + value: streamedTodo, + }, + ], + }) +} diff --git a/examples/react/start-ssr-e2e/src/main.tsx b/examples/react/start-ssr-e2e/src/main.tsx new file mode 100644 index 0000000000..7c9866dcd3 --- /dev/null +++ b/examples/react/start-ssr-e2e/src/main.tsx @@ -0,0 +1,12 @@ +import { StrictMode } from 'react' +import { createRoot } from 'react-dom/client' +import { RouterProvider } from '@tanstack/react-router' +import { getRouter } from './router' + +const router = getRouter() + +createRoot(document.getElementById(`root`)!).render( + + + , +) diff --git a/examples/react/start-ssr-e2e/src/routeTree.gen.ts b/examples/react/start-ssr-e2e/src/routeTree.gen.ts new file mode 100644 index 0000000000..b6d5a30439 --- /dev/null +++ b/examples/react/start-ssr-e2e/src/routeTree.gen.ts @@ -0,0 +1,105 @@ +/* eslint-disable */ + +// @ts-nocheck + +// noinspection JSUnusedGlobalSymbols + +// This file was automatically generated by TanStack Router. +// You should NOT make any changes in this file as it will be overwritten. +// Additionally, you should also exclude this file from your linter and/or formatter to prevent it from being checked or modified. + +import { Route as rootRouteImport } from './routes/__root' +import { Route as SsrDbStreamRouteImport } from './routes/ssr-db-stream' +import { Route as SsrDbRouteImport } from './routes/ssr-db' +import { Route as IndexRouteImport } from './routes/index' + +const SsrDbStreamRoute = SsrDbStreamRouteImport.update({ + id: '/ssr-db-stream', + path: '/ssr-db-stream', + getParentRoute: () => rootRouteImport, +} as any) +const SsrDbRoute = SsrDbRouteImport.update({ + id: '/ssr-db', + path: '/ssr-db', + getParentRoute: () => rootRouteImport, +} as any) +const IndexRoute = IndexRouteImport.update({ + id: '/', + path: '/', + getParentRoute: () => rootRouteImport, +} as any) + +export interface FileRoutesByFullPath { + '/': typeof IndexRoute + '/ssr-db': typeof SsrDbRoute + '/ssr-db-stream': typeof SsrDbStreamRoute +} +export interface FileRoutesByTo { + '/': typeof IndexRoute + '/ssr-db': typeof SsrDbRoute + '/ssr-db-stream': typeof SsrDbStreamRoute +} +export interface FileRoutesById { + __root__: typeof rootRouteImport + '/': typeof IndexRoute + '/ssr-db': typeof SsrDbRoute + '/ssr-db-stream': typeof SsrDbStreamRoute +} +export interface FileRouteTypes { + fileRoutesByFullPath: FileRoutesByFullPath + fullPaths: '/' | '/ssr-db' | '/ssr-db-stream' + fileRoutesByTo: FileRoutesByTo + to: '/' | '/ssr-db' | '/ssr-db-stream' + id: '__root__' | '/' | '/ssr-db' | '/ssr-db-stream' + fileRoutesById: FileRoutesById +} +export interface RootRouteChildren { + IndexRoute: typeof IndexRoute + SsrDbRoute: typeof SsrDbRoute + SsrDbStreamRoute: typeof SsrDbStreamRoute +} + +declare module '@tanstack/react-router' { + interface FileRoutesByPath { + '/ssr-db-stream': { + id: '/ssr-db-stream' + path: '/ssr-db-stream' + fullPath: '/ssr-db-stream' + preLoaderRoute: typeof SsrDbStreamRouteImport + parentRoute: typeof rootRouteImport + } + '/ssr-db': { + id: '/ssr-db' + path: '/ssr-db' + fullPath: '/ssr-db' + preLoaderRoute: typeof SsrDbRouteImport + parentRoute: typeof rootRouteImport + } + '/': { + id: '/' + path: '/' + fullPath: '/' + preLoaderRoute: typeof IndexRouteImport + parentRoute: typeof rootRouteImport + } + } +} + +const rootRouteChildren: RootRouteChildren = { + IndexRoute: IndexRoute, + SsrDbRoute: SsrDbRoute, + SsrDbStreamRoute: SsrDbStreamRoute, +} +export const routeTree = rootRouteImport + ._addFileChildren(rootRouteChildren) + ._addFileTypes() + +import type { getRouter } from './router.tsx' +import type { startInstance } from './start.tsx' +declare module '@tanstack/react-start' { + interface Register { + ssr: true + router: Awaited> + config: Awaited> + } +} diff --git a/examples/react/start-ssr-e2e/src/router.tsx b/examples/react/start-ssr-e2e/src/router.tsx new file mode 100644 index 0000000000..b091853cb9 --- /dev/null +++ b/examples/react/start-ssr-e2e/src/router.tsx @@ -0,0 +1,27 @@ +import { createRouter as createTanstackRouter } from '@tanstack/react-router' +import { createIsomorphicFn } from '@tanstack/react-start' +import { DbClient } from '@tanstack/react-db' +import { routerWithDbClient } from '@tanstack/react-router-with-db' +import { routeTree } from './routeTree.gen' +import './styles.css' + +export type RouterContext = { + dbClient: DbClient +} + +const getRuntime = createIsomorphicFn() + .server(() => `server` as const) + .client(() => `browser` as const) + +export function getRouter() { + const dbClient = new DbClient({ + runtime: getRuntime(), + }) + const router = createTanstackRouter({ + routeTree, + context: { dbClient }, + scrollRestoration: true, + }) + + return routerWithDbClient(router, dbClient) +} diff --git a/examples/react/start-ssr-e2e/src/routes/__root.tsx b/examples/react/start-ssr-e2e/src/routes/__root.tsx new file mode 100644 index 0000000000..ad71067417 --- /dev/null +++ b/examples/react/start-ssr-e2e/src/routes/__root.tsx @@ -0,0 +1,48 @@ +import * as React from 'react' +import { + HeadContent, + Outlet, + Scripts, + createRootRouteWithContext, +} from '@tanstack/react-router' +import appCss from '../styles.css?url' +import type { RouterContext } from '../router' + +export const Route = createRootRouteWithContext()({ + head: () => ({ + meta: [ + { + charSet: `utf-8`, + }, + { + name: `viewport`, + content: `width=device-width, initial-scale=1`, + }, + { + title: `TanStack DB Start SSR E2E`, + }, + ], + links: [ + { + rel: `stylesheet`, + href: appCss, + }, + ], + }), + shellComponent: RootDocument, + component: () => , +}) + +function RootDocument({ children }: { children: React.ReactNode }) { + return ( + + + + + + {children} + + + + ) +} diff --git a/examples/react/start-ssr-e2e/src/routes/index.tsx b/examples/react/start-ssr-e2e/src/routes/index.tsx new file mode 100644 index 0000000000..1131099207 --- /dev/null +++ b/examples/react/start-ssr-e2e/src/routes/index.tsx @@ -0,0 +1,14 @@ +import { Link, createFileRoute } from '@tanstack/react-router' + +export const Route = createFileRoute(`/`)({ + component: HomePage, +}) + +function HomePage() { + return ( +
                +

                TanStack DB Start SSR E2E

                + Open SSR DB route +
                + ) +} diff --git a/examples/react/start-ssr-e2e/src/routes/ssr-db-stream.tsx b/examples/react/start-ssr-e2e/src/routes/ssr-db-stream.tsx new file mode 100644 index 0000000000..14c3aea8e1 --- /dev/null +++ b/examples/react/start-ssr-e2e/src/routes/ssr-db-stream.tsx @@ -0,0 +1,131 @@ +import * as React from 'react' +import { createFileRoute } from '@tanstack/react-router' +import { + collectionOptions, + eq, + useLiveQuery, + useLiveSuspenseQuery, +} from '@tanstack/react-db' +import { preloadSsrTodos, ssrTodoCollection } from '../lib/ssr-fixture' + +type StreamedTodo = { + id: string + text: string + status: `open` | `done` + sourcePayload: string +} + +const serverTodo: StreamedTodo = { + id: `streamed-server-1`, + text: `Streamed while rendering`, + status: `open`, + sourcePayload: `SOURCE_ONLY_DO_NOT_TRANSPORT`, +} + +const browserTodo: StreamedTodo = { + id: `streamed-browser-1`, + text: `Reconciled from browser sync`, + status: `open`, + sourcePayload: `BROWSER_SOURCE_ONLY_DO_NOT_TRANSPORT`, +} + +const streamedTodoCollection = collectionOptions( + `ssr-suspense-stream-todos`, + (client) => { + const runtime = client.requireDependency<`server` | `browser`>(`runtime`) + + return { + id: `ssr-suspense-stream-todos`, + getKey: (todo: StreamedTodo) => todo.id, + syncMode: `on-demand` as const, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + + return { + loadSubset: () => { + return new Promise((resolve) => { + setTimeout( + () => { + begin({ immediate: true }) + write({ + type: `insert`, + value: runtime === `server` ? serverTodo : browserTodo, + }) + commit() + resolve() + }, + runtime === `server` ? 1000 : 1500, + ) + }) + }, + } + }, + }, + } + }, +) + +export const Route = createFileRoute(`/ssr-db-stream`)({ + loader: async ({ context }) => { + await preloadSsrTodos(context.dbClient) + }, + component: SsrDbStreamRoute, +}) + +function SsrDbStreamRoute() { + return ( +
                +

                TanStack DB Suspense Streaming

                + + Loading todos

                } + > + +                 +
                + ) +} + +function CriticalTodoList() { + const { data: todos } = useLiveQuery({ + query: (q) => + q + .from({ todo: ssrTodoCollection }) + .where(({ todo }) => eq(todo.status, `open`)), + }) + + return ( +
                  + {todos.map((todo) => ( +
                • + {todo.text} +
                  • + ))} +
                + ) +} + +function StreamedTodoList() { + const { data: todos } = useLiveSuspenseQuery({ + query: (q) => + q + .from({ todo: streamedTodoCollection }) + .where(({ todo }) => eq(todo.status, `open`)) + .select(({ todo }) => ({ + id: todo.id, + text: todo.text, + status: todo.status, + })), + }) + + return ( +
                  + {todos.map((todo) => ( +
                • + {todo.text} +
                  • + ))} +
                + ) +} diff --git a/examples/react/start-ssr-e2e/src/routes/ssr-db.tsx b/examples/react/start-ssr-e2e/src/routes/ssr-db.tsx new file mode 100644 index 0000000000..6b31a87460 --- /dev/null +++ b/examples/react/start-ssr-e2e/src/routes/ssr-db.tsx @@ -0,0 +1,110 @@ +import * as React from 'react' +import { createFileRoute } from '@tanstack/react-router' +import { eq, useLiveQuery } from '@tanstack/react-db' +import { + applyStreamedTodo, + createDehydratedSsrTodoState, + ssrTodoCollection, +} from '../lib/ssr-fixture' +import type { DbClient } from '@tanstack/react-db' + +export const Route = createFileRoute(`/ssr-db`)({ + loader: async () => { + return { + dbState: await createDehydratedSsrTodoState(), + } + }, + component: SsrDbRoute, +}) + +function SsrDbRoute() { + const { dbState } = Route.useLoaderData() + const { dbClient } = Route.useRouteContext() + const [hydratedDbClient] = React.useState(() => { + dbClient.hydrate(dbState) + return dbClient + }) + + return +} + +function SsrDbTodos({ dbClient }: { dbClient: DbClient }) { + const [hydrated, setHydrated] = React.useState(false) + const [streamed, setStreamed] = React.useState(false) + const { data: todos, isReady } = useLiveQuery({ + query: (q) => + q + .from({ todo: ssrTodoCollection }) + .where(({ todo }) => eq(todo.status, `open`)) + .orderBy(({ todo }) => todo.id, `asc`), + }) + + React.useEffect(() => { + setHydrated(true) + }, []) + + return ( +
                +
                +

                TanStack DB SSR

                + +
                + + {hydrated ? `hydrated` : `ssr`} + + {isReady ? `ready` : `loading`} + + {streamed ? `streamed` : `waiting`} + + + rows: {todos.length} + +
                + +
                  + {todos.map((todo) => ( +
                • + {todo.text} ({todo.source}) +
                  • + ))} +
                + + +
                +
                + ) +} diff --git a/examples/react/start-ssr-e2e/src/start.tsx b/examples/react/start-ssr-e2e/src/start.tsx new file mode 100644 index 0000000000..bb197cafb1 --- /dev/null +++ b/examples/react/start-ssr-e2e/src/start.tsx @@ -0,0 +1,7 @@ +import { createStart } from '@tanstack/react-start' + +export const startInstance = createStart(() => { + return { + defaultSsr: true, + } +}) diff --git a/examples/react/start-ssr-e2e/src/styles.css b/examples/react/start-ssr-e2e/src/styles.css new file mode 100644 index 0000000000..251e05865f --- /dev/null +++ b/examples/react/start-ssr-e2e/src/styles.css @@ -0,0 +1,15 @@ +body { + margin: 0; + font-family: + Inter, + ui-sans-serif, + system-ui, + -apple-system, + BlinkMacSystemFont, + 'Segoe UI', + sans-serif; +} + +button { + font: inherit; +} diff --git a/examples/react/start-ssr-e2e/tsconfig.json b/examples/react/start-ssr-e2e/tsconfig.json new file mode 100644 index 0000000000..19dcb2d948 --- /dev/null +++ b/examples/react/start-ssr-e2e/tsconfig.json @@ -0,0 +1,20 @@ +{ + "extends": "../../../tsconfig.json", + "compilerOptions": { + "baseUrl": ".", + "module": "ES2022", + "moduleResolution": "bundler", + "jsx": "react-jsx", + "paths": { + "@/*": ["./src/*"] + } + }, + "include": [ + "e2e/**/*.ts", + "playwright.config.ts", + "src/**/*.ts", + "src/**/*.tsx", + "vite.config.ts" + ], + "exclude": ["dist", "node_modules"] +} diff --git a/examples/react/start-ssr-e2e/vite.config.ts b/examples/react/start-ssr-e2e/vite.config.ts new file mode 100644 index 0000000000..856428d501 --- /dev/null +++ b/examples/react/start-ssr-e2e/vite.config.ts @@ -0,0 +1,17 @@ +import { defineConfig } from 'vite' +import react from '@vitejs/plugin-react' +import { tanstackStart } from '@tanstack/react-start/plugin/vite' +import viteTsConfigPaths from 'vite-tsconfig-paths' + +export default defineConfig({ + plugins: [ + viteTsConfigPaths({ + projects: [`./tsconfig.json`], + }), + tanstackStart({ + srcDirectory: `src`, + start: { entry: `./start.tsx` }, + }), + react(), + ], +}) diff --git a/examples/react/todo/CHANGELOG.md b/examples/react/todo/CHANGELOG.md index f2858046be..ad7c91b2a7 100644 --- a/examples/react/todo/CHANGELOG.md +++ b/examples/react/todo/CHANGELOG.md @@ -1,5 +1,25 @@ # examples/react/todo +## 0.1.27 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/react-db@0.3.0 + - @tanstack/electric-db-collection@0.4.0 + - @tanstack/query-db-collection@1.2.5 + - @tanstack/trailbase-db-collection@0.1.99 + +## 0.1.26 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/react-db@0.2.0 + - @tanstack/electric-db-collection@0.3.17 + - @tanstack/query-db-collection@1.2.3 + - @tanstack/trailbase-db-collection@0.1.97 + ## 0.1.25 ### Patch Changes diff --git a/examples/react/todo/package.json b/examples/react/todo/package.json index a656ae38ed..fe683412b6 100644 --- a/examples/react/todo/package.json +++ b/examples/react/todo/package.json @@ -1,15 +1,15 @@ { "name": "@tanstack/db-example-react-todo", "private": true, - "version": "0.1.25", + "version": "0.1.27", "dependencies": { - "@tanstack/electric-db-collection": "^0.3.6", + "@tanstack/electric-db-collection": "^0.4.8", "@tanstack/query-core": "^5.90.20", - "@tanstack/query-db-collection": "^1.0.40", - "@tanstack/react-db": "^0.1.86", + "@tanstack/query-db-collection": "^1.2.13", + "@tanstack/react-db": "^0.3.8", "@tanstack/react-router": "^1.159.5", "@tanstack/react-start": "^1.159.5", - "@tanstack/trailbase-db-collection": "^0.1.86", + "@tanstack/trailbase-db-collection": "^0.1.107", "cors": "^2.8.6", "drizzle-orm": "^0.45.1", "drizzle-zod": "^0.8.3", diff --git a/examples/react/todo/src/routes/electric.tsx b/examples/react/todo/src/routes/electric.tsx index 61629b81f2..16da41b9ff 100644 --- a/examples/react/todo/src/routes/electric.tsx +++ b/examples/react/todo/src/routes/electric.tsx @@ -24,15 +24,16 @@ export const Route = createFileRoute(`/electric`)({ function ElectricPage() { // Get data using live queries with Electric collections - const { data: todos } = useLiveQuery((q) => - q - .from({ todo: electricTodoCollection }) - .orderBy(({ todo }) => todo.created_at, `asc`), - ) + const { data: todos } = useLiveQuery({ + query: (q) => + q + .from({ todo: electricTodoCollection }) + .orderBy(({ todo }) => todo.created_at, `asc`), + }) - const { data: configData } = useLiveQuery((q) => - q.from({ config: electricConfigCollection }), - ) + const { data: configData } = useLiveQuery({ + query: (q) => q.from({ config: electricConfigCollection }), + }) // Electric collections use txid to track sync const configMutationFn = async ({ diff --git a/examples/react/todo/src/routes/query.tsx b/examples/react/todo/src/routes/query.tsx index 62c0ad37dc..5cbf4f28de 100644 --- a/examples/react/todo/src/routes/query.tsx +++ b/examples/react/todo/src/routes/query.tsx @@ -21,15 +21,16 @@ export const Route = createFileRoute(`/query`)({ function QueryPage() { // Get data using live queries with Query collections - const { data: todos } = useLiveQuery((q) => - q - .from({ todo: queryTodoCollection }) - .orderBy(({ todo }) => todo.created_at, `asc`), - ) + const { data: todos } = useLiveQuery({ + query: (q) => + q + .from({ todo: queryTodoCollection }) + .orderBy(({ todo }) => todo.created_at, `asc`), + }) - const { data: configData } = useLiveQuery((q) => - q.from({ config: queryConfigCollection }), - ) + const { data: configData } = useLiveQuery({ + query: (q) => q.from({ config: queryConfigCollection }), + }) // Query collections automatically refetch after handler completes const configMutationFn = async ({ diff --git a/examples/react/todo/src/routes/trailbase.tsx b/examples/react/todo/src/routes/trailbase.tsx index 96e05e11ac..d4b5b46556 100644 --- a/examples/react/todo/src/routes/trailbase.tsx +++ b/examples/react/todo/src/routes/trailbase.tsx @@ -22,15 +22,16 @@ export const Route = createFileRoute(`/trailbase`)({ function TrailBasePage() { // Get data using live queries with TrailBase collections - const { data: todos } = useLiveQuery((q) => - q - .from({ todo: trailBaseTodoCollection }) - .orderBy(({ todo }) => todo.created_at, `asc`), - ) + const { data: todos } = useLiveQuery({ + query: (q) => + q + .from({ todo: trailBaseTodoCollection }) + .orderBy(({ todo }) => todo.created_at, `asc`), + }) - const { data: configData } = useLiveQuery((q) => - q.from({ config: trailBaseConfigCollection }), - ) + const { data: configData } = useLiveQuery({ + query: (q) => q.from({ config: trailBaseConfigCollection }), + }) // Note: TrailBase collections use recordApi internally, which is not exposed // as a collection utility. For this example, we're not using serialized diff --git a/examples/solid/todo/CHANGELOG.md b/examples/solid/todo/CHANGELOG.md index 760666c0ee..1995734e18 100644 --- a/examples/solid/todo/CHANGELOG.md +++ b/examples/solid/todo/CHANGELOG.md @@ -1,5 +1,15 @@ # examples/react/todo +## 0.0.36 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/electric-db-collection@0.4.0 + - @tanstack/query-db-collection@1.2.5 + - @tanstack/trailbase-db-collection@0.1.99 + - @tanstack/solid-db@0.2.35 + ## 0.0.35 ### Patch Changes diff --git a/examples/solid/todo/package.json b/examples/solid/todo/package.json index 215c98e075..86e214f820 100644 --- a/examples/solid/todo/package.json +++ b/examples/solid/todo/package.json @@ -1,15 +1,15 @@ { "name": "@tanstack/db-example-solid-todo", "private": true, - "version": "0.0.35", + "version": "0.0.36", "dependencies": { - "@tanstack/electric-db-collection": "^0.3.6", + "@tanstack/electric-db-collection": "^0.4.8", "@tanstack/query-core": "^5.90.20", - "@tanstack/query-db-collection": "^1.0.40", - "@tanstack/solid-db": "^0.2.22", + "@tanstack/query-db-collection": "^1.2.13", + "@tanstack/solid-db": "^0.2.43", "@tanstack/solid-router": "^1.159.5", "@tanstack/solid-start": "^1.159.5", - "@tanstack/trailbase-db-collection": "^0.1.86", + "@tanstack/trailbase-db-collection": "^0.1.107", "cors": "^2.8.6", "drizzle-orm": "^0.45.1", "drizzle-zod": "^0.8.3", diff --git a/media/header_db.png b/media/header_db.png deleted file mode 100644 index 7ece5a83e8..0000000000 Binary files a/media/header_db.png and /dev/null differ diff --git a/notes/rfc-1657-server-pagination.md b/notes/rfc-1657-server-pagination.md new file mode 100644 index 0000000000..cff1151f1e --- /dev/null +++ b/notes/rfc-1657-server-pagination.md @@ -0,0 +1,172 @@ +# RFC 1657 follow-up: server pagination investigation + +Base: `origin/main` at `ad043b745` (verified after fetch on 2026-09-10). +Worktree: `codex-rfc-server-pagination`. + +## Accepted implementation + +The user approved option A: remove the silently ignored `getNextPageParam` +option, reject it before query construction for untyped callers, and explain +the existing on-demand server-pagination protocol. This is an API migration, +not a new InfiniteQueryObserver bridge or remote cursor registry. + +- [x] Remove the callback from React, Vue, and Svelte config types. Keep generic + config wrappers source-compatible; unrelated return types stay unchanged. +- [x] RED first: all three hook guards failed against baseline with the query + callback reached before rejection. Logs: `/private/tmp/968-{react,vue,svelte}-red.log`. +- [x] GREEN: reject the removed option before hook resources/query construction. + Type tests also assert the callback is not advertised. +- [x] Preserve existing successful hook tests without the ignored callback. +- [x] Expand the actual Query DB/React boundary to 36 cells: server page sizes + 1/2/3/5 × UI sizes 1/2/5 × row counts 0/1/8. Each checkpoint compares full + visible IDs and hasNextPage to an independent array slice. +- [x] Keep eager transport, prefix retention, invalid capped-provider, page-label, + and explicit-refetch ownership controls. The capped provider violates the + request protocol; its characterization is not a successful pagination oracle. +- [x] Replace the misleading manual-append pagination example with an on-demand + fixed-server-page drain example; explain ordering/filter translation, + cancellation, exhaustion, unlimited loads, and opaque-cursor boundaries. +- [x] Fix the React overview to stop recommending the ignored callback. +- [x] Package Vitest gates (with each package's configured type checking): + React reports 220 passing checks; Vue 96; Svelte 101. Query DB's final + log reports 370 passes / 371 discovered checks, no failures or type errors. + These are Vitest runtime/type reports, not standalone `tsc` claims. + Electric declarations were built before the final Query DB gate, clearing + the earlier dependency blocker recorded in the historical handoff below. +- [x] Focused lint: no errors or warnings. No core/adapter runtime edits. +- [x] Prepare minor framework changesets and focused PR publication. + +Verification logs: `/private/tmp/968-react-full.log`, +`/private/tmp/968-vue-green.log`, `/private/tmp/968-svelte-green.log`, +`/private/tmp/968-query-full.log`, `/private/tmp/968-lint.log`. + +Test gap: previous framework tests materialized all rows or supplied the callback +without asserting invocation. The new tests cross the real QueryObserver and +collection/window boundary. They model numeric ascending ID/rank queries only, +not arbitrary endpoint ordering, cursor protocols, retries, or changing datasets. + +## Original investigation (historical evidence) + +## Work queue + +- [x] Read AGENTS and live ARCHITECTURE in full. +- [x] Read report #968 and the current disposition of RFC #1657. +- [x] Read proposed documentation in #1355 without merging it. +- [x] Reproduce eager Query DB + real React useLiveInfiniteQuery behavior. +- [x] Verify the documented on-demand prefix path through the real adapter. +- [x] Cross fixed server page sizes against live-query page growth. +- [x] Identify API decisions and minimal alternatives below. +- [x] Run focused runtime gates and finalize findings (type limits below). +- [x] Parent/user selects API migration vs additional server-page feature. + +## Evidence + +Seven React integration probes pass on main. They use a real QueryClient, +QueryObserver, Query Collection, live-query compiler/window controller, and +useLiveInfiniteQuery. A separate test fixture evaluates small numeric predicates +with an independent test evaluator, not production expression evaluation. + +1. Report shape: eager source initially returns four of eight available rows. + A page size of two reveals two then four rows. Further fetches do nothing; + `hasNextPage` is false, QueryFn ran once, `pageParam` is undefined, and the + supplied `getNextPageParam` callback was never invoked. +2. On-demand provider fulfilling prefix demand: pages grow 2, 4, 6, 8 correctly; + finite requests have limits 3, 5, 7, 9. Tie requests can also occur. Prior rows + remain visible across these distinct Query cache entries. +3. A capped provider returning two rows for a limit-three request underfills + the local window and yields false `hasNextPage` despite eight remote rows. + This is a deliberately nonconforming adapter, not proof that core can know + unknown server extent. Main's exact request contract requires fulfilling + the request (or exhausting that source) before successful settlement. +4. Adapter-local page draining with explicit endpoint `nextPage` works through + the same actual hook for server page sizes 1, 2, 3, 5. The hook pages remain + size two; its first request peeks a third row. Later requests use offsets, + and page-number conversion/draining stays in QueryFn. Core needs no new state. + +Why previous tests missed the report: the React hook tests use fully materialized +mock collections. They exercise local peek-ahead, not QueryObserver's transport +context or an endpoint whose page size differs from the requested live window. +The existing test that passes `getNextPageParam` never asserts it runs. + +## Minimal coherent options + +### A. Clarify existing API and reject or warn on the no-op callback + +Recommended scope. Query DB uses QueryObserver, not InfiniteQueryObserver. Its +queryFn receives `meta.loadSubsetOptions`, not a server pageParam. The live hook +widens a local ordered query. `initialPageParam` labels result pageParams; it is +not a server cursor or an initial remote offset. + +Document on-demand limit/offset/filter/order translation and show a fixed-page +endpoint adapter that drains to fulfill the requested window, stopping only on +authoritative endpoint exhaustion. Keep arbitrary opaque cursor caching and +resume state inside that adapter. It may need to refetch earlier pages to honor +random offsets or independent queries; that is the honest cost of that endpoint. + +Decision needed: remove `getNextPageParam` (pre-1.0 API cleanup plus clear runtime +error for untyped callers) or retain it with a once-per-hook warning. A silent +compatibility option promises more compatibility than exists. It is currently +defined by React, Vue, and Svelte hooks; its comment and generated docs call it a no-op. +No decision is applied in this spike. + +Runtime state cost: zero for docs/adapter sample, one warning guard if warning +chosen. A runtime rejection needs no retained state. Test-only cross-package +fixture avoids adding Query dependencies to the React runtime. + +### B. New hook/server-page bridge + +Not a bug fix to the existing QueryObserver contract. Requires defining which +collection owns cursor history, how independent filters/windows share pages, +restart/invalidation semantics, and how UI hasNextPage maps to remote extent. +An arbitrary query may join/filter/group remote rows; a source's next page is +not necessarily another public result page. This must not be smuggled in as a +boolean or restored coverage registry. No runtime implementation proposed. + +## Review of #1355's docs proposal + +Useful: on-demand is the recommended alternative; direct writes need an explicit +ownership/refetch policy; function Query keys must distinguish relevant demand. + +Do not copy unchanged: + +- `staleTime: Infinity` prevents staleness-driven refetch, not explicit refetch, + invalidation, or configured polling. Those can still replace appended rows. +- `enabled: false` suppresses initial automatic acquisition too, not merely + later refresh. It needs a separately managed loading path. +- Examples omit orderBy and imply one exact Query cache entry per UI page. + Real loading can issue prefixes, suffixes, and tie requests; include an + explicit total order and describe requests rather than promising UI-page keys. +- The adapter must honor filters/order and drain server caps, not simply forward + limit/offset if the endpoint can silently truncate them. +- Appending with writeInsert fails on duplicate row IDs; writeUpsert can express + incremental page merging, but a later full-state snapshot still owns removal. + +## Boundaries retained + +No coverage algebra, outcome registry, new public extent facts, or inferential +claim that a short arbitrary response establishes source exhaustion. The +fixture's nextPage is endpoint-authoritative test data used only inside QueryFn. +The intentionally broken capped-provider probe characterizes missing information; +it must not become a green oracle accepting this provider as correct. + +## Historical probe handoff — before the accepted migration (2026-09-10) + +The following records the initial investigation only. It was superseded by the +accepted implementation and package gates above; it is not the PR's final state. + +From packages/react-db: +`pnpm exec vitest run tests/server-pagination-probe.test.tsx --coverage.enabled=false --maxWorkers=2` +passed all seven probes with no errors from Vitest's configured React type check. + +From packages/query-db-collection: +`pnpm exec vitest run tests/server-pagination-boundary.test.ts --coverage.enabled=false --typecheck.enabled=false --maxWorkers=2` +checked that explicit refetch removes a manually appended eager row despite +staleTime Infinity. Runtime passed. At this early stage the Query DB type pass was blocked by +unbuilt Electric declarations imported by existing cross-package E2E suites. +A fixture inference error found by that pass was corrected with an explicit +page-response type; no production code changed. + +At that early handoff only db-ivm/db builds and local probes were complete. +The accepted migration subsequently shipped in PR #1806 with tests, docs and +minor framework changesets; these files and this document are now committed. +The separate parent-owned rfc-next-work-plan.md remains untracked. diff --git a/notes/rfc-1657-trailbase.md b/notes/rfc-1657-trailbase.md new file mode 100644 index 0000000000..b9628d6dc7 --- /dev/null +++ b/notes/rfc-1657-trailbase.md @@ -0,0 +1,135 @@ +# RFC 1657 follow-up: TrailBase stream termination + +Base: origin/main ad043b745. This is the narrow error-handling bug found while +examining #1521; it does not implement that PR's polling policy. + +- [x] Reproduce with the actual Collection and TrailBase adapter. +- [x] Add close/error matrix checking reported errors, unhandled rejections, + reader lock, cleanup timer, retained rows/status, and absence of extra loads. +- [x] RED: normal close passes; errored stream leaks an unhandled rejection and + keeps its reader locked. Later cleanup can reject again when canceling it. + /private/tmp/trailbase-stream-red.log. +- [x] Fix: observe both settlements of reader.closed, clear interval, release + reader, and clear only the matching active-reader reference. The existing + listen catch remains the error reporter. +- [x] GREEN: all12 package runtime tests pass; no type errors. ESLint no errors + and one pre-existing require-await warning. Prettier unchanged. + /private/tmp/trailbase-stream-green.log. +- [ ] Release note and PR after integration review. No implementation pushed. + +## Why the tests missed it + +Existing tests close streams normally or cancel them deliberately. None errors +a live stream after startup. The listen() rejection handler did not observe the +separate promise returned by reader.closed.finally(). Normal close and rejected +close therefore need separate laws, including resource cleanup after failure. + +## Boundaries preserved + +Initial subscribe failure still rejects readiness. Post-start disconnection +keeps the last ready rows. No polling/reconnect behavior, automatic refetch, +mutation confirmation policy, or core change is added. + +The broader #1521 proposal claims a polling cycle which does not exist, marks +ready even after required list failure, and skips acknowledgement waits based +only on initial subscription availability. Do not transplant it. Stale same-ID +ack evidence and actual degradation/recovery policy remain separate decisions. + +## PR preparation review + +Fetched origin/main: still ad043b745. Simplification review found no worthwhile +cuts. Correctness review found a regression in the initial fix: reader.closed +can settle before listen() drains its last queued event. Releasing the lock at +that point turns graceful buffered closure into a spurious subscription error. + +- [x] Add buffered-close to the close/error matrix. Enqueue two inserts before + closing; assert both rows applied, no error report, timer cleared, reader + released and later collection cleanup safe. RED: one fail, two controls pass. + `/private/tmp/trailbase-buffered-close-red.log`. +- [x] Retire the reader after listen() settles, with a caught finally chain. + reader.closed now only clears the timer and observes both settlements. +- [x] Full package: 13 runtime tests plus one type test pass, no type errors. + Built the missing Electric declaration dependency before the full type gate. + ESLint: no errors, one pre-existing require-await warning. + `/private/tmp/trailbase-prep-final-{green,lint}.log`. +- [ ] Commit review follow-up, changeset and PR after preparation approval. + +The original matrix terminated an idle reader after readiness. It did not cross +buffered delivery with close. This is why it missed the race; no new recovery +policy or retry loop is needed to correct reader ownership. + +The follow-up review found that retiring the reader after listen() rejects must +also cancel a still-open source: parsing/writing can fail without a stream read +failure. Added an actual parse-failure test with an underlying cancel spy. RED: +zero source cancellations. The listener's handled finally now awaits cancellation +(preserving the original error on an already-errored stream) before releasing its +lock. Buffered-close, normal-close and read-error controls remain GREEN. + +Final gate: 14 runtime tests and one type test pass; no type errors, lint errors, +or new warnings. RED log: /private/tmp/trailbase-processing-failure-red.log. +Final GREEN/lint logs reuse the prep-final paths above. + +Adjacent pre-existing limitation, not claimed solved: erroring the stream and +calling collection.cleanup() in the same turn can still expose the unobserved +promise from cancelEventReader(). Its cancellation path is unchanged. The +reviewer also verified that an error during pending initial loading now retires +the reader before a later, settled cleanup. Do not equate that with the same-turn +cleanup case. Track the latter as a further cancellation test/fix before claiming +full termination coverage. + +## Same-turn cleanup follow-up + +User approved including the adjacent cancellation failure. Added the four-cell +loading/ready × close/error matrix with no microtask between stream termination +and collection cleanup. Both error cells RED, both close controls GREEN. +`/private/tmp/trailbase-same-turn-red.log`. + +cancelEventReader now observes its cancellation promise, since native errored +streams reject cancellation with the original stream error. The reader is still +retired synchronously. The tests observe preload immediately, resolve delayed +loading after cleanup, and assert no unhandled rejection, no late rows, no new +fetch/subscription, an unlocked stream and cleaned-up status. + +TrailBase has no dedicated fast-check/model oracle yet. Its local tests are +example/matrix integration tests. Its service-backed E2E entry runs shared +predicate, pagination, join, deduplication, collation, mutation, live-update and +progressive suites; these are not a generated lifecycle model. E2E was inspected, +not run during this preparation. A bounded future lifecycle oracle should vary +startup/list completion, event delivery, stream termination, processing failures, +cleanup and restart, with independent row/status/resource observations. Do not +add polling semantics to that model without a separate policy decision. + +Full local gate: 18 runtime tests plus one type test pass; no type errors or +lint errors (one pre-existing require-await warning). Logs: +`/private/tmp/trailbase-same-turn-{green,lint}.log`. No new commit or push yet. + +## Lifecycle oracle (user-approved) + +- [x] Drive the actual adapter and collection with controlled subscribe/list + promises and native streams; model visible rows independently. +- [x] Cross eager/on-demand, startup failures, delayed old-session settlements, + row edits, graceful/buffered close, read/parse failures, cleanup and restart. + Keep existing unit and loading-time matrices. Extract only the shared mock. +- [x] Add 32 corpus cases plus fixed and fresh-seed fast-check campaigns using + the existing replay configuration, not a second seed parser. +- [x] New bug RED in both generated campaigns and both modes' fixed witnesses: + old canceled subscribe rejects after restart and cancels the current reader. + Fixed seed 714203, path 21:0:2:2; random seed -951227069, path 39:0:3:2:2:2. + Log: /private/tmp/trailbase-oracle-first.log. +- [x] Guard canceled startup before touching shared reader ownership. GREEN. +- [x] Mutation assay: all five known stream fault variants rejected by this + oracle itself. Logs: /private/tmp/trailbase-oracle-assay-\*.log. +- [x] 10× campaign: 800 generated histories plus 32 fixed cases pass, random + seed 127535183. Log: /private/tmp/trailbase-oracle-stress.log. +- [x] Document laws, exclusions, replay commands and mutation evidence in + packages/trailbase-db-collection/tests/ORACLE.md. + +The gap that exposed the new bug was ownership across sessions: testing only +cleanup followed by old settlement leaves no replacement resource to corrupt. +The oracle settles old work while the next stream is live, then sends new edits. +This is a small lifecycle oracle, not complete TrailBase adapter coverage. +No polling, reconnect, pagination or mutation-confirmation policy was added. + +Final local gate: 52 runtime tests plus one type test pass, no type errors. +Lint has no errors and only the existing require-await warning. Logs: +/private/tmp/trailbase-oracle-{green,lint}.log. Changes remain local, uncommitted. diff --git a/package.json b/package.json index 93eae3610d..43675440d6 100644 --- a/package.json +++ b/package.json @@ -18,6 +18,7 @@ "lint-all": "eslint . --fix", "prepare": "husky", "test": "pnpm --filter \"./packages/**\" test", + "test:oracles": "pnpm --filter @tanstack/db test:oracles && pnpm --filter @tanstack/query-db-collection test:oracles", "test:docs": "node scripts/verify-links.ts", "test:sherif": "sherif -i zod -p offline-transactions-react-native -p shopping-list-react-native", "generate-docs": "node scripts/generate-docs.ts" diff --git a/packages/angular-db/CHANGELOG.md b/packages/angular-db/CHANGELOG.md index 10a322326a..b724d640b4 100644 --- a/packages/angular-db/CHANGELOG.md +++ b/packages/angular-db/CHANGELOG.md @@ -1,5 +1,170 @@ # @tanstack/angular-db +## 0.1.89 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 0.1.88 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 0.1.87 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 0.1.86 + +### Patch Changes + +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 0.1.85 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 0.1.84 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 0.1.83 + +### Patch Changes + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 0.1.82 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 0.1.81 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 0.1.80 + +### Patch Changes + +- Update agent skills to match current APIs and behavior. ([#1696](https://github.com/TanStack/db/pull/1696)) + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 0.1.79 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 0.1.78 + +### Patch Changes + +- Add an internal shared live-query observer and migrate all five framework adapters to it ([#1642](https://github.com/TanStack/db/pull/1642)) + + Introduces `createLiveQueryObserver` in `@tanstack/db`: given a resolved live-query collection (or `null` for a disabled query) it owns the subscription lifecycle every adapter used to re-implement — change and status subscriptions, a snapshot with stable identity per state revision for wholesale consumers, and delivery of the raw `ChangeMessage[]` for granular consumers. React, Vue, Svelte, Solid, and Angular's live-query hooks now materialize from the observer instead of their own hand-rolled subscription/status/snapshot machinery, keeping each adapter's native reactivity and each adapter's data-loading policy (wholesale adapters subscribe without initial state; granular adapters seed from it). + + The observer is an **internal, unstable contract** for TanStack DB's official adapters — it is exported so the adapter packages can consume it, but it is not a public extension point yet and its API may change in any release. + + The migration also fixes several live-query lifecycle defects: status-only transitions (`error`, `cleaned-up`) now reach mounted consumers; snapshot identity is stable across unsubscribe/resubscribe and stays fresh while detached; dispatch is FIFO and non-reentrant with subscriptions identified by record rather than callback; disposing during the synchronous initial replay no longer leaks the collection subscription; subscribing after dispose throws instead of registering a dead listener; Solid guards its async resource continuations against superseded collections; and constructing an observer no longer activates sync. Observers activate on their first committed subscription unless an adapter has already started a pre-created collection supplied directly or returned from a callback. + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 0.1.77 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 0.1.76 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 0.1.75 + +### Patch Changes + +- fix(angular-db): a `{ query }` config-object passed to `injectLiveQuery` now syncs ([#1638](https://github.com/TanStack/db/pull/1638)) + + The config-object branch called `createLiveQueryCollection(opts)` without defaulting `startSync`, unlike the query-function and reactive-options branches (which force `startSync: true`), so a bare `{ query }` never started syncing and produced no data. It now defaults `startSync: true` and `gcTime: 0` while still honoring any explicit values in the config. + +- Extract shared live-query adapter helpers into `@tanstack/db` ([#1641](https://github.com/TanStack/db/pull/1641)) + + Adds `isCollection`, `isSingleResultCollection`, and `getLiveQueryStatusFlags` to `@tanstack/db` and migrates all five framework adapters to use them. `isCollection` replaces the per-adapter duck-typing and Solid's `instanceof CollectionImpl` with one structural, multi-realm-safe guard (the `instanceof` form gave false negatives across dual-package boundaries). No behavior change; internal deduplication only. + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 0.1.74 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 0.1.73 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 0.1.72 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 0.1.71 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 0.1.70 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 0.1.69 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 0.1.68 ### Patch Changes diff --git a/packages/angular-db/README.md b/packages/angular-db/README.md index a2e3b29647..7aba438cba 100644 --- a/packages/angular-db/README.md +++ b/packages/angular-db/README.md @@ -1,3 +1,20 @@ +
                + + + + TanStack Angular DB + +
                # @tanstack/angular-db Angular hooks for TanStack DB. See [TanStack/db](https://github.com/TanStack/db) for more details. diff --git a/packages/angular-db/package.json b/packages/angular-db/package.json index e8ee4b5620..f9acc45c78 100644 --- a/packages/angular-db/package.json +++ b/packages/angular-db/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/angular-db", - "version": "0.1.68", + "version": "0.1.89", "description": "Angular integration for @tanstack/db", "author": "Ethan McDaniel", "license": "MIT", diff --git a/packages/angular-db/skills/angular-db/SKILL.md b/packages/angular-db/skills/angular-db/SKILL.md index e6e17f85b3..11f6ac3e4f 100644 --- a/packages/angular-db/skills/angular-db/SKILL.md +++ b/packages/angular-db/skills/angular-db/SKILL.md @@ -10,7 +10,7 @@ description: > type: framework library: db framework: angular -library_version: '0.6.0' +library_version: '0.6.17' requires: - db-core sources: @@ -101,6 +101,10 @@ const query = injectLiveQuery({ }) ``` +A bare `{ query }` config defaults to `startSync: true` and `gcTime: 0`, like +the query-function overload. Explicit values in the config override those +defaults. + ## Angular-Specific Patterns ### Reactive params with signals diff --git a/packages/angular-db/src/index.ts b/packages/angular-db/src/index.ts index 1bce5c6843..ba579c31a6 100644 --- a/packages/angular-db/src/index.ts +++ b/packages/angular-db/src/index.ts @@ -6,11 +6,15 @@ import { inject, signal, } from '@angular/core' -import { BaseQueryBuilder, createLiveQueryCollection } from '@tanstack/db' +import { + BaseQueryBuilder, + createLiveQueryCollection, + createLiveQueryObserver, + isCollection, + isSingleResultCollection, +} from '@tanstack/db' import type { - ChangeMessage, Collection, - CollectionConfigSingleRowOption, CollectionStatus, Context, GetResult, @@ -139,14 +143,7 @@ export function injectLiveQuery(opts: any) { const collection = computed(() => { // Check if it's an existing collection - const isExistingCollection = - opts && - typeof opts === `object` && - typeof opts.subscribeChanges === `function` && - typeof opts.startSyncImmediate === `function` && - typeof opts.id === `string` - - if (isExistingCollection) { + if (isCollection(opts)) { return opts } @@ -194,9 +191,11 @@ export function injectLiveQuery(opts: any) { }) } - // Handle LiveQueryCollectionConfig objects + // Handle LiveQueryCollectionConfig objects. Default startSync/gcTime to + // match the query-fn and reactive-options paths, but let an explicit value + // in the config win — otherwise a bare `{ query }` never syncs. if (opts && typeof opts === `object` && typeof opts.query === `function`) { - return createLiveQueryCollection(opts) + return createLiveQueryCollection({ startSync: true, gcTime: 0, ...opts }) } throw new Error(`Invalid options provided to injectLiveQuery`) @@ -214,10 +213,9 @@ export function injectLiveQuery(opts: any) { if (!currentCollection) { return internalData() } - const config = currentCollection.config as - | CollectionConfigSingleRowOption - | undefined - return config?.singleResult ? internalData()[0] : internalData() + return isSingleResultCollection(currentCollection) + ? internalData()[0] + : internalData() }) const syncDataFromCollection = ( @@ -251,28 +249,25 @@ export function injectLiveQuery(opts: any) { cleanup() - // Initialize immediately with current state - syncDataFromCollection(currentCollection) - - // Start sync if idle - if (currentCollection.status === `idle`) { - currentCollection.startSyncImmediate() - // Update status after starting sync - status.set(currentCollection.status) - } - - // Subscribe to changes - const subscription = currentCollection.subscribeChanges( - (_: Array>) => { - syncDataFromCollection(currentCollection) - }, - ) - unsub = subscription.unsubscribe.bind(subscription) + // The shared observer owns sync start, subscription, the ready-race, and + // status transitions; Angular re-reads the whole collection on each notify + // (wholesale) into its signals. + // Angular re-reads the collection on notify; wholesale mode preserves its + // pre-observer loading policy (no initial-state snapshot request). + const observer = createLiveQueryObserver(currentCollection, { + mode: `wholesale`, + }) - // Handle ready state - currentCollection.onFirstReady(() => { - status.set(currentCollection.status) + const unsubscribe = observer.subscribe(() => { + syncDataFromCollection(currentCollection) }) + // Wholesale attach suppresses listener calls raised by synchronous sync + // startup. Read once after subscribe returns to capture that final state. + syncDataFromCollection(currentCollection) + unsub = () => { + unsubscribe() + observer.dispose() + } onCleanup(cleanup) }) @@ -282,7 +277,9 @@ export function injectLiveQuery(opts: any) { return { state, data, - collection, + // Loosely typed so the impl return stays compatible with every overload + // (the shared `isCollection` guard narrows the computed to `Collection | null`). + collection: collection as Signal, status, isLoading: computed(() => status() === `loading`), isReady: computed(() => status() === `ready` || status() === `disabled`), diff --git a/packages/angular-db/tests/conformance.test.ts b/packages/angular-db/tests/conformance.test.ts new file mode 100644 index 0000000000..6a5b5ba66a --- /dev/null +++ b/packages/angular-db/tests/conformance.test.ts @@ -0,0 +1,222 @@ +/** + * Angular driver for the shared live-query conformance suite. + * + * `injectLiveQuery` needs an injection context, so each mount runs inside a + * child `EnvironmentInjector` created off TestBed's; `unmount` calls + * `injector.destroy()`, firing the `DestroyRef` cleanup. Result signals are read + * after settling. Controllable inputs use Angular's reactive `{ params, query }` + * form driven by a signal. + * + * `knownGaps` is populated empirically from the run below. + */ +import { + EnvironmentInjector, + createEnvironmentInjector, + runInInjectionContext, + signal, +} from '@angular/core' +import { TestBed } from '@angular/core/testing' +import { + coalesce, + count, + createCollection, + createLiveQueryCollection, + createOptimisticAction, + eq, + gt, + sum, +} from '@tanstack/db' +import { + mockSyncCollectionOptions, + mockSyncCollectionOptionsNoInitialState, +} from '../../db/tests/utils' +import { injectLiveQuery } from '../src/index' +import { runSuite } from '../../db/tests/conformance/suite' +import type { + ConformanceResult, + ControllableHandle, + DeferredSourceHandle, + LiveQueryDriver, + LiveQueryHandle, + QueryBuild, + SourceHandle, +} from '../../db/tests/conformance/contract' + +let sourceSeq = 0 + +function writer(collection: any) { + return (type: `insert` | `update` | `delete`, value: T) => { + collection.utils.begin() + collection.utils.write({ type, value }) + collection.utils.commit() + } +} + +function makeSource( + initialData: ReadonlyArray, +): SourceHandle { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `conformance-angular-${sourceSeq++}`, + getKey: (r) => r.id, + initialData: [...initialData], + }), + ) + const write = writer(collection) + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + } +} + +function makeDeferredSource< + T extends { id: string }, +>(): DeferredSourceHandle { + const collection = createCollection( + mockSyncCollectionOptionsNoInitialState({ + id: `conformance-angular-${sourceSeq++}`, + getKey: (r) => r.id, + }), + ) + collection.startSyncImmediate() + const write = writer(collection) + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + emit: (rows) => { + collection.utils.begin() + rows.forEach((value) => collection.utils.write({ type: `insert`, value })) + collection.utils.commit() + }, + markReady: () => collection.utils.markReady(), + } +} + +function makePrecreated(build: QueryBuild, opts?: { startSync?: boolean }) { + const collection = createLiveQueryCollection({ + query: build as any, + startSync: opts?.startSync ?? true, + }) + return { collection } +} + +function makeErrorSource() { + const collection = createCollection<{ id: string }>({ + id: `conformance-angular-err-${sourceSeq++}`, + getKey: (r) => r.id, + startSync: false, + sync: { + sync: () => { + throw new Error(`conformance: sync failure`) + }, + }, + }) + try { + collection.startSyncImmediate() + } catch { + // expected: engine catches the sync error and sets status to `error` + } + return { collection } +} + +async function settle() { + await new Promise((resolve) => setTimeout(resolve, 0)) + await new Promise((resolve) => setTimeout(resolve, 50)) +} + +function makeHandle(result: any, destroy: () => void): LiveQueryHandle { + return { + current(): ConformanceResult { + return { + data: result.data(), + state: result.state(), + status: result.status(), + isReady: Boolean(result.isReady()), + isError: Boolean(result.isError()), + // angular-db exposes no `isEnabled`; derive it from status (status-derived). + isEnabled: result.status() !== `disabled`, + } + }, + flush: settle, + async apply(fn: () => void) { + fn() + await settle() + }, + unmount() { + destroy() + }, + } +} + +function inCtx(fn: () => any): { result: any; destroy: () => void } { + const parent = TestBed.inject(EnvironmentInjector) + const injector = createEnvironmentInjector([], parent) + let result: any + runInInjectionContext(injector, () => { + result = fn() + }) + return { result, destroy: () => injector.destroy() } +} + +function mount(build: QueryBuild) { + const { result, destroy } = inCtx(() => injectLiveQuery(build as any)) + return makeHandle(result, destroy) +} + +function mountCollection(collection: any) { + const { result, destroy } = inCtx(() => injectLiveQuery(collection)) + return makeHandle(result, destroy) +} + +function mountConfig(build: QueryBuild) { + const { result, destroy } = inCtx(() => injectLiveQuery({ query: build })) + return makeHandle(result, destroy) +} + +function mountDisabled() { + const { result, destroy } = inCtx(() => injectLiveQuery(() => null)) + return makeHandle(result, destroy) +} + +function mountControllable

                ( + build: (q: any, param: P) => any, + initial: P, +): ControllableHandle

                { + const param = signal

                (initial) + const { result, destroy } = inCtx(() => + injectLiveQuery({ + params: () => ({ value: param() }), + query: ({ params, q }: any) => build(q, params.value), + }), + ) + const handle = makeHandle(result, destroy) + return { + ...handle, + async setParam(next: P) { + param.set(next) + await settle() + }, + } +} + +const angularDriver: LiveQueryDriver = { + name: `angular`, + ops: { eq, gt, count, sum, coalesce, createOptimisticAction }, + makeSource, + makeDeferredSource, + makePrecreated, + makeErrorSource, + mount, + mountControllable, + mountCollection, + mountConfig, + mountDisabled, + knownGaps: [], + features: { serverSnapshot: false, suspense: false }, +} + +runSuite(angularDriver) diff --git a/packages/angular-db/tests/inject-live-query.test.ts b/packages/angular-db/tests/inject-live-query.test.ts index 81fbb12a92..a2dceb0591 100644 --- a/packages/angular-db/tests/inject-live-query.test.ts +++ b/packages/angular-db/tests/inject-live-query.test.ts @@ -80,14 +80,25 @@ function createMockCollection( } let status: CollectionStatus = initialStatus + let stateRevision = 0 const subs = new Set<(changes: Array) => void>() const readySubs = new Set<() => void>() + const statusSubs = new Set<(event: any) => void>() const id = `mock-col-` + Math.random().toString(36).slice(2) + // Mirrors the real collection contract: committed changes advance the + // state revision before they are emitted. const notify = (changes: Array = []) => { + if (changes.length > 0) stateRevision++ for (const cb of subs) cb(changes) } + const emitStatusChange = (previousStatus: CollectionStatus) => { + for (const cb of statusSubs) { + cb({ type: `status:change`, previousStatus, status }) + } + } + const notifyReady = () => { for (const cb of readySubs) cb() } @@ -97,6 +108,13 @@ function createMockCollection( get status() { return status }, + get _stateRevision() { + return stateRevision + }, + on: (event: string, cb: (e: any) => void) => { + if (event === `status:change`) statusSubs.add(cb) + return () => statusSubs.delete(cb) + }, entries: () => Array.from(map.entries()), values: () => Array.from(map.values()), get: (key: K) => map.get(key), @@ -104,6 +122,8 @@ function createMockCollection( size: () => map.size, subscribeChanges: (cb: (changes: Array) => void) => { subs.add(cb) + // Real collections start sync when the first subscriber attaches. + api.startSyncImmediate() return { unsubscribe: () => subs.delete(cb), } @@ -118,26 +138,33 @@ function createMockCollection( }, preload: () => Promise.resolve(), startSyncImmediate: () => { - const wasNotReady = status !== `ready` + const previousStatus = status if (status === `idle`) { status = `ready` - } - if (wasNotReady && status === `ready`) { + emitStatusChange(previousStatus) setTimeout(notifyReady, 0) } }, __setStatus: (s: CollectionStatus) => { + const previousStatus = status const wasNotReady = status !== `ready` status = s - notify([]) + emitStatusChange(previousStatus) if (wasNotReady && status === `ready`) { setTimeout(notifyReady, 0) } }, __replaceAll: (rows: Array>) => { + const changes: Array = [] + for (const [key, value] of map.entries()) { + changes.push({ type: `delete`, key, value }) + } map.clear() - for (const r of rows) map.set(r.id, r) - notify([]) + for (const r of rows) { + map.set(r.id, r) + changes.push({ type: `insert`, key: r.id, value: r }) + } + notify(changes) }, __upsert: (row: T & Record<`id`, K>) => { const isUpdate = map.has(row.id) diff --git a/packages/browser-db-sqlite-persistence/CHANGELOG.md b/packages/browser-db-sqlite-persistence/CHANGELOG.md index fafcb0bb1f..6a532f4709 100644 --- a/packages/browser-db-sqlite-persistence/CHANGELOG.md +++ b/packages/browser-db-sqlite-persistence/CHANGELOG.md @@ -1,5 +1,154 @@ # @tanstack/browser-db-sqlite-persistence +## 0.2.21 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db-sqlite-persistence-core@0.2.21 + +## 0.2.20 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.20 + +## 0.2.19 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.19 + +## 0.2.18 + +### Patch Changes + +- Updated dependencies [[`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db-sqlite-persistence-core@0.2.18 + +## 0.2.17 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.17 + +## 0.2.16 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.16 + +## 0.2.15 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.15 + +## 0.2.14 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.14 + +## 0.2.13 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db-sqlite-persistence-core@0.2.13 + +## 0.2.12 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.12 + +## 0.2.11 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.11 + +## 0.2.10 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.10 + +## 0.2.9 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.9 + +## 0.2.8 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.8 + +## 0.2.7 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.7 + +## 0.2.6 + +### Patch Changes + +- Updated dependencies [[`f7da776`](https://github.com/TanStack/db/commit/f7da77660b16cbfe30817fb5c938267d696c8d1c)]: + - @tanstack/db-sqlite-persistence-core@0.2.6 + +## 0.2.5 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.5 + +## 0.2.4 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.4 + +## 0.2.3 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.3 + +## 0.2.2 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.2 + +## 0.2.1 + +### Patch Changes + +- Use a safe `randomUUID` helper that falls back to `crypto.getRandomValues` when `crypto.randomUUID` is unavailable (non-secure browser contexts such as dev servers reached via a LAN IP over HTTP). Fixes #1541. ([#1593](https://github.com/TanStack/db/pull/1593)) + +- Updated dependencies [[`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db-sqlite-persistence-core@0.2.1 + ## 0.2.0 ### Minor Changes diff --git a/packages/browser-db-sqlite-persistence/README.md b/packages/browser-db-sqlite-persistence/README.md index b14c83b32f..84175bf7cd 100644 --- a/packages/browser-db-sqlite-persistence/README.md +++ b/packages/browser-db-sqlite-persistence/README.md @@ -1,6 +1,9 @@ # @tanstack/browser-db-sqlite-persistence -Thin browser SQLite persistence for TanStack DB using `wa-sqlite` + OPFS. +Browser SQLite persistence for TanStack DB using `wa-sqlite` + OPFS. + +Supports both single-tab (default) and multi-tab usage. Multi-tab coordination +is opt-in by passing a `BrowserCollectionCoordinator`. ## Public API @@ -8,7 +11,12 @@ Thin browser SQLite persistence for TanStack DB using `wa-sqlite` + OPFS. - `openBrowserWASQLiteOPFSDatabase(...)` - `persistedCollectionOptions(...)` (re-exported from core) -## Quick start +## Quick start (single-tab) + +By default, `createBrowserWASQLitePersistence` uses `SingleProcessCoordinator` +semantics — no leader election, no `BroadcastChannel`, no Web Locks. This is +the right choice when your app is only ever open in one tab at a time, or when +each tab uses its own database. ```ts import { createCollection } from '@tanstack/db' @@ -42,13 +50,60 @@ export const todosCollection = createCollection( ) ``` +## Multi-tab usage + +To safely share a single OPFS database across multiple tabs of the same +origin, pass a `BrowserCollectionCoordinator` via the `coordinator` option. +The coordinator uses the Web Locks API to elect a leader tab, and +`BroadcastChannel` to fan out committed transactions to follower tabs. +Follower tabs forward writes to the leader via RPC over the channel. + +```ts +import { createCollection } from '@tanstack/db' +import { + BrowserCollectionCoordinator, + createBrowserWASQLitePersistence, + openBrowserWASQLiteOPFSDatabase, + persistedCollectionOptions, +} from '@tanstack/browser-db-sqlite-persistence' + +const database = await openBrowserWASQLiteOPFSDatabase({ + databaseName: `tanstack-db.sqlite`, +}) + +const coordinator = new BrowserCollectionCoordinator({ + dbName: `tanstack-db`, +}) + +const persistence = createBrowserWASQLitePersistence({ + database, + coordinator, +}) + +export const todosCollection = createCollection( + persistedCollectionOptions({ + id: `todos`, + getKey: (todo) => todo.id, + persistence, + schemaVersion: 1, + }), +) + +// On teardown: +// coordinator.dispose() +// await database.close?.() +``` + +See [`examples/react/offline-transactions`](../../examples/react/offline-transactions/src/db/persisted-todos.ts) +for a full multi-tab example. + ## Notes -- This package is Phase 7 single-tab browser wiring: it uses - `SingleProcessCoordinator` semantics by default. - `openBrowserWASQLiteOPFSDatabase(...)` starts a dedicated Web Worker and routes SQL operations through it. OPFS sync access handle APIs are used in that worker context. -- Single-tab mode does not require BroadcastChannel or Web Locks for +- Single-tab mode does not require `BroadcastChannel` or Web Locks for correctness. +- Multi-tab mode requires `BroadcastChannel` and the Web Locks API; both are + available in all modern browsers. - OPFS capability failures are surfaced as `PersistenceUnavailableError`. diff --git a/packages/browser-db-sqlite-persistence/package.json b/packages/browser-db-sqlite-persistence/package.json index 8e94ebedf9..4d2d04c025 100644 --- a/packages/browser-db-sqlite-persistence/package.json +++ b/packages/browser-db-sqlite-persistence/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/browser-db-sqlite-persistence", - "version": "0.2.0", + "version": "0.2.21", "description": "Browser wa-sqlite persisted collection adapter for TanStack DB", "author": "TanStack Team", "license": "MIT", diff --git a/packages/browser-db-sqlite-persistence/src/browser-coordinator.ts b/packages/browser-db-sqlite-persistence/src/browser-coordinator.ts index 5b518ea387..1babddc5a7 100644 --- a/packages/browser-db-sqlite-persistence/src/browser-coordinator.ts +++ b/packages/browser-db-sqlite-persistence/src/browser-coordinator.ts @@ -1,3 +1,4 @@ +import { safeRandomUUID } from '@tanstack/db-sqlite-persistence-core' import type { ApplyLocalMutationsResponse, PersistedCollectionCoordinator, @@ -118,7 +119,7 @@ export type BrowserCollectionCoordinatorOptions = { // --------------------------------------------------------------------------- export class BrowserCollectionCoordinator implements PersistedCollectionCoordinator { - private readonly nodeId = crypto.randomUUID() + private readonly nodeId = safeRandomUUID() private readonly dbName: string private adapter: AdapterWithPullSince | null private readonly channel: BroadcastChannel @@ -205,7 +206,7 @@ export class BrowserCollectionCoordinator implements PersistedCollectionCoordina error?: string }>(collectionId, { type: `rpc:ensureRemoteSubset:req`, - rpcId: crypto.randomUUID(), + rpcId: safeRandomUUID(), options, }) @@ -233,7 +234,7 @@ export class BrowserCollectionCoordinator implements PersistedCollectionCoordina error?: string }>(collectionId, { type: `rpc:ensurePersistedIndex:req`, - rpcId: crypto.randomUUID(), + rpcId: safeRandomUUID(), signature, spec, }) @@ -252,16 +253,16 @@ export class BrowserCollectionCoordinator implements PersistedCollectionCoordina if (this.isLeader(collectionId)) { return this.handleApplyLocalMutations(collectionId, { type: `rpc:applyLocalMutations:req`, - rpcId: crypto.randomUUID(), - envelopeId: crypto.randomUUID(), + rpcId: safeRandomUUID(), + envelopeId: safeRandomUUID(), mutations, }) } return this.sendRPC(collectionId, { type: `rpc:applyLocalMutations:req`, - rpcId: crypto.randomUUID(), - envelopeId: crypto.randomUUID(), + rpcId: safeRandomUUID(), + envelopeId: safeRandomUUID(), mutations, }) } @@ -273,14 +274,14 @@ export class BrowserCollectionCoordinator implements PersistedCollectionCoordina if (this.isLeader(collectionId)) { return this.handlePullSince(collectionId, { type: `rpc:pullSince:req`, - rpcId: crypto.randomUUID(), + rpcId: safeRandomUUID(), fromRowVersion, }) } return this.sendRPC(collectionId, { type: `rpc:pullSince:req`, - rpcId: crypto.randomUUID(), + rpcId: safeRandomUUID(), fromRowVersion, }) } @@ -663,7 +664,7 @@ export class BrowserCollectionCoordinator implements PersistedCollectionCoordina // Build and apply the persisted transaction const tx = { - txId: crypto.randomUUID(), + txId: safeRandomUUID(), term, seq, rowVersion, diff --git a/packages/capacitor-db-sqlite-persistence/CHANGELOG.md b/packages/capacitor-db-sqlite-persistence/CHANGELOG.md index 127512e3aa..c0f975415a 100644 --- a/packages/capacitor-db-sqlite-persistence/CHANGELOG.md +++ b/packages/capacitor-db-sqlite-persistence/CHANGELOG.md @@ -1,5 +1,152 @@ # @tanstack/capacitor-db-sqlite-persistence +## 0.2.21 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db-sqlite-persistence-core@0.2.21 + +## 0.2.20 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.20 + +## 0.2.19 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.19 + +## 0.2.18 + +### Patch Changes + +- Updated dependencies [[`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db-sqlite-persistence-core@0.2.18 + +## 0.2.17 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.17 + +## 0.2.16 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.16 + +## 0.2.15 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.15 + +## 0.2.14 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.14 + +## 0.2.13 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db-sqlite-persistence-core@0.2.13 + +## 0.2.12 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.12 + +## 0.2.11 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.11 + +## 0.2.10 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.10 + +## 0.2.9 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.9 + +## 0.2.8 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.8 + +## 0.2.7 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.7 + +## 0.2.6 + +### Patch Changes + +- Updated dependencies [[`f7da776`](https://github.com/TanStack/db/commit/f7da77660b16cbfe30817fb5c938267d696c8d1c)]: + - @tanstack/db-sqlite-persistence-core@0.2.6 + +## 0.2.5 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.5 + +## 0.2.4 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.4 + +## 0.2.3 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.3 + +## 0.2.2 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.2 + +## 0.2.1 + +### Patch Changes + +- Updated dependencies [[`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db-sqlite-persistence-core@0.2.1 + ## 0.2.0 ### Minor Changes diff --git a/packages/capacitor-db-sqlite-persistence/e2e/app/CHANGELOG.md b/packages/capacitor-db-sqlite-persistence/e2e/app/CHANGELOG.md index 118201f804..d6c85b91cd 100644 --- a/packages/capacitor-db-sqlite-persistence/e2e/app/CHANGELOG.md +++ b/packages/capacitor-db-sqlite-persistence/e2e/app/CHANGELOG.md @@ -1,5 +1,173 @@ # @tanstack/capacitor-db-sqlite-persistence-e2e-app +## 0.0.33 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + - @tanstack/capacitor-db-sqlite-persistence@0.2.21 + +## 0.0.32 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + - @tanstack/capacitor-db-sqlite-persistence@0.2.20 + +## 0.0.31 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + - @tanstack/capacitor-db-sqlite-persistence@0.2.19 + +## 0.0.30 + +### Patch Changes + +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + - @tanstack/capacitor-db-sqlite-persistence@0.2.18 + +## 0.0.29 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + - @tanstack/capacitor-db-sqlite-persistence@0.2.17 + +## 0.0.28 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + - @tanstack/capacitor-db-sqlite-persistence@0.2.16 + +## 0.0.27 + +### Patch Changes + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + - @tanstack/capacitor-db-sqlite-persistence@0.2.15 + +## 0.0.26 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + - @tanstack/capacitor-db-sqlite-persistence@0.2.14 + +## 0.0.25 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + - @tanstack/capacitor-db-sqlite-persistence@0.2.13 + +## 0.0.24 + +### Patch Changes + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + - @tanstack/capacitor-db-sqlite-persistence@0.2.12 + +## 0.0.23 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + - @tanstack/capacitor-db-sqlite-persistence@0.2.11 + +## 0.0.22 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + - @tanstack/capacitor-db-sqlite-persistence@0.2.10 + +## 0.0.21 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + - @tanstack/capacitor-db-sqlite-persistence@0.2.9 + +## 0.0.20 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + - @tanstack/capacitor-db-sqlite-persistence@0.2.8 + +## 0.0.19 + +### Patch Changes + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + - @tanstack/capacitor-db-sqlite-persistence@0.2.7 + +## 0.0.18 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + - @tanstack/capacitor-db-sqlite-persistence@0.2.6 + +## 0.0.17 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + - @tanstack/capacitor-db-sqlite-persistence@0.2.5 + +## 0.0.16 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + - @tanstack/capacitor-db-sqlite-persistence@0.2.4 + +## 0.0.15 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + - @tanstack/capacitor-db-sqlite-persistence@0.2.3 + +## 0.0.14 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + - @tanstack/capacitor-db-sqlite-persistence@0.2.2 + +## 0.0.13 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + - @tanstack/capacitor-db-sqlite-persistence@0.2.1 + ## 0.0.12 ### Patch Changes diff --git a/packages/capacitor-db-sqlite-persistence/e2e/app/package.json b/packages/capacitor-db-sqlite-persistence/e2e/app/package.json index 7c8cacdf76..d51e9dddfa 100644 --- a/packages/capacitor-db-sqlite-persistence/e2e/app/package.json +++ b/packages/capacitor-db-sqlite-persistence/e2e/app/package.json @@ -1,7 +1,7 @@ { "name": "@tanstack/capacitor-db-sqlite-persistence-e2e-app", "private": true, - "version": "0.0.12", + "version": "0.0.33", "type": "module", "scripts": { "build": "vite build", diff --git a/packages/capacitor-db-sqlite-persistence/package.json b/packages/capacitor-db-sqlite-persistence/package.json index 8ea6d338f6..cdd6556862 100644 --- a/packages/capacitor-db-sqlite-persistence/package.json +++ b/packages/capacitor-db-sqlite-persistence/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/capacitor-db-sqlite-persistence", - "version": "0.2.0", + "version": "0.2.21", "description": "Capacitor SQLite persisted collection adapter for TanStack DB", "author": "TanStack Team", "license": "MIT", diff --git a/packages/cloudflare-durable-objects-db-sqlite-persistence/CHANGELOG.md b/packages/cloudflare-durable-objects-db-sqlite-persistence/CHANGELOG.md index 87f4d18c83..e51b1ceee5 100644 --- a/packages/cloudflare-durable-objects-db-sqlite-persistence/CHANGELOG.md +++ b/packages/cloudflare-durable-objects-db-sqlite-persistence/CHANGELOG.md @@ -1,5 +1,152 @@ # @tanstack/cloudflare-durable-objects-db-sqlite-persistence +## 0.2.21 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db-sqlite-persistence-core@0.2.21 + +## 0.2.20 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.20 + +## 0.2.19 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.19 + +## 0.2.18 + +### Patch Changes + +- Updated dependencies [[`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db-sqlite-persistence-core@0.2.18 + +## 0.2.17 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.17 + +## 0.2.16 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.16 + +## 0.2.15 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.15 + +## 0.2.14 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.14 + +## 0.2.13 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db-sqlite-persistence-core@0.2.13 + +## 0.2.12 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.12 + +## 0.2.11 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.11 + +## 0.2.10 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.10 + +## 0.2.9 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.9 + +## 0.2.8 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.8 + +## 0.2.7 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.7 + +## 0.2.6 + +### Patch Changes + +- Updated dependencies [[`f7da776`](https://github.com/TanStack/db/commit/f7da77660b16cbfe30817fb5c938267d696c8d1c)]: + - @tanstack/db-sqlite-persistence-core@0.2.6 + +## 0.2.5 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.5 + +## 0.2.4 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.4 + +## 0.2.3 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.3 + +## 0.2.2 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.2 + +## 0.2.1 + +### Patch Changes + +- Updated dependencies [[`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db-sqlite-persistence-core@0.2.1 + ## 0.2.0 ### Minor Changes diff --git a/packages/cloudflare-durable-objects-db-sqlite-persistence/package.json b/packages/cloudflare-durable-objects-db-sqlite-persistence/package.json index 257183d06e..254672a1f1 100644 --- a/packages/cloudflare-durable-objects-db-sqlite-persistence/package.json +++ b/packages/cloudflare-durable-objects-db-sqlite-persistence/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/cloudflare-durable-objects-db-sqlite-persistence", - "version": "0.2.0", + "version": "0.2.21", "description": "Cloudflare Durable Object SQLite persisted collection adapter for TanStack DB", "author": "TanStack Team", "license": "MIT", diff --git a/packages/db-ivm/CHANGELOG.md b/packages/db-ivm/CHANGELOG.md index 1eb5a34491..80d93fe76a 100644 --- a/packages/db-ivm/CHANGELOG.md +++ b/packages/db-ivm/CHANGELOG.md @@ -1,5 +1,55 @@ # @tanstack/db-ivm +## 0.1.20 + +### Patch Changes + +- Fix on-demand load settlement, ordered pagination, and replay to preserve coherent results across cancellation, failure, cleanup, and restart. Preserve subset results and ownership across Electric, PowerSync, Query, and SQLite persistence adapters. Correct live-query grouping, include projections, value identity, and indexed comparisons. D2 hashing now rejects structural cycles and excessive traversal depth or work with an explicit error; Collection handles retain object-reference identity without traversing their mutable contents. ([#1797](https://github.com/TanStack/db/pull/1797)) + + Remove the unused public subset-algebra helpers: `isWhereSubset`, `unionWherePredicates`, `minusWherePredicates`, `isOrderBySubset`, `isLimitSubset`, `isOffsetLimitSubset`, `isPredicateSubset`, and `isLoadSubsetRequestSubsumedBy`. Apps that import these helpers must remove those imports; normal queries and adapters are unaffected. `DeduplicatedLoadSubset` remains available and shares only exact demand identities. + + Reject compiled Collection-valued includes as `fn.select()` inputs, including nested descendants, before invoking the callback. Use `toArray()` or `materialize()` in the upstream `.select()` for child-value calculations. To keep live child Collections, use expression `.select()` or perform parent-only functional work before adding the includes. Ordinary Collection-valued includes remain supported. + + Remove proxy `DEBUG` logging and automatic index timing statistics to avoid diagnostic work on reads and writes. Remove `getStats()` and `IndexStats`; use `index.keyCount` for the current entry count, and instrument index methods externally when profiling. Custom index subclasses must remove calls to the retired `trackLookup()` and `updateTimestamp()` helpers. + + Fix mutation drafts so Map/Set `forEach` calls each callback once and read-only iteration reports no changes. Track nested Map `for...of` edits through `collection.update()`. Keep Set entries in place during nested edits, preserving live iteration and usable `has`, `delete`, and `add` handles without duplicate entries or repeated visits. Reverting one entry no longer discards another entry's pending changes. + + Preserve shared values within a row's draft. Newly supplied objects use normal shared references during the update callback, including Map values and Set members; editing through either handle changes the same new object. Copy completed changes at callback return so later caller edits cannot mutate stored data. Existing collection values remain isolated. Copy a new caller-owned value before insertion if it must remain untouched during the callback. + + Keep rejected local-storage mutations out of later successful saves. Stage insert, update, delete, and manual transaction writes before persisting, then promote the shared cache only after storage succeeds. + + Deliver live-query observer publications to peer listeners even when one listener throws. Preserve queued publication order and stop delivery on disposal; report the first listener failure after delivery. + + Wait for PowerSync demands admitted during tracking finalization before reporting them loaded. Preserve untouched object and array key identity when they contain `NaN`. Retire every failed ordered acquisition on explicit retry, while retaining a full-source demand repaired by replay. + + After authoritative ordered recovery succeeds, retire settled page and tie demands so later truncate replay fetches only the full-source replacement. Keep unfinished requests observed until settlement and preserve independent query ownership. + + Remove the live-query `utils.getRunCount()` diagnostic and its runtime counter. Apps that call this diagnostic must remove those calls. Query scheduling and results are unchanged. + + Remove test-only index inspection getters (`indexedKeysSet`, `valueMapData`, `orderedEntriesArray`, and `orderedEntriesArrayReversed`) and unused scheduler diagnostics. `ReverseIndex` now exposes only the `IndexReader` lookup, range, and forward traversal surface returned by `findIndexForField`; mutate the original index instead. Export `IndexReader` for callers that name this return type. Remove the unused subscription `releaseLoadSubset()` method; request owners use the release callback supplied by `onLoadSubsetResult`. Ordinary query and adapter APIs remain unchanged by these removals. + + Remove unused internal helpers and the unused public error classes `WhereClauseConversionError`, `SubscriptionNotFoundError`, and `AggregateNotSupportedError`. These classes have no remaining runtime throw sites; remove any imports of them. Keep the existing query and index behavior and exercise identity/evaluation tests through the production entry points. + + Ensure failed mutations roll back even when their rejection value cannot be converted to a string. Preserve ordinary Error instances; report unprintable rejection values as `Unknown error`. + + Make reentrant effect disposal share the active cleanup result, including calls from abort listeners or source release callbacks. A release failure reaches every waiting disposer while each source still receives one release attempt. + + Reject starting or preloading a collection from inside its active cleanup callbacks with a clear `CollectionStateError`. Nested cleanup cannot admit replacement work that the old teardown would discard. Restart after cleanup completes, or from its final `cleaned-up` status event, remains supported. + + Prevent older page or tie-boundary completions from clearing a newer full-source failure or starting redundant loading. Failed window moves retain their settled public snapshot; an explicit retry releases the failed acquisition once and publishes the completed replacement. + + Restrict direct subscription `requestLimitedSnapshot()` cursor inputs to one order term and one `minValues` entry. Composite and partial-composite cursor inputs now throw before local delivery or adapter work. Use normal live-query ordering and window APIs for multi-column pagination; those remain supported through prefix-and-tie loading. Existing adapters need no changes. + + Treat `LoadSubsetOptions` and their nested request data as immutable from submission onward. Core no longer copies expression trees or mutable constant payloads at the sync and deduplication boundaries. Create a new Date, byte array, membership array, or options object when changing a demand instead of mutating submitted data. Adapters must also leave request data unchanged. Use stable data properties rather than stateful getters. `AbortSignal` cancellation and subscription release remain live. + + Replace replay acquisitions sequentially: release the prior physical lease before starting its replacement. The logical demand and last complete public result remain retained. A failed release prevents replacement startup; failed startup leaves demand available for a later authoritative replay. Custom adapters must support a release/load gap and preserve resources still held by other owners; a sole underlying resource may stop and restart. + +## 0.1.19 + +### Patch Changes + +- Rebuild correlated include materialization as one D2 graph, fixing stale or missing nested results across route changes, batching, lazy loading, optimistic updates, and layered queries. Add canonical structural relation keys, abortable subset demand, and coherent publication for Collection-valued includes. Dispose delayed PowerSync subset hooks after cleanup, and prevent released Query Collection cache results from reaching the collection. ([#1740](https://github.com/TanStack/db/pull/1740)) + ## 0.1.18 ### Patch Changes diff --git a/packages/db-ivm/package.json b/packages/db-ivm/package.json index 4bd57398ce..812c2f3b66 100644 --- a/packages/db-ivm/package.json +++ b/packages/db-ivm/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/db-ivm", - "version": "0.1.18", + "version": "0.1.20", "description": "Incremental View Maintenance for TanStack DB based on Differential Dataflow", "author": "Sam Willis", "license": "MIT", @@ -22,7 +22,8 @@ "build": "vite build", "dev": "vite build --watch", "lint": "eslint . --fix", - "test": "vitest --run" + "test": "vitest --run", + "bench:hash": "vitest bench --run tests/hash.bench.ts --coverage.enabled=false" }, "type": "module", "main": "dist/cjs/index.cjs", diff --git a/packages/db-ivm/src/hashing/hash.ts b/packages/db-ivm/src/hashing/hash.ts index 813e4ed35c..51c33b43f1 100644 --- a/packages/db-ivm/src/hashing/hash.ts +++ b/packages/db-ivm/src/hashing/hash.ts @@ -1,4 +1,4 @@ -import { MurmurHashStream, randomHash } from './murmur.js' +import { MurmurHashStream, getSymbolIdentity, randomHash } from './murmur.js' import type { Hasher } from './murmur.js' /* @@ -19,6 +19,10 @@ const MAP_MARKER = randomHash() const SET_MARKER = randomHash() const UINT8ARRAY_MARKER = randomHash() const TEMPORAL_MARKER = randomHash() +// Bound structural recursion and value visits. Shared acyclic subtrees are +// cached; cycles are rejected rather than given context-dependent hashes. +const MAX_STRUCTURAL_HASH_WORK = 1_000_000 +const MAX_STRUCTURAL_HASH_DEPTH = 768 const temporalTypes = new Set([ `Temporal.Duration`, @@ -48,63 +52,65 @@ const UINT8ARRAY_CONTENT_HASH_THRESHOLD = 128 const hashCache = new WeakMap() +/** @internal Register a mutable handle before it enters a structural value. */ +export function registerOpaqueHash(value: object): void { + cachedReferenceHash(value) +} + +type HashContext = { + activeObjects: Set + work: number + pendingHashes: Map +} + export function hash(input: any): number { const hasher = new MurmurHashStream() updateHasher(hasher, input) return hasher.digest() } -function hashObject(input: object): number { - const cachedHash = hashCache.get(input) - if (cachedHash !== undefined) { - return cachedHash +function hashObject(input: object, context: HashContext): number { + if (context.activeObjects.size >= MAX_STRUCTURAL_HASH_DEPTH) { + throw new RangeError( + `Value is too complex to hash safely: structural depth`, + ) } + context.activeObjects.add(input) + let valueHash: number | undefined - if (input instanceof Date) { - valueHash = hashDate(input) - } else if ( - // Check if input is a Uint8Array or Buffer - (typeof Buffer !== `undefined` && input instanceof Buffer) || - input instanceof Uint8Array - ) { - // For small Uint8Arrays/Buffers (e.g., ULIDs, UUIDs), hash by content - // to enable proper equality comparisons. For large arrays, hash by reference - // to avoid performance costs. - if (input.byteLength <= UINT8ARRAY_CONTENT_HASH_THRESHOLD) { + try { + if (input instanceof Date) { + valueHash = hashDate(input) + } else if (isBinaryValue(input)) { valueHash = hashUint8Array(input) + } else if (isTemporal(input)) { + valueHash = hashTemporal(input) } else { - // Deeply hashing large arrays would be too costly - // so we track them by reference and cache them in a weak map - return cachedReferenceHash(input) - } - } else if (input instanceof File) { - // Files are always hashed by reference due to their potentially large size - return cachedReferenceHash(input) - } else if (isTemporal(input)) { - valueHash = hashTemporal(input) - } else { - let plainObjectInput = input - let marker = OBJECT_MARKER - - if (input instanceof Array) { - marker = ARRAY_MARKER - } + let plainObjectInput = input + let marker = OBJECT_MARKER - if (input instanceof Map) { - marker = MAP_MARKER - plainObjectInput = [...input.entries()] - } + if (input instanceof Array) { + marker = ARRAY_MARKER + } - if (input instanceof Set) { - marker = SET_MARKER - plainObjectInput = [...input.entries()] - } + if (input instanceof Map) { + marker = MAP_MARKER + plainObjectInput = [...input.entries()] + } - valueHash = hashPlainObject(plainObjectInput, marker) + if (input instanceof Set) { + marker = SET_MARKER + plainObjectInput = [...input.entries()] + } + + valueHash = hashPlainObject(plainObjectInput, marker, context) + } + } finally { + context.activeObjects.delete(input) } - hashCache.set(input, valueHash) + context.pendingHashes.set(input, valueHash) return valueHash } @@ -135,7 +141,11 @@ function hashTemporal(input: TemporalLike): number { return hasher.digest() } -function hashPlainObject(input: object, marker: number): number { +function hashPlainObject( + input: object, + marker: number, + context: HashContext, +): number { const hasher = new MurmurHashStream() // Mark the type of the input @@ -145,13 +155,28 @@ function hashPlainObject(input: object, marker: number): number { for (const key of keys) { hasher.update(KEY) hasher.update(key) - updateHasher(hasher, input[key as keyof typeof input]) + updateHasher(hasher, input[key as keyof typeof input], context) + } + const symbolKeys = Object.getOwnPropertySymbols(input) + .filter((key) => Object.prototype.propertyIsEnumerable.call(input, key)) + .sort((left, right) => getSymbolIdentity(left) - getSymbolIdentity(right)) + for (const key of symbolKeys) { + hasher.update(KEY) + hasher.update(key) + updateHasher(hasher, input[key as keyof typeof input], context) } return hasher.digest() } -function updateHasher(hasher: Hasher, input: unknown): void { +function updateHasher( + hasher: Hasher, + input: unknown, + context?: HashContext, +): void { + if (context && ++context.work > MAX_STRUCTURAL_HASH_WORK) { + throw new RangeError(`Value is too complex to hash safely: structural work`) + } if (input === null) { hasher.update(NULL) return @@ -173,7 +198,7 @@ function updateHasher(hasher: Hasher, input: unknown): void { hasher.update(input) return case `object`: - hasher.update(getCachedHash(input)) + hasher.update(getCachedHash(input, context)) return case `function`: // Functions are assigned a globally unique ID @@ -187,12 +212,53 @@ function updateHasher(hasher: Hasher, input: unknown): void { } } -function getCachedHash(input: object): number { - let valueHash = hashCache.get(input) - if (valueHash === undefined) { - valueHash = hashObject(input) +function getCachedHash(input: object, context?: HashContext): number { + if (!context) { + const cached = hashCache.get(input) + if (cached !== undefined) return cached + if (isReferenceHashedObject(input)) return cachedReferenceHash(input) + + // Only an uncached structural root needs graph traversal state. Commit its + // cache entries after success so a failed traversal cannot poison retries. + context = { + activeObjects: new Set(), + work: 0, + pendingHashes: new Map(), + } + const result = hashObject(input, context) + for (const [object, valueHash] of context.pendingHashes) { + hashCache.set(object, valueHash) + } + return result } - return valueHash + + if (context.activeObjects.has(input)) { + throw new TypeError(`Cannot hash cyclic structural values`) + } + + // Opaque leaves cannot contain structural back-references. Resolve them + // before entering structural recursion, even when they have user properties. + if (isReferenceHashedObject(input)) return cachedReferenceHash(input) + + const valueHash = hashCache.get(input) ?? context.pendingHashes.get(input) + if (valueHash !== undefined) return valueHash + + return hashObject(input, context) +} + +function isReferenceHashedObject(input: object): boolean { + return ( + input instanceof File || + (isBinaryValue(input) && + input.byteLength > UINT8ARRAY_CONTENT_HASH_THRESHOLD) + ) +} + +function isBinaryValue(input: object): input is Uint8Array { + return ( + (typeof Buffer !== `undefined` && input instanceof Buffer) || + input instanceof Uint8Array + ) } let nextRefId = 1 diff --git a/packages/db-ivm/src/hashing/murmur.ts b/packages/db-ivm/src/hashing/murmur.ts index 9ce68be312..cd40030694 100644 --- a/packages/db-ivm/src/hashing/murmur.ts +++ b/packages/db-ivm/src/hashing/murmur.ts @@ -9,6 +9,49 @@ const BIG_INT_MARKER = randomHash() const NEG_BIG_INT_MARKER = randomHash() const SYMBOL_MARKER = randomHash() +type SymbolIdStore = { + get: (key: symbol) => number | undefined + set: (key: symbol, value: number) => unknown +} + +const symbolIds = createSymbolIdStore() +const registeredSymbolIds = new Map() +let nextSymbolId = 0 + +export function getSymbolIdentity(symbol: symbol): number { + const registeredKey = Symbol.keyFor(symbol) + if (registeredKey !== undefined) { + let id = registeredSymbolIds.get(registeredKey) + if (id === undefined) { + id = ++nextSymbolId + registeredSymbolIds.set(registeredKey, id) + } + return id + } + + let id = symbolIds.get(symbol) + if (id === undefined) { + id = ++nextSymbolId + symbolIds.set(symbol, id) + } + return id +} + +function createSymbolIdStore(): SymbolIdStore { + const weakIds = new WeakMap() as unknown as SymbolIdStore + const probe = Symbol() + + try { + weakIds.set(probe, 0) + if (weakIds.get(probe) === 0) return weakIds + } catch { + // Older runtimes reject symbols as weak keys. Retain them rather than + // merge distinct symbols and corrupt differential state. + } + + return new Map() +} + export type Hash = number export function randomHash() { @@ -67,16 +110,7 @@ export class MurmurHashStream implements Hasher { switch (typeof chunk) { case `symbol`: { this.update(SYMBOL_MARKER) - const description = chunk.description - if (!description) { - return - } - - for (let i = 0; i < description.length; i++) { - const code = description.charCodeAt(i) - this.writeByte(code & 0xff) - this.writeByte((code >>> 8) & 0xff) - } + this.update(getSymbolIdentity(chunk)) return } case `string`: diff --git a/packages/db-ivm/src/index.ts b/packages/db-ivm/src/index.ts index cae148b46e..441da1e484 100644 --- a/packages/db-ivm/src/index.ts +++ b/packages/db-ivm/src/index.ts @@ -3,3 +3,4 @@ export * from './multiset.js' export * from './operators/index.js' export * from './types.js' export { compareKeys, serializeValue } from './utils.js' +export { registerOpaqueHash } from './hashing/hash.js' diff --git a/packages/db-ivm/src/operators/groupBy.ts b/packages/db-ivm/src/operators/groupBy.ts index 9c2fe1e357..435156d25d 100644 --- a/packages/db-ivm/src/operators/groupBy.ts +++ b/packages/db-ivm/src/operators/groupBy.ts @@ -62,7 +62,7 @@ export function groupBy< stream: IStreamBuilder, ): IStreamBuilder> => { // Special key to store the original key object - const KEY_SENTINEL = `__original_key__` + const KEY_SENTINEL = Symbol(`original_group_key`) // First map to extract keys and pre-aggregate values const withKeysAndValues = stream.pipe( @@ -71,7 +71,7 @@ export function groupBy< const keyString = serializeValue(key) // Create values object with pre-aggregated values - const values: Record = {} + const values: Record = {} // Store the original key object values[KEY_SENTINEL] = key @@ -81,7 +81,10 @@ export function groupBy< values[name] = aggregate.preMap(data) } - return [keyString, values] as KeyValue> + return [keyString, values] as KeyValue< + string, + Record + > }), ) @@ -99,7 +102,7 @@ export function groupBy< return [] } - const result: Record = {} + const result: Record = {} // Get the original key from first value in group const originalKey = values[0]?.[0]?.[KEY_SENTINEL] diff --git a/packages/db-ivm/src/operators/orderByBTree.ts b/packages/db-ivm/src/operators/orderByBTree.ts deleted file mode 100644 index db95b2deac..0000000000 --- a/packages/db-ivm/src/operators/orderByBTree.ts +++ /dev/null @@ -1,20 +0,0 @@ -import { orderByWithFractionalIndexBase } from './orderBy.js' -import { topKWithFractionalIndexBTree } from './topKWithFractionalIndexBTree.js' -import type { KeyValue } from '../types.js' -import type { OrderByOptions } from './orderBy.js' - -export function orderByWithFractionalIndexBTree< - T extends KeyValue, - Ve = unknown, ->( - valueExtractor: ( - value: T extends KeyValue ? V : never, - ) => Ve, - options?: OrderByOptions, -) { - return orderByWithFractionalIndexBase( - topKWithFractionalIndexBTree, - valueExtractor, - options, - ) -} diff --git a/packages/db-ivm/src/utils.ts b/packages/db-ivm/src/utils.ts index 70cfda48c6..d7569ba211 100644 --- a/packages/db-ivm/src/utils.ts +++ b/packages/db-ivm/src/utils.ts @@ -185,6 +185,14 @@ function range(start: number, end: number): Array { export function compareKeys(a: string | number, b: string | number): number { // Same type: compare directly if (typeof a === typeof b) { + if (typeof a === `number` && typeof b === `number`) { + const aIsNaN = Number.isNaN(a) + const bIsNaN = Number.isNaN(b) + if (aIsNaN || bIsNaN) { + if (aIsNaN && bIsNaN) return 0 + return aIsNaN ? 1 : -1 + } + } if (a < b) return -1 if (a > b) return 1 return 0 @@ -193,19 +201,214 @@ export function compareKeys(a: string | number, b: string | number): number { return typeof a === `string` ? -1 : 1 } +type CanonicalValue = + | readonly [`undefined`] + | readonly [`null`] + | readonly [`boolean`, boolean] + | readonly [`number`, number | `NaN` | `Infinity` | `-Infinity`] + | readonly [`bigint`, string] + | readonly [`string`, string] + | readonly [`date`, number | `Invalid`] + | readonly [`regexp`, string, string] + | readonly [`bytes`, Array] + | readonly [`array`, Array] + | readonly [`map`, Array] + | readonly [`set`, Array] + | readonly [`object`, Array] + /** - * Serializes a value for use as a key, handling BigInt and Date values that JSON.stringify cannot handle. - * Uses JSON.stringify with a replacer function to convert BigInt values to strings and Date values to ISO strings. - * This is used for creating string keys in groupBy operations. + * Serializes a supported query value into one canonical key. + * + * JSON's native encoding is not suitable for relation keys: it merges BigInt + * with strings when a replacer is used, merges NaN with null, drops undefined, + * and depends on object insertion order. Ordinary JSON values keep their + * established wire form. Values that need richer types use a reserved prefix + * plus a structural, type-tagged encoding. */ export function serializeValue(value: unknown): string { - return JSON.stringify(value, (_, val) => { - if (typeof val === 'bigint') { - return val.toString() + if (isJsonSafeStructuralValue(value, new Set())) { + return JSON.stringify(toStableJsonValue(value)) + } + + return `~${JSON.stringify(toCanonicalValue(value, new Set()))}` +} + +type JsonValue = + | null + | boolean + | number + | string + | Array + | { [key: string]: JsonValue } + +function isJsonSafeStructuralValue( + value: unknown, + ancestors: Set, +): boolean { + if (value === null) return true + + switch (typeof value) { + case `boolean`: + case `string`: + return true + case `number`: + return Number.isFinite(value) + case `undefined`: + case `bigint`: + case `symbol`: + case `function`: + return false + } + + return withAcyclicValue(value, ancestors, () => { + if ( + value instanceof Date || + value instanceof RegExp || + value instanceof Uint8Array || + value instanceof Map || + value instanceof Set + ) { + return false } - if (val instanceof Date) { - return val.toISOString() + + return Array.isArray(value) + ? value.every((item) => isJsonSafeStructuralValue(item, ancestors)) + : Object.keys(value).every((key) => + isJsonSafeStructuralValue( + (value as Record)[key], + ancestors, + ), + ) + }) +} + +function toStableJsonValue(value: unknown): JsonValue { + if ( + value === null || + typeof value === `boolean` || + typeof value === `number` || + typeof value === `string` + ) { + return value + } + + if (Array.isArray(value)) { + return value.map(toStableJsonValue) + } + + return Object.fromEntries( + Object.keys(value as object) + .sort() + .map((key) => [ + key, + toStableJsonValue((value as Record)[key]), + ]), + ) +} + +function toCanonicalValue( + value: unknown, + ancestors: Set, +): CanonicalValue { + if (value === undefined) return [`undefined`] + if (value === null) return [`null`] + + switch (typeof value) { + case `boolean`: + return [`boolean`, value] + case `number`: + if (Number.isNaN(value)) return [`number`, `NaN`] + if (value === Infinity) return [`number`, `Infinity`] + if (value === -Infinity) return [`number`, `-Infinity`] + return [`number`, value === 0 ? 0 : value] + case `bigint`: + return [`bigint`, value.toString()] + case `string`: + return [`string`, value] + case `symbol`: + case `function`: + throw new TypeError( + `Cannot serialize ${typeof value} as a structural relation key`, + ) + } + + return withAcyclicValue(value, ancestors, () => { + if (value instanceof Date) { + const timestamp = value.getTime() + return Number.isNaN(timestamp) + ? ([`date`, `Invalid`] as const) + : ([`date`, timestamp] as const) + } + + if (value instanceof RegExp) { + return [`regexp`, value.source, value.flags] + } + + if (value instanceof Uint8Array) { + return [`bytes`, Array.from(value)] + } + + if (Array.isArray(value)) { + return [`array`, value.map((item) => toCanonicalValue(item, ancestors))] + } + + if (value instanceof Map) { + const entries = [...value.entries()].map( + ([key, entryValue]) => + [ + toCanonicalValue(key, ancestors), + toCanonicalValue(entryValue, ancestors), + ] as const, + ) + entries.sort((left, right) => + compareSerializedValues(JSON.stringify(left), JSON.stringify(right)), + ) + return [`map`, entries] } - return val + + if (value instanceof Set) { + const entries = [...value].map((entry) => + toCanonicalValue(entry, ancestors), + ) + entries.sort((left, right) => + compareSerializedValues(JSON.stringify(left), JSON.stringify(right)), + ) + return [`set`, entries] + } + + const entries = Object.keys(value) + .sort() + .map( + (key) => + [ + key, + toCanonicalValue( + (value as Record)[key], + ancestors, + ), + ] as const, + ) + return [`object`, entries] }) } + +function compareSerializedValues(left: string, right: string): number { + return left < right ? -1 : left > right ? 1 : 0 +} + +function withAcyclicValue( + value: object, + ancestors: Set, + encode: () => T, +): T { + if (ancestors.has(value)) { + throw new TypeError(`Cannot serialize a cyclic structural relation key`) + } + + ancestors.add(value) + try { + return encode() + } finally { + ancestors.delete(value) + } +} diff --git a/packages/db-ivm/tests/hash-graph.property.test.ts b/packages/db-ivm/tests/hash-graph.property.test.ts new file mode 100644 index 0000000000..c04dc7574f --- /dev/null +++ b/packages/db-ivm/tests/hash-graph.property.test.ts @@ -0,0 +1,119 @@ +import { describe, expect, it } from 'vitest' +import { fc } from '@fast-check/vitest' +import { hash } from '../src/hashing/hash' + +// Kahn's algorithm checks the reachable graph without using the hasher's +// recursive active-path algorithm. Unreachable cycles do not affect the root. +function isAcyclic(edges: Array>): boolean { + const reachable = new Set([0]) + for (const node of reachable) { + for (const target of edges[node]!) reachable.add(target) + } + const incoming = new Map([...reachable].map((node) => [node, 0])) + for (const node of reachable) { + for (const target of edges[node]!) { + incoming.set(target, incoming.get(target)! + 1) + } + } + const ready = [...reachable].filter((node) => incoming.get(node) === 0) + for (const node of ready) { + for (const target of edges[node]!) { + const remaining = incoming.get(target)! - 1 + incoming.set(target, remaining) + if (remaining === 0) ready.push(target) + } + } + return ready.length === reachable.size +} + +const graphArbitrary = fc + .array(fc.array(fc.nat({ max: 5 }), { maxLength: 3 }), { + minLength: 1, + maxLength: 6, + }) + .map((edges) => + edges.map((targets) => targets.map((target) => target % edges.length)), + ) + +describe(`structural hash graph boundary`, () => { + it.each([ + `object`, + `array`, + `map-key`, + `map-value`, + `set`, + `symbol`, + ] as const)(`rejects a cycle through %s on every attempt`, (kind) => { + const record: Record = {} + const array: Array = [] + const map = new Map() + const set = new Set() + const input = + kind === `array` + ? array + : kind.startsWith(`map`) + ? map + : kind === `set` + ? set + : record + if (kind === `object`) record.self = input + if (kind === `symbol`) record[Symbol(`self`)] = input + if (kind === `array`) array.push(input) + if (kind === `map-key`) map.set(input, 1) + if (kind === `map-value`) map.set(1, input) + if (kind === `set`) set.add(input) + for (let attempt = 0; attempt < 2; attempt++) { + expect(() => hash(input)).toThrow(`Cannot hash cyclic structural values`) + } + }) + + for (const seed of [1657019, undefined]) { + it(`matches reachable graph cycles and shared DAGs (${seed ?? `random`})`, () => { + fc.assert( + fc.property(graphArbitrary, (edges) => { + const nodes = edges.map((_, value) => ({ + value, + children: [] as Array, + })) + edges.forEach((targets, index) => { + nodes[index]!.children = targets.map((target) => nodes[target]) + }) + if (!isAcyclic(edges)) { + expect(() => hash(nodes[0])).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(nodes[0])).toThrow( + `Cannot hash cyclic structural values`, + ) + return + } + // Unfold sharing into equal but distinct subtrees. Hash identity must + // depend on values, not whether the graph reused an object reference. + const unfold = (node: number): unknown => ({ + value: node, + children: edges[node]!.map(unfold), + }) + expect(hash(nodes[0])).toBe(hash(unfold(0))) + }), + { numRuns: 300, ...(seed === undefined ? {} : { seed }) }, + ) + }) + } + + it(`leaves completed siblings uncached after a cycle rejects the root`, () => { + let reads = 0 + const sibling = { + get value() { + return ++reads + }, + } + const root: Record = { a: sibling } + root.z = root + expect(() => hash(root)).toThrow(`Cannot hash cyclic structural values`) + expect(() => hash(root)).toThrow(`Cannot hash cyclic structural values`) + expect(reads).toBe(2) + delete root.z + expect(hash(root)).toBe(hash({ a: { value: 3 } })) + expect(reads).toBe(3) + }) +}) diff --git a/packages/db-ivm/tests/hash-work.test.ts b/packages/db-ivm/tests/hash-work.test.ts new file mode 100644 index 0000000000..e0054c652f --- /dev/null +++ b/packages/db-ivm/tests/hash-work.test.ts @@ -0,0 +1,126 @@ +import { describe, expect, it, vi } from 'vitest' +import { hash, registerOpaqueHash } from '../src/hashing/hash' + +function countTraversalAllocations(run: () => void): number { + let allocations = 0 + for (const name of [`Map`, `Set`, `WeakMap`] as const) { + vi.stubGlobal( + name, + new Proxy(globalThis[name], { + construct(target, args) { + allocations++ + return Reflect.construct(target, args) + }, + }), + ) + } + try { + run() + } finally { + vi.unstubAllGlobals() + } + return allocations +} + +describe(`hash traversal work`, () => { + it.each([`object`, `array`] as const)( + `does not let a rejected %s traversal subsidize its own retry`, + (kind) => { + const left = Array.from({ length: 500_001 }, () => 0) + const right = Array.from({ length: 500_001 }, () => 0) + const root = kind === `object` ? { left, right } : [left, right] + // Either child fits, but this fresh root exceeds the combined work cap. + // Keeping the completed left child's cache after failure lets retry pass. + for (let attempt = 0; attempt < 2; attempt++) { + expect(() => hash(root)).toThrow( + `Value is too complex to hash safely: structural work`, + ) + } + }, + ) + + it(`does not allocate traversal collections for primitive and cached inputs`, () => { + const cached = { id: 1, title: `cached` } + hash(cached) + const inputs = [null, undefined, false, 0, 1n, `row`, Symbol(`key`), cached] + expect( + countTraversalAllocations(() => { + for (const input of inputs) hash(input) + }), + ).toBe(0) + }) + + it(`measures traversal collections for fresh structural inputs`, () => { + expect(countTraversalAllocations(() => hash({ id: 1 }))).toBeGreaterThan(0) + }) + + it(`uses identity for registered handles without traversing mutable internals`, () => { + const first: Record = {} + const second: Record = {} + for (const value of [first, second]) { + value.self = value + Object.defineProperty(value, `state`, { + enumerable: true, + get() { + throw new Error(`must not read handle state`) + }, + }) + registerOpaqueHash(value) + } + const before = hash({ handle: first }) + first.changed = true + expect(hash({ handle: first })).toBe(before) + expect(hash({ handle: second })).not.toBe(before) + expect( + countTraversalAllocations(() => { + hash(first) + hash(second) + }), + ).toBe(0) + }) + + it(`visits each shared acyclic subtree once`, () => { + let reads = 0 + let root: object = { value: 1 } + for (let depth = 0; depth < 200; depth++) { + const child = root + root = { + get left() { + reads++ + return child + }, + get right() { + reads++ + return child + }, + } + } + const result = hash(root) + expect(reads).toBe(400) + expect(hash(root)).toBe(result) + expect(reads).toBe(400) + }) + + it(`bounds value visits without publishing partial structural caches`, () => { + let reads = 0 + const shared = {} + const root = { + a: { + get value() { + reads++ + return 1 + }, + }, + // Repeated references must still count as work, even when their hashes + // are cached; no expanded tree is needed to reach the bound. + z: Array.from({ length: 1_000_001 }, () => shared), + } + for (let attempt = 1; attempt <= 2; attempt++) { + expect(() => hash(root)).toThrow( + `Value is too complex to hash safely: structural work`, + ) + expect(reads).toBe(attempt) + } + expect(hash({ value: 1 })).toBe(hash({ value: 1 })) + }) +}) diff --git a/packages/db-ivm/tests/hash.bench.ts b/packages/db-ivm/tests/hash.bench.ts new file mode 100644 index 0000000000..d870d968c3 --- /dev/null +++ b/packages/db-ivm/tests/hash.bench.ts @@ -0,0 +1,23 @@ +import { bench, describe } from 'vitest' +import { hash } from '../src/hashing/hash' + +const cached = { id: 1, title: `row`, active: true } +hash(cached) +let result = 0 + +describe(`hash input paths`, () => { + bench(`primitive`, () => { + result ^= hash(42) + }) + bench(`cached row`, () => { + result ^= hash(cached) + }) + bench(`fresh row`, () => { + result ^= hash({ id: 1, title: `row`, active: true }) + }) +}) + +// Keep benchmark results observable without adding work inside each sample. +export function getHashBenchmarkResult(): number { + return result +} diff --git a/packages/db-ivm/tests/operators/groupBy.test.ts b/packages/db-ivm/tests/operators/groupBy.test.ts index fbe50fb33a..52b0fac653 100644 --- a/packages/db-ivm/tests/operators/groupBy.test.ts +++ b/packages/db-ivm/tests/operators/groupBy.test.ts @@ -12,6 +12,7 @@ import { sum, } from '../../src/operators/groupBy.js' import { output } from '../../src/operators/index.js' +import { serializeValue } from '../../src/utils.js' describe(`Operators`, () => { describe(`GroupBy operation`, () => { @@ -50,7 +51,7 @@ describe(`Operators`, () => { const expectedResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, }, @@ -59,7 +60,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"B"}`, + serializeValue({ category: `B` }), { category: `B`, }, @@ -108,7 +109,7 @@ describe(`Operators`, () => { const expectedResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { total: 30, category: `A`, @@ -118,7 +119,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"B"}`, + serializeValue({ category: `B` }), { total: 30, category: `B`, @@ -131,6 +132,39 @@ describe(`Operators`, () => { expect(result).toEqual(expectedResult) }) + test(`does not reserve an aggregate name for its original group key`, () => { + const graph = new D2() + const input = graph.newInput<{ category: string }>() + let latestMessage: MultiSet | undefined + + input.pipe( + groupBy((data) => ({ category: data.category }), { + __original_key__: count(), + }), + output((message) => { + latestMessage = message + }), + ) + graph.finalize() + input.sendData( + new MultiSet([ + [{ category: `A` }, 1], + [{ category: `A` }, 1], + ]), + ) + graph.run() + + expect(latestMessage?.getInner()).toEqual([ + [ + [ + serializeValue({ category: `A` }), + { category: `A`, __original_key__: 2 }, + ], + 1, + ], + ]) + }) + test(`with sum and count aggregates`, () => { const graph = new D2() const input = graph.newInput<{ @@ -177,7 +211,7 @@ describe(`Operators`, () => { const expectedResult = [ [ [ - `{"category":"A","region":"East"}`, + serializeValue({ category: `A`, region: `East` }), { total: 30, count: 2, @@ -189,7 +223,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"A","region":"West"}`, + serializeValue({ category: `A`, region: `West` }), { total: 30, count: 1, @@ -201,7 +235,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"B","region":"East"}`, + serializeValue({ category: `B`, region: `East` }), { total: 40, count: 1, @@ -228,7 +262,7 @@ describe(`Operators`, () => { const expectedAddResult = [ [ [ - `{"category":"A","region":"East"}`, + serializeValue({ category: `A`, region: `East` }), { category: `A`, region: `East`, @@ -240,7 +274,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"A","region":"East"}`, + serializeValue({ category: `A`, region: `East` }), { category: `A`, region: `East`, @@ -252,7 +286,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"B","region":"West"}`, + serializeValue({ category: `B`, region: `West` }), { category: `B`, region: `West`, @@ -277,7 +311,7 @@ describe(`Operators`, () => { const expectedDeleteResult = [ [ [ - `{"category":"A","region":"East"}`, + serializeValue({ category: `A`, region: `East` }), { category: `A`, region: `East`, @@ -289,7 +323,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"A","region":"East"}`, + serializeValue({ category: `A`, region: `East` }), { category: `A`, region: `East`, @@ -349,7 +383,7 @@ describe(`Operators`, () => { const expectedResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, countNotNull: 1, @@ -360,7 +394,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"B"}`, + serializeValue({ category: `B` }), { category: `B`, countNotNull: 1, @@ -412,7 +446,7 @@ describe(`Operators`, () => { const expectedResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, average: 15, @@ -423,7 +457,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"B"}`, + serializeValue({ category: `B` }), { category: `B`, average: 30, @@ -448,7 +482,7 @@ describe(`Operators`, () => { const expectedAddResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, average: 15, @@ -459,7 +493,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, average: 20, @@ -470,7 +504,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"C"}`, + serializeValue({ category: `C` }), { category: `C`, average: 50, @@ -494,7 +528,7 @@ describe(`Operators`, () => { const expectedDeleteResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, average: 20, @@ -505,7 +539,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, average: 25, @@ -561,7 +595,7 @@ describe(`Operators`, () => { const expectedResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, minimum: 5, @@ -574,7 +608,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"B"}`, + serializeValue({ category: `B` }), { category: `B`, minimum: 15, @@ -637,7 +671,7 @@ describe(`Operators`, () => { const expectedResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, middle: 20, @@ -648,7 +682,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"B"}`, + serializeValue({ category: `B` }), { category: `B`, middle: 12.5, @@ -699,7 +733,7 @@ describe(`Operators`, () => { // Find the group for category A const categoryAGroup = result.find( - ([key]: any) => key[0] === `{"category":"A"}`, + ([key]: any) => key[0] === serializeValue({ category: `A` }), ) expect(categoryAGroup).toBeDefined() expect(categoryAGroup[0][1].total).toBe(30) // Sum of 10 + 20 @@ -722,7 +756,7 @@ describe(`Operators`, () => { const expectedResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, total: 30, @@ -737,7 +771,8 @@ describe(`Operators`, () => { // Verify no new group with total: 0 was created by checking that // we don't have any positive weight entries for category A const positiveCategoryAEntries = result.filter( - ([key, , weight]: any) => key[0] === `{"category":"A"}` && weight > 0, + ([key, , weight]: any) => + key[0] === serializeValue({ category: `A` }) && weight > 0, ) expect(positiveCategoryAEntries).toHaveLength(0) }) @@ -788,7 +823,8 @@ describe(`Operators`, () => { // Find the group for category A, region East const categoryAEastGroup = result.find( - ([key]: any) => key[0] === `{"category":"A","region":"East"}`, + ([key]: any) => + key[0] === serializeValue({ category: `A`, region: `East` }), ) expect(categoryAEastGroup).toBeDefined() expect(categoryAEastGroup[0][1]).toEqual({ @@ -816,7 +852,7 @@ describe(`Operators`, () => { const expectedResult = [ [ [ - `{"category":"A","region":"East"}`, + serializeValue({ category: `A`, region: `East` }), { category: `A`, region: `East`, @@ -834,7 +870,8 @@ describe(`Operators`, () => { // Verify no new group with zero/empty values was created const positiveCategoryAEastEntries = result.filter( ([key, , weight]: any) => - key[0] === `{"category":"A","region":"East"}` && weight > 0, + key[0] === serializeValue({ category: `A`, region: `East` }) && + weight > 0, ) expect(positiveCategoryAEastEntries).toHaveLength(0) }) @@ -875,7 +912,7 @@ describe(`Operators`, () => { // Find the group for category A const categoryAGroup = result.find( - ([key]: any) => key[0] === `{"category":"A"}`, + ([key]: any) => key[0] === serializeValue({ category: `A` }), ) expect(categoryAGroup).toBeDefined() expect(categoryAGroup[0][1].total).toBe(30) // Sum of 10 + 20 @@ -894,7 +931,7 @@ describe(`Operators`, () => { const expectedRemovalResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, total: 30, @@ -919,7 +956,7 @@ describe(`Operators`, () => { const expectedReAdditionResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, total: 75, // 50 + 25 (new values, not the old 30) @@ -939,7 +976,7 @@ describe(`Operators`, () => { const expectedUpdateResult = [ [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, total: 75, // Previous total @@ -949,7 +986,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"A"}`, + serializeValue({ category: `A` }), { category: `A`, total: 90, // 75 + 15 @@ -1009,7 +1046,8 @@ describe(`Operators`, () => { // Find the group for category A, region East const categoryAEastGroup = result.find( - ([key]: any) => key[0] === `{"category":"A","region":"East"}`, + ([key]: any) => + key[0] === serializeValue({ category: `A`, region: `East` }), ) expect(categoryAEastGroup).toBeDefined() expect(categoryAEastGroup[0][1]).toEqual({ @@ -1037,7 +1075,7 @@ describe(`Operators`, () => { const expectedRemovalResult = [ [ [ - `{"category":"A","region":"East"}`, + serializeValue({ category: `A`, region: `East` }), { category: `A`, region: `East`, @@ -1069,7 +1107,7 @@ describe(`Operators`, () => { const expectedReAdditionResult = [ [ [ - `{"category":"A","region":"East"}`, + serializeValue({ category: `A`, region: `East` }), { category: `A`, region: `East`, @@ -1098,7 +1136,7 @@ describe(`Operators`, () => { const expectedPartialRemovalResult = [ [ [ - `{"category":"A","region":"East"}`, + serializeValue({ category: `A`, region: `East` }), { category: `A`, region: `East`, @@ -1113,7 +1151,7 @@ describe(`Operators`, () => { ], [ [ - `{"category":"A","region":"East"}`, + serializeValue({ category: `A`, region: `East` }), { category: `A`, region: `East`, diff --git a/packages/db-ivm/tests/operators/orderByWithFractionalIndex.test.ts b/packages/db-ivm/tests/operators/orderByWithFractionalIndex.test.ts index 4dc16430fb..e341b84d5e 100644 --- a/packages/db-ivm/tests/operators/orderByWithFractionalIndex.test.ts +++ b/packages/db-ivm/tests/operators/orderByWithFractionalIndex.test.ts @@ -5,11 +5,20 @@ import { orderByWithFractionalIndex, output, } from '../../src/operators/index.js' -import { orderByWithFractionalIndexBTree } from '../../src/operators/orderByBTree.js' -import { loadBTree } from '../../src/operators/topKWithFractionalIndexBTree.js' +import { orderByWithFractionalIndexBase } from '../../src/operators/orderBy.js' +import { + loadBTree, + topKWithFractionalIndexBTree, +} from '../../src/operators/topKWithFractionalIndexBTree.js' import { MessageTracker, compareFractionalIndex } from '../test-utils.js' import type { KeyValue } from '../../src/types.js' +const orderByWithBTree: typeof orderByWithFractionalIndex = ( + extract, + options, +) => + orderByWithFractionalIndexBase(topKWithFractionalIndexBTree, extract, options) + const stripFractionalIndex = ([[key, [value, _index]], multiplicity]: any) => [ key, value, @@ -27,7 +36,7 @@ beforeAll(async () => { describe(`Operators`, () => { describe.each([ [`with array`, { orderBy: orderByWithFractionalIndex }], - [`with B+ tree`, { orderBy: orderByWithFractionalIndexBTree }], + [`with B+ tree`, { orderBy: orderByWithBTree }], ])(`OrderByWithFractionalIndex operator %s`, (_, { orderBy }) => { test(`initial results with default comparator`, () => { const graph = new D2() diff --git a/packages/db-ivm/tests/utils.test.ts b/packages/db-ivm/tests/utils.test.ts index a3eb0685bb..c9e35c0ca2 100644 --- a/packages/db-ivm/tests/utils.test.ts +++ b/packages/db-ivm/tests/utils.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from 'vitest' import { Temporal } from 'temporal-polyfill' -import { DefaultMap } from '../src/utils.js' +import { DefaultMap, compareKeys, serializeValue } from '../src/utils.js' import { hash } from '../src/hashing/index.js' describe(`DefaultMap`, () => { @@ -30,6 +30,37 @@ describe(`DefaultMap`, () => { }) }) +describe(`compareKeys`, () => { + it(`orders finite numeric keys before NaN`, () => { + expect(compareKeys(1, Number.NaN)).toBeLessThan(0) + expect(compareKeys(Number.NaN, 1)).toBeGreaterThan(0) + expect(compareKeys(Number.NaN, Number.NaN)).toBe(0) + }) +}) + +describe(`serializeValue`, () => { + it(`preserves the established JSON form for ordinary keys`, () => { + expect(serializeValue(`user1`)).toBe(`"user1"`) + expect(serializeValue([1, `completed`])).toBe(`[1,"completed"]`) + }) + + it(`keeps distinct primitive types and special numbers distinct`, () => { + expect(serializeValue(1n)).not.toBe(serializeValue(`1`)) + expect(serializeValue(Number.NaN)).not.toBe(serializeValue(null)) + expect(serializeValue(undefined)).not.toBe(serializeValue(null)) + expect(serializeValue(new Date(0))).not.toBe( + serializeValue(`1970-01-01T00:00:00.000Z`), + ) + expect(serializeValue(new Date(Number.NaN))).not.toBe( + serializeValue(Number.NaN), + ) + }) + + it(`canonicalizes plain-object property order`, () => { + expect(serializeValue({ a: 1, b: 2 })).toBe(serializeValue({ b: 2, a: 1 })) + }) +}) + const hashType = `number` describe(`hash`, () => { describe(`primitive types`, () => { @@ -122,12 +153,21 @@ describe(`hash`, () => { expect(typeof result1).toBe(hashType) expect(typeof result2).toBe(hashType) expect(typeof result3).toBe(hashType) - // Note: Different symbol instances with same description have same hash - expect(result1).toBe(result2) + expect(result1).not.toBe(result2) expect(result1).not.toBe(result3) - expect(result4).toBe(result5) + expect(result4).not.toBe(result5) expect(result1).not.toBe(result4) }) + + it(`should hash registered symbols`, () => { + const first = Symbol.for(`tanstack-db-ivm-hash-first`) + const same = Symbol.for(`tanstack-db-ivm-hash-first`) + const second = Symbol.for(`tanstack-db-ivm-hash-second`) + + expect(hash(first)).toBe(hash(same)) + expect(hash(first)).not.toBe(hash(second)) + expect(hash({ [first]: 1 })).not.toBe(hash({ [second]: 1 })) + }) }) describe(`object types`, () => { @@ -143,6 +183,355 @@ describe(`hash`, () => { // Note: Different key orders might produce different hashes depending on JSON.stringify behavior }) + it(`includes enumerable symbol keys and values`, () => { + const key = Symbol(`key`) + + expect(hash({ [key]: `before` })).not.toBe(hash({ [key]: `after` })) + expect(hash({ [Symbol(`key`)]: `value` })).not.toBe( + hash({ [Symbol(`key`)]: `value` }), + ) + }) + + it(`rejects self and mutual cycles through symbol keys`, () => { + const key = Symbol(`cycle`) + const first: Record = {} + const second: Record = {} + first[key] = first + second[key] = second + + const firstPeer: Record = {} + const secondPeer: Record = {} + firstPeer[key] = secondPeer + secondPeer[key] = firstPeer + + for (const input of [first, second, firstPeer, secondPeer]) { + expect(() => hash(input)).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(input)).toThrow( + `Cannot hash cyclic structural values`, + ) + } + }) + + it.each([`object`, `map`] as const)( + `rejects shared cyclic branches through %s with bounded work`, + (container) => { + const size = 14 + let reads = 0 + const nodes: Array | Map> = + Array.from({ length: size }, (_, value) => + container === `object` + ? { value } + : new Map([[`value`, value]]), + ) + + for (let index = 0; index < size; index++) { + const node = nodes[index]! + const next = nodes[(index + 1) % size]! + for (const key of [`left`, `right`] as const) { + const wrapper = Object.defineProperty({}, `next`, { + enumerable: true, + get: () => { + reads++ + return next + }, + }) + if (node instanceof Map) node.set(key, wrapper) + else node[key] = wrapper + } + } + + expect(() => hash(nodes[0]!)).toThrow( + `Cannot hash cyclic structural values`, + ) + const firstReads = reads + const copy = structuredClone(nodes[0]!) + + expect(() => hash(copy)).toThrow(`Cannot hash cyclic structural values`) + expect(firstReads).toBeLessThanOrEqual(size * 2) + }, + ) + + it(`rejects a shared child that cycles to either ancestor`, () => { + const createGraph = (backBranch: `left` | `right`) => { + const root: Record = {} + const left: Record = {} + const right: Record = {} + const shared: Record = {} + root.left = left + root.right = right + left.next = shared + right.next = shared + shared.back = backBranch === `left` ? left : right + return root + } + + const left = createGraph(`left`) + const equalLeft = createGraph(`left`) + const right = createGraph(`right`) + + for (const input of [left, equalLeft, right]) { + expect(() => hash(input)).toThrow( + `Cannot hash cyclic structural values`, + ) + } + }) + + it(`rejects cyclic graphs with exponentially many ancestor contexts`, () => { + const depth = 11 + const shared = Array.from( + { length: depth + 1 }, + (_, level) => ({ level }) as Record, + ) + const left = Array.from({ length: depth }, (_, level) => ({ + side: `left`, + level, + next: shared[level + 1], + })) + const right = Array.from({ length: depth }, (_, level) => ({ + side: `right`, + level, + next: shared[level + 1], + })) + for (let level = 0; level < depth; level++) { + shared[level]!.left = left[level] + shared[level]!.right = right[level] + shared[depth]![`left${level}`] = left[level] + } + + expect(() => hash(shared[0])).toThrow(TypeError) + expect(() => hash(shared[0])).toThrow( + `Cannot hash cyclic structural values`, + ) + + const ring = Array.from( + { length: 600 }, + (_, value) => ({ value }) as { value: number; next?: unknown }, + ) + for (let index = 0; index < ring.length; index++) { + ring[index]!.next = ring[(index + 1) % ring.length] + } + expect(() => hash(structuredClone(ring[0]))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(ring[0])).toThrow( + `Cannot hash cyclic structural values`, + ) + + const independent: Record = {} + for (let index = 0; index < 600; index++) { + const cycle: { self?: unknown } = {} + cycle.self = cycle + independent[String(index)] = cycle + } + expect(() => hash(structuredClone(independent))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(independent)).toThrow( + `Cannot hash cyclic structural values`, + ) + + const independentDiamonds: Record = {} + for (let index = 0; index < 600; index++) { + const diamondCenter: Record = {} + const leftIngress = { next: diamondCenter } + const rightIngress = { next: diamondCenter } + diamondCenter.back = leftIngress + independentDiamonds[`left${index}`] = leftIngress + independentDiamonds[`right${index}`] = rightIngress + } + expect(() => hash(structuredClone(independentDiamonds))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(independentDiamonds)).toThrow( + `Cannot hash cyclic structural values`, + ) + + const small: { self?: unknown } = {} + small.self = small + expect(() => hash(structuredClone(small))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(small)).toThrow(`Cannot hash cyclic structural values`) + }) + + it(`rejects both small and large repeated cyclic traversals`, () => { + const createGraph = (size: number) => { + const nodes = Array.from( + { length: size }, + (_, value) => ({ value }) as Record, + ) + for (let index = 0; index < size; index++) { + const next = nodes[(index + 1) % size]! + nodes[index]!.left = { next } + nodes[index]!.right = { next } + } + return nodes[0] + } + + expect(() => hash(createGraph(20))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(createGraph(300))).toThrow( + `Cannot hash cyclic structural values`, + ) + }) + + it(`does not warm structural caches when a hash is rejected`, () => { + let reads = 0 + const sentinel = Object.defineProperty({}, `value`, { + enumerable: true, + get: () => ++reads, + }) + const shared: Record = { + payload: Array.from({ length: 66_000 }, (_, value) => ({ value })), + } + const left = { next: shared } + const right = { next: shared } + shared.back = left + const root = { aSentinel: sentinel, left, right } + + expect(() => hash(root)).toThrow(`Cannot hash cyclic structural values`) + expect(() => hash(root)).toThrow(`Cannot hash cyclic structural values`) + expect(reads).toBe(2) + }) + + it.each([ + [`Buffer`, () => Buffer.alloc(129)], + [`Uint8Array`, () => new Uint8Array(129)], + [`File`, () => new File([`opaque`], `opaque.bin`)], + ])( + `treats a large %s as an opaque leaf before structural work`, + (_name, createLeaf) => { + const leaves = Array.from({ length: 700 }, createLeaf) + for (const leaf of leaves) Object.assign(leaf, { self: leaf }) + const createChain = () => { + const ring = leaves.map((leaf, value) => ({ + value, + leaf, + next: undefined as unknown, + })) + for (let index = 0; index < ring.length; index++) { + ring[index]!.next = ring[index + 1] + } + return ring[0] + } + + const first = createChain() + const expectedHash = hash(first) + expect(hash(first)).toBe(expectedHash) + expect(hash(createChain())).toBe(expectedHash) + + let atDepthBoundary: unknown = createLeaf() + for (let index = 0; index < 768; index++) { + atDepthBoundary = { next: atDepthBoundary } + } + expect(() => hash(atDepthBoundary)).not.toThrow() + + const adoptionLeaves = Array.from({ length: 20 }, createLeaf) + const createAdoptionGraph = () => { + const nodes = adoptionLeaves.map((leaf, value) => ({ + value, + leaf, + })) as Array> + for (let index = 0; index < nodes.length; index++) { + const next = nodes[index + 1] + nodes[index]!.left = { next } + nodes[index]!.right = { next } + } + return nodes[0] + } + expect(hash(createAdoptionGraph())).toBe(hash(createAdoptionGraph())) + }, + ) + + it(`rejects deep structural recursion before the JavaScript stack overflows`, () => { + let reads = 0 + const sentinel = Object.defineProperty({}, `value`, { + enumerable: true, + get: () => ++reads, + }) + const ring = Array.from( + { length: 800 }, + (_, value) => ({ value }) as { value: number; next?: unknown }, + ) + for (let index = 0; index < ring.length; index++) { + ring[index]!.next = ring[(index + 1) % ring.length] + } + Object.defineProperty(ring[0]!, `aSentinel`, { + enumerable: true, + value: sentinel, + }) + + expect(() => hash(ring[0])).toThrow( + `Value is too complex to hash safely: structural depth`, + ) + expect(() => hash(ring[0])).toThrow( + `Value is too complex to hash safely: structural depth`, + ) + expect(reads).toBe(2) + + const createChain = (size: number) => { + const root: { next?: unknown } = {} + let tail = root + for (let index = 0; index < size; index++) { + const next: { next?: unknown } = {} + tail.next = next + tail = next + } + return root + } + const accepted = createChain(600) + expect(hash(structuredClone(accepted))).toBe(hash(accepted)) + + const root = createChain(800) + expect(() => hash(root)).toThrow( + `Value is too complex to hash safely: structural depth`, + ) + }) + + it(`rejects dense ancestor back-references without warming siblings`, () => { + const createGraph = (size: number) => { + const nodes: Array> = [] + for (let index = 0; index < size; index++) { + const node: Record = { index } + if (index > 0) nodes[index - 1]!.next = node + for (let ancestor = 0; ancestor < index; ancestor++) { + node[`ancestor${ancestor}`] = nodes[ancestor] + } + nodes.push(node) + } + return nodes[0]! + } + const accepted = createGraph(50) + expect(() => hash(structuredClone(accepted))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(accepted)).toThrow( + `Cannot hash cyclic structural values`, + ) + + let reads = 0 + const sentinel = Object.defineProperty({}, `value`, { + enumerable: true, + get: () => ++reads, + }) + const rejected = createGraph(450) + Object.defineProperty(rejected, `aSentinel`, { + enumerable: true, + value: sentinel, + }) + + expect(() => hash(rejected)).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(rejected)).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(reads).toBe(2) + }) + it(`should hash arrays`, () => { const arr1 = [1, 2, 3] const arr2 = [1, 2, 3] diff --git a/packages/db-sqlite-persistence-core/CHANGELOG.md b/packages/db-sqlite-persistence-core/CHANGELOG.md index e1721d2e3a..6481eb6406 100644 --- a/packages/db-sqlite-persistence-core/CHANGELOG.md +++ b/packages/db-sqlite-persistence-core/CHANGELOG.md @@ -1,5 +1,214 @@ # @tanstack/db-sqlite-persistence-core +## 0.2.21 + +### Patch Changes + +- Fix on-demand load settlement, ordered pagination, and replay to preserve coherent results across cancellation, failure, cleanup, and restart. Preserve subset results and ownership across Electric, PowerSync, Query, and SQLite persistence adapters. Correct live-query grouping, include projections, value identity, and indexed comparisons. D2 hashing now rejects structural cycles and excessive traversal depth or work with an explicit error; Collection handles retain object-reference identity without traversing their mutable contents. ([#1797](https://github.com/TanStack/db/pull/1797)) + + Remove the unused public subset-algebra helpers: `isWhereSubset`, `unionWherePredicates`, `minusWherePredicates`, `isOrderBySubset`, `isLimitSubset`, `isOffsetLimitSubset`, `isPredicateSubset`, and `isLoadSubsetRequestSubsumedBy`. Apps that import these helpers must remove those imports; normal queries and adapters are unaffected. `DeduplicatedLoadSubset` remains available and shares only exact demand identities. + + Reject compiled Collection-valued includes as `fn.select()` inputs, including nested descendants, before invoking the callback. Use `toArray()` or `materialize()` in the upstream `.select()` for child-value calculations. To keep live child Collections, use expression `.select()` or perform parent-only functional work before adding the includes. Ordinary Collection-valued includes remain supported. + + Remove proxy `DEBUG` logging and automatic index timing statistics to avoid diagnostic work on reads and writes. Remove `getStats()` and `IndexStats`; use `index.keyCount` for the current entry count, and instrument index methods externally when profiling. Custom index subclasses must remove calls to the retired `trackLookup()` and `updateTimestamp()` helpers. + + Fix mutation drafts so Map/Set `forEach` calls each callback once and read-only iteration reports no changes. Track nested Map `for...of` edits through `collection.update()`. Keep Set entries in place during nested edits, preserving live iteration and usable `has`, `delete`, and `add` handles without duplicate entries or repeated visits. Reverting one entry no longer discards another entry's pending changes. + + Preserve shared values within a row's draft. Newly supplied objects use normal shared references during the update callback, including Map values and Set members; editing through either handle changes the same new object. Copy completed changes at callback return so later caller edits cannot mutate stored data. Existing collection values remain isolated. Copy a new caller-owned value before insertion if it must remain untouched during the callback. + + Keep rejected local-storage mutations out of later successful saves. Stage insert, update, delete, and manual transaction writes before persisting, then promote the shared cache only after storage succeeds. + + Deliver live-query observer publications to peer listeners even when one listener throws. Preserve queued publication order and stop delivery on disposal; report the first listener failure after delivery. + + Wait for PowerSync demands admitted during tracking finalization before reporting them loaded. Preserve untouched object and array key identity when they contain `NaN`. Retire every failed ordered acquisition on explicit retry, while retaining a full-source demand repaired by replay. + + After authoritative ordered recovery succeeds, retire settled page and tie demands so later truncate replay fetches only the full-source replacement. Keep unfinished requests observed until settlement and preserve independent query ownership. + + Remove the live-query `utils.getRunCount()` diagnostic and its runtime counter. Apps that call this diagnostic must remove those calls. Query scheduling and results are unchanged. + + Remove test-only index inspection getters (`indexedKeysSet`, `valueMapData`, `orderedEntriesArray`, and `orderedEntriesArrayReversed`) and unused scheduler diagnostics. `ReverseIndex` now exposes only the `IndexReader` lookup, range, and forward traversal surface returned by `findIndexForField`; mutate the original index instead. Export `IndexReader` for callers that name this return type. Remove the unused subscription `releaseLoadSubset()` method; request owners use the release callback supplied by `onLoadSubsetResult`. Ordinary query and adapter APIs remain unchanged by these removals. + + Remove unused internal helpers and the unused public error classes `WhereClauseConversionError`, `SubscriptionNotFoundError`, and `AggregateNotSupportedError`. These classes have no remaining runtime throw sites; remove any imports of them. Keep the existing query and index behavior and exercise identity/evaluation tests through the production entry points. + + Ensure failed mutations roll back even when their rejection value cannot be converted to a string. Preserve ordinary Error instances; report unprintable rejection values as `Unknown error`. + + Make reentrant effect disposal share the active cleanup result, including calls from abort listeners or source release callbacks. A release failure reaches every waiting disposer while each source still receives one release attempt. + + Reject starting or preloading a collection from inside its active cleanup callbacks with a clear `CollectionStateError`. Nested cleanup cannot admit replacement work that the old teardown would discard. Restart after cleanup completes, or from its final `cleaned-up` status event, remains supported. + + Prevent older page or tie-boundary completions from clearing a newer full-source failure or starting redundant loading. Failed window moves retain their settled public snapshot; an explicit retry releases the failed acquisition once and publishes the completed replacement. + + Restrict direct subscription `requestLimitedSnapshot()` cursor inputs to one order term and one `minValues` entry. Composite and partial-composite cursor inputs now throw before local delivery or adapter work. Use normal live-query ordering and window APIs for multi-column pagination; those remain supported through prefix-and-tie loading. Existing adapters need no changes. + + Treat `LoadSubsetOptions` and their nested request data as immutable from submission onward. Core no longer copies expression trees or mutable constant payloads at the sync and deduplication boundaries. Create a new Date, byte array, membership array, or options object when changing a demand instead of mutating submitted data. Adapters must also leave request data unchanged. Use stable data properties rather than stateful getters. `AbortSignal` cancellation and subscription release remain live. + + Replace replay acquisitions sequentially: release the prior physical lease before starting its replacement. The logical demand and last complete public result remain retained. A failed release prevents replacement startup; failed startup leaves demand available for a later authoritative replay. Custom adapters must support a release/load gap and preserve resources still held by other owners; a sole underlying resource may stop and restart. + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 0.2.20 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 0.2.19 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 0.2.18 + +### Patch Changes + +- Settle subset loads only after their committed rows and events are visible. A ([#1769](https://github.com/TanStack/db/pull/1769)) + commit receipt now rejects with `AbortError` when cancellation wins before + application and ignores later aborts. Preserve causal publication, + cancellation, persistence, and error handling across the affected sync + adapters. +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 0.2.17 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 0.2.16 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 0.2.15 + +### Patch Changes + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 0.2.14 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 0.2.13 + +### Patch Changes + +- Add SSR through request-scoped `DbClient` instances, collection descriptors, ([#1564](https://github.com/TanStack/db/pull/1564)) + explicit collection-row hydration, live-query result snapshots, adapter sync + metadata, and React and Svelte descriptor resolution. + + React live queries now derive identity from structured query IR. Opaque queries + can provide `queryKey`; legacy dependency arrays and unkeyed opaque queries keep + working with development warnings until 1.0. + + Add TanStack Router integration that streams live queries discovered during a + Suspense render as pending promises which resolve to ordered result snapshots. + The browser starts normal source sync and atomically replaces the snapshot when + its live result is ready. + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 0.2.12 + +### Patch Changes + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 0.2.11 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 0.2.10 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 0.2.9 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 0.2.8 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 0.2.7 + +### Patch Changes + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 0.2.6 + +### Patch Changes + +- Fixed bug where internal metadata would get reset on insert causing stale items to remain ([#1626](https://github.com/TanStack/db/pull/1626)) + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 0.2.5 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 0.2.4 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 0.2.3 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 0.2.2 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 0.2.1 + +### Patch Changes + +- Use a safe `randomUUID` helper that falls back to `crypto.getRandomValues` when `crypto.randomUUID` is unavailable (non-secure browser contexts such as dev servers reached via a LAN IP over HTTP). Fixes #1541. ([#1593](https://github.com/TanStack/db/pull/1593)) + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 0.2.0 ### Minor Changes diff --git a/packages/db-sqlite-persistence-core/package.json b/packages/db-sqlite-persistence-core/package.json index ddcd5847d0..70ec2a80ab 100644 --- a/packages/db-sqlite-persistence-core/package.json +++ b/packages/db-sqlite-persistence-core/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/db-sqlite-persistence-core", - "version": "0.2.0", + "version": "0.2.21", "description": "SQLite persisted collection core for TanStack DB", "author": "TanStack Team", "license": "MIT", diff --git a/packages/db-sqlite-persistence-core/src/index.ts b/packages/db-sqlite-persistence-core/src/index.ts index 921b92f3b0..9e2bb9faae 100644 --- a/packages/db-sqlite-persistence-core/src/index.ts +++ b/packages/db-sqlite-persistence-core/src/index.ts @@ -1,3 +1,5 @@ export * from './persisted' export * from './errors' export * from './sqlite-core-adapter' +// Re-export for use in non-secure browser contexts (see #1541) +export { safeRandomUUID } from '@tanstack/db' diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 7c20e96883..0f1e4112ac 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -1,4 +1,10 @@ -import { compileSingleRowExpression, toBooleanPredicate } from '@tanstack/db' +import { + SyncTransactionAbortedError, + compileSingleRowExpression, + safeRandomUUID, + toBooleanPredicate, + withCollectionConfigFactory, +} from '@tanstack/db' import { InvalidPersistedCollectionConfigError, InvalidPersistedCollectionCoordinatorError, @@ -15,8 +21,10 @@ import type { CollectionIndexMetadata, DeleteMutationFnParams, InsertMutationFnParams, + LoadSubsetFn, LoadSubsetOptions, PendingMutation, + SyncAppliedReceipt, SyncConfig, SyncConfigRes, SyncMetadataApi, @@ -344,6 +352,7 @@ export interface PersistedCollectionUtils extends UtilsRecord { mutations: Array>> }) => Promise | void getLeadershipState?: () => PersistedCollectionLeadershipState + /** Hydrate once without acquiring a new ongoing subset lease. */ forceReloadSubset?: (options: LoadSubsetOptions) => Promise | void } @@ -428,7 +437,7 @@ type SyncControlFns = { | { type: `delete`; key: TKey }, ) => void) | null - commit: (() => void) | null + commit: ((signal?: AbortSignal) => SyncAppliedReceipt) | null truncate: (() => void) | null metadata: SyncMetadataApi | null } @@ -440,7 +449,7 @@ type SyncControlFns = { export class SingleProcessCoordinator implements PersistedCollectionCoordinator { private readonly nodeId: string - constructor(nodeId: string = crypto.randomUUID()) { + constructor(nodeId: string = safeRandomUUID()) { this.nodeId = nodeId } @@ -467,7 +476,7 @@ export class SingleProcessCoordinator implements PersistedCollectionCoordinator public pullSince(): Promise { return Promise.resolve({ type: `rpc:pullSince:res`, - rpcId: crypto.randomUUID(), + rpcId: safeRandomUUID(), ok: true, latestTerm: 1, latestSeq: 0, @@ -581,6 +590,9 @@ type BufferedSyncTransaction = { > truncate: boolean internal: boolean + signal?: AbortSignal + resolveApplied?: () => void + rejectApplied?: (error: unknown) => void } type OpenSyncTransaction< @@ -696,18 +708,6 @@ function stableSerialize(value: unknown): string { return JSON.stringify(toStableSerializable(value) ?? null) } -function normalizeSubsetOptionsForKey( - options: LoadSubsetOptions, -): Record { - return { - where: toStableSerializable(options.where), - orderBy: toStableSerializable(options.orderBy), - limit: options.limit, - cursor: toStableSerializable(options.cursor), - offset: options.offset, - } -} - function normalizeSyncFnResult(result: void | (() => void) | SyncConfigRes) { if (typeof result === `function`) { return { cleanup: result } satisfies SyncConfigRes @@ -791,7 +791,7 @@ class PersistedCollectionRuntime< BufferedSyncTransaction > = [] private readonly queuedTxCommitted: Array = [] - private readonly subscriptionIds = new WeakMap() + private readonly requestIds = new WeakMap() private collection: Collection | null = null @@ -805,13 +805,17 @@ class PersistedCollectionRuntime< private started = false private startupMetadataPromise: Promise | null = null private startPromise: Promise | null = null + private resumeBaselinePromise: Promise | null = null + private lifecycleGeneration = 0 private internalApplyDepth = 0 - private isHydrating = false + private appliedReceiptSequence = 0 + private readonly pendingAppliedReceipts = new Map>() + private hydratingGeneration: number | null = null private coordinatorUnsubscribe: (() => void) | null = null private indexAddedUnsubscribe: (() => void) | null = null private indexRemovedUnsubscribe: (() => void) | null = null private remoteEnsureRetryTimer: ReturnType | null = null - private nextSubscriptionId = 0 + private nextRequestId = 0 private latestTerm = 0 private latestSeq = 0 @@ -829,7 +833,34 @@ class PersistedCollectionRuntime< ) {} setSyncControls(syncControls: SyncControlFns): void { - this.syncControls = syncControls + this.advanceLifecycle() + + const commit = syncControls.commit + this.syncControls = { + ...syncControls, + commit: commit + ? (signal) => this.trackAppliedReceipt(commit(signal)) + : null, + } + } + + private trackAppliedReceipt(receipt: SyncAppliedReceipt): SyncAppliedReceipt { + const sequence = ++this.appliedReceiptSequence + if (receipt === true) { + return true + } + this.pendingAppliedReceipts.set(sequence, receipt) + const removeReceipt = () => this.pendingAppliedReceipts.delete(sequence) + void receipt.then(removeReceipt, removeReceipt) + return receipt + } + + private async waitForAppliedReceiptsAfter(cursor: number): Promise { + await Promise.all( + Array.from(this.pendingAppliedReceipts, ([sequence, receipt]) => + sequence > cursor ? receipt : undefined, + ), + ) } clearSyncControls(): void { @@ -843,7 +874,7 @@ class PersistedCollectionRuntime< } isHydratingNow(): boolean { - return this.isHydrating + return this.hydratingGeneration === this.lifecycleGeneration } isApplyingInternally(): boolean { @@ -873,20 +904,56 @@ class PersistedCollectionRuntime< return this.startPromise } - this.startPromise = this.startInternal() + const lifecycleGeneration = this.lifecycleGeneration + this.startPromise = this.startInternal(lifecycleGeneration) return this.startPromise } + ensureResumeBaselineHydrated(): Promise { + if (this.resumeBaselinePromise) { + return this.resumeBaselinePromise + } + + const lifecycleGeneration = this.lifecycleGeneration + this.resumeBaselinePromise = (async () => { + await this.ensureStarted() + if (lifecycleGeneration !== this.lifecycleGeneration) return + if (this.syncMode !== `on-demand`) return + + await this.hydrateBaseline(lifecycleGeneration) + })() + return this.resumeBaselinePromise + } + + private async hydrateBaseline(lifecycleGeneration: number): Promise { + if (lifecycleGeneration !== this.lifecycleGeneration) return + + const baseline = {} + this.activeSubsets.set(this.getSubsetKey(baseline), baseline) + const appliedCursor = this.appliedReceiptSequence + await this.applyMutex.run(async () => { + if (lifecycleGeneration !== this.lifecycleGeneration) return + await this.hydrateSubsetUnsafe(baseline, { + requestRemoteEnsure: false, + lifecycleGeneration, + }) + }) + if (lifecycleGeneration !== this.lifecycleGeneration) return + await this.waitForAppliedReceiptsAfter(appliedCursor) + } + async ensureStartupMetadataLoaded(): Promise { if (this.startupMetadataPromise) { return this.startupMetadataPromise } - this.startupMetadataPromise = this.loadStartupMetadataInternal() + const lifecycleGeneration = this.lifecycleGeneration + this.startupMetadataPromise = + this.loadStartupMetadataInternal(lifecycleGeneration) return this.startupMetadataPromise } - private async startInternal(): Promise { + private async startInternal(lifecycleGeneration: number): Promise { if (this.started) { return } @@ -894,26 +961,28 @@ class PersistedCollectionRuntime< this.started = true await this.ensureStartupMetadataLoaded() + if (lifecycleGeneration !== this.lifecycleGeneration) return const indexBootstrapSnapshot = this.collection?.getIndexMetadata() ?? [] this.attachIndexLifecycleListeners() await this.bootstrapPersistedIndexes(indexBootstrapSnapshot) + if (lifecycleGeneration !== this.lifecycleGeneration) return if (this.syncMode !== `on-demand`) { - this.activeSubsets.set(this.getSubsetKey({}), {}) - await this.applyMutex.run(() => - this.hydrateSubsetUnsafe({}, { requestRemoteEnsure: false }), - ) + await this.hydrateBaseline(lifecycleGeneration) } } - private async loadStartupMetadataInternal(): Promise { + private async loadStartupMetadataInternal( + lifecycleGeneration: number, + ): Promise { // Restore stream position from the database so that new mutations // don't collide with previously applied transactions. if (this.persistence.adapter.getStreamPosition) { const position = await this.persistence.adapter.getStreamPosition( this.collectionId, ) + if (lifecycleGeneration !== this.lifecycleGeneration) return this.observeStreamPosition( position.latestTerm, position.latestSeq, @@ -921,11 +990,8 @@ class PersistedCollectionRuntime< ) } - await this.loadCollectionMetadataIntoCollection() - } - - private async loadCollectionMetadataIntoCollection(): Promise { const collectionMetadata = await this.loadCollectionMetadataSnapshot() + if (lifecycleGeneration !== this.lifecycleGeneration) return this.replaceCollectionMetadataSnapshot(collectionMetadata) } @@ -976,31 +1042,39 @@ class PersistedCollectionRuntime< async loadSubset( options: LoadSubsetOptions, - upstreamLoadSubset?: (options: LoadSubsetOptions) => true | Promise, + upstreamLoadSubset?: LoadSubsetFn, ): Promise { + const lifecycleGeneration = this.lifecycleGeneration this.activeSubsets.set(this.getSubsetKey(options), options) + const appliedCursor = this.appliedReceiptSequence await this.applyMutex.run(() => this.hydrateSubsetUnsafe(options, { requestRemoteEnsure: this.mode === `sync-present`, + lifecycleGeneration, }), ) + if (lifecycleGeneration !== this.lifecycleGeneration) return + await this.waitForAppliedReceiptsAfter(appliedCursor) if (upstreamLoadSubset) { try { - const maybePromise = upstreamLoadSubset(options) - if (maybePromise instanceof Promise) { - maybePromise.catch((error) => { - console.warn( - `Failed to load remote subset in persisted wrapper:`, - error, - ) - this.queueRemoteSubsetEnsure(options) - }) - } + await upstreamLoadSubset(options) } catch (error) { + if ( + options.signal?.aborted || + (typeof error === `object` && + error !== null && + `name` in error && + error.name === `AbortError`) + ) { + this.pendingRemoteSubsetEnsures.delete(this.getSubsetKey(options)) + throw error + } console.warn(`Failed to trigger remote subset load:`, error) this.queueRemoteSubsetEnsure(options) + // Hydration remains readable, but it does not satisfy remote demand. + throw error } } } @@ -1010,13 +1084,18 @@ class PersistedCollectionRuntime< upstreamUnloadSubset?: (options: LoadSubsetOptions) => void, ): void { this.activeSubsets.delete(this.getSubsetKey(options)) + this.pendingRemoteSubsetEnsures.delete(this.getSubsetKey(options)) upstreamUnloadSubset?.(options) } async forceReloadSubset(options: LoadSubsetOptions): Promise { - this.activeSubsets.set(this.getSubsetKey(options), options) + const lifecycleGeneration = this.lifecycleGeneration + // A one-shot refresh does not acquire an enduring subscription lease. await this.applyMutex.run(() => - this.hydrateSubsetUnsafe(options, { requestRemoteEnsure: false }), + this.hydrateSubsetUnsafe(options, { + requestRemoteEnsure: false, + lifecycleGeneration, + }), ) } @@ -1135,6 +1214,8 @@ class PersistedCollectionRuntime< } cleanup(): void { + this.advanceLifecycle() + this.coordinatorUnsubscribe?.() this.coordinatorUnsubscribe = null @@ -1151,15 +1232,27 @@ class PersistedCollectionRuntime< this.pendingRemoteSubsetEnsures.clear() this.activeSubsets.clear() + for (const transaction of this.queuedHydrationTransactions) { + transaction.rejectApplied?.(new SyncTransactionAbortedError()) + } this.queuedHydrationTransactions.length = 0 this.queuedTxCommitted.length = 0 this.clearSyncControls() + this.collection = null } - private withInternalApply(task: () => void): void { + private advanceLifecycle(): void { + this.lifecycleGeneration++ + this.started = false + this.startupMetadataPromise = null + this.startPromise = null + this.resumeBaselinePromise = null + } + + private withInternalApply(task: () => TResult): TResult { this.internalApplyDepth++ try { - task() + return task() } finally { this.internalApplyDepth-- } @@ -1206,15 +1299,19 @@ class PersistedCollectionRuntime< options: LoadSubsetOptions, config: { requestRemoteEnsure: boolean + lifecycleGeneration: number }, ): Promise { - this.isHydrating = true + this.hydratingGeneration = config.lifecycleGeneration try { const rows = await this.loadSubsetRowsUnsafe(options) + if (config.lifecycleGeneration !== this.lifecycleGeneration) return this.applyRowsToCollection(rows) } finally { - this.isHydrating = false + if (this.hydratingGeneration === config.lifecycleGeneration) { + this.hydratingGeneration = null + } } await this.flushQueuedHydrationTransactionsUnsafe() @@ -1240,6 +1337,9 @@ class PersistedCollectionRuntime< this.syncControls.begin?.({ immediate: true }) for (const row of rows) { + if (this.collection?._hasHydratedKey(row.key)) { + continue + } this.syncControls.write?.({ type: `update`, value: row.value, @@ -1303,36 +1403,48 @@ class PersistedCollectionRuntime< if (!transaction) { continue } - await this.applyBufferedSyncTransactionUnsafe(transaction) + try { + await this.applyBufferedSyncTransactionUnsafe(transaction) + } catch (error) { + transaction.rejectApplied?.(error) + for (const abandoned of this.queuedHydrationTransactions) { + abandoned.rejectApplied?.(error) + } + this.queuedHydrationTransactions.length = 0 + throw error + } } } private async applyBufferedSyncTransactionUnsafe( transaction: BufferedSyncTransaction, ): Promise { - if ( - !this.syncControls.begin || - !this.syncControls.write || - !this.syncControls.commit - ) { + if (transaction.signal?.aborted) { + transaction.rejectApplied?.(new SyncTransactionAbortedError()) return } - const applyToCollection = () => { - this.syncControls.begin?.() + const { begin, write, commit, truncate, metadata } = this.syncControls + if (!begin || !write || !commit) { + transaction.rejectApplied?.(new SyncTransactionAbortedError()) + return + } + + const applyToCollection = (): SyncAppliedReceipt => { + begin() if (transaction.truncate) { - this.syncControls.truncate?.() + truncate?.() } for (const operation of transaction.operations) { if (operation.type === `delete`) { - this.syncControls.write?.({ + write({ type: `delete`, key: operation.key, }) } else { - this.syncControls.write?.({ + write({ type: `update`, value: operation.value, metadata: operation.metadata, @@ -1342,30 +1454,39 @@ class PersistedCollectionRuntime< for (const [key, metadataWrite] of transaction.rowMetadataWrites) { if (metadataWrite.type === `delete`) { - this.syncControls.metadata?.row.delete(key) + metadata?.row.delete(key) } else { - this.syncControls.metadata?.row.set(key, metadataWrite.value) + metadata?.row.set(key, metadataWrite.value) } } for (const [key, metadataWrite] of transaction.collectionMetadataWrites) { if (metadataWrite.type === `delete`) { - this.syncControls.metadata?.collection.delete(key) + metadata?.collection.delete(key) } else { - this.syncControls.metadata?.collection.set(key, metadataWrite.value) + metadata?.collection.set(key, metadataWrite.value) } } - this.syncControls.commit?.() + return commit(transaction.signal) } - if (transaction.internal) { - this.withInternalApply(applyToCollection) - return - } + try { + const applied = transaction.internal + ? this.withInternalApply(applyToCollection) + : applyToCollection() + if (applied !== true) { + await applied + } - applyToCollection() - await this.persistAndBroadcastExternalSyncTransactionUnsafe(transaction) + if (!transaction.internal) { + await this.persistAndBroadcastExternalSyncTransactionUnsafe(transaction) + } + transaction.resolveApplied?.() + } catch (error) { + transaction.rejectApplied?.(error) + throw error + } } private async persistAndBroadcastExternalSyncTransactionUnsafe( @@ -1387,7 +1508,7 @@ class PersistedCollectionRuntime< this.createTxCommittedPayload({ term: streamPosition.term, seq: streamPosition.seq, - txId: crypto.randomUUID(), + txId: safeRandomUUID(), latestRowVersion: streamPosition.rowVersion, changedRows: [], deletedKeys: [], @@ -1427,7 +1548,7 @@ class PersistedCollectionRuntime< streamPosition: { term: number; seq: number; rowVersion: number }, ): PersistedTx { return { - txId: crypto.randomUUID(), + txId: safeRandomUUID(), term: streamPosition.term, seq: streamPosition.seq, rowVersion: streamPosition.rowVersion, @@ -1471,7 +1592,7 @@ class PersistedCollectionRuntime< streamPosition: { term: number; seq: number; rowVersion: number }, ): PersistedTx { return { - txId: crypto.randomUUID(), + txId: safeRandomUUID(), term: streamPosition.term, seq: streamPosition.seq, rowVersion: streamPosition.rowVersion, @@ -1737,26 +1858,21 @@ class PersistedCollectionRuntime< } private getSubsetKey(options: LoadSubsetOptions): string { - const subscription = options.subscription as object | undefined - if (subscription && typeof subscription === `object`) { - const existingId = this.subscriptionIds.get(subscription) - if (existingId) { - return existingId - } - - this.nextSubscriptionId++ - const id = `sub:${this.nextSubscriptionId}` - this.subscriptionIds.set(subscription, id) - return id + // A subscription can own several independent acquisitions, including + // identical requests. Only releasing this options object ends its lease. + let id = this.requestIds.get(options) + if (id === undefined) { + id = `request:${++this.nextRequestId}` + this.requestIds.set(options, id) } - - return `opts:${stableSerialize(normalizeSubsetOptionsForKey(options))}` + return id } private queueRemoteSubsetEnsure(options: LoadSubsetOptions): void { if ( this.mode !== `sync-present` || - !this.persistence.coordinator.requestEnsureRemoteSubset + !this.persistence.coordinator.requestEnsureRemoteSubset || + this.activeSubsets.get(this.getSubsetKey(options)) !== options ) { return } @@ -1839,7 +1955,7 @@ class PersistedCollectionRuntime< // of both local and remote mutations. The seq dedup in // processCommittedTxUnsafe prevents double-processing of our own writes. if (isTxCommittedPayload(payload)) { - if (this.isHydrating) { + if (this.isHydratingNow()) { this.queuedTxCommitted.push(payload) return } @@ -2064,17 +2180,20 @@ class PersistedCollectionRuntime< } private async reloadActiveSubsetsUnsafe(): Promise { + const lifecycleGeneration = this.lifecycleGeneration const activeSubsetOptions = this.activeSubsets.size > 0 ? Array.from(this.activeSubsets.values()) : [{}] - this.isHydrating = true + this.hydratingGeneration = lifecycleGeneration try { const mergedRows = new Map() const collectionMetadata = await this.loadCollectionMetadataSnapshot() + if (lifecycleGeneration !== this.lifecycleGeneration) return for (const options of activeSubsetOptions) { const subsetRows = await this.loadSubsetRowsUnsafe(options) + if (lifecycleGeneration !== this.lifecycleGeneration) return for (const row of subsetRows) { mergedRows.set(row.key, { value: row.value, @@ -2092,7 +2211,9 @@ class PersistedCollectionRuntime< collectionMetadata, ) } finally { - this.isHydrating = false + if (this.hydratingGeneration === lifecycleGeneration) { + this.hydratingGeneration = null + } } await this.flushQueuedHydrationTransactionsUnsafe() @@ -2208,24 +2329,8 @@ function createWrappedSyncConfig< const getOpenTransaction = () => transactionStack[transactionStack.length - 1] let fullStartPromise: Promise | null = null - const cancelledLoadKeys = new Set() - const loadSubscriptionIds = new WeakMap() - let nextLoadSubscriptionId = 0 - const getLoadKey = (options: LoadSubsetOptions) => { - const subscription = options.subscription as object | undefined - if (subscription && typeof subscription === `object`) { - const existingId = loadSubscriptionIds.get(subscription) - if (existingId) { - return `sub:${existingId}` - } - nextLoadSubscriptionId++ - const nextId = String(nextLoadSubscriptionId) - loadSubscriptionIds.set(subscription, nextId) - return `sub:${nextId}` - } - - return `opts:${stableSerialize(normalizeSubsetOptionsForKey(options))}` - } + const startupState = { cleanedUp: false } + const acquisitions = new Map() runtime.setSyncControls({ begin: params.begin, write: params.write as SyncControlFns[`write`], @@ -2240,11 +2345,14 @@ function createWrappedSyncConfig< const wrappedParams = { ...params, markReady: () => { + if (startupState.cleanedUp) return void (fullStartPromise ?? runtime.ensureStarted()) .then(() => { + if (startupState.cleanedUp) return params.markReady() }) .catch((error) => { + if (startupState.cleanedUp) return console.warn( `Failed persisted sync startup before markReady:`, error, @@ -2253,6 +2361,7 @@ function createWrappedSyncConfig< }) }, begin: (options?: { immediate?: boolean }) => { + if (startupState.cleanedUp) return const transaction: OpenSyncTransaction = { operations: [], rowMetadataWrites: new Map(), @@ -2269,6 +2378,7 @@ function createWrappedSyncConfig< } }, write: (message: ChangeMessageOrDeleteKeyMessage) => { + if (startupState.cleanedUp) return const normalization = runtime.normalizeSyncWriteMessage(message) const openTransaction = getOpenTransaction() @@ -2286,9 +2396,21 @@ function createWrappedSyncConfig< message.type === `insert` && normalization.operation.metadata === undefined ) { - openTransaction.rowMetadataWrites.set(normalization.operation.key, { - type: `delete`, - }) + // Reset stale metadata for a fresh insert, but don't clobber an + // explicit metadata write already queued for this key in the same + // transaction (e.g. query reconcile stamps owners, then inserts). + if ( + !openTransaction.rowMetadataWrites.has( + normalization.operation.key, + ) + ) { + openTransaction.rowMetadataWrites.set( + normalization.operation.key, + { + type: `delete`, + }, + ) + } } else if (normalization.operation.metadata !== undefined) { openTransaction.rowMetadataWrites.set(normalization.operation.key, { type: `set`, @@ -2302,7 +2424,12 @@ function createWrappedSyncConfig< metadata: params.metadata ? { row: { + whenHydrated: () => + startupState.cleanedUp + ? Promise.resolve() + : runtime.ensureResumeBaselineHydrated(), get: (key: TKey) => { + if (startupState.cleanedUp) return undefined const openTransaction = getOpenTransaction() const pendingWrite = openTransaction?.rowMetadataWrites.get(key) @@ -2317,8 +2444,11 @@ function createWrappedSyncConfig< return params.metadata!.row.get(key) }, scanPersisted: (options?: PersistedRowScanOptions) => - runtime.scanPersistedRows(options), + startupState.cleanedUp + ? Promise.resolve([]) + : runtime.scanPersistedRows(options), set: (key: TKey, value: unknown) => { + if (startupState.cleanedUp) return const openTransaction = getOpenTransaction() if (!openTransaction) { throw new InvalidPersistedCollectionConfigError( @@ -2334,6 +2464,7 @@ function createWrappedSyncConfig< } }, delete: (key: TKey) => { + if (startupState.cleanedUp) return const openTransaction = getOpenTransaction() if (!openTransaction) { throw new InvalidPersistedCollectionConfigError( @@ -2350,6 +2481,7 @@ function createWrappedSyncConfig< }, collection: { get: (key: string) => { + if (startupState.cleanedUp) return undefined const openTransaction = getOpenTransaction() const pendingWrite = openTransaction?.collectionMetadataWrites.get(key) @@ -2361,6 +2493,7 @@ function createWrappedSyncConfig< return params.metadata!.collection.get(key) }, set: (key: string, value: unknown) => { + if (startupState.cleanedUp) return const openTransaction = getOpenTransaction() if (!openTransaction) { throw new InvalidPersistedCollectionConfigError( @@ -2376,6 +2509,7 @@ function createWrappedSyncConfig< } }, delete: (key: string) => { + if (startupState.cleanedUp) return const openTransaction = getOpenTransaction() if (!openTransaction) { throw new InvalidPersistedCollectionConfigError( @@ -2390,6 +2524,7 @@ function createWrappedSyncConfig< } }, list: (prefix?: string) => { + if (startupState.cleanedUp) return [] const merged = new Map( params .metadata!.collection.list() @@ -2420,6 +2555,7 @@ function createWrappedSyncConfig< } : undefined, truncate: () => { + if (startupState.cleanedUp) return const openTransaction = getOpenTransaction() if (!openTransaction) { params.truncate() @@ -2437,14 +2573,26 @@ function createWrappedSyncConfig< params.truncate() } }, - commit: () => { + commit: (signal?: AbortSignal) => { + if (startupState.cleanedUp) return true const openTransaction = transactionStack.pop() if (!openTransaction) { - params.commit() - return + return params.commit(signal) } if (openTransaction.queuedBecauseHydrating) { + if (signal?.aborted) { + const aborted = Promise.reject(new SyncTransactionAbortedError()) + void aborted.catch(() => undefined) + return aborted + } + let resolveApplied!: () => void + let rejectApplied!: (error: unknown) => void + const applied = new Promise((resolve, reject) => { + resolveApplied = resolve + rejectApplied = reject + }) + void applied.catch(() => undefined) runtime.queueHydrationBufferedTransaction({ operations: openTransaction.operations, rowMetadataWrites: openTransaction.rowMetadataWrites, @@ -2452,14 +2600,18 @@ function createWrappedSyncConfig< openTransaction.collectionMetadataWrites, truncate: openTransaction.truncate, internal: openTransaction.internal, + signal, + resolveApplied, + rejectApplied, }) - return + return applied } - params.commit() + const applied = params.commit(signal) if (!openTransaction.internal) { - void runtime - .persistAndBroadcastExternalSyncTransaction({ + const persistAfterApplication = async () => { + if (applied !== true) await applied + await runtime.persistAndBroadcastExternalSyncTransaction({ operations: openTransaction.operations, rowMetadataWrites: openTransaction.rowMetadataWrites, collectionMetadataWrites: @@ -2467,18 +2619,16 @@ function createWrappedSyncConfig< truncate: openTransaction.truncate, internal: false, }) - .catch((error) => { - console.warn( - `Failed to persist wrapped sync transaction:`, - error, - ) - }) + } + const persisted = persistAfterApplication() + void persisted.catch(() => undefined) + return persisted } + return applied }, } let sourceResult: SyncConfigRes = {} - const startupState = { cleanedUp: false } fullStartPromise = runtime.ensureStarted() const sourceResultPromise = (async () => { await runtime.ensureStartupMetadataLoaded() @@ -2496,23 +2646,50 @@ function createWrappedSyncConfig< return { cleanup: () => { startupState.cleanedUp = true + acquisitions.clear() sourceResult.cleanup?.() runtime.cleanup() runtime.clearSyncControls() }, loadSubset: async (options: LoadSubsetOptions) => { - const loadKey = getLoadKey(options) - cancelledLoadKeys.delete(loadKey) + const acquisition = { forwarded: false } + acquisitions.set(options, acquisition) await fullStartPromise const resolvedSourceResult = await sourceResultPromise - if (startupState.cleanedUp || cancelledLoadKeys.has(loadKey)) { + if ( + startupState.cleanedUp || + acquisitions.get(options) !== acquisition + ) { return } - await runtime.loadSubset(options, resolvedSourceResult.loadSubset) + return runtime.loadSubset(options, (loadOptions) => { + // Hydration is another async boundary. A release before this + // point owns no upstream lease and must not start one later. + if ( + startupState.cleanedUp || + acquisitions.get(options) !== acquisition + ) { + return true + } + if (!resolvedSourceResult.loadSubset) return true + acquisition.forwarded = true + try { + // Returning a promise transfers its lease even if it rejects. + // Only a synchronous throw leaves no upstream lease to release. + return resolvedSourceResult.loadSubset(loadOptions) + } catch (error) { + acquisition.forwarded = false + throw error + } + }) }, unloadSubset: (options: LoadSubsetOptions) => { - cancelledLoadKeys.add(getLoadKey(options)) - runtime.unloadSubset(options, sourceResult.unloadSubset) + const acquisition = acquisitions.get(options) + acquisitions.delete(options) + runtime.unloadSubset( + options, + acquisition?.forwarded ? sourceResult.unloadSubset : undefined, + ) }, } }, @@ -2607,7 +2784,7 @@ export function persistedCollectionOptions< const { schemaVersion, ...syncOptions } = options const collectionId = - syncOptions.id ?? `persisted-collection:${crypto.randomUUID()}` + syncOptions.id ?? `persisted-collection:${safeRandomUUID()}` const persistence = resolvePersistenceForCollection( syncOptions.persistence, { @@ -2625,17 +2802,26 @@ export function persistedCollectionOptions< collectionId, ) - return { + const result = { ...syncOptions, id: collectionId, sync: createWrappedSyncConfig(syncOptions.sync, runtime), persistence, } + + return withCollectionConfigFactory( + result, + () => + persistedCollectionOptions({ + ...options, + id: collectionId, + }) as typeof result, + ) } const { schemaVersion, ...localOnlyOptions } = options const collectionId = - localOnlyOptions.id ?? `persisted-collection:${crypto.randomUUID()}` + localOnlyOptions.id ?? `persisted-collection:${safeRandomUUID()}` const persistence = resolvePersistenceForCollection( localOnlyOptions.persistence, { @@ -2718,7 +2904,7 @@ export function persistedCollectionOptions< ...persistedUtils, } - return { + const result = { ...localOnlyOptions, id: collectionId, persistence, @@ -2730,6 +2916,15 @@ export function persistedCollectionOptions< startSync: true, gcTime: localOnlyOptions.gcTime ?? 0, } + + return withCollectionConfigFactory( + result, + () => + persistedCollectionOptions({ + ...options, + id: collectionId, + }) as typeof result, + ) } export function encodePersistedStorageKey(key: string | number): string { diff --git a/packages/db-sqlite-persistence-core/tests/persisted.test.ts b/packages/db-sqlite-persistence-core/tests/persisted.test.ts index 57c11d8442..24f4e7dc92 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted.test.ts @@ -1,7 +1,9 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' import { BasicIndex, + DbClient, IR, + collectionOptions, createCollection, createTransaction, } from '@tanstack/db' @@ -816,6 +818,243 @@ describe(`persistedCollectionOptions`, () => { ) }) + it(`does not apply or persist a wrapped sync transaction committed with an aborted signal`, async () => { + const adapter = createRecordingAdapter() + let remoteBegin: (() => void) | undefined + let remoteWrite: + | ((message: { type: `insert`; value: Todo }) => void) + | undefined + let remoteCommit: + | ((signal?: AbortSignal) => true | Promise) + | undefined + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present-aborted-commit`, + getKey: (item) => item.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + remoteBegin = begin + remoteWrite = write as (message: { + type: `insert` + value: Todo + }) => void + remoteCommit = commit + markReady() + }, + }, + persistence: { adapter }, + }), + ) + + try { + await collection.stateWhenReady() + const abortController = new AbortController() + abortController.abort() + remoteBegin?.() + remoteWrite?.({ + type: `insert`, + value: { id: `aborted`, title: `Must not publish` }, + }) + await expect( + remoteCommit?.(abortController.signal), + ).rejects.toMatchObject({ name: `AbortError` }) + await flushAsyncWork() + + expect(collection.get(`aborted`)).toBeUndefined() + expect(adapter.applyCommittedTxCalls).toHaveLength(0) + } finally { + await collection.cleanup() + } + }) + + it(`persists a wrapped sync transaction when abort follows application`, async () => { + const adapter = createRecordingAdapter() + let remoteBegin: (() => void) | undefined + let remoteWrite: + | ((message: { type: `insert`; value: Todo }) => void) + | undefined + let remoteCommit: + | ((signal?: AbortSignal) => true | Promise) + | undefined + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present-abort-after-application`, + getKey: (item) => item.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + remoteBegin = begin + remoteWrite = write as (message: { + type: `insert` + value: Todo + }) => void + remoteCommit = commit + markReady() + }, + }, + persistence: { adapter }, + }), + ) + let releaseMutation!: () => void + const mutationGate = new Promise((resolve) => { + releaseMutation = resolve + }) + const transaction = createTransaction({ + mutationFn: () => mutationGate, + }) + + try { + await collection.stateWhenReady() + transaction.mutate(() => { + collection.insert({ id: `local`, title: `Optimistic gate` }) + }) + + const abortController = new AbortController() + const subscription = collection.subscribeChanges((changes) => { + if (changes.some((change) => change.key === `remote`)) { + abortController.abort() + } + }) + remoteBegin?.() + remoteWrite?.({ + type: `insert`, + value: { id: `remote`, title: `Already visible` }, + }) + const receipt = remoteCommit?.(abortController.signal) + expect(receipt).toBeInstanceOf(Promise) + + releaseMutation() + await transaction.isPersisted.promise + await receipt + subscription.unsubscribe() + + expect(stripVirtualProps(collection.get(`remote`))).toEqual({ + id: `remote`, + title: `Already visible`, + }) + expect(adapter.applyCommittedTxCalls).toHaveLength(1) + } finally { + releaseMutation() + await transaction.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } + }) + + it(`rejects a wrapped sync receipt when persistence fails`, async () => { + const adapter = createRecordingAdapter() + const persistenceError = new Error(`persistence failed`) + adapter.applyCommittedTx = () => Promise.reject(persistenceError) + let remoteBegin: (() => void) | undefined + let remoteWrite: + | ((message: { type: `insert`; value: Todo }) => void) + | undefined + let remoteCommit: (() => true | Promise) | undefined + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present-persistence-error`, + getKey: (item) => item.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + remoteBegin = begin + remoteWrite = write as (message: { + type: `insert` + value: Todo + }) => void + remoteCommit = commit + markReady() + }, + }, + persistence: { adapter }, + }), + ) + + try { + await collection.stateWhenReady() + remoteBegin?.() + remoteWrite?.({ + type: `insert`, + value: { id: `failed`, title: `Not durable` }, + }) + + await expect(Promise.resolve(remoteCommit?.())).rejects.toBe( + persistenceError, + ) + } finally { + await collection.cleanup() + } + }) + + it(`preserves row metadata set before a metadata-less insert in the same sync transaction`, async () => { + const adapter = createRecordingAdapter() + const ownership = { queryCollection: { owners: [`gc:q1`] } } + const sync: SyncConfig = { + sync: ({ begin, write, commit, markReady, metadata }) => { + begin() + metadata?.row.set(`remote-1`, ownership) + write({ + type: `insert`, + value: { + id: `remote-1`, + title: `From remote`, + }, + }) + commit() + markReady() + }, + } + + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present`, + getKey: (item: Todo) => item.id, + sync, + persistence: { + adapter, + }, + }), + ) + + await collection.stateWhenReady() + await flushAsyncWork() + + expect(adapter.rowMetadata.get(`remote-1`)).toEqual(ownership) + expect(collection._state.syncedMetadata.get(`remote-1`)).toEqual(ownership) + }) + + it(`resets stale row metadata for a metadata-less insert with no queued metadata`, async () => { + const adapter = createRecordingAdapter() + adapter.rowMetadata.set(`remote-1`, { stale: true }) + const sync: SyncConfig = { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { + id: `remote-1`, + title: `From remote`, + }, + }) + commit() + markReady() + }, + } + + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present`, + getKey: (item: Todo) => item.id, + sync, + persistence: { + adapter, + }, + }), + ) + + await collection.stateWhenReady() + await flushAsyncWork() + + expect(adapter.rowMetadata.has(`remote-1`)).toBe(false) + }) + it(`uses a stable generated collection id in sync-present mode when id is omitted`, async () => { const adapter = createRecordingAdapter() const options = persistedCollectionOptions({ @@ -840,6 +1079,51 @@ describe(`persistedCollectionOptions`, () => { expect(adapter.loadSubsetCalls[0]?.collectionId).toBe(collection.id) }) + it(`keeps hydrated rows ahead of persisted startup rows`, async () => { + const adapter = createRecordingAdapter([ + { id: `1`, title: `Persisted title` }, + ]) + const descriptor = collectionOptions( + persistedCollectionOptions({ + id: `hydration-precedence`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReady() + }, + }, + persistence: { + adapter, + }, + }), + ) + const client = new DbClient() + client.hydrate({ + collections: [ + { + collectionId: descriptor.id, + rows: [ + { + key: `1`, + value: { id: `1`, title: `SSR title` }, + }, + ], + }, + ], + }) + + const collection = client.collection(descriptor) + await collection.stateWhenReady() + await flushAsyncWork() + + expect(collection.get(`1`)).toMatchObject({ + id: `1`, + title: `SSR title`, + }) + + await client.cleanup() + }) + it(`bootstraps and tracks persisted index lifecycle in sync-present mode`, async () => { const adapter = createRecordingAdapter() const collection = createCollection( @@ -980,6 +1264,151 @@ describe(`persistedCollectionOptions`, () => { }) }) + it(`discards a hydration-buffered transaction aborted before replay`, async () => { + const adapter = createRecordingAdapter() + let resolveLoadSubset: (() => void) | undefined + adapter.loadSubset = async () => { + await new Promise((resolve) => { + resolveLoadSubset = resolve + }) + return [] + } + let remoteBegin: (() => void) | undefined + let remoteWrite: + | ((message: { type: `insert`; value: Todo }) => void) + | undefined + let remoteCommit: + | ((signal?: AbortSignal) => true | Promise) + | undefined + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present-aborted-hydration-queue`, + getKey: (item) => item.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + remoteBegin = begin + remoteWrite = write as (message: { + type: `insert` + value: Todo + }) => void + remoteCommit = commit + markReady() + }, + }, + persistence: { adapter }, + }), + ) + + try { + const ready = collection.stateWhenReady() + for (let attempt = 0; attempt < 20 && !resolveLoadSubset; attempt++) { + await flushAsyncWork() + } + const abortController = new AbortController() + remoteBegin?.() + remoteWrite?.({ + type: `insert`, + value: { id: `aborted`, title: `Must not replay` }, + }) + const applied = remoteCommit?.(abortController.signal) + abortController.abort() + resolveLoadSubset?.() + await ready + if (applied !== true) { + await expect(applied).rejects.toMatchObject({ name: `AbortError` }) + } + await flushAsyncWork() + + expect(collection.get(`aborted`)).toBeUndefined() + expect(adapter.applyCommittedTxCalls).toHaveLength(0) + } finally { + resolveLoadSubset?.() + await collection.cleanup() + } + }) + + it(`rejects every hydration-buffered receipt when replay fails`, async () => { + const adapter = createRecordingAdapter() + let resolveLoadSubset: (() => void) | undefined + adapter.loadSubset = async () => { + await new Promise((resolve) => { + resolveLoadSubset = resolve + }) + return [] + } + + const replayError = new Error(`replay key failed`) + let bufferedRowKeyReads = 0 + let remoteBegin: (() => void) | undefined + let remoteWrite: + | ((message: { type: `insert`; value: Todo }) => void) + | undefined + let remoteCommit: (() => true | Promise) | undefined + + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present-replay-failure-receipt`, + getKey: (item) => { + if (item.id === `during-hydrate`) { + bufferedRowKeyReads++ + if (bufferedRowKeyReads === 2) { + throw replayError + } + } + return item.id + }, + sync: { + sync: ({ begin, write, commit, markReady }) => { + remoteBegin = begin + remoteWrite = write as (message: { + type: `insert` + value: Todo + }) => void + remoteCommit = commit + markReady() + return {} + }, + }, + persistence: { + adapter, + }, + }), + ) + + const readyPromise = collection.stateWhenReady() + for (let attempt = 0; attempt < 20 && !resolveLoadSubset; attempt++) { + await flushAsyncWork() + } + + remoteBegin?.() + remoteWrite?.({ + type: `insert`, + value: { id: `during-hydrate`, title: `During hydrate` }, + }) + const failingReceipt = remoteCommit?.() + remoteBegin?.() + remoteWrite?.({ + type: `insert`, + value: { id: `sibling`, title: `Sibling` }, + }) + const siblingReceipt = remoteCommit?.() + expect(failingReceipt).toBeInstanceOf(Promise) + expect(siblingReceipt).toBeInstanceOf(Promise) + const failingExpectation = expect( + Promise.resolve(failingReceipt), + ).rejects.toBe(replayError) + const siblingExpectation = expect( + Promise.resolve(siblingReceipt), + ).rejects.toBe(replayError) + + resolveLoadSubset?.() + await readyPromise + await failingExpectation + await siblingExpectation + + await collection.cleanup() + }) + it(`marks ready even when persisted startup fails before markReady`, async () => { const adapter = createRecordingAdapter() adapter.loadSubset = async () => { @@ -1315,6 +1744,487 @@ describe(`persistedCollectionOptions`, () => { expect(collection.get(`2`)).toBeUndefined() }) + it(`does not let a stale invalidation reload overwrite a restarted lifecycle`, async () => { + const adapter = createRecordingAdapter([{ id: `1`, title: `Initial` }]) + const coordinator = createCoordinatorHarness() + const originalLoadSubset = adapter.loadSubset.bind(adapter) + let loadCalls = 0 + let releaseStaleReload!: () => void + let releaseFreshReload!: () => void + const staleReloadGate = new Promise((resolve) => { + releaseStaleReload = resolve + }) + const freshReloadGate = new Promise((resolve) => { + releaseFreshReload = resolve + }) + adapter.loadSubset = async (...args) => { + loadCalls++ + if (loadCalls === 2) { + await staleReloadGate + return [ + { + key: `1`, + value: { id: `1`, title: `Stale reload` }, + }, + ] + } + if (loadCalls === 3) await freshReloadGate + return originalLoadSubset(...args) + } + + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReady() + }, + }, + persistence: { adapter, coordinator }, + }), + ) + + await collection.preload() + await flushAsyncWork() + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: 1, + txId: `tx-stale-reload`, + latestRowVersion: 1, + requiresFullReload: true, + }) + for (let attempt = 0; attempt < 20 && loadCalls < 2; attempt++) { + await flushAsyncWork() + } + expect(loadCalls).toBe(2) + + await collection.cleanup() + adapter.rows.set(`1`, { id: `1`, title: `Restarted` }) + collection.startSyncImmediate() + releaseStaleReload() + for (let attempt = 0; attempt < 20 && loadCalls < 3; attempt++) { + await flushAsyncWork() + } + expect(loadCalls).toBe(3) + expect(collection.get(`1`)?.title).not.toBe(`Stale reload`) + + releaseFreshReload() + for ( + let attempt = 0; + attempt < 20 && collection.get(`1`)?.title !== `Restarted`; + attempt++ + ) { + await flushAsyncWork() + } + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + title: `Restarted`, + }) + await collection.cleanup() + }) + + it(`does not let stale reload metadata start row loading after restart`, async () => { + const adapter = createRecordingAdapter([{ id: `1`, title: `Initial` }]) + const coordinator = createCoordinatorHarness() + const originalLoadCollectionMetadata = + adapter.loadCollectionMetadata!.bind(adapter) + const originalLoadSubset = adapter.loadSubset.bind(adapter) + let metadataCalls = 0 + let subsetCalls = 0 + let releaseStaleMetadata!: () => void + const staleMetadataGate = new Promise((resolve) => { + releaseStaleMetadata = resolve + }) + adapter.loadCollectionMetadata = async (...args) => { + metadataCalls++ + if (metadataCalls === 2) await staleMetadataGate + return originalLoadCollectionMetadata(...args) + } + adapter.loadSubset = async (...args) => { + subsetCalls++ + return originalLoadSubset(...args) + } + + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReady() + }, + }, + persistence: { adapter, coordinator }, + }), + ) + + await collection.preload() + await flushAsyncWork() + expect(metadataCalls).toBe(1) + expect(subsetCalls).toBe(1) + + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: 1, + txId: `tx-stale-metadata`, + latestRowVersion: 1, + requiresFullReload: true, + }) + for (let attempt = 0; attempt < 20 && metadataCalls < 2; attempt++) { + await flushAsyncWork() + } + expect(metadataCalls).toBe(2) + + await collection.cleanup() + adapter.rows.set(`1`, { id: `1`, title: `Restarted` }) + collection.startSyncImmediate() + releaseStaleMetadata() + for ( + let attempt = 0; + attempt < 20 && (metadataCalls < 3 || subsetCalls < 2); + attempt++ + ) { + await flushAsyncWork() + } + + expect(metadataCalls).toBe(3) + expect(subsetCalls).toBe(2) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + title: `Restarted`, + }) + await collection.cleanup() + }) + + it.each( + [false, true].flatMap((sharedSubscription) => + [false, true].map((identical) => ({ sharedSubscription, identical })), + ), + )( + `keeps sibling requests owned after one release: %j`, + async ({ sharedSubscription, identical }) => { + const adapter = createRecordingAdapter([{ id: `1`, title: `Before` }]) + const coordinator = createCoordinatorHarness() + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset: () => true } + }, + }, + persistence: { adapter, coordinator }, + }), + ) + collection.startSyncImmediate() + const subscription = collection.subscribeChanges(() => {}) + const owner = sharedSubscription ? { subscription } : {} + const page: LoadSubsetOptions = { + ...owner, + ...(identical ? {} : { limit: 1 }), + } + const all: LoadSubsetOptions = { ...owner } + try { + await collection._sync.loadSubset(page) + await collection._sync.loadSubset(all) + collection._sync.unloadSubset(page) + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: 1, + txId: `sibling-update`, + latestRowVersion: 1, + requiresFullReload: false, + changedRows: [{ key: `1`, value: { id: `1`, title: `After` } }], + deletedKeys: [], + }) + await flushAsyncWork() + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + title: `After`, + }) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`does not retain refresh history as permanent subset demand`, async () => { + const adapter = createRecordingAdapter([{ id: `1`, title: `Before` }]) + const coordinator = createCoordinatorHarness() + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present`, + getKey: (row) => row.id, + syncMode: `on-demand`, + persistence: { adapter, coordinator }, + }), + ) + collection.startSyncImmediate() + try { + await collection._sync.loadSubset({ limit: 1 }) + for (let i = 0; i < 20; i++) + await collection.utils.forceReloadSubset!({ limit: 1 }) + const before = adapter.loadSubsetCalls.length + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: 1, + txId: `refresh-invalidation`, + latestRowVersion: 1, + requiresFullReload: true, + }) + await flushAsyncWork() + await flushAsyncWork() + expect(adapter.loadSubsetCalls.length - before).toBe(1) + } finally { + await collection.cleanup() + } + }) + + it(`reports a non-abort upstream failure rather than treating hydration as remote success`, async () => { + const failure = new Error(`remote acquisition failed`) + const warn = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const collection = createCollection( + persistedCollectionOptions({ + id: `remote-acquisition-failure`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => Promise.reject(failure), + } + }, + }, + persistence: { + adapter: createRecordingAdapter([ + { id: `cached`, title: `Last known row` }, + ]), + }, + }), + ) + collection.startSyncImmediate() + try { + await expect( + Promise.resolve(collection._sync.loadSubset({})), + ).rejects.toBe(failure) + expect(stripVirtualProps(collection.get(`cached`))).toEqual({ + id: `cached`, + title: `Last known row`, + }) + } finally { + warn.mockRestore() + await collection.cleanup() + } + }) + + it.each([`throw`, `reject`] as const)( + `releases only transferred upstream ownership after a load %s`, + async (mode) => { + const failure = new Error(`failed upstream load`) + const warn = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const peer: LoadSubsetOptions = { limit: 1 } + const failed: LoadSubsetOptions = { limit: 1 } + const leases = new Set() + let publish!: (title: string) => Promise + const unload = vi.fn((options: LoadSubsetOptions) => { + leases.delete(options) + }) + const collection = createCollection( + persistedCollectionOptions({ + id: `failed-load-ownership-${mode}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + publish = async (title) => { + if (!leases.has(peer)) return + begin() + write({ + type: collection.has(`live`) ? `update` : `insert`, + value: { id: `live`, title }, + }) + await commit() + } + markReady() + return { + loadSubset: (options) => { + if (options === failed && mode === `throw`) throw failure + // Returning a promise transfers the ongoing lease, even if + // fetching its initial snapshot subsequently fails. + leases.add(options) + return options === failed ? Promise.reject(failure) : true + }, + unloadSubset: unload, + } + }, + }, + persistence: { adapter: createRecordingAdapter() }, + }), + ) + collection.startSyncImmediate() + try { + await collection._sync.loadSubset(peer) + await expect(collection._sync.loadSubset(failed)).rejects.toBe(failure) + expect(leases.has(failed)).toBe(mode === `reject`) + collection._sync.unloadSubset(failed) + expect(unload.mock.calls.map(([options]) => options)).toEqual( + mode === `reject` ? [failed] : [], + ) + expect(leases).toEqual(new Set([peer])) + await publish(`Peer still live`) + expect(collection.get(`live`)?.title).toBe(`Peer still live`) + collection._sync.unloadSubset(peer) + expect(leases.size).toBe(0) + await publish(`Must not arrive`) + expect(collection.get(`live`)?.title).toBe(`Peer still live`) + } finally { + warn.mockRestore() + await collection.cleanup() + } + }, + ) + + it(`does not release or acquire an upstream lease cancelled during hydration`, async () => { + const adapter = createRecordingAdapter() + const hydrate = adapter.loadSubset + let blocked = false + let enterHydration!: () => void + let finishHydration!: () => void + const entered = new Promise((resolve) => { + enterHydration = resolve + }) + const gate = new Promise((resolve) => { + finishHydration = resolve + }) + adapter.loadSubset = async (...args) => { + if (blocked) { + enterHydration() + await gate + } + return hydrate(...args) + } + let leases = 0 + let loads = 0 + const collection = createCollection( + persistedCollectionOptions({ + id: `cancelled-hydration-lease`, + syncMode: `on-demand`, + getKey: (row) => row.id, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + leases++ + return true + }, + unloadSubset: () => { + leases-- + }, + } + }, + }, + persistence: { adapter }, + }), + ) + collection.startSyncImmediate() + const first: LoadSubsetOptions = { limit: 1 } + const second: LoadSubsetOptions = { limit: 1 } + try { + await collection._sync.loadSubset(first) + expect(leases).toBe(1) + blocked = true + const pending = collection._sync.loadSubset(second) + await entered + collection._sync.unloadSubset(second) + expect(leases).toBe(1) + finishHydration() + await pending + expect(loads).toBe(1) + collection._sync.unloadSubset(first) + expect(leases).toBe(0) + } finally { + finishHydration() + await collection.cleanup() + } + }) + + it.each([`abort`, `release`, `offline`] as const)( + `handles remote ensure after %s without resurrecting cancelled demand`, + async (action) => { + vi.useFakeTimers() + const warning = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const failure = Object.assign(new Error(action), { + name: action === `abort` ? `AbortError` : `Error`, + }) + const ensure = vi.fn(async () => { + throw new Error(`offline`) + }) + const coordinator: PersistedCollectionCoordinator = { + getNodeId: () => `cancel-ensure`, + subscribe: () => () => {}, + publish: () => {}, + isLeader: () => true, + ensureLeadership: async () => {}, + requestEnsurePersistedIndex: async () => {}, + requestEnsureRemoteSubset: ensure, + } + const collection = createCollection( + persistedCollectionOptions({ + id: `cancel-ensure-${action}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: async () => { + throw failure + }, + } + }, + }, + persistence: { adapter: createRecordingAdapter(), coordinator }, + }), + ) + const options = { limit: 1 } + try { + collection.startSyncImmediate() + const result = await Promise.resolve( + collection._sync.loadSubset(options), + ).then( + () => `ready`, + (error: unknown) => error, + ) + if (action === `release`) collection._sync.unloadSubset(options) + const callsBeforeRetry = ensure.mock.calls.length + await vi.advanceTimersByTimeAsync(200) + if (action === `offline`) { + expect(result).toBe(failure) + expect(ensure.mock.calls.length).toBeGreaterThan(callsBeforeRetry) + } else { + if (action === `abort`) expect(result).toBe(failure) + expect(ensure).toHaveBeenCalledTimes(callsBeforeRetry) + } + } finally { + await collection.cleanup() + warning.mockRestore() + vi.useRealTimers() + } + }, + ) + it(`retries queued remote subset ensure after transient failures`, async () => { const adapter = createRecordingAdapter() let ensureCalls = 0 @@ -1568,6 +2478,89 @@ describe(`persistedCollectionOptions`, () => { title: `Updated`, }) }) + + it(`keeps a hydrated resume baseline across narrow full reloads`, async () => { + const adapter = createRecordingAdapter([ + { id: `1`, title: `Narrow` }, + { id: `2`, title: `Baseline only` }, + ]) + const loadSubset = adapter.loadSubset.bind(adapter) + adapter.loadSubset = async (...args) => { + const rows = await loadSubset(...args) + return args[1].where ? rows.filter((row) => row.key === `1`) : rows + } + const coordinator = createCoordinatorHarness() + let hydrateBaseline: (() => Promise) | undefined + const collection = createCollection( + persistedCollectionOptions({ + id: `sync-present`, + syncMode: `on-demand`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady, metadata }) => { + hydrateBaseline = ( + metadata?.row as + | { whenHydrated?: () => Promise } + | undefined + )?.whenHydrated + markReady() + return { loadSubset: () => true } + }, + }, + persistence: { adapter, coordinator }, + }), + ) + + collection.startSyncImmediate() + await vi.waitFor(() => expect(hydrateBaseline).toBeTypeOf(`function`)) + await hydrateBaseline!() + expect(collection.has(`2`)).toBe(true) + + await collection._sync.loadSubset({ + where: new IR.Func(`eq`, [new IR.PropRef([`id`]), new IR.Value(`1`)]), + }) + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: 1, + txId: `full-reload`, + latestRowVersion: 1, + requiresFullReload: true, + }) + await flushAsyncWork() + await flushAsyncWork() + + expect(collection.has(`2`)).toBe(true) + await collection.cleanup() + }) + + it(`ignores late wrapped sync writes after cleanup`, async () => { + let lateWrite!: (message: { type: `insert`; value: Todo }) => void + const collection = createCollection( + persistedCollectionOptions({ + id: `late-write-after-cleanup`, + getKey: (item) => item.id, + sync: { + sync: ({ write, markReady }) => { + lateWrite = (message) => write(message) + markReady() + }, + }, + persistence: { adapter: createNoopAdapter() }, + }), + ) + + await collection.preload() + await collection.cleanup() + + expect(() => + lateWrite({ + type: `insert`, + value: { id: `late`, title: `Late` }, + }), + ).not.toThrow() + expect(collection.has(`late`)).toBe(false) + }) }) describe(`persisted key and identifier helpers`, () => { diff --git a/packages/db/CHANGELOG.md b/packages/db/CHANGELOG.md index d59f3991e6..058e12eb7f 100644 --- a/packages/db/CHANGELOG.md +++ b/packages/db/CHANGELOG.md @@ -1,5 +1,302 @@ # @tanstack/db +## 0.9.0 + +### Minor Changes + +- Fix on-demand load settlement, ordered pagination, and replay to preserve coherent results across cancellation, failure, cleanup, and restart. Preserve subset results and ownership across Electric, PowerSync, Query, and SQLite persistence adapters. Correct live-query grouping, include projections, value identity, and indexed comparisons. D2 hashing now rejects structural cycles and excessive traversal depth or work with an explicit error; Collection handles retain object-reference identity without traversing their mutable contents. ([#1797](https://github.com/TanStack/db/pull/1797)) + + Remove the unused public subset-algebra helpers: `isWhereSubset`, `unionWherePredicates`, `minusWherePredicates`, `isOrderBySubset`, `isLimitSubset`, `isOffsetLimitSubset`, `isPredicateSubset`, and `isLoadSubsetRequestSubsumedBy`. Apps that import these helpers must remove those imports; normal queries and adapters are unaffected. `DeduplicatedLoadSubset` remains available and shares only exact demand identities. + + Reject compiled Collection-valued includes as `fn.select()` inputs, including nested descendants, before invoking the callback. Use `toArray()` or `materialize()` in the upstream `.select()` for child-value calculations. To keep live child Collections, use expression `.select()` or perform parent-only functional work before adding the includes. Ordinary Collection-valued includes remain supported. + + Remove proxy `DEBUG` logging and automatic index timing statistics to avoid diagnostic work on reads and writes. Remove `getStats()` and `IndexStats`; use `index.keyCount` for the current entry count, and instrument index methods externally when profiling. Custom index subclasses must remove calls to the retired `trackLookup()` and `updateTimestamp()` helpers. + + Fix mutation drafts so Map/Set `forEach` calls each callback once and read-only iteration reports no changes. Track nested Map `for...of` edits through `collection.update()`. Keep Set entries in place during nested edits, preserving live iteration and usable `has`, `delete`, and `add` handles without duplicate entries or repeated visits. Reverting one entry no longer discards another entry's pending changes. + + Preserve shared values within a row's draft. Newly supplied objects use normal shared references during the update callback, including Map values and Set members; editing through either handle changes the same new object. Copy completed changes at callback return so later caller edits cannot mutate stored data. Existing collection values remain isolated. Copy a new caller-owned value before insertion if it must remain untouched during the callback. + + Keep rejected local-storage mutations out of later successful saves. Stage insert, update, delete, and manual transaction writes before persisting, then promote the shared cache only after storage succeeds. + + Deliver live-query observer publications to peer listeners even when one listener throws. Preserve queued publication order and stop delivery on disposal; report the first listener failure after delivery. + + Wait for PowerSync demands admitted during tracking finalization before reporting them loaded. Preserve untouched object and array key identity when they contain `NaN`. Retire every failed ordered acquisition on explicit retry, while retaining a full-source demand repaired by replay. + + After authoritative ordered recovery succeeds, retire settled page and tie demands so later truncate replay fetches only the full-source replacement. Keep unfinished requests observed until settlement and preserve independent query ownership. + + Remove the live-query `utils.getRunCount()` diagnostic and its runtime counter. Apps that call this diagnostic must remove those calls. Query scheduling and results are unchanged. + + Remove test-only index inspection getters (`indexedKeysSet`, `valueMapData`, `orderedEntriesArray`, and `orderedEntriesArrayReversed`) and unused scheduler diagnostics. `ReverseIndex` now exposes only the `IndexReader` lookup, range, and forward traversal surface returned by `findIndexForField`; mutate the original index instead. Export `IndexReader` for callers that name this return type. Remove the unused subscription `releaseLoadSubset()` method; request owners use the release callback supplied by `onLoadSubsetResult`. Ordinary query and adapter APIs remain unchanged by these removals. + + Remove unused internal helpers and the unused public error classes `WhereClauseConversionError`, `SubscriptionNotFoundError`, and `AggregateNotSupportedError`. These classes have no remaining runtime throw sites; remove any imports of them. Keep the existing query and index behavior and exercise identity/evaluation tests through the production entry points. + + Ensure failed mutations roll back even when their rejection value cannot be converted to a string. Preserve ordinary Error instances; report unprintable rejection values as `Unknown error`. + + Make reentrant effect disposal share the active cleanup result, including calls from abort listeners or source release callbacks. A release failure reaches every waiting disposer while each source still receives one release attempt. + + Reject starting or preloading a collection from inside its active cleanup callbacks with a clear `CollectionStateError`. Nested cleanup cannot admit replacement work that the old teardown would discard. Restart after cleanup completes, or from its final `cleaned-up` status event, remains supported. + + Prevent older page or tie-boundary completions from clearing a newer full-source failure or starting redundant loading. Failed window moves retain their settled public snapshot; an explicit retry releases the failed acquisition once and publishes the completed replacement. + + Restrict direct subscription `requestLimitedSnapshot()` cursor inputs to one order term and one `minValues` entry. Composite and partial-composite cursor inputs now throw before local delivery or adapter work. Use normal live-query ordering and window APIs for multi-column pagination; those remain supported through prefix-and-tie loading. Existing adapters need no changes. + + Treat `LoadSubsetOptions` and their nested request data as immutable from submission onward. Core no longer copies expression trees or mutable constant payloads at the sync and deduplication boundaries. Create a new Date, byte array, membership array, or options object when changing a demand instead of mutating submitted data. Adapters must also leave request data unchanged. Use stable data properties rather than stateful getters. `AbortSignal` cancellation and subscription release remain live. + + Replace replay acquisitions sequentially: release the prior physical lease before starting its replacement. The logical demand and last complete public result remain retained. A failed release prevents replacement startup; failed startup leaves demand available for a later authoritative replay. Custom adapters must support a release/load gap and preserve resources still held by other owners; a sole underlying resource may stop and restart. + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db-ivm@0.1.20 + +## 0.8.7 + +### Patch Changes + +- Match index collation options by their effective values so indexes remain reusable when optional locale fields are omitted, set to `undefined`, or use equivalent locale identifiers. ([#1788](https://github.com/TanStack/db/pull/1788)) + +## 0.8.6 + +### Patch Changes + +- Lazily initialize runtime reference identities to avoid generating random values during Cloudflare Worker module evaluation. ([#1782](https://github.com/TanStack/db/pull/1782)) + +## 0.8.5 + +### Patch Changes + +- Canonicalize equivalent loadSubset queries to one demand identity while preserving observable output aliases, exact projected values, and distinct ordered windows. Query DB now reuses the same canonical identity for its on-demand cache keys. ([#1768](https://github.com/TanStack/db/pull/1768)) + +- Reuse materialized collections when new collection descriptors have the same id. This lets callers recreate dynamic descriptors without creating duplicate collections. ([#1770](https://github.com/TanStack/db/pull/1770)) + +- Settle subset loads only after their committed rows and events are visible. A ([#1769](https://github.com/TanStack/db/pull/1769)) + commit receipt now rejects with `AbortError` when cancellation wins before + application and ignores later aborts. Preserve causal publication, + cancellation, persistence, and error handling across the affected sync + adapters. + +## 0.8.4 + +### Patch Changes + +- Preserve parent-specific include results through nested includes, subqueries, unions, joins, ordering, projections, aggregates, and having clauses. ([#1761](https://github.com/TanStack/db/pull/1761)) + +- Report incremental subset-load failures through subscriptions, live-query utilities, and effects while keeping cached source rows available. Recover cleanly from failed or overlapping must-refetch replays, collection cleanup, effect teardown errors, and cooperative adapter cancellation. Electric's shared-stream snapshot path still depends on upstream request identity or cancellation support to prevent rows from an aborted request from arriving before the request Promise settles. ([#1756](https://github.com/TanStack/db/pull/1756)) + +## 0.8.3 + +### Patch Changes + +- Reject child query builders and query-construction helpers returned from `fn.select()` with type and runtime errors instead of exposing internal query objects. ([#1760](https://github.com/TanStack/db/pull/1760)) + +- Support disabling live queries declared with the `{ query }` config syntax by returning `undefined` or `null` from the query callback. ([#1757](https://github.com/TanStack/db/pull/1757)) + +## 0.8.2 + +### Patch Changes + +- Propagate initial query sync failures through dependent live queries and readiness promises, including recovery and late subscribers, while preserving a ready cached snapshot on later refetch failures. Let sync adapters pass the original failure to `markError(error)` so readiness promises reject with that cause. Isolate adapter callbacks by sync session, preserve synchronous startup errors, and prevent rejected deduplicated subset requests from creating detached promise rejections. ([#1751](https://github.com/TanStack/db/pull/1751)) + +## 0.8.1 + +### Patch Changes + +- Rebuild correlated include materialization as one D2 graph, fixing stale or missing nested results across route changes, batching, lazy loading, optimistic updates, and layered queries. Add canonical structural relation keys, abortable subset demand, and coherent publication for Collection-valued includes. Dispose delayed PowerSync subset hooks after cleanup, and prevent released Query Collection cache results from reaching the collection. ([#1740](https://github.com/TanStack/db/pull/1740)) + +- Support Temporal values in the `gt`/`gte`/`lt`/`lte` query operators. Comparisons now dispatch to the Temporal types' static `compare()` instead of the native relational operators, which throw on Temporal objects (`valueOf()` is designed to throw). `orderBy` uses the same logic, so filtering and ordering now agree — previously `orderBy` compared Temporal values lexicographically by their `toString()`, which mis-ordered equivalent `Duration` forms (`PT60M` vs `PT1H`) and same-instant `ZonedDateTime` values in different zones. ([#1519](https://github.com/TanStack/db/pull/1519)) + + Note two intentional behavior changes for `orderBy` over Temporal columns: + - Ordering `Temporal.PlainMonthDay` values now throws a `TypeError`, since the type has no defined ordering (previously they were silently ordered by string). + - Ordering mixed Temporal types (e.g. `PlainDate` vs `PlainDateTime`) now throws a `TypeError` instead of comparing their string forms. + + Equality (`eq`) is unchanged: `ZonedDateTime` equality still treats the zone as part of identity, and equivalent `Duration` forms remain unequal, mirroring Temporal's `.equals()` vs `.compare()` semantics. + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827)]: + - @tanstack/db-ivm@0.1.19 + +## 0.8.0 + +### Minor Changes + +- Add SSR through request-scoped `DbClient` instances, collection descriptors, ([#1564](https://github.com/TanStack/db/pull/1564)) + explicit collection-row hydration, live-query result snapshots, adapter sync + metadata, and React and Svelte descriptor resolution. + + React live queries now derive identity from structured query IR. Opaque queries + can provide `queryKey`; legacy dependency arrays and unkeyed opaque queries keep + working with development warnings until 1.0. + + Add TanStack Router integration that streams live queries discovered during a + Suspense render as pending promises which resolve to ordered result snapshots. + The browser starts normal source sync and atomically replaces the snapshot when + its live result is ready. + +## 0.7.2 + +### Patch Changes + +- Update agent skills to match current APIs and behavior. ([#1696](https://github.com/TanStack/db/pull/1696)) + +## 0.7.1 + +### Patch Changes + +- Add `useLiveInfiniteQuery` as a Vue binding over the shared live-query window controller. Align infinite-query behavior across React, Vue, and Svelte, including awaitable page fetches, safe page sizes, reactive page-depth preservation, ordered collection validation, shared input resolution, and shared-window cleanup. ([#1724](https://github.com/TanStack/db/pull/1724)) + +## 0.7.0 + +### Minor Changes + +- Add an internal shared live-query observer and migrate all five framework adapters to it ([#1642](https://github.com/TanStack/db/pull/1642)) + + Introduces `createLiveQueryObserver` in `@tanstack/db`: given a resolved live-query collection (or `null` for a disabled query) it owns the subscription lifecycle every adapter used to re-implement — change and status subscriptions, a snapshot with stable identity per state revision for wholesale consumers, and delivery of the raw `ChangeMessage[]` for granular consumers. React, Vue, Svelte, Solid, and Angular's live-query hooks now materialize from the observer instead of their own hand-rolled subscription/status/snapshot machinery, keeping each adapter's native reactivity and each adapter's data-loading policy (wholesale adapters subscribe without initial state; granular adapters seed from it). + + The observer is an **internal, unstable contract** for TanStack DB's official adapters — it is exported so the adapter packages can consume it, but it is not a public extension point yet and its API may change in any release. + + The migration also fixes several live-query lifecycle defects: status-only transitions (`error`, `cleaned-up`) now reach mounted consumers; snapshot identity is stable across unsubscribe/resubscribe and stays fresh while detached; dispatch is FIFO and non-reentrant with subscriptions identified by record rather than callback; disposing during the synchronous initial replay no longer leaks the collection subscription; subscribing after dispose throws instead of registering a dead listener; Solid guards its async resource continuations against superseded collections; and constructing an observer no longer activates sync. Observers activate on their first committed subscription unless an adapter has already started a pre-created collection supplied directly or returned from a callback. + +### Patch Changes + +- fix(db): republish ordered live queries on an order-only move ([#1669](https://github.com/TanStack/db/pull/1669)) + + An `orderBy` live query that reordered its rows without changing any projected + row value (an "order-only move") previously emitted nothing, so `useLiveQuery` + kept rendering the stale order. The live-query collection now publishes an + explicit layout-change notification when this happens, and the shared live-query + observer snapshot exposes a `layoutRevision` that increments on any visible + membership, ordering, or order-only-move change. All five framework adapters + pick this up via their existing wholesale re-read. + +- Add the unstable, internal `createLiveQueryWindowController` primitive for ([#1675](https://github.com/TanStack/db/pull/1675)) + forward pagination. It coordinates collection-scoped window leases, commits + pages only after subset loads succeed, restores windows after failures and + cleanup, and lets React's `useLiveInfiniteQuery` become a thin binding without + changing its public API or resetting pages for structurally equal dependencies. + +## 0.6.17 + +### Patch Changes + +- Skip unchanged index writes while preserving index bookkeeping after failed ([#1691](https://github.com/TanStack/db/pull/1691)) + removals. Cache index evaluators and avoid object normalization work for + primitive values to reduce update overhead. + +## 0.6.16 + +### Patch Changes + +- Fix queries failing to typecheck when the collection's row type is a generic type parameter. Refs inside where/join/select callbacks now expose the properties guaranteed by the type parameter's constraint, and subqueries over generic collections can be used as join sources again (regression introduced in 0.6.6). ([#1678](https://github.com/TanStack/db/pull/1678)) + +- Preserve an explicit `gcTime: 0` on live query collections. The live query config builder used `this.config.gcTime || 5000`, so a `gcTime` of `0` (which disables garbage collection) was treated as unset and silently replaced by the 5s default, causing the collection to be garbage collected instead of kept alive. Use `??` so only `undefined` falls back to the default. ([#1660](https://github.com/TanStack/db/pull/1660)) + +## 0.6.15 + +### Patch Changes + +- Clarify local write status documentation for `$synced` and `isPersisted.promise`, and add core coverage for queued ambiguous server-key sync while optimistic temp-key inserts are pending. ([#1652](https://github.com/TanStack/db/pull/1652)) + +- Extract shared live-query adapter helpers into `@tanstack/db` ([#1641](https://github.com/TanStack/db/pull/1641)) + + Adds `isCollection`, `isSingleResultCollection`, and `getLiveQueryStatusFlags` to `@tanstack/db` and migrates all five framework adapters to use them. `isCollection` replaces the per-adapter duck-typing and Solid's `instanceof CollectionImpl` with one structural, multi-realm-safe guard (the `instanceof` form gave false negatives across dual-package boundaries). No behavior change; internal deduplication only. + +## 0.6.14 + +### Patch Changes + +- Avoid full row origin snapshots during incremental collection updates and make bulk mutation merging linear. ([#1640](https://github.com/TanStack/db/pull/1640)) + +## 0.6.13 + +### Patch Changes + +- Fix `.select()` collapsing discriminated-union fields to the intersection of common keys (#1511). `Ref` now distributes over `T` so `keyof (A | B | C)` no longer reduces the union to its common keys, and `ExtractRef` now distinguishes a real branded `Ref` (where the underlying user type `U` can be returned directly) from a spread-produced inline object (which still needs to be projected through `ResultTypeFromSelect`). This preserves discriminated unions both when the field is selected at the top level and when the field is nested inside another selected object. The real-`Ref` detection uses a strict structural equivalence against the canonical `Ref` shape, so spread-derived objects that keep the same keys but change a field's type (e.g. `{ ...u, code: u.slug }`) or drop an optional key (e.g. `const { nickname, ...rest } = u`) are projected through `ResultTypeFromSelect` instead of being collapsed back to `U`. ([#1597](https://github.com/TanStack/db/pull/1597)) + +- fix(db): keep deeply nested includes in sync when sibling groups share nested correlation keys ([#1607](https://github.com/TanStack/db/pull/1607)) + + Deeply nested includes could drop or stop updating nested rows when sibling parent groups shared the same nested correlation key, especially when one sibling group was inserted after the initial load. Shared nested pipeline buffers were being drained through route state that was scoped too narrowly, so one branch could consume a buffered update before other branches that referenced the same nested row received it. + + Nested route state is now shared at the same scope as the nested buffer and routes updates to every concrete destination branch before clearing the buffer. Snapshot replay still seeds late-arriving sibling groups with already-materialized rows, and recursive pending-change detection ensures deeper routed updates are flushed back up through the result tree. + +## 0.6.12 + +### Patch Changes + +- Fix live query includes reconciliation so updates that re-emit existing child rows update internal child collections instead of attempting duplicate inserts, and ensure duplicate-key sync errors handle collection configs without live query internals. ([#1600](https://github.com/TanStack/db/pull/1600)) + +## 0.6.11 + +### Patch Changes + +- Fix incorrect results from index-optimized `where` clauses that combine indexed and non-indexed conditions. ([#1582](https://github.com/TanStack/db/pull/1582)) + - `OR` expressions are now only served from indexes when every disjunct can use an index; otherwise the query falls back to a full scan. Previously, rows matched only by a non-indexed disjunct were missing from the result. + - `AND` expressions still use indexes for the conditions that have them, but the remaining conditions are now enforced by re-checking each candidate row against the full expression. Previously, non-indexed conditions were silently dropped, returning rows that did not match the query. + - Compound range conditions (e.g. `age > 5 AND age < 10`) combined with conditions on other fields no longer ignore those other conditions. + - Compound range conditions sharing the same boundary value (e.g. `age >= 5 AND age > 5`) now apply the strictest bound regardless of the order the conditions appear in, using the same value comparison semantics as the indexes (dates, locale strings, ...). + - Compound range conditions that only bound one side (e.g. `age > 5 AND age >= 8`) no longer return an empty result. + - Strict range comparisons (`gt`/`lt`) on BTree-indexed fields holding normalized values such as dates now correctly exclude the boundary value. + - Compound range conditions with a `null`/`undefined` bound (e.g. `gt(score, undefined)`) now re-filter against the full expression instead of returning index-ordered rows, matching the semantics of a full scan (a comparison against `null`/`undefined` is never true). + - Index-optimized `eq`, `IN`, and range queries on a field that has rows with `null`/`undefined` values no longer leak those rows into results. BTree indexes store and return such rows (they sort as the smallest key), but a comparison against `null`/`undefined` is never true, so these results are now re-filtered against the full expression to stay equivalent to a full scan. + - String range conditions (`gt`/`gte`/`lt`/`lte`) on a collection using locale string collation (the default) are no longer served by the index. The index orders strings with `localeCompare` while the `where` evaluator compares them with standard relational operators, so an index range lookup could omit matching rows; these conditions now fall back to a full scan. + - Range conditions whose operand is not ordered the same way by the index and the `where` evaluator (arrays, plain objects, Temporal values) now fall back to a full scan instead of using the index, which could otherwise omit matching rows. + - Range conditions on an index created with a custom comparator now fall back to a full scan, since the comparator's ordering may not match the `where` evaluator's relational operators. + +- fix(query): drive lazy-join loading through the collection the join key resolves to ([#1614](https://github.com/TanStack/db/pull/1614)) + + When a subquery used in a JOIN clause selects its join key from a _joined_ source rather than from its own `from` clause, the lazy-join loader subscribed to the wrong inner source: it used the subquery's `from` alias while computing the index requirement against the collection the key actually resolves to. This produced a misleading `Join requires an index` warning naming an already-indexed collection and an unnecessary full-load fallback. `followRef` now reports the resolved source alias, so lazy loading subscribes to the correct collection and loads through its index. + +- Adopt PostgreSQL float semantics for `NaN` in `where` clauses and ordering. ([#1582](https://github.com/TanStack/db/pull/1582)) + + `NaN` (and invalid `Date` values, whose timestamp is `NaN`) previously had no consistent order — `NaN === NaN` is `false` in JavaScript, so `NaN` compared unequal to everything and could not be sorted or indexed deterministically. Following PostgreSQL, `NaN` is now treated as **equal to itself** and **greater than every other non-null value**: + - `eq(row.value, NaN)` matches rows whose value is `NaN`; `inArray(row.value, [NaN, ...])` matches them too. + - Range comparisons treat `NaN` as the greatest value: `gt`/`gte` include it, `lt`/`lte` exclude it. + - Ordering by a field containing `NaN` is now deterministic, with `NaN` sorting last (and `null` still ordered by `NULLS FIRST`/`NULLS LAST`). + + `null`/`undefined` are unaffected: they continue to use three-valued logic (a comparison with `null` yields `UNKNOWN`). + + This makes results independent of whether a query is served from an index or a full scan. + +- Fix prototype pollution via `select()` alias paths. Aliases were split on `.` and walked into the result object without sanitization, so a query like `select(() => ({ ['__proto__.polluted']: ... }))` (or any segment matching `__proto__`, `prototype`, or `constructor`) could mutate `Object.prototype`. The select compiler now rejects unsafe alias path segments with a new `UnsafeAliasPathError`. Fixes #1584. ([#1595](https://github.com/TanStack/db/pull/1595)) + +## 0.6.10 + +### Patch Changes + +- Fix live query `preload()` hanging forever after a source collection was cleaned up (#1576) ([#1606](https://github.com/TanStack/db/pull/1606)) + + When a source collection is cleaned up while a live query depends on it, the live query transitions to an error state and latches an internal `isInErrorState` flag. That flag was never reset, so restarting sync (e.g. calling `preload()` again after cleanup when switching profiles) left the live query unable to become ready and the returned promise never resolved. The flag is now cleared at the start of each sync session so the live query can recover. + +## 0.6.9 + +### Patch Changes + +- Add `subtract`, `multiply`, and `divide` math functions for computed columns ([#1151](https://github.com/TanStack/db/pull/1151)) + + These functions enable complex calculations in `select` and `orderBy` clauses, such as ranking algorithms that combine multiple factors (e.g., HN-style scoring that balances recency and rating). + + ```ts + import { subtract, multiply, divide } from '@tanstack/db' + + // Example: Sort by computed ranking score + const ranked = createLiveQueryCollection((q) => + q + .from({ r: recipesCollection }) + .orderBy( + ({ r }) => + subtract( + multiply(r.rating, r.timesMade), + divide(r.ageInMs, 86400000), + ), + 'desc', + ), + ) + ``` + + - `subtract(a, b)` - Subtraction + - `multiply(a, b)` - Multiplication + - `divide(a, b)` - Division (returns `null` on divide-by-zero) + +- Use a safe `randomUUID` helper that falls back to `crypto.getRandomValues` when `crypto.randomUUID` is unavailable (non-secure browser contexts such as dev servers reached via a LAN IP over HTTP). Fixes #1541. ([#1593](https://github.com/TanStack/db/pull/1593)) + ## 0.6.8 ### Patch Changes diff --git a/packages/db/package.json b/packages/db/package.json index 218beb8072..c9374a4bf9 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/db", - "version": "0.6.8", + "version": "0.9.0", "description": "A reactive client store for building super fast apps on sync", "author": "Kyle Mathews", "license": "MIT", @@ -20,7 +20,10 @@ "build:minified": "vite build --minify", "dev": "vite build --watch", "lint": "eslint . --fix", - "test": "vitest --run" + "test": "vitest --run", + "test:facade-retention": "node --expose-gc --import tsx tests/facade-retention.probe.ts", + "test:oracles": "vitest --run tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-lifecycle-publication.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state.test.ts tests/query/ordered-demand-retirement.test.ts tests/query/ordered-default-work.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts tests/query/includes-space-oracle.test.ts", + "bench:nested-includes": "vitest bench tests/query/includes-performance.bench.ts --run" }, "type": "module", "main": "dist/cjs/index.cjs", diff --git a/packages/db/skills/db-core/SKILL.md b/packages/db/skills/db-core/SKILL.md index 19c2ac27d4..496c7405bc 100644 --- a/packages/db/skills/db-core/SKILL.md +++ b/packages/db/skills/db-core/SKILL.md @@ -3,14 +3,14 @@ name: db-core description: > TanStack DB core concepts: createCollection with queryCollectionOptions, electricCollectionOptions, powerSyncCollectionOptions, rxdbCollectionOptions, - trailbaseCollectionOptions, localOnlyCollectionOptions. Live queries via + trailBaseCollectionOptions, localOnlyCollectionOptions. Live queries via query builder (from, where, join, select, groupBy, orderBy, limit). Optimistic mutations with draft proxy (collection.insert, collection.update, collection.delete). createOptimisticAction, createTransaction, createPacedMutations. Entry point for all TanStack DB skills. type: core library: db -library_version: '0.6.0' +library_version: '0.6.17' --- # TanStack DB — Core Concepts @@ -60,4 +60,4 @@ For framework-specific hooks: ## Version -Targets @tanstack/db v0.6.0. +Targets @tanstack/db v0.6.17. diff --git a/packages/db/skills/db-core/collection-setup/SKILL.md b/packages/db/skills/db-core/collection-setup/SKILL.md index 26d76501b2..9673266c20 100644 --- a/packages/db/skills/db-core/collection-setup/SKILL.md +++ b/packages/db/skills/db-core/collection-setup/SKILL.md @@ -4,16 +4,17 @@ description: > Creating typed collections with createCollection. Adapter selection: queryCollectionOptions (REST/TanStack Query), electricCollectionOptions (ElectricSQL real-time sync), powerSyncCollectionOptions (PowerSync SQLite), - rxdbCollectionOptions (RxDB), trailbaseCollectionOptions (TrailBase), + rxdbCollectionOptions (RxDB), trailBaseCollectionOptions (TrailBase), localOnlyCollectionOptions, localStorageCollectionOptions. CollectionConfig options: getKey, schema, sync, gcTime, autoIndex (default off), defaultIndexType, syncMode (eager/on-demand, plus progressive for Electric). StandardSchema validation with Zod/Valibot/ArkType. Collection lifecycle (idle/loading/ready/error). Adapter-specific sync patterns including Electric txid tracking, Query direct - writes, and PowerSync query-driven sync with onLoad/onLoadSubset hooks. + writes, Query initial data and scoped factories, and PowerSync query-driven + sync with onLoad/onLoadSubset hooks. type: sub-skill library: db -library_version: '0.6.0' +library_version: '0.6.17' sources: - 'TanStack/db:docs/overview.md' - 'TanStack/db:docs/guides/schemas.md' @@ -51,8 +52,8 @@ const todoSchema = z.object({ const todoCollection = createCollection( queryCollectionOptions({ queryKey: ['todos'], - queryFn: async () => { - const res = await fetch('/api/todos') + queryFn: async (ctx) => { + const res = await fetch('/api/todos', { signal: ctx.signal }) return res.json() }, queryClient, @@ -60,16 +61,13 @@ const todoCollection = createCollection( schema: todoSchema, onInsert: async ({ transaction }) => { await api.todos.create(transaction.mutations[0].modified) - await todoCollection.utils.refetch() }, onUpdate: async ({ transaction }) => { const mut = transaction.mutations[0] await api.todos.update(mut.key, mut.changes) - await todoCollection.utils.refetch() }, onDelete: async ({ transaction }) => { await api.todos.delete(transaction.mutations[0].key) - await todoCollection.utils.refetch() }, }), ) @@ -83,7 +81,7 @@ const todoCollection = createCollection( | ElectricSQL (real-time Postgres) | `electricCollectionOptions` | `@tanstack/electric-db-collection` | | PowerSync (SQLite offline) | `powerSyncCollectionOptions` | `@tanstack/powersync-db-collection` | | RxDB (reactive database) | `rxdbCollectionOptions` | `@tanstack/rxdb-db-collection` | -| TrailBase (event streaming) | `trailbaseCollectionOptions` | `@tanstack/trailbase-db-collection` | +| TrailBase (event streaming) | `trailBaseCollectionOptions` | `@tanstack/trailbase-db-collection` | | No backend (UI state) | `localOnlyCollectionOptions` | `@tanstack/db` | | Browser localStorage | `localStorageCollectionOptions` | `@tanstack/db` | @@ -105,6 +103,13 @@ queryCollectionOptions({ | `on-demand` | Search, catalogs, large tables | >50k rows | | `progressive` | Collaborative apps needing instant first paint (Electric only) | Any | +Calling `collection.preload()` on an on-demand collection is a no-op. Create +the live query for the required subset and call `liveQuery.preload()` instead. + +For Query Collection request cancellation, cleanup boundaries, and shared +`QueryClient` behavior, read +[the Query adapter reference](references/query-adapter.md#request-cancellation-and-cleanup). + ## Indexing Indexing is opt-in. The `autoIndex` option defaults to `"off"`. To enable automatic indexing, set `autoIndex: "eager"` and provide a `defaultIndexType`: @@ -214,7 +219,10 @@ queryCollectionOptions({ }) ``` -`queryFn` result is treated as complete server state. Returning `[]` means "server has no items", deleting all existing collection data. +In eager mode, `queryFn` is complete collection state. Returning `[]` means +"the server has no items" and removes all rows. In on-demand mode, a result is +complete only for that exact subset/Query key; an empty result releases that +subset's ownership, while overlapping subsets can keep shared rows. Source: docs/collections/query-collection.md @@ -305,7 +313,9 @@ queryCollectionOptions({ }) ``` -`queryFn` result replaces all collection data. For incremental fetches, merge with existing data. +An eager `queryFn` result replaces all collection data. For incremental eager +fetches, merge with existing data. In on-demand mode, return the complete state +for the requested subset instead. Source: docs/collections/query-collection.md @@ -386,11 +396,20 @@ When a schema transforms types, `TInput` must accept both the pre-transform and Source: docs/guides/schemas.md -### HIGH React Native missing crypto.randomUUID polyfill +### HIGH Runtime has no secure random number generator -TanStack DB uses `crypto.randomUUID()` internally. React Native doesn't provide this. Install `react-native-random-uuid` and import it at your app entry point. +TanStack DB's `safeRandomUUID()` uses `crypto.randomUUID()` when available and +falls back to `crypto.getRandomValues()`, including on non-secure HTTP origins. +Add a Web Crypto polyfill only in runtimes, including some React Native +versions, that provide neither API. -Source: docs/overview.md +```ts +import { safeRandomUUID } from '@tanstack/db' + +collection.insert({ id: safeRandomUUID(), text: 'New item' }) +``` + +Source: packages/db/src/utils/uuid.ts, packages/db/tests/uuid.test.ts ### MEDIUM Providing both explicit type parameter and schema diff --git a/packages/db/skills/db-core/collection-setup/references/electric-adapter.md b/packages/db/skills/db-core/collection-setup/references/electric-adapter.md index 38f0e07b20..5886791c35 100644 --- a/packages/db/skills/db-core/collection-setup/references/electric-adapter.md +++ b/packages/db/skills/db-core/collection-setup/references/electric-adapter.md @@ -30,10 +30,23 @@ const collection = createCollection( | `id` | (none) | Unique collection identifier | | `schema` | (none) | StandardSchema validator | | `shapeOptions.params` | (none) | Additional shape params (e.g. `{ table: 'todos' }`) | +| `syncMode` | `eager` | `eager`, `on-demand`, or `progressive` | | `onInsert` | (none) | Persistence handler; should return `{ txid }` | | `onUpdate` | (none) | Persistence handler; should return `{ txid }` | | `onDelete` | (none) | Persistence handler; should return `{ txid }` | +## Sync Modes + +- `eager` loads the full shape before use. +- `on-demand` loads only the subsets requested by live queries. +- `progressive` loads the requested subset first, then completes the full sync + in the background. + +With SQLite persistence, progressive resume keeps hydrated rows visible while +the full Electric stream resumes. On-demand subset snapshots use a bounded +refresh wait, so a stalled native long-poll refresh does not block loading +forever. + ## Three Sync Strategies ### 1. Txid Return (Recommended) diff --git a/packages/db/skills/db-core/collection-setup/references/local-adapters.md b/packages/db/skills/db-core/collection-setup/references/local-adapters.md index cc3ffea28e..f145aa8aad 100644 --- a/packages/db/skills/db-core/collection-setup/references/local-adapters.md +++ b/packages/db/skills/db-core/collection-setup/references/local-adapters.md @@ -24,19 +24,18 @@ import { const collection = createCollection( localOnlyCollectionOptions({ - id: 'ui-state', getKey: (item) => item.id, }), ) ``` -- `id` -- unique collection identifier - `getKey` -- extracts unique key from each item ### Optional Config | Option | Default | Description | | ------------- | ------- | -------------------------------------- | +| `id` | UUID | Unique collection identifier | | `schema` | (none) | StandardSchema validator | | `initialData` | (none) | Array of items to populate on creation | | `onInsert` | (none) | Handler before confirming inserts | @@ -101,27 +100,26 @@ import { const collection = createCollection( localStorageCollectionOptions({ - id: 'user-preferences', storageKey: 'app-user-prefs', getKey: (item) => item.id, }), ) ``` -- `id` -- unique collection identifier - `storageKey` -- localStorage key for all collection data - `getKey` -- extracts unique key from each item ### Optional Config -| Option | Default | Description | -| ----------------- | -------------- | -------------------------------------------------------------------- | -| `schema` | (none) | StandardSchema validator | -| `storage` | `localStorage` | Custom storage (`sessionStorage` or any localStorage-compatible API) | -| `storageEventApi` | `window` | Event API for cross-tab sync | -| `onInsert` | (none) | Handler on insert | -| `onUpdate` | (none) | Handler on update | -| `onDelete` | (none) | Handler on delete | +| Option | Default | Description | +| ----------------- | ---------------- | -------------------------------------------------------------------- | +| `id` | From storage key | `local-collection:${storageKey}` | +| `schema` | (none) | StandardSchema validator | +| `storage` | `localStorage` | Custom storage (`sessionStorage` or any localStorage-compatible API) | +| `storageEventApi` | `window` | Event API for cross-tab sync | +| `onInsert` | (none) | Handler on insert | +| `onUpdate` | (none) | Handler on update | +| `onDelete` | (none) | Handler on delete | ### Using sessionStorage diff --git a/packages/db/skills/db-core/collection-setup/references/powersync-adapter.md b/packages/db/skills/db-core/collection-setup/references/powersync-adapter.md index 8a61414c88..72e6b6e82a 100644 --- a/packages/db/skills/db-core/collection-setup/references/powersync-adapter.md +++ b/packages/db/skills/db-core/collection-setup/references/powersync-adapter.md @@ -9,7 +9,7 @@ pnpm add @tanstack/powersync-db-collection @powersync/web @journeyapps/wa-sqlite ## Required Config ```typescript -import { createCollection } from '@tanstack/react-db' +import { createCollection, safeRandomUUID } from '@tanstack/react-db' import { powerSyncCollectionOptions } from '@tanstack/powersync-db-collection' import { Schema, Table, column, PowerSyncDatabase } from '@powersync/web' @@ -164,7 +164,7 @@ const APP_SCHEMA = new Schema({ }) await collection.insert( - { id: crypto.randomUUID(), name: 'Report' }, + { id: safeRandomUUID(), name: 'Report' }, { metadata: { source: 'web-app', userId: 'user-123' } }, ).isPersisted.promise ``` @@ -187,7 +187,7 @@ const tx = createTransaction({ }) tx.mutate(() => { documentsCollection.insert({ - id: crypto.randomUUID(), + id: safeRandomUUID(), name: 'Doc 1', created_at: new Date().toISOString(), }) diff --git a/packages/db/skills/db-core/collection-setup/references/query-adapter.md b/packages/db/skills/db-core/collection-setup/references/query-adapter.md index dc979223c0..3f0b07f08e 100644 --- a/packages/db/skills/db-core/collection-setup/references/query-adapter.md +++ b/packages/db/skills/db-core/collection-setup/references/query-adapter.md @@ -17,7 +17,8 @@ const queryClient = new QueryClient() const collection = createCollection( queryCollectionOptions({ queryKey: ['todos'], - queryFn: async () => fetch('/api/todos').then((r) => r.json()), + queryFn: async (ctx) => + fetch('/api/todos', { signal: ctx.signal }).then((r) => r.json()), queryClient, getKey: (item) => item.id, }), @@ -31,26 +32,35 @@ const collection = createCollection( ## Optional Config (with defaults) -| Option | Default | Description | -| ----------------- | ------------ | ----------------------------------------------- | -| `id` | (none) | Unique collection identifier | -| `schema` | (none) | StandardSchema validator | -| `select` | (none) | Extracts array items when wrapped with metadata | -| `enabled` | `true` | Whether query runs automatically | -| `refetchInterval` | `0` | Polling interval in ms; 0 = disabled | -| `retry` | (TQ default) | Retry config for failed queries | -| `retryDelay` | (TQ default) | Delay between retries | -| `staleTime` | (TQ default) | How long data is considered fresh | -| `meta` | (none) | Metadata passed to queryFn context | -| `startSync` | `true` | Start syncing immediately | -| `syncMode` | (none) | Set `"on-demand"` for predicate push-down | +| Option | Default | Description | +| ---------------------- | ------------ | ----------------------------------------------- | +| `id` | (none) | Unique collection identifier | +| `schema` | (none) | StandardSchema validator | +| `select` | (none) | Extracts rows from the original response shape | +| `enabled` | `true` | Whether query runs automatically | +| `refetchInterval` | (TQ default) | Polling interval | +| `retry` / `retryDelay` | (TQ default) | Retry policy | +| `staleTime` / `gcTime` | (TQ default) | Freshness and unused-cache retention | +| `refetchOnWindowFocus` | (TQ default) | Refetch when the window regains focus | +| `refetchOnReconnect` | (TQ default) | Refetch after reconnecting | +| `refetchOnMount` | (TQ default) | Refetch when the observer mounts | +| `networkMode` | (TQ default) | TanStack Query network mode | +| `initialData` | (none) | Initial response for eager collections | +| `initialDataUpdatedAt` | (none) | Timestamp used to judge initial-data freshness | +| `meta` | (none) | Metadata merged into the query function context | +| `startSync` | `true` | Start syncing immediately | +| `syncMode` | `eager` | Set `"on-demand"` for predicate push-down | + +Query Client defaults apply when these pass-through fields are omitted. +`placeholderData` is intentionally unsupported: it is observer-local UI state, +not cache data, so it must not become collection-wide rows. ### Persistence Handlers ```typescript onInsert: async ({ transaction }) => { await api.createTodos(transaction.mutations.map((m) => m.modified)) - // return nothing or { refetch: true } to trigger refetch + // Query Collection automatically refetches and awaits the result. // return { refetch: false } to skip refetch }, onUpdate: async ({ transaction }) => { @@ -80,6 +90,159 @@ collection.utils.writeBatch(() => { }) ``` +## Response Shape, Initial Data, and Query Options + +`select` is a Query Collection row-extraction hook, not TanStack Query's +observer-level projection. The Query cache keeps the original response while +the collection materializes the returned row array: + +```typescript +const collection = createCollection( + queryCollectionOptions({ + queryKey: ['todos'], + queryFn: fetchTodosResponse, + initialData: { + items: [{ id: '1', title: 'Initial todo' }], + total: 1, + }, + initialDataUpdatedAt: Date.now(), + staleTime: 60_000, + select: (response) => response.items, + queryClient, + getKey: (todo) => todo.id, + }), +) +``` + +The same `select` applies to fetched and initial responses. Cached or hydrated +data for the same exact Query key takes precedence over a later `initialData` +value. `initialData` is supported only in eager mode; seed the exact derived +Query cache entries for on-demand subsets. + +Passing either `initialData` or `initialDataUpdatedAt` in on-demand mode throws +`InitialDataInOnDemandModeError`. When stale initial data triggers a fetch, the +rows remain visible while it runs. A failed fetch retains them; a successful +fetch reconciles them normally. + +Direct writes can preserve simple wrappers such as `{ data: [...] }`, +`{ items: [...] }`, or `{ results: [...] }`. For a derived projection such as +`response.edges.map((edge) => edge.node)`, refetch or invalidate when the +wrapped cache must reflect the write exactly. + +You may spread compatible `queryOptions(...)` output into +`queryCollectionOptions`, but provide `queryFn` explicitly. Do not pass a +TanStack Query observer-level `select`; Query Collection gives that name the +row-extraction contract above. + +## Runtime QueryClient and Business Scopes + +When a `QueryClient` is request-, router-, tenant-, or test-scoped, put shared +options in a factory and create one stable collection per client and business +scope: + +```typescript +function createProjectTodosCollection( + queryClient: QueryClient, + projectId: string, +) { + return createCollection( + queryCollectionOptions({ + queryKey: ['projects', projectId, 'todos'], + queryFn: () => fetchProjectTodos(projectId), + queryClient, + getKey: (todo) => todo.id, + }), + ) +} + +type ProjectTodosCollection = ReturnType + +const projectCollections = new WeakMap< + QueryClient, + Map +>() + +export function getProjectTodosCollection( + queryClient: QueryClient, + projectId: string, +): ProjectTodosCollection { + let collectionsByProject = projectCollections.get(queryClient) + if (!collectionsByProject) { + collectionsByProject = new Map() + projectCollections.set(queryClient, collectionsByProject) + } + + let collection = collectionsByProject.get(projectId) + if (!collection) { + collection = createProjectTodosCollection(queryClient, projectId) + collectionsByProject.set(projectId, collection) + } + + return collection +} + +export async function removeProjectTodosCollection( + queryClient: QueryClient, + projectId: string, +): Promise { + const collectionsByProject = projectCollections.get(queryClient) + if (!collectionsByProject) return + + const collection = collectionsByProject.get(projectId) + if (!collection) return + + collectionsByProject.delete(projectId) + if (collectionsByProject.size === 0) { + projectCollections.delete(queryClient) + } + await collection.cleanup() +} +``` + +Memoize by `QueryClient` and every scope value. Do not create the collection +during every render or in each consumer. Clean up and remove unused entries +from long-lived scope maps when your application owns their lifecycle. + +A business scope names a distinct server resource. A relational subset +(`where`, `orderBy`, `limit`, or `offset`) stays within that collection and, in +on-demand mode, reaches `queryFn` as `ctx.meta.loadSubsetOptions`. Do not create +a collection for each relational subset. + +## Request Cancellation and Cleanup + +TanStack Query passes an `AbortSignal` through the query function context. +Forward it to `fetch` or another abortable client: + +```typescript +queryFn: async (ctx) => { + const response = await fetch('/api/todos', { signal: ctx.signal }) + return response.json() +}, +``` + +Explicit `collection.cleanup()` cancels each exact Query key the collection is +currently tracking, then removes it from the Query cache. The underlying client +stops work only when it consumes `ctx.signal`. + +An unloaded on-demand subset is no longer tracked, so later collection cleanup +does not revisit its Query key. Unloading does not explicitly call +`queryClient.cancelQueries()`: it removes the subset's Query observer. If that +was the final observer and the query function consumed `ctx.signal`, Query Core +aborts the request. If the signal was ignored, or another observer still uses +the same exact key, the request may finish and remain cached until `gcTime`. + +Query cache entries are shared within a `QueryClient`. Explicit cleanup can +cancel or remove entries used by another collection or Query consumer with the +same exact key. + +## Query Invalidation + +Exact-key and prefix invalidation refetch active eager and on-demand queries, +then rematerialize their results. Overlapping subsets keep rows that another +active subset still owns. A failed refetch retains current rows and records the +error in `collection.utils.lastError`. Cleaned-up or otherwise inactive queries +do not rematerialize. + ## Predicate Push-Down (syncMode: "on-demand") Query predicates (where, orderBy, limit, offset) passed to `queryFn` via `ctx.meta.loadSubsetOptions`. @@ -151,7 +314,9 @@ const productsCollection = createCollection( ) } if (limit) params.set('limit', String(limit)) - return fetch(`/api/products?${params}`).then((r) => r.json()) + return fetch(`/api/products?${params}`, { + signal: ctx.signal, + }).then((r) => r.json()) }, onInsert: async ({ transaction }) => { const serverItems = await api.createProducts( @@ -210,6 +375,10 @@ When using a function-based `queryKey`, all derived keys must share the base key ## Key Behaviors -- `queryFn` result is treated as **complete state** -- missing items are deleted -- Empty array from `queryFn` deletes all items +- In eager mode, each `queryFn` result is complete collection state +- In on-demand mode, it is complete state for that exact subset/Query key +- An empty subset removes that subset's ownership; overlapping subsets can keep + the same rows materialized - Direct writes update TQ cache but are overridden by subsequent `queryFn` results +- Persistence handlers automatically refetch unless they return `{ refetch: false }` +- On-demand `collection.preload()` is a no-op; preload the live query instead diff --git a/packages/db/skills/db-core/collection-setup/references/rxdb-adapter.md b/packages/db/skills/db-core/collection-setup/references/rxdb-adapter.md index fcdcf84b02..18147bb70e 100644 --- a/packages/db/skills/db-core/collection-setup/references/rxdb-adapter.md +++ b/packages/db/skills/db-core/collection-setup/references/rxdb-adapter.md @@ -23,15 +23,15 @@ const todosCollection = createCollection( ## Optional Config (with defaults) -| Option | Default | Description | -| --------------- | ----------------------- | -------------------------------------------------------------------------------------------------- | -| `id` | (none) | Unique collection identifier | -| `schema` | (none) | StandardSchema validator (RxDB has its own validation; this adds TanStack DB-side validation) | -| `startSync` | `true` | Start ingesting RxDB data immediately | -| `syncBatchSize` | `1000` | Max documents per batch during initial sync from RxDB; only affects initial load, not live updates | -| `onInsert` | (default: `bulkUpsert`) | Override default insert persistence | -| `onUpdate` | (default: `patch`) | Override default update persistence | -| `onDelete` | (default: `bulkRemove`) | Override default delete persistence | +| Option | Default | Description | +| --------------- | ------- | -------------------------------------------------------------------------------------------------- | +| `id` | (none) | Unique collection identifier | +| `schema` | (none) | StandardSchema validator (RxDB has its own validation; this adds TanStack DB-side validation) | +| `startSync` | `false` | Start ingesting RxDB data immediately | +| `syncBatchSize` | `1000` | Max documents per batch during initial sync from RxDB; only affects initial load, not live updates | + +The adapter owns `onInsert`, `onUpdate`, and `onDelete` so writes persist to +RxDB. Those handlers cannot be overridden in `RxDBCollectionConfig`. ## Key Behavior: String Keys @@ -96,7 +96,7 @@ RxDB schema indexes do not affect TanStack DB query performance (queries run in- ```typescript import { createRxDatabase } from 'rxdb/plugins/core' import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage' -import { createCollection } from '@tanstack/react-db' +import { createCollection, safeRandomUUID } from '@tanstack/react-db' import { rxdbCollectionOptions } from '@tanstack/rxdb-db-collection' import { z } from 'zod' @@ -141,7 +141,7 @@ const todosCollection = createCollection( // Usage todosCollection.insert({ - id: crypto.randomUUID(), + id: safeRandomUUID(), text: 'Buy milk', completed: false, }) diff --git a/packages/db/skills/db-core/collection-setup/references/trailbase-adapter.md b/packages/db/skills/db-core/collection-setup/references/trailbase-adapter.md index a01b29ee6b..0ded8f17f7 100644 --- a/packages/db/skills/db-core/collection-setup/references/trailbase-adapter.md +++ b/packages/db/skills/db-core/collection-setup/references/trailbase-adapter.md @@ -17,27 +17,27 @@ const trailBaseClient = initClient('https://your-trailbase-instance.com') const todosCollection = createCollection( trailBaseCollectionOptions({ - id: 'todos', recordApi: trailBaseClient.records('todos'), getKey: (item) => item.id, + parse: {}, + serialize: {}, }), ) ``` -- `id` -- unique collection identifier - `recordApi` -- TrailBase Record API instance from `trailBaseClient.records(tableName)` - `getKey` -- extracts unique key from each item +- `parse` -- field conversions from TrailBase records to collection rows +- `serialize` -- field conversions from collection rows to TrailBase records + +Use empty objects for `parse` and `serialize` when both shapes are identical. ## Optional Config -| Option | Default | Description | -| ----------- | ------- | --------------------------------------------------------------------------------- | -| `schema` | (none) | StandardSchema validator | -| `parse` | (none) | Object mapping field names to functions that transform data coming FROM TrailBase | -| `serialize` | (none) | Object mapping field names to functions that transform data going TO TrailBase | -| `onInsert` | (none) | Handler called on insert | -| `onUpdate` | (none) | Handler called on update | -| `onDelete` | (none) | Handler called on delete | +| Option | Default | Description | +| ---------- | ------- | ---------------------------- | +| `id` | (none) | Unique collection identifier | +| `syncMode` | `eager` | `eager` or `on-demand` | ## Conversions (parse/serialize) @@ -58,8 +58,8 @@ type Todo = { completed: boolean } -const collection = createCollection( - trailBaseCollectionOptions({ +const collection = createCollection( + trailBaseCollectionOptions({ id: 'todos', recordApi: trailBaseClient.records('todos'), getKey: (item) => item.id, @@ -79,36 +79,28 @@ Automatic when `enable_subscriptions` is enabled on the TrailBase server. No add ## Persistence Handlers -```typescript -onInsert: async ({ transaction }) => { - const newItem = transaction.mutations[0].modified -}, -onUpdate: async ({ transaction }) => { - const { original, modified } = transaction.mutations[0] -}, -onDelete: async ({ transaction }) => { - const deletedItem = transaction.mutations[0].original -}, -``` +TrailBase owns `onInsert`, `onUpdate`, and `onDelete`. The adapter writes +through the Record API and waits until subscription events confirm the affected +IDs before removing the optimistic overlay. Custom mutation handlers and +`schema` are not part of `TrailBaseCollectionConfig`. -TrailBase handles persistence through the Record API automatically. Custom handlers are for additional logic only. +Call `collection.utils.cancel()` to cancel the active TrailBase event reader. ## Complete Example ```typescript -import { createCollection } from '@tanstack/react-db' +import { createCollection, safeRandomUUID } from '@tanstack/react-db' import { trailBaseCollectionOptions } from '@tanstack/trailbase-db-collection' import { initClient } from 'trailbase' -import { z } from 'zod' const trailBaseClient = initClient('https://your-trailbase-instance.com') -const todoSchema = z.object({ - id: z.string(), - text: z.string(), - completed: z.boolean(), - created_at: z.date(), -}) +type Todo = { + id: string + text: string + completed: boolean + created_at: Date +} type SelectTodo = { id: string @@ -117,29 +109,23 @@ type SelectTodo = { created_at: number } -type Todo = z.infer - -const todosCollection = createCollection( - trailBaseCollectionOptions({ +const todosCollection = createCollection( + trailBaseCollectionOptions({ id: 'todos', recordApi: trailBaseClient.records('todos'), getKey: (item) => item.id, - schema: todoSchema, parse: { created_at: (ts) => new Date(ts * 1000), }, serialize: { created_at: (date) => Math.floor(date.valueOf() / 1000), }, - onInsert: async ({ transaction }) => { - console.log('Created:', transaction.mutations[0].modified) - }, }), ) // Usage todosCollection.insert({ - id: crypto.randomUUID(), + id: safeRandomUUID(), text: 'Review PR', completed: false, created_at: new Date(), diff --git a/packages/db/skills/db-core/custom-adapter/SKILL.md b/packages/db/skills/db-core/custom-adapter/SKILL.md index 8304c75e23..212f3662c2 100644 --- a/packages/db/skills/db-core/custom-adapter/SKILL.md +++ b/packages/db/skills/db-core/custom-adapter/SKILL.md @@ -2,17 +2,19 @@ name: db-core/custom-adapter description: > Building custom collection adapters for new backends. SyncConfig interface: - sync function receiving begin, write, commit, markReady, truncate, metadata - primitives. ChangeMessage format (insert, update, delete). loadSubset for - on-demand sync. LoadSubsetOptions (where, orderBy, limit, cursor). Expression - parsing: parseWhereExpression, parseOrderByExpression, + sync function receiving begin, write, commit, markReady, markError, truncate, metadata + primitives and returning cleanup, loadSubset, and optional unloadSubset + handlers. + ChangeMessage format (insert, update, delete). On-demand LoadSubsetOptions + (where, orderBy, limit, offset, cursor). Expression parsing: + parseWhereExpression, parseOrderByExpression, extractSimpleComparisons, parseLoadSubsetOptions. Collection options creator pattern. rowUpdateMode (partial vs full). Subscription lifecycle and cleanup functions. Persisted sync metadata API (metadata.row and metadata.collection) for storing per-row and per-collection adapter state. type: sub-skill library: db -library_version: '0.6.0' +library_version: '0.6.17' sources: - 'TanStack/db:docs/guides/collection-options-creator.md' - 'TanStack/db:packages/db/src/collection/sync.ts' @@ -26,23 +28,30 @@ This skill builds on db-core and db-core/collection-setup. Read those first. ```ts import { createCollection } from '@tanstack/db' -import type { SyncConfig, CollectionConfig } from '@tanstack/db' +import type { CollectionConfig } from '@tanstack/db' interface MyItem { id: string name: string } -function myBackendCollectionOptions(config: { +interface BackendEvent { + type: 'insert' | 'update' | 'delete' + id: string + data: T +} + +function myBackendCollectionOptions(config: { endpoint: string getKey: (item: T) => string -}): CollectionConfig { +}): CollectionConfig { return { getKey: config.getKey, sync: { - sync: ({ begin, write, commit, markReady, metadata, collection }) => { + sync: ({ begin, write, commit, markReady, markError, collection }) => { let isInitialSyncComplete = false - const bufferedEvents: Array = [] + const bufferedEvents: Array> = [] + const initialSyncAbort = new AbortController() // 1. Subscribe to real-time events FIRST const unsubscribe = myWebSocket.subscribe(config.endpoint, (event) => { @@ -56,50 +65,65 @@ function myBackendCollectionOptions(config: { }) // 2. Fetch initial data - fetch(config.endpoint).then(async (res) => { - const items = await res.json() - begin() - for (const item of items) { - write({ type: 'insert', value: item }) - } - commit() - - // 3. Process buffered events - isInitialSyncComplete = true - for (const event of bufferedEvents) { + void fetch(config.endpoint, { signal: initialSyncAbort.signal }) + .then(async (res) => { + const items = await res.json() begin() - write({ type: event.type, key: event.id, value: event.data }) + for (const item of items) { + write({ type: 'insert', value: item }) + } commit() - } - // 4. Signal readiness - markReady() - }) + // 3. Process buffered events + isInitialSyncComplete = true + for (const event of bufferedEvents) { + begin() + write({ type: event.type, key: event.id, value: event.data }) + commit() + } + + // 4. Signal that a usable snapshot exists + markReady() + }) + .catch((error) => { + if (initialSyncAbort.signal.aborted) return + console.error('Initial sync failed:', error) + // Only initial startup owns collection readiness. A later refetch + // failure must keep the last ready snapshot usable. + if (collection.status === 'loading') markError(error) + }) // 5. Return cleanup function return () => { + initialSyncAbort.abort() unsubscribe() } }, rowUpdateMode: 'partial', }, onInsert: async ({ transaction }) => { - await fetch(config.endpoint, { + const response = await fetch(config.endpoint, { method: 'POST', body: JSON.stringify(transaction.mutations[0].modified), }) + await waitForServerObservation(response) }, onUpdate: async ({ transaction }) => { const mut = transaction.mutations[0] - await fetch(`${config.endpoint}/${mut.key}`, { + const response = await fetch(`${config.endpoint}/${mut.key}`, { method: 'PATCH', body: JSON.stringify(mut.changes), }) + await waitForServerObservation(response) }, onDelete: async ({ transaction }) => { - await fetch(`${config.endpoint}/${transaction.mutations[0].key}`, { - method: 'DELETE', - }) + const response = await fetch( + `${config.endpoint}/${transaction.mutations[0].key}`, + { + method: 'DELETE', + }, + ) + await waitForServerObservation(response) }, } } @@ -127,28 +151,61 @@ write({ type: 'delete', key: itemId, value: item }) ### On-demand sync with loadSubset ```ts -import { parseLoadSubsetOptions } from "@tanstack/db" +import { parseLoadSubsetOptions } from '@tanstack/db' +syncMode: 'on-demand', sync: { - sync: ({ begin, write, commit, markReady }) => { - // Initial sync... + sync: ({ begin, write, commit, markReady, collection }) => { + const stopSync = subscribeToBackendChanges() markReady() - return () => {} - }, - loadSubset: async (options) => { - const { filters, sorts, limit, offset } = parseLoadSubsetOptions(options) - // filters: [{ field: ['category'], operator: 'eq', value: 'electronics' }] - // sorts: [{ field: ['price'], direction: 'asc', nulls: 'last' }] - const params = new URLSearchParams() - for (const f of filters) { - params.set(f.field.join("."), `${f.operator}:${f.value}`) + + return { + cleanup: stopSync, + loadSubset: async (options) => { + const { filters, sorts, limit } = parseLoadSubsetOptions(options) + const items = await api.items.list({ + filters, + sorts, + limit, + offset: options.offset, + // Translate cursor.whereFrom/whereCurrent expressions for your API. + cursor: translateCursorExpressions(options.cursor), + }) + + begin() + for (const item of items) { + const key = collection.config.getKey(item) + write( + collection.has(key) + ? { type: 'update', key, value: item } + : { type: 'insert', value: item }, + ) + } + commit() + }, } - const res = await fetch(`/api/items?${params}`) - return res.json() }, + rowUpdateMode: 'full', } ``` +`sync()` returns the handlers in a `SyncConfigRes` object. `loadSubset()` must +write fetched rows through `begin()` → `write()` → `commit()` and resolve +`void` (or return `true` for an immediate synchronous result); it does not +return the fetched rows. `parseLoadSubsetOptions()` returns only `filters`, +`sorts`, and `limit`. Read `offset` and `cursor` from the original options. +`cursor` contains query expressions (`whereFrom` and `whereCurrent`), not an +opaque backend cursor; translate or combine those expressions for your API. +Return `unloadSubset` only when `loadSubset` creates an ongoing resource, such +as a per-subset server subscription, that must be released. +Ownership transfers to core only when `loadSubset` returns `true` or a promise. +If it throws synchronously after partial setup, release that partial resource +before throwing; core will not call `unloadSubset` for a request that never +returned. A must-refetch can call `loadSubset` again with the same options. Each +successful return is a fresh acquisition: core releases the previous +acquisition when its replacement returns, then releases the current one when +the demand ends. + ### Managing optimistic state duration Mutation handlers must not resolve until server changes have synced back to the collection. Five strategies: @@ -163,10 +220,16 @@ Mutation handlers must not resolve until server changes have synced back to the The `metadata` API on the sync config allows adapters to store per-row and per-collection metadata that persists across sync transactions. This is useful for tracking resume tokens, cursors, LSNs, or other adapter-specific state. -The `metadata` object is available as a property on the sync config argument alongside `begin`, `write`, `commit`, etc. It is always provided, but without persistence the metadata is in-memory only and does not survive reloads. With persistence, metadata is durable across sessions. +The `metadata` object is available on the sync config argument alongside +`begin`, `write`, and `commit`. Core supplies it at runtime, but its public type +is optional, so strict TypeScript code must guard it or assert its presence. +Without persistence the metadata is in-memory only and does not survive +reloads. With persistence, it is durable across sessions. ```ts -sync: ({ begin, write, commit, markReady, metadata }) => { +sync: ({ begin, write, commit, markReady, markError, metadata }) => { + if (!metadata) throw new Error('Sync metadata API is unavailable') + // Row metadata: store per-row state (e.g. server version, ETag) metadata.row.get(key) // => unknown | undefined metadata.row.set(key, { version: 3, etag: 'abc' }) @@ -181,7 +244,11 @@ sync: ({ begin, write, commit, markReady, metadata }) => { } ``` -Row metadata writes are tied to the current transaction. When a row is deleted via `write({ type: 'delete', ... })`, its row metadata is automatically deleted. When a row is inserted, its metadata is set from `message.metadata` if provided, or deleted otherwise. +Row metadata writes are tied to the current transaction. Deleting a row also +deletes its metadata. An insert sets metadata from `message.metadata`. A +metadata-less insert deletes stale metadata unless `metadata.row.set()` already +queued an explicit value for that key in the same transaction; that queued +value wins. Collection metadata writes staged before `truncate()` are preserved and commit atomically with the truncate transaction. @@ -189,6 +256,8 @@ Collection metadata writes staged before `truncate()` are preserved and commit a ```ts sync: ({ begin, write, commit, markReady, metadata }) => { + if (!metadata) throw new Error('Sync metadata API is unavailable') + const lastCursor = metadata.collection.get('cursor') as string | undefined const stream = subscribeFromCursor(lastCursor) @@ -202,6 +271,7 @@ sync: ({ begin, write, commit, markReady, metadata }) => { }) stream.on('ready', () => markReady()) + stream.on('initial-error', (error) => markError(error)) return () => stream.close() } ``` @@ -225,6 +295,21 @@ const orderBy = parseOrderByExpression(options.orderBy) ## Common Mistakes +### CRITICAL Defining loadSubset beside sync() + +Wrong: + +```ts +sync: { + sync: ({ markReady }) => markReady(), + loadSubset: async () => fetch('/items').then((response) => response.json()), +} +``` + +Correct: return `{ loadSubset, cleanup }` from `sync()` and apply loaded rows +with the sync transaction primitives, as shown above. Add `unloadSubset` when +each loaded subset owns a resource that must be released. + ### CRITICAL Not calling markReady() in sync implementation Wrong: @@ -255,6 +340,12 @@ sync: ({ begin, write, commit, markReady }) => { `markReady()` transitions the collection to "ready" status. Without it, live queries never resolve and `useLiveSuspenseQuery` hangs forever in Suspense. +If initial sync fails before it produces a usable snapshot, call +`markError(error)` instead. This rejects readiness waits with the supplied cause +and moves dependent live queries to the error state. Calling `markError()` +without a cause remains supported and rejects with a generic collection-state +error. A later successful sync can call `markReady()` to recover. + Source: docs/guides/collection-options-creator.md ### HIGH Race condition: subscribing after initial fetch @@ -327,6 +418,14 @@ Sync data must be written within a transaction (`begin` → `write` → `commit` Source: packages/db/src/collection/sync.ts:110 +### HIGH Inserting a different value for an existing synced key + +An `insert` for an existing synced key is normalized to an update only when +the value is unchanged. A different value throws `DuplicateKeySyncError`, +including for plain custom configs with no `utils`. + +Emit an `update`, or delete/truncate the old row before inserting the new one. + ## Tension: Simplicity vs. Correctness in Sync Getting-started simplicity (localOnly, eager mode) conflicts with production correctness (on-demand sync, race condition prevention, proper markReady handling). Agents optimizing for quick setup tend to skip buffering, markReady, and cleanup functions. diff --git a/packages/db/skills/db-core/live-queries/SKILL.md b/packages/db/skills/db-core/live-queries/SKILL.md index fe55967ca9..92f234fcfc 100644 --- a/packages/db/skills/db-core/live-queries/SKILL.md +++ b/packages/db/skills/db-core/live-queries/SKILL.md @@ -5,16 +5,18 @@ description: > fullJoin, select, fn.select, groupBy, having, orderBy, limit, offset, distinct, findOne. Operators: eq, gt, gte, lt, lte, like, ilike, inArray, isNull, isUndefined, and, or, not. Aggregates: count, sum, avg, min, max. String - functions: upper, lower, length, concat. Utility: coalesce, caseWhen. Math: add. + functions: upper, lower, length, concat. Utility: coalesce, caseWhen. Math: + add, subtract, multiply, divide. $selected namespace. createLiveQueryCollection. Derived collections. Predicate push-down. Incremental view maintenance via differential dataflow (d2ts). Virtual properties ($synced, $origin, $key, $collectionId). Includes subqueries - for hierarchical data. toArray and concat(toArray(...)) scalar includes. + for hierarchical data. Collection, toArray, materialize, and + concat(toArray(...)) include modes. queryOnce for one-shot queries. createEffect for reactive side effects (onEnter, onUpdate, onExit, onBatch). type: sub-skill library: db -library_version: '0.6.0' +library_version: '0.6.17' sources: - 'TanStack/db:docs/guides/live-queries.md' - 'TanStack/db:packages/db/src/query/builder/index.ts' @@ -106,6 +108,11 @@ Boolean column references work directly: .where(({ user }) => not(user.suspended)) // negated boolean ref ``` +Comparisons follow PostgreSQL semantics. Comparisons involving `null` or +`undefined` are unknown and do not match; use `isNull()` or `isUndefined()`. +`NaN` (and an invalid `Date`) equals itself and sorts after every other +non-null value. + ### 2. Joining two collections Join conditions **must** use `eq()` (equality only -- IVM constraint). Default join type is `left`. Convenience methods: `leftJoin`, `rightJoin`, `innerJoin`, `fullJoin`. @@ -195,11 +202,25 @@ const activeUserPosts = createLiveQueryCollection((q) => Create derived collections once at module scope and reuse them. Do not recreate on every render or navigation. +Live query collections default to `gcTime: 5_000`. An explicit `gcTime: 0` is +preserved and disables garbage collection for that derived collection -- +including the reclamation of a collection that started syncing and never gained +a subscriber. Note this is the opposite of `gcTime: 0` in TanStack Query, where +it collects as soon as the query goes inactive; use a small positive value if +you want prompt collection here. + +Sync started without subscribers has a minimum 50ms GC grace period. Pending +`preload()` calls retain the collection until they settle; the unused retention +period then starts. Preloading an already-ready collection refreshes that +period. Explicit `cleanup()` can still abort a pending preload. + ## Virtual Properties Live query results include computed, read-only virtual properties on every row: -- `$synced`: `true` when the row is confirmed by sync; `false` when it is still optimistic. +- `$synced`: `true` when no pending local optimistic write affects the row; + `false` while one does. This is local mutation status, not proof that a + backend uploaded, confirmed, or read back the row. - `$origin`: `"local"` if the last confirmed change came from this client, otherwise `"remote"`. - `$key`: the row key for the result. - `$collectionId`: the source collection ID. @@ -208,7 +229,9 @@ These props are added automatically and can be used in `where`, `select`, and `o ## Includes (Subqueries in Select) -Embed a correlated subquery inside `select()` to produce hierarchical (nested) data. The subquery must contain a `where` with an `eq()` that correlates a parent field with a child field. Three materialization modes are available. +Embed a correlated subquery inside `select()` to produce hierarchical (nested) +data. The subquery must contain a `where` with an `eq()` that correlates a +parent field with a child field. ### Collection includes (default) @@ -239,7 +262,7 @@ for (const project of projectsWithIssues) { ### Array includes with toArray() -Wrap the subquery in `toArray()` to get a plain array of scalar values instead of a Collection: +Wrap the subquery in `toArray()` to get a plain array instead of a Collection: ```ts import { eq, toArray, createLiveQueryCollection } from '@tanstack/db' @@ -259,6 +282,32 @@ const messagesWithParts = createLiveQueryCollection((q) => // row.contentParts is string[] ``` +### Plain values with materialize() + +Use `materialize()` when the parent row should hold a plain snapshot rather +than a child collection: + +```ts +import { eq, materialize, createLiveQueryCollection } from '@tanstack/db' + +const issuesWithProject = createLiveQueryCollection((q) => + q.from({ issue: issuesCollection }).select(({ issue }) => ({ + ...issue, + project: materialize( + q + .from({ project: projectsCollection }) + .where(({ project }) => eq(project.id, issue.projectId)) + .findOne(), + ), + })), +) +// row.project is Project | undefined +``` + +For a multi-row subquery, `materialize()` returns `Array` like `toArray()`. +For a subquery ending in `findOne()`, it returns `T | undefined`. In both cases, +the parent row is re-emitted when the child result changes. + ### Concatenated scalar with concat(toArray()) Wrap `toArray()` in `concat()` to join the scalar results into a single string: @@ -287,6 +336,9 @@ const messagesWithContent = createLiveQueryCollection((q) => - The subquery **must** have a `where` clause with an `eq()` correlating a parent alias with a child alias. The library extracts this automatically as the join condition. - `toArray()` works with both scalar selects (e.g., `select(({ c }) => c.text)` → `string[]`) and object selects (e.g., `select(({ c }) => ({ id: c.id, title: c.title }))` → `Array<{id, title}>`). +- `materialize()` returns an array, or one value for a `findOne()` subquery. + Like `toArray()`, it must be a top-level value in `select()` and cannot be + nested inside `coalesce()`, `eq()`, or another expression. - `concat(toArray())` requires a **scalar** `select` to concatenate into a string. - Collection includes (bare subquery) require an **object** `select`. - Includes subqueries are compiled into the same incremental pipeline as the parent query -- they are not separate live queries. @@ -372,20 +424,26 @@ JS `.filter()` / `.map()` on the result array throws away incremental maintenanc ```ts // WRONG -- re-runs filter on every change -const { data } = useLiveQuery((q) => q.from({ todos: todosCollection })) +const { data } = useLiveQuery({ + query: (q) => q.from({ todos: todosCollection }), +}) const active = data.filter((t) => t.completed === false) // CORRECT -- incrementally maintained -const { data } = useLiveQuery((q) => - q - .from({ todos: todosCollection }) - .where(({ todos }) => eq(todos.completed, false)), -) +const { data } = useLiveQuery({ + query: (q) => + q + .from({ todos: todosCollection }) + .where(({ todos }) => eq(todos.completed, false)), +}) ``` ### HIGH: Not using the full operator set -The library provides string functions (`upper`, `lower`, `length`, `concat`), math (`add`), utility functions (`coalesce`, `caseWhen`), and aggregates (`count`, `sum`, `avg`, `min`, `max`). All are incrementally maintained. Prefer them over JS equivalents. +The library provides string functions (`upper`, `lower`, `length`, `concat`), +math (`add`, `subtract`, `multiply`, `divide`), utility functions (`coalesce`, +`caseWhen`), and aggregates (`count`, `sum`, `avg`, `min`, `max`). All are +incrementally maintained. Prefer them over JS equivalents. ```ts // WRONG @@ -402,6 +460,11 @@ The library provides string functions (`upper`, `lower`, `length`, `concat`), ma })) ``` +Math expressions also work in `orderBy()`. When a computed expression is used +with `limit()`, lazy-loading optimization is skipped and all matching rows load +before sorting. Literal values such as `Date.now()` are captured when the query +is created; recreate the query when the value must advance. + ### HIGH: Missing conditional expression helpers Use `coalesce()` for null/undefined fallbacks and `caseWhen()` for conditional @@ -506,6 +569,12 @@ q.from(usersCollection) q.from({ users: usersCollection }) ``` +### MEDIUM: Using unsafe select alias paths + +Select alias path segments named `__proto__`, `prototype`, or `constructor` +throw `UnsafeAliasPathError`. Use ordinary data-field names; do not suppress +this prototype-pollution guard. + ## Tension: Query expressiveness vs. IVM constraints The query builder looks like SQL but has constraints that SQL does not: diff --git a/packages/db/skills/db-core/live-queries/references/operators.md b/packages/db/skills/db-core/live-queries/references/operators.md index be494de0d0..8d4082ea5c 100644 --- a/packages/db/skills/db-core/live-queries/references/operators.md +++ b/packages/db/skills/db-core/live-queries/references/operators.md @@ -32,6 +32,9 @@ import { concat, // Math add, + subtract, + multiply, + divide, // Utility coalesce, } from '@tanstack/db' @@ -112,6 +115,17 @@ Check if value is `undefined` (absent). Especially useful after left joins where isUndefined(profile) // no matching profile in left join ``` +### Comparison semantics + +Comparisons involving `null` or `undefined` evaluate as unknown and do not +match. Use `isNull()` or `isUndefined()` instead of `eq(value, null)` or +`eq(value, undefined)`. + +`NaN` follows PostgreSQL rather than JavaScript semantics: it equals itself and +is greater than every other non-null value. Invalid `Date` values behave the +same way. This applies to equality, `inArray()`, range comparisons, and +ordering. + --- ## Logical Operators @@ -208,15 +222,32 @@ concat(user.firstName, ' ', user.lastName) ## Math Functions -### add(left, right) -> BasicExpression\ +### add, subtract, multiply (left, right) -> BasicExpression\ -Add two numeric values. +Apply the named operation to two numeric values. ```ts add(order.price, order.tax) -add(user.salary, coalesce(user.bonus, 0)) +subtract(user.salary, user.deductions) +multiply(item.price, item.quantity) +``` + +Nullish operands are treated as `0`. + +### divide(left, right) -> BasicExpression\ + +Divide two numeric values. Nullish operands are treated as `0`; a zero or +nullish divisor returns `null`. + +```ts +divide(order.total, order.itemCount) ``` +These functions may be used in `orderBy()`. With a computed `orderBy()` and +`limit()`, all matching rows load before sorting because lazy-loading +optimization cannot apply. Literal values such as `Date.now()` are captured +when the query is built. + --- ## Utility Functions diff --git a/packages/db/skills/db-core/mutations-optimistic/SKILL.md b/packages/db/skills/db-core/mutations-optimistic/SKILL.md index eced517738..b3dda1471d 100644 --- a/packages/db/skills/db-core/mutations-optimistic/SKILL.md +++ b/packages/db/skills/db-core/mutations-optimistic/SKILL.md @@ -9,7 +9,7 @@ description: > onInsert/onUpdate/onDelete handlers. PendingMutation type. Transaction.isPersisted. type: sub-skill library: db -library_version: '0.6.0' +library_version: '0.6.17' sources: - 'TanStack/db:docs/guides/mutations.md' - 'TanStack/db:packages/db/src/transactions.ts' @@ -24,7 +24,7 @@ sources: > handlers) before you can mutate. TanStack DB mutations follow a unidirectional loop: -**optimistic mutation -> handler persists to backend -> sync back -> confirmed state**. +**optimistic mutation -> handler persists -> handler waits for sync/ack -> confirmed state**. Optimistic state is applied in the current tick and dropped when the handler resolves. --- @@ -34,17 +34,19 @@ Optimistic state is applied in the current tick and dropped when the handler res ### insert ```ts +import { safeRandomUUID } from '@tanstack/db' + // Single item todoCollection.insert({ - id: crypto.randomUUID(), + id: safeRandomUUID(), text: 'Buy groceries', completed: false, }) // Multiple items todoCollection.insert([ - { id: crypto.randomUUID(), text: 'Buy groceries', completed: false }, - { id: crypto.randomUUID(), text: 'Walk dog', completed: false }, + { id: safeRandomUUID(), text: 'Buy groceries', completed: false }, + { id: safeRandomUUID(), text: 'Walk dog', completed: false }, ]) // With metadata / non-optimistic @@ -87,7 +89,14 @@ todoCollection.delete(todo.id, { metadata: { reason: 'completed' } }) ``` All three return a `Transaction` object. Use `tx.isPersisted.promise` to await -persistence or catch rollback errors. +settlement or catch rollback errors. For a non-empty transaction, this normally +means its `mutationFn` returned; it proves upload, confirmation, or read-back +only when that function waits for the backend observation before returning. + +Do not start or await collection preloads, live-query preloads, or direct +`loadSubset()` calls inside `mutationFn`. Sync commits queue behind mutation +persistence, so the preload can wait on the mutation that is waiting on it. +Use the collection adapter's documented mutation acknowledgement pattern. --- @@ -127,7 +136,7 @@ Multi-collection example: const createProject = createOptimisticAction<{ name: string; ownerId: string }>( { onMutate: ({ name, ownerId }) => { - projectCollection.insert({ id: crypto.randomUUID(), name, ownerId }) + projectCollection.insert({ id: safeRandomUUID(), name, ownerId }) userCollection.update(ownerId, (d) => { d.projectCount += 1 }) @@ -207,7 +216,9 @@ await tx.commit() Inside `tx.mutate(() => { ... })`, the transaction is pushed onto an ambient stack. Any `collection.insert/update/delete` call joins the ambient transaction -automatically via `getActiveTransaction()`. +automatically via `getActiveTransaction()`. That scope is synchronous: +collection operations after an `await` do not join it. Put async work in +`mutationFn`, or call `mutate()` again before committing. For mutations captured by a manual transaction, collection-level `onInsert`/`onUpdate`/`onDelete` handlers are not invoked automatically. The @@ -217,7 +228,7 @@ a good fit for draft-style flows where local state updates immediately but the server call waits for Save/Blur; call `tx.rollback()` to discard the optimistic changes. -### 4. Mutation handler with refetch (QueryCollection pattern) +### 4. Mutation handlers with automatic refetch (QueryCollection pattern) ```ts const todoCollection = createCollection( @@ -229,8 +240,7 @@ const todoCollection = createCollection( await Promise.all( transaction.mutations.map((m) => api.todos.create(m.modified)), ) - // IMPORTANT: handler must not resolve until server state is synced back - // QueryCollection auto-refetches after handler completes + // Query Collection refetches after the handler completes and awaits it. }, onUpdate: async ({ transaction }) => { await Promise.all( @@ -308,7 +318,7 @@ createOptimisticAction({ // CORRECT createOptimisticAction({ onMutate: (text) => { - collection.insert({ id: crypto.randomUUID(), text }) + collection.insert({ id: safeRandomUUID(), text }) }, ... }) @@ -338,29 +348,31 @@ re-insert. ### HIGH: Inserting item with duplicate key If an item with the same key already exists (synced or optimistic), throws -`DuplicateKeyError`. Always generate a unique key (e.g. `crypto.randomUUID()`) +`DuplicateKeyError`. Always generate a unique key (e.g. `safeRandomUUID()`) or check before inserting. -### HIGH: Not awaiting refetch after mutation in query collection handler +### HIGH: Manually refetching inside a Query Collection handler -The optimistic state is held only until the handler resolves. If the handler -returns before server state has synced back, optimistic state is dropped and -users see a flash of missing data. +Query Collection automatically refetches after `onInsert`, `onUpdate`, and +`onDelete` complete, and waits for that refetch before the mutation finishes. +Calling `utils.refetch()` inside the handler sends a redundant request. ```ts -// WRONG -- optimistic state dropped before new server state arrives +// WRONG -- causes one manual and one automatic refetch onInsert: async ({ transaction }) => { await api.createTodo(transaction.mutations[0].modified) - // missing: await collection.utils.refetch() + await collection.utils.refetch() } -// CORRECT +// CORRECT -- automatic refetch is awaited after this returns onInsert: async ({ transaction }) => { await api.createTodo(transaction.mutations[0].modified) - await collection.utils.refetch() } ``` +When the handler writes the confirmed server result with direct-write utilities, +return `{ refetch: false }` to skip the automatic refetch. + --- ## Tension: Optimistic Speed vs. Data Consistency diff --git a/packages/db/skills/db-core/mutations-optimistic/references/transaction-api.md b/packages/db/skills/db-core/mutations-optimistic/references/transaction-api.md index 5c8e918489..e4cff2abcf 100644 --- a/packages/db/skills/db-core/mutations-optimistic/references/transaction-api.md +++ b/packages/db/skills/db-core/mutations-optimistic/references/transaction-api.md @@ -6,7 +6,7 @@ import { createTransaction } from "@tanstack/db" const tx = createTransaction({ - id?: string, // defaults to crypto.randomUUID() + id?: string, // defaults to safeRandomUUID() autoCommit?: boolean, // default true -- commit after mutate() mutationFn: MutationFn, // (params: { transaction }) => Promise metadata?: Record, // custom data attached to the transaction @@ -26,7 +26,8 @@ interface Transaction { metadata: Record error?: { message: string; error: Error } - // Deferred promise -- resolves when mutationFn completes, rejects on failure + // Deferred promise -- resolves when the transaction settles, rejects on + // mutation failure or rollback isPersisted: { promise: Promise> resolve: (value: Transaction) => void @@ -51,6 +52,7 @@ interface Transaction { - `rollback()` allowed in `pending` or `persisting` (throws `TransactionAlreadyCompletedRollbackError` if completed) - Failed `mutationFn` automatically triggers `rollback()` - Rollback cascades to other pending transactions sharing the same item keys +- An empty or fully cancelled transaction completes without calling `mutationFn` ## PendingMutation Type @@ -102,6 +104,11 @@ stack. Any `collection.insert/update/delete` call automatically joins the topmost ambient transaction. This is how `createOptimisticAction` and `createPacedMutations` wire collection operations into their transactions. +The ambient scope lasts only for the synchronous `mutate()` callback. A +collection operation after an `await` does not join that transaction. Put async +work in `mutationFn`, or call `mutate()` again while the transaction is still +pending. + ## createOptimisticAction ```ts @@ -116,7 +123,7 @@ const action = createOptimisticAction({ // Optional: same as createTransaction config id?: string, - autoCommit?: boolean, // always true (commit happens after mutate) + autoCommit?: boolean, // default true; false requires manual commit() metadata?: Record, }) @@ -203,5 +210,10 @@ try { The promise is a `Deferred` -- it is created at transaction construction time and settled when `commit()` completes or `rollback()` is called. For -`autoCommit: true` transactions, the promise settles shortly after `mutate()` -returns (the commit runs asynchronously). +`autoCommit: true` transactions, commit starts after `mutate()` returns; the +promise can remain pending as long as `mutationFn` does. + +For a non-empty commit, `mutationFn` is the normal success boundary. +`isPersisted.promise` does not by itself prove that a backend uploaded, +confirmed, or read back the write. It proves those stronger guarantees only +when `mutationFn` waits for them before returning. diff --git a/packages/db/skills/db-core/persistence/SKILL.md b/packages/db/skills/db-core/persistence/SKILL.md index 70d2b9eb69..8a5f657af0 100644 --- a/packages/db/skills/db-core/persistence/SKILL.md +++ b/packages/db/skills/db-core/persistence/SKILL.md @@ -8,10 +8,11 @@ description: > Cloudflare Durable Objects. Multi-tab/multi-process coordination via BrowserCollectionCoordinator / ElectronCollectionCoordinator / SingleProcessCoordinator. schemaVersion for migration resets. Local-only mode - for offline-first without a server. + for offline-first without a server. Applied transaction log pruning and safe + full-reload recovery. type: sub-skill library: db -library_version: '0.6.0' +library_version: '0.6.17' sources: - 'TanStack/db:packages/db-sqlite-persistence-core/src/persisted.ts' - 'TanStack/db:packages/browser-db-sqlite-persistence/src/index.ts' @@ -51,7 +52,6 @@ For purely local data with no sync backend: ```ts import { createCollection } from '@tanstack/react-db' import { - BrowserCollectionCoordinator, createBrowserWASQLitePersistence, openBrowserWASQLiteOPFSDatabase, persistedCollectionOptions, @@ -61,13 +61,8 @@ const database = await openBrowserWASQLiteOPFSDatabase({ databaseName: 'my-app.sqlite', }) -const coordinator = new BrowserCollectionCoordinator({ - dbName: 'my-app', -}) - const persistence = createBrowserWASQLitePersistence({ database, - coordinator, }) const draftsCollection = createCollection( @@ -115,11 +110,15 @@ This works with any adapter: `electricCollectionOptions`, `queryCollectionOption Coordinators handle leader election and cross-instance communication so only one tab/process owns the database writer. -| Platform | Coordinator | Mechanism | -| ------------------------------------- | ------------------------------- | ---------------------------------------------- | -| Browser | `BrowserCollectionCoordinator` | BroadcastChannel + Web Locks | -| Electron | `ElectronCollectionCoordinator` | IPC (main holds DB, renderer accesses via RPC) | -| Single-process (RN, Expo, Node, etc.) | `SingleProcessCoordinator` | No-op (always leader) | +| Platform | Coordinator | Mechanism | +| ------------------------------------- | ------------------------------- | ---------------------------- | +| Browser | `BrowserCollectionCoordinator` | BroadcastChannel + Web Locks | +| Electron | `ElectronCollectionCoordinator` | BroadcastChannel + Web Locks | +| Single-process (RN, Expo, Node, etc.) | `SingleProcessCoordinator` | No-op (always leader) | + +Browser persistence uses single-process semantics by default. That is correct +when the app runs in one tab at a time or each tab has its own database. Pass a +`BrowserCollectionCoordinator` only when multiple tabs share one OPFS database. Browser example: @@ -142,7 +141,12 @@ Electron requires setup in both processes: ```ts // Main process import { exposeElectronSQLitePersistence } from '@tanstack/electron-db-sqlite-persistence' -exposeElectronSQLitePersistence({ persistence, ipcMain }) +import { app, ipcMain } from 'electron' + +const disposeIpc = exposeElectronSQLitePersistence({ persistence, ipcMain }) +app.on('before-quit', () => { + disposeIpc() +}) // Renderer process import { @@ -157,6 +161,10 @@ const persistence = createElectronSQLitePersistence({ }) ``` +Electron persistence calls cross the renderer/main boundary through IPC. The +`ElectronCollectionCoordinator` separately coordinates renderer instances with +`BroadcastChannel` and Web Locks. + ## Schema Versioning `schemaVersion` tracks the shape of persisted data. When the stored version doesn't match the code, the collection resets (drops and reloads from server for synced collections, or throws for local-only). @@ -170,6 +178,36 @@ persistedCollectionOptions({ There is no custom migration function -- a version mismatch triggers a full reset. For synced collections this is safe because the server re-supplies the data. +## Applied Transaction Log Pruning + +The SQLite `applied_tx` log is a replay cache, not permanent history. Browser, +Capacitor, Cloudflare Durable Objects, Expo, Node, React Native, and Tauri +wrappers prune it inside write transactions by default, per collection: + +- `appliedTxPruneMaxRows: 1_000` +- `appliedTxPruneMaxAgeSeconds: 86_400` (24 hours) + +Set either option to `0` to disable that limit, or raise it to retain a longer +replay window: + +```ts +const persistence = createNodeSQLitePersistence({ + database, + appliedTxPruneMaxRows: 5_000, + appliedTxPruneMaxAgeSeconds: 0, +}) +``` + +If a follower asks to recover from a point older than the retained log, it +falls back to a full reload. Pruning does not itself shrink the SQLite file; +use SQLite vacuum settings or separate maintenance when disk reclamation +matters. The defaults are exported as +`DEFAULT_APPLIED_TX_PRUNE_MAX_ROWS` and +`DEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDS`. + +Raw `createSQLiteCorePersistenceAdapter` calls do not inject these defaults. +Electron uses whichever persistence adapter the main process supplies. + ## Key Options | Option | Type | Description | @@ -204,13 +242,13 @@ persistedCollectionOptions({ Without an explicit `id`, the code generates a random UUID each session, so persisted data is silently abandoned on every reload. Local-only persisted collections must always provide an `id`. Synced collections derive it from the adapter config. -### HIGH Forgetting the coordinator in multi-tab apps +### HIGH Sharing one browser database across tabs without a coordinator Wrong: ```ts const persistence = createBrowserWASQLitePersistence({ database }) -// No coordinator — concurrent tabs corrupt the database +// Unsafe if multiple tabs share this database ``` Correct: @@ -220,7 +258,9 @@ const coordinator = new BrowserCollectionCoordinator({ dbName: 'my-app' }) const persistence = createBrowserWASQLitePersistence({ database, coordinator }) ``` -Without a coordinator, multiple browser tabs write to SQLite concurrently, causing data corruption. Always use `BrowserCollectionCoordinator` in browser environments. +Without a coordinator, multiple browser tabs that share one OPFS database can +write concurrently. Use `BrowserCollectionCoordinator` for that case. Do not +add it to a single-tab app merely because the runtime is a browser. ### HIGH Not bumping schemaVersion after changing data shape diff --git a/packages/db/skills/meta-framework/SKILL.md b/packages/db/skills/meta-framework/SKILL.md index 04046f6896..bcc3c7a5dd 100644 --- a/packages/db/skills/meta-framework/SKILL.md +++ b/packages/db/skills/meta-framework/SKILL.md @@ -3,13 +3,13 @@ name: meta-framework description: > Integrating TanStack DB with meta-frameworks (TanStack Start, Next.js, Remix, Nuxt, SvelteKit). Client-side only: SSR is NOT supported — routes - must disable SSR. Preloading collections in route loaders with - collection.preload(). Pattern: ssr: false + await collection.preload() in - loader. Multiple collection preloading with Promise.all. Framework-specific - loader APIs. + must disable SSR. Preloading eager collections in route loaders with + collection.preload(). On-demand Query Collections require preloading the + live query because source collection preload is a no-op. Multiple collection + preloading with Promise.all. Framework-specific loader APIs. type: composition library: db -library_version: '0.6.0' +library_version: '0.6.17' requires: - db-core - db-core/collection-setup @@ -28,7 +28,7 @@ This skill builds on db-core. Read it first for collection setup and query build TanStack DB collections are **client-side only**. SSR is not implemented. Routes using TanStack DB **must disable SSR**. The setup pattern is: 1. Set `ssr: false` on the route -2. Call `collection.preload()` in the route loader +2. Preload the eager collection, or preload the live query for an on-demand source 3. Use `useLiveQuery` in the component ## TanStack Start @@ -62,7 +62,9 @@ export const Route = createFileRoute('/todos')({ }) function TodoPage() { - const { data: todos } = useLiveQuery((q) => q.from({ todo: todoCollection })) + const { data: todos } = useLiveQuery({ + query: (q) => q.from({ todo: todoCollection }), + }) return (
                  {todos.map((t) => ( @@ -73,6 +75,35 @@ function TodoPage() { } ``` +### On-demand Query Collection preload + +Calling `preload()` on an on-demand source collection is a no-op. Define the +live query once, preload it in the loader, and pass that same collection to the +framework hook: + +```tsx +import { createLiveQueryCollection, eq } from '@tanstack/db' +import { useLiveQuery } from '@tanstack/react-db' + +const activeTodos = createLiveQueryCollection((q) => + q + .from({ todo: todoCollection }) + .where(({ todo }) => eq(todo.completed, false)), +) + +export const Route = createFileRoute('/todos')({ + ssr: false, + loader: async () => { + await activeTodos.preload() + return null + }, + component: () => { + const { data } = useLiveQuery(activeTodos) + // ... + }, +}) +``` + ### Multiple collection preloading ```tsx @@ -98,9 +129,9 @@ import { useEffect, useState } from 'react' import { useLiveQuery } from '@tanstack/react-db' export default function TodoPage() { - const { data: todos, isLoading } = useLiveQuery((q) => - q.from({ todo: todoCollection }), - ) + const { data: todos, isLoading } = useLiveQuery({ + query: (q) => q.from({ todo: todoCollection }), + }) if (isLoading) return
                  Loading...
                  return ( @@ -128,7 +159,9 @@ import { useLiveQuery } from '@tanstack/react-db' const preloadPromise = todoCollection.preload() export default function TodoPage() { - const { data: todos } = useLiveQuery((q) => q.from({ todo: todoCollection })) + const { data: todos } = useLiveQuery({ + query: (q) => q.from({ todo: todoCollection }), + }) return (
                    {todos.map((t) => ( @@ -157,7 +190,9 @@ export const clientLoader = async ({ request }: ClientLoaderFunctionArgs) => { export const loader = () => null export default function TodoPage() { - const { data: todos } = useLiveQuery((q) => q.from({ todo: todoCollection })) + const { data: todos } = useLiveQuery({ + query: (q) => q.from({ todo: todoCollection }), + }) return (
                      {todos.map((t) => ( @@ -227,7 +262,9 @@ export const ssr = false ### What preload() does -`collection.preload()` starts the sync process and returns a promise that resolves when the collection reaches "ready" status. This means: +For eager collections, `collection.preload()` starts the sync process and +returns a promise that resolves when the collection reaches "ready" status. +This means: 1. The sync function connects to the backend 2. Initial data is fetched and written to the collection @@ -236,9 +273,14 @@ export const ssr = false Subsequent calls to `preload()` on an already-ready collection return immediately. -### Collection module pattern +For on-demand collections, source `collection.preload()` warns and does +nothing because no subset has been requested. Create the required live query +and await `liveQuery.preload()`. + +### Stable collection ownership -Define collections in a shared module, import in both loaders and components: +For one global `QueryClient` and one global server resource, define the +collection in a shared module and import it in loaders and components: ```ts // lib/collections.ts @@ -260,12 +302,23 @@ export const Route = createFileRoute('/todos')({ return null }, component: () => { - const { data } = useLiveQuery((q) => q.from({ todo: todoCollection })) + const { data } = useLiveQuery({ + query: (q) => q.from({ todo: todoCollection }), + }) // ... }, }) ``` +When the `QueryClient`, tenant, project, account, or route parameter defines +the resource, create one stable collection per `QueryClient` and business +scope. Memoize it and put it in router/request context rather than using a +process-global collection. Remove unused entries and call +`collection.cleanup()` in long-lived scope maps. + +See the +[Query adapter runtime and business-scope pattern](../db-core/collection-setup/references/query-adapter.md#runtime-queryclient-and-business-scopes). + ## Server-Side Integration This skill covers the **client-side** read path only (preloading, live queries). For server-side concerns: @@ -328,9 +381,9 @@ export const Route = createFileRoute('/todos')({ }) ``` -Without preloading, the collection starts syncing only when the component mounts, causing a loading flash. Preloading in the route loader starts sync during navigation, making data available immediately when the component renders. +Without preloading, the collection starts syncing when the component first renders, causing a loading flash. Preloading in the route loader starts sync during navigation, so the data is already there on that first render. -### MEDIUM Creating separate collection instances +### MEDIUM Creating separate collection instances in one scope Wrong: @@ -342,7 +395,9 @@ export const Route = createFileRoute('/todos')({ ssr: false, loader: async () => { await todoCollection.preload() }, component: () => { - const { data } = useLiveQuery((q) => q.from({ todo: todoCollection })) + const { data } = useLiveQuery({ + query: (q) => q.from({ todo: todoCollection }), + }) }, }) ``` @@ -350,11 +405,14 @@ export const Route = createFileRoute('/todos')({ Correct: ```ts -// lib/collections.ts — single shared instance +// lib/collections.ts — shared for a global QueryClient and global resource export const todoCollection = createCollection(queryCollectionOptions({ ... })) ``` -Collections are singletons. Creating multiple instances for the same data causes duplicate syncs, wasted bandwidth, and inconsistent state between components. +Collections are stable within a `QueryClient` and business scope; they are not +universal singletons. Creating several instances in one scope causes duplicate +syncs and split state. A request-, router-, tenant-, or route-scoped client +needs a scoped factory instead of the global module pattern. See also: react-db/SKILL.md, vue-db/SKILL.md, svelte-db/SKILL.md, solid-db/SKILL.md, angular-db/SKILL.md — for framework-specific hook usage. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts new file mode 100644 index 0000000000..6765b76dc1 --- /dev/null +++ b/packages/db/src/client.ts @@ -0,0 +1,906 @@ +import { createCollection } from './collection/index.js' +import { + collectionOptionsBrand, + collectionOptionsFactory, + hasCollectionOptionsBrand, +} from './collection-options.js' +import { TransactionScope } from './transactions.js' +import { getBuilderFromConfig } from './query/live/collection-registry.js' +import { createLiveQueryCollection } from './query/live-query-collection.js' +import { createLiveQueryObserver } from './live-query-observer.js' +import { createDeferred } from './deferred.js' +import { + getLiveQueryHash, + prepareLiveQueryValue, +} from './live-query-options.js' +import type { StandardSchemaV1 } from '@standard-schema/spec' +import type { Collection } from './collection/index.js' +import type { CollectionOptionsIdentity } from './collection-options.js' +import type { + CollectionConfig, + InferSchemaInput, + InferSchemaOutput, + NonSingleResult, + SingleResult, + TransactionConfig, + UtilsRecord, +} from './types.js' +import type { + DeferredLiveQueryCollections, + LiveQueryOptions, +} from './live-query-options.js' + +const collectionConfigFactory: unique symbol = Symbol.for( + `@tanstack/db.collectionConfig.factory`, +) as never + +type AnyCollectionConfig = CollectionConfig + +export type CollectionOptions< + T extends object = Record, + TKey extends string | number = string | number, + TSchema extends StandardSchemaV1 = never, + TUtils extends UtilsRecord = UtilsRecord, +> = CollectionOptionsIdentity + +type AnyCollectionOptions = CollectionOptions +type AnyCollection = Collection + +type DescriptorFromConfig = + TConfig extends { + getKey: (item: infer T) => infer TKey + } + ? CollectionOptions< + Extract, + Extract, + TConfig extends { + schema: infer TSchema extends StandardSchemaV1 + } + ? TSchema + : never, + TConfig extends { + utils: infer TUtils extends UtilsRecord + } + ? TUtils + : UtilsRecord + > & + (TConfig extends SingleResult ? SingleResult : NonSingleResult) + : never + +type CollectionConfigWithFactory = + TConfig & { + readonly [collectionConfigFactory]: (client: DbClient) => TConfig + } + +/** + * Adds a fresh-config materializer to an adapter options object. + * + * Adapter option creators should use this so a module-scoped descriptor can be + * materialized safely by more than one DbClient. + */ +export function withCollectionConfigFactory< + TConfig extends AnyCollectionConfig, +>( + config: TConfig, + factory: (client: DbClient) => TConfig, +): CollectionConfigWithFactory { + Object.defineProperty(config, collectionConfigFactory, { + value: factory, + enumerable: false, + }) + return config as CollectionConfigWithFactory +} + +export type CollectionMaterializeOptions = { + initialData?: Array +} + +export type DehydratedCollectionRow< + T extends object = Record, + TKey extends string | number = string | number, +> = { + key: TKey + value: T + metadata?: unknown +} + +export type DehydratedCollectionChunk< + T extends object = Record, + TKey extends string | number = string | number, +> = { + collectionId: string + rows: Array> + syncMeta?: unknown +} + +export type DehydratedLiveQuery = { + queryHash: string + dehydratedAt: number + snapshot?: DehydratedLiveQueryResult + promise?: Promise +} + +export type DehydratedLiveQueryResult< + T extends object = object, + TKey extends string | number = string | number, +> = { + rows: Array> +} + +export type DehydratedDbState = { + collections: Array + liveQueries?: Array +} + +export type DbClientLiveQueryState = `pending` | `success` | `error` + +export type DbClientLiveQuery = { + readonly queryHash: string + readonly dehydratedAt: number + readonly status: DbClientLiveQueryState + readonly promise: Promise + readonly snapshot?: DehydratedLiveQueryResult + readonly error?: unknown +} + +export type DbClientEvent = + | { + type: `liveQueryAdded` | `liveQueryUpdated` + query: DbClientLiveQuery + } + | { + type: `liveQueryStreamError` + error: unknown + } + +export type DehydrateDbClientOptions = { + shouldDehydrateCollection?: (collection: Collection) => boolean + shouldDehydrateLiveQuery?: (query: DbClientLiveQuery) => boolean +} + +type CollectionRecord = { + collection: AnyCollection + shouldDehydrate: boolean +} + +type LiveQueryRecord = { + queryHash: string + dehydratedAt: number + status: DbClientLiveQueryState + promise: Promise + resultPromise: Promise + succeed: (snapshot: DehydratedLiveQueryResult) => void + fail: (error: unknown) => void + snapshot?: DehydratedLiveQueryResult + error?: unknown +} + +export type DbClientOptions = Record + +export function collectionOptions< + T extends StandardSchemaV1, + TKey extends string | number, + TUtils extends UtilsRecord, +>( + options: CollectionConfig, TKey, T, TUtils> & { + schema: T + } & NonSingleResult, +): CollectionOptions, TKey, T, TUtils> & NonSingleResult +export function collectionOptions< + T extends StandardSchemaV1, + TKey extends string | number, + TUtils extends UtilsRecord, +>( + options: CollectionConfig, TKey, T, TUtils> & { + schema: T + } & SingleResult, +): CollectionOptions, TKey, T, TUtils> & SingleResult +export function collectionOptions< + T extends object, + TKey extends string | number = string | number, + TUtils extends UtilsRecord = UtilsRecord, +>( + options: CollectionConfig & { + schema?: never + } & NonSingleResult, +): CollectionOptions & NonSingleResult +export function collectionOptions< + T extends object, + TKey extends string | number = string | number, + TUtils extends UtilsRecord = UtilsRecord, +>( + options: CollectionConfig & { + schema?: never + } & SingleResult, +): CollectionOptions & SingleResult +export function collectionOptions( + id: string, + factory: (client: DbClient) => TConfig, +): DescriptorFromConfig +export function collectionOptions( + optionsOrId: AnyCollectionConfig | string, + explicitFactory?: (client: DbClient) => unknown, +): any { + const config = typeof optionsOrId === `string` ? undefined : optionsOrId + const id = typeof optionsOrId === `string` ? optionsOrId : optionsOrId.id + + if (!id) { + throw new Error( + `collectionOptions requires a non-empty explicit id so the descriptor is stable across DbClient instances and SSR boundaries.`, + ) + } + + if (typeof optionsOrId === `string` && !explicitFactory) { + throw new Error( + `collectionOptions("${id}") requires a factory as its second argument.`, + ) + } + + const reusableFactory: + | ((client: DbClient) => AnyCollectionConfig) + | undefined = config + ? (config as CollectionConfigWithFactory)[ + collectionConfigFactory + ] + : (explicitFactory as + | ((client: DbClient) => AnyCollectionConfig) + | undefined) + + let owner: DbClient | undefined + const materialize = (client: DbClient): AnyCollectionConfig => { + let materialized: AnyCollectionConfig + + if (reusableFactory) { + materialized = reusableFactory(client) + } else { + if (owner && owner !== client) { + throw new Error( + `Collection descriptor "${id}" was created from a concrete config that cannot be safely reused across DbClient instances. ` + + `Use collectionOptions("${id}", (client) => adapterCollectionOptions(...)) or an adapter options creator that supports DbClient materialization.`, + ) + } + owner = client + materialized = config! + } + + if (materialized.id !== undefined && materialized.id !== id) { + throw new Error( + `Collection descriptor "${id}" materialized a config with id "${materialized.id}". Descriptor and collection ids must match.`, + ) + } + + return materialized.id === id ? materialized : { ...materialized, id } + } + + const descriptor = { + id, + ...((config as { singleResult?: boolean } | undefined)?.singleResult === + true + ? { singleResult: true as const } + : {}), + } as Record + + Object.defineProperties(descriptor, { + [collectionOptionsBrand]: { + value: true, + enumerable: false, + }, + [collectionOptionsFactory]: { + value: materialize, + enumerable: false, + }, + }) + + return Object.freeze(descriptor) as CollectionOptions< + any, + string | number, + any, + UtilsRecord + > +} + +export function isCollectionOptions( + value: unknown, +): value is CollectionOptions { + return hasCollectionOptionsBrand(value) +} + +export class DbClient { + private collectionsById = new Map() + private pendingHydration = new Map>() + private liveQueries = new Map() + private preloadedLiveQueries = new Map< + string, + { + collection: AnyCollection + observer: { dispose: () => void } + } + >() + private liveQueryResources = new Map Promise>() + private listeners = new Set<(event: DbClientEvent) => void>() + private ssrStreamingEnabled = false + private ssrServerCleanupEnabled = false + private lastLiveQueryTimestamp = 0 + private readonly transactionScope = new TransactionScope() + + constructor(private readonly options: DbClientOptions = {}) {} + + getDependency(key: string): T | undefined { + return this.options[key] as T | undefined + } + + requireDependency(key: string): T { + const dependency = this.getDependency(key) + if (dependency === undefined) { + throw new Error( + `DbClient is missing the required "${key}" dependency. Pass it explicitly when constructing the client: new DbClient({ ${key} }).`, + ) + } + return dependency + } + + get activeTransaction() { + return this.transactionScope.getActiveTransaction() + } + + createTransaction>( + config: TransactionConfig, + ) { + return this.transactionScope.createTransaction(config) + } + + preloadLiveQuery(options: LiveQueryOptions): Promise { + const deferredCollections: DeferredLiveQueryCollections = new Set() + try { + const prepared = prepareLiveQueryValue(options, this, deferredCollections) + const queryHash = getLiveQueryHash(prepared, options.queryKey) + const existing = this.liveQueries.get(queryHash) + if (existing && existing.status !== `error`) return existing.promise + + const failedPreload = this.preloadedLiveQueries.get(queryHash) + if (failedPreload) { + failedPreload.observer.dispose() + void failedPreload.collection.cleanup().catch(() => {}) + this.preloadedLiveQueries.delete(queryHash) + } + + const collection = createLiveQueryCollection({ + ...(prepared as LiveQueryOptions), + startSync: true, + }) as AnyCollection + const observer = createLiveQueryObserver(collection, { + client: this, + queryHash, + mode: `wholesale`, + }) + this.preloadedLiveQueries.set(queryHash, { collection, observer }) + + return this._registerLiveQuery( + queryHash, + collection.preload().then(() => observer.dehydrate()), + ) + } finally { + for (const source of deferredCollections) source._resumeSyncStart() + deferredCollections.clear() + } + } + + collection< + T extends StandardSchemaV1, + TKey extends string | number, + TUtils extends UtilsRecord, + >( + options: CollectionOptions, TKey, T, TUtils> & + NonSingleResult, + materializeOptions?: CollectionMaterializeOptions>, + ): Collection, TKey, TUtils, T, InferSchemaInput> & + NonSingleResult + collection< + T extends StandardSchemaV1, + TKey extends string | number, + TUtils extends UtilsRecord, + >( + options: CollectionOptions, TKey, T, TUtils> & + SingleResult, + materializeOptions?: CollectionMaterializeOptions>, + ): Collection, TKey, TUtils, T, InferSchemaInput> & + SingleResult + collection< + T extends object, + TKey extends string | number = string | number, + TUtils extends UtilsRecord = UtilsRecord, + >( + options: CollectionOptions & NonSingleResult, + materializeOptions?: CollectionMaterializeOptions, + ): Collection & NonSingleResult + collection< + T extends object, + TKey extends string | number = string | number, + TUtils extends UtilsRecord = UtilsRecord, + >( + options: CollectionOptions & SingleResult, + materializeOptions?: CollectionMaterializeOptions, + ): Collection & SingleResult + collection( + options: AnyCollectionOptions, + materializeOptions?: CollectionMaterializeOptions, + ): AnyCollection { + return this.materializeCollection(options, materializeOptions, false) + } + + /** @internal */ + _materializeCollectionForRender< + T extends object, + TKey extends string | number, + TSchema extends StandardSchemaV1, + TUtils extends UtilsRecord, + >( + options: CollectionOptions, + ): Collection< + T, + TKey, + TUtils, + TSchema, + [TSchema] extends [never] ? T : InferSchemaInput + > { + return this.materializeCollection(options, undefined, true) + } + + private materializeCollection( + options: AnyCollectionOptions, + materializeOptions: CollectionMaterializeOptions | undefined, + deferSyncStart: boolean, + ): AnyCollection { + const existing = this.collectionsById.get(options.id) + if (existing) { + return this.reuseMaterializedCollection(existing, deferSyncStart) + } + + const config = options[collectionOptionsFactory](this) + const materializedDuringFactory = this.collectionsById.get(options.id) + if (materializedDuringFactory) { + return this.reuseMaterializedCollection( + materializedDuringFactory, + deferSyncStart, + ) + } + + const shouldStartSync = config.startSync === true + const collection = createCollection({ + ...config, + startSync: false, + } as any) + collection._setTransactionScope(this.transactionScope) + if (deferSyncStart) { + collection._deferSyncStart() + } + + this.collectionsById.set(collection.id, { + collection, + shouldDehydrate: !deferSyncStart, + }) + + if (materializeOptions?.initialData?.length) { + this.applyRows( + collection, + { + collectionId: collection.id, + rows: materializeOptions.initialData.map((value) => ({ value })), + }, + `initialData`, + ) + } + + const pendingChunks = this.pendingHydration.get(collection.id) + if (pendingChunks) { + for (const chunk of pendingChunks) { + this.applyRows(collection, chunk, `hydration`) + } + this.pendingHydration.delete(collection.id) + } + + if (shouldStartSync) { + collection.startSyncImmediate() + } + + return collection + } + + private reuseMaterializedCollection( + record: CollectionRecord, + deferSyncStart: boolean, + ): AnyCollection { + if (deferSyncStart) { + record.collection._deferSyncStart() + } else { + record.shouldDehydrate = true + } + return record.collection + } + + dehydrate(options: DehydrateDbClientOptions = {}): DehydratedDbState { + const collections: Array = [] + + for (const { + collection, + shouldDehydrate, + } of this.collectionsById.values()) { + const collectionDecision = options.shouldDehydrateCollection?.(collection) + if ( + getBuilderFromConfig(collection.config) || + collectionDecision === false || + (!shouldDehydrate && collectionDecision !== true) + ) { + continue + } + + const rows = Array.from(collection._state.syncedData.entries()).map( + ([key, value]) => { + const metadata = collection._state.syncedMetadata.get(key) + return { + key, + value, + ...(metadata === undefined ? {} : { metadata }), + } + }, + ) + + collections.push({ + collectionId: collection.id, + rows, + syncMeta: collection.config.sync.exportSyncMeta?.(), + }) + } + + const liveQueries = Array.from(this.liveQueries.values()).flatMap( + (query): Array => { + const shouldDehydrate = + options.shouldDehydrateLiveQuery?.(query) ?? + query.status === `success` + if (!shouldDehydrate || query.status === `error`) { + return [] + } + + return [ + { + queryHash: query.queryHash, + dehydratedAt: query.dehydratedAt, + ...(query.snapshot + ? { snapshot: query.snapshot } + : { promise: query.resultPromise }), + }, + ] + }, + ) + + return { + collections, + ...(liveQueries.length > 0 ? { liveQueries } : {}), + } + } + + hydrate(state: DehydratedDbState): void { + for (const chunk of state.collections) { + const record = this.collectionsById.get(chunk.collectionId) + if (record) { + this.applyRows(record.collection, chunk, `hydration`) + continue + } + + const pendingChunks = this.pendingHydration.get(chunk.collectionId) ?? [] + pendingChunks.push(chunk) + this.pendingHydration.set(chunk.collectionId, pendingChunks) + } + + for (const dehydratedQuery of state.liveQueries ?? []) { + this.hydrateLiveQuery(dehydratedQuery) + } + } + + applyCollectionChunk(chunk: DehydratedCollectionChunk): void { + this.hydrate({ collections: [chunk] }) + } + + subscribe(listener: (event: DbClientEvent) => void): () => void { + this.listeners.add(listener) + return () => this.listeners.delete(listener) + } + + /** @internal */ + _setSsrStreamingEnabled(enabled: boolean): void { + this.ssrStreamingEnabled = enabled + } + + /** @internal */ + _isSsrStreamingEnabled(): boolean { + return this.ssrStreamingEnabled + } + + /** @internal */ + _setSsrServerCleanupEnabled(enabled: boolean): void { + this.ssrServerCleanupEnabled = enabled + } + + /** @internal */ + _isSsrServerCleanupEnabled(): boolean { + return this.ssrServerCleanupEnabled + } + + /** @internal */ + _getLiveQuery(queryHash: string): DbClientLiveQuery | undefined { + return this.liveQueries.get(queryHash) + } + + /** @internal */ + _consumeLiveQueryResult(queryHash: string, dehydratedAt: number): void { + const record = this.liveQueries.get(queryHash) + if (record?.dehydratedAt === dehydratedAt) { + this.liveQueries.delete(queryHash) + } + } + + /** @internal */ + _registerLiveQuery( + queryHash: string, + promise: Promise, + ): Promise { + const existing = this.liveQueries.get(queryHash) + if (existing && existing.status !== `error`) { + void Promise.resolve(promise).catch(() => {}) + return existing.promise + } + + const record = this.createLiveQueryRecord( + queryHash, + this.nextLiveQueryTimestamp(), + ) + + this.liveQueries.set(queryHash, record) + this.emit({ type: `liveQueryAdded`, query: record }) + Promise.resolve(promise).then(record.succeed, record.fail) + return record.promise + } + + /** @internal */ + _registerLiveQueryResource( + owner: object, + cleanup: () => Promise, + ): () => void { + this.liveQueryResources.set(owner, cleanup) + return () => { + if (this.liveQueryResources.get(owner) === cleanup) { + this.liveQueryResources.delete(owner) + } + } + } + + /** @internal */ + _failPendingLiveQueries(error: unknown): void { + for (const record of this.liveQueries.values()) { + if (record.status === `pending`) record.fail(error) + } + this.emit({ type: `liveQueryStreamError`, error }) + } + + async cleanup(): Promise { + try { + const materializedCollections = Array.from( + this.collectionsById.values(), + ({ collection }) => collection, + ) + const preloadedQueries = Array.from(this.preloadedLiveQueries.values()) + const liveQueryCollections = new Set([ + ...preloadedQueries.map(({ collection }) => collection), + ...materializedCollections.filter((collection) => + getBuilderFromConfig(collection.config), + ), + ]) + + for (const { observer } of preloadedQueries) observer.dispose() + + const cleanupResults = [ + ...(await Promise.allSettled( + Array.from(this.liveQueryResources.values(), (cleanup) => cleanup()), + )), + ...(await Promise.allSettled( + Array.from(liveQueryCollections, (collection) => + collection.cleanup(), + ), + )), + ...(await Promise.allSettled( + materializedCollections + .filter((collection) => !liveQueryCollections.has(collection)) + .map((collection) => collection.cleanup()), + )), + ] + const failure = cleanupResults.find( + (result): result is PromiseRejectedResult => + result.status === `rejected`, + ) + if (failure) throw failure.reason + } finally { + this.transactionScope.clear() + this.collectionsById.clear() + this.pendingHydration.clear() + this.liveQueries.clear() + this.preloadedLiveQueries.clear() + this.liveQueryResources.clear() + this.listeners.clear() + this.ssrStreamingEnabled = false + this.ssrServerCleanupEnabled = false + this.lastLiveQueryTimestamp = 0 + } + } + + private hydrateLiveQuery(dehydratedQuery: DehydratedLiveQuery): void { + const existing = this.liveQueries.get(dehydratedQuery.queryHash) + if (existing && existing.dehydratedAt >= dehydratedQuery.dehydratedAt) { + return + } + + const record = this.createLiveQueryRecord( + dehydratedQuery.queryHash, + dehydratedQuery.dehydratedAt, + ) + this.liveQueries.set(record.queryHash, record) + if (existing?.status === `pending`) { + void record.resultPromise.then(existing.succeed, existing.fail) + } + this.emit({ type: `liveQueryAdded`, query: record }) + + if (dehydratedQuery.snapshot) { + record.succeed(dehydratedQuery.snapshot) + } else if (dehydratedQuery.promise) { + Promise.resolve(dehydratedQuery.promise).then(record.succeed, record.fail) + } else { + record.fail( + new Error( + `Dehydrated live query "${dehydratedQuery.queryHash}" has neither a snapshot nor a promise.`, + ), + ) + } + } + + private nextLiveQueryTimestamp(): number { + this.lastLiveQueryTimestamp = Math.max( + Date.now(), + this.lastLiveQueryTimestamp + 1, + ) + return this.lastLiveQueryTimestamp + } + + private createLiveQueryRecord( + queryHash: string, + dehydratedAt: number, + ): LiveQueryRecord { + this.lastLiveQueryTimestamp = Math.max( + this.lastLiveQueryTimestamp, + dehydratedAt, + ) + + let resolveResult!: (snapshot: DehydratedLiveQueryResult) => void + let rejectResult!: (error: unknown) => void + let settled = false + const resultPromise = new Promise( + (resolve, reject) => { + resolveResult = resolve + rejectResult = reject + }, + ) + const promise = resultPromise.then(() => undefined) + resultPromise.catch(() => {}) + promise.catch(() => {}) + + const record: LiveQueryRecord = { + queryHash, + dehydratedAt, + status: `pending`, + promise, + resultPromise, + succeed: (snapshot) => { + if (settled) return + settled = true + record.status = `success` + record.snapshot = snapshot + resolveResult(snapshot) + if (this.liveQueries.get(queryHash) === record) { + this.emit({ type: `liveQueryUpdated`, query: record }) + } + }, + fail: (error) => { + if (settled) return + settled = true + record.status = `error` + record.error = error + rejectResult(error) + if (this.liveQueries.get(queryHash) === record) { + this.emit({ type: `liveQueryUpdated`, query: record }) + } + }, + } + + return record + } + + private emit(event: DbClientEvent): void { + for (const listener of this.listeners) { + listener(event) + } + } + + private applyRows( + collection: Collection, + chunk: Omit & { + rows: Array< + Omit & { + key?: string | number + } + > + }, + seedKind?: `initialData` | `hydration`, + ): void { + const rows = chunk.rows.flatMap((row) => { + const value = collection.validateData(row.value, `insert`) + const key = collection.config.getKey(value) + const isAdapterAuthoritative = + seedKind === `hydration` && + collection._state.syncedData.has(key) && + !collection._state.hydrationSeedKeys.has(key) + + return isAdapterAuthoritative ? [] : [{ ...row, key, value }] + }) + const rowMetadataWrites = new Map< + string | number, + { type: `set`; value: unknown } | { type: `delete` } + >() + + for (const row of rows) { + if (row.metadata !== undefined) { + rowMetadataWrites.set(row.key, { type: `set`, value: row.metadata }) + } + } + + if (seedKind) { + for (const row of rows) { + collection._state.hydrationSeedKeys.add(row.key) + if (seedKind === `hydration`) { + collection._state.hydratedKeys.add(row.key) + } + } + } + + if (rows.length > 0) { + collection._state.pendingSyncedTransactions.push({ + committed: true, + applicationStarted: false, + layoutChanged: false, + operations: rows.map((row) => ({ + type: collection._state.syncedData.has(row.key) + ? (`update` as const) + : (`insert` as const), + key: row.key, + value: row.value, + })), + deletedKeys: new Set(), + rowMetadataWrites, + collectionMetadataWrites: new Map(), + applied: createDeferred(), + immediate: true, + preserveHydrationSeedKeys: seedKind !== undefined, + }) + collection._state.commitPendingTransactions() + } + + if (chunk.syncMeta !== undefined) { + const currentMeta = collection.config.sync.exportSyncMeta?.() + const mergedMeta = + currentMeta === undefined + ? chunk.syncMeta + : (collection.config.sync.mergeSyncMeta?.( + currentMeta, + chunk.syncMeta, + ) ?? chunk.syncMeta) + collection.config.sync.importSyncMeta?.(mergedMeta) + } + } +} diff --git a/packages/db/src/collection-options.ts b/packages/db/src/collection-options.ts new file mode 100644 index 0000000000..148841c4a6 --- /dev/null +++ b/packages/db/src/collection-options.ts @@ -0,0 +1,34 @@ +import type { StandardSchemaV1 } from '@standard-schema/spec' +import type { CollectionConfig, UtilsRecord } from './types.js' + +export const collectionOptionsBrand: unique symbol = Symbol.for( + `@tanstack/db.collectionOptions`, +) as never + +export const collectionOptionsFactory: unique symbol = Symbol.for( + `@tanstack/db.collectionOptions.factory`, +) as never + +export type CollectionOptionsIdentity< + T extends object = Record, + TKey extends string | number = string | number, + TSchema extends StandardSchemaV1 = never, + TUtils extends UtilsRecord = UtilsRecord, + TClient = unknown, +> = { + readonly id: string + readonly [collectionOptionsBrand]: true + readonly [collectionOptionsFactory]: ( + client: TClient, + ) => CollectionConfig +} + +export function hasCollectionOptionsBrand( + value: unknown, +): value is CollectionOptionsIdentity { + return ( + typeof value === `object` && + value !== null && + (value as Record)[collectionOptionsBrand] === true + ) +} diff --git a/packages/db/src/collection/change-events.ts b/packages/db/src/collection/change-events.ts index 6afac412ce..eca99275d9 100644 --- a/packages/db/src/collection/change-events.ts +++ b/packages/db/src/collection/change-events.ts @@ -1,7 +1,3 @@ -import { - createSingleRowRefProxy, - toExpression, -} from '../query/builder/ref-proxy' import { compileSingleRowExpression, toBooleanPredicate, @@ -20,7 +16,6 @@ import type { SubscribeChangesOptions, } from '../types' import type { CollectionImpl } from './index.js' -import type { SingleRowRefProxy } from '../query/builder/ref-proxy' import type { BasicExpression, OrderBy } from '../query/ir.js' import type { WithVirtualProps } from '../virtual-props.js' @@ -138,11 +133,16 @@ export function currentStateAsChanges< ) if (optimizationResult.canOptimize) { - // Use index optimization + // Use index optimization. When the index lookup is inexact, the keys + // are a superset of the true result (some conditions could not be + // served by an index), so re-check each row against the full expression. + const filterFn = optimizationResult.isExact + ? undefined + : createFilterFunctionFromExpression(expression) const result: Array, TKey>> = [] for (const key of optimizationResult.matchingKeys) { const value = collection.get(key) - if (value !== undefined) { + if (value !== undefined && (filterFn?.(value) ?? true)) { result.push({ type: `insert`, key, @@ -176,44 +176,6 @@ export function currentStateAsChanges< } } -/** - * Creates a filter function from a where callback - * @param whereCallback - The callback function that defines the filter condition - * @returns A function that takes an item and returns true if it matches the filter - */ -export function createFilterFunction( - whereCallback: (row: SingleRowRefProxy) => any, -): (item: T) => boolean { - return (item: T): boolean => { - try { - // First try the RefProxy approach for query builder functions - const singleRowRefProxy = createSingleRowRefProxy() - const whereExpression = whereCallback(singleRowRefProxy) - const expression = toExpression(whereExpression) - const evaluator = compileSingleRowExpression(expression) - const result = evaluator(item as Record) - // WHERE clauses should always evaluate to boolean predicates (Kevin's feedback) - return toBooleanPredicate(result) - } catch { - // If RefProxy approach fails (e.g., arithmetic operations), fall back to direct evaluation - try { - // Create a simple proxy that returns actual values for arithmetic operations - const simpleProxy = new Proxy(item as any, { - get(target, prop) { - return target[prop] - }, - }) as SingleRowRefProxy - - const result = whereCallback(simpleProxy) - return toBooleanPredicate(result) - } catch { - // If both approaches fail, exclude the item - return false - } - } - } -} - /** * Creates a filter function from a pre-compiled expression * @param expression - The pre-compiled expression to evaluate @@ -248,7 +210,7 @@ export function createFilteredCallback< >( originalCallback: (changes: Array>) => void, options: SubscribeChangesOptions, -): (changes: Array>) => void { +): (changes: Array>) => boolean { const filterFn = createFilterFunctionFromExpression(options.whereExpression!) return (changes: Array>) => { @@ -298,7 +260,9 @@ export function createFilteredCallback< // if the original changes array was empty (which indicates a ready signal) if (filteredChanges.length > 0 || changes.length === 0) { originalCallback(filteredChanges) + return true } + return false } } diff --git a/packages/db/src/collection/changes.ts b/packages/db/src/collection/changes.ts index dc07cd3f18..962c7fff6c 100644 --- a/packages/db/src/collection/changes.ts +++ b/packages/db/src/collection/changes.ts @@ -1,4 +1,6 @@ import { NegativeActiveSubscribersError } from '../errors' +import { recordPublicationError, withPublicationContext } from '../scheduler.js' +import { runAllCallbacks } from '../utils/callbacks.js' import { createSingleRowRefProxy, toExpression, @@ -13,6 +15,11 @@ import type { CollectionImpl } from './index.js' import type { CollectionStateManager } from './state.js' import type { WithVirtualProps } from '../virtual-props.js' +export type PublicationDeferral = { + publish: () => void + discard: () => void +} + export class CollectionChangesManager< TOutput extends object = Record, TKey extends string | number = string | number, @@ -29,6 +36,30 @@ export class CollectionChangesManager< public changeSubscriptions = new Set() public batchedEvents: Array> = [] public shouldBatchEvents = false + private publicationDeferralDepth = 0 + private discardDeferredPublications = false + private deferredStateRevision = 0 + private deferredLayoutRevision = 0 + private deferredPublications: Array<{ + changes: Array> + layoutChanged: boolean + }> = [] + private layoutChangeListeners = new Set<() => void>() + + /** + * Monotonic revision of the collection's visible state, advanced once per + * committed batch of changes and cleanup — including while nothing is subscribed. + * Lets consumers (the live-query observer) cheaply detect "did the data + * change" without subscribing, and stays untouched by subscription + * bootstrap replays, which do not go through emitEvents. + */ + public stateRevision = 0 + + /** + * Monotonic revision advanced only for explicit layout-only publications. + * Observers use it to detect reordered rows whose values did not change. + */ + public layoutRevision = 0 /** * Creates a new CollectionChangesManager instance @@ -54,10 +85,17 @@ export class CollectionChangesManager< * This bypasses the normal empty array check in emitEvents */ public emitEmptyReadyEvent(): void { - // Emit empty array directly to all subscribers - for (const subscription of this.changeSubscriptions) { - subscription.emitEvents([]) - } + withPublicationContext(() => { + try { + runAllCallbacks( + [...this.changeSubscriptions].map( + (subscription) => () => subscription.emitEvents([]), + ), + ) + } catch (error) { + recordPublicationError(error) + } + }) } /** @@ -76,7 +114,13 @@ export class CollectionChangesManager< public emitEvents( changes: Array>, forceEmit = false, + layoutChanged = false, ): void { + // The visible state was already committed by the caller, so the revision + // advances even when the events below end up batched for later emission. + if (changes.length > 0) this.stateRevision++ + if (layoutChanged) this.layoutRevision++ + // Skip batching for user actions (forceEmit=true) to keep UI responsive if (this.shouldBatchEvents && !forceEmit) { // Add events to the batch @@ -92,13 +136,81 @@ export class CollectionChangesManager< // buffered optimistic events with the final changes so subscribers see the // whole picture, even if the sync diff is empty. if (this.batchedEvents.length > 0) { - rawEvents = [...this.batchedEvents, ...changes] + const combined = new Map( + this.batchedEvents.map((change) => [change.key, change]), + ) + for (const change of changes) { + const pending = combined.get(change.key) + // A buffered removal was never delivered. Re-insertion replaces the + // subscriber's old row rather than inserting an already-sent key. + combined.set( + change.key, + pending?.type === `delete` && change.type === `insert` + ? { ...change, type: `update`, previousValue: pending.value } + : change, + ) + } + rawEvents = [...combined.values()] } this.batchedEvents = [] this.shouldBatchEvents = false } - if (rawEvents.length === 0) { + if (this.publicationDeferralDepth > 0) { + this.deferredPublications.push({ changes: rawEvents, layoutChanged }) + return + } + + this.publishEvents(rawEvents, layoutChanged) + } + + /** + * Defers subscriber delivery while a coherent multi-Collection publication + * installs all of its visible state. State and indexes still commit at their + * normal transaction boundaries. + */ + public deferPublication(): PublicationDeferral { + if (this.publicationDeferralDepth === 0) { + this.deferredStateRevision = this.stateRevision + this.deferredLayoutRevision = this.layoutRevision + } + this.publicationDeferralDepth++ + let closed = false + + const close = (discard: boolean) => { + if (closed) return + closed = true + if (this.publicationDeferralDepth === 0) return + this.discardDeferredPublications ||= discard + + this.publicationDeferralDepth-- + if (this.publicationDeferralDepth > 0) return + + const publications = this.deferredPublications + this.deferredPublications = [] + if (this.discardDeferredPublications) { + this.discardDeferredPublications = false + this.stateRevision = this.deferredStateRevision + this.layoutRevision = this.deferredLayoutRevision + return + } + this.publishEvents( + publications.flatMap(({ changes }) => changes), + publications.some(({ layoutChanged }) => layoutChanged), + ) + } + + return { + publish: () => close(false), + discard: () => close(true), + } + } + + private publishEvents( + rawEvents: Array>, + layoutChanged: boolean, + ): void { + if (rawEvents.length === 0 && !layoutChanged) { return } @@ -108,10 +220,29 @@ export class CollectionChangesManager< ChangeMessage, TKey> > = rawEvents.map((change) => this.enrichChangeWithVirtualProps(change)) - // Emit to all listeners - for (const subscription of this.changeSubscriptions) { - subscription.emitEvents(enrichedEvents) - } + // Every subscriber sees one committed source batch before dependent query + // graphs run. This keeps repeated aliases and sibling subqueries coherent. + const layoutListeners = [...this.layoutChangeListeners] + const subscriptions = [...this.changeSubscriptions] + withPublicationContext(() => { + const callbacks: Array<() => void> = subscriptions.map( + (subscription) => () => subscription.emitEvents(enrichedEvents), + ) + if (rawEvents.length === 0) { + callbacks.unshift(...layoutListeners) + } + try { + runAllCallbacks(callbacks) + } catch (error) { + recordPublicationError(error) + } + }) + } + + /** Subscribe to layout-only publications. Internal observer channel. */ + public subscribeLayoutChanges(listener: () => void): () => void { + this.layoutChangeListeners.add(listener) + return () => this.layoutChangeListeners.delete(listener) } /** @@ -123,9 +254,6 @@ export class CollectionChangesManager< ) => void, options: SubscribeChangesOptions = {}, ): CollectionSubscription { - // Start sync and track subscriber - this.addSubscriber() - // Compile where callback to whereExpression if provided if (options.where && options.whereExpression) { throw new Error( @@ -141,37 +269,58 @@ export class CollectionChangesManager< whereExpression = toExpression(result) } - const subscription = new CollectionSubscription(this.collection, callback, { - ...opts, - whereExpression, - onUnsubscribe: () => { - this.removeSubscriber() - this.changeSubscriptions.delete(subscription) - }, - }) - - // Register status listener BEFORE requesting snapshot to avoid race condition. - // This ensures the listener catches all status transitions, even if the - // loadSubset promise resolves synchronously or very quickly. - if (options.onStatusChange) { - subscription.on(`status:change`, options.onStatusChange) - } + // Acquire ownership only after all fallible option validation and + // user-provided predicate compilation has completed. + this.addSubscriber() - if (options.includeInitialState) { - subscription.requestSnapshot({ - trackLoadSubsetPromise: false, - orderBy: options.orderBy, - limit: options.limit, - onLoadSubsetResult: options.onLoadSubsetResult, + let subscription: CollectionSubscription | undefined + const setupState = { closed: false } + try { + subscription = new CollectionSubscription(this.collection, callback, { + ...opts, + whereExpression, + onUnsubscribe: () => { + setupState.closed = true + this.removeSubscriber() + if (subscription) this.changeSubscriptions.delete(subscription) + }, }) - } else if (options.includeInitialState === false) { - // When explicitly set to false (not just undefined), mark all state as "seen" - // so that all future changes (including deletes) pass through unfiltered. - subscription.markAllStateAsSeen() - } - // Add to batched listeners - this.changeSubscriptions.add(subscription) + // Register status listener BEFORE requesting snapshot to avoid race condition. + // This ensures the listener catches all status transitions, even if the + // loadSubset promise resolves synchronously or very quickly. + if (options.onStatusChange) { + subscription.on(`status:change`, options.onStatusChange) + } + + if (options.includeInitialState) { + subscription.requestSnapshot({ + trackLoadSubsetPromise: false, + orderBy: options.orderBy, + limit: options.limit, + onLoadSubsetResult: options.onLoadSubsetResult, + }) + } else if (options.includeInitialState === false) { + // When explicitly set to false (not just undefined), mark all state as "seen" + // so that all future changes (including deletes) pass through unfiltered. + subscription.markAllStateAsSeen() + } + + // Add to batched listeners + if (!setupState.closed) this.changeSubscriptions.add(subscription) + } catch (error) { + if (subscription) { + try { + subscription.unsubscribe() + } catch { + // Preserve the setup error. Cleanup still releases subscriber + // ownership and attempts every subset unload before it throws. + } + } else { + this.removeSubscriber() + } + throw error + } return subscription } @@ -184,12 +333,20 @@ export class CollectionChangesManager< this.activeSubscribersCount++ this.lifecycle.cancelGCTimer() - // Start sync if collection was cleaned up - if ( - this.lifecycle.status === `cleaned-up` || - this.lifecycle.status === `idle` - ) { - this.sync.startSync() + try { + // Start sync if collection was cleaned up + if ( + this.lifecycle.status === `cleaned-up` || + this.lifecycle.status === `idle` + ) { + this.sync.startSync() + } + } catch (error) { + this.activeSubscribersCount = previousSubscriberCount + if (this.activeSubscribersCount === 0) { + this.lifecycle.startGCTimer() + } + throw error } this.events.emitSubscribersChange( @@ -222,7 +379,12 @@ export class CollectionChangesManager< * This can be called manually or automatically by garbage collection */ public cleanup(): void { + // Cleanup clears visible state without publishing row changes. Detached + // consumers may miss every status transition before an empty restart. + this.stateRevision++ this.batchedEvents = [] this.shouldBatchEvents = false + this.deferredPublications = [] + this.publicationDeferralDepth = 0 } } diff --git a/packages/db/src/collection/cleanup-queue.ts b/packages/db/src/collection/cleanup-queue.ts index 1acf7751ae..ce43b8dc09 100644 --- a/packages/db/src/collection/cleanup-queue.ts +++ b/packages/db/src/collection/cleanup-queue.ts @@ -42,6 +42,15 @@ export class CleanupQueue { public cancel(key: unknown): void { this.tasks.delete(key) + + // Retire the root timer with the last task. A non-empty queue keeps its + // timer even when the cancelled task was the earliest: it wakes early, + // finds nothing due and reschedules, which costs less than rescanning + // every task on each cancellation. + if (this.tasks.size === 0 && this.timeoutId !== null) { + clearTimeout(this.timeoutId) + this.timeoutId = null + } } /** @@ -66,6 +75,9 @@ export class CleanupQueue { const delay = Math.max(0, earliestTime - Date.now()) this.timeoutId = setTimeout(() => this.process(), delay) + // Background collection GC must not keep an otherwise finished Node + // process alive. Browsers return a numeric timer handle. + if (typeof this.timeoutId === `object`) this.timeoutId.unref() } /** @@ -90,16 +102,4 @@ export class CleanupQueue { this.updateTimeout() } } - - /** - * Resets the singleton instance for tests. - */ - public static resetInstance(): void { - if (CleanupQueue.instance) { - if (CleanupQueue.instance.timeoutId !== null) { - clearTimeout(CleanupQueue.instance.timeoutId) - } - CleanupQueue.instance = null - } - } } diff --git a/packages/db/src/collection/events.ts b/packages/db/src/collection/events.ts index 8058e70b21..1846535737 100644 --- a/packages/db/src/collection/events.ts +++ b/packages/db/src/collection/events.ts @@ -107,15 +107,6 @@ export type AllCollectionEvents = { [K in CollectionStatus as `status:${K}`]: CollectionStatusEvent } -export type CollectionEvent = - | AllCollectionEvents[keyof AllCollectionEvents] - | CollectionStatusChangeEvent - | CollectionSubscribersChangeEvent - | CollectionLoadingSubsetChangeEvent - | CollectionTruncateEvent - | CollectionIndexAddedEvent - | CollectionIndexRemovedEvent - export type CollectionEventHandler = ( event: AllCollectionEvents[T], ) => void @@ -145,22 +136,32 @@ export class CollectionEventsManager extends EventEmitter { emitStatusChange( status: T, previousStatus: CollectionStatus, + isCurrent: () => boolean, ) { - this.emit(`status:change`, { - type: `status:change`, - collection: this.collection, - previousStatus, - status, - }) + this.emitInnerWhile( + `status:change`, + { + type: `status:change`, + collection: this.collection, + previousStatus, + status, + }, + isCurrent, + ) + if (!isCurrent()) return // Emit specific status event using type assertion const eventKey: `status:${T}` = `status:${status}` - this.emit(eventKey, { - type: eventKey, - collection: this.collection, - previousStatus, - status, - } as AllCollectionEvents[`status:${T}`]) + this.emitInnerWhile( + eventKey, + { + type: eventKey, + collection: this.collection, + previousStatus, + status, + } as AllCollectionEvents[`status:${T}`], + isCurrent, + ) } emitSubscribersChange( diff --git a/packages/db/src/collection/index.ts b/packages/db/src/collection/index.ts index e51eb998d6..71a9ed5eb0 100644 --- a/packages/db/src/collection/index.ts +++ b/packages/db/src/collection/index.ts @@ -1,3 +1,5 @@ +import { registerOpaqueHash } from '@tanstack/db-ivm' +import { safeRandomUUID } from '../utils/uuid' import { CollectionConfigurationError, CollectionRequiresConfigError, @@ -12,6 +14,7 @@ import { CollectionSyncManager } from './sync' import { CollectionIndexesManager } from './indexes' import { CollectionMutationsManager } from './mutations' import { CollectionEventsManager } from './events.js' +import type { PublicationDeferral } from './changes' import type { CollectionSubscription } from './subscription' import type { AllCollectionEvents, @@ -41,9 +44,76 @@ import type { import type { SingleRowRefProxy } from '../query/builder/ref-proxy' import type { StandardSchemaV1 } from '@standard-schema/spec' import type { WithVirtualProps } from '../virtual-props.js' +import type { TransactionScope } from '../transactions.js' export type { CollectionIndexMetadata } from './events.js' +const collectionSyncConfigFactory: unique symbol = Symbol.for( + `@tanstack/db.collectionSyncConfig.factory`, +) as never +const collectionSyncConfigCleanup: unique symbol = Symbol.for( + `@tanstack/db.collectionSyncConfig.cleanup`, +) as never + +type CollectionSyncConfigWithFactory = TSync & { + readonly [collectionSyncConfigFactory]: ( + this: TSync, + utilities: object, + ) => TSync +} + +/** @internal Lets adapters bind a sync config to each collection instance. */ +export function withCollectionSyncConfigFactory( + sync: TSync, + factory: (source: TSync, utilities: object) => TSync, +): CollectionSyncConfigWithFactory { + Object.defineProperty(sync, collectionSyncConfigFactory, { + value(this: TSync, utilities: object) { + return factory(this, utilities) + }, + // Preserve the hook when callers wrap a sync config with object spread. + enumerable: true, + }) + return sync as CollectionSyncConfigWithFactory +} + +/** @internal Registers work owned before an adapter sync starts. */ +export function withCollectionSyncConfigCleanup( + sync: TSync, + cleanup: () => void, +): TSync { + Object.defineProperty(sync, collectionSyncConfigCleanup, { + value: cleanup, + enumerable: false, + }) + return sync +} + +function materializeCollectionSyncConfig< + TSync extends object, + TUtils extends object, +>(sync: TSync, utilities: TUtils): { sync: TSync; utilities: TUtils } { + const factory = ( + sync as unknown as Partial> + )[collectionSyncConfigFactory] + if (!factory) return { sync, utilities } + // Binding mutates adapter utilities. Reused/spread descriptors must not + // retarget helpers that already belong to another Collection. Preserve + // accessors and the prototype rather than evaluating them during a spread. + const ownedUtilities = Object.create( + Object.getPrototypeOf(utilities), + Object.getOwnPropertyDescriptors(utilities), + ) as TUtils + return { sync: factory.call(sync, ownedUtilities), utilities: ownedUtilities } +} + +function cleanupCollectionSyncConfig(sync: object): void { + const cleanup = ( + sync as unknown as { [collectionSyncConfigCleanup]?: () => void } + )[collectionSyncConfigCleanup] + cleanup?.() +} + /** * Enhanced Collection interface that includes both data type T and utilities TUtils * @template T - The type of items in the collection @@ -258,14 +328,6 @@ export function createCollection( const collection = new CollectionImpl( options, ) - - // Attach utils to collection - if (options.utils) { - collection.utils = options.utils - } else { - collection.utils = {} - } - return collection } @@ -329,14 +391,21 @@ export class CollectionImpl< if (config.id) { this.id = config.id } else { - this.id = crypto.randomUUID() + this.id = safeRandomUUID() } // Set default values for optional config properties + const { sync: collectionSync, utilities: collectionUtils } = + materializeCollectionSyncConfig(config.sync, config.utils ?? {}) this.config = { ...config, + sync: collectionSync, autoIndex: config.autoIndex ?? `off`, + utils: collectionUtils, } + // Attach utilities before eager sync starts so adapters can bind helpers + // during sync setup. Preserve the adapter's object identity by default. + this.utils = collectionUtils if (this.config.autoIndex === `eager` && !config.defaultIndexType) { throw new CollectionConfigurationError( @@ -347,13 +416,18 @@ export class CollectionImpl< ) } + // Collections are mutable handles, not structural rows. Downstream queries + // must not hash their internal state or follow its ownership cycles. + registerOpaqueHash(this) this._changes = new CollectionChangesManager() this._events = new CollectionEventsManager() this._indexes = new CollectionIndexesManager() - this._lifecycle = new CollectionLifecycleManager(config, this.id) - this._mutations = new CollectionMutationsManager(config, this.id) - this._state = new CollectionStateManager(config) - this._sync = new CollectionSyncManager(config, this.id) + this._lifecycle = new CollectionLifecycleManager(this.config, this.id, () => + cleanupCollectionSyncConfig(this.config.sync), + ) + this._mutations = new CollectionMutationsManager(this.config, this.id) + this._state = new CollectionStateManager(this.config) + this._sync = new CollectionSyncManager(this.config, this.id) this.comparisonOpts = buildCompareOptionsFromConfig(config) @@ -419,9 +493,46 @@ export class CollectionImpl< return this._changes.activeSubscribersCount } + /** + * Monotonic revision of the collection's visible state; advances once per + * committed batch of changes and cleanup, even while nothing is subscribed. + * Internal — used by the live-query observer's snapshot cache. + */ + public get _stateRevision(): number { + return this._changes.stateRevision + } + + /** + * Monotonic revision of explicit layout-only publications. + * Internal — used to distinguish them from empty ready events. + */ + public get _layoutRevision(): number { + return this._changes.layoutRevision + } + + /** Subscribe to layout-only publications. Internal observer channel. */ + public _subscribeLayoutChanges(listener: () => void): () => void { + return this._changes.subscribeLayoutChanges(listener) + } + + /** Mark the active sync transaction as layout-changing. Internal. */ + public _markLayoutChange(): void { + this._sync.markLayoutChange() + } + + /** Defer subscriber events until a coherent multi-Collection commit ends. */ + public _deferPublication(): PublicationDeferral { + return this._changes.deferPublication() + } + /** * Register a callback to be executed when the collection first becomes ready * Useful for preloading collections + * Every callback queued before the transition runs. Because ready state is + * established first, callbacks registered during or after delivery run + * immediately. If one throws, the collection remains ready. Direct sync + * startup rethrows the first failure; preload resolves from ready state. + * Cleanup discards pending callbacks without invoking them. * @param callback Function to call when the collection first becomes ready * @example * collection.onFirstReady(() => { @@ -429,7 +540,7 @@ export class CollectionImpl< * // Safe to access collection.state now * }) */ - public onFirstReady(callback: () => void): void { + public onFirstReady(callback: () => void): () => void { return this._lifecycle.onFirstReady(callback) } @@ -460,11 +571,32 @@ export class CollectionImpl< /** * Start sync immediately - internal method for compiled queries * This bypasses lazy loading for special cases like live query results + * Throws during active cleanup; restart after cleanup completes instead. */ public startSyncImmediate(): void { this._sync.startSync() } + /** @internal */ + public _setTransactionScope(transactionScope: TransactionScope): void { + this._mutations.setTransactionScope(transactionScope) + } + + /** @internal */ + public _hasHydratedKey(key: TKey): boolean { + return this._state.hydratedKeys.has(key) + } + + /** @internal */ + public _deferSyncStart(): boolean { + return this._sync.deferStart() + } + + /** @internal */ + public _resumeSyncStart(): void { + this._sync.resumeStart() + } + /** * Preload the collection data by starting sync if not already started * Multiple concurrent calls will share the same promise @@ -984,6 +1116,8 @@ export class CollectionImpl< /** * Clean up the collection by stopping sync and clearing data * This can be called manually or automatically by garbage collection + * Cleanup callbacks must not restart this collection or call its preload(). + * Wait until cleanup completes before starting a new sync session. */ public async cleanup(): Promise { this._lifecycle.cleanup() diff --git a/packages/db/src/collection/lifecycle.ts b/packages/db/src/collection/lifecycle.ts index a9454ddca7..31ce2b986b 100644 --- a/packages/db/src/collection/lifecycle.ts +++ b/packages/db/src/collection/lifecycle.ts @@ -7,6 +7,7 @@ import { safeCancelIdleCallback, safeRequestIdleCallback, } from '../utils/browser-polyfills' +import { runAllCallbacks } from '../utils/callbacks' import { CleanupQueue } from './cleanup-queue' import type { IdleCallbackDeadline } from '../utils/browser-polyfills' import type { StandardSchemaV1 } from '@standard-schema/spec' @@ -17,6 +18,17 @@ import type { CollectionChangesManager } from './changes' import type { CollectionSyncManager } from './sync' import type { CollectionStateManager } from './state' +/** + * Floor applied to the GC delay of a collection that started syncing before + * anything subscribed. Adapters build their live query while rendering and + * subscribe when that render commits. This grace period reduces cleanup + * during that gap; a later subscriber can still restart sync. Adapters pass + * a near-zero `gcTime` to make teardown on unmount immediate. Does not apply + * to the timer armed when the last subscriber leaves, which still honours + * `gcTime` exactly. + */ +const UNSUBSCRIBED_GC_FLOOR_MS = 50 + export class CollectionLifecycleManager< TOutput extends object = Record, TKey extends string | number = string | number, @@ -36,13 +48,22 @@ export class CollectionLifecycleManager< public hasReceivedFirstCommit = false public onFirstReadyCallbacks: Array<() => void> = [] private idleCallbackId: number | null = null + private syncError: unknown + private cleanupConfig: () => void + private statusRevision = 0 + private cleaningUp = false /** * Creates a new CollectionLifecycleManager instance */ - constructor(config: CollectionConfig, id: string) { + constructor( + config: CollectionConfig, + id: string, + cleanupConfig: () => void = () => {}, + ) { this.config = config this.id = id + this.cleanupConfig = cleanupConfig } setDeps(deps: { @@ -77,7 +98,7 @@ export class CollectionLifecycleManager< idle: [`loading`, `error`, `cleaned-up`], loading: [`ready`, `error`, `cleaned-up`], ready: [`cleaned-up`, `error`], - error: [`cleaned-up`, `idle`], + error: [`ready`, `cleaned-up`, `idle`], 'cleaned-up': [`loading`, `error`], } @@ -103,11 +124,16 @@ export class CollectionLifecycleManager< ) } this.validateStatusTransition(this.status, newStatus) + const revision = ++this.statusRevision const previousStatus = this.status this.status = newStatus // Emit event - this.events.emitStatusChange(newStatus, previousStatus) + this.events.emitStatusChange( + newStatus, + previousStatus, + () => this.statusRevision === revision, + ) } /** @@ -132,11 +158,34 @@ export class CollectionLifecycleManager< * @private - Should only be called by sync implementations */ public markReady(): void { + const failure = this.applyReadyTransition() + if (failure) throw failure.error + } + + /** @internal Capture ready-effect failures while the sync entry completes. */ + public markReadyDuringSyncStart(): { error: unknown } | undefined { + return this.applyReadyTransition() + } + + private applyReadyTransition(): { error: unknown } | undefined { this.validateStatusTransition(this.status, `ready`) - // Can transition to ready from loading state - if (this.status === `loading`) { + // A successful initial sync or recovery establishes a ready snapshot. + if (this.status === `loading` || this.status === `error`) { + this.syncError = undefined + const readyRevision = this.statusRevision + 1 this.setStatus(`ready`, true) + // A status listener can synchronously supersede this transition, even + // when it restarts the Collection back to ready before returning. + if ( + (this.status as CollectionStatus) !== `ready` || + this.statusRevision !== readyRevision + ) { + return undefined + } + + const readyEffects: Array<() => void> = [] + // Call any registered first ready callbacks (only on first time becoming ready) if (!this.hasBeenReady) { this.hasBeenReady = true @@ -146,23 +195,64 @@ export class CollectionLifecycleManager< this.hasReceivedFirstCommit = true } - const callbacks = [...this.onFirstReadyCallbacks] + readyEffects.push(...this.onFirstReadyCallbacks) this.onFirstReadyCallbacks = [] - callbacks.forEach((callback) => callback()) } // Notify dependents when markReady is called, after status is set // This ensures live queries get notified when their dependencies become ready - if (this.changes.changeSubscriptions.size > 0) { - this.changes.emitEmptyReadyEvent() + readyEffects.push(() => this.changes.emitEmptyReadyEvent()) + try { + runAllCallbacks(readyEffects) + } catch (error) { + return { error } } } + return undefined + } + + /** Mark an asynchronous sync failure after sync has started. */ + public markError(error?: unknown): void { + this.validateStatusTransition(this.status, `error`) + this.syncError = error + this.setStatus(`error`) + } + + /** Return the cause supplied by the current sync session, if any. */ + public getSyncError(): unknown { + return this.syncError + } + + public assertCanStartSync(): void { + if (this.cleaningUp) { + throw new CollectionStateError( + `Cannot start collection "${this.id}" during cleanup. Restart after cleanup() completes.`, + ) + } + } + + /** + * Start the garbage collection timer for a collection with no subscribers + * Called when sync starts outside a subscription + */ + public startGCTimerIfUnsubscribed(): void { + this.startGCTimer(UNSUBSCRIBED_GC_FLOOR_MS) + } + + private canGarbageCollect(): boolean { + return ( + !this.cleaningUp && + this.changes.activeSubscribersCount === 0 && + !this.sync.hasPendingPreload + ) } /** * Start the garbage collection timer * Called when the collection becomes inactive (no subscribers) */ - public startGCTimer(): void { + public startGCTimer(minDelay = 0): void { + if (!this.canGarbageCollect()) return + const gcTime = this.config.gcTime ?? 300000 // 5 minutes default // If gcTime is 0, negative, or non-finite (Infinity, -Infinity, NaN), GC is disabled. @@ -172,12 +262,16 @@ export class CollectionLifecycleManager< return } - CleanupQueue.getInstance().schedule(this, gcTime, () => { - if (this.changes.activeSubscribersCount === 0) { - // Schedule cleanup during idle time to avoid blocking the UI thread - this.scheduleIdleCleanup() - } - }) + CleanupQueue.getInstance().schedule( + this, + Math.max(gcTime, minDelay), + () => { + if (this.canGarbageCollect()) { + // Schedule cleanup during idle time to avoid blocking the UI thread + this.scheduleIdleCleanup() + } + }, + ) } /** @@ -208,7 +302,7 @@ export class CollectionLifecycleManager< this.idleCallbackId = safeRequestIdleCallback( (deadline) => { // Perform cleanup if we still have no subscribers - if (this.changes.activeSubscribersCount === 0) { + if (this.canGarbageCollect()) { const cleanupCompleted = this.performCleanup(deadline) // Only clear the callback ID if cleanup actually completed if (cleanupCompleted) { @@ -228,43 +322,44 @@ export class CollectionLifecycleManager< * @returns true if cleanup was completed, false if it was rescheduled */ private performCleanup(deadline?: IdleCallbackDeadline): boolean { + // Nested cleanup belongs to this retirement, not a new lifecycle turn. + if (this.cleaningUp) return true // If we have a deadline, we can potentially split cleanup into chunks // For now, we'll do all cleanup at once but check if we have time const hasTime = !deadline || deadline.timeRemaining() > 0 || deadline.didTimeout if (hasTime) { - // Perform all cleanup operations except events - this.sync.cleanup() - this.state.cleanup() - this.changes.cleanup() - this.indexes.cleanup() - - CleanupQueue.getInstance().cancel(this) - - this.hasBeenReady = false - - // Call any pending onFirstReady callbacks before clearing them. - // This ensures preload() promises resolve during cleanup instead of hanging. - const callbacks = [...this.onFirstReadyCallbacks] - this.onFirstReadyCallbacks = [] - callbacks.forEach((callback) => { - try { - callback() - } catch (error) { - console.error( - `${this.config.id ? `[${this.config.id}] ` : ``}Error in onFirstReady callback during cleanup:`, - error, - ) - } - }) + this.cleaningUp = true + try { + // Perform all cleanup operations except events + this.cleanupConfig() + this.sync.cleanup() + this.state.cleanup() + this.changes.cleanup() + this.indexes.cleanup() + + CleanupQueue.getInstance().cancel(this) + + this.hasBeenReady = false + this.syncError = undefined + + // Cleanup is not readiness. Sync cleanup rejects pending preload callers; + // first-ready listeners belong to the discarded run. + this.onFirstReadyCallbacks = [] + } finally { + this.cleaningUp = false + } // Set status to cleaned-up after everything is cleaned up // This fires the status:change event to notify listeners this.setStatus(`cleaned-up`) - // Finally, cleanup event handlers after the event has been fired - this.events.cleanup() + // Active collection subscriptions still depend on lifecycle events. + // Once the last subscriber leaves, its GC cleanup clears the handlers. + if (this.changes.activeSubscribersCount === 0) { + this.events.cleanup() + } return true } else { @@ -279,14 +374,20 @@ export class CollectionLifecycleManager< * Useful for preloading collections * @param callback Function to call when the collection first becomes ready */ - public onFirstReady(callback: () => void): void { + public onFirstReady(callback: () => void): () => void { // If already ready, call immediately if (this.hasBeenReady) { callback() - return + return () => {} } this.onFirstReadyCallbacks.push(callback) + return () => { + const index = this.onFirstReadyCallbacks.indexOf(callback) + if (index !== -1) { + this.onFirstReadyCallbacks.splice(index, 1) + } + } } public cleanup(): void { diff --git a/packages/db/src/collection/mutations.ts b/packages/db/src/collection/mutations.ts index 765e409ef6..328d481657 100644 --- a/packages/db/src/collection/mutations.ts +++ b/packages/db/src/collection/mutations.ts @@ -1,4 +1,5 @@ import { withArrayChangeTracking, withChangeTracking } from '../proxy' +import { safeRandomUUID } from '../utils/uuid' import { createTransaction, getActiveTransaction } from '../transactions' import { DeleteKeyNotFoundError, @@ -26,11 +27,13 @@ import type { OperationConfig, PendingMutation, StandardSchema, + TransactionConfig, Transaction as TransactionType, TransactionWithMutations, UtilsRecord, WritableDeep, } from '../types' +import type { TransactionScope } from '../transactions' import type { CollectionLifecycleManager } from './lifecycle' import type { CollectionStateManager } from './state' @@ -45,6 +48,7 @@ export class CollectionMutationsManager< private state!: CollectionStateManager private collection!: CollectionImpl private config!: CollectionConfig + private transactionScope?: TransactionScope private id: string constructor(config: CollectionConfig, id: string) { @@ -62,6 +66,22 @@ export class CollectionMutationsManager< this.collection = deps.collection } + setTransactionScope(transactionScope: TransactionScope): void { + this.transactionScope = transactionScope + } + + private getActiveTransaction() { + return this.transactionScope + ? this.transactionScope.getActiveTransactionForCollection() + : getActiveTransaction() + } + + private createTransaction(config: TransactionConfig) { + return this.transactionScope + ? this.transactionScope.createTransaction(config) + : createTransaction(config) + } + private ensureStandardSchema(schema: unknown): StandardSchema { // If the schema already implements the standard-schema interface, return it if (schema && `~standard` in (schema as {})) { @@ -154,11 +174,13 @@ export class CollectionMutationsManager< return `KEY::${this.id}/${key}` } - private markPendingLocalOrigins( + private markPendingLocalChanges( mutations: Array>, ): void { for (const mutation of mutations) { - this.state.pendingLocalOrigins.add(mutation.key as TKey) + // The handler can sync synchronously before its transaction is registered. + // This is provisional; only completed mutations retain a local origin. + this.state.pendingLocalChanges.add(mutation.key as TKey) } } @@ -168,7 +190,7 @@ export class CollectionMutationsManager< insert = (data: TInput | Array, config?: InsertConfig) => { this.lifecycle.validateCollectionUsable(`insert`) const state = this.state - const ambientTransaction = getActiveTransaction() + const ambientTransaction = this.getActiveTransaction() // If no ambient transaction exists, check for an onInsert handler early if (!ambientTransaction && !this.config.onInsert) { @@ -193,7 +215,7 @@ export class CollectionMutationsManager< const globalKey = this.generateGlobalKey(key, item) const mutation: PendingMutation = { - mutationId: crypto.randomUUID(), + mutationId: safeRandomUUID(), original: {}, modified: validatedData, // Pick the values from validatedData based on what's passed in - this is for cases @@ -230,7 +252,7 @@ export class CollectionMutationsManager< return ambientTransaction } else { // Create a new transaction with a mutation function that calls the onInsert handler - const directOpTransaction = createTransaction({ + const directOpTransaction = this.createTransaction({ metadata: { [DIRECT_TRANSACTION_METADATA_KEY]: true }, mutationFn: async (params) => { // Call the onInsert handler with the transaction and collection @@ -247,7 +269,7 @@ export class CollectionMutationsManager< // Apply mutations to the new transaction directOpTransaction.applyMutations(mutations) - this.markPendingLocalOrigins(mutations) + this.markPendingLocalChanges(mutations) // Errors still reject tx.isPersisted.promise; this catch only prevents global unhandled rejections directOpTransaction.commit().catch(() => undefined) @@ -280,7 +302,7 @@ export class CollectionMutationsManager< const state = this.state this.lifecycle.validateCollectionUsable(`update`) - const ambientTransaction = getActiveTransaction() + const ambientTransaction = this.getActiveTransaction() // If no ambient transaction exists, check for an onUpdate handler early if (!ambientTransaction && !this.config.onUpdate) { @@ -366,7 +388,7 @@ export class CollectionMutationsManager< const globalKey = this.generateGlobalKey(modifiedItemId, modifiedItem) return { - mutationId: crypto.randomUUID(), + mutationId: safeRandomUUID(), original: originalItem, modified: modifiedItem, // Pick the values from modifiedItem based on what's passed in - this is for cases @@ -403,7 +425,7 @@ export class CollectionMutationsManager< // If no changes were made, return an empty transaction early if (mutations.length === 0) { - const emptyTransaction = createTransaction({ + const emptyTransaction = this.createTransaction({ mutationFn: async () => {}, }) // Errors still propagate through tx.isPersisted.promise; suppress the background commit from warning @@ -427,7 +449,7 @@ export class CollectionMutationsManager< // No need to check for onUpdate handler here as we've already checked at the beginning // Create a new transaction with a mutation function that calls the onUpdate handler - const directOpTransaction = createTransaction({ + const directOpTransaction = this.createTransaction({ metadata: { [DIRECT_TRANSACTION_METADATA_KEY]: true }, mutationFn: async (params) => { // Call the onUpdate handler with the transaction and collection @@ -444,7 +466,7 @@ export class CollectionMutationsManager< // Apply mutations to the new transaction directOpTransaction.applyMutations(mutations) - this.markPendingLocalOrigins(mutations) + this.markPendingLocalChanges(mutations) // Errors still hit tx.isPersisted.promise; avoid leaking an unhandled rejection from the fire-and-forget commit directOpTransaction.commit().catch(() => undefined) @@ -467,7 +489,7 @@ export class CollectionMutationsManager< const state = this.state this.lifecycle.validateCollectionUsable(`delete`) - const ambientTransaction = getActiveTransaction() + const ambientTransaction = this.getActiveTransaction() // If no ambient transaction exists, check for an onDelete handler early if (!ambientTransaction && !this.config.onDelete) { @@ -497,7 +519,7 @@ export class CollectionMutationsManager< `delete`, CollectionImpl > = { - mutationId: crypto.randomUUID(), + mutationId: safeRandomUUID(), original: this.state.get(key)!, modified: this.state.get(key)!, changes: this.state.get(key)!, @@ -530,7 +552,7 @@ export class CollectionMutationsManager< } // Create a new transaction with a mutation function that calls the onDelete handler - const directOpTransaction = createTransaction({ + const directOpTransaction = this.createTransaction({ autoCommit: true, metadata: { [DIRECT_TRANSACTION_METADATA_KEY]: true }, mutationFn: async (params) => { @@ -548,7 +570,7 @@ export class CollectionMutationsManager< // Apply mutations to the new transaction directOpTransaction.applyMutations(mutations) - this.markPendingLocalOrigins(mutations) + this.markPendingLocalChanges(mutations) // Errors still reject tx.isPersisted.promise; silence the internal commit promise to prevent test noise directOpTransaction.commit().catch(() => undefined) diff --git a/packages/db/src/collection/state.ts b/packages/db/src/collection/state.ts index 9cbdebb234..aac2e7c84f 100644 --- a/packages/db/src/collection/state.ts +++ b/packages/db/src/collection/state.ts @@ -1,6 +1,7 @@ import { deepEquals } from '../utils' import { SortedMap } from '../SortedMap' import { enrichRowWithVirtualProps } from '../virtual-props.js' +import { SyncTransactionAbortedError } from '../errors.js' import { DIRECT_TRANSACTION_METADATA_KEY } from './transaction-metadata.js' import type { VirtualOrigin, @@ -13,27 +14,34 @@ import type { ChangeMessage, CollectionConfig, OptimisticChangeMessage, + PendingMutation, } from '../types' import type { CollectionImpl } from './index.js' import type { CollectionLifecycleManager } from './lifecycle' import type { CollectionChangesManager } from './changes' import type { CollectionIndexesManager } from './indexes' import type { CollectionEventsManager } from './events' +import type { Deferred } from '../deferred' interface PendingSyncedTransaction< T extends object = Record, TKey extends string | number = string | number, > { committed: boolean + applicationStarted: boolean + layoutChanged: boolean operations: Array> truncate?: boolean deletedKeys: Set rowMetadataWrites: Map collectionMetadataWrites: Map + /** Resolves after application and rejects if canceled before application. */ + applied: Deferred optimisticSnapshot?: { upserts: Map deletes: Set } + preserveHydrationSeedKeys?: boolean /** * When true, this transaction should be processed immediately even if there * are persisting user transactions. Used by manual write operations (writeInsert, @@ -44,6 +52,11 @@ interface PendingSyncedTransaction< type PendingMetadataWrite = { type: `set`; value: unknown } | { type: `delete` } +type OptimisticUpsert = Pick< + PendingMutation, + `key` | `modified` +> & { insert?: object } + type InternalChangeMessage< T extends object = Record, TKey extends string | number = string | number, @@ -75,14 +88,18 @@ export class CollectionStateManager< public syncedData: SortedMap public syncedMetadata = new Map() public syncedCollectionMetadata = new Map() + public hydrationSeedKeys = new Set() + public hydratedKeys = new Set() // Optimistic state tracking - make public for testing public optimisticUpserts = new Map() public optimisticDeletes = new Set() - public pendingOptimisticUpserts = new Map() + + public pendingOptimisticUpserts = new Map>() public pendingOptimisticDeletes = new Set() public pendingOptimisticDirectUpserts = new Set() public pendingOptimisticDirectDeletes = new Set() + private acknowledgedInserts = new WeakSet() /** * Tracks the origin of confirmed changes for each row. @@ -101,6 +118,8 @@ export class CollectionStateManager< * When sync confirms data for a key with pending local changes, it keeps 'local' origin. */ public pendingLocalChanges = new Set() + // Successful mutations retain attribution until sync applies. Active or + // failed mutations must not add to, or erase a sibling's entry in, this set. public pendingLocalOrigins = new Set() private virtualPropsCache = new WeakMap< @@ -118,11 +137,13 @@ export class CollectionStateManager< public size = 0 // State used for computing the change events - public syncedKeys = new Set() public preSyncVisibleState = new Map() + public preSyncVirtualState = new Map>() public recentlySyncedKeys = new Set() public hasReceivedFirstCommit = false public isCommittingSyncTransactions = false + private isDrainingSyncTransactions = false + private syncSessionGeneration = 0 public isLocalOnly = false /** @@ -154,7 +175,12 @@ export class CollectionStateManager< } /** - * Checks if a row has pending optimistic mutations (not yet confirmed by sync). + * Checks whether this row currently has no pending local optimistic writes. + * + * This is local mutation status, not backend confirmation: `true` means the + * row is not currently affected by an optimistic transaction in this + * collection's visible state. + * * Used to compute the $synced virtual property. */ public isRowSynced(key: TKey): boolean { @@ -226,6 +252,21 @@ export class CollectionStateManager< }) } + private snapshotRowOriginsForKeys( + keys: Iterable, + ): Map { + const rowOrigins = new Map() + + for (const key of keys) { + const origin = this.rowOrigins.get(key) + if (origin !== undefined) { + rowOrigins.set(key, origin) + } + } + + return rowOrigins + } + private enrichWithVirtualPropsSnapshot( row: TOutput, virtualProps: VirtualRowProps, @@ -476,9 +517,12 @@ export class CollectionStateManager< const previousState = new Map(this.optimisticUpserts) const previousDeletes = new Set(this.optimisticDeletes) - const previousRowOrigins = new Map(this.rowOrigins) + const previousRowOrigins = this.rowOrigins - // Update pending optimistic state for completed/failed transactions + // A retained update can depend on an unconfirmed insert, not just its key. + // Keep that exact dependency so settlement and key reuse cannot conflate rows. + const pendingInserts = new Map() + // Retain successful contributions; failed/active work is recomputed below. for (const transaction of this.transactions.values()) { const isDirectTransaction = transaction.metadata[DIRECT_TRANSACTION_METADATA_KEY] === true @@ -487,17 +531,37 @@ export class CollectionStateManager< if (!this.isThisCollection(mutation.collection)) { continue } + // Only a sync write during this insertion acknowledges it. A stale + // base row can also exist after an optimistic delete and reinsert. + if ( + isDirectTransaction && + mutation.type === `insert` && + this.acknowledgedInserts.has(mutation) + ) { + continue + } this.pendingLocalOrigins.add(mutation.key) if (!mutation.optimistic) { continue } switch (mutation.type) { case `insert`: - case `update`: - this.pendingOptimisticUpserts.set( - mutation.key, - mutation.modified as TOutput, - ) + case `update`: { + // Retain whole snapshots, not fields rebased over newer synced + // values. A slow insert must not replace its accepted dependent + // update with the older insertion snapshot. + const previous = this.pendingOptimisticUpserts.get(mutation.key) + const confirmsInsert = previous?.insert === mutation + this.pendingOptimisticUpserts.set(mutation.key, { + key: mutation.key, + modified: confirmsInsert + ? previous.modified + : mutation.modified, + insert: + mutation.type === `update` + ? (previous?.insert ?? pendingInserts.get(mutation.key)) + : undefined, + }) this.pendingOptimisticDeletes.delete(mutation.key) if (isDirectTransaction) { this.pendingOptimisticDirectUpserts.add(mutation.key) @@ -507,6 +571,7 @@ export class CollectionStateManager< this.pendingOptimisticDirectDeletes.delete(mutation.key) } break + } case `delete`: this.pendingOptimisticUpserts.delete(mutation.key) this.pendingOptimisticDeletes.add(mutation.key) @@ -520,17 +585,26 @@ export class CollectionStateManager< break } } - } else if (transaction.state === `failed`) { + } else { for (const mutation of transaction.mutations) { - if (!this.isThisCollection(mutation.collection)) { + if ( + !this.isThisCollection(mutation.collection) || + mutation.type !== `insert` || + !mutation.optimistic + ) continue - } - this.pendingLocalOrigins.delete(mutation.key) - if (mutation.optimistic) { + if ( + transaction.state !== `failed` && + !this.acknowledgedInserts.has(mutation) + ) { + pendingInserts.set(mutation.key, mutation) + } else if ( + this.pendingOptimisticUpserts.get(mutation.key)?.insert === mutation + ) { + // Drop only the dependent row, never a later same-key insertion or + // successful sibling attribution. Failed transactions remain listed. this.pendingOptimisticUpserts.delete(mutation.key) - this.pendingOptimisticDeletes.delete(mutation.key) this.pendingOptimisticDirectUpserts.delete(mutation.key) - this.pendingOptimisticDirectDeletes.delete(mutation.key) } } } @@ -554,7 +628,7 @@ export class CollectionStateManager< pendingSyncKeys.has(key) || this.pendingOptimisticDirectUpserts.has(key) ) { - this.optimisticUpserts.set(key, value) + this.optimisticUpserts.set(key, this.resolveOptimisticUpsert(value)) } else { staleOptimisticUpserts.push(key) } @@ -603,7 +677,7 @@ export class CollectionStateManager< case `update`: this.optimisticUpserts.set( mutation.key, - mutation.modified as TOutput, + this.resolveOptimisticUpsert(mutation), ) this.optimisticDeletes.delete(mutation.key) break @@ -648,7 +722,7 @@ export class CollectionStateManager< // Filter out redundant delete events if there are pending sync transactions // that will immediately restore the same data, but only for completed transactions // IMPORTANT: Skip complex filtering for user-triggered actions to prevent UI blocking - if (this.pendingSyncedTransactions.length > 0 && !triggeredByUserAction) { + if (this.changes.shouldBatchEvents && !triggeredByUserAction) { const pendingSyncKeysForFilter = new Set() // Collect keys from pending sync operations @@ -762,7 +836,9 @@ export class CollectionStateManager< } else if ( previousValue !== undefined && currentValue !== undefined && - previousValue !== currentValue + (!deepEquals(previousValue, currentValue) || + previousVirtualProps.$origin !== nextVirtualProps.$origin || + previousVirtualProps.$synced !== nextVirtualProps.$synced) ) { events.push({ type: `update`, @@ -778,6 +854,37 @@ export class CollectionStateManager< } } + // Optimistic mutations keep their full validated snapshot. Mixing in newer + // synced fields could produce a row neither the user nor the server created. + private resolveOptimisticUpsert( + mutation: OptimisticUpsert, + ): TOutput { + const key = mutation.key as TKey + const dependent = this.pendingOptimisticUpserts.get(key) + return dependent?.insert === mutation + ? dependent.modified + : mutation.modified + } + + /** Build once per output flush; queued membership excludes optimistic edits. */ + createSyncedKeyLookup(): (key: TKey) => boolean { + if (this.pendingSyncedTransactions.length === 0) + return (key) => this.syncedData.has(key) + const queued = new Map() + let truncated = false + for (const transaction of this.pendingSyncedTransactions) { + if (!transaction.committed) continue + if (transaction.truncate) { + queued.clear() + truncated = true + } + for (const operation of transaction.operations) { + queued.set(operation.key as TKey, operation.type !== `delete`) + } + } + return (key) => queued.get(key) ?? (!truncated && this.syncedData.has(key)) + } + /** * Get the previous value for a key given previous optimistic state */ @@ -800,6 +907,30 @@ export class CollectionStateManager< * This method processes operations from pending transactions and applies them to the synced data */ commitPendingTransactions = () => { + if (this.isDrainingSyncTransactions) return + this.isDrainingSyncTransactions = true + let failed = false + let firstError: unknown + try { + let result: { processed: boolean; failure?: { error: unknown } } + do { + result = this.commitNextPendingTransactionBatch() + if (result.failure && !failed) { + failed = true + firstError = result.failure.error + } + } while (result.processed) + } finally { + this.isDrainingSyncTransactions = false + } + if (failed) throw firstError + } + + private commitNextPendingTransactionBatch(): { + processed: boolean + failure?: { error: unknown } + } { + const syncSessionGeneration = this.syncSessionGeneration // Check if there are any persisting transaction let hasPersistingTransaction = false for (const transaction of this.transactions.values()) { @@ -816,10 +947,12 @@ export class CollectionStateManager< uncommittedSyncedTransactions, hasTruncateSync, hasImmediateSync, + layoutChanged, } = this.pendingSyncedTransactions.reduce( (acc, t) => { if (t.committed) { acc.committedSyncedTransactions.push(t) + acc.layoutChanged ||= t.layoutChanged if (t.truncate) { acc.hasTruncateSync = true } @@ -840,9 +973,14 @@ export class CollectionStateManager< >, hasTruncateSync: false, hasImmediateSync: false, + layoutChanged: false, }, ) + if (committedSyncedTransactions.length === 0) { + return { processed: false } + } + // Process committed transactions if: // 1. No persisting user transaction (normal sync flow), OR // 2. There's a truncate operation (must be processed immediately), OR @@ -854,13 +992,19 @@ export class CollectionStateManager< // non-immediate transactions would be applied later and could overwrite newer state. // Processing all committed transactions together preserves causal ordering. if (!hasPersistingTransaction || hasTruncateSync || hasImmediateSync) { + const previousLayout = layoutChanged ? [...this.keys()] : undefined + this.pendingSyncedTransactions = uncommittedSyncedTransactions + + // Application is now the point of no return. Event listeners run before + // the receipts resolve, so a signal aborted from one of those listeners + // must not cancel writes that are already becoming visible. + for (const transaction of committedSyncedTransactions) { + transaction.applicationStarted = true + } + // Set flag to prevent redundant optimistic state recalculations this.isCommittingSyncTransactions = true - const previousRowOrigins = new Map(this.rowOrigins) - const previousOptimisticUpserts = new Map(this.optimisticUpserts) - const previousOptimisticDeletes = new Set(this.optimisticDeletes) - // Get the optimistic snapshot from the truncate transaction (captured when truncate() was called) const truncateOptimisticSnapshot = hasTruncateSync ? committedSyncedTransactions.find((t) => t.truncate) @@ -871,15 +1015,36 @@ export class CollectionStateManager< // First collect all keys that will be affected by sync operations const changedKeys = new Set() + const syncedInsertedOrUpdatedKeys = new Set() for (const transaction of committedSyncedTransactions) { for (const operation of transaction.operations) { changedKeys.add(operation.key as TKey) + if (operation.type !== `delete`) + syncedInsertedOrUpdatedKeys.add(operation.key as TKey) } for (const [key] of transaction.rowMetadataWrites) { changedKeys.add(key) } } + const virtualSnapshotKeys = new Set(changedKeys) + for (const key of this.pendingOptimisticDirectUpserts) { + virtualSnapshotKeys.add(key) + } + for (const key of this.pendingOptimisticDirectDeletes) { + virtualSnapshotKeys.add(key) + } + const previousRowOrigins = + this.snapshotRowOriginsForKeys(virtualSnapshotKeys) + const previousOptimisticUpserts = new Map(this.optimisticUpserts) + const previousOptimisticDeletes = new Set(this.optimisticDeletes) + const completedDirectUpserts = new Set( + this.pendingOptimisticDirectUpserts, + ) + const completedDirectDeletes = new Set( + this.pendingOptimisticDirectDeletes, + ) + // Use pre-captured state if available (from optimistic scenarios), // otherwise capture current state (for pure sync scenarios) let currentVisibleState = this.preSyncVisibleState @@ -946,7 +1111,8 @@ export class CollectionStateManager< truncatePendingLocalOrigins = new Set(this.pendingLocalOrigins) this.syncedData.clear() this.syncedMetadata.clear() - this.syncedKeys.clear() + this.hydrationSeedKeys.clear() + this.hydratedKeys.clear() this.clearOriginTrackingState() // 3) Clear currentVisibleState for truncated keys to ensure subsequent operations @@ -963,19 +1129,27 @@ export class CollectionStateManager< }) } + // Attribution belongs to the whole atomic batch. A repeated write must + // not forget the local acknowledgement consumed by its first operation. + const localKeys = new Set() for (const operation of transaction.operations) { const key = operation.key as TKey - this.syncedKeys.add(key) // Determine origin: 'local' for local-only collections or pending local changes + const retainedLocalOrigin = + truncatePendingLocalChanges?.has(key) === true || + (truncatePendingLocalOrigins?.has(key) === true && + !completedDirectUpserts.has(key) && + !completedDirectDeletes.has(key)) const origin: VirtualOrigin = this.isLocalOnly || this.pendingLocalChanges.has(key) || this.pendingLocalOrigins.has(key) || - truncatePendingLocalChanges?.has(key) === true || - truncatePendingLocalOrigins?.has(key) === true + localKeys.has(key) || + retainedLocalOrigin ? 'local' : 'remote' + if (origin === `local`) localKeys.add(key) // Update synced data switch (operation.type) { @@ -1024,6 +1198,10 @@ export class CollectionStateManager< this.pendingOptimisticDirectDeletes.delete(key) break } + if (!transaction.preserveHydrationSeedKeys) { + this.hydrationSeedKeys.delete(key) + this.hydratedKeys.delete(key) + } } for (const [key, metadataWrite] of transaction.rowMetadataWrites) { @@ -1046,70 +1224,45 @@ export class CollectionStateManager< } } - // After applying synced operations, if this commit included a truncate, - // re-apply optimistic mutations on top of the fresh synced base. This ensures - // the UI preserves local intent while respecting server rebuild semantics. - // Ordering: deletes (above) -> server ops (just applied) -> optimistic upserts. - if (hasTruncateSync) { - // Avoid duplicating keys that were inserted/updated by synced operations in this commit - const syncedInsertedOrUpdatedKeys = new Set() - for (const t of committedSyncedTransactions) { - for (const op of t.operations) { - if (op.type === `insert` || op.type === `update`) { - syncedInsertedOrUpdatedKeys.add(op.key as TKey) - } - } - } - - // Build re-apply sets from the snapshot taken at the start of this function. - // This prevents losing optimistic state if transactions complete during truncate processing. - const reapplyUpserts = new Map( - truncateOptimisticSnapshot!.upserts, - ) - const reapplyDeletes = new Set( - truncateOptimisticSnapshot!.deletes, + // A completed optimistic insert may have used a temporary client key while + // the sync confirmation used a different server-generated key. Once a + // sync commit has been applied, stop retaining completed optimistic keys + // that were not confirmed by this commit so the temporary row is removed. + for (const key of this.pendingOptimisticDirectUpserts) { + // Truncate republishes this captured snapshot. Keep its existing + // retention marker so the next sync can also publish its removal. + if ( + hasTruncateSync && + truncateOptimisticSnapshot?.upserts.has(key) && + !changedKeys.has(key) ) - - // Emit inserts for re-applied upserts, skipping any keys that have an optimistic delete. - // If the server also inserted/updated the same key in this batch, override that value - // with the optimistic value to preserve local intent. - for (const [key, value] of reapplyUpserts) { - if (reapplyDeletes.has(key)) continue - if (syncedInsertedOrUpdatedKeys.has(key)) { - let foundInsert = false - for (let i = events.length - 1; i >= 0; i--) { - const evt = events[i]! - if (evt.key === key && evt.type === `insert`) { - evt.value = value - foundInsert = true - break - } - } - if (!foundInsert) { - events.push({ type: `insert`, key, value }) - } - } else { - events.push({ type: `insert`, key, value }) - } - } - - // Finally, ensure we do NOT insert keys that have an outstanding optimistic delete. - if (events.length > 0 && reapplyDeletes.size > 0) { - const filtered: Array> = [] - for (const evt of events) { - if (evt.type === `insert` && reapplyDeletes.has(evt.key)) { - continue + continue + if (!changedKeys.has(key)) { + changedKeys.add(key) + if (!currentVisibleState.has(key)) { + const previousValue = previousOptimisticUpserts.get(key) + if (previousValue !== undefined) { + currentVisibleState.set(key, previousValue) } - filtered.push(evt) } - events.length = 0 - events.push(...filtered) + this.pendingOptimisticUpserts.delete(key) + this.pendingLocalOrigins.delete(key) } - - // Ensure listeners are active before emitting this critical batch - if (this.lifecycle.status !== `ready`) { - this.lifecycle.markReady() + this.pendingOptimisticDirectUpserts.delete(key) + } + for (const key of this.pendingOptimisticDirectDeletes) { + if ( + hasTruncateSync && + truncateOptimisticSnapshot?.deletes.has(key) && + !changedKeys.has(key) + ) + continue + if (!changedKeys.has(key)) { + changedKeys.add(key) } + this.pendingOptimisticDeletes.delete(key) + this.pendingLocalOrigins.delete(key) + this.pendingOptimisticDirectDeletes.delete(key) } // Maintain optimistic state appropriately @@ -1125,9 +1278,11 @@ export class CollectionStateManager< // This includes items from transactions that may have completed during processing if (hasTruncateSync && truncateOptimisticSnapshot) { for (const [key, value] of truncateOptimisticSnapshot.upserts) { + if (completedDirectUpserts.has(key) && changedKeys.has(key)) continue this.optimisticUpserts.set(key, value) } for (const key of truncateOptimisticSnapshot.deletes) { + if (completedDirectDeletes.has(key) && changedKeys.has(key)) continue this.optimisticDeletes.add(key) } } @@ -1137,6 +1292,17 @@ export class CollectionStateManager< for (const transaction of this.transactions.values()) { if (![`completed`, `failed`].includes(transaction.state)) { for (const mutation of transaction.mutations) { + // Truncate clears attribution with the old base, not the still-live + // local requests. Preserve them for later source acknowledgements. + if (this.isThisCollection(mutation.collection)) + this.pendingLocalChanges.add(mutation.key) + if ( + this.isThisCollection(mutation.collection) && + mutation.type === `insert` && + syncedInsertedOrUpdatedKeys.has(mutation.key) + ) { + this.acknowledgedInserts.add(mutation) + } if ( this.isThisCollection(mutation.collection) && mutation.optimistic @@ -1146,7 +1312,7 @@ export class CollectionStateManager< case `update`: this.optimisticUpserts.set( mutation.key, - mutation.modified as TOutput, + this.resolveOptimisticUpsert(mutation), ) this.optimisticDeletes.delete(mutation.key) break @@ -1160,43 +1326,69 @@ export class CollectionStateManager< } } - // A completed optimistic insert may have used a temporary client key while - // the sync confirmation used a different server-generated key. Once a - // sync commit has been applied, stop retaining completed optimistic keys - // that were not confirmed by this commit so the temporary row is removed. - for (const key of this.pendingOptimisticDirectUpserts) { - if (!changedKeys.has(key)) { - changedKeys.add(key) - if (!currentVisibleState.has(key)) { - const previousValue = previousOptimisticUpserts.get(key) - if (previousValue !== undefined) { - currentVisibleState.set(key, previousValue) + // After applying synced operations, if this commit included a truncate, + // re-apply optimistic mutations on top of the fresh synced base. This ensures + // the UI preserves local intent while respecting server rebuild semantics. + // Ordering: deletes (above) -> server ops (just applied) -> optimistic upserts. + if (hasTruncateSync) { + // Events use the same rebuilt overlay as synchronous reads. + const reapplyUpserts = this.optimisticUpserts + const reapplyDeletes = this.optimisticDeletes + + // Emit inserts for re-applied upserts, skipping any keys that have an optimistic delete. + // If the server also inserted/updated the same key in this batch, override that value + // with the optimistic value to preserve local intent. + for (const [key, value] of reapplyUpserts) { + if (reapplyDeletes.has(key)) continue + if (syncedInsertedOrUpdatedKeys.has(key)) { + let foundInsert = false + for (let i = events.length - 1; i >= 0; i--) { + const evt = events[i]! + if (evt.key === key && evt.type === `insert`) { + evt.value = value + foundInsert = true + break + } } + if (!foundInsert) { + events.push({ type: `insert`, key, value }) + } + } else { + events.push({ type: `insert`, key, value }) } - this.pendingOptimisticUpserts.delete(key) - this.pendingLocalOrigins.delete(key) } - } - for (const key of this.pendingOptimisticDirectDeletes) { - if (!changedKeys.has(key)) { - changedKeys.add(key) + + // Finally, ensure we do NOT insert keys that have an outstanding optimistic delete. + if (events.length > 0 && reapplyDeletes.size > 0) { + const filtered: Array> = [] + for (const evt of events) { + if (evt.type === `insert` && reapplyDeletes.has(evt.key)) { + continue + } + filtered.push(evt) + } + events.length = 0 + events.push(...filtered) + } + + // Ensure listeners are active before emitting this critical batch + if (this.lifecycle.status !== `ready`) { + this.lifecycle.markReady() } - this.pendingOptimisticDeletes.delete(key) - this.pendingLocalOrigins.delete(key) } - this.pendingOptimisticDirectUpserts.clear() - this.pendingOptimisticDirectDeletes.clear() // Now check what actually changed in the final visible state for (const key of changedKeys) { const previousVisibleValue = currentVisibleState.get(key) const newVisibleValue = this.get(key) // This returns the new derived state - const previousVirtualProps = this.getVirtualPropsSnapshotForState(key, { - rowOrigins: previousRowOrigins, - optimisticUpserts: previousOptimisticUpserts, - optimisticDeletes: previousOptimisticDeletes, - completedOptimisticKeys: completedOptimisticOps, - }) + const previousVirtualProps = + this.preSyncVirtualState.get(key) ?? + this.getVirtualPropsSnapshotForState(key, { + rowOrigins: previousRowOrigins, + optimisticUpserts: previousOptimisticUpserts, + optimisticDeletes: previousOptimisticDeletes, + completedOptimisticKeys: completedOptimisticOps, + }) const nextVirtualProps = this.getVirtualPropsSnapshotForState(key) const virtualChanged = previousVirtualProps.$synced !== nextVirtualProps.$synced || @@ -1212,36 +1404,12 @@ export class CollectionStateManager< ) : undefined - // Check if this sync operation is redundant with a completed optimistic operation - const completedOp = completedOptimisticOps.get(key) - let isRedundantSync = false - - if (completedOp) { - if ( - completedOp.type === `delete` && - previousVisibleValue !== undefined && - newVisibleValue === undefined && - deepEquals(completedOp.value, previousVisibleValue) - ) { - isRedundantSync = true - } else if ( - newVisibleValue !== undefined && - deepEquals(completedOp.value, newVisibleValue) - ) { - isRedundantSync = true - } - } - const shouldEmitVirtualUpdate = virtualChanged && previousVisibleValue !== undefined && newVisibleValue !== undefined && deepEquals(previousVisibleValue, newVisibleValue) - if (isRedundantSync && !shouldEmitVirtualUpdate) { - continue - } - if ( previousVisibleValue === undefined && newVisibleValue !== undefined @@ -1303,23 +1471,77 @@ export class CollectionStateManager< } // End batching and emit all events (combines any batched events with sync events) - this.changes.emitEvents(events, true) + let failure: { error: unknown } | undefined + try { + const visibleLayoutChanged = + previousLayout !== undefined && + (previousLayout.length !== this.size || + [...this.keys()].some( + (key, index) => key !== previousLayout[index], + )) + this.changes.emitEvents(events, true, visibleLayoutChanged) + } catch (error) { + failure = { error } + } - this.pendingSyncedTransactions = uncommittedSyncedTransactions + if (this.syncSessionGeneration === syncSessionGeneration) { + this.preSyncVisibleState.clear() + this.preSyncVirtualState.clear() + Promise.resolve().then(() => { + if (this.syncSessionGeneration === syncSessionGeneration) { + this.recentlySyncedKeys.clear() + } + }) + if (!this.hasReceivedFirstCommit) this.hasReceivedFirstCommit = true + } - // Clear the pre-sync state since sync operations are complete - this.preSyncVisibleState.clear() + for (const transaction of committedSyncedTransactions) { + transaction.applied.resolve() + } - // Clear recently synced keys after a microtask to allow recomputeOptimisticState to see them - Promise.resolve().then(() => { - this.recentlySyncedKeys.clear() - }) + return { processed: true, failure } + } + + return { processed: false } + } + + /** Abandons one committed transaction before it becomes visible. */ + public cancelPendingSyncedTransaction( + transaction: PendingSyncedTransaction, + ): void { + if (transaction.applicationStarted) return + + const index = this.pendingSyncedTransactions.indexOf(transaction) + if (index === -1) return - // Mark that we've received the first commit (for tracking purposes) - if (!this.hasReceivedFirstCommit) { - this.hasReceivedFirstCommit = true + this.pendingSyncedTransactions.splice(index, 1) + transaction.applied.reject(new SyncTransactionAbortedError()) + + const remainingPendingKeys = new Set() + for (const pending of this.pendingSyncedTransactions) { + for (const operation of pending.operations) { + remainingPendingKeys.add(operation.key as TKey) + } + } + for (const operation of transaction.operations) { + const key = operation.key as TKey + if (!remainingPendingKeys.has(key)) { + this.recentlySyncedKeys.delete(key) + this.preSyncVisibleState.delete(key) + this.preSyncVirtualState.delete(key) } } + + if (this.pendingSyncedTransactions.length === 0) { + this.preSyncVisibleState.clear() + this.preSyncVirtualState.clear() + this.recentlySyncedKeys.clear() + this.changes.emitEvents([], true) + } else { + // Recompute after removing the canceled keys so optimistic cleanup is + // no longer suppressed by a sync transaction that will never publish. + this.recomputeOptimisticState(false) + } } /** @@ -1373,6 +1595,10 @@ export class CollectionStateManager< const currentValue = this.get(key) if (currentValue !== undefined) { this.preSyncVisibleState.set(key, currentValue) + this.preSyncVirtualState.set( + key, + this.getVirtualPropsSnapshotForState(key), + ) } } } @@ -1383,12 +1609,21 @@ export class CollectionStateManager< * This method should be called by the Transaction class when state changes */ public onTransactionStateChange(): void { - // Check if commitPendingTransactions will be called after this - // by checking if there are pending sync transactions (same logic as in transactions.ts) - this.changes.shouldBatchEvents = this.pendingSyncedTransactions.length > 0 + // Batch only when the next sync drain can actually publish. A persisting + // sibling can keep normal sync queued; it must not hide this rollback. + const hasPersistingTransaction = [...this.transactions.values()].some( + (transaction) => transaction.state === `persisting`, + ) + this.changes.shouldBatchEvents = this.pendingSyncedTransactions.some( + (transaction) => + transaction.committed && + (!hasPersistingTransaction || + transaction.immediate || + transaction.truncate), + ) // CRITICAL: Capture visible state BEFORE clearing optimistic state - this.capturePreSyncVisibleState() + if (this.changes.shouldBatchEvents) this.capturePreSyncVisibleState() this.recomputeOptimisticState(false) } @@ -1398,6 +1633,10 @@ export class CollectionStateManager< * This can be called manually or automatically by garbage collection */ public cleanup(): void { + this.syncSessionGeneration++ + for (const transaction of this.pendingSyncedTransactions) { + transaction.applied.reject(new SyncTransactionAbortedError()) + } this.syncedData.clear() this.syncedMetadata.clear() this.syncedCollectionMetadata.clear() @@ -1407,11 +1646,15 @@ export class CollectionStateManager< this.pendingOptimisticDeletes.clear() this.pendingOptimisticDirectUpserts.clear() this.pendingOptimisticDirectDeletes.clear() + this.hydrationSeedKeys.clear() + this.hydratedKeys.clear() this.clearOriginTrackingState() this.isLocalOnly = false this.size = 0 this.pendingSyncedTransactions = [] - this.syncedKeys.clear() + this.preSyncVisibleState.clear() + this.preSyncVirtualState.clear() + this.recentlySyncedKeys.clear() this.hasReceivedFirstCommit = false } } diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index 2d48add4b6..8d4e0489ad 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -1,56 +1,133 @@ import { ensureIndexForExpression } from '../indexes/auto-index.js' -import { and, eq, gte, lt } from '../query/builder/functions.js' +import { and, eq } from '../query/builder/functions.js' import { PropRef, Value } from '../query/ir.js' import { EventEmitter } from '../event-emitter.js' import { compileExpression } from '../query/compiler/evaluators.js' -import { buildCursor } from '../utils/cursor.js' +import { buildCursor, buildCursorCurrent } from '../utils/cursor.js' +import { deepEquals } from '../utils.js' +import { normalizeError } from '../utils/error.js' +import { runAllCallbacks } from '../utils/callbacks.js' +import { createDeferred } from '../deferred.js' +import { LoadSubsetOperationAbortedError } from '../errors.js' import { createFilterFunctionFromExpression, createFilteredCallback, } from './change-events.js' import type { BasicExpression, OrderBy } from '../query/ir.js' -import type { IndexInterface } from '../indexes/base-index.js' +import type { IndexReader } from '../indexes/base-index.js' import type { ChangeMessage, LoadSubsetOptions, + LoadSubsetRequestResult, Subscription, SubscriptionEvents, + SubscriptionLoadSubsetErrorEvent, SubscriptionStatus, SubscriptionUnsubscribedEvent, } from '../types.js' import type { CollectionImpl } from './index.js' +import type { Deferred } from '../deferred.js' type RequestSnapshotOptions = { where?: BasicExpression + signal?: AbortSignal optimizedOnly?: boolean trackLoadSubsetPromise?: boolean /** Optional orderBy to pass to loadSubset for backend optimization */ orderBy?: OrderBy /** Optional limit to pass to loadSubset for backend optimization */ limit?: number - /** Callback that receives the raw loadSubset result for external tracking */ - onLoadSubsetResult?: (result: Promise | true) => void + /** Callback that receives the normalized loadSubset result for internal tracking */ + onLoadSubsetResult?: SubsetResultObserver + /** Called when the local snapshot must fall back from an index to a scan. */ + onUnoptimized?: () => void } type RequestLimitedSnapshotOptions = { orderBy: OrderBy limit: number - /** All column values for cursor (first value used for local index, all values for sync layer) */ + /** A single cursor value; composite cursor inputs are rejected. */ minValues?: Array /** Row offset for offset-based pagination (passed to sync layer) */ offset?: number /** Whether to track the loadSubset promise on this subscription (default: true) */ trackLoadSubsetPromise?: boolean - /** Callback that receives the raw loadSubset result for external tracking */ - onLoadSubsetResult?: (result: Promise | true) => void + /** Callback that receives the normalized loadSubset result for internal tracking */ + onLoadSubsetResult?: SubsetResultObserver } +export type ReleaseLoadSubset = (primaryFailure?: { error: unknown }) => void + +type SubsetResultObserver = ( + result: LoadSubsetRequestResult, + options: LoadSubsetOptions, + release: ReleaseLoadSubset, +) => void + type CollectionSubscriptionOptions = { includeInitialState?: boolean /** Pre-compiled expression for filtering changes */ whereExpression?: BasicExpression /** Callback to call when the subscription is unsubscribed */ onUnsubscribe?: (event: SubscriptionUnsubscribedEvent) => void + /** Callback for subset-load failures scoped to this subscription. */ + onLoadSubsetError?: (event: SubscriptionLoadSubsetErrorEvent) => void + truncateReplayPublication?: TruncateReplayPublicationControl +} + +type TruncateReplayPublicationControl = Readonly<{ + start: () => void + succeed: () => void +}> + +type TruncatePublicationState = { + loadedInitialState: boolean + snapshotSent: boolean + limitedSnapshotRowCount: number + lastSentKey: string | number | undefined +} + +type SubsetAcquisition = { + options: LoadSubsetOptions + loadSubsetSession: number + abortController?: AbortController + removeRequestAbortListener?: () => void + releaseAttempted?: true +} + +type SubsetDemand = { + requestOptions: LoadSubsetOptions + acquisition: SubsetAcquisition + acquisitionState: `starting` | `active` | `detached` + initialResult?: Deferred +} + +type TruncateReplayAttempt = { + pendingCount: number + setupComplete: boolean +} + +type TruncateReplaySession = { + loadSubsetSession: number + publicationState: TruncatePublicationState + /** Direct subscribers buffer the replacement here; delegated publication has no buffer. */ + privateRows: Map | undefined + pending: Set<{ demand: SubsetDemand; attempt: TruncateReplayAttempt }> + pendingSetups: number + currentAttempt: TruncateReplayAttempt + failures: Map + completion: Deferred +} + +function createReplayCompletion(): Deferred { + const completion = createDeferred() + void completion.promise.catch(() => {}) + return completion +} + +function cancelAcquisition(acquisition: SubsetAcquisition): void { + acquisition.abortController?.abort() + acquisition.removeRequestAbortListener?.() } export class CollectionSubscription @@ -72,10 +149,17 @@ export class CollectionSubscription * Track all loadSubset calls made by this subscription so we can unload them on cleanup. * We store the exact LoadSubsetOptions we passed to loadSubset to ensure symmetric unload. */ - private loadedSubsets: Array = [] + private subsetDemands: Array = [] + private primaryFailureDeliveryDepth = 0 + private readonly requestedSubsetWhere = new WeakMap< + LoadSubsetOptions, + BasicExpression + >() // Keep track of the keys we've sent (needed for join and orderBy optimizations) private sentKeys = new Set() + private publishedRows = new Map() + private stalePublishedRows = new Map() // Track the count of rows sent via requestLimitedSnapshot for offset-based pagination private limitedSnapshotRowCount = 0 @@ -83,28 +167,42 @@ export class CollectionSubscription // Track the last key sent via requestLimitedSnapshot for cursor-based pagination private lastSentKey: string | number | undefined - private filteredCallback: (changes: Array>) => void + private filteredCallback: (changes: Array>) => boolean - private orderByIndex: IndexInterface | undefined + private orderByIndex: IndexReader | undefined // Status tracking private _status: SubscriptionStatus = `ready` - private pendingLoadSubsetPromises: Set> = new Set() + private statusRevision = 0 + private _lastError: unknown | undefined + private pendingLoadSubsetParticipants = new Set<{ + demand: SubsetDemand + promise: Promise + }>() // Cleanup function for truncate event listener private truncateCleanup: (() => void) | undefined - - // Truncate buffering state - // When a truncate occurs, we buffer changes until all loadSubset refetches complete - // This prevents a flash of missing content between deletes and new inserts - private isBufferingForTruncate = false - private truncateBuffer: Array>> = [] - private pendingTruncateRefetches: Set> = new Set() + private collectionCleanup: (() => void) | undefined + private collectionRestartCleanup: (() => void) | undefined + + // One replay session owns the publication baseline, overlapping attempts, + // and buffered changes until every attempt settles. + private truncateReplaySession: TruncateReplaySession | undefined + private readonly loadSubsetPromiseErrors = new WeakMap< + Promise, + Error + >() + private truncateReplacementPending = false + private unsubscribed = false public get status(): SubscriptionStatus { return this._status } + public get lastError(): unknown | undefined { + return this._lastError + } + constructor( private collection: CollectionImpl, private callback: (changes: Array>) => void, @@ -112,7 +210,10 @@ export class CollectionSubscription ) { super() if (options.onUnsubscribe) { - this.on(`unsubscribed`, (event) => options.onUnsubscribe!(event)) + this.on(`unsubscribed`, options.onUnsubscribe) + } + if (options.onLoadSubsetError) { + this.on(`loadSubset:error`, options.onLoadSubsetError) } // Auto-index for where expressions if enabled @@ -123,8 +224,9 @@ export class CollectionSubscription const callbackWithSentKeysTracking = ( changes: Array>, ) => { - callback(changes) + this.trackPublishedRows(changes) this.trackSentKeys(changes) + callback(changes) } this.callback = callbackWithSentKeysTracking @@ -132,7 +234,10 @@ export class CollectionSubscription // Create a filtered callback if where clause is provided this.filteredCallback = options.whereExpression ? createFilteredCallback(this.callback, options) - : this.callback + : (changes) => { + this.callback(changes) + return true + } // Listen for truncate events to re-request data after must-refetch // When a truncate happens (e.g., from a 409 must-refetch), all collection data is cleared. @@ -140,6 +245,96 @@ export class CollectionSubscription this.truncateCleanup = this.collection.on(`truncate`, () => { this.handleTruncate() }) + this.collectionCleanup = this.collection.on(`status:cleaned-up`, () => { + this.handleCollectionCleanup() + }) + this.collectionRestartCleanup = this.collection.on( + `status:change`, + ({ status }) => { + if (status !== `loading` && status !== `ready`) return + const loadSubsetSession = this.collection._sync.getLoadSubsetSession() + const replaySession = this.truncateReplaySession + if ( + this.subsetDemands.some( + (demand) => demand.acquisitionState === `detached`, + ) + ) { + this.setStatus(`loadingSubset`) + } + queueMicrotask(() => { + if (this.truncateReplaySession === replaySession) { + this.restartDetachedDemands(loadSubsetSession) + } + }) + }, + ) + } + + /** Detach logical demand from work owned by a discarded sync session. */ + private handleCollectionCleanup(): void { + this.discardTruncateReplay() + this.stalePublishedRows = new Map(this.publishedRows) + this.pendingLoadSubsetParticipants.clear() + + for (const demand of [...this.subsetDemands]) { + demand.initialResult?.reject(new LoadSubsetOperationAbortedError()) + cancelAcquisition(demand.acquisition) + if (demand.acquisitionState === `starting`) { + const index = this.subsetDemands.indexOf(demand) + if (index !== -1) this.subsetDemands.splice(index, 1) + } else { + demand.acquisitionState = `detached` + demand.acquisition = { + options: demand.requestOptions, + loadSubsetSession: demand.acquisition.loadSubsetSession, + } + } + } + this.setReadyIfIdle() + } + + /** Acquire detached demand after startup or initial-error recovery. */ + private restartDetachedDemands(loadSubsetSession: number): void { + if ( + this.unsubscribed || + !this.isLoadSubsetSessionCurrent(loadSubsetSession) + ) { + return + } + if ( + this.collection.status === `error` || + this.collection._sync.syncLoadSubsetFn === null + ) { + this.setReadyIfIdle() + return + } + const demands = this.subsetDemands.filter( + (demand) => + demand.acquisitionState === `detached` && + !demand.requestOptions.signal?.aborted, + ) + if (demands.length === 0) { + this.setReadyIfIdle() + return + } + + const session = this.createTruncateReplaySession(loadSubsetSession, () => { + const currentRows = this.collection.currentStateAsChanges({ + optimizedOnly: false, + }) + return new Map( + // The API returns void for unavailable snapshots, not just undefined. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + (currentRows ?? []) + .filter((change) => change.type !== `delete`) + .map((change) => [change.key, change.value]), + ) + }) + const attempt = session.currentAttempt + this.truncateReplaySession = session + this.setStatus(`loadingSubset`) + if (this.truncateReplaySession !== session) return + this.startTruncateReplayAttempt(session, attempt, demands) } /** @@ -147,199 +342,788 @@ export class CollectionSubscription * This is called when the sync layer receives a must-refetch and clears all data. * * To prevent a flash of missing content, we buffer all changes (deletes from truncate - * and inserts from refetch) until all loadSubset promises resolve, then emit them together. + * and inserts from refetch) until all loadSubset calls succeed, then emit them together. + * A failed replay keeps the last published snapshot private until a later + * authoritative replay succeeds. */ private handleTruncate() { - // Copy the loaded subsets before clearing (we'll re-request them) - const subsetsToReload = [...this.loadedSubsets] - - // Only buffer if there's an actual loadSubset handler that can do async work. - // Without a loadSubset handler, there's nothing to re-request and no reason to buffer. - // This prevents unnecessary buffering in eager sync mode or when loadSubset isn't implemented. + // Without a loader, replay only reconciles rows retained across cleanup. const hasLoadSubsetHandler = this.collection._sync.syncLoadSubsetFn !== null + const demandsToReload = hasLoadSubsetHandler ? [...this.subsetDemands] : [] - // If there are no subsets to reload OR no loadSubset handler, just reset state - if (subsetsToReload.length === 0 || !hasLoadSubsetHandler) { - this.snapshotSent = false - this.loadedInitialState = false - this.limitedSnapshotRowCount = 0 - this.lastSentKey = undefined - this.loadedSubsets = [] + // Retained rows still need the committed replacement even without demand. + if (demandsToReload.length === 0 && this.stalePublishedRows.size === 0) { + this.resetSnapshotTracking() return } - // Start buffering BEFORE we receive the delete events from the truncate commit - // This ensures we capture both the deletes and subsequent inserts - this.isBufferingForTruncate = true - this.truncateBuffer = [] - this.pendingTruncateRefetches.clear() + let session = this.truncateReplaySession + if (session) { + if (!session.completion.isPending()) { + session.completion = createReplayCompletion() + } + // Setup itself holds publication: adapter/status callbacks may reenter + // before a request returns its promise and joins the pending set. + session.pendingSetups++ + session.failures.clear() + session.currentAttempt = { pendingCount: 0, setupComplete: false } + } else { + // Every overlapping attempt shares one publication baseline and buffer. + session = this.createTruncateReplaySession( + this.collection._sync.getLoadSubsetSession(), + () => new Map(this.publishedRows), + ) + this.truncateReplaySession = session + } + const attempt = session.currentAttempt + this.setStatus(`loadingSubset`) - // Reset snapshot/pagination tracking state - // Note: We don't need to populate sentKeys here because filterAndFlipChanges - // will skip the delete filter when isBufferingForTruncate is true - this.snapshotSent = false - this.loadedInitialState = false - this.limitedSnapshotRowCount = 0 - this.lastSentKey = undefined + if (this.truncateReplaySession !== session) return + + if (this.options.truncateReplayPublication) { + this.truncateReplacementPending = true + this.options.truncateReplayPublication.start() + } + + // A newer replay replaces every prior acquisition for these demands. Abort + // the old work before it can install rows into the new generation. + for (const demand of demandsToReload) { + demand.acquisition.abortController?.abort() + } - // Clear the loadedSubsets array since we're re-requesting fresh - this.loadedSubsets = [] + // Reset snapshot/pagination tracking for the replacement snapshot. Rows + // retained from an earlier failed replay stay marked until this attempt + // either replaces them or proves they are absent. + this.resetSnapshotTracking() - // Defer the loadSubset calls to a microtask so the truncate commit's delete events - // are buffered BEFORE the loadSubset calls potentially trigger nested commits. - // This ensures correct event ordering: deletes first, then inserts. + // Defer the requests so the truncate commit's deletes enter the session + // buffer before a synchronous adapter can publish replacement rows. queueMicrotask(() => { - // Check if we were unsubscribed while waiting - if (!this.isBufferingForTruncate) { + if (this.truncateReplaySession !== session) return + if (!this.isLoadSubsetSessionCurrent(session.loadSubsetSession)) { + this.retireStaleTruncateReplay(session) + return + } + // A newer truncate that arrived before this attempt began source work + // already captured the active demands. Starting them now would place the + // obsolete acquisition outside the newer abort sweep. + this.startTruncateReplayAttempt( + session, + attempt, + session.currentAttempt === attempt ? demandsToReload : [], + ) + }) + } + + /** Make tentative replay ownership visible before adapter code can reenter. */ + private startTruncateReplayDemand( + session: TruncateReplaySession, + attempt: TruncateReplayAttempt, + demand: SubsetDemand, + ): void { + const isCurrentAttempt = () => + this.truncateReplaySession === session && + session.currentAttempt === attempt + const isCurrent = () => + isCurrentAttempt() && + this.isLoadSubsetSessionCurrent(session.loadSubsetSession) && + this.isDemandActive(demand) + const fail = (error: unknown) => { + if (isCurrent()) session.failures.set(demand, normalizeError(error)) + } + if (demand.initialResult) { + void session.completion.promise.then( + demand.initialResult.resolve, + demand.initialResult.reject, + ) + } + + // Sequential handoff: retire the old physical lease while retaining its + // logical demand. Callback reentry cannot release that lease twice. + const previous = demand.acquisition + const hadPreviousAcquisition = demand.acquisitionState === `active` + demand.acquisitionState = `detached` + if (hadPreviousAcquisition) { + try { + this.releaseAcquisition(previous) + } catch (error) { + fail(error) return } + } + if (!isCurrent() || demand.requestOptions.signal?.aborted) return + + const next = this.createSubsetAcquisition(demand) + demand.acquisition = next + demand.acquisitionState = `starting` + let result: LoadSubsetRequestResult + try { + result = this.loadSubset(next.options, isCurrent) + } catch (error) { + if (demand.acquisition === next) demand.acquisitionState = `detached` + cancelAcquisition(next) + fail(error) + return + } + + if (!isCurrent()) { + if (demand.acquisition === next) demand.acquisitionState = `detached` + try { + this.releaseAcquisition(next) + } catch (error) { + fail(error) + } + return + } + + demand.acquisitionState = `active` + this.trackTruncateReplayParticipant(session, attempt, demand, result) + this.observeLoadSubsetResult( + result, + demand, + next.options, + true, + () => isCurrent() && !next.options.signal?.aborted, + ) + } + + private settleTruncateReplay( + session: TruncateReplaySession, + pending: { demand: SubsetDemand; attempt: TruncateReplayAttempt }, + ): void { + try { + if (this.truncateReplaySession !== session) return + if (!this.isLoadSubsetSessionCurrent(session.loadSubsetSession)) { + this.retireStaleTruncateReplay(session) + return + } + if (session.pending.delete(pending)) pending.attempt.pendingCount-- + this.checkTruncateReplayComplete(session) + } catch (error) { + // Replay settlement runs from a Promise callback, so throwing here would + // create an unobserved derived rejection. Surface subscriber errors like + // other async collection events instead. + queueMicrotask(() => { + throw error + }) + } + } + + /** Keep every acquisition begun during recovery inside its publication barrier. */ + private trackTruncateReplayParticipant( + session: TruncateReplaySession, + attempt: TruncateReplayAttempt, + demand: SubsetDemand, + result: LoadSubsetRequestResult, + ): void { + if ( + this.truncateReplaySession !== session || + (session.currentAttempt !== attempt && + attempt.setupComplete && + attempt.pendingCount === 0) || + !(result instanceof Promise) + ) { + return + } - // Re-request all previously loaded subsets and track their promises - for (const options of subsetsToReload) { - const syncResult = this.collection._sync.loadSubset(options) - - // Track this loadSubset call so we can unload it later - this.loadedSubsets.push(options) - this.trackLoadSubsetPromise(syncResult) - - // Track the promise for buffer flushing - if (syncResult instanceof Promise) { - this.pendingTruncateRefetches.add(syncResult) - syncResult - .catch(() => { - // Ignore errors - we still want to flush the buffer even if some requests fail - }) - .finally(() => { - this.pendingTruncateRefetches.delete(syncResult) - this.checkTruncateRefetchComplete() - }) + // An older attempt can still accept returning startup work while setup or + // another participant retains it. Once drained, it cannot reopen. Shared + // promises still get one participant per logical acquisition. + const pending = { demand, attempt } + attempt.pendingCount++ + session.pending.add(pending) + void result.then( + () => this.settleTruncateReplay(session, pending), + (error: unknown) => { + // A released demand no longer participates in this replacement. Its + // cooperative AbortError must not discard rows from active demands. + if ( + this.truncateReplaySession === session && + session.currentAttempt === attempt && + this.isLoadSubsetSessionCurrent(session.loadSubsetSession) && + this.subsetDemands.includes(demand) + ) { + const normalized = this.normalizeLoadSubsetPromiseError(result, error) + session.failures.set(demand, normalized) } + this.settleTruncateReplay(session, pending) + }, + ) + } + + /** Stop obsolete logical demand from pinning a replay barrier. */ + private removeTruncateReplayParticipant(demand: SubsetDemand): void { + const session = this.truncateReplaySession + if (!session) return + session.failures.delete(demand) + for (const pending of session.pending) { + if (pending.demand === demand) { + session.pending.delete(pending) + pending.attempt.pendingCount-- } + } + } - // If all loadSubset calls were synchronous (returned true), flush now - // At this point, delete events have already been buffered from the truncate commit - if (this.pendingTruncateRefetches.size === 0) { - this.flushTruncateBuffer() + /** Publish only after every overlapping replay attempt has settled. */ + private checkTruncateReplayComplete(session: TruncateReplaySession): void { + if (this.truncateReplaySession !== session) return + if (session.pendingSetups > 0 || session.pending.size > 0) return + + const activeFailure = [...session.failures].find(([demand]) => + this.subsetDemands.includes(demand), + ) + try { + if (activeFailure) { + this.abandonTruncateReplay(session, activeFailure[1]) + } else { + this.flushTruncateReplay(session) } - }) + } finally { + this.setReadyIfIdle() + } } /** - * Check if all truncate refetch promises have completed and flush buffer if so + * Keep an incomplete replay private. The source no longer proves a complete + * state, so only a later successful truncate replay may reopen publication. */ - private checkTruncateRefetchComplete() { + private abandonTruncateReplay( + session: TruncateReplaySession, + failure: Error, + ): void { + if (this.truncateReplaySession !== session) return + session.completion.reject(failure) + // Delegated publication already delivered its rows. Only a private buffer + // returns the caller's pagination position to the public snapshot; the + // private rows and their sent-key tracking stay together for a retry. + if (!session.privateRows) return + const publicationState = session.publicationState + this.loadedInitialState = publicationState.loadedInitialState + this.snapshotSent = publicationState.snapshotSent + this.limitedSnapshotRowCount = publicationState.limitedSnapshotRowCount + this.lastSentKey = publicationState.lastSentKey + } + + /** Publish the buffered replacement as one batch, or release the delegate. */ + private flushTruncateReplay(session: TruncateReplaySession): void { + if (this.truncateReplaySession !== session) return + this.truncateReplaySession = undefined + this.truncateReplacementPending = false + + // Retained rows the source never re-delivered leave the replacement. + const { privateRows } = session + for (const key of this.stalePublishedRows.keys()) privateRows?.delete(key) + this.stalePublishedRows.clear() + try { + if (privateRows) { + // Diff the retained public snapshot against the applied source replacement. + const replacement = this.createStateDiff( + this.publishedRows, + privateRows, + ) + if (replacement.length > 0) this.filteredCallback(replacement) + } + } finally { + // Restore tracking even when a subscriber rejects the replacement. + this.restorePublishedSnapshotTracking() + session.completion.resolve() + this.options.truncateReplayPublication?.succeed() + } + } + + private restorePublishedSnapshotTracking(): void { + this.sentKeys = new Set(this.publishedRows.keys()) + if (!this.orderByIndex) return + + this.limitedSnapshotRowCount = this.sentKeys.size + const orderedSentKeys = this.orderByIndex.takeFromStart( + this.sentKeys.size, + (key) => this.sentKeys.has(key), + ) + this.lastSentKey = orderedSentKeys.at(-1) + } + + /** Fold changes into the private replacement; false when they publish now. */ + private bufferPrivately( + changes: ReadonlyArray>, + ): boolean { + const privateRows = this.truncateReplaySession?.privateRows + if (!privateRows) return false + for (const change of changes) { + if (change.type === `delete`) privateRows.delete(change.key) + else privateRows.set(change.key, change.value) + } + return true + } + + private createStateDiff( + baseline: ReadonlyMap, + finalRows: ReadonlyMap, + ): Array> { + const replacement: Array> = [] + for (const [key, previousValue] of baseline) { + const value = finalRows.get(key) + if (value === undefined) { + replacement.push({ + type: `delete`, + key, + value: previousValue, + }) + } else if (!deepEquals(value, previousValue)) { + replacement.push({ + type: `update`, + key, + value, + previousValue, + }) + } + } + for (const [key, value] of finalRows) { + if (!baseline.has(key)) replacement.push({ type: `insert`, key, value }) + } + return replacement + } + + private get isBufferingForTruncate(): boolean { + return this.truncateReplaySession !== undefined + } + + private setReadyIfIdle(): void { + const session = this.truncateReplaySession + const hasPendingReplayWork = + session && (session.pendingSetups > 0 || session.pending.size > 0) if ( - this.pendingTruncateRefetches.size === 0 && - this.isBufferingForTruncate + this.pendingLoadSubsetParticipants.size === 0 && + !hasPendingReplayWork ) { - this.flushTruncateBuffer() + this.setStatus(`ready`) } } - /** - * Flush the truncate buffer, emitting all buffered changes to the callback - */ - private flushTruncateBuffer() { - this.isBufferingForTruncate = false + private isLoadSubsetSessionCurrent(session: number): boolean { + return session === this.collection._sync.getLoadSubsetSession() + } + + private retireStaleTruncateReplay(session: TruncateReplaySession): void { + if (this.truncateReplaySession !== session) return + this.discardTruncateReplay() + this.stalePublishedRows.clear() + } - // Flatten all buffered changes into a single array for atomic emission - // This ensures consumers see all truncate changes (deletes + inserts) in one callback - const merged = this.truncateBuffer.flat() - if (merged.length > 0) { - this.filteredCallback(merged) + /** Drop the replay without publishing; an unfinished wait rejects as aborted. */ + private discardTruncateReplay(): void { + const session = this.truncateReplaySession + if (session?.completion.isPending()) { + session.completion.reject(new LoadSubsetOperationAbortedError()) } + this.truncateReplaySession = undefined + this.truncateReplacementPending = false + } - this.truncateBuffer = [] + private resetSnapshotTracking(): void { + this.snapshotSent = false + this.loadedInitialState = false + this.limitedSnapshotRowCount = 0 + this.lastSentKey = undefined } - setOrderByIndex(index: IndexInterface) { - this.orderByIndex = index + /** One replay session; only direct subscribers buffer a private replacement. */ + private createTruncateReplaySession( + loadSubsetSession: number, + privateRows: () => Map, + ): TruncateReplaySession { + return { + loadSubsetSession, + publicationState: { + loadedInitialState: this.loadedInitialState, + snapshotSent: this.snapshotSent, + limitedSnapshotRowCount: this.limitedSnapshotRowCount, + lastSentKey: this.lastSentKey, + }, + privateRows: this.options.truncateReplayPublication + ? undefined + : privateRows(), + pending: new Set(), + // Setup itself holds publication: adapter/status callbacks may reenter + // before a request returns its promise and joins the pending set. + pendingSetups: 1, + currentAttempt: { pendingCount: 0, setupComplete: false }, + failures: new Map(), + completion: createReplayCompletion(), + } } - /** - * Check if an orderBy index has been set for this subscription - */ - hasOrderByIndex(): boolean { - return this.orderByIndex !== undefined + /** Start one attempt's demands, then release the setup hold on publication. */ + private startTruncateReplayAttempt( + session: TruncateReplaySession, + attempt: TruncateReplayAttempt, + demands: ReadonlyArray, + ): void { + for (const demand of demands) { + if (!this.subsetDemands.includes(demand)) continue + this.startTruncateReplayDemand(session, attempt, demand) + if ( + this.truncateReplaySession !== session || + session.currentAttempt !== attempt + ) { + break + } + } + attempt.setupComplete = true + session.pendingSetups-- + this.checkTruncateReplayComplete(session) + } + + public get hasPendingTruncateReplacement(): boolean { + return this.truncateReplacementPending + } + + public get pendingTruncateReplacement(): Promise | undefined { + const completion = this.truncateReplaySession?.completion + return completion?.isPending() ? completion.promise : undefined + } + + public get hasFailedTruncateReplacement(): boolean { + const completion = this.truncateReplaySession?.completion + return ( + this.truncateReplacementPending && + completion !== undefined && + !completion.isPending() + ) + } + + setOrderByIndex(index: IndexReader) { + this.orderByIndex = index } /** * Set subscription status and emit events if changed */ private setStatus(newStatus: SubscriptionStatus) { + if (this.unsubscribed) return if (this._status === newStatus) { return // No change } const previousStatus = this._status this._status = newStatus + const revision = ++this.statusRevision // Emit status:change event - this.emitInner(`status:change`, { - type: `status:change`, - subscription: this, - previousStatus, - status: newStatus, - }) + this.emitInnerWhile( + `status:change`, + { + type: `status:change`, + subscription: this, + previousStatus, + status: newStatus, + }, + () => this.statusRevision === revision, + ) + + // A listener may synchronously start or release demand. Do not follow that + // newer transition with a stale specific event. + if (this.statusRevision !== revision) return // Emit specific status event const eventKey: `status:${SubscriptionStatus}` = `status:${newStatus}` - this.emitInner(eventKey, { - type: eventKey, - subscription: this, - previousStatus, - status: newStatus, - } as SubscriptionEvents[typeof eventKey]) + this.emitInnerWhile( + eventKey, + { + type: eventKey, + subscription: this, + previousStatus, + status: newStatus, + } as SubscriptionEvents[typeof eventKey], + () => this.statusRevision === revision, + ) } - /** - * Track a loadSubset promise and manage loading status - */ - private trackLoadSubsetPromise(syncResult: Promise | true) { - // Track the promise if it's actually a promise (async work) - if (syncResult instanceof Promise) { - this.pendingLoadSubsetPromises.add(syncResult) + /** Observe an asynchronous subset load and restore status on settlement. */ + private observeLoadSubsetResult( + syncResult: LoadSubsetRequestResult, + demand: SubsetDemand, + options: LoadSubsetOptions, + trackStatus: boolean, + shouldReportError: () => boolean = () => true, + ): void { + if (!(syncResult instanceof Promise)) return + + const loadSubsetSession = this.collection._sync.getLoadSubsetSession() + const participant = { demand, promise: syncResult } + + if (trackStatus) { + this.pendingLoadSubsetParticipants.add(participant) this.setStatus(`loadingSubset`) + } - syncResult.finally(() => { - this.pendingLoadSubsetPromises.delete(syncResult) - if (this.pendingLoadSubsetPromises.size === 0) { - this.setStatus(`ready`) + const finish = () => { + if (trackStatus) { + this.pendingLoadSubsetParticipants.delete(participant) + if (this.isLoadSubsetSessionCurrent(loadSubsetSession)) { + this.setReadyIfIdle() } - }) + } } + + void syncResult.then(finish, (error: unknown) => { + if ( + this.isLoadSubsetSessionCurrent(loadSubsetSession) && + shouldReportError() + ) { + this.recordLoadSubsetError( + options, + this.normalizeLoadSubsetPromiseError(syncResult, error), + ) + } + finish() + }) } - hasLoadedInitialState() { - return this.loadedInitialState + /** Give every logical observer of one transport rejection the same Error. */ + private normalizeLoadSubsetPromiseError( + promise: Promise, + error: unknown, + ): Error { + const existing = this.loadSubsetPromiseErrors.get(promise) + if (existing) return existing + const normalized = normalizeError(error) + this.loadSubsetPromiseErrors.set(promise, normalized) + return normalized } - hasSentAtLeastOneSnapshot() { - return this.snapshotSent + private stopDemandStatusParticipants(demand: SubsetDemand): void { + for (const participant of this.pendingLoadSubsetParticipants) { + if (participant.demand === demand) { + this.pendingLoadSubsetParticipants.delete(participant) + } + } + this.setReadyIfIdle() } - emitEvents(changes: Array>) { - const newChanges = this.filterAndFlipChanges(changes) + private loadSubset( + options: LoadSubsetOptions, + shouldReportError: () => boolean = () => true, + ): LoadSubsetRequestResult { + try { + return this.collection._sync.loadSubset(options) + } catch (error) { + const normalized = normalizeError(error) + if (shouldReportError()) this.recordLoadSubsetError(options, normalized) + throw normalized + } + } + + /** Create a fresh, abortable adapter acquisition for a replay generation. */ + private createSubsetAcquisition( + demand: SubsetDemand, + ): SubsetAcquisition & { abortController: AbortController } { + const abortController = new AbortController() + const requestSignal = demand.requestOptions.signal + let removeRequestAbortListener: (() => void) | undefined + + if (requestSignal?.aborted) { + abortController.abort(requestSignal.reason) + } else if (requestSignal) { + const abort = () => abortController.abort(requestSignal.reason) + requestSignal.addEventListener(`abort`, abort, { once: true }) + removeRequestAbortListener = () => + requestSignal.removeEventListener(`abort`, abort) + } + + return { + options: { + ...demand.requestOptions, + signal: abortController.signal, + }, + loadSubsetSession: this.collection._sync.getLoadSubsetSession(), + abortController, + removeRequestAbortListener, + } + } - if (this.isBufferingForTruncate) { - // Buffer the changes instead of emitting immediately - // This prevents a flash of missing content during truncate/refetch - if (newChanges.length > 0) { - this.truncateBuffer.push(newChanges) + /** Retire an acquisition before user code; failed cleanup is not retryable. */ + private releaseAcquisition( + acquisition: SubsetAcquisition, + reportReleaseError = this.primaryFailureDeliveryDepth === 0, + ): void { + if (acquisition.releaseAttempted) return + acquisition.releaseAttempted = true + try { + acquisition.abortController?.abort() + if (this.isLoadSubsetSessionCurrent(acquisition.loadSubsetSession)) { + this.collection._sync.unloadSubset(acquisition.options) } - } else { - this.filteredCallback(newChanges) + } catch (error) { + const normalized = reportReleaseError + ? this.recordLoadSubsetError( + acquisition.options, + normalizeError(error), + true, + ) + : normalizeError(error) + throw normalized + } finally { + acquisition.removeRequestAbortListener?.() } } + /** Start and retain the first acquisition for one logical subset demand. */ + private startSubsetDemand(requestOptions: LoadSubsetOptions): { + demand: SubsetDemand + result: LoadSubsetRequestResult + started: boolean + } { + const demand: SubsetDemand = { + requestOptions, + acquisition: { + options: requestOptions, + loadSubsetSession: this.collection._sync.getLoadSubsetSession(), + }, + acquisitionState: `starting`, + } + if ( + this.collection.status === `cleaned-up` || + // Ready/error callbacks can run before sync returns its loader. Idle + // deferred starts still acquire through the sync manager's queue. + (this.collection.config.syncMode === `on-demand` && + (this.collection.status === `error` || + (this.collection.status !== `idle` && + this.collection._sync.syncLoadSubsetFn === null))) + ) { + demand.acquisitionState = `detached` + this.subsetDemands.push(demand) + const initialResult = createDeferred() + demand.initialResult = initialResult + const abort = () => + initialResult.reject(new LoadSubsetOperationAbortedError()) + requestOptions.signal?.addEventListener(`abort`, abort, { once: true }) + const finish = () => { + requestOptions.signal?.removeEventListener(`abort`, abort) + demand.initialResult = undefined + } + void initialResult.promise.then(finish, finish) + return { demand, result: initialResult.promise, started: false } + } + const acquisition = this.createSubsetAcquisition(demand) + demand.acquisition = acquisition + const replaySession = this.truncateReplaySession + const replayAttempt = replaySession?.currentAttempt + const loadSubsetSession = this.collection._sync.getLoadSubsetSession() + // Reentrant release must see the exact acquisition before adapter work + // starts. A genuine load throw removes this tentative logical owner below. + this.subsetDemands.push(demand) + let result: LoadSubsetRequestResult + try { + result = this.loadSubset( + acquisition.options, + () => + this.isLoadSubsetSessionCurrent(loadSubsetSession) && + this.subsetDemands.includes(demand) && + (replaySession === undefined || + (this.truncateReplaySession === replaySession && + replaySession.currentAttempt === replayAttempt)), + ) + } catch (error) { + const demandIndex = this.subsetDemands.indexOf(demand) + if (demandIndex !== -1) { + if ( + replaySession && + replayAttempt && + this.truncateReplaySession === replaySession && + replaySession.currentAttempt === replayAttempt + ) { + replaySession.failures.set(demand, normalizeError(error)) + } + this.subsetDemands.splice(demandIndex, 1) + } + cancelAcquisition(acquisition) + throw error + } + + if (!this.isLoadSubsetSessionCurrent(loadSubsetSession)) { + const demandIndex = this.subsetDemands.indexOf(demand) + if (demandIndex !== -1) this.subsetDemands.splice(demandIndex, 1) + cancelAcquisition(acquisition) + return { demand, result, started: true } + } + + demand.acquisitionState = `active` + if (!this.subsetDemands.includes(demand)) { + this.releaseAcquisition(acquisition) + return { demand, result, started: true } + } + + if (replaySession && replayAttempt) { + this.trackTruncateReplayParticipant( + replaySession, + replayAttempt, + demand, + result, + ) + } + return { demand, result, started: true } + } + + /** Re-check ownership after adapter and event callbacks that may reenter. */ + private isDemandActive(demand: SubsetDemand): boolean { + return !this.unsubscribed && this.subsetDemands.includes(demand) + } + + private recordLoadSubsetError( + options: LoadSubsetOptions, + error: unknown, + reportAborted = false, + ): Error { + const normalized = normalizeError(error) + // Aborted subset requests are obsolete demand, not load failures. The + // request may reject after its route has already been released. + if (options.signal?.aborted && !reportAborted) return normalized + + this._lastError = normalized + this.primaryFailureDeliveryDepth++ + try { + this.emitInner(`loadSubset:error`, { + type: `loadSubset:error`, + subscription: this, + options, + error: normalized, + }) + } finally { + this.primaryFailureDeliveryDepth-- + } + return normalized + } + + emitEvents(changes: Array>): boolean { + if (this.unsubscribed) return false + const newChanges = this.filterAndFlipChanges(changes) + + // Reconciliation can reduce a source delta to no visible change. Do not + // wake subscribers for an empty semantic batch. + if (changes.length > 0 && newChanges.length === 0) return false + + // A direct subscriber sees the replacement as one batch, not a flash of + // missing content. Delegated publication keeps its private D2 contributions. + if (this.bufferPrivately(newChanges)) return false + return this.filteredCallback(newChanges) + } + + /** Keep direct snapshot reads private while an authoritative replay is open. */ + private publishSnapshot(changes: Array>): void { + if (!this.bufferPrivately(changes)) this.callback(changes) + } + /** * Sends the snapshot to the callback. * Returns a boolean indicating if it succeeded. * It can only fail if there is no index to fulfill the request * and the optimizedOnly option is set to true, - * or, the entire state was already loaded. + * or, the entire state was already loaded or the request was cancelled. */ requestSnapshot(opts?: RequestSnapshotOptions): boolean { + // Cancel before acquiring ownership or publishing a local snapshot. + if (this.unsubscribed || opts?.signal?.aborted) return false if (this.loadedInitialState) { // Subscription was deoptimized so we already sent the entire initial state return false @@ -371,35 +1155,77 @@ export class CollectionSubscription // don't await it, we will load the data into the collection when it comes in const loadOptions: LoadSubsetOptions = { where: stateOpts.where, + signal: opts?.signal, subscription: this, // Include orderBy and limit if provided so sync layer can optimize the query orderBy: opts?.orderBy, limit: opts?.limit, } - const syncResult = this.collection._sync.loadSubset(loadOptions) - // Pass the raw loadSubset result to the caller for external tracking - opts?.onLoadSubsetResult?.(syncResult) - - // Track this loadSubset call so we can unload it later - this.loadedSubsets.push(loadOptions) - - const trackLoadSubsetPromise = opts?.trackLoadSubsetPromise ?? true - if (trackLoadSubsetPromise) { - this.trackLoadSubsetPromise(syncResult) + const { + demand, + result: syncResult, + started, + } = this.startSubsetDemand(loadOptions) + if (!this.isDemandActive(demand)) return false + if (opts?.where) this.requestedSubsetWhere.set(loadOptions, opts.where) + + // Report the result synchronously, including a wait for an unavailable loader. + opts?.onLoadSubsetResult?.( + syncResult, + demand.acquisition.options, + (primaryFailure) => this.releaseDemand(demand, primaryFailure), + ) + if (!this.isDemandActive(demand)) return false + + if (started) { + this.observeLoadSubsetResult( + syncResult, + demand, + demand.acquisition.options, + opts?.trackLoadSubsetPromise ?? true, + ) } + if (!this.isDemandActive(demand)) return false // Also load data immediately from the collection - const snapshot = this.collection.currentStateAsChanges(stateOpts) + let snapshot: Array> | void + if (opts?.onUnoptimized) { + snapshot = this.collection.currentStateAsChanges({ + ...stateOpts, + optimizedOnly: true, + }) + if (snapshot === undefined) { + opts.onUnoptimized() + // The callback can unsubscribe; TypeScript retains the pre-call narrowing. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (this.unsubscribed) return false + snapshot = this.collection.currentStateAsChanges({ + ...stateOpts, + optimizedOnly: false, + }) + } + } else { + snapshot = this.collection.currentStateAsChanges(stateOpts) + } + // Snapshot evaluation may call user code that tears down the subscription. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (this.unsubscribed) return false if (snapshot === undefined) { // Couldn't load from indexes return false } - // Only send changes that have not been sent yet + // Skip known rows, except retained rows from an abandoned replay: a new + // snapshot must reconcile those with the source, not suppress their update. + const knownRows = + this.truncateReplaySession?.privateRows ?? this.publishedRows const filteredSnapshot = snapshot.filter( - (change) => !this.sentKeys.has(change.key), + (change) => + (!this.isBufferingForTruncate && + this.stalePublishedRows.has(change.key)) || + (!this.sentKeys.has(change.key) && !knownRows.has(change.key)), ) // Add keys to sentKeys BEFORE calling callback to prevent race condition. @@ -410,18 +1236,116 @@ export class CollectionSubscription } this.snapshotSent = true - this.callback(filteredSnapshot) + this.publishSnapshot( + this.isBufferingForTruncate + ? filteredSnapshot + : this.reconcileStalePublishedChanges(filteredSnapshot), + ) return true } + /** Release one exact subset request while keeping the subscription alive. */ + releaseSnapshot(where: BasicExpression): void { + const index = this.subsetDemands.findIndex( + (demand) => + demand.requestOptions.where === where || + this.requestedSubsetWhere.get(demand.requestOptions) === where, + ) + if (index === -1) return + + this.releaseDemandAt(index) + } + + private releaseDemand( + demand: SubsetDemand, + primaryFailure?: { error: unknown }, + ): void { + if (!primaryFailure) { + const index = this.subsetDemands.indexOf(demand) + if (index !== -1) this.releaseDemandAt(index) + return + } + + try { + this.recordLoadSubsetError( + demand.acquisition.options, + primaryFailure.error, + true, + ) + } finally { + // The failed request remains the public error, even if cleanup also fails. + const index = this.subsetDemands.indexOf(demand) + if (index !== -1) this.releaseDemandAt(index, false) + } + } + + private releaseDemandAt( + index: number, + reportReleaseError = this.primaryFailureDeliveryDepth === 0, + ): void { + const demand = this.subsetDemands[index] + if (!demand) return + const replaySession = this.truncateReplaySession + const acquisition = demand.acquisition + this.subsetDemands.splice(index, 1) + demand.initialResult?.reject(new LoadSubsetOperationAbortedError()) + const releaseCallbacks = [ + () => this.removeTruncateReplayParticipant(demand), + ...(demand.acquisitionState === `active` + ? [ + // Adapter release is a supported reentrancy boundary. A demand + // started from unload joins this replacement before completion. + () => this.releaseAcquisition(acquisition, reportReleaseError), + ] + : []), + () => this.retireEmptyReplay(), + () => { + if (replaySession) this.checkTruncateReplayComplete(replaySession) + }, + // Ready follows replacement publication, never the delete half of it. + () => this.stopDemandStatusParticipants(demand), + ] + runAllCallbacks(releaseCallbacks) + } + + /** A replay with no remaining logical demand cannot establish more rows. */ + private retireEmptyReplay(): void { + if (this.subsetDemands.length !== 0 || !this.truncateReplaySession) { + return + } + this.discardTruncateReplay() + this.stalePublishedRows = new Map(this.publishedRows) + this.restorePublishedSnapshotTracking() + this.options.truncateReplayPublication?.succeed() + } + + /** Read the applied rows in an ordered acquisition without starting demand. */ + readOrderedSnapshot( + options: LoadSubsetOptions, + ): Array, string | number>> { + const predicates = [ + this.options.whereExpression, + options.where, + options.cursor?.whereFrom, + ].filter((where) => where !== undefined) + const snapshot = this.collection.currentStateAsChanges({ + orderBy: options.orderBy, + limit: options.limit, + where: + predicates.length > 0 + ? predicates.reduce((left, right) => and(left, right)) + : undefined, + }) + return Array.isArray(snapshot) ? snapshot : [] + } + /** * Sends a snapshot that fulfills the `where` clause and all rows are bigger or equal to the cursor. * Requires a range index to be set with `setOrderByIndex` prior to calling this method. * It uses that range index to load the items in the order of the index. * - * For multi-column orderBy: - * - Uses first value from `minValues` for LOCAL index operations (wide bounds, ensures no missed rows) - * - Uses all `minValues` to build a precise composite cursor for SYNC layer loadSubset + * Cursor requests support one order term and one minValue. Multi-column + * queries use the ordered loader's prefix-and-tie fallback instead. * * Note 1: it may load more rows than the provided LIMIT because it loads all values equal to the first cursor value + limit values greater. * This is needed to ensure that it does not accidentally skip duplicate values when the limit falls in the middle of some duplicated values. @@ -435,6 +1359,7 @@ export class CollectionSubscription trackLoadSubsetPromise: shouldTrackLoadSubsetPromise = true, onLoadSubsetResult, }: RequestLimitedSnapshotOptions) { + if (this.unsubscribed) return if (!limit) throw new Error(`limit is required`) if (!this.orderByIndex) { @@ -443,6 +1368,11 @@ export class CollectionSubscription ) } + // Validate cursor input before local delivery changes sent keys or calls user code. + const whereFromCursor = minValues + ? buildCursor(orderBy, minValues) + : undefined + // Check if minValues has a first element (regardless of its value) // This distinguishes between "no min value provided" vs "min value is undefined" const hasMinValue = minValues !== undefined && minValues.length > 0 @@ -479,9 +1409,6 @@ export class CollectionSubscription // so if minValue is 3 then the previous snapshot may not have included all 3s // e.g. if it was offset 0 and limit 3 it would only have loaded the first 3 // so we load all rows equal to minValue first, to be sure we don't skip any duplicate values - // - // For multi-column orderBy, we use the first column value for index operations (wide bounds) - // This may load some duplicates but ensures we never miss any rows. let keys: Array = [] if (hasMinValue) { // First, get all items with the same FIRST COLUMN value as minValue @@ -525,8 +1452,6 @@ export class CollectionSubscription : null while (valuesNeeded() > 0 && !collectionExhausted()) { - const insertedKeys = new Set() // Track keys we add to `changes` in this iteration - for (const key of keys) { const value = this.collection.get(key)! changes.push({ @@ -537,7 +1462,6 @@ export class CollectionSubscription // Extract the indexed value (e.g., salary) from the row, not the full row // This is needed for index.take() to work correctly with the BTree comparator biggestObservedValue = valueExtractor ? valueExtractor(value) : value - insertedKeys.add(key) // Track this key } keys = index.take(valuesNeeded(), biggestObservedValue!, filterFn) @@ -554,10 +1478,16 @@ export class CollectionSubscription this.sentKeys.add(change.key) } - this.callback(changes) + this.publishSnapshot(changes) + // A subscriber callback can synchronously tear down this subscription. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (this.unsubscribed) return // Update the row count and last key after sending (for next call's offset/cursor) - this.limitedSnapshotRowCount += changes.length + this.limitedSnapshotRowCount = Math.max( + this.limitedSnapshotRowCount, + currentOffset + changes.length, + ) if (changes.length > 0) { this.lastSentKey = changes[changes.length - 1]!.key } @@ -573,27 +1503,9 @@ export class CollectionSubscription } | undefined - if (minValues !== undefined && minValues.length > 0) { - const whereFromCursor = buildCursor(orderBy, minValues) - - if (whereFromCursor) { - const { expression } = orderBy[0]! - const cursorMinValue = minValues[0] - - // Build the whereCurrent expression for the first orderBy column - // For Date values, we need to handle precision differences between JS (ms) and backends (μs) - // A JS Date represents a 1ms range, so we query for all values within that range - let whereCurrentCursor: BasicExpression - if (cursorMinValue instanceof Date) { - const cursorMinValuePlus1ms = new Date(cursorMinValue.getTime() + 1) - whereCurrentCursor = and( - gte(expression, new Value(cursorMinValue)), - lt(expression, new Value(cursorMinValuePlus1ms)), - ) - } else { - whereCurrentCursor = eq(expression, new Value(cursorMinValue)) - } - + if (whereFromCursor && minValues) { + const whereCurrentCursor = buildCursorCurrent(orderBy, minValues) + if (whereCurrentCursor) { cursorExpressions = { whereFrom: whereFromCursor, whereCurrent: whereCurrentCursor, @@ -614,16 +1526,30 @@ export class CollectionSubscription offset: offset ?? currentOffset, // Use provided offset, or auto-tracked offset subscription: this, } - const syncResult = this.collection._sync.loadSubset(loadOptions) - // Pass the raw loadSubset result to the caller for external tracking - onLoadSubsetResult?.(syncResult) - - // Track this loadSubset call - this.loadedSubsets.push(loadOptions) - if (shouldTrackLoadSubsetPromise) { - this.trackLoadSubsetPromise(syncResult) + const { + demand, + result: syncResult, + started, + } = this.startSubsetDemand(loadOptions) + if (!this.isDemandActive(demand)) return + + // Report the result synchronously, including a wait for an unavailable loader. + onLoadSubsetResult?.( + syncResult, + demand.acquisition.options, + (primaryFailure) => this.releaseDemand(demand, primaryFailure), + ) + if (!this.isDemandActive(demand)) return + if (started) { + this.observeLoadSubsetResult( + syncResult, + demand, + demand.acquisition.options, + shouldTrackLoadSubsetPromise, + ) } + if (!this.isDemandActive(demand)) return } // TODO: also add similar test but that checks that it can also load it from the collection's loadSubset function @@ -636,6 +1562,8 @@ export class CollectionSubscription * Duplicate inserts are filtered out to prevent D2 multiplicity > 1. */ private filterAndFlipChanges(changes: Array>) { + changes = this.reconcileStalePublishedChanges(changes) + if (this.loadedInitialState || this.skipFiltering) { // We loaded the entire initial state or filtering is explicitly skipped // so no need to filter or flip changes @@ -658,14 +1586,16 @@ export class CollectionSubscription if (!keyInSentKeys) { if (change.type === `update`) { newChange = { ...change, type: `insert`, previousValue: undefined } + this.sentKeys.add(change.key) } else if (change.type === `delete`) { // Filter out deletes for keys that have not been sent, // UNLESS we're buffering for truncate (where all deletes should pass through) if (!skipDeleteFilter) { continue } + } else { + this.sentKeys.add(change.key) } - this.sentKeys.add(change.key) } else { // Key was already sent - handle based on change type if (change.type === `insert`) { @@ -685,6 +1615,67 @@ export class CollectionSubscription return newChanges } + /** + * After a failed replay, the source collection is empty but subscribers still + * hold the last good publication. Reconcile the first later source delta for + * each retained key against that publication instead of treating it as a + * duplicate insert. + */ + private reconcileStalePublishedChanges( + changes: Array>, + ): Array> { + if (this.stalePublishedRows.size === 0) return changes + + const reconciled: Array> = [] + for (const change of changes) { + const previous = this.stalePublishedRows.get(change.key) + if (previous === undefined) { + reconciled.push(change) + continue + } + + this.stalePublishedRows.delete(change.key) + if (change.type === `delete`) { + reconciled.push({ + ...change, + value: previous, + previousValue: undefined, + }) + } else if (!deepEquals(previous, change.value)) { + reconciled.push({ + ...change, + type: `update`, + previousValue: previous, + }) + } + } + // Cleanup discards rows without publishing deletes. Eager sources publish + // their installed state; subset sources must first finish reacquisition. + if ( + this.collection.config.syncMode !== `on-demand` && + !this.isBufferingForTruncate + ) { + for (const [key, value] of this.stalePublishedRows) { + if (this.collection.has(key)) continue + this.stalePublishedRows.delete(key) + reconciled.push({ type: `delete`, key, value }) + } + } + return reconciled + } + + private trackPublishedRows( + changes: Array>, + ): void { + for (const change of changes) { + if (change.type === `delete`) { + this.publishedRows.delete(change.key) + } else { + this.publishedRows.set(change.key, change.value) + } + } + } + private trackSentKeys(changes: Array>) { if (this.loadedInitialState || this.skipFiltering) { // No need to track sent keys if we loaded the entire state or filtering is skipped. @@ -721,27 +1712,52 @@ export class CollectionSubscription } unsubscribe() { - // Clean up truncate event listener - this.truncateCleanup?.() + if (this.unsubscribed) return + this.unsubscribed = true + // Stop any status listener set already being iterated. Clearing the + // emitter's map cannot invalidate that captured Set by itself. + this.statusRevision++ + const sourceListenerCleanups = [ + this.truncateCleanup, + this.collectionCleanup, + this.collectionRestartCleanup, + ] this.truncateCleanup = undefined - - // Clean up truncate buffer state - this.isBufferingForTruncate = false - this.truncateBuffer = [] - this.pendingTruncateRefetches.clear() - - // Unload all subsets that this subscription loaded - // We pass the exact same LoadSubsetOptions we used for loadSubset - for (const options of this.loadedSubsets) { - this.collection._sync.unloadSubset(options) - } - this.loadedSubsets = [] - - this.emitInner(`unsubscribed`, { - type: `unsubscribed`, - subscription: this, - }) - // Clear all event listeners to prevent memory leaks - this.clearListeners() + this.collectionCleanup = undefined + this.collectionRestartCleanup = undefined + + runAllCallbacks([ + ...sourceListenerCleanups.map((cleanup) => () => cleanup?.()), + () => { + // Stop any buffered replay from publishing after unsubscription. + this.discardTruncateReplay() + this.stalePublishedRows.clear() + + // Retire every owner before an unload can reenter teardown. + const acquisitions = this.subsetDemands + .filter((demand) => demand.acquisitionState === `active`) + .map((demand) => demand.acquisition) + for (const demand of this.subsetDemands) { + demand.initialResult?.reject(new LoadSubsetOperationAbortedError()) + this.stopDemandStatusParticipants(demand) + if (demand.acquisitionState === `starting`) { + cancelAcquisition(demand.acquisition) + } + } + this.subsetDemands = [] + runAllCallbacks( + acquisitions.map( + (acquisition) => () => this.releaseAcquisition(acquisition), + ), + ) + }, + () => + this.emitInner(`unsubscribed`, { + type: `unsubscribed`, + subscription: this, + }), + // Clear all event listeners to prevent memory leaks + () => this.clearListeners(), + ]) } } diff --git a/packages/db/src/collection/sync.ts b/packages/db/src/collection/sync.ts index 82fffb7728..cecf93b957 100644 --- a/packages/db/src/collection/sync.ts +++ b/packages/db/src/collection/sync.ts @@ -1,13 +1,16 @@ import { CollectionConfigurationError, CollectionIsInErrorStateError, + CollectionPreloadAbortedError, DuplicateKeySyncError, + LoadSubsetOperationAbortedError, NoPendingSyncTransactionCommitError, NoPendingSyncTransactionWriteError, SyncCleanupError, SyncTransactionAlreadyCommittedError, SyncTransactionAlreadyCommittedWriteError, } from '../errors' +import { createDeferred } from '../deferred' import { deepEquals } from '../utils' import { LIVE_QUERY_INTERNAL } from '../query/live/internal.js' import type { StandardSchemaV1 } from '@standard-schema/spec' @@ -15,7 +18,9 @@ import type { ChangeMessageOrDeleteKeyMessage, CleanupFn, CollectionConfig, + LoadSubsetFn, LoadSubsetOptions, + LoadSubsetRequestResult, OptimisticChangeMessage, SyncConfigRes, SyncMetadataApi, @@ -25,6 +30,21 @@ import type { CollectionStateManager } from './state' import type { CollectionLifecycleManager } from './lifecycle' import type { CollectionEventsManager } from './events.js' import type { LiveQueryCollectionUtils } from '../query/live/collection-config-builder.js' +import type { Deferred } from '../deferred' + +type DeferredLoadSubset = { + options: LoadSubsetOptions + deferred: Deferred +} + +type LoadSubsetOperation = { + pending: Set> + waiting: boolean + completed: boolean + hasError: boolean + error?: unknown + deferred?: Deferred +} export class CollectionSyncManager< TOutput extends object = Record, @@ -41,14 +61,20 @@ export class CollectionSyncManager< private syncMode: `eager` | `on-demand` public preloadPromise: Promise | null = null + private rejectPreload?: (error: unknown) => void public syncCleanupFn: (() => void) | null = null - public syncLoadSubsetFn: - | ((options: LoadSubsetOptions) => true | Promise) - | null = null + public syncLoadSubsetFn: LoadSubsetFn | null = null public syncUnloadSubsetFn: ((options: LoadSubsetOptions) => void) | null = null - private pendingLoadSubsetPromises: Set> = new Set() + private pendingLoadSubsetPromises: Set> = new Set() + private activeLoadSubsetOperation: LoadSubsetOperation | undefined + private loadSubsetOperations = new Set() + private syncStartDeferred = false + private syncStartRequested = false + private deferredLoadSubsets: Array = [] + private syncEpoch = 0 + private loadSubsetSession = 0 /** * Creates a new CollectionSyncManager instance @@ -71,11 +97,17 @@ export class CollectionSyncManager< this._events = deps.events } + /** Mark the active sync transaction as changing collection layout. */ + public markLayoutChange(): void { + this.getActivePendingSyncTransaction().layoutChanged = true + } + /** * Start the sync process for this collection * This is called when the collection is first accessed or preloaded */ public startSync(): void { + this.lifecycle.assertCanStartSync() if ( this.lifecycle.status !== `idle` && this.lifecycle.status !== `cleaned-up` @@ -83,20 +115,39 @@ export class CollectionSyncManager< return // Already started or in progress } + if (this.syncStartDeferred) { + this.syncStartRequested = true + return + } + + const syncEpoch = ++this.syncEpoch + const isCurrentSync = () => syncEpoch === this.syncEpoch this.lifecycle.setStatus(`loading`) + if (!isCurrentSync()) return + let syncEntryActive = true + let readyEffectFailure: { error: unknown } | undefined try { const syncRes = normalizeSyncFnResult( this.config.sync.sync({ collection: this.collection, begin: (options?: { immediate?: boolean }) => { + if (!isCurrentSync()) return + const applied = createDeferred() + // A source may ignore a stream receipt. Keep cancellation from + // becoming an unhandled rejection while preserving the original + // promise's rejection for callers that do await it. + void applied.promise.catch(() => undefined) this.state.pendingSyncedTransactions.push({ committed: false, + applicationStarted: false, + layoutChanged: false, operations: [], deletedKeys: new Set(), rowMetadataWrites: new Map(), collectionMetadataWrites: new Map(), immediate: options?.immediate, + applied, }) }, write: ( @@ -105,6 +156,7 @@ export class CollectionSyncManager< TKey >, ) => { + if (!isCurrentSync()) return const pendingTransaction = this.state.pendingSyncedTransactions[ this.state.pendingSyncedTransactions.length - 1 @@ -123,10 +175,6 @@ export class CollectionSyncManager< key = this.config.getKey(messageWithOptionalKey.value) } - if (this.state.pendingLocalChanges.has(key)) { - this.state.pendingLocalOrigins.add(key) - } - let messageType = messageWithOptionalKey.type // Check if an item with this key already exists when inserting @@ -145,15 +193,17 @@ export class CollectionSyncManager< const valuesEqual = existingValue !== undefined && deepEquals(existingValue, messageWithOptionalKey.value) - if (valuesEqual) { + if (valuesEqual || this.state.hydrationSeedKeys.has(key)) { // The "insert" is an echo of a value we already have locally. - // Treat it as an update so we preserve optimistic intent without - // throwing a duplicate-key error during reconciliation. + // Hydration and initialData are also provisional base state, so + // accept the adapter's first authoritative value as an update + // using the configured rowUpdateMode semantics. messageType = `update` } else { - const utils = this.config - .utils as Partial - const internal = utils[LIVE_QUERY_INTERNAL] + const utils = this.config.utils as + | Partial + | undefined + const internal = utils?.[LIVE_QUERY_INTERNAL] throw new DuplicateKeySyncError(key, this.id, { hasCustomGetKey: internal?.hasCustomGetKey ?? false, hasJoins: internal?.hasJoins ?? false, @@ -191,7 +241,8 @@ export class CollectionSyncManager< }) } }, - commit: () => { + commit: (signal?: AbortSignal) => { + if (!isCurrentSync()) return true const pendingTransaction = this.state.pendingSyncedTransactions[ this.state.pendingSyncedTransactions.length - 1 @@ -203,14 +254,46 @@ export class CollectionSyncManager< throw new SyncTransactionAlreadyCommittedError() } + if (signal?.aborted) { + this.state.cancelPendingSyncedTransaction(pendingTransaction) + return pendingTransaction.applied.promise + } + pendingTransaction.committed = true + const cancel = () => { + this.state.cancelPendingSyncedTransaction(pendingTransaction) + } + signal?.addEventListener(`abort`, cancel, { once: true }) + this.state.commitPendingTransactions() + if (!pendingTransaction.applied.isPending()) { + signal?.removeEventListener(`abort`, cancel) + return true + } + + const receipt = pendingTransaction.applied.promise + if (signal) { + const removeAbortListener = () => { + signal.removeEventListener(`abort`, cancel) + } + void receipt.then(removeAbortListener, removeAbortListener) + } + return receipt }, markReady: () => { - this.lifecycle.markReady() + if (!isCurrentSync()) return + if (syncEntryActive) { + readyEffectFailure ??= this.lifecycle.markReadyDuringSyncStart() + } else { + this.lifecycle.markReady() + } + }, + markError: (error?: unknown) => { + if (isCurrentSync()) this.lifecycle.markError(error) }, truncate: () => { + if (!isCurrentSync()) return const pendingTransaction = this.state.pendingSyncedTransactions[ this.state.pendingSyncedTransactions.length - 1 @@ -245,9 +328,16 @@ export class CollectionSyncManager< deletes: new Set(this.state.optimisticDeletes), } }, - metadata: this.createSyncMetadataApi(), + metadata: this.createSyncMetadataApi(isCurrentSync), }), ) + syncEntryActive = false + + if (!isCurrentSync()) { + syncRes?.cleanup?.() + if (readyEffectFailure) throw readyEffectFailure.error + return + } // Store cleanup function if provided this.syncCleanupFn = syncRes?.cleanup ?? null @@ -265,10 +355,78 @@ export class CollectionSyncManager< `Either provide a loadSubset handler or use syncMode "eager".`, ) } + + // Every route into sync passes through here, so it is the one place + // that sees sync start ahead of the subscriber that would justify it. + // `addSubscriber` counts itself in before calling us, so a subscription + // starting sync leaves the timer alone. + this.lifecycle.startGCTimerIfUnsubscribed() } catch (error) { - this.lifecycle.setStatus(`error`) + syncEntryActive = false + if (isCurrentSync()) this.lifecycle.markError(error) throw error } + if (readyEffectFailure) throw readyEffectFailure.error + } + + public deferStart(): boolean { + if ( + this.lifecycle.status !== `idle` && + this.lifecycle.status !== `cleaned-up` + ) { + return false + } + + this.syncStartDeferred = true + return true + } + + public resumeStart(): void { + if (!this.syncStartDeferred) { + return + } + + this.syncStartDeferred = false + const shouldStart = + this.syncStartRequested || this.deferredLoadSubsets.length > 0 + this.syncStartRequested = false + const deferredLoadSubsets = this.deferredLoadSubsets + this.deferredLoadSubsets = [] + const loadSubsetSession = this.loadSubsetSession + + try { + if (shouldStart) { + this.startSync() + } + } catch (error) { + for (const { deferred } of deferredLoadSubsets) { + deferred.reject(error) + } + throw error + } + + for (const { options, deferred } of deferredLoadSubsets) { + const loadSubset = this.syncLoadSubsetFn + try { + if ( + loadSubsetSession !== this.loadSubsetSession || + options.signal?.aborted + ) { + throw new LoadSubsetOperationAbortedError() + } + const result = loadSubset?.(options) ?? true + if (result instanceof Promise) { + void result.then( + (sourceResult) => deferred.resolve(sourceResult), + (error: unknown) => deferred.reject(error), + ) + } else { + deferred.resolve(undefined) + } + } catch (error) { + deferred.reject(error) + } + } } private getActivePendingSyncTransaction() { @@ -287,10 +445,13 @@ export class CollectionSyncManager< return pendingTransaction } - private createSyncMetadataApi(): SyncMetadataApi { + private createSyncMetadataApi( + isCurrentSync: () => boolean, + ): SyncMetadataApi { return { row: { get: (key) => { + if (!isCurrentSync()) return undefined const pendingTransaction = this.state.pendingSyncedTransactions[ this.state.pendingSyncedTransactions.length - 1 @@ -307,6 +468,7 @@ export class CollectionSyncManager< return this.state.syncedMetadata.get(key) }, set: (key, metadata) => { + if (!isCurrentSync()) return const pendingTransaction = this.getActivePendingSyncTransaction() pendingTransaction.rowMetadataWrites.set(key, { type: `set`, @@ -314,6 +476,7 @@ export class CollectionSyncManager< }) }, delete: (key) => { + if (!isCurrentSync()) return const pendingTransaction = this.getActivePendingSyncTransaction() pendingTransaction.rowMetadataWrites.set(key, { type: `delete`, @@ -322,6 +485,7 @@ export class CollectionSyncManager< }, collection: { get: (key) => { + if (!isCurrentSync()) return undefined const pendingTransaction = this.state.pendingSyncedTransactions[ this.state.pendingSyncedTransactions.length - 1 @@ -336,6 +500,7 @@ export class CollectionSyncManager< return this.state.syncedCollectionMetadata.get(key) }, set: (key, value) => { + if (!isCurrentSync()) return const pendingTransaction = this.getActivePendingSyncTransaction() pendingTransaction.collectionMetadataWrites.set(key, { type: `set`, @@ -343,12 +508,14 @@ export class CollectionSyncManager< }) }, delete: (key) => { + if (!isCurrentSync()) return const pendingTransaction = this.getActivePendingSyncTransaction() pendingTransaction.collectionMetadataWrites.set(key, { type: `delete`, }) }, list: (prefix) => { + if (!isCurrentSync()) return [] const merged = new Map(this.state.syncedCollectionMetadata) const pendingTransaction = this.state.pendingSyncedTransactions[ @@ -378,11 +545,27 @@ export class CollectionSyncManager< } } + /** Whether a caller is still waiting for the initial sync to finish. */ + public get hasPendingPreload(): boolean { + return this.rejectPreload !== undefined + } + /** * Preload the collection data by starting sync if not already started * Multiple concurrent calls will share the same promise */ public preload(): Promise { + try { + this.lifecycle.assertCanStartSync() + } catch (error) { + return Promise.reject(error) + } + // Warm preloads need the same handoff time as a load that just finished, + // including when the previous GC deadline already queued idle cleanup. + if (this.lifecycle.status === `ready`) { + this.lifecycle.cancelGCTimer() + this.lifecycle.startGCTimerIfUnsubscribed() + } if (this.preloadPromise) { return this.preloadPromise } @@ -397,20 +580,54 @@ export class CollectionSyncManager< ) } - this.preloadPromise = new Promise((resolve, reject) => { + const attempt = new Promise((resolve, reject) => { if (this.lifecycle.status === `ready`) { resolve() return } if (this.lifecycle.status === `error`) { - reject(new CollectionIsInErrorStateError()) + reject(this.getPreloadError()) return } - // Register callback BEFORE starting sync to avoid race condition - this.lifecycle.onFirstReady(() => { + let settled = false + const syncStartState = { active: false, ready: false } + let unsubscribeError = () => {} + let unsubscribeReady = () => {} + const finishPreload = () => { + settled = true + unsubscribeError() + unsubscribeReady() + if (this.rejectPreload === rejectError) this.rejectPreload = undefined + this.lifecycle.startGCTimerIfUnsubscribed() + } + const resolveReady = () => { + if (syncStartState.active) { + syncStartState.ready = true + return + } + if (settled) return + finishPreload() resolve() + } + const rejectError = (error: unknown) => { + if (settled) return + finishPreload() + reject(error) + } + + // Register callback BEFORE starting sync to avoid race condition + this.rejectPreload = rejectError + // An awaited preload owns this sync run until it settles, including + // when GC has already queued the destructive idle callback. + this.lifecycle.cancelGCTimer() + unsubscribeReady = this.lifecycle.onFirstReady(resolveReady) + unsubscribeError = this.collection.on(`status:error`, () => { + if (syncStartState.active) { + return + } + rejectError(this.getPreloadError()) }) // Start sync if collection hasn't started yet or was cleaned up @@ -418,16 +635,42 @@ export class CollectionSyncManager< this.lifecycle.status === `idle` || this.lifecycle.status === `cleaned-up` ) { + syncStartState.active = true + let startFailure: { error: unknown } | undefined try { this.startSync() } catch (error) { - reject(error) - return + startFailure = { error } + } finally { + syncStartState.active = false + } + if (this.collection.status === `error`) { + rejectError(this.getPreloadError()) + } else if (syncStartState.ready) { + // A first-ready listener can throw after readiness is established. + // That failure still escapes direct startSync(), but preload follows + // the final collection state after synchronous adapter entry. + resolveReady() + } else if (startFailure) { + rejectError(startFailure.error) } } }) - return this.preloadPromise + this.preloadPromise = attempt + void attempt.then(undefined, () => { + if (this.preloadPromise === attempt) { + this.preloadPromise = null + } + }) + return attempt + } + + private getPreloadError(): unknown { + const syncError = this.lifecycle.getSyncError() + return syncError === undefined + ? new CollectionIsInErrorStateError() + : syncError } /** @@ -437,13 +680,109 @@ export class CollectionSyncManager< return this.pendingLoadSubsetPromises.size > 0 } + /** @internal Observe subset requests caused by one imperative operation. */ + public beginLoadSubsetOperation(): { + wait: () => true | Promise + cancel: () => void + } { + const previousOperation = this.activeLoadSubsetOperation + const operation: LoadSubsetOperation = { + pending: new Set(), + waiting: false, + completed: false, + hasError: false, + } + // A new imperative operation owns future requests. Older operations keep + // waiting for the promises they already acquired, but cannot absorb work + // caused by a superseding physical window. + this.activeLoadSubsetOperation = operation + this.loadSubsetOperations.add(operation) + return { + wait: () => this.waitForLoadSubsetOperation(operation), + cancel: () => { + operation.completed = true + this.loadSubsetOperations.delete(operation) + if (this.activeLoadSubsetOperation === operation) { + this.activeLoadSubsetOperation = previousOperation?.completed + ? undefined + : previousOperation + } + }, + } + } + + private waitForLoadSubsetOperation( + operation: LoadSubsetOperation, + ): true | Promise { + operation.waiting = true + if (operation.pending.size === 0) { + operation.completed = true + this.loadSubsetOperations.delete(operation) + if (this.activeLoadSubsetOperation === operation) { + this.activeLoadSubsetOperation = undefined + } + return operation.hasError ? Promise.reject(operation.error) : true + } + operation.deferred = createDeferred() + return operation.deferred.promise + } + + private settleLoadSubsetOperation( + operation: LoadSubsetOperation, + promise: Promise, + outcome: { ok: true } | { ok: false; error: unknown }, + ): void { + if (operation.completed) return + operation.pending.delete(promise) + if (!outcome.ok && !operation.hasError) { + operation.hasError = true + operation.error = outcome.error + } + if (!operation.waiting || operation.pending.size > 0) return + + // A resolved request can synchronously publish source rows that register + // follow-up loads. Let those registrations join this operation before it + // is considered complete. + queueMicrotask(() => { + if (operation.completed || operation.pending.size > 0) return + operation.completed = true + this.loadSubsetOperations.delete(operation) + if (this.activeLoadSubsetOperation === operation) { + this.activeLoadSubsetOperation = undefined + } + if (operation.hasError) { + operation.deferred!.reject(operation.error) + } else { + operation.deferred!.resolve() + } + }) + } + + /** @internal Attach a relevant existing request to the active operation. */ + public trackLoadSubsetOperationPromise(promise: Promise): void { + const operation = this.activeLoadSubsetOperation + if (!operation || operation.pending.has(promise)) return + + operation.pending.add(promise) + void promise.then( + () => this.settleLoadSubsetOperation(operation, promise, { ok: true }), + (error) => + this.settleLoadSubsetOperation(operation, promise, { + ok: false, + error, + }), + ) + } + /** * Tracks a load promise for isLoadingSubset state. * @internal This is for internal coordination (e.g., live-query glue code), not for general use. */ - public trackLoadPromise(promise: Promise): void { + public trackLoadPromise(promise: Promise): void { + const loadSubsetSession = this.loadSubsetSession const loadingStarting = !this.isLoadingSubset this.pendingLoadSubsetPromises.add(promise) + this.trackLoadSubsetOperationPromise(promise) if (loadingStarting) { this._events.emit(`loadingSubset:change`, { @@ -455,7 +794,9 @@ export class CollectionSyncManager< }) } - promise.finally(() => { + const finish = () => { + if (loadSubsetSession !== this.loadSubsetSession) return + const loadingEnding = this.pendingLoadSubsetPromises.size === 1 && this.pendingLoadSubsetPromises.has(promise) @@ -470,7 +811,13 @@ export class CollectionSyncManager< loadingSubsetTransition: `end`, }) } - }) + } + void promise.then(finish, finish) + } + + /** @internal Generation fence for subscription-owned async work. */ + public getLoadSubsetSession(): number { + return this.loadSubsetSession } /** @@ -479,12 +826,24 @@ export class CollectionSyncManager< * @returns If data loading is asynchronous, this method returns a promise that resolves when the data is loaded. * Returns true if no sync function is configured, if syncMode is 'eager', or if there is no work to do. */ - public loadSubset(options: LoadSubsetOptions): Promise | true { + public loadSubset(options: LoadSubsetOptions): LoadSubsetRequestResult { + if (options.signal?.aborted) { + return Promise.reject(new LoadSubsetOperationAbortedError()) + } + // Bypass loadSubset when syncMode is 'eager' if (this.syncMode === `eager`) { return true } + if (this.syncStartDeferred) { + this.syncStartRequested = true + const deferred = createDeferred() + this.deferredLoadSubsets.push({ options, deferred }) + this.trackLoadPromise(deferred.promise) + return deferred.promise + } + if (this.syncLoadSubsetFn) { const result = this.syncLoadSubsetFn(options) // If the result is a promise, track it @@ -502,18 +861,42 @@ export class CollectionSyncManager< * @param options Options that identify what data is being unloaded */ public unloadSubset(options: LoadSubsetOptions): void { + // Eager loading bypasses subset acquisition, so there is no lease to release. + if (this.syncMode === `eager`) return + + if (this.syncStartDeferred) { + this.deferredLoadSubsets = this.deferredLoadSubsets.filter((request) => { + if (request.options !== options) { + return true + } + + request.deferred.reject(new LoadSubsetOperationAbortedError()) + return false + }) + return + } + if (this.syncUnloadSubsetFn) { this.syncUnloadSubsetFn(options) } } public cleanup(): void { + // Invalidate callbacks retained by asynchronous work from this session + // before invoking adapter cleanup or allowing a new session to start. + const cleanupEpoch = ++this.syncEpoch + this.loadSubsetSession++ + this.rejectPreload?.(new CollectionPreloadAbortedError()) + const cleanup = this.syncCleanupFn + this.syncCleanupFn = null + this.syncLoadSubsetFn = null + this.syncUnloadSubsetFn = null try { - if (this.syncCleanupFn) { - this.syncCleanupFn() - this.syncCleanupFn = null - } + cleanup?.() } catch (error) { + // Keep failed cleanup retryable, but never overwrite a replacement + // session installed by reentrant adapter code. + if (this.syncEpoch === cleanupEpoch) this.syncCleanupFn = cleanup // Re-throw in a microtask to surface the error after cleanup completes queueMicrotask(() => { if (error instanceof Error) { @@ -528,6 +911,35 @@ export class CollectionSyncManager< }) } this.preloadPromise = null + this.syncStartDeferred = false + this.syncStartRequested = false + const wasLoadingSubset = this.pendingLoadSubsetPromises.size > 0 + this.pendingLoadSubsetPromises.clear() + if (wasLoadingSubset) { + this._events.emit(`loadingSubset:change`, { + type: `loadingSubset:change`, + collection: this.collection, + isLoadingSubset: false, + previousIsLoadingSubset: true, + loadingSubsetTransition: `end`, + }) + } + this.activeLoadSubsetOperation = undefined + for (const operation of this.loadSubsetOperations) { + if (!operation.completed) { + operation.completed = true + operation.pending.clear() + operation.hasError = true + operation.error = new LoadSubsetOperationAbortedError() + operation.deferred?.reject(operation.error) + } + } + this.loadSubsetOperations.clear() + const deferredLoadSubsets = this.deferredLoadSubsets + this.deferredLoadSubsets = [] + for (const request of deferredLoadSubsets) { + request.deferred.reject(new LoadSubsetOperationAbortedError()) + } } } diff --git a/packages/db/src/errors.ts b/packages/db/src/errors.ts index 472fcabd12..a97cb117b7 100644 --- a/packages/db/src/errors.ts +++ b/packages/db/src/errors.ts @@ -135,6 +135,18 @@ export class NegativeActiveSubscribersError extends CollectionStateError { } } +export class LiveQueryObserverDisposedError extends CollectionStateError { + constructor() { + super(`Cannot subscribe to a disposed LiveQueryObserver`) + } +} + +export class LiveQueryWindowControllerDisposedError extends CollectionStateError { + constructor() { + super(`Cannot subscribe to a disposed LiveQueryWindowController`) + } +} + // Collection Operation Errors export class CollectionOperationError extends TanStackDBError { constructor(message: string) { @@ -442,6 +454,16 @@ export class QueryCompilationError extends TanStackDBError { } } +export class UnsafeAliasPathError extends QueryCompilationError { + constructor(segment: string) { + super( + `Unsafe alias path segment "${segment}" is not allowed in .select(). ` + + `Aliases must not contain "__proto__", "prototype", or "constructor".`, + ) + this.name = `UnsafeAliasPathError` + } +} + export class DistinctRequiresSelectError extends QueryCompilationError { constructor() { super(`DISTINCT requires a SELECT clause.`) @@ -459,6 +481,16 @@ export class FnSelectWithGroupByError extends QueryCompilationError { } } +export class UnsupportedFnSelectResultError extends QueryCompilationError { + constructor(valueDescription: string) { + super( + `fn.select() cannot return ${valueDescription}. ` + + `Child query builders, query expressions, and helpers such as eq(), toArray(), materialize(), concat(toArray()), and caseWhen() are query-construction values. ` + + `Use them as direct fields in .select() instead.`, + ) + } +} + export class UnsupportedRootScalarSelectError extends QueryCompilationError { constructor() { super( @@ -697,56 +729,41 @@ export class SyncCleanupError extends TanStackDBError { } } -// Query Optimizer Errors -export class QueryOptimizerError extends TanStackDBError { - constructor(message: string) { - super(message) - this.name = `QueryOptimizerError` +/** A sync transaction was canceled before its writes became visible. */ +export class SyncTransactionAbortedError extends Error { + constructor() { + super(`Sync transaction was aborted before application`) + this.name = `AbortError` } } -export class CannotCombineEmptyExpressionListError extends QueryOptimizerError { +/** A collection was cleaned up before its initial preload became ready. */ +export class CollectionPreloadAbortedError extends Error { constructor() { - super(`Cannot combine empty expression list`) + super(`Collection preload was abandoned during cleanup`) + this.name = `AbortError` } } -/** - * Internal error when the query optimizer fails to convert a WHERE clause to a collection filter. - */ -export class WhereClauseConversionError extends QueryOptimizerError { - constructor(collectionId: string, alias: string) { - super( - `Failed to convert WHERE clause to collection filter for collection '${collectionId}' alias '${alias}'. This indicates a bug in the query optimization logic.`, - ) +/** A subset operation was canceled before its result became visible. */ +export class LoadSubsetOperationAbortedError extends Error { + constructor() { + super(`Load subset operation was aborted before its result became visible`) + this.name = `AbortError` } } -/** - * Error when a subscription cannot be found during lazy join processing. - * For subqueries, aliases may be remapped (e.g., 'activeUser' → 'user'). - */ -export class SubscriptionNotFoundError extends QueryCompilationError { - constructor( - resolvedAlias: string, - originalAlias: string, - collectionId: string, - availableAliases: Array, - ) { - super( - `Internal error: subscription for alias '${resolvedAlias}' (remapped from '${originalAlias}', collection '${collectionId}') is missing in join pipeline. Available aliases: ${availableAliases.join(`, `)}. This indicates a bug in alias tracking.`, - ) +// Query Optimizer Errors +export class QueryOptimizerError extends TanStackDBError { + constructor(message: string) { + super(message) + this.name = `QueryOptimizerError` } } -/** - * Error thrown when aggregate expressions are used outside of a GROUP BY context. - */ -export class AggregateNotSupportedError extends QueryCompilationError { +export class CannotCombineEmptyExpressionListError extends QueryOptimizerError { constructor() { - super( - `Aggregate expressions are not supported in this context. Use GROUP BY clause for aggregates.`, - ) + super(`Cannot combine empty expression list`) } } @@ -774,3 +791,13 @@ export class SetWindowRequiresOrderByError extends QueryCompilationError { ) } } + +/** Error thrown when setWindow is called from inside another setWindow call. */ +export class SetWindowReentrancyError extends TanStackDBError { + constructor() { + super( + `setWindow() cannot run reentrantly. Wait for the current window operation to return before starting another one.`, + ) + this.name = `SetWindowReentrancyError` + } +} diff --git a/packages/db/src/event-emitter.ts b/packages/db/src/event-emitter.ts index 6d7ad90aa2..06fec5adb0 100644 --- a/packages/db/src/event-emitter.ts +++ b/packages/db/src/event-emitter.ts @@ -5,7 +5,11 @@ export class EventEmitter> { private listeners = new Map< keyof TEvents, - Set<(event: TEvents[keyof TEvents]) => void> + Map<(event: TEvents[keyof TEvents]) => void, object> + >() + private onceCallbacks = new WeakMap< + (event: TEvents[keyof TEvents]) => void, + (event: TEvents[keyof TEvents]) => void >() /** @@ -19,12 +23,21 @@ export class EventEmitter> { callback: (event: TEvents[T]) => void, ): () => void { if (!this.listeners.has(event)) { - this.listeners.set(event, new Set()) + this.listeners.set(event, new Map()) + } + const listeners = this.listeners.get(event)! + const registered = callback as (event: any) => void + let registration = listeners.get(registered) + if (!registration) { + registration = {} + listeners.set(registered, registration) } - this.listeners.get(event)!.add(callback as (event: any) => void) return () => { - this.listeners.get(event)?.delete(callback as (event: any) => void) + const current = this.listeners.get(event) + if (current?.get(registered) === registration) { + current.delete(registered) + } } } @@ -38,10 +51,16 @@ export class EventEmitter> { event: T, callback: (event: TEvents[T]) => void, ): () => void { - const unsubscribe = this.on(event, (eventPayload) => { - callback(eventPayload) + let unsubscribe = () => {} + const listener = (eventPayload: TEvents[T]) => { unsubscribe() - }) + callback(eventPayload) + } + this.onceCallbacks.set( + listener as (event: TEvents[keyof TEvents]) => void, + callback as (event: TEvents[keyof TEvents]) => void, + ) + unsubscribe = this.on(event, listener) return unsubscribe } @@ -54,7 +73,16 @@ export class EventEmitter> { event: T, callback: (event: TEvents[T]) => void, ): void { - this.listeners.get(event)?.delete(callback as (event: any) => void) + const listeners = this.listeners.get(event) + if (!listeners) return + for (const listener of listeners.keys()) { + if ( + listener === callback || + this.onceCallbacks.get(listener) === callback + ) { + listeners.delete(listener) + } + } } /** @@ -97,7 +125,20 @@ export class EventEmitter> { event: T, eventPayload: TEvents[T], ): void { - this.listeners.get(event)?.forEach((listener) => { + this.emitInnerWhile(event, eventPayload, () => true) + } + + /** Emit until a reentrant callback invalidates the event being delivered. */ + protected emitInnerWhile( + event: T, + eventPayload: TEvents[T], + isCurrent: () => boolean, + ): void { + const listeners = this.listeners.get(event) + if (!listeners) return + for (const [listener, registration] of [...listeners]) { + if (!isCurrent()) break + if (this.listeners.get(event)?.get(listener) !== registration) continue try { listener(eventPayload) } catch (error) { @@ -106,7 +147,7 @@ export class EventEmitter> { throw error }) } - }) + } } /** diff --git a/packages/db/src/index.ts b/packages/db/src/index.ts index ec1e229665..18a6bee3b6 100644 --- a/packages/db/src/index.ts +++ b/packages/db/src/index.ts @@ -6,10 +6,17 @@ import * as IR from './query/ir.js' export * from './collection/index.js' export * from './SortedMap' export * from './transactions' +export * from './client.js' +export { withCollectionConfigFactory } from './client.js' export * from './types' export * from './proxy' export * from './query/index.js' export * from './optimistic-action' +export * from './live-query-adapter' +export * from './live-query-observer' +export * from './live-query-options' +/** @internal Unstable adapter primitive for RFC #1623. */ +export * from './live-query-window-controller' export * from './local-only' export * from './local-storage' export * from './errors' @@ -31,8 +38,8 @@ export { BaseIndex } from './indexes/base-index.js' export type { IndexInterface, IndexConstructor, - IndexStats, IndexOperation, + IndexReader, } from './indexes/base-index.js' export { type IndexOptions } from './indexes/index-options.js' @@ -80,6 +87,9 @@ export { type EffectQueryInput, } from './query/effect.js' +// UUID helper (safe in non-secure browser contexts, see #1541) +export { safeRandomUUID } from './utils/uuid.js' + // Re-export some stuff explicitly to ensure the type & value is exported export type { Collection } from './collection/index.js' export { IR } diff --git a/packages/db/src/indexes/auto-index.ts b/packages/db/src/indexes/auto-index.ts index 350b469a43..6e6f1896ed 100644 --- a/packages/db/src/indexes/auto-index.ts +++ b/packages/db/src/indexes/auto-index.ts @@ -5,10 +5,6 @@ import type { CompareOptions } from '../query/builder/types' import type { BasicExpression } from '../query/ir' import type { CollectionImpl } from '../collection/index.js' -export interface AutoIndexConfig { - autoIndex?: `off` | `eager` -} - function shouldAutoIndex(collection: CollectionImpl) { // Only proceed if auto-indexing is enabled // Note: autoIndex: 'eager' without defaultIndexType is caught at construction time diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index 4346450f90..777d89381a 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -1,10 +1,33 @@ import { compileSingleRowExpression } from '../query/compiler/evaluators.js' import { comparisonFunctions } from '../query/builder/functions.js' import { DEFAULT_COMPARE_OPTIONS, deepEquals } from '../utils.js' +import type { CompiledSingleRowExpression } from '../query/compiler/evaluators.js' import type { RangeQueryOptions } from './btree-index.js' import type { CompareOptions } from '../query/builder/types.js' import type { BasicExpression, OrderByDirection } from '../query/ir.js' +function normalizeLocaleOptions(options: object | undefined): object { + return Object.fromEntries( + Object.entries(options ?? {}).filter(([, value]) => value !== undefined), + ) +} + +function canonicalizeLocale(locale: string | undefined): string | undefined { + return locale === undefined ? undefined : Intl.getCanonicalLocales(locale)[0] +} + +type LocaleCompareOptions = CompareOptions & { + stringSort?: `locale` + locale?: string + localeOptions?: object +} + +function usesLocaleCollation( + options: CompareOptions, +): options is LocaleCompareOptions { + return (options.stringSort ?? DEFAULT_COMPARE_OPTIONS.stringSort) === `locale` +} + /** * Operations that indexes can support, imported from available comparison functions */ @@ -15,15 +38,18 @@ export const IndexOperation = comparisonFunctions */ export type IndexOperation = (typeof comparisonFunctions)[number] -/** - * Statistics about index usage and performance - */ -export interface IndexStats { - readonly entryCount: number - readonly lookupCount: number - readonly averageLookupTime: number - readonly lastUpdated: Date -} +/** The read-side surface consumers use on a resolved (possibly reversed) index. */ +export type IndexReader = Pick< + IndexInterface, + | `lookup` + | `rangeQuery` + | `take` + | `takeFromStart` + | `keyCount` + | `supports` + | `supportsRangeOptimization` + | `canOptimizeRangeFor` +> export interface IndexInterface< TKey extends string | number = string | number, @@ -45,13 +71,13 @@ export interface IndexInterface< take: ( n: number, - from: TKey, + from: unknown, filterFn?: (key: TKey) => boolean, ) => Array takeFromStart: (n: number, filterFn?: (key: TKey) => boolean) => Array takeReversed: ( n: number, - from: TKey, + from: unknown, filterFn?: (key: TKey) => boolean, ) => Array takeReversedFromEnd: ( @@ -60,19 +86,27 @@ export interface IndexInterface< ) => Array get keyCount(): number - get orderedEntriesArray(): Array<[any, Set]> - get orderedEntriesArrayReversed(): Array<[any, Set]> + supports: (operation: IndexOperation) => boolean - get indexedKeysSet(): Set - get valueMapData(): Map> + /** + * Whether range lookups (gt/gte/lt/lte) on this index can be trusted to + * return every matching key. Range traversal relies on the index ordering, so + * it is unsafe when the index uses a custom comparator, whose order may not + * match the WHERE evaluator's relational operators. Callers must fall back to + * a full scan when this is `false`. + */ + get supportsRangeOptimization(): boolean - supports: (operation: IndexOperation) => boolean + /** + * Whether the live values in this index share the predicate operand's + * relational domain. Mixed domains can sort differently in the index and + * WHERE evaluator, which can make a range lookup omit matching rows. + */ + canOptimizeRangeFor?: (value: unknown) => boolean matchesField: (fieldPath: Array) => boolean matchesCompareOptions: (compareOptions: CompareOptions) => boolean matchesDirection: (direction: OrderByDirection) => boolean - - getStats: () => IndexStats } /** @@ -85,11 +119,14 @@ export abstract class BaseIndex< public readonly name?: string public readonly expression: BasicExpression public abstract readonly supportedOperations: Set - - protected lookupCount = 0 - protected totalLookupTime = 0 - protected lastUpdated = new Date() protected compareOptions: CompareOptions + private compiledIndexEvaluator: CompiledSingleRowExpression | undefined + /** + * Set by subclasses when constructed with a user-supplied comparator, whose + * ordering may not match the WHERE evaluator's relational operators. + */ + protected hasCustomComparator = false + private rangeValueDomains = new Map() constructor( id: number, @@ -113,7 +150,7 @@ export abstract class BaseIndex< abstract lookup(operation: IndexOperation, value: any): Set abstract take( n: number, - from: TKey, + from: unknown, filterFn?: (key: TKey) => boolean, ): Array abstract takeFromStart( @@ -122,7 +159,7 @@ export abstract class BaseIndex< ): Array abstract takeReversed( n: number, - from: TKey, + from: unknown, filterFn?: (key: TKey) => boolean, ): Array abstract takeReversedFromEnd( @@ -133,17 +170,62 @@ export abstract class BaseIndex< abstract equalityLookup(value: any): Set abstract inArrayLookup(values: Array): Set abstract rangeQuery(options: RangeQueryOptions): Set - abstract rangeQueryReversed(options: RangeQueryOptions): Set - abstract get orderedEntriesArray(): Array<[any, Set]> - abstract get orderedEntriesArrayReversed(): Array<[any, Set]> - abstract get indexedKeysSet(): Set - abstract get valueMapData(): Map> // Common methods + rangeQueryReversed(options: RangeQueryOptions = {}): Set { + const { from, to, fromInclusive = true, toInclusive = true } = options + const reversed: RangeQueryOptions = {} + if (`to` in options) { + reversed.from = to + reversed.fromInclusive = toInclusive + } + if (`from` in options) { + reversed.to = from + reversed.toInclusive = fromInclusive + } + return this.rangeQuery(reversed) + } + supports(operation: IndexOperation): boolean { return this.supportedOperations.has(operation) } + get supportsRangeOptimization(): boolean { + return !this.hasCustomComparator + } + + protected addRangeValue(value: unknown): void { + const domain = rangeValueDomain(value) + if (domain === undefined) return + this.rangeValueDomains.set( + domain, + (this.rangeValueDomains.get(domain) ?? 0) + 1, + ) + } + + protected removeRangeValue(value: unknown): void { + const domain = rangeValueDomain(value) + if (domain === undefined) return + const count = this.rangeValueDomains.get(domain) + if (count === undefined) return + if (count === 1) this.rangeValueDomains.delete(domain) + else this.rangeValueDomains.set(domain, count - 1) + } + + protected clearRangeValues(): void { + this.rangeValueDomains.clear() + } + + canOptimizeRangeFor(value: unknown): boolean { + const domain = rangeValueDomain(value) + if (domain === undefined) return true + if (!isNativeRangeDomain(domain)) return false + return ( + this.rangeValueDomains.size === 0 || + (this.rangeValueDomains.size === 1 && this.rangeValueDomains.has(domain)) + ) + } + matchesField(fieldPath: Array): boolean { return ( this.expression.type === `ref` && @@ -157,18 +239,28 @@ export abstract class BaseIndex< * The direction is ignored because the index can be reversed if the direction is different. */ matchesCompareOptions(compareOptions: CompareOptions): boolean { - const thisCompareOptionsWithoutDirection = { - ...this.compareOptions, - direction: undefined, + const indexCompareOptions = this.compareOptions + const indexUsesLocale = usesLocaleCollation(indexCompareOptions) + const requestedUsesLocale = usesLocaleCollation(compareOptions) + + if ( + indexCompareOptions.nulls !== compareOptions.nulls || + indexUsesLocale !== requestedUsesLocale + ) { + return false } - const compareOptionsWithoutDirection = { - ...compareOptions, - direction: undefined, + + if (!indexUsesLocale || !requestedUsesLocale) { + return true } - return deepEquals( - thisCompareOptionsWithoutDirection, - compareOptionsWithoutDirection, + return ( + canonicalizeLocale(indexCompareOptions.locale) === + canonicalizeLocale(compareOptions.locale) && + deepEquals( + normalizeLocaleOptions(indexCompareOptions.localeOptions), + normalizeLocaleOptions(compareOptions.localeOptions), + ) ) } @@ -179,32 +271,29 @@ export abstract class BaseIndex< return this.compareOptions.direction === direction } - getStats(): IndexStats { - return { - entryCount: this.keyCount, - lookupCount: this.lookupCount, - averageLookupTime: - this.lookupCount > 0 ? this.totalLookupTime / this.lookupCount : 0, - lastUpdated: this.lastUpdated, - } - } - protected abstract initialize(options?: any): void protected evaluateIndexExpression(item: any): any { - const evaluator = compileSingleRowExpression(this.expression) + const evaluator = (this.compiledIndexEvaluator ??= + compileSingleRowExpression(this.expression)) return evaluator(item as Record) } +} - protected trackLookup(startTime: number): void { - const duration = performance.now() - startTime - this.lookupCount++ - this.totalLookupTime += duration - } +function rangeValueDomain(value: unknown): string | undefined { + if (value == null) return undefined + if (value instanceof Date) return `date` + return typeof value +} - protected updateTimestamp(): void { - this.lastUpdated = new Date() - } +function isNativeRangeDomain(domain: string): boolean { + return ( + domain === `number` || + domain === `bigint` || + domain === `boolean` || + domain === `string` || + domain === `date` + ) } /** diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index b80f7fb439..8f47e1b665 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -1,6 +1,12 @@ -import { defaultComparator, normalizeValue } from '../utils/comparison.js' +import { compareKeys } from '@tanstack/db-ivm' import { - deleteInSortedArray, + areSameValueZeroEqual, + defaultComparator, + makeComparator, + normalizeValue, +} from '../utils/comparison.js' +import { + compareKeysReversed, findInsertPositionInArray, } from '../utils/array-utils.js' import { BaseIndex } from './base-index.js' @@ -64,10 +70,11 @@ export class BasicIndex< options?: any, ) { super(id, expression, name, options) - this.compareFn = options?.compareFn ?? defaultComparator if (options?.compareOptions) { this.compareOptions = options!.compareOptions } + this.compareFn = options?.compareFn ?? makeComparator(this.compareOptions) + this.hasCustomComparator = options?.compareFn != null } protected initialize(_options?: BasicIndexOptions): void {} @@ -88,9 +95,17 @@ export class BasicIndex< const normalizedValue = normalizeValue(indexedValue) - if (this.valueMap.has(normalizedValue)) { + this.addToBucket(key, normalizedValue) + this.addRangeValue(indexedValue) + + this.indexedKeys.add(key) + } + + private addToBucket(key: TKey, normalizedValue: unknown): void { + const keySet = this.valueMap.get(normalizedValue) + if (keySet) { // Value already exists, just add the key to the set - this.valueMap.get(normalizedValue)!.add(key) + keySet.add(key) } else { // New value - add to map and insert into sorted array this.valueMap.set(normalizedValue, new Set([key])) @@ -103,9 +118,6 @@ export class BasicIndex< ) this.sortedValues.splice(insertIdx, 0, normalizedValue) } - - this.indexedKeys.add(key) - this.updateTimestamp() } /** @@ -121,33 +133,82 @@ export class BasicIndex< error, ) this.indexedKeys.delete(key) - this.updateTimestamp() return } const normalizedValue = normalizeValue(indexedValue) - if (this.valueMap.has(normalizedValue)) { - const keySet = this.valueMap.get(normalizedValue)! + this.removeFromBucket(key, normalizedValue) + this.removeRangeValue(indexedValue) + + this.indexedKeys.delete(key) + } + + private removeFromBucket(key: TKey, normalizedValue: unknown): void { + const keySet = this.valueMap.get(normalizedValue) + if (keySet) { keySet.delete(key) if (keySet.size === 0) { // No more keys for this value, remove from map and sorted array this.valueMap.delete(normalizedValue) - deleteInSortedArray(this.sortedValues, normalizedValue, this.compareFn) + let sortedIndex = findInsertPositionInArray( + this.sortedValues, + normalizedValue, + this.compareFn, + ) + // Distinct equality keys may share one comparator position. + while ( + sortedIndex < this.sortedValues.length && + this.compareFn(this.sortedValues[sortedIndex], normalizedValue) === 0 + ) { + if ( + areSameValueZeroEqual( + this.sortedValues[sortedIndex], + normalizedValue, + ) + ) { + this.sortedValues.splice(sortedIndex, 1) + break + } + sortedIndex++ + } } } - - this.indexedKeys.delete(key) - this.updateTimestamp() } /** * Updates a value in the index */ update(key: TKey, oldItem: any, newItem: any): void { - this.remove(key, oldItem) - this.add(key, newItem) + let oldIndexedValue: unknown + let newIndexedValue: unknown + try { + oldIndexedValue = this.evaluateIndexExpression(oldItem) + newIndexedValue = this.evaluateIndexExpression(newItem) + } catch { + this.remove(key, oldItem) + this.add(key, newItem) + return + } + + const oldValue = normalizeValue(oldIndexedValue) + const newValue = normalizeValue(newIndexedValue) + if ( + areSameValueZeroEqual(oldValue, newValue) && + this.valueMap.get(newValue)?.has(key) && + this.indexedKeys.has(key) + ) { + this.removeRangeValue(oldIndexedValue) + this.addRangeValue(newIndexedValue) + return + } + + this.removeFromBucket(key, oldValue) + this.removeRangeValue(oldIndexedValue) + this.addToBucket(key, newValue) + this.addRangeValue(newIndexedValue) + this.indexedKeys.add(key) } /** @@ -169,6 +230,7 @@ export class BasicIndex< ) } entriesArray.push({ key, value: normalizeValue(indexedValue) }) + this.addRangeValue(indexedValue) this.indexedKeys.add(key) } @@ -183,8 +245,6 @@ export class BasicIndex< // Build sorted array from unique values this.sortedValues = Array.from(this.valueMap.keys()).sort(this.compareFn) - - this.updateTimestamp() } /** @@ -194,15 +254,13 @@ export class BasicIndex< this.valueMap.clear() this.sortedValues = [] this.indexedKeys.clear() - this.updateTimestamp() + this.clearRangeValues() } /** * Performs a lookup operation */ lookup(operation: IndexOperation, value: any): Set { - const startTime = performance.now() - let result: Set switch (operation) { @@ -227,8 +285,6 @@ export class BasicIndex< default: throw new Error(`Operation ${operation} not supported by BasicIndex`) } - - this.trackLookup(startTime) return result } @@ -260,17 +316,20 @@ export class BasicIndex< const normalizedFrom = normalizeValue(from) const normalizedTo = normalizeValue(to) + const hasFrom = `from` in options + const hasTo = `to` in options // Find start index let startIdx = 0 - if (normalizedFrom !== undefined) { + if (hasFrom) { startIdx = findInsertPositionInArray( this.sortedValues, normalizedFrom, this.compareFn, ) - // If not inclusive and we found exact match, skip it - if ( + // Comparator-equal values form one range boundary even when they are + // distinct equality keys. + while ( !fromInclusive && startIdx < this.sortedValues.length && this.compareFn(this.sortedValues[startIdx], normalizedFrom) === 0 @@ -281,14 +340,14 @@ export class BasicIndex< // Find end index let endIdx = this.sortedValues.length - if (normalizedTo !== undefined) { + if (hasTo) { endIdx = findInsertPositionInArray( this.sortedValues, normalizedTo, this.compareFn, ) - // If inclusive and we found the value, include it - if ( + // Include the whole comparator group at an inclusive upper boundary. + while ( toInclusive && endIdx < this.sortedValues.length && this.compareFn(this.sortedValues[endIdx], normalizedTo) === 0 @@ -308,71 +367,25 @@ export class BasicIndex< return result } - /** - * Performs a reversed range query - */ - rangeQueryReversed(options: RangeQueryOptions = {}): Set { - const { from, to, fromInclusive = true, toInclusive = true } = options - - // Swap from/to and fromInclusive/toInclusive to handle reversed ranges - // If to is undefined, we want to start from the end (max value) - // If from is undefined, we want to end at the beginning (min value) - const swappedFrom = - to ?? - (this.sortedValues.length > 0 - ? this.sortedValues[this.sortedValues.length - 1] - : undefined) - const swappedTo = - from ?? (this.sortedValues.length > 0 ? this.sortedValues[0] : undefined) - - return this.rangeQuery({ - from: swappedFrom, - to: swappedTo, - fromInclusive: toInclusive, - toInclusive: fromInclusive, - }) - } - /** * Returns the next n items in sorted order */ - take(n: number, from?: any, filterFn?: (key: TKey) => boolean): Array { - const result: Array = [] - - let startIdx = 0 - if (from !== undefined) { - const normalizedFrom = normalizeValue(from) - startIdx = findInsertPositionInArray( - this.sortedValues, - normalizedFrom, - this.compareFn, - ) - // Skip past the 'from' value (exclusive) - while ( - startIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[startIdx], normalizedFrom) <= 0 - ) { - startIdx++ - } - } - - for ( - let i = startIdx; - i < this.sortedValues.length && result.length < n; - i++ + take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { + const normalizedFrom = normalizeValue(from) + let startIdx = findInsertPositionInArray( + this.sortedValues, + normalizedFrom, + this.compareFn, + ) + // Skip past the 'from' value (exclusive) + while ( + startIdx < this.sortedValues.length && + this.compareFn(this.sortedValues[startIdx], normalizedFrom) <= 0 ) { - const keys = this.valueMap.get(this.sortedValues[i]) - if (keys) { - for (const key of keys) { - if (result.length >= n) break - if (!filterFn || filterFn(key)) { - result.push(key) - } - } - } + startIdx++ } - return result + return this.takeFromIndex(n, startIdx, 1, filterFn) } /** @@ -380,61 +393,32 @@ export class BasicIndex< */ takeReversed( n: number, - from?: any, + from: any, filterFn?: (key: TKey) => boolean, ): Array { - const result: Array = [] - - let startIdx = this.sortedValues.length - 1 - if (from !== undefined) { - const normalizedFrom = normalizeValue(from) - startIdx = - findInsertPositionInArray( - this.sortedValues, - normalizedFrom, - this.compareFn, - ) - 1 - // Skip past the 'from' value (exclusive) - while ( - startIdx >= 0 && - this.compareFn(this.sortedValues[startIdx], normalizedFrom) >= 0 - ) { - startIdx-- - } - } - - for (let i = startIdx; i >= 0 && result.length < n; i--) { - const keys = this.valueMap.get(this.sortedValues[i]) - if (keys) { - for (const key of keys) { - if (result.length >= n) break - if (!filterFn || filterFn(key)) { - result.push(key) - } - } - } + const normalizedFrom = normalizeValue(from) + let startIdx = + findInsertPositionInArray( + this.sortedValues, + normalizedFrom, + this.compareFn, + ) - 1 + // Skip past the 'from' value (exclusive) + while ( + startIdx >= 0 && + this.compareFn(this.sortedValues[startIdx], normalizedFrom) >= 0 + ) { + startIdx-- } - return result + return this.takeFromIndex(n, startIdx, -1, filterFn) } /** * Returns the first n items in sorted order (from the start) */ takeFromStart(n: number, filterFn?: (key: TKey) => boolean): Array { - const result: Array = [] - for (let i = 0; i < this.sortedValues.length && result.length < n; i++) { - const keys = this.valueMap.get(this.sortedValues[i]) - if (keys) { - for (const key of keys) { - if (result.length >= n) break - if (!filterFn || filterFn(key)) { - result.push(key) - } - } - } - } - return result + return this.takeFromIndex(n, 0, 1, filterFn) } /** @@ -443,21 +427,39 @@ export class BasicIndex< takeReversedFromEnd( n: number, filterFn?: (key: TKey) => boolean, + ): Array { + return this.takeFromIndex(n, this.sortedValues.length - 1, -1, filterFn) + } + + private takeFromIndex( + n: number, + startIndex: number, + step: 1 | -1, + filterFn?: (key: TKey) => boolean, ): Array { const result: Array = [] - for ( - let i = this.sortedValues.length - 1; - i >= 0 && result.length < n; - i-- + let index = startIndex + while ( + index >= 0 && + index < this.sortedValues.length && + result.length < n ) { - const keys = this.valueMap.get(this.sortedValues[i]) - if (keys) { - for (const key of keys) { - if (result.length >= n) break - if (!filterFn || filterFn(key)) { - result.push(key) - } + const groupValue = this.sortedValues[index] + const groupKeys: Array = [] + do { + for (const key of this.valueMap.get(this.sortedValues[index]) ?? []) { + groupKeys.push(key) } + index += step + } while ( + index >= 0 && + index < this.sortedValues.length && + this.compareFn(this.sortedValues[index], groupValue) === 0 + ) + groupKeys.sort(step === 1 ? compareKeys : compareKeysReversed) + for (const key of groupKeys) { + if (filterFn?.(key) ?? true) result.push(key) + if (result.length >= n) break } } return result @@ -479,29 +481,4 @@ export class BasicIndex< return result } - - // Getter methods for testing/compatibility - get indexedKeysSet(): Set { - return this.indexedKeys - } - - get orderedEntriesArray(): Array<[any, Set]> { - return this.sortedValues.map((value) => [ - value, - this.valueMap.get(value) ?? new Set(), - ]) - } - - get orderedEntriesArrayReversed(): Array<[any, Set]> { - const result: Array<[any, Set]> = [] - for (let i = this.sortedValues.length - 1; i >= 0; i--) { - const value = this.sortedValues[i] - result.push([value, this.valueMap.get(value) ?? new Set()]) - } - return result - } - - get valueMapData(): Map> { - return this.valueMap - } } diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index 17608950a7..ecafef5dfe 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -1,8 +1,11 @@ import { compareKeys } from '@tanstack/db-ivm' +import { compareKeysReversed } from '../utils/array-utils.js' import { BTree } from '../utils/btree.js' import { + areSameValueZeroEqual, defaultComparator, denormalizeUndefined, + makeComparator, normalizeForBTree, } from '../utils/comparison.js' import { BaseIndex } from './base-index.js' @@ -28,6 +31,12 @@ export interface RangeQueryOptions { toInclusive?: boolean } +type OrderedBucket = { + representative: unknown + exactValues: Set + keys: Set +} + /** * B+Tree index for sorted data with range queries * This maintains items in sorted order and provides efficient range operations @@ -45,10 +54,13 @@ export class BTreeIndex< ]) // Internal data structures - private to hide implementation details - // The `orderedEntries` B+ tree is used for efficient range queries - // The `valueMap` is used for O(1) lookups of PKs by indexed value - private orderedEntries: BTree // we don't associate values with the keys of the B+ tree (the keys are indexed values) - private valueMap = new Map>() // instead we store a mapping of indexed values to a set of PKs + // The `orderedEntries` B+ tree groups values that occupy the same comparator + // position. The `valueMap` keeps exact values separate for equality lookups. + private orderedEntries: BTree> + private valueMap = new Map< + unknown, + { keys: Set; ordered: OrderedBucket } + >() private indexedKeys = new Set() private compareFn: (a: any, b: any) => number = defaultComparator @@ -60,8 +72,14 @@ export class BTreeIndex< ) { super(id, expression, name, options) + if (options?.compareOptions) { + this.compareOptions = options!.compareOptions + } + // Get the base compare function - const baseCompareFn = options?.compareFn ?? defaultComparator + const baseCompareFn = + options?.compareFn ?? makeComparator(this.compareOptions) + this.hasCustomComparator = options?.compareFn != null // Wrap it to denormalize sentinels before comparison // This ensures UNDEFINED_SENTINEL is converted back to undefined @@ -69,9 +87,6 @@ export class BTreeIndex< this.compareFn = (a: any, b: any) => baseCompareFn(denormalizeUndefined(a), denormalizeUndefined(b)) - if (options?.compareOptions) { - this.compareOptions = options!.compareOptions - } this.orderedEntries = new BTree(this.compareFn) } @@ -93,19 +108,36 @@ export class BTreeIndex< // Normalize the value for Map key usage const normalizedValue = normalizeForBTree(indexedValue) - // Check if this value already exists - if (this.valueMap.has(normalizedValue)) { - // Add to existing set - this.valueMap.get(normalizedValue)!.add(key) - } else { - // Create new set for this value - const keySet = new Set([key]) - this.valueMap.set(normalizedValue, keySet) - this.orderedEntries.set(normalizedValue, undefined) - } + this.addToBucket(key, normalizedValue) + this.addRangeValue(indexedValue) this.indexedKeys.add(key) - this.updateTimestamp() + } + + private addToBucket(key: TKey, normalizedValue: unknown): void { + const exact = this.valueMap.get(normalizedValue) + if (exact) { + exact.keys.add(key) + exact.ordered.keys.add(key) + return + } + + let orderedBucket = this.orderedEntries.get(normalizedValue) + if (orderedBucket) { + orderedBucket.keys.add(key) + orderedBucket.exactValues.add(normalizedValue) + } else { + orderedBucket = { + representative: normalizedValue, + exactValues: new Set([normalizedValue]), + keys: new Set([key]), + } + this.orderedEntries.set(normalizedValue, orderedBucket) + } + this.valueMap.set(normalizedValue, { + keys: new Set([key]), + ordered: orderedBucket, + }) } /** @@ -126,29 +158,65 @@ export class BTreeIndex< // Normalize the value for Map key usage const normalizedValue = normalizeForBTree(indexedValue) - if (this.valueMap.has(normalizedValue)) { - const keySet = this.valueMap.get(normalizedValue)! - keySet.delete(key) + this.removeFromBucket(key, normalizedValue) + this.removeRangeValue(indexedValue) - // If set is now empty, remove the entry entirely - if (keySet.size === 0) { - this.valueMap.delete(normalizedValue) + this.indexedKeys.delete(key) + } - // Remove from ordered entries - this.orderedEntries.delete(normalizedValue) - } + private removeFromBucket(key: TKey, normalizedValue: unknown): void { + const exact = this.valueMap.get(normalizedValue) + if (!exact || !exact.keys.delete(key)) return + const removedExactValue = exact.keys.size === 0 + if (removedExactValue) this.valueMap.delete(normalizedValue) + const orderedBucket = exact.ordered + orderedBucket.keys.delete(key) + if (removedExactValue) orderedBucket.exactValues.delete(normalizedValue) + + if (orderedBucket.keys.size === 0) { + this.orderedEntries.delete(normalizedValue) + } else if ( + removedExactValue && + areSameValueZeroEqual(orderedBucket.representative, normalizedValue) + ) { + this.orderedEntries.delete(normalizedValue) + const representative = orderedBucket.exactValues.values().next().value + orderedBucket.representative = representative + this.orderedEntries.set(representative, orderedBucket) } - - this.indexedKeys.delete(key) - this.updateTimestamp() } /** * Updates a value in the index */ update(key: TKey, oldItem: any, newItem: any): void { - this.remove(key, oldItem) - this.add(key, newItem) + let oldIndexedValue: unknown + let newIndexedValue: unknown + try { + oldIndexedValue = this.evaluateIndexExpression(oldItem) + newIndexedValue = this.evaluateIndexExpression(newItem) + } catch { + this.remove(key, oldItem) + this.add(key, newItem) + return + } + + const oldValue = normalizeForBTree(oldIndexedValue) + const newValue = normalizeForBTree(newIndexedValue) + if ( + areSameValueZeroEqual(oldValue, newValue) && + this.valueMap.get(newValue)?.keys.has(key) + ) { + this.removeRangeValue(oldIndexedValue) + this.addRangeValue(newIndexedValue) + return + } + + this.removeFromBucket(key, oldValue) + this.removeRangeValue(oldIndexedValue) + this.addToBucket(key, newValue) + this.addRangeValue(newIndexedValue) + this.indexedKeys.add(key) } /** @@ -169,15 +237,13 @@ export class BTreeIndex< this.orderedEntries.clear() this.valueMap.clear() this.indexedKeys.clear() - this.updateTimestamp() + this.clearRangeValues() } /** * Performs a lookup operation */ lookup(operation: IndexOperation, value: any): Set { - const startTime = performance.now() - let result: Set switch (operation) { @@ -202,8 +268,6 @@ export class BTreeIndex< default: throw new Error(`Operation ${operation} not supported by BTreeIndex`) } - - this.trackLookup(startTime) return result } @@ -221,7 +285,7 @@ export class BTreeIndex< */ equalityLookup(value: any): Set { const normalizedValue = normalizeForBTree(value) - return new Set(this.valueMap.get(normalizedValue) ?? []) + return new Set(this.valueMap.get(normalizedValue)?.keys ?? []) } /** @@ -246,40 +310,29 @@ export class BTreeIndex< fromKey, toKey, toInclusive, - (indexedValue, _) => { - if (!fromInclusive && this.compareFn(indexedValue, from) === 0) { + (indexedValue, bucket) => { + // Only exclude the boundary when an exclusive lower bound was + // actually provided. Without a `from` bound, `fromKey` defaults to + // the minimum key and must not be dropped. Compare against the + // normalized key since indexed values are stored normalized + // (e.g. dates as timestamps), so the raw `from` would never match. + if ( + hasFrom && + !fromInclusive && + this.compareFn(indexedValue, fromKey) === 0 + ) { // the B+ tree `forRange` method does not support exclusive lower bounds // so we need to exclude it manually return } - const keys = this.valueMap.get(indexedValue) - if (keys) { - keys.forEach((key) => result.add(key)) - } + bucket.keys.forEach((key) => result.add(key)) }, ) return result } - /** - * Performs a reversed range query - */ - rangeQueryReversed(options: RangeQueryOptions = {}): Set { - const { from, to, fromInclusive = true, toInclusive = true } = options - const hasFrom = `from` in options - const hasTo = `to` in options - - // Swap from/to for reversed query, respecting explicit undefined values - return this.rangeQuery({ - from: hasTo ? to : this.orderedEntries.maxKey(), - to: hasFrom ? from : this.orderedEntries.minKey(), - fromInclusive: toInclusive, - toInclusive: fromInclusive, - }) - } - /** * Internal method for taking items from the index. * @param n - The number of items to return @@ -290,32 +343,25 @@ export class BTreeIndex< */ private takeInternal( n: number, - nextPair: (k?: any) => [any, any] | undefined, + nextPair: (k?: any) => [any, OrderedBucket] | undefined, from: any, filterFn?: (key: TKey) => boolean, reversed: boolean = false, ): Array { - const keysInResult: Set = new Set() const result: Array = [] - let pair: [any, any] | undefined + let pair: [any, OrderedBucket] | undefined let key = from // Use as-is - it's already normalized by the caller + // Every key owns exactly one bucket, so the walk never repeats a key. while ((pair = nextPair(key)) !== undefined && result.length < n) { key = pair[0] - const keys = this.valueMap.get(key) as - | Set> - | undefined - if (keys && keys.size > 0) { - // Sort keys for deterministic order, reverse if needed - const sorted = Array.from(keys).sort(compareKeys) - if (reversed) sorted.reverse() - for (const ks of sorted) { - if (result.length >= n) break - if (!keysInResult.has(ks) && (filterFn?.(ks) ?? true)) { - result.push(ks) - keysInResult.add(ks) - } - } + // Sort keys for deterministic order within a comparator position. + const sorted = Array.from(pair[1].keys).sort( + reversed ? compareKeysReversed : compareKeys, + ) + for (const ks of sorted) { + if (result.length >= n) break + if (filterFn?.(ks) ?? true) result.push(ks) } } @@ -387,7 +433,7 @@ export class BTreeIndex< for (const value of values) { const normalizedValue = normalizeForBTree(value) - const keys = this.valueMap.get(normalizedValue) + const keys = this.valueMap.get(normalizedValue)?.keys if (keys) { keys.forEach((key) => result.add(key)) } @@ -395,34 +441,4 @@ export class BTreeIndex< return result } - - // Getter methods for testing compatibility - get indexedKeysSet(): Set { - return this.indexedKeys - } - - get orderedEntriesArray(): Array<[any, Set]> { - return this.orderedEntries - .keysArray() - .map((key) => [ - denormalizeUndefined(key), - this.valueMap.get(key) ?? new Set(), - ]) - } - - get orderedEntriesArrayReversed(): Array<[any, Set]> { - return this.takeReversedFromEnd(this.orderedEntries.size).map((key) => [ - denormalizeUndefined(key), - this.valueMap.get(key) ?? new Set(), - ]) - } - - get valueMapData(): Map> { - // Return a new Map with denormalized keys - const result = new Map>() - for (const [key, value] of this.valueMap) { - result.set(denormalizeUndefined(key), value) - } - return result - } } diff --git a/packages/db/src/indexes/reverse-index.ts b/packages/db/src/indexes/reverse-index.ts index 8999b2801c..3cfda9f2e4 100644 --- a/packages/db/src/indexes/reverse-index.ts +++ b/packages/db/src/indexes/reverse-index.ts @@ -1,11 +1,9 @@ -import type { CompareOptions } from '../query/builder/types' -import type { OrderByDirection } from '../query/ir' -import type { IndexInterface, IndexOperation, IndexStats } from './base-index' +import type { IndexInterface, IndexOperation, IndexReader } from './base-index' import type { RangeQueryOptions } from './btree-index' export class ReverseIndex< TKey extends string | number, -> implements IndexInterface { +> implements IndexReader { private originalIndex: IndexInterface constructor(index: IndexInterface) { @@ -32,10 +30,6 @@ export class ReverseIndex< return this.originalIndex.rangeQueryReversed(options) } - rangeQueryReversed(options: RangeQueryOptions = {}): Set { - return this.originalIndex.rangeQuery(options) - } - take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { return this.originalIndex.takeReversed(n, from, filterFn) } @@ -44,88 +38,21 @@ export class ReverseIndex< return this.originalIndex.takeReversedFromEnd(n, filterFn) } - takeReversed( - n: number, - from: any, - filterFn?: (key: TKey) => boolean, - ): Array { - return this.originalIndex.take(n, from, filterFn) - } - - takeReversedFromEnd( - n: number, - filterFn?: (key: TKey) => boolean, - ): Array { - return this.originalIndex.takeFromStart(n, filterFn) - } - - get orderedEntriesArray(): Array<[any, Set]> { - return this.originalIndex.orderedEntriesArrayReversed - } - - get orderedEntriesArrayReversed(): Array<[any, Set]> { - return this.originalIndex.orderedEntriesArray - } - // All operations below delegate to the original index supports(operation: IndexOperation): boolean { return this.originalIndex.supports(operation) } - matchesField(fieldPath: Array): boolean { - return this.originalIndex.matchesField(fieldPath) - } - - matchesCompareOptions(compareOptions: CompareOptions): boolean { - return this.originalIndex.matchesCompareOptions(compareOptions) - } - - matchesDirection(direction: OrderByDirection): boolean { - return this.originalIndex.matchesDirection(direction) - } - - getStats(): IndexStats { - return this.originalIndex.getStats() + get supportsRangeOptimization(): boolean { + return this.originalIndex.supportsRangeOptimization } - add(key: TKey, item: any): void { - this.originalIndex.add(key, item) - } - - remove(key: TKey, item: any): void { - this.originalIndex.remove(key, item) - } - - update(key: TKey, oldItem: any, newItem: any): void { - this.originalIndex.update(key, oldItem, newItem) - } - - build(entries: Iterable<[TKey, any]>): void { - this.originalIndex.build(entries) - } - - clear(): void { - this.originalIndex.clear() + canOptimizeRangeFor(value: unknown): boolean { + return this.originalIndex.canOptimizeRangeFor?.(value) ?? true } get keyCount(): number { return this.originalIndex.keyCount } - - equalityLookup(value: any): Set { - return this.originalIndex.equalityLookup(value) - } - - inArrayLookup(values: Array): Set { - return this.originalIndex.inArrayLookup(values) - } - - get indexedKeysSet(): Set { - return this.originalIndex.indexedKeysSet - } - - get valueMapData(): Map> { - return this.originalIndex.valueMapData - } } diff --git a/packages/db/src/live-query-adapter.ts b/packages/db/src/live-query-adapter.ts new file mode 100644 index 0000000000..b13001d0a8 --- /dev/null +++ b/packages/db/src/live-query-adapter.ts @@ -0,0 +1,68 @@ +import type { Collection } from './collection/index.js' +import type { CollectionStatus } from './types.js' + +/** + * Shared helpers for the first-party framework adapters (`@tanstack/react-db`, + * `@tanstack/vue-db`, `@tanstack/svelte-db`, `@tanstack/solid-db`, + * `@tanstack/angular-db`). + * + * These centralize small pieces of logic every adapter used to duplicate, so + * they stay consistent across frameworks. They are intended for the official + * adapters; treat them as unstable for external use. + */ + +/** + * Structural check for a live-query/`Collection` instance. + * + * Uses duck typing rather than `instanceof CollectionImpl` on purpose: adapters + * and core can resolve to different copies of `@tanstack/db` (dual-package / + * multi-realm), where `instanceof` gives false negatives. The three methods + * below uniquely identify a Collection. + */ +export function isCollection( + value: unknown, +): value is Collection { + return ( + typeof value === `object` && + value !== null && + typeof (value as any).subscribeChanges === `function` && + typeof (value as any).startSyncImmediate === `function` && + typeof (value as any).id === `string` + ) +} + +/** Whether a collection yields a single result (`findOne`) rather than an array. */ +export function isSingleResultCollection( + collection: Collection, +): boolean { + return ( + (collection.config as { singleResult?: boolean } | undefined) + ?.singleResult === true + ) +} + +/** The derived boolean status flags every adapter exposes for a query. */ +export interface LiveQueryStatusFlags { + isLoading: boolean + isReady: boolean + isIdle: boolean + isError: boolean + isCleanedUp: boolean +} + +/** + * Derive the boolean status flags from a collection status. Adapters represent + * a disabled query separately (with `isReady: true`); this covers the real + * `CollectionStatus` values. + */ +export function getLiveQueryStatusFlags( + status: CollectionStatus, +): LiveQueryStatusFlags { + return { + isLoading: status === `loading`, + isReady: status === `ready`, + isIdle: status === `idle`, + isError: status === `error`, + isCleanedUp: status === `cleaned-up`, + } +} diff --git a/packages/db/src/live-query-observer.ts b/packages/db/src/live-query-observer.ts new file mode 100644 index 0000000000..13d31c9e9e --- /dev/null +++ b/packages/db/src/live-query-observer.ts @@ -0,0 +1,898 @@ +import { LiveQueryObserverDisposedError } from './errors.js' +import { + getLiveQueryStatusFlags, + isSingleResultCollection, +} from './live-query-adapter.js' +import { getBuilderFromConfig } from './query/live/collection-registry.js' +import type { Collection } from './collection/index.js' +import type { DbClient, DehydratedLiveQueryResult } from './client.js' +import type { ChangeMessage, CollectionStatus } from './types.js' + +/** + * The canonical, adapter-agnostic view of a live query at a point in time. + * + * `getSnapshot()` returns a stable object identity that only changes when the + * query changes, so `useSyncExternalStore`-style consumers can compare by + * reference. Each snapshot owns a captured view of `state`/`data`, so reading + * an older snapshot cannot expose rows from a later revision. + */ +export interface LiveQuerySnapshot< + T extends object, + TKey extends string | number, +> { + /** Keyed results, or `undefined` for a disabled query. */ + state: ReadonlyMap | undefined + /** Ordered results (single row for `findOne`), or `undefined` when disabled. */ + data: T | ReadonlyArray | undefined + /** The underlying collection, or `undefined` when disabled. */ + collection: Collection | undefined + /** + * Monotonic counter bumped whenever the visible layout (the ordered key + * sequence) changes — membership, ordering, or an order-only move. Lets + * consumers detect a reorder that changed no row value (which `data`/`state` + * identity alone can't express once row values are structurally shared). + * + * It is NOT in lockstep with snapshot identity: a value-only update produces a + * new snapshot while `layoutRevision` stays put. A `layoutRevision` change + * always accompanies a new snapshot, but not vice versa. + */ + layoutRevision: number + status: CollectionStatus | `disabled` + isLoading: boolean + isReady: boolean + isIdle: boolean + isError: boolean + isCleanedUp: boolean + isEnabled: boolean +} + +/** + * Listener payload: changes, `[]` for an internal layout-only publication, or + * `undefined` for a synthetic status/ready notification. + */ +export type LiveQueryObserverListener< + T extends object, + TKey extends string | number, +> = (changes: Array> | undefined) => void + +/** + * Wraps a resolved live-query `Collection` (or `null` for a disabled query) with + * the shared lifecycle every framework adapter needs: start sync on first + * subscribe, subscribe to changes and status transitions, expose a stable + * snapshot for wholesale consumers, and deliver the raw change set for + * granular consumers. + * + * Input resolution (query fn / config / collection / disabled) stays in the + * adapter — it is framework-reactive. The observer owns everything after the + * input is resolved to a concrete collection. + * + * @internal Unstable contract for TanStack DB's official framework adapters — + * not a public extension point yet; may change in any release. + */ +export interface LiveQueryObserver< + T extends object, + TKey extends string | number, +> { + /** Stable per-revision snapshot for wholesale materialization. */ + getSnapshot: () => LiveQuerySnapshot + /** Stable server snapshot used by useSyncExternalStore-style adapters. */ + getServerSnapshot: () => LiveQuerySnapshot + /** + * Subscribe to changes. The listener receives the change set (or `undefined` + * for the synthetic notify a ready collection emits on attach). Granular + * adapters apply the changes; wholesale adapters can ignore them and re-read + * `getSnapshot()`. Returns an unsubscribe function. + */ + subscribe: (listener: LiveQueryObserverListener) => () => void + /** Resolve once the collection has loaded its first data. */ + preload: () => Promise + /** The transport or preload error for this query, if it has not produced data. */ + getError: () => unknown + /** Capture the ordered query result without serializing its source collections. */ + dehydrate: () => DehydratedLiveQueryResult + /** Idempotent teardown. */ + dispose: () => void +} + +/** + * One logical subscription. Records — not raw callbacks — identify + * subscriptions, so the same listener function can be subscribed twice and + * each subscription tears down independently. + */ +interface SubscriptionRecord { + listener: LiveQueryObserverListener + active: boolean +} + +interface Publication { + changes: Array> | undefined + targets: Array> + entries?: Array<[TKey, T]> + status: CollectionStatus + collectionRevision?: number + collectionLayoutRevision?: number + layoutChanged: boolean +} + +const DISABLED_SNAPSHOT: LiveQuerySnapshot = { + state: undefined, + data: undefined, + collection: undefined, + layoutRevision: 0, + status: `disabled`, + isLoading: false, + isReady: true, + isIdle: false, + isError: false, + isCleanedUp: false, + isEnabled: false, +} + +class LiveQueryObserverImpl< + T extends object, + TKey extends string | number, +> implements LiveQueryObserver { + private readonly collection: Collection | null + private readonly wholesale: boolean + private readonly client: DbClient | undefined + private readonly queryHash: string | undefined + private readonly onPreload: (() => void) | undefined + private visibleStatus: CollectionStatus | undefined + private cachedEntries: Array<[TKey, T]> | undefined + private cachedCollectionRevision: number | undefined + private cachedCollectionLayoutRevision: number | undefined + private snapshotDirty = true + private cachedSnapshot: LiveQuerySnapshot = DISABLED_SNAPSHOT + private layoutRevision = 0 + private lastLayoutKeys: Array | undefined + private deliveredLayoutRevision: number | undefined + private readonly subscriptions = new Set>() + // Publications are dispatched FIFO: an emit that happens while another + // publication is being delivered (a listener mutating the collection + // synchronously) is queued, never delivered reentrantly. + private readonly publicationQueue: Array> = [] + private dispatching = false + private blockDelivery = false + private attached = false + private collectionUnsub: (() => void) | null = null + private unregisterClientResource: (() => void) | undefined + private hydrationSeed: + | { + dehydratedAt: number + entries: Array<[TKey, T]> + } + | undefined + private hydrationError: unknown + private hasHydrationError = false + private liveResultIsAuthoritative = false + private handoffScheduled = false + private preloadPromise: Promise | undefined + private disposed = false + + // Sync activation belongs to the first subscription (attach), so building + // an observer cannot activate collection resources on its own. Server + // request clients still record ownership here because React may render an + // observer without ever subscribing to it. + constructor( + collection: Collection | null, + wholesale: boolean, + client: DbClient | undefined, + queryHash: string | undefined, + onPreload: (() => void) | undefined, + ) { + this.collection = collection + this.wholesale = wholesale + this.client = client + this.queryHash = queryHash + this.onPreload = onPreload + this.registerClientResource() + } + + getSnapshot(): LiveQuerySnapshot { + const collection = this.collection + if (!collection) return DISABLED_SNAPSHOT + + this.syncHydrationState() + if (!this.attached) this.refreshDetachedState(collection) + + if (this.snapshotDirty) { + const entries = this.getVisibleEntries(collection) + const state = new Map(entries) + const data = entries.map(([, value]) => value) + const singleResult = isSingleResultCollection(collection) + const liveStatus = this.visibleStatus ?? collection.status + const status = + this.hasHydrationError || liveStatus === `error` + ? (`error` as const) + : this.hasHydrationSeed() + ? (`ready` as const) + : liveStatus + + // Bump the layout revision when the ordered key sequence changes + // (membership, ordering, or an order-only move). Compare the key sequence + // directly rather than via a serialized signature: a joined-with-separator + // signature can collide when a key value equals the concatenation of + // neighboring keys around the separator. Comparing keys also avoids + // materializing a large string on every rebuild; a new key array is only + // allocated when the layout actually moved. + const prevKeys = this.lastLayoutKeys + let layoutChanged = + prevKeys === undefined || prevKeys.length !== entries.length + if (!layoutChanged) { + for (let i = 0; i < entries.length; i++) { + if (prevKeys![i] !== entries[i]![0]) { + layoutChanged = true + break + } + } + } + if (layoutChanged) { + this.lastLayoutKeys = entries.map(([key]) => key) + this.layoutRevision++ + } + + this.cachedSnapshot = { + state, + data: singleResult ? data[0] : data, + collection, + layoutRevision: this.layoutRevision, + status, + ...getLiveQueryStatusFlags(status), + isEnabled: true, + } + this.snapshotDirty = false + } + return this.cachedSnapshot + } + + getServerSnapshot(): LiveQuerySnapshot { + return this.getSnapshot() + } + + getError(): unknown { + this.syncHydrationState() + return this.hasHydrationError ? this.hydrationError : undefined + } + + dehydrate(): DehydratedLiveQueryResult { + const collection = this.collection + if (!collection) return { rows: [] } + + const entries = this.hasHydrationSeed() + ? this.hydrationSeed!.entries + : this.readEntries(collection).entries + + return { + rows: entries.map(([key, value]) => ({ + key, + value, + })), + } + } + + private hasHydrationSeed(): boolean { + return this.hydrationSeed !== undefined && !this.liveResultIsAuthoritative + } + + private getVisibleEntries( + collection: Collection, + ): Array<[TKey, T]> { + if (this.hasHydrationSeed()) return this.hydrationSeed!.entries + return this.cachedEntries ?? this.captureEntries(collection).entries + } + + private syncHydrationState(): boolean { + if (!this.client || !this.queryHash || this.liveResultIsAuthoritative) { + return false + } + + const query = this.client._getLiveQuery(this.queryHash) + if (!query) return false + + if ( + this.attached && + !this.hydrationSeed && + this.collection?.status === `ready` && + !this.collection.isLoadingSubset + ) { + return this.markLiveResultAuthoritative(query.dehydratedAt) + } + + if (query.status === `error`) { + const changed = + !this.hasHydrationError || this.hydrationError !== query.error + this.hydrationError = query.error + this.hasHydrationError = true + if (changed) this.snapshotDirty = true + return changed + } + + if ( + query.status !== `success` || + !query.snapshot || + (this.hydrationSeed && + this.hydrationSeed.dehydratedAt >= query.dehydratedAt) + ) { + return false + } + + this.hydrationSeed = { + dehydratedAt: query.dehydratedAt, + entries: query.snapshot.rows.map((row) => [ + row.key as TKey, + row.value as T, + ]), + } + this.hydrationError = undefined + this.hasHydrationError = false + this.snapshotDirty = true + return true + } + + private diffEntries( + previous: Array<[TKey, T]>, + next: Array<[TKey, T]>, + ): Array> { + const previousByKey = new Map(previous) + const nextByKey = new Map(next) + const changes: Array> = [] + + for (const [key, value] of previous) { + if (!nextByKey.has(key)) changes.push({ type: `delete`, key, value }) + } + for (const [key, value] of next) { + const previousValue = previousByKey.get(key) + if (previousValue === undefined) { + changes.push({ type: `insert`, key, value }) + } else if (previousValue !== value) { + changes.push({ + type: `update`, + key, + value, + previousValue, + }) + } + } + + return changes + } + + private handoffHydrationSeed(collection: Collection): { + changes: Array> + entries: Array<[TKey, T]> + revision?: number + } { + const previous = this.hydrationSeed?.entries ?? [] + const dehydratedAt = this.hydrationSeed?.dehydratedAt + const { entries, revision } = this.readEntries(collection) + this.hydrationSeed = undefined + this.markLiveResultAuthoritative(dehydratedAt) + this.updateCachedEntries(entries, revision) + this.snapshotDirty = true + return { + changes: this.diffEntries(previous, entries), + entries, + revision, + } + } + + private markLiveResultAuthoritative(dehydratedAt?: number): boolean { + const changed = this.hasHydrationError + this.hydrationError = undefined + this.hasHydrationError = false + this.liveResultIsAuthoritative = true + if (dehydratedAt !== undefined && this.queryHash) { + this.client?._consumeLiveQueryResult(this.queryHash, dehydratedAt) + } + if (changed) this.snapshotDirty = true + return changed + } + + private scheduleHydrationHandoff(): void { + if (this.handoffScheduled) return + this.handoffScheduled = true + + queueMicrotask(() => { + this.handoffScheduled = false + const collection = this.collection + if ( + this.disposed || + !this.attached || + !collection || + !this.hasHydrationSeed() || + collection.status !== `ready` || + collection.isLoadingSubset + ) { + return + } + + const handoff = this.handoffHydrationSeed(collection) + this.emit( + this.wholesale ? undefined : handoff.changes, + undefined, + handoff.entries, + collection.status, + handoff.revision, + this.getCollectionLayoutRevision(collection), + true, + ) + }) + } + + private getCollectionRevision( + collection: Collection, + ): number | undefined { + const revision = (collection as { _stateRevision?: unknown })._stateRevision + return typeof revision === `number` ? revision : undefined + } + + private getCollectionLayoutRevision( + collection: Collection, + ): number | undefined { + const revision = (collection as { _layoutRevision?: unknown }) + ._layoutRevision + return typeof revision === `number` ? revision : undefined + } + + private readEntries(collection: Collection): { + entries: Array<[TKey, T]> + revision?: number + } { + const entries = Array.from(collection.entries()) as Array<[TKey, T]> + const revision = this.getCollectionRevision(collection) + return { entries, revision } + } + + private captureEntries(collection: Collection): { + entries: Array<[TKey, T]> + revision?: number + } { + const { entries, revision } = this.readEntries(collection) + this.updateCachedEntries(entries, revision) + return { entries, revision } + } + + private updateCachedEntries( + entries: Array<[TKey, T]>, + revision: number | undefined, + ): void { + const changed = + revision !== undefined + ? this.cachedEntries === undefined || + revision !== this.cachedCollectionRevision + : !this.entriesEqual(this.cachedEntries, entries) + + this.cachedEntries = entries + this.cachedCollectionRevision = revision + if (changed) this.snapshotDirty = true + } + + private entriesEqual( + left: Array<[TKey, T]> | undefined, + right: Array<[TKey, T]>, + ): boolean { + if (!left || left.length !== right.length) return false + return left.every( + ([key, value], index) => + right[index]![0] === key && right[index]![1] === value, + ) + } + + /** + * While detached there is no delivered-publication clock, so fall back to + * the collection revision. Compatible cross-copy collections that predate + * `_stateRevision` are compared structurally instead. + */ + private refreshDetachedState(collection: Collection): void { + const status = collection.status + const revision = this.getCollectionRevision(collection) + const layoutRevision = this.getCollectionLayoutRevision(collection) + + if (revision !== undefined) { + if ( + this.cachedEntries === undefined || + revision !== this.cachedCollectionRevision || + layoutRevision !== this.cachedCollectionLayoutRevision + ) { + this.captureEntries(collection) + this.cachedCollectionLayoutRevision = layoutRevision + this.snapshotDirty = true + } + } else { + const entries = Array.from(collection.entries()) as Array<[TKey, T]> + this.updateCachedEntries(entries, undefined) + } + + if (this.visibleStatus !== status) { + this.visibleStatus = status + this.snapshotDirty = true + } + } + + subscribe(listener: LiveQueryObserverListener): () => void { + if (this.disposed) throw new LiveQueryObserverDisposedError() + + const record: SubscriptionRecord = { listener, active: true } + this.subscriptions.add(record) + if (this.subscriptions.size === 1) { + this.attach() + } else { + // The initial-state replay only happens on attach, so a granular + // subscriber that arrives while already attached is seeded with the + // current rows — delivered to this subscription alone, without advancing + // the observer's revision (the collection state did not change). + // Wholesale consumers read getSnapshot() instead and need no seed. + if (!this.wholesale) this.seed(record) + } + + return () => { + if (!record.active) return + record.active = false + this.subscriptions.delete(record) + if (this.subscriptions.size === 0) this.detach() + } + } + + /** Deliver the collection's current rows to one late subscription as inserts. */ + private seed(record: SubscriptionRecord): void { + const collection = this.collection + if (!collection) return + + const seedChanges: Array> = [] + for (const [key, value] of this.getVisibleEntries(collection)) { + seedChanges.push({ type: `insert`, key, value }) + } + if (seedChanges.length === 0) return + + this.emit(seedChanges, [record]) + } + + private attach(): void { + const collection = this.collection + if (!collection || this.disposed) return + this.registerClientResource() + this.syncHydrationState() + this.refreshDetachedState(collection) + this.attached = true + this.visibleStatus ??= collection.status + this.deliveredLayoutRevision = this.getCollectionLayoutRevision(collection) + const attachedWithHydrationSeed = this.hasHydrationSeed() + this.blockDelivery = this.wholesale || attachedWithHydrationSeed + + // Sync activation happens inside subscribeChanges (addSubscriber starts + // an idle/cleaned-up collection) — the same startSync path the old + // constructor-time startSyncImmediate() took, but now owned by the first + // committed subscription and observed by the status listener below. + + // Granular consumers subscribe with initial state so they receive the + // current rows as inserts followed by deltas through one consistent + // channel (the collection's per-subscriber change stream requires this to + // align deltas). Wholesale consumers subscribe WITHOUT initial state — + // preserving their pre-observer loading policy: no snapshot request means + // no unfiltered loadSubset({ where: undefined }) against on-demand + // collections. The explicit `false` marks all state as seen so deletes + // still flow through as notifies. + const notify = ( + changes: Array> | undefined, + status: CollectionStatus = collection.status, + explicitLayoutChange = false, + ) => { + if (this.disposed || this.subscriptions.size === 0) return + + if (this.hasHydrationSeed()) { + if (status === `ready`) this.scheduleHydrationHandoff() + if (status !== `error`) return + } + + if ( + status === `ready` && + !collection.isLoadingSubset && + !this.liveResultIsAuthoritative && + this.client && + this.queryHash + ) { + const query = this.client._getLiveQuery(this.queryHash) + this.markLiveResultAuthoritative(query?.dehydratedAt) + } + + const layoutRevision = this.getCollectionLayoutRevision(collection) + let layoutChanged = explicitLayoutChange + if ( + !explicitLayoutChange && + changes !== undefined && + changes.length === 0 + ) { + // Empty ready events predate the explicit layout signal and share its + // empty-array payload. Only forward an empty batch when the collection + // confirms that a new layout-only publication occurred. + if ( + layoutRevision === undefined || + layoutRevision === this.deliveredLayoutRevision + ) { + return + } + layoutChanged = true + } + if (changes !== undefined && layoutRevision !== undefined) { + this.deliveredLayoutRevision = layoutRevision + } + const captured = + changes !== undefined + ? this.readEntries(collection) + : status === `cleaned-up` + ? this.readEntries(collection) + : undefined + this.emit( + changes, + undefined, + captured?.entries, + status, + captured?.revision, + layoutRevision, + layoutChanged, + ) + } + + // Status transitions that carry no change events (loading→ready with no + // rows, error, cleaned-up) are part of the canonical publication path: + // any status change publishes a synthetic notify so consumers re-read the + // snapshot. Unlike onFirstReady, `on` returns a real unsubscribe, so a + // detached attachment leaves nothing behind. + const statusUnsub = collection.on(`status:change`, ({ status }) => + notify(undefined, status), + ) + const subscribeLayoutChanges = ( + collection as Collection & { + _subscribeLayoutChanges?: (listener: () => void) => () => void + } + )._subscribeLayoutChanges + const layoutUnsub = + typeof subscribeLayoutChanges === `function` + ? subscribeLayoutChanges.call(collection, () => + notify([], collection.status, true), + ) + : () => {} + + // `subscribeChanges` delivers the initial state synchronously, so a + // listener can dispose the observer while the collection subscription is + // still being created. Register the release hook up front; if detach() + // ran during that replay (collectionUnsub no longer points at our hook), + // undo the subscription as soon as the call returns. + let subscription: { unsubscribe: () => void } | null = null + const clientUnsub = + this.client && this.queryHash + ? this.client.subscribe((event) => { + if ( + event.type === `liveQueryStreamError` || + event.query.queryHash !== this.queryHash + ) { + return + } + + const previousEntries = this.getVisibleEntries(collection) + if (!this.syncHydrationState()) return + const nextEntries = this.getVisibleEntries(collection) + this.emit( + this.wholesale + ? undefined + : this.diffEntries(previousEntries, nextEntries), + ) + }) + : () => {} + const release = () => { + clientUnsub() + statusUnsub() + layoutUnsub() + subscription?.unsubscribe() + } + this.collectionUnsub = release + subscription = collection.subscribeChanges( + (changes) => notify(changes as Array>), + { includeInitialState: !this.wholesale && !attachedWithHydrationSeed }, + ) + this.blockDelivery = false + if (this.collectionUnsub !== release) { + subscription.unsubscribe() + return + } + if (this.wholesale || attachedWithHydrationSeed) { + // Publications raised while subscribeChanges starts sync are part of the + // subscribe handshake. Apply their final snapshot state now, but suppress + // listener delivery: useSyncExternalStore performs its consistency read + // immediately after subscribe returns. + this.flushPublications(!this.wholesale) + const { entries, revision } = this.readEntries(collection) + this.updateCachedEntries(entries, revision) + } + if (this.hasHydrationSeed()) { + if (!this.wholesale) this.seed(Array.from(this.subscriptions)[0]!) + if (collection.status === `ready`) this.scheduleHydrationHandoff() + } + } + + private detach(): void { + this.collectionUnsub?.() + this.collectionUnsub = null + this.attached = false + this.blockDelivery = false + this.publicationQueue.length = 0 + this.unregisterClientResource?.() + this.unregisterClientResource = undefined + } + + private registerClientResource(): void { + if ( + this.unregisterClientResource || + !this.client?._isSsrServerCleanupEnabled() || + !this.collection || + !getBuilderFromConfig(this.collection.config) + ) { + return + } + + this.unregisterClientResource = this.client._registerLiveQueryResource( + this, + async () => { + const collection = this.collection + this.dispose() + await collection?.cleanup() + }, + ) + } + + private emit( + changes: Array> | undefined, + targets = Array.from(this.subscriptions), + entries?: Array<[TKey, T]>, + status = this.collection?.status ?? `cleaned-up`, + collectionRevision?: number, + collectionLayoutRevision?: number, + layoutChanged = false, + ): void { + this.publicationQueue.push({ + changes, + targets, + entries, + status, + collectionRevision, + collectionLayoutRevision, + layoutChanged, + }) + if (this.dispatching || this.blockDelivery) return + + this.flushPublications() + } + + private flushPublications(deliver = true): void { + if (this.dispatching) return + + let failure: { error: unknown } | undefined + this.dispatching = true + try { + // A dispose() during dispatch empties the queue, ending this loop. + while (this.publicationQueue.length > 0) { + const publication = this.publicationQueue.shift()! + if (publication.entries) { + this.updateCachedEntries( + publication.entries, + publication.collectionRevision, + ) + } + if (publication.collectionLayoutRevision !== undefined) { + this.cachedCollectionLayoutRevision = + publication.collectionLayoutRevision + } + if (publication.layoutChanged) { + this.snapshotDirty = true + } + if (this.visibleStatus !== publication.status) { + this.visibleStatus = publication.status + this.snapshotDirty = true + } + // Targets are captured when the publication is queued: a subscription + // removed mid-delivery still receives the in-flight publication, and + // one added later does not. Late-subscriber seeds use the same queue. + if (deliver) { + for (const subRecord of publication.targets) { + if (this.disposed) break + try { + subRecord.listener(publication.changes) + } catch (error) { + failure ??= { error } + } + } + } + } + } finally { + this.dispatching = false + } + if (failure) throw failure.error + } + + preload(): Promise { + if (this.preloadPromise) return this.preloadPromise + + if (this.client && this.queryHash) { + const query = this.client._getLiveQuery(this.queryHash) + if (query?.status === `pending`) return query.promise + if (query?.status === `success`) return Promise.resolve() + } + + this.registerClientResource() + this.onPreload?.() + const collectionPromise = this.collection?.preload() ?? Promise.resolve() + const preloadPromise = + this.client?._isSsrStreamingEnabled() && this.queryHash + ? this.client._registerLiveQuery( + this.queryHash, + collectionPromise.then(() => this.dehydrate()), + ) + : collectionPromise + this.preloadPromise = preloadPromise + const clearPreload = () => { + if (this.preloadPromise === preloadPromise) { + this.preloadPromise = undefined + } + } + void preloadPromise.then(clearPreload, clearPreload) + return preloadPromise + } + + dispose(): void { + if (this.disposed) return + this.disposed = true + this.detach() + for (const subRecord of this.subscriptions) subRecord.active = false + this.subscriptions.clear() + this.publicationQueue.length = 0 + } +} + +export interface CreateLiveQueryObserverOptions { + /** + * How subscribers consume the observer: + * + * - `granular` (default): subscribers apply the delivered `ChangeMessage[]` + * deltas to their own keyed state (Vue/Svelte/Solid). The observer + * subscribes with initial state and seeds late subscribers, so every + * subscriber converges from deltas alone. + * - `wholesale`: subscribers treat notifications as a wake-up and re-read + * `getSnapshot()` (React/Angular). The observer subscribes WITHOUT initial + * state, preserving those adapters' loading policy — no snapshot request, + * so no unfiltered `loadSubset` against on-demand collections. Nothing is + * delivered synchronously during `subscribe`, which keeps + * `useSyncExternalStore`-style consumers safe by construction. + */ + mode?: `granular` | `wholesale` + /** DbClient cache that owns SSR snapshots for this query identity. */ + client?: DbClient + /** Stable live-query identity used for dehydration and hydration. */ + queryHash?: string + /** Resume framework-deferred query sources before a server preload. */ + onPreload?: () => void +} + +/** + * Create a {@link LiveQueryObserver} for a resolved live-query collection, or a + * disabled observer when `collection` is `null`/`undefined`. + * + * @internal This is an unstable contract shared by TanStack DB's official + * framework adapters. It is exported so the adapter packages can use it, but + * it is not a public extension point yet: its API may change in any release + * without a semver major. + */ +export function createLiveQueryObserver< + T extends object, + TKey extends string | number, +>( + collection: Collection | null | undefined, + options: CreateLiveQueryObserverOptions = {}, +): LiveQueryObserver { + return new LiveQueryObserverImpl( + collection ?? null, + options.mode === `wholesale`, + options.client, + options.queryHash, + options.onPreload, + ) +} diff --git a/packages/db/src/live-query-options.ts b/packages/db/src/live-query-options.ts new file mode 100644 index 0000000000..7439b2f143 --- /dev/null +++ b/packages/db/src/live-query-options.ts @@ -0,0 +1,140 @@ +import { BaseQueryBuilder } from './query/builder/index.js' +import { isCollection } from './live-query-adapter.js' +import { + getStableQueryBuilderHash, + getStableValueHash, +} from './query/ir-stable-identity.js' +import type { CollectionImpl } from './collection/index.js' +import type { CollectionOptionsIdentity } from './collection-options.js' +import type { CollectionOptions, DbClient } from './client.js' +import type { + Context, + InitialQueryBuilder, + LiveQueryCollectionConfig, + QueryBuilder, +} from './query/index.js' + +export type LiveQueryKey = ReadonlyArray + +export type LiveQueryOptions = LiveQueryCollectionConfig & { + queryKey?: LiveQueryKey +} + +export type DeferredLiveQueryCollections = Set< + CollectionImpl +> + +type PreparedLiveQueryConfigInput = Omit< + LiveQueryCollectionConfig, + `query` +> & { + query: + | QueryBuilder + | ((q: InitialQueryBuilder) => QueryBuilder | undefined | null) + queryKey?: LiveQueryKey + client?: DbClient +} + +function createInitialQueryBuilder( + dbClient: DbClient | undefined, + deferredCollections: DeferredLiveQueryCollections, +): InitialQueryBuilder { + return new BaseQueryBuilder( + {}, + dbClient + ? ( + options: CollectionOptionsIdentity< + any, + string | number, + any, + any, + any + >, + ) => { + const collection = dbClient._materializeCollectionForRender( + options as CollectionOptions, + ) as CollectionImpl + if (collection._deferSyncStart()) deferredCollections.add(collection) + return collection + } + : undefined, + ) as InitialQueryBuilder +} + +export function prepareLiveQueryValue( + value: unknown, + dbClient: DbClient | undefined, + deferredCollections: DeferredLiveQueryCollections, +): unknown { + if (typeof value === `function`) { + return prepareLiveQueryValue( + value(createInitialQueryBuilder(dbClient, deferredCollections)), + dbClient, + deferredCollections, + ) + } + + if ( + value && + typeof value === `object` && + !isCollection(value) && + !(value instanceof BaseQueryBuilder) && + `query` in value + ) { + const { + query, + queryKey: _queryKey, + client: _client, + ...config + } = value as PreparedLiveQueryConfigInput + + const preparedQuery = + typeof query === `function` + ? query(createInitialQueryBuilder(dbClient, deferredCollections)) + : query + + if (preparedQuery === undefined || preparedQuery === null) { + return preparedQuery + } + + return { + ...config, + query: preparedQuery, + } + } + + return value +} + +export function getPreparedLiveQueryIdentity(value: unknown): unknown { + if (isCollection(value)) return [`collection`, value.id] + if (value instanceof BaseQueryBuilder) { + return [`query`, getStableQueryBuilderHash(value)] + } + if (value && typeof value === `object` && `query` in value) { + const config = value as LiveQueryCollectionConfig + return [ + `config`, + getPreparedLiveQueryIdentity(config.query), + [`getKey`, config.getKey], + [`schema`, config.schema], + [`singleResult`, config.singleResult === true], + [`defaultStringCollation`, config.defaultStringCollation], + ] + } + if (value === undefined || value === null) return [`disabled`] + return [`value`, value] +} + +export function getLiveQueryHash( + preparedValue: unknown, + queryKey?: LiveQueryKey, +): string { + const identity = queryKey?.length + ? [`queryKey`, queryKey] + : isCollection(preparedValue) + ? [`collection`, preparedValue.id] + : [`derived`, getPreparedLiveQueryIdentity(preparedValue)] + + return getStableValueHash(identity, `queryKey`) +} diff --git a/packages/db/src/live-query-window-controller.ts b/packages/db/src/live-query-window-controller.ts new file mode 100644 index 0000000000..1736eaa901 --- /dev/null +++ b/packages/db/src/live-query-window-controller.ts @@ -0,0 +1,1043 @@ +import { + LiveQueryWindowControllerDisposedError, + SetWindowRequiresOrderByError, +} from './errors.js' +import { + getLiveQueryStatusFlags, + isCollection, + isSingleResultCollection, +} from './live-query-adapter.js' +import { createLiveQueryObserver } from './live-query-observer.js' +import { BaseQueryBuilder } from './query/builder/index.js' +import { deepEquals } from './utils.js' +import type { + LiveQueryObserver, + LiveQuerySnapshot, +} from './live-query-observer.js' +import type { Collection } from './collection/index.js' +import type { CollectionStatus } from './types.js' +import type { + Context, + InitialQueryBuilder, + QueryBuilder, +} from './query/builder/index.js' + +const DEFAULT_PAGE_SIZE = 20 + +export type LiveQueryWindowInputKind = `collection` | `query` + +/** @internal The supported, enabled input forms for infinite-query adapters. */ +export type ResolvedLiveQueryWindowInput = + | { kind: `collection`; collection: Collection } + | { kind: `query`; query: QueryBuilder } + +/** + * Classify an infinite-query input without invoking its query callback. + * Frameworks use this during lifecycle comparison so unchanged React renders + * do not execute the callback again. + * + * @internal This contract is unstable while RFC #1623 is being implemented. + */ +export function getLiveQueryWindowInputKind( + input: unknown, +): LiveQueryWindowInputKind { + if (isCollection(input)) return `collection` + if (typeof input === `function`) return `query` + throw new Error( + `useLiveInfiniteQuery: First argument must be either a pre-created live query collection or a query function. ` + + `Received: ${typeof input}`, + ) +} + +/** + * Resolve a supported infinite-query input and invoke a query callback once. + * A function may resolve to a collection for framework getter compatibility. + * Nullable/disabled and config-object inputs are intentionally not supported. + * + * @internal This contract is unstable while RFC #1623 is being implemented. + */ +export function resolveLiveQueryWindowInput( + input: unknown, +): ResolvedLiveQueryWindowInput { + if (getLiveQueryWindowInputKind(input) === `collection`) { + return { + kind: `collection`, + collection: input as Collection, + } + } + + const value = ( + input as (q: InitialQueryBuilder) => QueryBuilder | unknown + )(new BaseQueryBuilder() as InitialQueryBuilder) + if (isCollection(value)) { + return { kind: `collection`, collection: value } + } + if ( + typeof value !== `object` || + value === null || + typeof (value as { limit?: unknown }).limit !== `function` || + typeof (value as { offset?: unknown }).offset !== `function` + ) { + throw new Error( + `useLiveInfiniteQuery: Query function must return a query builder. ` + + `Disabled null or undefined queries are not supported.`, + ) + } + return { kind: `query`, query: value as QueryBuilder } +} + +/** @internal This contract is unstable while RFC #1623 is being implemented. */ +export function normalizeLiveQueryWindowPageSize( + pageSize: number | undefined, +): number { + if ( + pageSize === undefined || + !Number.isSafeInteger(pageSize) || + pageSize <= 0 || + pageSize >= Number.MAX_SAFE_INTEGER + ) { + return DEFAULT_PAGE_SIZE + } + return pageSize +} + +type WindowResult = true | Promise + +type LiveQueryWindow = { offset: number; limit: number } + +/** @internal Shared adapter view of a collection with an ordered window. */ +export type LiveQueryWindowCollection = Collection & { + utils: { + setWindow: (options: LiveQueryWindow) => WindowResult + getWindow: () => LiveQueryWindow | undefined + } +} + +type WindowTarget = object & { + utils?: { + setWindow?: (options: { offset: number; limit: number }) => WindowResult + getWindow?: () => { offset: number; limit: number } | undefined + } +} + +type PendingWindow = { + generation: number + limit: number + promise: Promise +} + +class WindowCoordinator { + private readonly leases = new Map() + private readonly leaseVersions = new Map() + private baselineWindow: { offset: number; limit: number } | undefined + private retainedWindow: { offset: number; limit: number } | undefined + private shouldCaptureBaseline = true + private appliedLimit: number | undefined + private pending: PendingWindow | undefined + private generation = 0 + private leaseVersion = 0 + + constructor(private readonly target: WindowTarget) {} + + request(lease: symbol, limit: number): WindowResult { + if (this.leases.size === 0) { + const currentWindow = this.target.utils?.getWindow?.() + const retainedWindowChanged = + this.retainedWindow !== undefined && + (currentWindow?.offset !== this.retainedWindow.offset || + currentWindow.limit !== this.retainedWindow.limit) + if (this.shouldCaptureBaseline || retainedWindowChanged) { + this.baselineWindow = currentWindow + this.shouldCaptureBaseline = false + } + this.retainedWindow = undefined + } + const previousLimit = this.leases.get(lease) + const previousVersion = this.leaseVersions.get(lease) + const version = ++this.leaseVersion + this.leases.set(lease, limit) + this.leaseVersions.set(lease, version) + + let result: WindowResult + try { + result = this.applyDesiredWindow() + } catch (error) { + this.rollbackLease(lease, version, previousLimit, previousVersion) + this.appliedLimit = undefined + if (this.leases.size === 0) this.shouldCaptureBaseline = true + throw error + } + + if (result === true) return true + return result.catch(async (error: unknown) => { + if (this.rollbackLease(lease, version, previousLimit, previousVersion)) { + this.generation++ + this.pending = undefined + this.appliedLimit = undefined + try { + if (this.leases.size === 0) { + this.restoreInitialWindow() + } else { + const rollback = this.applyDesiredWindow() + if (rollback !== true) await rollback + } + } catch { + // Preserve the failure from the requested window. + } + } + throw error + }) + } + + getLeaseResult(lease: symbol, minimumLimit: number): WindowResult | false { + const limit = this.leases.get(lease) + if (limit === undefined || limit < minimumLimit) return false + const desiredLimit = this.getDesiredLimit() + // getWindow reports settled state; the current lease may still be loading. + if (this.pending && this.pending.limit === desiredLimit) + return this.pending.promise + const currentWindow = this.target.utils?.getWindow?.() + return ( + currentWindow === undefined || + (currentWindow.offset === 0 && currentWindow.limit === desiredLimit) + ) + } + + hasLeases(): boolean { + return this.leases.size > 0 + } + + release(lease: symbol, restoreWhenEmpty: boolean): void { + if (!this.leases.delete(lease)) return + this.leaseVersions.delete(lease) + + // A pending request may still mutate the physical operator, but it no longer + // establishes the accepted window for the remaining lease set. + this.generation++ + this.pending = undefined + this.appliedLimit = undefined + + if (this.leases.size === 0) { + if (restoreWhenEmpty) { + this.restoreInitialWindow() + } else { + this.retainedWindow = this.target.utils?.getWindow?.() + } + return + } + + try { + const result = this.applyDesiredWindow() + if (result !== true) { + void result.catch(() => { + // Unsubscribe has no async error channel. Leave the physical window + // unaccepted so the next request retries it. + this.appliedLimit = undefined + }) + } + } catch { + // The remaining controller will retry on its next request. + this.appliedLimit = undefined + } + } + + private getDesiredLimit(): number | undefined { + let desired: number | undefined + for (const limit of this.leases.values()) { + desired = desired === undefined ? limit : Math.max(desired, limit) + } + return desired + } + + private rollbackLease( + lease: symbol, + version: number, + previousLimit: number | undefined, + previousVersion: number | undefined, + ): boolean { + if (this.leaseVersions.get(lease) !== version) return false + if (previousLimit === undefined) { + this.leases.delete(lease) + this.leaseVersions.delete(lease) + } else { + this.leases.set(lease, previousLimit) + if (previousVersion === undefined) { + this.leaseVersions.delete(lease) + } else { + this.leaseVersions.set(lease, previousVersion) + } + } + return true + } + + private restoreInitialWindow(): void { + const setWindow = this.target.utils?.setWindow + const baselineWindow = this.baselineWindow + this.retainedWindow = undefined + this.shouldCaptureBaseline = false + if (!baselineWindow || typeof setWindow !== `function`) { + this.shouldCaptureBaseline = true + return + } + const generation = this.generation + const markRestored = () => { + if (generation === this.generation && this.leases.size === 0) { + this.shouldCaptureBaseline = true + } + } + try { + const result = setWindow.call(this.target.utils, baselineWindow) + if (result === true) { + markRestored() + } else { + void result.then(markRestored, () => { + // Keep the original baseline so a later release can retry it. + }) + } + } catch { + // Release has no error channel. Keep the baseline for a later retry. + } + } + + private applyDesiredWindow(): WindowResult { + const limit = this.getDesiredLimit() + if (limit === undefined) return true + if (this.pending?.limit === limit) return this.pending.promise + if (this.pending) { + // `setWindow` mutates the physical operator before its load promise + // settles. A different desired window must therefore be applied again, + // even when it matches the last settled limit. + this.generation++ + this.pending = undefined + this.appliedLimit = undefined + } + const currentWindow = this.target.utils?.getWindow?.() + if ( + limit === this.appliedLimit && + currentWindow?.offset === 0 && + currentWindow.limit === limit + ) { + return true + } + + const setWindow = this.target.utils?.setWindow + if (typeof setWindow !== `function`) { + throw new SetWindowRequiresOrderByError() + } + + const generation = ++this.generation + const result = setWindow.call(this.target.utils, { offset: 0, limit }) + if (result === true) { + if (generation === this.generation && this.getDesiredLimit() === limit) { + this.appliedLimit = limit + } + return true + } + + const promise = result.then( + () => { + if ( + generation === this.generation && + this.getDesiredLimit() === limit + ) { + this.appliedLimit = limit + } + if (this.pending?.generation === generation) { + this.pending = undefined + } + }, + (error: unknown) => { + if (this.pending?.generation === generation) { + this.pending = undefined + } + throw error + }, + ) + this.pending = { generation, limit, promise } + return promise + } +} + +const windowCoordinators = new WeakMap() + +function getWindowCoordinator(target: WindowTarget): WindowCoordinator { + let coordinator = windowCoordinators.get(target) + if (!coordinator) { + coordinator = new WindowCoordinator(target) + windowCoordinators.set(target, coordinator) + } + return coordinator +} + +/** @internal Whether an infinite-query controller currently owns this window. */ +export function hasLiveQueryWindowLeases(target: object): boolean { + return windowCoordinators.get(target)?.hasLeases() ?? false +} + +/** @internal Shared validation for infinite-query adapters. */ +export function assertLiveQueryWindowManyResult( + collection: Collection, +): void { + if (isSingleResultCollection(collection)) { + throw new Error( + `useLiveInfiniteQuery: Infinite queries do not support single-result queries. Remove .findOne().`, + ) + } +} + +/** @internal Whether a collection exposes an active ordered window. */ +export function isLiveQueryWindowCollection( + collection: Collection, +): collection is LiveQueryWindowCollection { + return ( + typeof collection.utils?.setWindow === `function` && + collection.utils.getWindow?.() !== undefined + ) +} + +/** + * Validate a pre-created infinite-query collection and describe any window + * adjustment the adapter should warn about. + * + * @internal Shared validation for infinite-query adapters. + */ +export function getLiveQueryWindowCollectionWarning( + collection: Collection, + expectedLimit: number, +): string | undefined { + assertLiveQueryWindowManyResult(collection) + if (!isLiveQueryWindowCollection(collection)) { + throw new Error( + `useLiveInfiniteQuery: Pre-created live query collection must have an ORDER BY (orderBy) clause for infinite pagination to work. ` + + `Please add .orderBy() to your createLiveQueryCollection query.`, + ) + } + + const currentWindow = collection.utils.getWindow() + if ( + !currentWindow || + hasLiveQueryWindowLeases(collection) || + (currentWindow.offset === 0 && currentWindow.limit === expectedLimit) + ) { + return undefined + } + + return ( + `useLiveInfiniteQuery: Pre-created collection has window {offset: ${currentWindow.offset}, limit: ${currentWindow.limit}} ` + + `but the hook expects {offset: 0, limit: ${expectedLimit}}. Adjusting window now.` + ) +} + +/** @internal Compare adapter dependencies by identity and structure. */ +export function compareLiveQueryWindowDependencies( + previous: ReadonlyArray | null | undefined, + current: ReadonlyArray, +): { changed: boolean; structurallyEqual: boolean } { + const changed = + previous === null || + previous === undefined || + previous.length !== current.length || + previous.some((dependency, index) => dependency !== current[index]) + return { + changed, + structurallyEqual: + previous !== null && + previous !== undefined && + deepEquals(previous, current), + } +} + +/** @internal Shared page-depth preservation policy for framework adapters. */ +export function shouldPreserveLiveQueryWindowPageCount(options: { + hasPreviousController: boolean + previousInputKind: `collection` | `query` | undefined + inputKind: `collection` | `query` + sameCollection: boolean + dependenciesChanged: boolean + dependenciesStructurallyEqual: boolean + pageShapeChanged: boolean +}): boolean { + if ( + !options.hasPreviousController || + options.previousInputKind !== options.inputKind + ) { + return false + } + if (options.inputKind === `collection`) return options.sameCollection + return options.dependenciesChanged + ? options.dependenciesStructurallyEqual + : options.pageShapeChanged +} + +/** + * A page-windowed view of a live query at a point in time. + * + * @internal This contract is unstable while RFC #1623 is being implemented. + */ +export interface LiveQueryWindowSnapshot< + T extends object, + TKey extends string | number, +> { + /** Rows across all committed pages, with the peek-ahead row removed. */ + data: ReadonlyArray + /** Rows grouped into committed pages of `pageSize`. */ + pages: ReadonlyArray> + /** `initialPageParam + i` for each committed page. */ + pageParams: ReadonlyArray + hasNextPage: boolean + isFetchingNextPage: boolean + /** The last pagination failure, cleared when a retry begins. */ + error: unknown + /** Keyed results for the physical window, or `undefined` when disabled. */ + state: ReadonlyMap | undefined + collection: Collection | undefined + status: CollectionStatus | `disabled` + isLoading: boolean + isReady: boolean + isIdle: boolean + isError: boolean + isCleanedUp: boolean + isEnabled: boolean +} + +/** @internal This contract is unstable while RFC #1623 is being implemented. */ +export interface CreateLiveQueryWindowControllerOptions { + /** Rows per page (default 20). Invalid values use the default. */ + pageSize?: number + /** Value of the first page's `pageParam` (default 0). */ + initialPageParam?: number + /** Committed pages to preserve when a framework binding changes page shape. */ + initialPageCount?: number +} + +/** @internal This contract is unstable while RFC #1623 is being implemented. */ +export interface LiveQueryWindowController< + T extends object, + TKey extends string | number, +> { + getSnapshot: () => LiveQueryWindowSnapshot + subscribe: (listener: () => void) => () => void + /** Load one more page, resolving only after that page is committed. */ + fetchNextPage: () => Promise + /** Reset to the first page, resolving after the smaller window is accepted. */ + reset: () => Promise + preload: () => Promise + dispose: () => void +} + +/** + * Run an adapter-facing page fetch. The controller records failures in its + * snapshot; consuming the rejection here keeps event handlers safe while the + * returned promise still settles with the request. + * + * @internal This contract is unstable while RFC #1623 is being implemented. + */ +export function fetchNextLiveQueryWindowPage( + controller: Pick< + LiveQueryWindowController, + `fetchNextPage` + >, +): Promise { + return controller.fetchNextPage().catch(() => {}) +} + +interface CachedFrom { + observerSnapshot: unknown + committedPageCount: number + isFetchingNextPage: boolean + hasPaginationError: boolean + paginationError: unknown + failedHasNextPage: boolean +} + +interface SubscriptionRecord { + listener: () => void + active: boolean +} + +interface Publication { + targets: Array +} + +class LiveQueryWindowControllerImpl< + T extends object, + TKey extends string | number, +> implements LiveQueryWindowController { + private readonly observer: LiveQueryObserver + private readonly collection: Collection | null + private readonly coordinator: WindowCoordinator | null + private readonly lease = Symbol(`liveQueryWindowLease`) + private readonly pageSize: number + private readonly initialPageParam: number + + private committedPageCount: number + private isFetchingNextPage = false + private hasPaginationError = false + private paginationError: unknown + private failedHasNextPage = false + private activeFetchPromise: Promise | null = null + private windowGeneration = 0 + private pendingWindowGeneration: number | undefined + private leaseActive = false + private leaseGeneration = 0 + private inFlightLeaseHolders = 0 + private restoreInitialWindowOnRelease = false + + private readonly subscriptions = new Set() + private readonly publicationQueue: Array = [] + private dispatching = false + private blockDelivery = false + private transitionDepth = 0 + private transitionNeedsNotify = false + private observerUnsub: (() => void) | null = null + private cachedSnapshot: LiveQueryWindowSnapshot | null = null + private cachedFrom: CachedFrom | null = null + private disposed = false + + constructor( + collection: Collection | null, + options: CreateLiveQueryWindowControllerOptions, + ) { + this.collection = collection + this.coordinator = collection + ? getWindowCoordinator(collection as unknown as WindowTarget) + : null + this.pageSize = normalizeLiveQueryWindowPageSize(options.pageSize) + this.initialPageParam = options.initialPageParam ?? 0 + const initialPageCount = Math.floor(options.initialPageCount ?? 1) + this.committedPageCount = Number.isFinite(initialPageCount) + ? Math.max(1, initialPageCount) + : 1 + // The controller listener carries no delta payload, so wholesale is the + // only coherent observer contract and guarantees non-reentrant subscribe. + this.observer = createLiveQueryObserver(collection, { + mode: `wholesale`, + }) + } + + getSnapshot(): LiveQueryWindowSnapshot { + const observerSnapshot = this.observer.getSnapshot() + const cached = this.cachedSnapshot + if ( + cached && + this.cachedFrom && + this.cachedFrom.observerSnapshot === observerSnapshot && + this.cachedFrom.committedPageCount === this.committedPageCount && + this.cachedFrom.isFetchingNextPage === this.isFetchingNextPage && + this.cachedFrom.hasPaginationError === this.hasPaginationError && + this.cachedFrom.paginationError === this.paginationError && + this.cachedFrom.failedHasNextPage === this.failedHasNextPage + ) { + return cached + } + + const enabled = observerSnapshot.isEnabled + const rows = + enabled && Array.isArray(observerSnapshot.data) + ? (observerSnapshot.data as ReadonlyArray) + : [] + const totalRequested = this.committedPageCount * this.pageSize + const computedHasNextPage = enabled && rows.length > totalRequested + const hasNextPage = this.hasPaginationError + ? this.failedHasNextPage + : computedHasNextPage + + const pageCount = enabled ? this.committedPageCount : 0 + const pages: Array> = [] + const pageParams: Array = [] + for (let i = 0; i < pageCount; i++) { + pages.push(rows.slice(i * this.pageSize, (i + 1) * this.pageSize)) + pageParams.push(this.initialPageParam + i) + } + + const status = this.hasPaginationError ? `error` : observerSnapshot.status + const statusFlags = this.hasPaginationError + ? getLiveQueryStatusFlags(`error`) + : observerSnapshot + this.cachedSnapshot = { + data: rows.slice(0, totalRequested), + pages, + pageParams, + hasNextPage, + isFetchingNextPage: this.isFetchingNextPage, + error: this.hasPaginationError ? this.paginationError : undefined, + state: observerSnapshot.state, + collection: observerSnapshot.collection, + status, + isLoading: statusFlags.isLoading, + isReady: statusFlags.isReady, + isIdle: statusFlags.isIdle, + isError: statusFlags.isError, + isCleanedUp: observerSnapshot.isCleanedUp, + isEnabled: observerSnapshot.isEnabled, + } + this.cachedFrom = { + observerSnapshot, + committedPageCount: this.committedPageCount, + isFetchingNextPage: this.isFetchingNextPage, + hasPaginationError: this.hasPaginationError, + paginationError: this.paginationError, + failedHasNextPage: this.failedHasNextPage, + } + return this.cachedSnapshot + } + + subscribe(listener: () => void): () => void { + if (this.disposed) throw new LiveQueryWindowControllerDisposedError() + + const record: SubscriptionRecord = { listener, active: true } + this.subscriptions.add(record) + if (this.subscriptions.size === 1) { + this.restoreInitialWindowOnRelease = false + this.blockDelivery = true + let observerUnsub: (() => void) | null = null + try { + // Store the desired physical window before observer activation can + // compile or restart the live-query pipeline. + const windowResult = this.ensureLeaseActive(this.committedPageCount) + const leaseGeneration = this.leaseGeneration + observerUnsub = this.observer.subscribe(() => this.onObserverNotify()) + this.observerUnsub = observerUnsub + if (windowResult !== true) { + this.trackAttachmentFailure(windowResult, leaseGeneration) + } + } catch (error) { + observerUnsub?.() + this.observerUnsub = null + this.deactivateLease(true) + record.active = false + this.subscriptions.delete(record) + throw error + } finally { + this.blockDelivery = false + } + } + + return () => { + if (!record.active) return + record.active = false + this.subscriptions.delete(record) + if (this.subscriptions.size === 0) { + this.restoreInitialWindowOnRelease = true + this.observerUnsub?.() + this.observerUnsub = null + if (this.inFlightLeaseHolders === 0) this.deactivateLease(true) + } + } + } + + fetchNextPage(): Promise { + if (this.disposed) return Promise.resolve() + if (this.activeFetchPromise) { + return this.activeFetchPromise + } + const snapshot = this.getSnapshot() + const awaitingInitialLoad = snapshot.isLoading || snapshot.isIdle + if (!snapshot.hasNextPage && !awaitingInitialLoad) return Promise.resolve() + + let resolveFetch!: () => void + let rejectFetch!: (error: unknown) => void + const activeFetchPromise = new Promise((resolve, reject) => { + resolveFetch = resolve + rejectFetch = reject + }) + this.activeFetchPromise = activeFetchPromise + + let request: Promise + try { + const generation = this.windowGeneration + // An unpublished initial snapshot cannot establish that there is no next + // page. Keep one fetch pending, then decide from the settled first page. + request = awaitingInitialLoad + ? this.preload().then(() => { + if ( + this.disposed || + generation !== this.windowGeneration || + !this.getSnapshot().hasNextPage + ) { + return + } + return this.requestPageCount(this.committedPageCount + 1, true) + }) + : this.requestPageCount(this.committedPageCount + 1, true) + } catch (error) { + this.activeFetchPromise = null + rejectFetch(error) + return activeFetchPromise + } + void request.then( + () => { + if (this.activeFetchPromise === activeFetchPromise) { + this.activeFetchPromise = null + } + resolveFetch() + }, + (error: unknown) => { + if (this.activeFetchPromise === activeFetchPromise) { + this.activeFetchPromise = null + } + rejectFetch(error) + }, + ) + return activeFetchPromise + } + + reset(): Promise { + if (this.disposed) return Promise.resolve() + if ( + this.committedPageCount === 1 && + !this.hasPaginationError && + !this.isFetchingNextPage && + !this.activeFetchPromise && + this.pendingWindowGeneration === undefined + ) { + return Promise.resolve() + } + this.activeFetchPromise = null + return this.requestPageCount(1, false) + } + + async preload(): Promise { + if (this.disposed) throw new LiveQueryWindowControllerDisposedError() + + const hadPaginationError = this.hasPaginationError + this.hasPaginationError = false + this.paginationError = undefined + this.acquireInFlightLease() + try { + const result = this.ensureLeaseActive(this.committedPageCount) + if (result !== true) await result + await this.observer.preload() + this.failedHasNextPage = false + if (hadPaginationError) this.notify() + } catch (error) { + this.hasPaginationError = true + this.paginationError = error + this.failedHasNextPage = this.getComputedHasNextPage() + this.notify() + throw error + } finally { + this.releaseInFlightLease() + } + } + + dispose(): void { + if (this.disposed) return + this.disposed = true + this.windowGeneration++ + this.pendingWindowGeneration = undefined + this.observerUnsub?.() + this.observerUnsub = null + this.deactivateLease(true) + this.observer.dispose() + for (const record of this.subscriptions) record.active = false + this.subscriptions.clear() + this.publicationQueue.length = 0 + } + + private requestPageCount( + requestedPageCount: number, + fetchingNextPage: boolean, + ): Promise { + const generation = ++this.windowGeneration + const previousHasNextPage = this.getSnapshot().hasNextPage + this.pendingWindowGeneration = undefined + this.acquireInFlightLease() + + this.beginTransition() + this.isFetchingNextPage = fetchingNextPage + this.hasPaginationError = false + this.paginationError = undefined + if (fetchingNextPage) this.notify() + + let result: WindowResult + try { + result = this.activateLease(requestedPageCount) + } catch (error) { + this.isFetchingNextPage = false + this.hasPaginationError = true + this.paginationError = error + this.failedHasNextPage = previousHasNextPage + this.notify() + this.endTransition() + this.releaseInFlightLease() + return Promise.reject(error) + } + + if (result === true) { + if (!this.disposed && generation === this.windowGeneration) { + this.committedPageCount = requestedPageCount + this.isFetchingNextPage = false + this.failedHasNextPage = false + this.notify() + } + this.endTransition() + this.releaseInFlightLease() + return Promise.resolve() + } + + this.pendingWindowGeneration = generation + this.endTransition() + + return result + .then( + () => { + if (this.disposed || generation !== this.windowGeneration) return + this.beginTransition() + this.pendingWindowGeneration = undefined + this.committedPageCount = requestedPageCount + this.isFetchingNextPage = false + this.failedHasNextPage = false + this.notify() + this.endTransition() + }, + (error: unknown) => { + if (!this.disposed && generation === this.windowGeneration) { + this.beginTransition() + this.pendingWindowGeneration = undefined + this.isFetchingNextPage = false + this.hasPaginationError = true + this.paginationError = error + this.failedHasNextPage = previousHasNextPage + this.notify() + this.endTransition() + } + throw error + }, + ) + .finally(() => { + this.releaseInFlightLease() + }) + } + + private acquireInFlightLease(): void { + this.inFlightLeaseHolders++ + } + + private releaseInFlightLease(): void { + this.inFlightLeaseHolders-- + if (this.inFlightLeaseHolders === 0 && this.subscriptions.size === 0) { + this.deactivateLease(this.restoreInitialWindowOnRelease) + } + } + + private activateLease(pageCount: number): WindowResult { + this.leaseGeneration++ + if (!this.coordinator || !this.collection) return true + this.leaseActive = true + return this.coordinator.request(this.lease, pageCount * this.pageSize + 1) + } + + private ensureLeaseActive(pageCount: number): WindowResult { + const minimumLimit = pageCount * this.pageSize + 1 + if (this.leaseActive) { + const result = this.coordinator?.getLeaseResult(this.lease, minimumLimit) + if (result) return result + } + return this.activateLease(pageCount) + } + + private deactivateLease(restoreWhenEmpty = false): void { + if (!this.leaseActive || !this.coordinator) return + this.leaseGeneration++ + this.leaseActive = false + this.restoreInitialWindowOnRelease = false + this.coordinator.release(this.lease, restoreWhenEmpty) + } + + private trackAttachmentFailure( + result: Promise, + leaseGeneration: number, + ): void { + void result.catch((error: unknown) => { + if ( + this.disposed || + !this.leaseActive || + leaseGeneration !== this.leaseGeneration + ) { + return + } + this.beginTransition() + this.hasPaginationError = true + this.paginationError = error + this.failedHasNextPage = this.getComputedHasNextPage() + this.notify() + this.endTransition() + }) + } + + private getComputedHasNextPage(): boolean { + const snapshot: LiveQuerySnapshot = this.observer.getSnapshot() + return ( + snapshot.isEnabled && + Array.isArray(snapshot.data) && + snapshot.data.length > this.committedPageCount * this.pageSize + ) + } + + private onObserverNotify(): void { + this.notify() + } + + private beginTransition(): void { + this.transitionDepth++ + } + + private endTransition(): void { + this.transitionDepth-- + if (this.transitionDepth === 0 && this.transitionNeedsNotify) { + this.transitionNeedsNotify = false + this.publish() + } + } + + private notify(): void { + if (this.transitionDepth > 0) { + this.transitionNeedsNotify = true + return + } + this.publish() + } + + private publish(): void { + if (this.disposed || this.blockDelivery || this.subscriptions.size === 0) { + return + } + + this.publicationQueue.push({ targets: [...this.subscriptions] }) + if (this.dispatching) return + + this.dispatching = true + try { + while (this.publicationQueue.length > 0) { + const publication = this.publicationQueue.shift()! + for (const record of publication.targets) { + if (this.hasBeenDisposed()) return + if (!record.active) continue + record.listener() + } + } + } finally { + this.dispatching = false + } + } + + private hasBeenDisposed(): boolean { + return this.disposed + } +} + +/** + * Create an internal forward-window controller for an ordered live query. + * + * @internal This factory is unstable while RFC #1623 is being implemented. + */ +export function createLiveQueryWindowController< + T extends object, + TKey extends string | number, +>( + collection: Collection | null | undefined, + options: CreateLiveQueryWindowControllerOptions = {}, +): LiveQueryWindowController { + return new LiveQueryWindowControllerImpl(collection ?? null, options) +} diff --git a/packages/db/src/local-only.ts b/packages/db/src/local-only.ts index d3a0a7f2ca..911b0cb924 100644 --- a/packages/db/src/local-only.ts +++ b/packages/db/src/local-only.ts @@ -1,3 +1,5 @@ +import { safeRandomUUID } from './utils/uuid' +import { withCollectionConfigFactory } from './client.js' import type { BaseCollectionConfig, CollectionConfig, @@ -182,7 +184,7 @@ export function localOnlyCollectionOptions< const { initialData, onInsert, onUpdate, onDelete, id, ...restConfig } = config - const collectionId = id ?? crypto.randomUUID() + const collectionId = id ?? safeRandomUUID() // Create the sync configuration with transaction confirmation capability const syncResult = createLocalOnlySync(initialData) @@ -263,7 +265,7 @@ export function localOnlyCollectionOptions< ) } - return { + const options = { ...restConfig, id: collectionId, sync: syncResult.sync, @@ -278,6 +280,17 @@ export function localOnlyCollectionOptions< } as LocalOnlyCollectionOptionsResult & { schema?: StandardSchemaV1 } + + return withCollectionConfigFactory(options, () => + ( + localOnlyCollectionOptions as ( + nextConfig: LocalOnlyCollectionConfig, + ) => typeof options + )({ + ...config, + id: collectionId, + }), + ) } /** diff --git a/packages/db/src/local-storage.ts b/packages/db/src/local-storage.ts index 3060b7ec61..be8e326a7f 100644 --- a/packages/db/src/local-storage.ts +++ b/packages/db/src/local-storage.ts @@ -1,3 +1,5 @@ +import { safeRandomUUID } from './utils/uuid' +import { withCollectionConfigFactory } from './client.js' import { InvalidStorageDataFormatError, InvalidStorageObjectFormatError, @@ -149,7 +151,7 @@ function validateJsonSerializable( * @returns A unique identifier string for tracking data versions */ function generateUuid(): string { - return crypto.randomUUID() + return safeRandomUUID() } /** @@ -431,6 +433,26 @@ export function localStorageCollectionOptions( return data ? new Blob([data]).size : 0 } + const persistMutations = ( + mutations: Array>>, + ): void => { + const staged = new Map(lastKnownData) + for (const mutation of mutations) { + if (mutation.type === `delete`) staged.delete(mutation.key) + else + staged.set(mutation.key, { + versionKey: generateUuid(), + data: mutation.modified, + }) + } + saveToStorage(staged) + // Sync and storage-event handling share this Map. Promote only after the + // write succeeds, so rejected mutations cannot contaminate a later save. + lastKnownData.clear() + for (const [key, value] of staged) lastKnownData.set(key, value) + sync.confirmOperationsSync(mutations) + } + /* * Create wrapper handlers for direct persistence operations that perform actual storage operations * Wraps the user's onInsert handler to also save changes to localStorage @@ -447,24 +469,7 @@ export function localStorageCollectionOptions( handlerResult = (await config.onInsert(params)) ?? {} } - // Always persist to storage - // Use lastKnownData (in-memory cache) instead of reading from storage - // Add new items with version keys - params.transaction.mutations.forEach((mutation) => { - // Use the engine's pre-computed key for consistency - const storedItem: StoredItem = { - versionKey: generateUuid(), - data: mutation.modified, - } - lastKnownData.set(mutation.key, storedItem) - }) - - // Save to storage - saveToStorage(lastKnownData) - - // Confirm mutations through sync interface (moves from optimistic to synced state) - // without reloading from storage - sync.confirmOperationsSync(params.transaction.mutations) + persistMutations(params.transaction.mutations) return handlerResult } @@ -481,24 +486,7 @@ export function localStorageCollectionOptions( handlerResult = (await config.onUpdate(params)) ?? {} } - // Always persist to storage - // Use lastKnownData (in-memory cache) instead of reading from storage - // Update items with new version keys - params.transaction.mutations.forEach((mutation) => { - // Use the engine's pre-computed key for consistency - const storedItem: StoredItem = { - versionKey: generateUuid(), - data: mutation.modified, - } - lastKnownData.set(mutation.key, storedItem) - }) - - // Save to storage - saveToStorage(lastKnownData) - - // Confirm mutations through sync interface (moves from optimistic to synced state) - // without reloading from storage - sync.confirmOperationsSync(params.transaction.mutations) + persistMutations(params.transaction.mutations) return handlerResult } @@ -510,20 +498,7 @@ export function localStorageCollectionOptions( handlerResult = (await config.onDelete(params)) ?? {} } - // Always persist to storage - // Use lastKnownData (in-memory cache) instead of reading from storage - // Remove items - params.transaction.mutations.forEach((mutation) => { - // Use the engine's pre-computed key for consistency - lastKnownData.delete(mutation.key) - }) - - // Save to storage - saveToStorage(lastKnownData) - - // Confirm mutations through sync interface (moves from optimistic to synced state) - // without reloading from storage - sync.confirmOperationsSync(params.transaction.mutations) + persistMutations(params.transaction.mutations) return handlerResult } @@ -577,36 +552,10 @@ export function localStorageCollectionOptions( } } - // Use lastKnownData (in-memory cache) instead of reading from storage - // Apply each mutation - for (const mutation of collectionMutations) { - // Use the engine's pre-computed key to avoid key derivation issues - switch (mutation.type) { - case `insert`: - case `update`: { - const storedItem: StoredItem> = { - versionKey: generateUuid(), - data: mutation.modified, - } - lastKnownData.set(mutation.key, storedItem) - break - } - case `delete`: { - lastKnownData.delete(mutation.key) - break - } - } - } - - // Save to storage - saveToStorage(lastKnownData) - - // Confirm the mutations in the collection to move them from optimistic to synced state - // This writes them through the sync interface to make them "synced" instead of "optimistic" - sync.confirmOperationsSync(collectionMutations) + persistMutations(collectionMutations) } - return { + const options = { ...restConfig, id: collectionId, sync, @@ -619,6 +568,15 @@ export function localStorageCollectionOptions( acceptMutations, }, } + + return withCollectionConfigFactory( + options, + () => + localStorageCollectionOptions({ + ...config, + id: collectionId, + }) as unknown as typeof options, + ) } /** diff --git a/packages/db/src/proxy.ts b/packages/db/src/proxy.ts index 57723e3cca..cecc2222a6 100644 --- a/packages/db/src/proxy.ts +++ b/packages/db/src/proxy.ts @@ -3,7 +3,15 @@ * and provides a way to retrieve those changes. */ -import { deepEquals, isTemporal } from './utils' +import { deepEquals, deepEqualsInternal, isTemporal } from './utils' + +// Resolve draft handles before calling native Map/Set membership methods. +const draftCopies = new WeakMap() +function unwrapDraft(value: unknown): unknown { + return value !== null && typeof value === `object` + ? (draftCopies.get(value) ?? value) + : value +} /** * Set of array methods that iterate with callbacks and may return elements. @@ -39,11 +47,6 @@ const ARRAY_MODIFYING_METHODS = new Set([ `copyWithin`, ]) -/** - * Set of Map/Set methods that modify the collection in place. - */ -const MAP_SET_MODIFYING_METHODS = new Set([`set`, `delete`, `clear`, `add`]) - /** * Set of Map/Set iterator methods. */ @@ -245,234 +248,67 @@ function createModifyingMethodHandler( } /** - * Creates handlers for Map/Set iterator methods (entries, keys, values, forEach). - * Returns proxied values for iteration to enable change tracking. + * Use the native live iterator, but expose tracked values. Editing an entry + * changes its owned draft copy in place; it must not delete/reinsert a Set slot. */ function createMapSetIteratorHandler( methodName: string, prop: string | symbol, - methodFn: (...args: Array) => unknown, - target: Map | Set, changeTracker: ChangeTracker, + collectionProxy: unknown, memoizedCreateChangeProxy: ( obj: Record, - parent?: { - tracker: ChangeTracker> - prop: string | symbol - }, + parent?: ChangeParent, ) => { proxy: Record }, - markChanged: (tracker: ChangeTracker) => void, ): ((...args: Array) => unknown) | undefined { - const isIteratorMethod = - MAP_SET_ITERATOR_METHODS.has(methodName) || prop === Symbol.iterator - - if (!isIteratorMethod) { + if (!MAP_SET_ITERATOR_METHODS.has(methodName) && prop !== Symbol.iterator) { return undefined } - return function (this: unknown, ...args: Array) { - const result = methodFn.apply(changeTracker.copy_, args) + return (...args) => { + const copy = changeTracker.copy_ as Map | Set + const isMap = copy instanceof Map + if (isMap && methodName === `keys`) return copy.keys() + + const track = (value: unknown) => + isProxiableObject(value) + ? memoizedCreateChangeProxy(value, { + tracker: changeTracker as unknown as ChangeTracker< + Record + >, + prop: ``, + retainIdentity: true, + }).proxy + : value - // For forEach, wrap the callback to track changes if (methodName === `forEach`) { const callback = args[0] - if (typeof callback === `function`) { - const wrappedCallback = function ( - this: unknown, - value: unknown, - key: unknown, - collection: unknown, - ) { - const cbresult = callback.call(this, value, key, collection) - markChanged(changeTracker) - return cbresult - } - return methodFn.apply(target, [wrappedCallback, ...args.slice(1)]) - } + if (typeof callback !== `function`) + throw new TypeError(`forEach callback must be a function`) + return copy.forEach((value, key) => { + const tracked = track(value) + callback.call(args[1], tracked, isMap ? key : tracked, collectionProxy) + }) } - // For iterators (entries, keys, values, Symbol.iterator) - const isValueIterator = - methodName === `entries` || - methodName === `values` || - methodName === Symbol.iterator.toString() || - prop === Symbol.iterator - - if (isValueIterator) { - const originalIterator = result as Iterator - - // For values() iterator on Maps, create a value-to-key mapping - const valueToKeyMap = new Map() - if (methodName === `values` && target instanceof Map) { - for (const [key, mapValue] of ( - changeTracker.copy_ as unknown as Map - ).entries()) { - valueToKeyMap.set(mapValue, key) - } - } - - // For Set iterators, create an original-to-modified mapping - const originalToModifiedMap = new Map() - if (target instanceof Set) { - for (const setValue of ( - changeTracker.copy_ as unknown as Set - ).values()) { - originalToModifiedMap.set(setValue, setValue) + const entries = copy.entries() + const pairs = + methodName === `entries` || (isMap && prop === Symbol.iterator) + return { + next() { + const result = entries.next() + if (result.done) return result + const [key, value] = result.value + const tracked = track(value) + return { + done: false, + value: pairs ? [isMap ? key : tracked, tracked] : tracked, } - } - - // Return a wrapped iterator that proxies values - return { - next() { - const nextResult = originalIterator.next() - - if ( - !nextResult.done && - nextResult.value && - typeof nextResult.value === `object` - ) { - // For entries, the value is a [key, value] pair - if ( - methodName === `entries` && - Array.isArray(nextResult.value) && - nextResult.value.length === 2 - ) { - if ( - nextResult.value[1] && - typeof nextResult.value[1] === `object` - ) { - const mapKey = nextResult.value[0] - const mapParent = { - tracker: changeTracker as unknown as ChangeTracker< - Record - >, - prop: mapKey as string | symbol, - updateMap: (newValue: unknown) => { - if (changeTracker.copy_ instanceof Map) { - ;(changeTracker.copy_ as Map).set( - mapKey, - newValue, - ) - } - }, - } - const { proxy: valueProxy } = memoizedCreateChangeProxy( - nextResult.value[1] as Record, - mapParent as unknown as { - tracker: ChangeTracker> - prop: string | symbol - }, - ) - nextResult.value[1] = valueProxy - } - } else if ( - methodName === `values` || - methodName === Symbol.iterator.toString() || - prop === Symbol.iterator - ) { - // For Map values(), use the key mapping - if (methodName === `values` && target instanceof Map) { - const mapKey = valueToKeyMap.get(nextResult.value) - if (mapKey !== undefined) { - const mapParent = { - tracker: changeTracker as unknown as ChangeTracker< - Record - >, - prop: mapKey as string | symbol, - updateMap: (newValue: unknown) => { - if (changeTracker.copy_ instanceof Map) { - ;(changeTracker.copy_ as Map).set( - mapKey, - newValue, - ) - } - }, - } - const { proxy: valueProxy } = memoizedCreateChangeProxy( - nextResult.value as Record, - mapParent as unknown as { - tracker: ChangeTracker> - prop: string | symbol - }, - ) - nextResult.value = valueProxy - } - } else if (target instanceof Set) { - // For Set, track modifications - const setOriginalValue = nextResult.value - const setParent = { - tracker: changeTracker as unknown as ChangeTracker< - Record - >, - prop: setOriginalValue as unknown as string | symbol, - updateSet: (newValue: unknown) => { - if (changeTracker.copy_ instanceof Set) { - ;(changeTracker.copy_ as Set).delete( - setOriginalValue, - ) - ;(changeTracker.copy_ as Set).add(newValue) - originalToModifiedMap.set(setOriginalValue, newValue) - } - }, - } - const { proxy: valueProxy } = memoizedCreateChangeProxy( - nextResult.value as Record, - setParent as unknown as { - tracker: ChangeTracker> - prop: string | symbol - }, - ) - nextResult.value = valueProxy - } else { - // For other cases, use a symbol placeholder - const tempKey = Symbol(`iterator-value`) - const { proxy: valueProxy } = memoizedCreateChangeProxy( - nextResult.value as Record, - { - tracker: changeTracker as unknown as ChangeTracker< - Record - >, - prop: tempKey, - }, - ) - nextResult.value = valueProxy - } - } - } - - return nextResult - }, - [Symbol.iterator]() { - return this - }, - } + }, + [Symbol.iterator]() { + return this + }, } - - return result - } -} - -/** - * Simple debug utility that only logs when debug mode is enabled - * Set DEBUG to true in localStorage to enable debug logging - */ -function debugLog(...args: Array): void { - // Check if we're in a browser environment - const isBrowser = - typeof window !== `undefined` && typeof localStorage !== `undefined` - - // In browser, check localStorage for debug flag - if (isBrowser && localStorage.getItem(`DEBUG`) === `true`) { - console.log(`[proxy]`, ...args) - } - // In Node.js environment, check for environment variable (though this is primarily for browser) - else if ( - // true - !isBrowser && - typeof process !== `undefined` && - process.env.DEBUG === `true` - ) { - console.log(`[proxy]`, ...args) } } @@ -483,27 +319,20 @@ interface TypedArray { } // Update type for ChangeTracker +interface ChangeParent { + tracker: ChangeTracker> + prop: string | symbol + // Map/Set entries already belong to the parent's private copy. + retainIdentity?: boolean +} + interface ChangeTracker { + valueCopies: WeakMap originalObject: T modified: boolean copy_: T - proxyCount: number assigned_: Record - parent?: - | { - tracker: ChangeTracker> - prop: string | symbol - } - | { - tracker: ChangeTracker> - prop: string | symbol - updateMap: (newValue: unknown) => void - } - | { - tracker: ChangeTracker> - prop: unknown - updateSet: (newValue: unknown) => void - } + parent?: ChangeParent target: T } @@ -514,7 +343,10 @@ interface ChangeTracker { function deepClone( obj: T, visited = new WeakMap(), + detach = false, ): T { + // A draft handle and its underlying copy must share one cycle identity. + obj = unwrapDraft(obj) as T // Handle null and undefined if (obj === null || obj === undefined) { return obj @@ -531,18 +363,29 @@ function deepClone( } if (obj instanceof Date) { - return new Date(obj.getTime()) as unknown as T + const clone = new Date(obj.getTime()) + visited.set(obj, clone) + return clone as T } if (obj instanceof RegExp) { - return new RegExp(obj.source, obj.flags) as unknown as T + const clone = new RegExp(obj.source, obj.flags) + clone.lastIndex = obj.lastIndex + visited.set(obj, clone) + return clone as T + } + + if (obj instanceof URL) { + const clone = new URL(obj.href) + visited.set(obj, clone) + return clone as T } if (Array.isArray(obj)) { const arrayClone = [] as Array visited.set(obj as object, arrayClone) obj.forEach((item, index) => { - arrayClone[index] = deepClone(item, visited) + arrayClone[index] = deepClone(item, visited, detach) }) return arrayClone as unknown as T } @@ -568,7 +411,7 @@ function deepClone( const clone = new Map() as Map visited.set(obj as object, clone) obj.forEach((value, key) => { - clone.set(key, deepClone(value, visited)) + clone.set(key, deepClone(value, visited, detach)) }) return clone as unknown as T } @@ -577,7 +420,7 @@ function deepClone( const clone = new Set() visited.set(obj as object, clone) obj.forEach((value) => { - clone.add(deepClone(value, visited)) + clone.add(deepClone(value, visited, detach)) }) return clone as unknown as T } @@ -589,6 +432,13 @@ function deepClone( return obj } + // Arbitrary instances may carry private/native state we cannot reconstruct. + // Keep them by reference at publication, rather than silently flattening them. + if (detach) { + const prototype = Object.getPrototypeOf(obj) + if (prototype !== Object.prototype && prototype !== null) return obj + } + const clone = {} as Record visited.set(obj as object, clone) @@ -597,6 +447,7 @@ function deepClone( clone[key] = deepClone( (obj as Record)[key], visited, + detach, ) } } @@ -606,18 +457,13 @@ function deepClone( clone[sym] = deepClone( (obj as Record)[sym], visited, + detach, ) } return clone as T } -let count = 0 -function getProxyCount() { - count += 1 - return count -} - /** * Creates a proxy that tracks changes to the target object * @@ -629,10 +475,7 @@ export function createChangeProxy< T extends Record, >( target: T, - parent?: { - tracker: ChangeTracker> - prop: string | symbol - }, + parent?: ChangeParent, ): { proxy: T @@ -644,15 +487,11 @@ export function createChangeProxy< TInner extends Record, >( innerTarget: TInner, - innerParent?: { - tracker: ChangeTracker> - prop: string | symbol - }, + innerParent?: ChangeParent, ): { proxy: TInner getChanges: () => Record } { - debugLog(`Object ID:`, innerTarget.constructor.name) if (changeProxyCache.has(innerTarget)) { return changeProxyCache.get(innerTarget) as { proxy: TInner @@ -669,22 +508,22 @@ export function createChangeProxy< // and handles circular references const proxyCache = new Map() - // Create a change tracker to track changes to the object + // Existing values share one private copy per row. Newly inserted objects + // retain normal references during the callback; the result is detached below. + const valueCopies = + parent?.tracker.valueCopies ?? new WeakMap() const changeTracker: ChangeTracker = { - copy_: deepClone(target), + valueCopies, + copy_: parent + ? ((valueCopies.get(target) ?? target) as T) + : deepClone(target, valueCopies), originalObject: deepClone(target), - proxyCount: getProxyCount(), modified: false, assigned_: {}, parent, target, // Store reference to the target object } - debugLog( - `createChangeProxy called for target`, - target, - changeTracker.proxyCount, - ) // Mark this object and all its ancestors as modified // Also propagate the actual changes up the chain function markChanged(state: ChangeTracker) { @@ -694,16 +533,7 @@ export function createChangeProxy< // Propagate the change up the parent chain if (state.parent) { - debugLog(`propagating change to parent`) - - // Check if this is a special Map parent with updateMap function - if (`updateMap` in state.parent) { - // Use the special updateMap function for Maps - state.parent.updateMap(state.copy_) - } else if (`updateSet` in state.parent) { - // Use the special updateSet function for Sets - state.parent.updateSet(state.copy_) - } else { + if (!state.parent.retainIdentity) { // Update parent's copy with this object's current state state.parent.tracker.copy_[state.parent.prop] = state.copy_ state.parent.tracker.assigned_[state.parent.prop] = true @@ -718,17 +548,22 @@ export function createChangeProxy< function checkIfReverted( state: ChangeTracker>, ): boolean { - debugLog( - `checkIfReverted called with assigned keys:`, - Object.keys(state.assigned_), - ) - + if (state.copy_ instanceof Map || state.copy_ instanceof Set) { + // Compare entry contents: these containers have no assigned properties. + return deepEquals( + Array.from(state.copy_), + Array.from( + state.originalObject as unknown as + | Map + | Set, + ), + ) + } // If there are no assigned properties, object is unchanged if ( Object.keys(state.assigned_).length === 0 && Object.getOwnPropertySymbols(state.assigned_).length === 0 ) { - debugLog(`No assigned properties, returning true`) return true } @@ -739,21 +574,12 @@ export function createChangeProxy< const currentValue = state.copy_[prop] const originalValue = (state.originalObject as any)[prop] - debugLog( - `Checking property ${String(prop)}, current:`, - currentValue, - `original:`, - originalValue, - ) - // If the value is not equal to original, something is still changed if (!deepEquals(currentValue, originalValue)) { - debugLog(`Property ${String(prop)} is different, returning false`) return false } } else if (state.assigned_[prop] === false) { // Property was deleted, so it's different from original - debugLog(`Property ${String(prop)} was deleted, returning false`) return false } } @@ -767,17 +593,14 @@ export function createChangeProxy< // If the value is not equal to original, something is still changed if (!deepEquals(currentValue, originalValue)) { - debugLog(`Symbol property is different, returning false`) return false } } else if (state.assigned_[sym] === false) { // Property was deleted, so it's different from original - debugLog(`Symbol property was deleted, returning false`) return false } } - debugLog(`All properties match original values, returning true`) // All assigned properties match their original values return true } @@ -785,49 +608,36 @@ export function createChangeProxy< // Update parent status based on child changes function checkParentStatus( parentState: ChangeTracker>, - childProp: string | symbol | unknown, ) { - debugLog(`checkParentStatus called for child prop:`, childProp) - // Check if all properties of the parent are reverted const isReverted = checkIfReverted(parentState) - debugLog(`Parent checkIfReverted returned:`, isReverted) if (isReverted) { - debugLog(`Parent is fully reverted, clearing tracking`) // If everything is reverted, clear the tracking parentState.modified = false parentState.assigned_ = {} // Continue up the chain if (parentState.parent) { - debugLog(`Continuing up the parent chain`) - checkParentStatus(parentState.parent.tracker, parentState.parent.prop) + checkParentStatus(parentState.parent.tracker) } } } // Create a proxy for the target object function createObjectProxy(obj: TObj): TObj { - debugLog(`createObjectProxy`, obj) // If we've already created a proxy for this object, return it if (proxyCache.has(obj)) { - debugLog(`proxyCache found match`) return proxyCache.get(obj) as TObj } // Create a proxy for the object const proxy = new Proxy(obj, { - get(ptarget, prop) { - debugLog(`get`, ptarget, prop) + get(ptarget, prop, receiver) { const value = changeTracker.copy_[prop as keyof T] ?? changeTracker.originalObject[prop as keyof T] - const originalValue = changeTracker.originalObject[prop as keyof T] - - debugLog(`value (at top of proxy get)`, value) - // If it's a getter, return the value directly const desc = Object.getOwnPropertyDescriptor(ptarget, prop) if (desc?.get) { @@ -872,7 +682,42 @@ export function createChangeProxy< if (ptarget instanceof Map || ptarget instanceof Set) { const methodName = prop.toString() - if (MAP_SET_MODIFYING_METHODS.has(methodName)) { + const resolveValue = (entry: unknown) => { + const raw = unwrapDraft(entry) + return raw !== null && typeof raw === `object` + ? (valueCopies.get(raw) ?? raw) + : raw + } + + if ( + methodName === `has` || + methodName === `delete` || + methodName === `add` || + methodName === `set` + ) { + return (...args: Array) => { + if (ptarget instanceof Set) args[0] = resolveValue(args[0]) + else if (methodName === `set`) args[1] = resolveValue(args[1]) + const result = value.apply(ptarget, args) + if (methodName !== `has`) markChanged(changeTracker) + return result === ptarget ? receiver : result + } + } + + if (ptarget instanceof Map && methodName === `get`) { + return (key: unknown) => { + const entry = ptarget.get(key) + return isProxiableObject(entry) + ? memoizedCreateChangeProxy(entry, { + tracker: changeTracker, + prop: ``, + retainIdentity: true, + }).proxy + : entry + } + } + + if (methodName === `clear`) { return createModifyingMethodHandler( value, changeTracker, @@ -884,11 +729,9 @@ export function createChangeProxy< const iteratorHandler = createMapSetIteratorHandler( methodName, prop, - value, - ptarget, changeTracker, + receiver, memoizedCreateChangeProxy, - markChanged, ) if (iteratorHandler) { return iteratorHandler @@ -907,7 +750,7 @@ export function createChangeProxy< // Create a proxy for the nested object const { proxy: nestedProxy } = memoizedCreateChangeProxy( - originalValue, + value, nestedParent, ) @@ -922,12 +765,6 @@ export function createChangeProxy< set(_sobj, prop, value) { const currentValue = changeTracker.copy_[prop as keyof T] - debugLog( - `set called for property ${String(prop)}, current:`, - currentValue, - `new:`, - value, - ) // Only track the change if the value is actually different if (!deepEquals(currentValue, value)) { @@ -935,48 +772,31 @@ export function createChangeProxy< // Important: Use the originalObject to get the true original value const originalValue = changeTracker.originalObject[prop as keyof T] const isRevertToOriginal = deepEquals(value, originalValue) - debugLog( - `value:`, - value, - `original:`, - originalValue, - `isRevertToOriginal:`, - isRevertToOriginal, - ) if (isRevertToOriginal) { - debugLog(`Reverting property ${String(prop)} to original value`) // If the value is reverted to its original state, remove it from changes delete changeTracker.assigned_[prop.toString()] // Make sure the copy is updated with the original value - debugLog(`Updating copy with original value for ${String(prop)}`) changeTracker.copy_[prop as keyof T] = deepClone(originalValue) // Check if all properties in this object have been reverted - debugLog(`Checking if all properties reverted`) const allReverted = checkIfReverted(changeTracker) - debugLog(`All reverted:`, allReverted) if (allReverted) { - debugLog(`All properties reverted, clearing tracking`) // If all have been reverted, clear tracking changeTracker.modified = false changeTracker.assigned_ = {} // If we're a nested object, check if the parent needs updating if (parent) { - debugLog(`Updating parent for property:`, parent.prop) - checkParentStatus(parent.tracker, parent.prop) + checkParentStatus(parent.tracker) } } else { // Some properties are still changed - debugLog(`Some properties still changed, keeping modified flag`) changeTracker.modified = true } } else { - debugLog(`Setting new value for property ${String(prop)}`) - // Set the value on the copy changeTracker.copy_[prop as keyof T] = value @@ -984,11 +804,8 @@ export function createChangeProxy< changeTracker.assigned_[prop.toString()] = true // Mark this object and its ancestors as modified - debugLog(`Marking object and ancestors as modified`, changeTracker) markChanged(changeTracker) } - } else { - debugLog(`Value unchanged, not tracking`) } return true @@ -1022,7 +839,6 @@ export function createChangeProxy< }, deleteProperty(dobj, prop) { - debugLog(`deleteProperty`, dobj, prop) const stringProp = typeof prop === `symbol` ? prop.toString() : prop if (stringProp in dobj) { @@ -1068,6 +884,7 @@ export function createChangeProxy< // Cache the proxy proxyCache.set(obj, proxy) + draftCopies.set(proxy, changeTracker.copy_) return proxy } @@ -1081,12 +898,8 @@ export function createChangeProxy< return { proxy, getChanges: () => { - debugLog(`getChanges called, modified:`, changeTracker.modified) - debugLog(changeTracker) - // First, check if the object is still considered modified if (!changeTracker.modified) { - debugLog(`Object not modified, returning empty object`) return {} } @@ -1104,18 +917,33 @@ export function createChangeProxy< } const result: Record = {} + const mayHaveChangedAliases = Object.keys(changeTracker.assigned_).some( + (key) => typeof changeTracker.copy_[key] === `object`, + ) + const pairedRoots = new Map([ + [changeTracker.copy_, changeTracker.originalObject], + ]) // Iterate through keys in keyObj for (const key in changeTracker.copy_) { - // If the key's value is true and the key exists in valueObj + const value: unknown = changeTracker.copy_[key] + const original: unknown = changeTracker.originalObject[key] + // Compare child contents, stopping only at paired root backedges. A + // child's own changes still count even when it also points to this row. if ( - changeTracker.assigned_[key] === true && + (changeTracker.assigned_[key] === true || + (mayHaveChangedAliases && + !deepEqualsInternal( + value instanceof Set ? Array.from(value) : value, + original instanceof Set ? Array.from(original) : original, + pairedRoots, + ))) && key in changeTracker.copy_ ) { result[key] = changeTracker.copy_[key] } } - debugLog(`Returning copy:`, result) + return result as unknown as Record }, } @@ -1157,7 +985,7 @@ export function withChangeTracking( callback(proxy) - return getChanges() + return deepClone(getChanges(), undefined, true) } /** @@ -1176,5 +1004,5 @@ export function withArrayChangeTracking( callback(proxies) - return getChanges() + return deepClone(getChanges(), undefined, true) } diff --git a/packages/db/src/query/builder/functions.ts b/packages/db/src/query/builder/functions.ts index 47e190162a..84c26e5ca4 100644 --- a/packages/db/src/query/builder/functions.ts +++ b/packages/db/src/query/builder/functions.ts @@ -125,31 +125,12 @@ type MapToNumber = T extends string | Array ? null : T -// Helper type for binary numeric operations (combines nullability of both operands) -type BinaryNumericReturnType = - ExtractType extends infer U1 - ? ExtractType extends infer U2 - ? U1 extends number - ? U2 extends number - ? BasicExpression - : U2 extends number | undefined - ? BasicExpression - : U2 extends number | null - ? BasicExpression - : BasicExpression - : U1 extends number | undefined - ? U2 extends number - ? BasicExpression - : U2 extends number | undefined - ? BasicExpression - : BasicExpression - : U1 extends number | null - ? U2 extends number - ? BasicExpression - : BasicExpression - : BasicExpression - : BasicExpression - : BasicExpression +// Helper type for binary numeric operations. +// Runtime coalesces nullish operands to 0 for these operations, so nullable +// operands don't make the result nullable. +type BinaryNumericReturnType = BasicExpression + +type DivideReturnType = BasicExpression // Operators @@ -620,11 +601,41 @@ export function caseWhen(...args: Array): any { export function add( left: T1, right: T2, -): BinaryNumericReturnType { +): BinaryNumericReturnType { return new Func(`add`, [ toExpression(left), toExpression(right), - ]) as BinaryNumericReturnType + ]) as BinaryNumericReturnType +} + +export function subtract( + left: T1, + right: T2, +): BinaryNumericReturnType { + return new Func(`subtract`, [ + toExpression(left), + toExpression(right), + ]) as BinaryNumericReturnType +} + +export function multiply( + left: T1, + right: T2, +): BinaryNumericReturnType { + return new Func(`multiply`, [ + toExpression(left), + toExpression(right), + ]) as BinaryNumericReturnType +} + +export function divide( + left: T1, + right: T2, +): DivideReturnType { + return new Func(`divide`, [ + toExpression(left), + toExpression(right), + ]) as DivideReturnType } // Aggregates @@ -690,6 +701,9 @@ export const operators = [ `concat`, // Numeric functions `add`, + `subtract`, + `multiply`, + `divide`, // Utility functions `coalesce`, `caseWhen`, diff --git a/packages/db/src/query/builder/index.ts b/packages/db/src/query/builder/index.ts index e8f370228f..c1097f9e9b 100644 --- a/packages/db/src/query/builder/index.ts +++ b/packages/db/src/query/builder/index.ts @@ -1,4 +1,5 @@ import { CollectionImpl } from '../../collection/index.js' +import { hasCollectionOptionsBrand } from '../../collection-options.js' import { Aggregate as AggregateExpr, CollectionRef, @@ -22,6 +23,7 @@ import { QueryMustHaveFromClauseError, SubQueryMustHaveFromClauseError, } from '../../errors.js' +import { getQueryIR } from './query-ir.js' import { createRefProxy, createRefProxyWithSelected, @@ -36,6 +38,7 @@ import { } from './functions.js' import type { SourceClauseContext } from '../../errors.js' import type { NamespacedRow, SingleResult } from '../../types.js' +import type { CollectionOptionsIdentity } from '../../collection-options.js' import type { Aggregate, BasicExpression, @@ -75,13 +78,77 @@ import type { const UNION_ALL_SOURCE_CONTEXT = `unionAll clause` satisfies SourceClauseContext +type CollectionResolver = ( + options: CollectionOptionsIdentity, +) => CollectionImpl + +type FnSelectQueryConstructionValue = + | QueryBuilder + | InitialQueryBuilder + | BasicExpression + | Aggregate + | ToArrayWrapper + | ConcatToArrayWrapper + | MaterializeWrapper + | CaseWhenWrapper + +type IsAnyType = 0 extends 1 & T ? true : false + +// Bound recursive inspection so deeply recursive result types do not exceed +// TypeScript's instantiation limit. The runtime check has no depth limit. +type ContainsFnSelectQueryConstructionValue< + T, + TDepth extends ReadonlyArray = [], +> = + IsAnyType extends true + ? false + : T extends FnSelectQueryConstructionValue + ? true + : TDepth[`length`] extends 8 + ? false + : T extends (...args: Array) => any + ? false + : T extends ReadonlyArray + ? ContainsFnSelectQueryConstructionValue< + TItem, + [...TDepth, unknown] + > + : T extends object + ? true extends { + [K in keyof T]-?: ContainsFnSelectQueryConstructionValue< + T[K], + [...TDepth, unknown] + > + }[keyof T] + ? true + : false + : false + +type InvalidFnSelectResult = { + readonly __tanstackDbFnSelectResultError__: `fn.select() cannot return child query builders, query expressions, or query helpers. Use them as direct fields in .select() instead.` +} + +type FnSelectQueryResult = + true extends ContainsFnSelectQueryConstructionValue + ? InvalidFnSelectResult + : QueryBuilder> + export class BaseQueryBuilder { private readonly query: Partial = {} - constructor(query: Partial = {}) { + constructor( + query: Partial = {}, + private readonly resolveCollection?: CollectionResolver, + ) { this.query = { ...query } } + private _clone( + query: Partial, + ): BaseQueryBuilder { + return new BaseQueryBuilder(query, this.resolveCollection) + } + /** * Creates a CollectionRef or QueryRef from a source object * @param source - An object with a single key-value pair @@ -140,6 +207,13 @@ export class BaseQueryBuilder { if (sourceValue instanceof CollectionImpl) { ref = new CollectionRef(sourceValue, alias) + } else if (hasCollectionOptionsBrand(sourceValue)) { + if (!this.resolveCollection) { + throw new Error( + `Cannot use collection descriptor "${alias}" as a query source without a DbClient resolver. In React, wrap your tree in .`, + ) + } + ref = new CollectionRef(this.resolveCollection(sourceValue), alias) } else if (sourceValue instanceof BaseQueryBuilder) { const subQuery = sourceValue._getQuery() if (!(subQuery as Partial).from) { @@ -177,7 +251,7 @@ export class BaseQueryBuilder { ): QueryBuilder> { const [, from] = this._createRefForSource(source, `from clause`) - return new BaseQueryBuilder({ + return this._clone({ ...this.query, from, }) as any @@ -209,7 +283,7 @@ export class BaseQueryBuilder { ...branches: Array> ): QueryBuilder { if (sourceOrBranch instanceof BaseQueryBuilder) { - return new BaseQueryBuilder({ + return this._clone({ ...this.query, from: new UnionAll( [sourceOrBranch, ...branches].map((branch) => @@ -226,7 +300,7 @@ export class BaseQueryBuilder { const from = refs.length === 1 ? refs[0]![1] : new UnionFrom(refs.map((r) => r[1])) - return new BaseQueryBuilder({ + return this._clone({ ...this.query, from, }) as any @@ -308,7 +382,7 @@ export class BaseQueryBuilder { const existingJoins = this.query.join || [] - return new BaseQueryBuilder({ + return this._clone({ ...this.query, join: [...existingJoins, joinClause], }) as any @@ -467,7 +541,7 @@ export class BaseQueryBuilder { const existingWhere = this.query.where || [] - return new BaseQueryBuilder({ + return this._clone({ ...this.query, where: [...existingWhere, expression], }) as any @@ -527,7 +601,7 @@ export class BaseQueryBuilder { const existingHaving = this.query.having || [] - return new BaseQueryBuilder({ + return this._clone({ ...this.query, having: [...existingHaving, expression], }) as any @@ -593,7 +667,7 @@ export class BaseQueryBuilder { const select = buildNestedSelect(selectObject, aliases) - return new BaseQueryBuilder({ + return this._clone({ ...this.query, select: select, fnSelect: undefined, // remove the fnSelect clause if it exists @@ -668,7 +742,7 @@ export class BaseQueryBuilder { const existingOrderBy: OrderBy = this.query.orderBy || [] - return new BaseQueryBuilder({ + return this._clone({ ...this.query, orderBy: [...existingOrderBy, ...orderByClauses], }) as any @@ -713,7 +787,7 @@ export class BaseQueryBuilder { // Extend existing groupBy expressions (multiple groupBy calls should accumulate) const existingGroupBy = this.query.groupBy || [] - return new BaseQueryBuilder({ + return this._clone({ ...this.query, groupBy: [...existingGroupBy, ...newExpressions], }) as any @@ -736,7 +810,7 @@ export class BaseQueryBuilder { * ``` */ limit(count: number): QueryBuilder { - return new BaseQueryBuilder({ + return this._clone({ ...this.query, limit: count, }) as any @@ -760,7 +834,7 @@ export class BaseQueryBuilder { * ``` */ offset(count: number): QueryBuilder { - return new BaseQueryBuilder({ + return this._clone({ ...this.query, offset: count, }) as any @@ -781,7 +855,7 @@ export class BaseQueryBuilder { * ``` */ distinct(): QueryBuilder { - return new BaseQueryBuilder({ + return this._clone({ ...this.query, distinct: true, }) as any @@ -801,7 +875,7 @@ export class BaseQueryBuilder { *``` */ findOne(): QueryBuilder { - return new BaseQueryBuilder({ + return this._clone({ ...this.query, // TODO: enforcing return only one result with also a default orderBy if none is specified // limit: 1, @@ -867,11 +941,21 @@ export class BaseQueryBuilder { * age: row.users.age + 1, * })) * ``` + * + * Child query builders, query expressions, and helpers such as eq(), + * toArray(), and materialize() cannot be returned from fn.select(). Use + * them as fields in select() so the compiler can add them to the query + * graph. + * + * Compiled Collection-valued includes cannot be inputs to fn.select(), + * including nested descendants. Use toArray() or materialize() in the + * upstream select(), or do parent-only functional work before adding + * live Collection includes with select(). */ select( callback: (row: TContext[`schema`]) => TFuncSelectResult, - ): QueryBuilder> { - return new BaseQueryBuilder({ + ): FnSelectQueryResult { + return builder._clone({ ...builder.query, select: undefined, // remove the select clause if it exists fnSelect: callback, @@ -895,7 +979,7 @@ export class BaseQueryBuilder { where( callback: (row: TContext[`schema`]) => any, ): QueryBuilder { - return new BaseQueryBuilder({ + return builder._clone({ ...builder.query, fnWhere: [ ...(builder.query.fnWhere || []), @@ -923,7 +1007,7 @@ export class BaseQueryBuilder { having( callback: (row: FunctionalHavingRow) => any, ): QueryBuilder { - return new BaseQueryBuilder({ + return builder._clone({ ...builder.query, fnHaving: [ ...(builder.query.fnHaving || []), @@ -1084,14 +1168,17 @@ function buildConditionalSelect( /** * Recursively collects all PropRef nodes from an expression tree. */ -function collectRefsFromExpression(expr: BasicExpression): Array { +function collectRefsFromExpression( + expr: BasicExpression | Aggregate, +): Array { const refs: Array = [] switch (expr.type) { case `ref`: refs.push(expr) break case `func`: - for (const arg of (expr as any).args ?? []) { + case `agg`: + for (const arg of expr.args) { refs.push(...collectRefsFromExpression(arg)) } break @@ -1101,6 +1188,164 @@ function collectRefsFromExpression(expr: BasicExpression): Array { return refs } +function collectRefsFromSelectValue(value: unknown): Array { + if ( + value instanceof PropRef || + value instanceof FuncExpr || + value instanceof AggregateExpr + ) { + return collectRefsFromExpression(value) + } + if (value instanceof ConditionalSelect) { + return [ + ...value.branches.flatMap((branch) => [ + ...collectRefsFromExpression(branch.condition), + ...collectRefsFromSelectValue(branch.value), + ]), + ...(value.defaultValue === undefined + ? [] + : collectRefsFromSelectValue(value.defaultValue)), + ] + } + if (value instanceof IncludesSubquery) { + return [ + value.correlationField, + ...(value.parentProjection ?? []), + ...collectExternalRefsFromQuery(value.query), + ] + } + if (!isPlainObject(value)) return [] + return Object.values(value).flatMap(collectRefsFromSelectValue) +} + +function collectExternalRefsFromQuery(query: QueryIR): Array { + const localAliases = new Set(collectQueryAliases(query)) + const refs: Array = [] + const addExpression = (expression: BasicExpression | Aggregate) => { + refs.push(...collectRefsFromExpression(expression)) + } + const addWhere = (where: Where) => { + addExpression( + typeof where === `object` && `expression` in where + ? where.expression + : where, + ) + } + + for (const where of query.where ?? []) addWhere(where) + for (const join of query.join ?? []) { + addExpression(join.left) + addExpression(join.right) + if (join.from.type === `queryRef`) { + refs.push(...collectExternalRefsFromQuery(join.from.query)) + } + } + for (const expression of query.groupBy ?? []) addExpression(expression) + for (const having of query.having ?? []) addWhere(having) + for (const { expression } of query.orderBy ?? []) addExpression(expression) + if (query.select) refs.push(...collectRefsFromSelectValue(query.select)) + + if (query.from.type === `queryRef`) { + refs.push(...collectExternalRefsFromQuery(query.from.query)) + } else if (query.from.type === `unionFrom`) { + for (const source of query.from.sources) { + if (source.type === `queryRef`) { + refs.push(...collectExternalRefsFromQuery(source.query)) + } + } + } else if (query.from.type === `unionAll`) { + for (const branch of query.from.queries) { + refs.push(...collectExternalRefsFromQuery(branch)) + } + } + + const seen = new Set() + return refs.filter((ref) => { + const alias = ref.path.length > 1 ? ref.path[0] : undefined + const path = ref.path.join(`.`) + if ( + alias == null || + alias === `$selected` || + localAliases.has(alias) || + seen.has(path) + ) { + return false + } + seen.add(path) + return true + }) +} + +function collectParentRefsFromQuery( + query: QueryIR, + parentAliases: Array, +): Array { + const refs: Array = [] + const addExpression = (expression: BasicExpression | Aggregate) => { + refs.push(...collectRefsFromExpression(expression)) + } + const addWhere = (where: Where) => { + addExpression( + typeof where === `object` && `expression` in where + ? where.expression + : where, + ) + } + + for (const where of query.where ?? []) addWhere(where) + for (const join of query.join ?? []) { + addExpression(join.left) + addExpression(join.right) + if (join.from.type === `queryRef`) { + refs.push(...collectParentRefsFromQuery(join.from.query, parentAliases)) + } + } + for (const expression of query.groupBy ?? []) addExpression(expression) + for (const having of query.having ?? []) addWhere(having) + for (const { expression } of query.orderBy ?? []) addExpression(expression) + if (query.select) { + refs.push(...collectRefsFromSelectValue(query.select)) + } + + if (query.from.type === `queryRef`) { + refs.push(...collectParentRefsFromQuery(query.from.query, parentAliases)) + } else if (query.from.type === `unionFrom`) { + for (const source of query.from.sources) { + if (source.type === `queryRef`) { + refs.push(...collectParentRefsFromQuery(source.query, parentAliases)) + } + } + } else if (query.from.type === `unionAll`) { + for (const branch of query.from.queries) { + refs.push(...collectParentRefsFromQuery(branch, parentAliases)) + } + } + + const seen = new Set() + return refs.filter((ref) => { + const path = ref.path.join(`.`) + if ( + ref.path[0] == null || + !parentAliases.includes(ref.path[0]) || + seen.has(path) + ) { + return false + } + seen.add(path) + return true + }) +} + +function collectExternalParentAliases(query: QueryIR): Array { + return [ + ...new Set( + collectExternalRefsFromQuery(query) + .map((ref) => ref.path[0]) + .filter((alias): alias is string => alias !== undefined), + ), + ] +} + /** * Checks whether a WHERE clause references any parent alias. */ @@ -1129,6 +1374,9 @@ function buildIncludesSubquery( // Collect child's own aliases const childAliases = collectQueryAliases(childQuery) + const visibleParentAliases = [ + ...new Set([...parentAliases, ...collectExternalParentAliases(childQuery)]), + ] // Walk child's WHERE clauses to find the correlation condition. // The correlation eq() may be a standalone WHERE or nested inside a top-level and(). @@ -1154,7 +1402,7 @@ function buildIncludesSubquery( const result = extractCorrelation( expr.args[0]!, expr.args[1]!, - parentAliases, + visibleParentAliases, childAliases, ) if (result) { @@ -1181,7 +1429,7 @@ function buildIncludesSubquery( const result = extractCorrelation( arg.args[0]!, arg.args[1]!, - parentAliases, + visibleParentAliases, childAliases, ) if (result) { @@ -1242,32 +1490,21 @@ function buildIncludesSubquery( const pureChildWhere: Array = [] const parentFilters: Array = [] for (const w of modifiedWhere) { - if (referencesParent(w, parentAliases)) { + if (referencesParent(w, visibleParentAliases)) { parentFilters.push(w) } else { pureChildWhere.push(w) } } - // Collect distinct parent PropRefs from parent-referencing filters - let parentProjection: Array | undefined - if (parentFilters.length > 0) { - const seen = new Set() - parentProjection = [] - for (const w of parentFilters) { - const expr = typeof w === `object` && `expression` in w ? w.expression : w - for (const ref of collectRefsFromExpression(expr)) { - if ( - ref.path[0] != null && - parentAliases.includes(ref.path[0]) && - !seen.has(ref.path.join(`.`)) - ) { - seen.add(ref.path.join(`.`)) - parentProjection.push(ref) - } - } - } - } + // Every parent input that can affect the child plan belongs to the route + // identity, not only the main equality key or residual filters. + const projectedParentRefs = collectParentRefsFromQuery( + { ...childQuery, where: modifiedWhere }, + visibleParentAliases, + ) + const parentProjection = + projectedParentRefs.length > 0 ? projectedParentRefs : undefined const modifiedQuery: QueryIR = { ...childQuery, @@ -1383,12 +1620,7 @@ export function buildQuery( return getQueryIR(result) } -// Internal function to get the QueryIR from a builder -export function getQueryIR( - builder: BaseQueryBuilder | QueryBuilder | InitialQueryBuilder, -): QueryIR { - return (builder as unknown as BaseQueryBuilder)._getQuery() -} +export { getQueryIR } // Type-only exports for the query builder export type InitialQueryBuilder = Pick< diff --git a/packages/db/src/query/builder/query-ir.ts b/packages/db/src/query/builder/query-ir.ts new file mode 100644 index 0000000000..2aefb0be91 --- /dev/null +++ b/packages/db/src/query/builder/query-ir.ts @@ -0,0 +1,13 @@ +import type { + BaseQueryBuilder, + InitialQueryBuilder, + QueryBuilder, +} from './index.js' +import type { QueryIR } from '../ir.js' + +// Keep IR access independent of Collection construction at runtime. +export function getQueryIR( + builder: BaseQueryBuilder | QueryBuilder | InitialQueryBuilder, +): QueryIR { + return (builder as unknown as BaseQueryBuilder)._getQuery() +} diff --git a/packages/db/src/query/builder/types.ts b/packages/db/src/query/builder/types.ts index ac6a95dac4..5adb14bc5b 100644 --- a/packages/db/src/query/builder/types.ts +++ b/packages/db/src/query/builder/types.ts @@ -1,4 +1,5 @@ import type { Collection, CollectionImpl } from '../../collection/index.js' +import type { CollectionOptionsIdentity } from '../../collection-options.js' import type { SingleResult, StringCollationConfig } from '../../types.js' import type { Aggregate, @@ -89,7 +90,10 @@ export type ContextSchema = Record * Example: `{ users: usersCollection }` */ export type Source = { - [alias: string]: CollectionImpl | QueryBuilder + [alias: string]: + | CollectionImpl + | CollectionOptionsIdentity + | QueryBuilder } /** @@ -101,7 +105,15 @@ export type Source = { export type InferCollectionType = T extends CollectionImpl ? WithVirtualProps - : never + : T extends CollectionOptionsIdentity< + infer TOutput, + infer TKey, + any, + any, + any + > + ? WithVirtualProps + : never /** * SchemaFromSource - Converts a Source definition into a ContextSchema @@ -116,9 +128,11 @@ export type InferCollectionType = export type SchemaFromSource = Prettify<{ [K in keyof T]: T[K] extends CollectionImpl ? InferCollectionType - : T[K] extends QueryBuilder - ? GetResult - : never + : T[K] extends CollectionOptionsIdentity + ? InferCollectionType + : T[K] extends QueryBuilder + ? GetRawResult + : never }> export type UnionRefsSchema = Prettify<{ @@ -154,7 +168,7 @@ export type ContextFromUnionSource = : ContextFromSource type ResultFromBranch = - TBranch extends QueryBuilder ? GetResult : never + TBranch extends QueryBuilder ? GetRawResult : never type UnionBranchResult>> = ResultFromBranch @@ -467,19 +481,65 @@ export type ResultTypeFromSelect = }> > -export type SelectResult = - IsPlainObject extends true - ? ResultTypeFromSelect - : ResultTypeFromSelectValue - // Distribute over caseWhen branch unions so projection branches remain a union // of branch result shapes instead of being merged as one object type. type ResultTypeFromCaseWhen = T extends unknown ? ResultTypeFromSelectValue : never -// Extract Ref or subobject with a spread or a Ref -type ExtractRef = Prettify>> +// Extract Ref or subobject with a spread or a Ref. +type ExtractRef = T extends unknown + ? IsTrueRef extends true + ? T extends RefLeaf + ? IsNullableRef extends true + ? DeepNullable + : U + : never + : Prettify>> + : never + +// A "true" Ref is one that is structurally equivalent to the canonical +// `Ref` shape the query builder produces for its underlying user type +// `U` (taking the ref's own nullability into account). When `T` is a true +// ref, `ExtractRef` can safely return `U` directly; otherwise it must fall +// through to the recursive projection. +// +// Checking only that `T` has "no extra keys" beyond `keyof U` (plus the +// brand/virtual props) is not sufficient. A spread-derived object can keep +// exactly the keys of `U` while: +// - changing a field's type, e.g. `{ ...u, code: u.slug }`, or +// - dropping an optional key, e.g. `const { nickname, ...rest } = u`. +// Both must be recursively projected, not collapsed back to `U`. We +// therefore require strict structural equivalence against the canonical ref +// shape rather than a one-directional key-subset check. +type IsTrueRef = + T extends RefLeaf + ? RefShapeMatches>> extends true + ? true + : false + : false + +// Strict structural equivalence between two ref shapes. Unlike plain +// bidirectional assignability, this is sensitive to *key presence* — an +// object that drops an optional key (e.g. `const { nickname, ...rest } = u`) +// is not considered equal to one that keeps `nickname?`, even though the two +// remain mutually assignable. A direct ref (`u.document`, a union member, +// etc.) is exactly the canonical `Ref` shape and matches here, so it returns +// `U` via the fast path; any spread-derived object differs (changed field +// types, dropped keys, or stripped `readonly` modifiers) and instead falls +// through to the recursive projection, which reconstructs the correct type. +type RefShapeMatches = + (() => G extends A ? 1 : 2) extends () => G extends B ? 1 : 2 + ? true + : false + +// Propagate nullable-join semantics into the user-data shape. +type DeepNullable = + T extends Record + ? IsPlainObject extends true + ? { [K in keyof T]: DeepNullable } + : T | undefined + : T | undefined // Helper type to extract the underlying type from various expression types type ExtractExpressionType = @@ -611,8 +671,11 @@ type ValueOfUnion = T extends unknown ? T[K] : never : never -type RefForContextValue = - IsPlainObject extends true ? Ref : RefLeaf +type RefForContextValue = T extends unknown + ? IsPlainObject extends true + ? Ref + : RefLeaf + : never type RefsSchemaForContext = IsExactlyUndefined extends true ? TContext[`schema`] @@ -770,7 +833,11 @@ type VirtualPropsRef = { * select(({ user }) => ({ ...user })) // Returns User type, not Ref types * ``` */ -export type Ref = { +export type Ref = T extends unknown + ? RefBranch + : never + +type RefBranch = { [K in keyof T]: IsNonExactOptional extends true ? IsNonExactNullable extends true ? // Both optional and nullable diff --git a/packages/db/src/query/compiler/evaluators.ts b/packages/db/src/query/compiler/evaluators.ts index 929ac56dfb..55b8c68f49 100644 --- a/packages/db/src/query/compiler/evaluators.ts +++ b/packages/db/src/query/compiler/evaluators.ts @@ -3,7 +3,13 @@ import { UnknownExpressionTypeError, UnknownFunctionError, } from '../../errors.js' -import { areValuesEqual, normalizeValue } from '../../utils/comparison.js' +import { + areValuesEqual, + compareValues, + isUint8Array, + isUnorderable, + normalizeValue, +} from '../../utils/comparison.js' import type { BasicExpression, Func, PropRef } from '../ir.js' import type { NamespacedRow } from '../../types.js' @@ -14,6 +20,24 @@ function isUnknown(value: any): boolean { return value === null || value === undefined } +function normalizeEqualityOperand(value: unknown): unknown { + // Byte comparison needs no Map-key encoding, even for large binary values. + return isUint8Array(value) ? value : normalizeValue(value) +} + +/** + * Equality that follows PostgreSQL float semantics for `NaN`/invalid Dates: + * such values are equal to one another and unequal to anything else. For all + * other values it defers to {@link areValuesEqual}. Operands must not be + * null/undefined (callers handle UNKNOWN first). + */ +function valuesEqual(a: any, b: any): boolean { + if (isUnorderable(a) || isUnorderable(b)) { + return isUnorderable(a) && isUnorderable(b) + } + return areValuesEqual(a, b) +} + function toDateValue(value: any): Date | null { if (value instanceof Date) { return Number.isNaN(value.getTime()) ? null : value @@ -227,14 +251,15 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { const argA = compiledArgs[0]! const argB = compiledArgs[1]! return (data) => { - const a = normalizeValue(argA(data)) - const b = normalizeValue(argB(data)) + const a = normalizeEqualityOperand(argA(data)) + const b = normalizeEqualityOperand(argB(data)) // In 3-valued logic, any comparison with null/undefined returns UNKNOWN if (isUnknown(a) || isUnknown(b)) { return null } - // Use areValuesEqual for proper Uint8Array/Buffer comparison - return areValuesEqual(a, b) + // NaN/invalid Dates are equal to one another (PostgreSQL semantics); + // otherwise use areValuesEqual for proper Uint8Array/Buffer comparison + return valuesEqual(a, b) } } case `gt`: { @@ -247,7 +272,10 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (isUnknown(a) || isUnknown(b)) { return null } - return a > b + if (isUnorderable(a) || isUnorderable(b)) { + return isUnorderable(a) && !isUnorderable(b) + } + return compareValues(a, b) > 0 } } case `gte`: { @@ -260,7 +288,10 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (isUnknown(a) || isUnknown(b)) { return null } - return a >= b + if (isUnorderable(a) || isUnorderable(b)) { + return isUnorderable(a) + } + return compareValues(a, b) >= 0 } } case `lt`: { @@ -273,7 +304,10 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (isUnknown(a) || isUnknown(b)) { return null } - return a < b + if (isUnorderable(a) || isUnorderable(b)) { + return isUnorderable(b) && !isUnorderable(a) + } + return compareValues(a, b) < 0 } } case `lte`: { @@ -286,7 +320,10 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (isUnknown(a) || isUnknown(b)) { return null } - return a <= b + if (isUnorderable(a) || isUnorderable(b)) { + return isUnorderable(b) + } + return compareValues(a, b) <= 0 } } @@ -361,7 +398,7 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { const valueEvaluator = compiledArgs[0]! const arrayEvaluator = compiledArgs[1]! return (data) => { - const value = normalizeValue(valueEvaluator(data)) + const value = normalizeEqualityOperand(valueEvaluator(data)) const array = arrayEvaluator(data) // In 3-valued logic, if the value is null/undefined, return UNKNOWN if (isUnknown(value)) { @@ -370,7 +407,9 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (!Array.isArray(array)) { return false } - return array.some((item) => normalizeValue(item) === value) + return array.some((item) => + valuesEqual(normalizeEqualityOperand(item), value), + ) } } diff --git a/packages/db/src/query/compiler/expressions.ts b/packages/db/src/query/compiler/expressions.ts index f2856ed7eb..a52b8d11e5 100644 --- a/packages/db/src/query/compiler/expressions.ts +++ b/packages/db/src/query/compiler/expressions.ts @@ -1,6 +1,27 @@ import { Func, PropRef, Value } from '../ir.js' import type { BasicExpression, OrderBy } from '../ir.js' +/** Extracts the source aliases referenced by an expression. */ +export function getSourceAliasesFromExpression( + expr: BasicExpression, +): Set { + switch (expr.type) { + case `ref`: + return new Set(expr.path[0] ? [expr.path[0]] : []) + case `func`: { + const sourceAliases = new Set() + for (const arg of expr.args) { + for (const alias of getSourceAliasesFromExpression(arg)) { + sourceAliases.add(alias) + } + } + return sourceAliases + } + default: + return new Set() + } +} + /** * Normalizes a WHERE clause expression by removing table aliases from property references. * diff --git a/packages/db/src/query/compiler/group-by.ts b/packages/db/src/query/compiler/group-by.ts index c670de9649..9ce3b2f6a0 100644 --- a/packages/db/src/query/compiler/group-by.ts +++ b/packages/db/src/query/compiler/group-by.ts @@ -18,11 +18,24 @@ import { UnknownHavingExpressionTypeError, UnsupportedAggregateFunctionError, } from '../../errors.js' +import { + getEqualityValueIdentity, + getParentContextIdentity, + getParentContextValue, +} from '../equality-value-identity.js' import { compileExpression, isCaseWhenConditionTrue, toBooleanPredicate, } from './evaluators.js' +import { + INCLUDES_PUBLIC_KEY, + attachRouteMetadata, + getNamespacedRouteMetadata, + stripInternalCallbackMetadata, +} from './route-metadata.js' +import type { ValueIdentity } from '../equality-value-identity.js' +import type { RouteMetadata } from './route-metadata.js' import type { Aggregate, BasicExpression, @@ -34,15 +47,171 @@ import type { import type { NamespacedAndKeyedStream, NamespacedRow } from '../../types.js' import type { VirtualOrigin } from '../../virtual-props.js' -const VIRTUAL_SYNCED_KEY = `__virtual_synced__` -const VIRTUAL_HAS_LOCAL_KEY = `__virtual_has_local__` -const GROUP_KEY_REF_PREFIX = `__group_key_` +const RAW_REPRESENTATIVE = Symbol(`raw_group_representative`) + +type InternalGroupFields = ReturnType + +function createInternalGroupFields(groupCount: number, selectClause?: Select) { + const aliases = Object.keys(selectClause ?? {}) + let prefix = `__tanstack_group_` + while (aliases.some((alias) => alias.startsWith(prefix))) prefix += `_` + + return { + virtual: `${prefix}virtual`, + route: `${prefix}route`, + correlationIdentity: `${prefix}correlation_identity`, + parentContextIdentity: `${prefix}parent_context_identity`, + singleGroup: `${prefix}single_group`, + aggregatePrefix: `${prefix}aggregate_`, + groupKeys: Array.from( + { length: groupCount }, + (_, i) => `${prefix}key_${i}`, + ), + groupValues: Array.from( + { length: groupCount }, + (_, i) => `${prefix}value_${i}`, + ), + groupKeyRefs: Array.from( + { length: groupCount }, + (_, i) => `${prefix}key_ref_${i}`, + ), + } +} type RowVirtualMetadata = { synced: boolean hasLocal: boolean } +type Representative = { + key: string + [RAW_REPRESENTATIVE]: T +} + +function createPublicGroupKey(values: Array): unknown { + const identities = values.map(getEqualityValueIdentity) + if (identities.length === 1) { + const identity = identities[0] + if ( + identity == null || + (typeof identity !== `object` && + typeof identity !== `function` && + typeof identity !== `symbol`) + ) { + return identity + } + } + return serializeValue(identities) +} + +function attachPublicGroupKey( + row: Record, + publicKey: unknown, +): void { + const keyedRow = row as Record + keyedRow[INCLUDES_PUBLIC_KEY] = publicKey +} + +function createRepresentative( + rowKey: string, + value: T, + identity: unknown, +): Representative { + // Encode once per contribution, not once per member on every group change. + const representative = { + key: serializeValue([rowKey, identity]), + } as Representative + Object.defineProperty(representative, RAW_REPRESENTATIVE, { value }) + return representative +} + +function getRepresentative( + values: Array<[Representative, number]>, +): Representative | undefined { + let selected: Representative | undefined + for (const [candidate, multiplicity] of values) { + if (multiplicity <= 0) continue + if (selected === undefined || candidate.key < selected.key) { + selected = candidate + } + } + return selected +} + +function unwrapRepresentative( + value: Representative | undefined, +): T | undefined { + return value?.[RAW_REPRESENTATIVE] +} + +function addCorrelationRouteIdentityToGroupKey( + key: Record, + row: NamespacedRow, + mainSource: string, + fields: InternalGroupFields, + valueIdentity: ValueIdentity, +): void { + const route = getNamespacedRouteMetadata(row, mainSource) + key[fields.correlationIdentity] = valueIdentity.equality( + route?.correlationKey, + ) + if (route?.parentContext != null) { + key[fields.parentContextIdentity] = getParentContextIdentity( + route.parentContext, + ) + } +} + +/** One representative carries the whole route so both parts come from one row. */ +function addCorrelationRouteAggregate( + aggregates: Record, + mainSource: string, + fields: InternalGroupFields, + valueIdentity: ValueIdentity, +): void { + aggregates[fields.route] = { + preMap: ([rowKey, row]: [string, NamespacedRow]) => { + const route = getNamespacedRouteMetadata(row, mainSource) + return createRepresentative(rowKey, route, [ + valueIdentity.exact(route?.correlationKey), + getParentContextIdentity(route?.parentContext), + ]) + }, + reduce: getRepresentative, + postMap: unwrapRepresentative, + } +} + +function getGroupRoute( + aggregatedRow: Record, + fields: InternalGroupFields, +): RouteMetadata | undefined { + return aggregatedRow[fields.route] as RouteMetadata | undefined +} + +function getCorrelationRouteIdentity( + aggregatedRow: Record, + fields: InternalGroupFields, +): unknown { + return getGroupRoute(aggregatedRow, fields)?.parentContext == null + ? aggregatedRow[fields.correlationIdentity] + : [ + aggregatedRow[fields.correlationIdentity], + aggregatedRow[fields.parentContextIdentity], + ] +} + +function getGroupEvaluationRow( + row: Record, + fields: InternalGroupFields, + selected = row.$selected as Record, +): NamespacedRow { + return { + ...getParentContextValue(getGroupRoute(row, fields)?.parentContext), + $selected: selected, + } +} + function getRowVirtualMetadata(row: NamespacedRow): RowVirtualMetadata { let found = false let allSynced = true @@ -74,14 +243,6 @@ function getRowVirtualMetadata(row: NamespacedRow): RowVirtualMetadata { const { sum, count, avg, min, max } = groupByOperators -/** - * Interface for caching the mapping between GROUP BY expressions and SELECT expressions - */ -interface GroupBySelectMapping { - selectToGroupByIndex: Map // Maps SELECT alias to GROUP BY expression index - groupByExpressions: Array // The GROUP BY expressions for reference -} - /** * Validates that all non-aggregate expressions in SELECT are present in GROUP BY * and creates a cached mapping for efficient lookup during processing @@ -89,12 +250,11 @@ interface GroupBySelectMapping { function validateAndCreateMapping( groupByClause: GroupBy, selectClause?: Select, -): GroupBySelectMapping { +): Map { const selectToGroupByIndex = new Map() - const groupByExpressions = [...groupByClause] if (!selectClause) { - return { selectToGroupByIndex, groupByExpressions } + return selectToGroupByIndex } // Validate each SELECT expression @@ -105,7 +265,7 @@ function validateAndCreateMapping( } // Non-aggregate expression must be in GROUP BY - const groupIndex = groupByExpressions.findIndex((groupExpr) => + const groupIndex = groupByClause.findIndex((groupExpr) => expressionsEqual(expr, groupExpr), ) @@ -117,7 +277,7 @@ function validateAndCreateMapping( selectToGroupByIndex.set(alias, groupIndex) } - return { selectToGroupByIndex, groupByExpressions } + return selectToGroupByIndex } /** @@ -127,206 +287,79 @@ function validateAndCreateMapping( export function processGroupBy( pipeline: NamespacedAndKeyedStream, groupByClause: GroupBy, + valueIdentity: ValueIdentity, havingClauses?: Array, selectClause?: Select, fnHavingClauses?: Array<(row: any) => any>, aggregateCollectionId?: string, mainSource?: string, + sanitizeCallbackRows = false, ): NamespacedAndKeyedStream { + const fields = createInternalGroupFields(groupByClause.length, selectClause) const virtualAggregates: Record = { - [VIRTUAL_SYNCED_KEY]: { - preMap: ([, row]: [string, NamespacedRow]) => - getRowVirtualMetadata(row).synced, - reduce: (values: Array<[boolean, number]>) => { - for (const [isSynced, multiplicity] of values) { - if (!isSynced && multiplicity > 0) { - return false - } + [fields.virtual]: { + preMap: ([, row]: [string, NamespacedRow]) => getRowVirtualMetadata(row), + reduce: (values: Array<[RowVirtualMetadata, number]>) => { + const group: RowVirtualMetadata = { synced: true, hasLocal: false } + for (const [metadata, multiplicity] of values) { + if (multiplicity <= 0) continue + if (!metadata.synced) group.synced = false + if (metadata.hasLocal) group.hasLocal = true } - return true - }, - }, - [VIRTUAL_HAS_LOCAL_KEY]: { - preMap: ([, row]: [string, NamespacedRow]) => - getRowVirtualMetadata(row).hasLocal, - reduce: (values: Array<[boolean, number]>) => { - for (const [isLocal, multiplicity] of values) { - if (isLocal && multiplicity > 0) { - return true - } - } - return false + return group }, }, } - // Handle empty GROUP BY (single-group aggregation) - if (groupByClause.length === 0) { - // For single-group aggregation, create a single group with all data - const aggregates: Record = virtualAggregates - - // Expressions that wrap aggregates (e.g. coalesce(count(...), 0)). - // Keys are the original SELECT aliases; values are pre-compiled evaluators - // over the transformed (aggregate-free) expression. - const wrappedAggExprs: Record any> = {} - const aggCounter = { value: 0 } - - if (selectClause) { - // Scan the SELECT clause for aggregate functions - for (const [alias, expr] of Object.entries(selectClause)) { - if (expr.type === `agg`) { - aggregates[alias] = getAggregateFunction(expr) - } else if (containsAggregate(expr)) { - const { transformed, extracted } = extractAndReplaceAggregates( - expr as SelectValueExpression, - aggCounter, - ) - for (const [syntheticAlias, aggExpr] of Object.entries(extracted)) { - aggregates[syntheticAlias] = getAggregateFunction(aggExpr) - } - wrappedAggExprs[alias] = compileGroupedSelectValue(transformed) - } - } - } - - // Use a constant key for single group. - // When mainSource is set (includes mode), include __correlationKey so that - // rows from different parents aggregate separately. - const keyExtractor = mainSource - ? ([, row]: [string, NamespacedRow]) => ({ - __singleGroup: true, - __correlationKey: (row as any)?.[mainSource]?.__correlationKey, - }) - : () => ({ __singleGroup: true }) - - // Apply the groupBy operator with single group - pipeline = pipeline.pipe( - groupBy(keyExtractor, aggregates), - ) as NamespacedAndKeyedStream - - // Update $selected to include aggregate values - pipeline = pipeline.pipe( - map(([, aggregatedRow]) => { - // Start with the existing $selected from early SELECT processing - const selectResults = (aggregatedRow as any).$selected || {} - const finalResults: Record = { ...selectResults } - - if (selectClause) { - // First pass: populate plain aggregate results and synthetic aliases - for (const [alias, expr] of Object.entries(selectClause)) { - if (expr.type === `agg`) { - finalResults[alias] = aggregatedRow[alias] - } - } - evaluateWrappedAggregates( - finalResults, - aggregatedRow as Record, - wrappedAggExprs, - ) - } - - // Use a single key for the result and update $selected. - // When in includes mode, restore the namespaced source structure with - // __correlationKey so output extraction can route results per-parent. - const correlationKey = mainSource - ? (aggregatedRow as any).__correlationKey - : undefined - const resultKey = - correlationKey !== undefined - ? `single_group_${serializeValue(correlationKey)}` - : `single_group` - const resultRow: Record = { - ...(aggregatedRow as Record), - $selected: finalResults, - } - const groupSynced = (aggregatedRow as Record)[ - VIRTUAL_SYNCED_KEY - ] - const groupHasLocal = (aggregatedRow as Record)[ - VIRTUAL_HAS_LOCAL_KEY - ] - resultRow.$synced = groupSynced ?? true - resultRow.$origin = ( - groupHasLocal ? `local` : `remote` - ) satisfies VirtualOrigin - resultRow.$key = resultKey - resultRow.$collectionId = - aggregateCollectionId ?? resultRow.$collectionId - if (mainSource && correlationKey !== undefined) { - resultRow[mainSource] = { __correlationKey: correlationKey } - } - return [resultKey, resultRow] as [unknown, Record] - }), + if (mainSource) { + addCorrelationRouteAggregate( + virtualAggregates, + mainSource, + fields, + valueIdentity, ) - - // Apply HAVING clauses if present - if (havingClauses && havingClauses.length > 0) { - for (const havingClause of havingClauses) { - const havingExpression = getHavingExpression(havingClause) - const transformedHavingClause = replaceAggregatesByRefs( - havingExpression, - selectClause || {}, - `$selected`, - ) - const compiledHaving = compileExpression(transformedHavingClause) - - pipeline = pipeline.pipe( - filter(([, row]) => { - // Create a namespaced row structure for HAVING evaluation - const namespacedRow = { $selected: (row as any).$selected } - return toBooleanPredicate(compiledHaving(namespacedRow)) - }), - ) - } - } - - // Apply functional HAVING clauses if present - if (fnHavingClauses && fnHavingClauses.length > 0) { - for (const fnHaving of fnHavingClauses) { - pipeline = pipeline.pipe( - filter(([, row]) => { - // Create a namespaced row structure for functional HAVING evaluation - const namespacedRow = { $selected: (row as any).$selected } - return toBooleanPredicate(fnHaving(namespacedRow)) - }), - ) - } - } - - return pipeline } - // Multi-group aggregation logic... - // Validate and create mapping for non-aggregate expressions in SELECT - const mapping = validateAndCreateMapping(groupByClause, selectClause) + const singleGroup = groupByClause.length === 0 + // Single-group aggregation accepts selections without grouping validation. + const mapping = singleGroup + ? undefined + : validateAndCreateMapping(groupByClause, selectClause) // Pre-compile groupBy expressions const compiledGroupByExpressions = groupByClause.map((e) => compileExpression(e), ) - // Create a key extractor function using simple __key_X format. - // When mainSource is set (includes mode), include __correlationKey so that - // rows from different parents with the same group key aggregate separately. + // Include the complete route so distinct parent inputs stay apart. const keyExtractor = ([, row]: [ string, NamespacedRow & { $selected?: any }, ]) => { // Use the original namespaced row for GROUP BY expressions, not $selected - const namespacedRow = { ...row } - delete (namespacedRow as any).$selected + const namespacedRow = singleGroup ? row : { ...row } + if (!singleGroup) delete namespacedRow.$selected - const key: Record = {} + const key: Record = singleGroup + ? { [fields.singleGroup]: true } + : {} - // Use simple __key_X format for each groupBy expression + // D2 must key groups by the same relation as the query evaluator. The raw + // representative is retained separately as an aggregate for projection. for (let i = 0; i < groupByClause.length; i++) { const compiledExpr = compiledGroupByExpressions[i]! const value = compiledExpr(namespacedRow) - key[`__key_${i}`] = value + key[fields.groupKeys[i]!] = valueIdentity.equality(value) } if (mainSource) { - key.__correlationKey = (row as any)?.[mainSource]?.__correlationKey + addCorrelationRouteIdentityToGroupKey( + key, + row, + mainSource, + fields, + valueIdentity, + ) } return key @@ -337,6 +370,18 @@ export function processGroupBy( const wrappedAggExprs: Record any> = {} const aggCounter = { value: 0 } + for (let i = 0; i < compiledGroupByExpressions.length; i++) { + const compiledExpr = compiledGroupByExpressions[i]! + aggregates[fields.groupValues[i]!] = { + preMap: ([rowKey, row]: [string, NamespacedRow]) => { + const value = compiledExpr(row) + return createRepresentative(rowKey, value, valueIdentity.exact(value)) + }, + reduce: getRepresentative, + postMap: unwrapRepresentative, + } + } + if (selectClause) { // Scan the SELECT clause for aggregate functions for (const [alias, expr] of Object.entries(selectClause)) { @@ -346,12 +391,19 @@ export function processGroupBy( const { transformed, extracted } = extractAndReplaceAggregates( expr as SelectValueExpression, aggCounter, + fields.aggregatePrefix, ) for (const [syntheticAlias, aggExpr] of Object.entries(extracted)) { aggregates[syntheticAlias] = getAggregateFunction(aggExpr) } wrappedAggExprs[alias] = compileGroupedSelectValue( - replaceGroupByRefsInSelectValue(transformed, groupByClause), + singleGroup + ? transformed + : replaceGroupByRefsInSelectValue( + transformed, + groupByClause, + fields.groupKeyRefs, + ), ) } } @@ -365,18 +417,21 @@ export function processGroupBy( map(([, aggregatedRow]) => { // Start with the existing $selected from early SELECT processing const selectResults = (aggregatedRow as any).$selected || {} - const finalResults: Record = {} + const finalResults: Record = singleGroup + ? { ...selectResults } + : {} if (selectClause) { // First pass: populate group keys, plain aggregates, and synthetic aliases for (const [alias, expr] of Object.entries(selectClause)) { if (expr.type === `agg`) { finalResults[alias] = aggregatedRow[alias] - } else if (!wrappedAggExprs[alias]) { + } else if (!singleGroup && !wrappedAggExprs[alias]) { // Use cached mapping to get the corresponding __key_X for non-aggregates - const groupIndex = mapping.selectToGroupByIndex.get(alias) + const groupIndex = mapping?.get(alias) if (groupIndex !== undefined) { - finalResults[alias] = aggregatedRow[`__key_${groupIndex}`] + finalResults[alias] = + aggregatedRow[fields.groupValues[groupIndex]!] } else { // Fallback to original SELECT results finalResults[alias] = selectResults[alias] @@ -387,53 +442,71 @@ export function processGroupBy( finalResults, aggregatedRow as Record, wrappedAggExprs, - groupByClause.length, + fields, ) } else { // No SELECT clause - just use the group keys for (let i = 0; i < groupByClause.length; i++) { - finalResults[`__key_${i}`] = aggregatedRow[`__key_${i}`] + finalResults[`__key_${i}`] = aggregatedRow[fields.groupValues[i]!] } } // Generate a simple key for the live collection using group values. - // When in includes mode, include the correlation key so that groups - // from different parents don't collide. - const correlationKey = mainSource - ? (aggregatedRow as any).__correlationKey + // In includes mode, add the complete route so correlated groups do not + // collide. + const route = mainSource + ? getGroupRoute(aggregatedRow, fields) + : undefined + const correlationKey = route?.correlationKey + const correlationRoute = mainSource + ? getCorrelationRouteIdentity(aggregatedRow, fields) : undefined const keyParts: Array = [] + const publicKeyParts: Array = [] for (let i = 0; i < groupByClause.length; i++) { - keyParts.push(aggregatedRow[`__key_${i}`]) + keyParts.push(aggregatedRow[fields.groupKeys[i]!]) + publicKeyParts.push(aggregatedRow[fields.groupValues[i]!]) } - if (correlationKey !== undefined) { - keyParts.push(correlationKey) + if (correlationRoute !== undefined) { + keyParts.push(correlationRoute) } - const finalKey = - keyParts.length === 1 ? keyParts[0] : serializeValue(keyParts) - - // When in includes mode, restore the namespaced source structure with - // __correlationKey so output extraction can route results per-parent. + const finalKey = singleGroup + ? correlationRoute !== undefined + ? `single_group_${serializeValue(correlationRoute)}` + : `single_group` + : keyParts.length === 1 + ? keyParts[0] + : serializeValue(keyParts) + const publicKey = singleGroup + ? `single_group` + : createPublicGroupKey(publicKeyParts) + + // When in includes mode, restore route metadata for output routing. const resultRow: Record = { ...(aggregatedRow as Record), $selected: finalResults, } - const groupSynced = (aggregatedRow as Record)[ - VIRTUAL_SYNCED_KEY - ] - const groupHasLocal = (aggregatedRow as Record)[ - VIRTUAL_HAS_LOCAL_KEY - ] - resultRow.$synced = groupSynced ?? true + const virtual = (aggregatedRow as Record)[fields.virtual] as + | RowVirtualMetadata + | undefined + resultRow.$synced = virtual?.synced ?? true resultRow.$origin = ( - groupHasLocal ? `local` : `remote` + virtual?.hasLocal ? `local` : `remote` ) satisfies VirtualOrigin - resultRow.$key = finalKey + resultRow.$key = publicKey resultRow.$collectionId = aggregateCollectionId ?? resultRow.$collectionId if (mainSource && correlationKey !== undefined) { - resultRow[mainSource] = { __correlationKey: correlationKey } + attachPublicGroupKey(resultRow, publicKey) + attachRouteMetadata( + resultRow, + correlationKey, + route?.parentContext ?? null, + ) } - return [finalKey, resultRow] as [unknown, Record] + return [mainSource ? finalKey : publicKey, resultRow] as [ + unknown, + Record, + ] }), ) @@ -449,9 +522,10 @@ export function processGroupBy( pipeline = pipeline.pipe( filter(([, row]) => { - // Create a namespaced row structure for HAVING evaluation - const namespacedRow = { $selected: (row as any).$selected } - return compiledHaving(namespacedRow) + const namespacedRow = getGroupEvaluationRow(row, fields) + const result = compiledHaving(namespacedRow) + // Preserve each path's coercion for unchecked nonboolean IR values. + return singleGroup ? toBooleanPredicate(result) : result }), ) } @@ -462,9 +536,11 @@ export function processGroupBy( for (const fnHaving of fnHavingClauses) { pipeline = pipeline.pipe( filter(([, row]) => { - // Create a namespaced row structure for functional HAVING evaluation - const namespacedRow = { $selected: (row as any).$selected } - return toBooleanPredicate(fnHaving(namespacedRow)) + const namespacedRow = getGroupEvaluationRow(row, fields) + const callbackRow = sanitizeCallbackRows + ? stripInternalCallbackMetadata(namespacedRow) + : namespacedRow + return toBooleanPredicate(fnHaving(callbackRow)) }), ) } @@ -636,21 +712,27 @@ function evaluateWrappedAggregates( finalResults: Record, aggregatedRow: Record, wrappedAggExprs: Record any>, - groupKeyCount: number = 0, + fields: InternalGroupFields, ): void { for (const key of Object.keys(aggregatedRow)) { - if (key.startsWith(`__agg_`)) { + if (key.startsWith(fields.aggregatePrefix)) { finalResults[key] = aggregatedRow[key] } } - for (let i = 0; i < groupKeyCount; i++) { - finalResults[`${GROUP_KEY_REF_PREFIX}${i}`] = aggregatedRow[`__key_${i}`] + for (let i = 0; i < fields.groupKeyRefs.length; i++) { + finalResults[fields.groupKeyRefs[i]!] = + aggregatedRow[fields.groupValues[i]!] } for (const [alias, evaluator] of Object.entries(wrappedAggExprs)) { - finalResults[alias] = evaluator({ $selected: finalResults }) + finalResults[alias] = evaluator( + getGroupEvaluationRow(aggregatedRow, fields, finalResults), + ) } for (const key of Object.keys(finalResults)) { - if (key.startsWith(`__agg_`) || key.startsWith(GROUP_KEY_REF_PREFIX)) { + if ( + key.startsWith(fields.aggregatePrefix) || + fields.groupKeyRefs.includes(key) + ) { delete finalResults[key] } } @@ -707,6 +789,7 @@ export function containsAggregate( function extractAndReplaceAggregates( expr: SelectValueExpression, counter: { value: number }, + aggregatePrefix: string, ): { transformed: SelectValueExpression extracted: Record @@ -716,7 +799,7 @@ function extractAndReplaceAggregates( } if (expr.type === `agg`) { - const alias = `__agg_${counter.value++}` + const alias = `${aggregatePrefix}${counter.value++}` return { transformed: new PropRef([`$selected`, alias]), extracted: { [alias]: expr }, @@ -726,7 +809,7 @@ function extractAndReplaceAggregates( if (expr.type === `func`) { const allExtracted: Record = {} const newArgs = expr.args.map((arg: BasicExpression | Aggregate) => { - const result = extractAndReplaceAggregates(arg, counter) + const result = extractAndReplaceAggregates(arg, counter, aggregatePrefix) Object.assign(allExtracted, result.extracted) return result.transformed as BasicExpression }) @@ -739,8 +822,16 @@ function extractAndReplaceAggregates( if (isConditionalSelect(expr)) { const allExtracted: Record = {} const branches = expr.branches.map((branch) => { - const condition = extractAndReplaceAggregates(branch.condition, counter) - const value = extractAndReplaceAggregates(branch.value, counter) + const condition = extractAndReplaceAggregates( + branch.condition, + counter, + aggregatePrefix, + ) + const value = extractAndReplaceAggregates( + branch.value, + counter, + aggregatePrefix, + ) Object.assign(allExtracted, condition.extracted, value.extracted) return { condition: condition.transformed as BasicExpression, @@ -750,7 +841,11 @@ function extractAndReplaceAggregates( const defaultValue = expr.defaultValue === undefined ? undefined - : extractAndReplaceAggregates(expr.defaultValue, counter) + : extractAndReplaceAggregates( + expr.defaultValue, + counter, + aggregatePrefix, + ) if (defaultValue) { Object.assign(allExtracted, defaultValue.extracted) @@ -770,6 +865,7 @@ function extractAndReplaceAggregates( const result = extractAndReplaceAggregates( value as SelectValueExpression, counter, + aggregatePrefix, ) Object.assign(allExtracted, result.extracted) transformed[key] = result.transformed @@ -785,6 +881,7 @@ function extractAndReplaceAggregates( function replaceGroupByRefsInSelectValue( value: SelectValueExpression, groupByClause: GroupBy, + groupKeyRefs: Array, ): SelectValueExpression { if (isConditionalSelect(value)) { return new ConditionalSelect( @@ -792,12 +889,21 @@ function replaceGroupByRefsInSelectValue( condition: replaceGroupByRefsInExpression( branch.condition, groupByClause, + groupKeyRefs, + ), + value: replaceGroupByRefsInSelectValue( + branch.value, + groupByClause, + groupKeyRefs, ), - value: replaceGroupByRefsInSelectValue(branch.value, groupByClause), })), value.defaultValue === undefined ? undefined - : replaceGroupByRefsInSelectValue(value.defaultValue, groupByClause), + : replaceGroupByRefsInSelectValue( + value.defaultValue, + groupByClause, + groupKeyRefs, + ), ) } @@ -807,6 +913,7 @@ function replaceGroupByRefsInSelectValue( transformed[key] = replaceGroupByRefsInSelectValue( entry as SelectValueExpression, groupByClause, + groupKeyRefs, ) } return transformed @@ -820,12 +927,13 @@ function replaceGroupByRefsInSelectValue( return value } - return replaceGroupByRefsInExpression(value, groupByClause) + return replaceGroupByRefsInExpression(value, groupByClause, groupKeyRefs) } function replaceGroupByRefsInExpression( expr: BasicExpression, groupByClause: GroupBy, + groupKeyRefs: Array, ): BasicExpression { if (expr.type === `ref`) { const groupIndex = groupByClause.findIndex((groupExpr) => @@ -833,14 +941,14 @@ function replaceGroupByRefsInExpression( ) return groupIndex === -1 ? expr - : new PropRef([`$selected`, `${GROUP_KEY_REF_PREFIX}${groupIndex}`]) + : new PropRef([`$selected`, groupKeyRefs[groupIndex]!]) } if (expr.type === `func`) { return new Func( expr.name, expr.args.map((arg) => - replaceGroupByRefsInExpression(arg, groupByClause), + replaceGroupByRefsInExpression(arg, groupByClause, groupKeyRefs), ), ) } diff --git a/packages/db/src/query/compiler/index.ts b/packages/db/src/query/compiler/index.ts index b7779a21d4..cc64fde363 100644 --- a/packages/db/src/query/compiler/index.ts +++ b/packages/db/src/query/compiler/index.ts @@ -5,9 +5,19 @@ import { join as joinOperator, map, reduce, + serializeValue, tap, } from '@tanstack/db-ivm' +import { isPlainObject } from '../../utils/type-guards.js' +import { getOrCreate } from '../../utils/get-or-create.js' import { optimizeQuery } from '../optimizer.js' +import { materializeCompilation } from '../live/materialized-pipeline.js' +import { + createParentContext, + createValueIdentity, + getParentContextIdentity, + getParentContextValue, +} from '../equality-value-identity.js' import { CollectionInputNotFoundError, DistinctRequiresSelectError, @@ -15,29 +25,54 @@ import { FnSelectWithGroupByError, HavingRequiresGroupByError, LimitOffsetRequireOrderByError, + UnsupportedFnSelectResultError, UnsupportedFromTypeError, } from '../../errors.js' import { VIRTUAL_PROP_NAMES } from '../../virtual-props.js' +import { BaseQueryBuilder } from '../builder/index.js' +import { + CaseWhenWrapper, + ConcatToArrayWrapper, + MaterializeWrapper, + ToArrayWrapper, +} from '../builder/functions.js' import { ConditionalSelect, IncludesSubquery, PropRef, Value as ValClass, + collectCollectionSources, + getFromSources, getWhereExpression, isExpressionLike, } from '../ir.js' import { ensureIndexForField } from '../../indexes/auto-index.js' -import { inArray } from '../builder/functions.js' +import { deepEquals } from '../../utils.js' +import { normalizeValue } from '../../utils/comparison.js' import { compileExpression, isCaseWhenConditionTrue, toBooleanPredicate, } from './evaluators.js' -import { processJoins } from './joins.js' +import { processJoins, registerLazyDemandPlan } from './joins.js' import { containsAggregate, processGroupBy } from './group-by.js' import { getLazyLoadTargets } from './lazy-targets.js' import { processOrderBy } from './order-by.js' +import { crossJoinParentRoutes } from './parent-routes.js' +import { + INCLUDES_PUBLIC_KEY, + INCLUDES_ROUTING, + attachRouteMetadata, + attachRouteMetadataToResult, + getNamespacedRouteMetadata, + getRouteMetadata, + getRoutedScalarMetadata, + stripInternalCallbackMetadata, + stripInternalRouteMetadata, + stripRouteMetadata, +} from './route-metadata.js' import { processSelect } from './select.js' +import type { ValueIdentity } from '../equality-value-identity.js' import type { CollectionSubscription } from '../../collection/subscription.js' import type { OrderByOptimizationInfo } from './order-by.js' import type { @@ -54,17 +89,60 @@ import type { Collection } from '../../collection/index.js' import type { KeyedStream, NamespacedAndKeyedStream, + NamespacedRow, ResultStream, } from '../../types.js' import type { QueryCache, QueryMapping, WindowOptions } from './types.js' export type { WindowOptions } from './types.js' +export { INCLUDES_PUBLIC_KEY, INCLUDES_ROUTING } from './route-metadata.js' -/** Symbol used to tag parent $selected with routing metadata for includes */ -export const INCLUDES_ROUTING = Symbol(`includesRouting`) -export const FN_SELECT_STATE = Symbol(`fnSelectState`) const SKIP_INCLUDE = Symbol(`skipInclude`) +function getUnsupportedFnSelectResultDescription( + value: unknown, + seen: Set = new Set(), +): string | undefined { + if (value instanceof BaseQueryBuilder) return `a child query builder` + if (value instanceof ToArrayWrapper) return `toArray()` + if (value instanceof ConcatToArrayWrapper) return `concat(toArray())` + if (value instanceof MaterializeWrapper) return `materialize()` + if (value instanceof CaseWhenWrapper) return `caseWhen()` + if (isExpressionLike(value)) { + return value && + typeof value === `object` && + `name` in value && + typeof value.name === `string` + ? `${value.name}()` + : `a query expression` + } + if (value === null || typeof value !== `object` || seen.has(value)) { + return undefined + } + + seen.add(value) + const keys = [ + ...Object.keys(value), + ...Object.getOwnPropertySymbols(value).filter((key) => + Object.prototype.propertyIsEnumerable.call(value, key), + ), + ] + for (const key of keys) { + const entry = (value as Record)[key] + const unsupported = getUnsupportedFnSelectResultDescription(entry, seen) + if (unsupported) return unsupported + } + return undefined +} + +export function validateFnSelectResult(value: unknown): void { + const unsupportedValueDescription = + getUnsupportedFnSelectResultDescription(value) + if (unsupportedValueDescription) { + throw new UnsupportedFnSelectResultError(unsupportedValueDescription) + } +} + type ConditionalSelectGuard = { condition: BasicExpression expected: boolean @@ -80,6 +158,121 @@ type ProjectedSourceIncludePath = { guards: Array } +type CompiledParentProjection = { + alias: string + field: Array + compiled: (row: NamespacedRow) => unknown +} + +function projectParentContext( + nsRow: NamespacedRow, + projections: Array, + valueIdentity: ValueIdentity, +): Record { + const inherited = getRouteMetadata(nsRow)?.parentContext + const inheritedValue = getParentContextValue(inherited) + const parentContext: Record = + inheritedValue === undefined ? {} : { ...inheritedValue } + const projectedIdentity: Array = [] + + for (const projection of projections) { + const projectedValue = projection.compiled(nsRow) + projectedIdentity.push([ + projection.alias, + projection.field, + valueIdentity.equality(projectedValue), + ]) + if (projection.field.length === 0) { + const projectedAlias = projectedValue + parentContext[projection.alias] = + projectedAlias != null && typeof projectedAlias === `object` + ? { ...projectedAlias } + : projectedAlias + continue + } + + const inheritedAlias = parentContext[projection.alias] + const aliasContext = + inheritedAlias != null && typeof inheritedAlias === `object` + ? { ...inheritedAlias } + : {} + parentContext[projection.alias] = aliasContext + + let target = aliasContext + for (let index = 0; index < projection.field.length - 1; index++) { + const segment = projection.field[index]! + const inheritedNested = target[segment] + const nested = + inheritedNested != null && typeof inheritedNested === `object` + ? { ...inheritedNested } + : {} + target[segment] = nested + target = nested + } + target[projection.field[projection.field.length - 1]!] = projectedValue + } + + return createParentContext(parentContext, [ + getParentContextIdentity(inherited), + projectedIdentity, + ]) +} + +function parameterizeByParentRoutes( + pipeline: NamespacedAndKeyedStream, + parentKeyStream: KeyedStream, + mainSource: string, + valueIdentity: ValueIdentity, +): NamespacedAndKeyedStream { + return crossJoinParentRoutes( + pipeline, + parentKeyStream, + (rowKey, row, correlationKey, parentContext) => { + const namespaced = { + ...(row as Record), + } as Record + namespaced[mainSource] = { + ...namespaced[mainSource], + [INCLUDES_PUBLIC_KEY]: + namespaced[mainSource]?.[INCLUDES_PUBLIC_KEY] ?? rowKey, + } + if (parentContext != null) { + Object.assign(namespaced, getParentContextValue(parentContext)) + } + attachRouteMetadata(namespaced, correlationKey, parentContext) + return [ + serializeValue([ + valueIdentity.equality(rowKey), + valueIdentity.equality(correlationKey), + getParentContextIdentity(parentContext), + ]), + namespaced, + ] as [string, NamespacedRow] + }, + ) as NamespacedAndKeyedStream +} + +function getRowCorrelationKey(row: NamespacedRow, mainSource: string): unknown { + return getNamespacedRouteMetadata(row, mainSource)?.correlationKey +} + +function getRowParentContext(row: NamespacedRow, mainSource: string): unknown { + return getNamespacedRouteMetadata(row, mainSource)?.parentContext ?? null +} + +function correlationValuesEqual(left: unknown, right: unknown): boolean { + if (left == null || right == null) return false + const normalizedLeft = normalizeValue(left) + const normalizedRight = normalizeValue(right) + return ( + Object.is(normalizedLeft, normalizedRight) || + (typeof normalizedLeft === `number` && + typeof normalizedRight === `number` && + Number.isNaN(normalizedLeft) && + Number.isNaN(normalizedRight)) + ) +} + /** * Result of compiling an includes subquery, including the child pipeline * and metadata needed to route child results to parent-scoped Collections. @@ -117,7 +310,10 @@ export interface CompilationResult { /** The compiled query pipeline (D2 stream) */ pipeline: ResultStream - /** Map of source aliases to their WHERE clauses for index optimization */ + /** Runtime identity scope owned by this compiled graph. */ + valueIdentity: ValueIdentity + + /** Map of opaque source IDs to their WHERE clauses for index optimization */ sourceWhereClauses: Map> /** @@ -146,6 +342,12 @@ export interface CompilationResult { includes?: Array } +const valueIdentitiesByCache = new WeakMap() + +function getCompilationValueIdentity(cache: QueryCache): ValueIdentity { + return getOrCreate(valueIdentitiesByCache, cache, createValueIdentity) +} + /** * Compiles a query IR into a D2 pipeline * @param rawQuery The query IR to compile @@ -153,8 +355,8 @@ export interface CompilationResult { * @param collections Mapping of collection IDs to Collection instances * @param subscriptions Mapping of source aliases to CollectionSubscription instances * @param callbacks Mapping of source aliases to lazy loading callbacks - * @param lazySources Set of source aliases that should load data lazily - * @param optimizableOrderByCollections Map of collection IDs to order-by optimization info + * @param lazySources Set of source identities that should load data lazily + * @param optimizableOrderByCollections Map of source IDs to order-by optimization info * @param cache Optional cache for compiled subqueries (used internally for recursion) * @param queryMapping Optional mapping from optimized queries to original queries * @returns A CompilationResult with the pipeline, source WHERE clauses, and alias metadata @@ -175,10 +377,12 @@ export function compileQuery( childCorrelationField?: PropRef, ): CompilationResult { // Check if the original raw query has already been compiled - const cachedResult = cache.get(rawQuery) + const cachedResult = + parentKeyStream === undefined ? cache.get(rawQuery) : undefined if (cachedResult) { return cachedResult } + const valueIdentity = getCompilationValueIdentity(cache) // Validate the raw query BEFORE optimization to check user's original structure. // This must happen before optimization because the optimizer may create internal @@ -196,6 +400,8 @@ export function compileQuery( // Create a copy of the inputs map to avoid modifying the original const allInputs = { ...inputs } + const rawSources = collectCollectionSources(rawQuery) + bindSourceInputs(rawSources, allInputs) // Track alias to collection id relationships discovered during compilation. // This includes all user-declared aliases plus inner aliases from subqueries. @@ -221,6 +427,7 @@ export function compileQuery( sourceIncludes, directIncludes, isUnionFrom, + isParentRouted, } = processFromClause( query.from, allInputs, @@ -235,45 +442,80 @@ export function compileQuery( aliasToCollectionId, aliasRemapping, sourceWhereClauses, + parentKeyStream, ) Object.assign(sources, fromSources) + const sourceCarriesInternalRouteState = + parentKeyStream !== undefined || + sourceIncludes.length > 0 || + directIncludes.length > 0 // If this is an includes child query, inner-join the raw input with parent keys. // This filters the child collection to only rows matching parents in the result set. // The inner join happens BEFORE namespace wrapping / WHERE / SELECT / ORDER BY, // so the child pipeline only processes rows that match parents. let pipeline: NamespacedAndKeyedStream = initialPipeline - if (!isUnionFrom && parentKeyStream && childCorrelationField) { + const childCorrelationAlias = childCorrelationField?.path[0] + const joinsParentDirectly = + !isUnionFrom && + !isParentRouted && + parentKeyStream !== undefined && + childCorrelationField !== undefined && + childCorrelationAlias === mainSource + if (parentKeyStream && childCorrelationField && joinsParentDirectly) { const mainInput = sources[mainSource]! let filteredMainInput = mainInput - // Re-key child input by correlation field: [correlationValue, [childKey, childRow]] + // Join on query equality rather than raw JavaScript identity. Keep the raw + // child value beside the row so result routing can still expose it. const childFieldPath = childCorrelationField.path.slice(1) // remove alias prefix const childRekeyed = mainInput.pipe( map(([key, row]: [unknown, any]) => { const correlationValue = getNestedValue(row, childFieldPath) - return [correlationValue, [key, row]] as [unknown, [unknown, any]] + return [ + valueIdentity.serializeEquality(correlationValue), + [key, row, correlationValue], + ] as [unknown, [unknown, any, unknown]] }), ) + const equalityParentKeys = parentKeyStream.pipe( + map(([correlationValue, parentContext]: [unknown, unknown]) => [ + valueIdentity.serializeEquality(correlationValue), + parentContext, + ]), + reduce((values: Array<[unknown, number]>) => + values.map(([value, multiplicity]) => [ + value, + multiplicity > 0 ? 1 : 0, + ]), + ), + ) + // Inner join: only children whose correlation key exists in parent keys pass through - const joined = childRekeyed.pipe(joinOperator(parentKeyStream, `inner`)) + const joined = childRekeyed.pipe(joinOperator(equalityParentKeys, `inner`)) // Extract: [correlationValue, [[childKey, childRow], parentContext]] → [childKey, childRow] - // Tag the row with __correlationKey for output routing - // If parentSide is non-null (parent context projected), attach as __parentContext + // Keep routing metadata outside the user-visible row namespace. filteredMainInput = joined.pipe( filter(([_correlationValue, [childSide]]: any) => { return childSide != null }), - map(([correlationValue, [childSide, parentSide]]: any) => { - const [childKey, childRow] = childSide - const tagged: any = { ...childRow, __correlationKey: correlationValue } - if (parentSide != null) { - tagged.__parentContext = parentSide - } + map(([_correlationIdentity, [childSide, parentSide]]: any) => { + const [childKey, childRow, correlationValue] = childSide + const tagged: any = attachRouteMetadata( + { + ...childRow, + [INCLUDES_PUBLIC_KEY]: childKey, + }, + correlationValue, + parentSide, + ) const effectiveKey = parentSide != null - ? `${String(childKey)}::${JSON.stringify(parentSide)}` + ? serializeValue([ + valueIdentity.equality(childKey), + getParentContextIdentity(parentSide), + ]) : childKey return [effectiveKey, tagged] }), @@ -283,6 +525,15 @@ export function compileQuery( sources[mainSource] = filteredMainInput pipeline = wrapInputWithAlias(filteredMainInput, mainSource) + } else if (parentKeyStream && !isParentRouted) { + // QueryRefs, unions, and joined-source correlations need the route before + // source-local joins, filters, grouping, ordering, or windows run. + pipeline = parameterizeByParentRoutes( + initialPipeline, + parentKeyStream, + mainSource, + valueIdentity, + ) } // Process JOIN clauses if they exist @@ -307,9 +558,27 @@ export function compileQuery( aliasToCollectionId, aliasRemapping, sourceWhereClauses, + parentKeyStream !== undefined, + valueIdentity, + parentKeyStream, ) } + // A recursively compiled source or a correlation owned by a joined source + // is already parameterized by route. Once the correlation field is visible, + // retain only the copy whose route key matches it. + if (parentKeyStream && childCorrelationField && !joinsParentDirectly) { + const compiledChildCorrelation = compileExpression(childCorrelationField) + pipeline = pipeline.pipe( + filter(([, row]) => + correlationValuesEqual( + compiledChildCorrelation(row), + getRowCorrelationKey(row, mainSource), + ), + ), + ) as NamespacedAndKeyedStream + } + // Process the WHERE clause if it exists if (query.where && query.where.length > 0) { // Apply each WHERE condition as a filter (they are ANDed together) @@ -329,7 +598,10 @@ export function compileQuery( for (const fnWhere of query.fnWhere) { pipeline = pipeline.pipe( filter(([_key, namespacedRow]) => { - return toBooleanPredicate(fnWhere(namespacedRow)) + const callbackRow = sourceCarriesInternalRouteState + ? (stripInternalCallbackMetadata(namespacedRow) as NamespacedRow) + : namespacedRow + return toBooleanPredicate(fnWhere(callbackRow)) }), ) } @@ -338,15 +610,17 @@ export function compileQuery( // Extract includes from SELECT, compile child pipelines, and replace with placeholders. // This must happen AFTER WHERE (so parent pipeline is filtered) but BEFORE processSelect // (so IncludesSubquery nodes are stripped before select compilation). - const includesResults: Array = !query.select + const inputIncludes = [ + ...directIncludes, + ...sourceIncludes.map(({ include }) => include), + ] + const materializeSelectInput = !!query.fnSelect && inputIncludes.length > 0 + let includesResults: Array = !query.select ? [...directIncludes] : [] - const includesRoutingFns: Array<{ + let includesRoutingFns: Array<{ fieldName: string - getRouting: (nsRow: any) => { - correlationKey: unknown - parentContext: Record | null - } + getRouting: (nsRow: any) => IncludeRouting }> = [] for (const { sourceAlias, include } of sourceIncludes) { const projectedPaths = @@ -356,7 +630,7 @@ export function compileQuery( sourceAlias, include.resultPath, ) - : query.fnSelect + : query.fnSelect && !materializeSelectInput ? [] : [ { @@ -374,29 +648,18 @@ export function compileQuery( `${sourceAlias}.${resultPath.join(`.`)}`, includesRoutingFns, ) - const compiledGuards = guards.map((guard) => ({ - condition: compileExpression(guard.condition), - expected: guard.expected, - })) includesResults.push({ ...include, fieldName, resultPath, }) - includesRoutingFns.push({ fieldName, - getRouting: (nsRow: any) => { - if (!matchesConditionalSelectGuards(compiledGuards, nsRow)) { - return { correlationKey: null, parentContext: null } - } - return ( - nsRow[sourceAlias]?.[INCLUDES_ROUTING]?.[include.fieldName] ?? { - correlationKey: null, - parentContext: null, - } - ) - }, + getRouting: compileGuardedRouting( + guards, + (nsRow) => + nsRow[sourceAlias]?.[INCLUDES_ROUTING]?.[include.fieldName], + ), }) } } @@ -412,30 +675,17 @@ export function compileQuery( resultPath.join(`.`), includesRoutingFns, ) - const compiledGuards = guards.map((guard) => ({ - condition: compileExpression(guard.condition), - expected: guard.expected, - })) - includesResults.push({ ...include, fieldName, resultPath, }) - includesRoutingFns.push({ fieldName, - getRouting: (nsRow: any) => { - if (!matchesConditionalSelectGuards(compiledGuards, nsRow)) { - return { correlationKey: null, parentContext: null } - } - return ( - nsRow[INCLUDES_ROUTING]?.[include.fieldName] ?? { - correlationKey: null, - parentContext: null, - } - ) - }, + getRouting: compileGuardedRouting( + guards, + (nsRow) => nsRow[INCLUDES_ROUTING]?.[include.fieldName], + ), }) } } @@ -450,52 +700,31 @@ export function compileQuery( // Branch parent pipeline: map to [correlationValue, parentContext] // When parentProjection exists, project referenced parent fields; otherwise null (zero overhead) const compiledCorrelation = compileExpression(subquery.correlationField) - const compiledGuards = guards.map((guard) => ({ - condition: compileExpression(guard.condition), - expected: guard.expected, - })) - let parentKeys: any - if (subquery.parentProjection && subquery.parentProjection.length > 0) { - const compiledProjections = subquery.parentProjection.map((ref) => ({ + const compiledProjections: Array = + subquery.parentProjection?.map((ref) => ({ alias: ref.path[0]!, field: ref.path.slice(1), compiled: compileExpression(ref), - })) - parentKeys = pipeline.pipe( - map(([_key, nsRow]: any) => { - if (!matchesConditionalSelectGuards(compiledGuards, nsRow)) { - return [SKIP_INCLUDE, null] as any - } - const parentContext: Record> = {} - for (const proj of compiledProjections) { - if (!parentContext[proj.alias]) { - parentContext[proj.alias] = {} - } - const value = proj.compiled(nsRow) - // Set nested field in the alias namespace - let target = parentContext[proj.alias]! - for (let i = 0; i < proj.field.length - 1; i++) { - if (!target[proj.field[i]!]) { - target[proj.field[i]!] = {} - } - target = target[proj.field[i]!] - } - target[proj.field[proj.field.length - 1]!] = value - } - return [compiledCorrelation(nsRow), parentContext] as any - }), - ) - } else { - parentKeys = pipeline.pipe( - map(([_key, nsRow]: any) => { - if (!matchesConditionalSelectGuards(compiledGuards, nsRow)) { - return [SKIP_INCLUDE, null] as any - } - return [compiledCorrelation(nsRow), null] as any - }), - ) - } - parentKeys = parentKeys.pipe( + })) ?? [] + // One routing function serves both the parent-key branch and the + // INCLUDES_ROUTING tag on $selected. + const getRouting = compileGuardedRouting(guards, (nsRow) => ({ + active: true, + correlationKey: compiledCorrelation(nsRow), + parentContext: + compiledProjections.length > 0 + ? projectParentContext(nsRow, compiledProjections, valueIdentity) + : null, + })) + let parentKeys: any = pipeline.pipe( + map(([_key, nsRow]: any) => { + const routing = getRouting(nsRow) + return ( + routing.active + ? [routing.correlationKey, routing.parentContext] + : [SKIP_INCLUDE, null] + ) as any + }), filter(([correlationValue]: any) => correlationValue !== SKIP_INCLUDE), ) @@ -511,7 +740,7 @@ export function compileQuery( // --- Includes lazy loading (mirrors join lazy loading in joins.ts) --- // Resolve the child correlation field to concrete collection targets so // subquery and union child sources can load by branch when it is safe. - const childCorrelationAlias = subquery.childCorrelationField.path[0]! + const childSourceAlias = subquery.childCorrelationField.path[0]! const directChildCollection = subquery.query.from.type === `collectionRef` ? subquery.query.from.collection @@ -519,7 +748,7 @@ export function compileQuery( const lazyTargets = getLazyLoadTargets( subquery.query, subquery.query.from, - childCorrelationAlias, + childSourceAlias, subquery.childCorrelationField, directChildCollection, aliasRemapping, @@ -528,7 +757,7 @@ export function compileQuery( if (lazyTargets.length > 0) { // 1. Mark child source as lazy so CollectionSubscriber skips initial full load for (const target of lazyTargets) { - lazySources.add(target.alias) + lazySources.add(target.sourceId) } // 2. Ensure an index on the correlation field for efficient lookups @@ -539,40 +768,46 @@ export function compileQuery( } } - // 3. Tap parent keys to intercept correlation values and request - // matching child rows on-demand via the child's subscription + const initialKeys = getStaticDemandKeys( + rawQuery, + subquery.correlationField, + ) + const demandPlans = lazyTargets.map((target) => + registerLazyDemandPlan(callbacks, target, initialKeys), + ) + const demandWeights = new Map< + string, + { key: unknown; weight: number } + >() + + // Keep the async demand adapter in sync with the current parent-key + // relation. Retired keys stop participating in readiness immediately. parentKeys = parentKeys.pipe( tap((data: any) => { - const joinKeys = [ - ...new Set( - data - .getInner() - .map( - ([[correlationValue]]: any) => correlationValue as unknown, - ) - .filter((joinKey: unknown) => joinKey != null), - ), - ] - - if (joinKeys.length === 0) { - return - } - - for (const target of lazyTargets) { - const lazySourceSubscription = subscriptions[target.alias] - - if (!lazySourceSubscription) { - continue - } - - if (lazySourceSubscription.hasLoadedInitialState()) { - continue + for (const [[correlationValue], weight] of data.getInner()) { + if (correlationValue == null) continue + const encoded = valueIdentity.serializeEquality(correlationValue) + const previous = demandWeights.get(encoded) + const nextWeight = (previous?.weight ?? 0) + weight + if (nextWeight === 0) { + demandWeights.delete(encoded) + } else { + demandWeights.set(encoded, { + key: correlationValue, + weight: nextWeight, + }) } + } - const lazyJoinRef = new PropRef(target.path) - lazySourceSubscription.requestSnapshot({ - where: inArray(lazyJoinRef, joinKeys), - }) + const keys = new Set( + [...demandWeights.values()] + .filter(({ weight }) => weight > 0) + .map(({ key: demandedKey }) => demandedKey), + ) + for (let index = 0; index < lazyTargets.length; index++) { + const target = lazyTargets[index]! + const plan = demandPlans[index]! + callbacks[target.sourceId]?.setDemand?.(plan, keys) } }), ) @@ -628,54 +863,7 @@ export function compileQuery( scalarField: subquery.scalarField, }) - // Capture routing function for INCLUDES_ROUTING tagging - if (subquery.parentProjection && subquery.parentProjection.length > 0) { - const compiledProjs = subquery.parentProjection.map((ref) => ({ - alias: ref.path[0]!, - field: ref.path.slice(1), - compiled: compileExpression(ref), - })) - const compiledCorr = compiledCorrelation - const compiledRoutingGuards = compiledGuards - includesRoutingFns.push({ - fieldName, - getRouting: (nsRow: any) => { - if (!matchesConditionalSelectGuards(compiledRoutingGuards, nsRow)) { - return { correlationKey: null, parentContext: null } - } - const parentContext: Record> = {} - for (const proj of compiledProjs) { - if (!parentContext[proj.alias]) { - parentContext[proj.alias] = {} - } - const value = proj.compiled(nsRow) - let target = parentContext[proj.alias]! - for (let i = 0; i < proj.field.length - 1; i++) { - if (!target[proj.field[i]!]) { - target[proj.field[i]!] = {} - } - target = target[proj.field[i]!] - } - target[proj.field[proj.field.length - 1]!] = value - } - return { correlationKey: compiledCorr(nsRow), parentContext } - }, - }) - } else { - const compiledRoutingGuards = compiledGuards - includesRoutingFns.push({ - fieldName, - getRouting: (nsRow: any) => { - if (!matchesConditionalSelectGuards(compiledRoutingGuards, nsRow)) { - return { correlationKey: null, parentContext: null } - } - return { - correlationKey: compiledCorrelation(nsRow), - parentContext: null, - } - }, - }) - } + includesRoutingFns.push({ fieldName, getRouting }) // Replace includes entry in select with a null placeholder query = { @@ -698,48 +886,111 @@ export function compileQuery( throw new FnSelectWithGroupByError() } + const selectHasAggregates = + query.select !== undefined && containsAggregate(query.select) + const routingFns = includesRoutingFns + const getRowIncludesRouting = (row: NamespacedRow) => + Object.fromEntries( + routingFns.map(({ fieldName, getRouting }) => [ + fieldName, + getRouting(row), + ]), + ) + if (materializeSelectInput) { + if (!inputIncludes.every(isInlineInclude)) { + throw new Error( + `fn.select() cannot consume Collection-valued includes. Use toArray() or materialize() in the upstream select(), or use an expression select() to keep live Collections.`, + ) + } + // Input paths belong before the callback: its arbitrary output may rename + // or discard them. Inline values need no public Collection boundary. + const inputPipeline = pipeline.pipe( + map( + ([key, row]: [ + unknown, + NamespacedRow & { [INCLUDES_ROUTING]?: object }, + ]) => [ + key, + [ + { + ...row, + [INCLUDES_ROUTING]: { + ...row[INCLUDES_ROUTING], + ...getRowIncludesRouting(row), + }, + }, + undefined, + ], + ], + ), + ) as ResultStream + const materializedInput = materializeCompilation({ + pipeline: inputPipeline, + includes: includesResults, + valueIdentity, + collectionId: mainCollectionId, + sourceWhereClauses, + aliasToCollectionId, + aliasRemapping, + }) + pipeline = materializedInput.pipeline.pipe( + map(([key, [value]]) => { + const row = { ...value } + delete row[INCLUDES_ROUTING] + return [key, row] + }), + ) as NamespacedAndKeyedStream + includesResults = [] + includesRoutingFns = [] + } + // Process the SELECT clause early - always create $selected // This eliminates duplication and allows for DISTINCT implementation if (query.fnSelect) { + const fnSelect = (row: NamespacedRow) => { + const selected = query.fnSelect!(row) + validateFnSelectResult(selected) + return selected + } // Handle functional select - apply the function to transform the row - pipeline = pipeline.pipe( - map(([key, namespacedRow]) => { - const selectResults = query.fnSelect!(namespacedRow) - if (selectResults && typeof selectResults === `object`) { - const routing = (namespacedRow as any)[INCLUDES_ROUTING] - if (routing) { - selectResults[INCLUDES_ROUTING] = routing - } - if (directIncludes.length > 0) { - Object.defineProperty(selectResults, FN_SELECT_STATE, { - value: { - sourceRow: namespacedRow, - fnSelect: query.fnSelect!, - }, - enumerable: true, - configurable: true, - }) - } + const projectRow = (namespacedRow: NamespacedRow) => { + const callbackRow = sourceCarriesInternalRouteState + ? (stripInternalCallbackMetadata(namespacedRow) as NamespacedRow) + : namespacedRow + const selectResults = fnSelect(callbackRow) + let selected = selectResults + if ( + selectResults && + typeof selectResults === `object` && + (Array.isArray(selectResults) || isPlainObject(selectResults)) + ) { + selected = Array.isArray(selectResults) + ? [...selectResults] + : { ...selectResults } + const routing = (namespacedRow as any)[INCLUDES_ROUTING] + if (routing) { + selected[INCLUDES_ROUTING] = routing } - return [ - key, - { - ...namespacedRow, - $selected: selectResults, - }, - ] as [string, typeof namespacedRow & { $selected: any }] - }), - ) + } + return { + ...namespacedRow, + $selected: selected, + } + } + pipeline = pipeline.pipe(map(([key, row]) => [key, projectRow(row)])) } else if (query.select) { pipeline = processSelect(pipeline, query.select, allInputs) } else { // If no SELECT clause, create $selected with the main table data pipeline = pipeline.pipe( map(([key, namespacedRow]) => { + const routedScalar = getRoutedScalarMetadata(namespacedRow) const selectResults = - !isUnionFrom && !query.join && !query.groupBy - ? namespacedRow[mainSource] - : namespacedRow + isUnionFrom && routedScalar + ? routedScalar.value + : !isUnionFrom && !query.join && !query.groupBy + ? namespacedRow[mainSource] + : namespacedRow return [ key, @@ -752,56 +1003,49 @@ export function compileQuery( ) } - // Tag $selected with routing metadata for includes. - // This lets collection-config-builder extract routing info (correlationKey + parentContext) - // from parent results without depending on the user's select. + // Tag $selected with routing metadata so the materialization graph can route + // children without depending on the user's projection. if (includesRoutingFns.length > 0) { pipeline = pipeline.pipe( map(([key, namespacedRow]: any) => { - const routing: Record< - string, - { correlationKey: unknown; parentContext: Record | null } - > = {} - for (const { fieldName, getRouting } of includesRoutingFns) { - routing[fieldName] = getRouting(namespacedRow) - } - namespacedRow.$selected[INCLUDES_ROUTING] = routing - return [key, namespacedRow] + const selected = Array.isArray(namespacedRow.$selected) + ? [...namespacedRow.$selected] + : { ...namespacedRow.$selected } + selected[INCLUDES_ROUTING] = getRowIncludesRouting(namespacedRow) + return [key, { ...namespacedRow, $selected: selected }] }), ) } // Process the GROUP BY clause if it exists. // When in includes mode (parentKeyStream), pass mainSource so that groupBy - // preserves __correlationKey for per-parent aggregation. + // preserves route metadata for per-parent aggregation. const groupByMainSource = parentKeyStream ? mainSource : undefined if (query.groupBy && query.groupBy.length > 0) { pipeline = processGroupBy( pipeline, query.groupBy, + valueIdentity, query.having, query.select, query.fnHaving, mainCollectionId, groupByMainSource, + sourceCarriesInternalRouteState || includesRoutingFns.length > 0, ) - } else if (query.select) { - // Check if SELECT contains aggregates but no GROUP BY (implicit single-group aggregation) - const hasAggregates = Object.values(query.select).some( - (expr) => expr.type === `agg` || containsAggregate(expr), + } else if (selectHasAggregates) { + // SELECT contains aggregates but no GROUP BY: implicit single-group aggregation + pipeline = processGroupBy( + pipeline, + [], // Empty group by means single group + valueIdentity, + query.having, + query.select, + query.fnHaving, + mainCollectionId, + groupByMainSource, + sourceCarriesInternalRouteState || includesRoutingFns.length > 0, ) - if (hasAggregates) { - // Handle implicit single-group aggregation - pipeline = processGroupBy( - pipeline, - [], // Empty group by means single group - query.having, - query.select, - query.fnHaving, - mainCollectionId, - groupByMainSource, - ) - } } // Process the HAVING clause if it exists (only applies after GROUP BY) @@ -826,18 +1070,70 @@ export function compileQuery( for (const fnHaving of query.fnHaving) { pipeline = pipeline.pipe( filter(([_key, namespacedRow]) => { - return fnHaving(namespacedRow) + const callbackRow = + sourceCarriesInternalRouteState || includesRoutingFns.length > 0 + ? (stripInternalCallbackMetadata(namespacedRow) as NamespacedRow) + : namespacedRow + return fnHaving(callbackRow) }), ) } } + // Normalize every logical row before DISTINCT and ordering. Those operators + // track visibility by row key, so an insert-before-delete replacement with + // the same key would otherwise keep the old value and hide route or order + // changes. Joined contributors may differ in unselected namespaces; only + // the public value and its route/order inputs must be congruent. + if (!selectHasAggregates) { + pipeline = canonicalizeSelectedRows( + pipeline, + query, + mainSource, + parentKeyStream !== undefined, + ) + } + + const keyedSourceWhereClauses = keyWhereClausesBySource( + rawSources, + sourceWhereClauses, + aliasRemapping, + ) + // Process the DISTINCT clause if it exists if (query.distinct) { pipeline = pipeline.pipe(distinct(([_key, row]) => row.$selected)) } - // Process orderBy parameter if it exists + const finalizeRow = ( + key: unknown, + row: Record, + orderByIndex: string | undefined, + ) => { + const finalResults = attachVirtualPropsToSelected( + unwrapValue(row.$selected), + row, + ) + // When in includes mode, embed the correlation key and parentContext + if (parentKeyStream) { + return [ + key, + [ + stripInternalRouteMetadata(finalResults), + orderByIndex, + getRowCorrelationKey(row, mainSource), + getRowParentContext(row, mainSource), + getIncludesPublicKey(row, mainSource, key), + ], + ] as any + } + return [key, [finalResults, orderByIndex]] as [ + unknown, + [any, string | undefined], + ] + } + + let resultPipeline: ResultStream if (query.orderBy && query.orderBy.length > 0) { // When in includes mode with limit/offset, use grouped ordering so that // the limit is applied per parent (per correlation key), not globally. @@ -845,16 +1141,25 @@ export function compileQuery( parentKeyStream && (query.limit !== undefined || query.offset !== undefined) ? (_key: unknown, row: unknown) => { - const correlationKey = (row as any)?.[mainSource]?.__correlationKey - const parentContext = (row as any)?.__parentContext + const correlationKey = getRowCorrelationKey( + row as NamespacedRow, + mainSource, + ) + const parentContext = getRowParentContext( + row as NamespacedRow, + mainSource, + ) if (parentContext != null) { - return JSON.stringify([correlationKey, parentContext]) + return serializeValue([ + valueIdentity.equality(correlationKey), + getParentContextIdentity(parentContext), + ]) } - return correlationKey + return valueIdentity.equality(correlationKey) } : undefined - const orderedPipeline = processOrderBy( + resultPipeline = processOrderBy( rawQuery, pipeline, query.orderBy, @@ -865,94 +1170,124 @@ export function compileQuery( query.limit, query.offset, includesGroupKeyFn, - ) - - // Final step: extract the $selected and include orderBy index - const resultPipeline: ResultStream = orderedPipeline.pipe( - map(([key, [row, orderByIndex]]) => { - // Extract the final results from $selected and include orderBy index - const raw = (row as any).$selected - const finalResults = attachVirtualPropsToSelected( - unwrapValue(raw), - row as Record, - ) - // When in includes mode, embed the correlation key and parentContext - if (parentKeyStream) { - const correlationKey = (row as any)[mainSource]?.__correlationKey - const parentContext = (row as any).__parentContext ?? null - // Strip internal routing properties that may leak via spread selects - delete finalResults.__correlationKey - delete finalResults.__parentContext - return [ - key, - [finalResults, orderByIndex, correlationKey, parentContext], - ] as any - } - return [key, [finalResults, orderByIndex]] as [unknown, [any, string]] - }), + ).pipe( + map(([key, [row, orderByIndex]]) => finalizeRow(key, row, orderByIndex)), ) as ResultStream - - const result = resultPipeline - // Cache the result before returning (use original query as key) - const compilationResult: CompilationResult = { - collectionId: mainCollectionId, - pipeline: result, - sourceWhereClauses, - aliasToCollectionId, - aliasRemapping, - includes: includesResults.length > 0 ? includesResults : undefined, - } - cache.set(rawQuery, compilationResult) - - return compilationResult } else if (query.limit !== undefined || query.offset !== undefined) { - // If there's a limit or offset without orderBy, throw an error throw new LimitOffsetRequireOrderByError() + } else { + resultPipeline = pipeline.pipe( + map(([key, row]) => finalizeRow(key, row, undefined)), + ) as ResultStream } - // Final step: extract the $selected and return tuple format (no orderBy) - const resultPipeline: ResultStream = pipeline.pipe( - map(([key, row]) => { - // Extract the final results from $selected and return [key, [results, undefined]] - const raw = (row as any).$selected - const finalResults = attachVirtualPropsToSelected( - unwrapValue(raw), - row as Record, - ) - // When in includes mode, embed the correlation key and parentContext - if (parentKeyStream) { - const correlationKey = (row as any)[mainSource]?.__correlationKey - const parentContext = (row as any).__parentContext ?? null - // Strip internal routing properties that may leak via spread selects - delete finalResults.__correlationKey - delete finalResults.__parentContext - return [ - key, - [finalResults, undefined, correlationKey, parentContext], - ] as any - } - return [key, [finalResults, undefined]] as [ - unknown, - [any, string | undefined], - ] - }), - ) - - const result = resultPipeline // Cache the result before returning (use original query as key) const compilationResult: CompilationResult = { collectionId: mainCollectionId, - pipeline: result, - sourceWhereClauses, + pipeline: resultPipeline, + valueIdentity, + sourceWhereClauses: keyedSourceWhereClauses, aliasToCollectionId, aliasRemapping, includes: includesResults.length > 0 ? includesResults : undefined, } - cache.set(rawQuery, compilationResult) + if (parentKeyStream === undefined) cache.set(rawQuery, compilationResult) return compilationResult } +function isInlineInclude(include: IncludesCompilationResult): boolean { + return ( + include.materialization !== `collection` && + (include.childCompilationResult.includes ?? []).every(isInlineInclude) + ) +} + +function keyWhereClausesBySource( + sources: Array, + clauses: Map>, + aliasRemapping: Record, +): Map> { + const sourceIds = new Set(sources.map(({ sourceId }) => sourceId)) + const result = new Map>() + for (const [key, clause] of clauses) { + if (sourceIds.has(key)) { + result.set(key, clause) + continue + } + + const alias = aliasRemapping[key] ?? key + for (const source of sources) { + if (source.alias === alias) result.set(source.sourceId, clause) + } + } + return result +} + +function bindSourceInputs( + sources: Array, + inputs: Record, +): void { + for (const source of sources) { + const input = inputs[source.sourceId] ?? inputs[source.alias] + if (!input) continue + inputs[source.sourceId] = input + inputs[source.alias] = input + } +} + +function canonicalizeSelectedRows( + pipeline: NamespacedAndKeyedStream, + query: QueryIR, + mainSource: string, + isIncludedRelation: boolean, +): NamespacedAndKeyedStream { + const compiledOrder = (query.orderBy ?? []).map(({ expression }) => + compileExpression(expression), + ) + const signature = (row: any) => ({ + value: row.$selected, + routing: row.$selected?.[INCLUDES_ROUTING], + outerCorrelation: isIncludedRelation + ? getRowCorrelationKey(row, mainSource) + : undefined, + parentContext: isIncludedRelation + ? getRowParentContext(row, mainSource) + : undefined, + order: compiledOrder.map((evaluate) => evaluate(row)), + }) + + return pipeline.pipe( + reduce((values: Array<[any, number]>) => { + const totalMultiplicity = values.reduce( + (total, [, multiplicity]) => total + multiplicity, + 0, + ) + if (totalMultiplicity === 0) return [] + if (totalMultiplicity < 0) { + throw new Error(`Query row has negative multiplicity`) + } + + const visible = values.find(([, multiplicity]) => multiplicity > 0)?.[0] + if (!visible) throw new Error(`Query row has no positive contributor`) + const visibleSignature = signature(visible) + + for (const [candidate, multiplicity] of values) { + if ( + multiplicity > 0 && + !deepEquals(visibleSignature, signature(candidate)) + ) { + throw new Error( + `Query contributors with the same row key are not congruent`, + ) + } + } + + return [[visible, 1]] + }), + ) as NamespacedAndKeyedStream +} + /** * Collects aliases used for DIRECT collection references (not subqueries). * Used to validate that subqueries don't reuse parent query collection aliases. @@ -1029,6 +1364,12 @@ function validateQueryStructure( } } } + + if (query.select) { + for (const { subquery } of extractIncludesFromSelect(query.select)) { + validateQueryStructure(subquery.query, combinedAliases) + } + } } /** @@ -1049,6 +1390,7 @@ function processFromClause( aliasToCollectionId: Record, aliasRemapping: Record, sourceWhereClauses: Map>, + parentKeyStream?: KeyedStream, ): { alias: string pipeline: NamespacedAndKeyedStream @@ -1057,7 +1399,9 @@ function processFromClause( sourceIncludes: Array directIncludes: Array isUnionFrom: boolean + isParentRouted: boolean } { + const valueIdentity = getCompilationValueIdentity(cache) if (from.type === `unionAll`) { return processUnionAll( from, @@ -1073,25 +1417,28 @@ function processFromClause( aliasToCollectionId, aliasRemapping, sourceWhereClauses, + parentKeyStream, ) } if (from.type !== `unionFrom`) { - const { alias, input, collectionId, sourceIncludes } = processFrom( - from, - allInputs, - collections, - subscriptions, - callbacks, - lazySources, - optimizableOrderByCollections, - setWindowFn, - cache, - queryMapping, - aliasToCollectionId, - aliasRemapping, - sourceWhereClauses, - ) + const { alias, input, collectionId, sourceIncludes, isParentRouted } = + processFrom( + from, + allInputs, + collections, + subscriptions, + callbacks, + lazySources, + optimizableOrderByCollections, + setWindowFn, + cache, + queryMapping, + aliasToCollectionId, + aliasRemapping, + sourceWhereClauses, + parentKeyStream, + ) return { alias, @@ -1101,6 +1448,7 @@ function processFromClause( sourceIncludes, directIncludes: [], isUnionFrom: false, + isParentRouted, } } @@ -1120,6 +1468,7 @@ function processFromClause( input, collectionId, sourceIncludes: childSourceIncludes, + isParentRouted, } = processFrom( source, allInputs, @@ -1134,6 +1483,7 @@ function processFromClause( aliasToCollectionId, aliasRemapping, sourceWhereClauses, + parentKeyStream, ) if (!mainAlias) { @@ -1143,12 +1493,32 @@ function processFromClause( sources[alias] = input sourceIncludes.push(...childSourceIncludes) - const branch = wrapInputWithAlias(input, alias).pipe( + const routedBranch = + parentKeyStream && !isParentRouted + ? parameterizeByParentRoutes( + wrapInputWithAlias(input, alias), + parentKeyStream, + alias, + valueIdentity, + ) + : wrapInputWithAlias(input, alias) + const branch = routedBranch.pipe( map(([key, row]) => { - return [`${alias}:${encodeKeyForUnionBranch(key)}`, row] as [ - string, - typeof row, - ] + const branchKey = `${alias}:${encodeKeyForUnionBranch(key)}` + const aliasRow = row[alias] as Record | undefined + const publicKey = aliasRow?.[INCLUDES_PUBLIC_KEY] ?? key + const branchPublicKey = `${alias}:${encodeKeyForUnionBranch(publicKey)}` + const branchRow = parentKeyStream + ? { + ...row, + [INCLUDES_PUBLIC_KEY]: branchPublicKey, + [alias]: { + ...row[alias], + [INCLUDES_PUBLIC_KEY]: branchPublicKey, + }, + } + : row + return [branchKey, branchRow] as [string, typeof row] }), ) @@ -1163,6 +1533,7 @@ function processFromClause( sourceIncludes, directIncludes: [], isUnionFrom: true, + isParentRouted: parentKeyStream !== undefined, } } @@ -1180,6 +1551,7 @@ function processUnionAll( aliasToCollectionId: Record, aliasRemapping: Record, sourceWhereClauses: Map>, + parentKeyStream?: KeyedStream, ): { alias: string pipeline: NamespacedAndKeyedStream @@ -1188,6 +1560,7 @@ function processUnionAll( sourceIncludes: Array directIncludes: Array isUnionFrom: boolean + isParentRouted: boolean } { if (from.queries.length === 0) { throw new UnsupportedFromTypeError(`empty unionAll`) @@ -1222,6 +1595,7 @@ function processUnionAll( setWindowFn, cache, queryMapping, + parentKeyStream, ) if (!mainCollectionId) { @@ -1236,11 +1610,22 @@ function processUnionAll( } const branchPipeline = branchResult.pipeline.pipe( - map(([key, [row]]) => { - return [`${index}:${encodeKeyForUnionBranch(key)}`, row] as [ - string, - Record, - ] + map((data: any) => { + const [key, [row, _order, correlationKey, parentContext, publicKey]] = + data + const branchKey = `${index}:${encodeKeyForUnionBranch(key)}` + const branchPublicKey = `${index}:${encodeKeyForUnionBranch( + publicKey ?? key, + )}` + const routedRow = parentKeyStream + ? attachRouteMetadataToResult( + row, + correlationKey, + parentContext, + branchPublicKey, + ) + : row + return [branchKey, routedRow] as [string, Record] }), ) @@ -1257,6 +1642,7 @@ function processUnionAll( sourceIncludes, directIncludes, isUnionFrom: true, + isParentRouted: parentKeyStream !== undefined, } } @@ -1266,14 +1652,42 @@ function wrapInputWithAlias( ): NamespacedAndKeyedStream { return input.pipe( map(([key, row]) => { - // Initialize the record with a nested structure. - // If __parentContext exists (from parent-referencing includes), merge parent - // aliases into the namespaced row so WHERE can resolve parent refs. - const { __parentContext, ...cleanRow } = row as any + const inputRow: unknown = row + const scalar = getRoutedScalarMetadata(inputRow) + if (scalar) { + const nsRow = attachRouteMetadata( + { + [alias]: scalar.value, + [INCLUDES_PUBLIC_KEY]: scalar.publicKey, + }, + scalar.correlationKey, + scalar.parentContext, + ) as unknown as NamespacedRow + if ( + scalar.parentContext != null && + typeof scalar.parentContext === `object` + ) { + Object.assign(nsRow, getParentContextValue(scalar.parentContext)) + } + return [key, nsRow] as [unknown, NamespacedRow] + } + + if (inputRow == null || typeof inputRow !== `object`) { + return [key, { [alias]: inputRow }] as [unknown, NamespacedRow] + } + + // Initialize the record with a nested structure. Route metadata remains + // outside the user namespace while projected parent aliases stay visible. + const route = getRouteMetadata(inputRow) + const cleanRow = route + ? stripRouteMetadata(inputRow as Record) + : inputRow const nsRow: Record = { [alias]: cleanRow } - if (__parentContext) { - Object.assign(nsRow, __parentContext) - ;(nsRow as any).__parentContext = __parentContext + if (route?.parentContext != null) { + Object.assign(nsRow, getParentContextValue(route.parentContext)) + } + if (route) { + attachRouteMetadata(nsRow, route.correlationKey, route.parentContext) } return [key, nsRow] as [unknown, Record] }), @@ -1307,15 +1721,17 @@ function processFrom( aliasToCollectionId: Record, aliasRemapping: Record, sourceWhereClauses: Map>, + parentKeyStream?: KeyedStream, ): { alias: string input: KeyedStream collectionId: string sourceIncludes: Array + isParentRouted: boolean } { switch (from.type) { case `collectionRef`: { - const input = allInputs[from.alias] + const input = allInputs[from.sourceId] ?? allInputs[from.alias] if (!input) { throw new CollectionInputNotFoundError( from.alias, @@ -1329,6 +1745,7 @@ function processFrom( input, collectionId: from.collection.id, sourceIncludes: [], + isParentRouted: false, } } case `queryRef`: { @@ -1347,6 +1764,7 @@ function processFrom( setWindowFn, cache, queryMapping, + parentKeyStream, ) // Pull up alias mappings from subquery to parent scope. @@ -1405,9 +1823,17 @@ function processFrom( // We need to extract just the value for use in parent queries const extractedInput = subQueryInput.pipe( map((data: any) => { - const [key, [value, _orderByIndex]] = data + const [ + key, + [value, _orderByIndex, correlationKey, parentContext, publicKey], + ] = data // Unwrap Value expressions that might have leaked through as the entire row - const unwrapped = unwrapValue(value) + const unwrapped = attachRouteMetadataToResult( + unwrapValue(value), + correlationKey, + parentContext, + publicKey, + ) return [key, unwrapped] as [unknown, any] }), ) @@ -1421,6 +1847,7 @@ function processFrom( sourceAlias: from.alias, include, })) ?? [], + isParentRouted: parentKeyStream !== undefined, } } default: @@ -1445,13 +1872,18 @@ function attachVirtualPropsToSelected( selected: any, row: Record, ): any { - if (!selected || typeof selected !== `object`) { + if ( + !selected || + typeof selected !== `object` || + (!Array.isArray(selected) && !isPlainObject(selected)) + ) { return selected } + const selectedRecord = selected as Record let needsMerge = false for (const prop of VIRTUAL_PROP_NAMES) { - if (selected[prop] == null && prop in row) { + if (selectedRecord[prop] == null && prop in row) { needsMerge = true break } @@ -1461,13 +1893,28 @@ function attachVirtualPropsToSelected( return selected } + const result = ( + Array.isArray(selected) ? [...selected] : { ...selected } + ) as Record for (const prop of VIRTUAL_PROP_NAMES) { - if (selected[prop] == null && prop in row) { - selected[prop] = row[prop] + if (selectedRecord[prop] == null && prop in row) { + result[prop] = row[prop] } } - return selected + return result +} + +function getIncludesPublicKey( + row: Record, + mainSource: string, + fallback: unknown, +): unknown { + return ( + row[mainSource]?.[INCLUDES_PUBLIC_KEY] ?? + (row as any)[INCLUDES_PUBLIC_KEY] ?? + fallback + ) } /** @@ -1508,35 +1955,6 @@ function mapNestedQueries( } } -function getRefFromAlias( - query: QueryIR, - alias: string, -): CollectionRef | QueryRef | void { - for (const source of getFromSources(query.from)) { - if (source.alias === alias) { - return source - } - } - - for (const join of query.join || []) { - if (join.from.alias === alias) { - return join.from - } - } -} - -function getFromSources( - from: QueryIR[`from`], -): Array { - if (from.type === `unionFrom`) { - return from.sources - } - if (from.type === `unionAll`) { - return [] - } - return [from] -} - function getAllSources(query: QueryIR): Array { return [ ...getFromSources(query.from), @@ -1703,57 +2121,6 @@ function mapNestedFromQueries( } } -/** - * Follows the given reference in a query - * until its finds the root field the reference points to. - * @returns The collection, its alias, and the path to the root field in this collection - */ -export function followRef( - query: QueryIR, - ref: PropRef, - collection: Collection, -): { collection: Collection; path: Array } | void { - if (ref.path.length === 0) { - return - } - - if (ref.path.length === 1) { - // This field should be part of this collection - const field = ref.path[0]! - // is it part of the select clause? - if (query.select) { - const selectedField = query.select[field] - if (selectedField && selectedField.type === `ref`) { - return followRef(query, selectedField, collection) - } - } - - // Either this field is not part of the select clause - // and thus it must be part of the collection itself - // or it is part of the select but is not a reference - // so we can stop here and don't have to follow it - return { collection, path: [field] } - } - - if (ref.path.length > 1) { - // This is a nested field - const [alias, ...rest] = ref.path - const aliasRef = getRefFromAlias(query, alias!) - if (!aliasRef) { - return - } - - if (aliasRef.type === `queryRef`) { - return followRef(aliasRef.query, new PropRef(rest), collection) - } else { - // This is a reference to a collection - // we can't follow it further - // so the field must be on the collection itself - return { collection: aliasRef.collection, path: rest } - } - } -} - /** * Walks a Select object to find IncludesSubquery entries. * Plain nested objects still reject includes, but ConditionalSelect branches can @@ -1895,7 +2262,8 @@ function isNestedSelectObject(value: any): value is Record { value != null && typeof value === `object` && !Array.isArray(value) && - !isExpressionLike(value) + !isExpressionLike(value) && + value.__refProxy !== true ) } @@ -2046,16 +2414,78 @@ function getNestedValue(obj: any, path: Array): any { return value } -function matchesConditionalSelectGuards( - guards: Array<{ - condition: (row: any) => any - expected: boolean - }>, - row: any, -): boolean { - return guards.every( - (guard) => isCaseWhenConditionTrue(guard.condition(row)) === guard.expected, - ) +type IncludeRouting = { + active: boolean + correlationKey: unknown + parentContext: Record | null +} + +/** + * Compiles a select-branch guard set once and resolves the include route only + * for rows whose guards hold. Every other row is routed as inactive. + */ +function compileGuardedRouting( + guards: Array, + resolve: (nsRow: any) => IncludeRouting | undefined, +): (nsRow: any) => IncludeRouting { + const compiledGuards = guards.map((guard) => ({ + condition: compileExpression(guard.condition), + expected: guard.expected, + })) + return (nsRow) => { + const active = compiledGuards.every( + (guard) => + isCaseWhenConditionTrue(guard.condition(nsRow)) === guard.expected, + ) + return ( + (active ? resolve(nsRow) : undefined) ?? { + active: false, + correlationKey: null, + parentContext: null, + } + ) + } } export type CompileQueryFn = typeof compileQuery + +function getStaticDemandKeys(query: QueryIR, ref: PropRef): Set { + const constraints: Array> = [] + const visit = (expression: BasicExpression): void => { + if (expression.type !== `func`) return + if (expression.name === `and`) { + expression.args.forEach(visit) + return + } + if (expression.name !== `eq` && expression.name !== `in`) return + + const [left, right] = expression.args + const value = + left?.type === `ref` && + pathsEqual(left.path, ref.path) && + right instanceof ValClass + ? right.value + : right?.type === `ref` && + pathsEqual(right.path, ref.path) && + left instanceof ValClass + ? left.value + : undefined + if (value === undefined) return + constraints.push(new Set(Array.isArray(value) ? value : [value])) + } + + query.where?.forEach((where) => visit(getWhereExpression(where))) + if (constraints.length === 0) return new Set() + return new Set( + [...constraints[0]!].filter((value) => + constraints.slice(1).every((constraint) => constraint.has(value)), + ), + ) +} + +function pathsEqual(left: Array, right: Array): boolean { + return ( + left.length === right.length && + left.every((segment, index) => segment === right[index]) + ) +} diff --git a/packages/db/src/query/compiler/joins.ts b/packages/db/src/query/compiler/joins.ts index 0c37e05f4e..4dfa935bc8 100644 --- a/packages/db/src/query/compiler/joins.ts +++ b/packages/db/src/query/compiler/joins.ts @@ -1,4 +1,10 @@ -import { filter, join as joinOperator, map, tap } from '@tanstack/db-ivm' +import { + filter, + join as joinOperator, + map, + serializeValue, + tap, +} from '@tanstack/db-ivm' import { CollectionInputNotFoundError, InvalidJoinCondition, @@ -7,16 +13,30 @@ import { InvalidJoinConditionSameSourceError, InvalidJoinConditionSourceMismatchError, JoinCollectionNotFoundError, - SubscriptionNotFoundError, UnsupportedJoinSourceTypeError, UnsupportedJoinTypeError, } from '../../errors.js' import { normalizeValue } from '../../utils/comparison.js' +import { + getParentContextIdentity, + getParentContextValue, +} from '../equality-value-identity.js' import { ensureIndexForField } from '../../indexes/auto-index.js' -import { PropRef } from '../ir.js' -import { inArray } from '../builder/functions.js' +import { getFromSources } from '../ir.js' import { compileExpression } from './evaluators.js' +import { getSourceAliasesFromExpression } from './expressions.js' import { getLazyLoadTargets } from './lazy-targets.js' +import { crossJoinParentRoutes } from './parent-routes.js' +import { + INCLUDES_PUBLIC_KEY, + attachRouteMetadata, + attachRouteMetadataToResult, + getNamespacedRouteMetadata, + getRouteMetadata, + getRoutedScalarMetadata, + stripRouteMetadata, +} from './route-metadata.js' +import type { ValueIdentity } from '../equality-value-identity.js' import type { CompileQueryFn } from './index.js' import type { OrderByOptimizationInfo } from './order-by.js' import type { @@ -36,13 +56,112 @@ import type { import type { QueryCache, QueryMapping, WindowOptions } from './types.js' import type { CollectionSubscription } from '../../collection/subscription.js' -/** Function type for loading specific keys into a lazy collection */ -export type LoadKeysFn = (key: Set) => void +export type LazyDemandPlan = { + id: string + path: Array + collectionId: string + initialKeys: Set +} /** Callbacks for managing lazy-loaded collections in optimized joins */ export type LazyCollectionCallbacks = { - loadKeys: LoadKeysFn - loadInitialState: () => void + plans?: Array + setDemand?: (plan: LazyDemandPlan, keys: Set) => void +} + +let nextLazyDemandPlanId = 0 + +function parameterizeJoinInputByParentRoutes( + input: KeyedStream, + parentKeyStream: KeyedStream, + valueIdentity: ValueIdentity, +): KeyedStream { + return crossJoinParentRoutes( + input, + parentKeyStream, + (rowKey, row, correlationKey, parentContext) => { + return [ + serializeValue([ + valueIdentity.equality(rowKey), + valueIdentity.equality(correlationKey), + getParentContextIdentity(parentContext), + ]), + attachRouteMetadata( + { + ...(row as Record), + }, + correlationKey, + parentContext, + ), + ] + }, + ) +} + +function wrapJoinedInputRow(alias: string, row: any): NamespacedRow { + const scalar = getRoutedScalarMetadata(row) + if (scalar) { + const namespaced = attachRouteMetadata( + { + [alias]: scalar.value, + [INCLUDES_PUBLIC_KEY]: scalar.publicKey, + }, + scalar.correlationKey, + scalar.parentContext, + ) as unknown as NamespacedRow + if ( + scalar.parentContext != null && + typeof scalar.parentContext === `object` + ) { + Object.assign(namespaced, getParentContextValue(scalar.parentContext)) + } + return namespaced + } + + if (row == null || typeof row !== `object`) { + return { [alias]: row } + } + + const route = getRouteMetadata(row) + const cleanRow = route ? stripRouteMetadata(row) : row + const namespaced: NamespacedRow = { [alias]: cleanRow } + if (route?.parentContext != null) { + Object.assign(namespaced, getParentContextValue(route.parentContext)) + } + if (route) { + attachRouteMetadata(namespaced, route.correlationKey, route.parentContext) + } + return namespaced +} + +function getRouteJoinKey( + row: NamespacedRow, + source: string, + value: unknown, + valueIdentity: ValueIdentity, +): string { + const route = getNamespacedRouteMetadata(row, source) + return serializeValue([ + valueIdentity.equality(route?.correlationKey), + getParentContextIdentity(route?.parentContext ?? null), + valueIdentity.equality(value), + ]) +} + +export function registerLazyDemandPlan( + callbacks: Record, + target: { sourceId: string; path: Array; collection: Collection }, + initialKeys: Set = new Set(), +): LazyDemandPlan { + const plan: LazyDemandPlan = { + id: `lazy-demand-${++nextLazyDemandPlanId}`, + path: target.path, + collectionId: target.collection.id, + initialKeys: new Set(initialKeys), + } + const state = (callbacks[target.sourceId] ??= {}) + ;(state.plans ??= []).push(plan) + return plan } /** @@ -69,6 +188,9 @@ export function processJoins( aliasToCollectionId: Record, aliasRemapping: Record, sourceWhereClauses: Map>, + mainSourceIsParentFiltered: boolean, + valueIdentity: ValueIdentity, + parentKeyStream?: KeyedStream, ): NamespacedAndKeyedStream { let resultPipeline = pipeline @@ -93,6 +215,9 @@ export function processJoins( aliasToCollectionId, aliasRemapping, sourceWhereClauses, + mainSourceIsParentFiltered, + valueIdentity, + parentKeyStream, ) } @@ -123,12 +248,32 @@ function processJoin( aliasToCollectionId: Record, aliasRemapping: Record, sourceWhereClauses: Map>, + mainSourceIsParentFiltered: boolean, + valueIdentity: ValueIdentity, + parentKeyStream?: KeyedStream, ): NamespacedAndKeyedStream { const isCollectionRef = joinClause.from.type === `collectionRef` + const joinedSource = joinClause.from.alias + const availableSources = [...Object.keys(sources), joinedSource] + const { mainExpr, joinedExpr } = analyzeJoinExpressions( + joinClause.left, + joinClause.right, + availableSources, + joinedSource, + rawQuery.from.type === `unionAll`, + ) + const joinedExpressionAliases = getSourceAliasesFromExpression(joinedExpr) + const joinedExpressionUsesParent = [...joinedExpressionAliases].some( + (alias) => alias !== joinedSource && !sources[alias], + ) + const routeJoinedSource = + parentKeyStream !== undefined && + (joinClause.from.type === `queryRef` || joinedExpressionUsesParent) + // Get the joined source alias and input stream const { - alias: joinedSource, + alias: processedJoinedSource, input: joinedInput, collectionId: joinedCollectionId, } = processJoinSource( @@ -146,8 +291,14 @@ function processJoin( aliasToCollectionId, aliasRemapping, sourceWhereClauses, + valueIdentity, + routeJoinedSource ? parentKeyStream : undefined, ) + if (processedJoinedSource !== joinedSource) { + throw new InvalidJoinConditionSourceMismatchError() + } + // Add the joined source to the sources map sources[joinedSource] = joinedInput if (isCollectionRef) { @@ -167,21 +318,16 @@ function processJoin( throw new JoinCollectionNotFoundError(joinedCollectionId) } - const { activeSource, lazySource } = getActiveAndLazySources( + const sourceActivity = getActiveAndLazySources( joinClause.type, mainCollection, joinedCollection, + mainSourceIsParentFiltered, ) - - // Analyze which source each expression refers to and swap if necessary - const availableSources = Object.keys(sources) - const { mainExpr, joinedExpr } = analyzeJoinExpressions( - joinClause.left, - joinClause.right, - availableSources, - joinedSource, - rawQuery.from.type === `unionAll`, - ) + const activeSource = routeJoinedSource + ? undefined + : sourceActivity.activeSource + const lazySource = sourceActivity.lazySource // Pre-compile the join expressions const compiledMainExpr = compileExpression(mainExpr) @@ -191,7 +337,10 @@ function processJoin( let mainPipeline = pipeline.pipe( map(([currentKey, namespacedRow]) => { // Extract the join key from the main source expression - const mainKey = normalizeValue(compiledMainExpr(namespacedRow)) + const value = normalizeValue(compiledMainExpr(namespacedRow)) + const mainKey = routeJoinedSource + ? getRouteJoinKey(namespacedRow, mainSource, value, valueIdentity) + : value // Return [joinKey, [originalKey, namespacedRow]] return [mainKey, [currentKey, namespacedRow]] as [ @@ -205,10 +354,13 @@ function processJoin( let joinedPipeline = joinedInput.pipe( map(([currentKey, row]) => { // Wrap the row in a namespaced structure - const namespacedRow: NamespacedRow = { [joinedSource]: row } + const namespacedRow = wrapJoinedInputRow(joinedSource, row) // Extract the join key from the joined source expression - const joinedKey = normalizeValue(compiledJoinedExpr(namespacedRow)) + const value = normalizeValue(compiledJoinedExpr(namespacedRow)) + const joinedKey = routeJoinedSource + ? getRouteJoinKey(namespacedRow, joinedSource, value, valueIdentity) + : value // Return [joinKey, [originalKey, namespacedRow]] return [joinedKey, [currentKey, namespacedRow]] as [ @@ -258,9 +410,14 @@ function processJoin( // such that the liveQueryCollection can check it after compilation // to know which source aliases should load data lazily (not initially) for (const target of lazyTargets) { - lazySources.add(target.alias) + lazySources.add(target.sourceId) } + const demandPlans = lazyTargets.map((target) => + registerLazyDemandPlan(callbacks, target), + ) + const demandWeights = new Map() + const activePipeline = activeSource === `main` ? mainPipeline : joinedPipeline @@ -277,55 +434,26 @@ function processJoin( [key: unknown, [originalKey: string, namespacedRow: NamespacedRow]] > = activePipeline.pipe( tap((data) => { - // Deduplicate and filter null keys before requesting snapshot - const joinKeys = [ - ...new Set( - data - .getInner() - .map(([[joinKey]]) => joinKey) - .filter((key) => key != null), - ), - ] - - if (joinKeys.length === 0) { - return - } - - for (const target of lazyTargets) { - const lazySourceSubscription = subscriptions[target.alias] - - if (!lazySourceSubscription) { - throw new SubscriptionNotFoundError( - target.alias, - lazyAlias, - target.collection.id, - Object.keys(subscriptions), - ) - } - - if (lazySourceSubscription.hasLoadedInitialState()) { - // Entire state was already loaded because we deoptimized the join - continue + for (const [[joinKey], weight] of data.getInner()) { + if (joinKey == null) continue + const encoded = valueIdentity.serializeEquality(joinKey) + const previous = demandWeights.get(encoded) + const nextWeight = (previous?.weight ?? 0) + weight + if (nextWeight === 0) { + demandWeights.delete(encoded) + } else { + demandWeights.set(encoded, { key: joinKey, weight: nextWeight }) } + } - const lazyJoinRef = new PropRef(target.path) - const loaded = lazySourceSubscription.requestSnapshot({ - where: inArray(lazyJoinRef, joinKeys), - optimizedOnly: true, - }) - - if (!loaded) { - // Snapshot wasn't sent because it could not be loaded from the indexes - const collectionId = target.collection.id - const fieldPath = target.path.join(`.`) - console.warn( - `[TanStack DB]${collectionId ? ` [${collectionId}]` : ``} Join requires an index on "${fieldPath}" for efficient loading. ` + - `Falling back to loading all data. ` + - `Consider creating an index on the collection with collection.createIndex((row) => row.${fieldPath}) ` + - `or enable auto-indexing with autoIndex: 'eager' and a defaultIndexType.`, - ) - lazySourceSubscription.requestSnapshot() - } + const keys = new Set( + [...demandWeights.values()] + .filter(({ weight }) => weight > 0) + .map(({ key }) => key), + ) + for (let index = 0; index < lazyTargets.length; index++) { + const target = lazyTargets[index]! + callbacks[target.sourceId]?.setDemand?.(demandPlans[index]!, keys) } }), ) @@ -425,30 +553,6 @@ function analyzeJoinExpressions( throw new InvalidJoinCondition() } -/** - * Extracts the source alias from a join expression - */ -function getSourceAliasesFromExpression(expr: BasicExpression): Set { - switch (expr.type) { - case `ref`: - // PropRef path has the source alias as the first element - return new Set(expr.path[0] ? [expr.path[0]] : []) - case `func`: { - // For function expressions, we need to check if all arguments refer to the same source - const sourceAliases = new Set() - for (const arg of expr.args) { - for (const alias of getSourceAliasesFromExpression(arg)) { - sourceAliases.add(alias) - } - } - return sourceAliases - } - default: - // Values (type='val') don't reference any source - return new Set() - } -} - /** * Processes the join source (collection or sub-query) */ @@ -467,10 +571,12 @@ function processJoinSource( aliasToCollectionId: Record, aliasRemapping: Record, sourceWhereClauses: Map>, + valueIdentity: ValueIdentity, + parentKeyStream?: KeyedStream, ): { alias: string; input: KeyedStream; collectionId: string } { switch (from.type) { case `collectionRef`: { - const input = allInputs[from.alias] + const input = allInputs[from.sourceId] ?? allInputs[from.alias] if (!input) { throw new CollectionInputNotFoundError( from.alias, @@ -479,7 +585,17 @@ function processJoinSource( ) } aliasToCollectionId[from.alias] = from.collection.id - return { alias: from.alias, input, collectionId: from.collection.id } + return { + alias: from.alias, + input: parentKeyStream + ? parameterizeJoinInputByParentRoutes( + input, + parentKeyStream, + valueIdentity, + ) + : input, + collectionId: from.collection.id, + } } case `queryRef`: { // Find the original query for caching purposes @@ -497,6 +613,7 @@ function processJoinSource( setWindowFn, cache, queryMapping, + parentKeyStream, ) // Pull up alias mappings from subquery to parent scope. @@ -554,8 +671,22 @@ function processJoinSource( // We need to extract just the value for use in parent queries const extractedInput = subQueryInput.pipe( map((data: any) => { - const [key, [value, _orderByIndex]] = data - return [key, value] as [unknown, any] + const [ + key, + [value, _orderByIndex, correlationKey, parentContext, publicKey], + ] = data + if (!parentKeyStream) { + return [key, value] as [unknown, any] + } + return [ + key, + attachRouteMetadataToResult( + value, + correlationKey, + parentContext, + publicKey, + ), + ] as [unknown, any] }), ) @@ -571,15 +702,7 @@ function processJoinSource( } function getFirstFromAlias(query: QueryIR): string | undefined { - if (query.from.type === `unionFrom`) { - return query.from.sources[0]?.alias - } - - if (query.from.type === `unionAll`) { - return undefined - } - - return query.from.alias + return getFromSources(query.from)[0]?.alias } /** @@ -663,6 +786,7 @@ function getActiveAndLazySources( joinType: JoinClause[`type`], leftCollection: Collection, rightCollection: Collection, + mainSourceIsParentFiltered: boolean, ): | { activeSource: `main` | `joined`; lazySource: Collection } | { activeSource: undefined; lazySource: undefined } { @@ -675,6 +799,13 @@ function getActiveAndLazySources( case `right`: return { activeSource: `joined`, lazySource: leftCollection } case `inner`: + // A correlated include has already reduced the main relation to active + // parent routes. Keep that relation active and load the joined side from + // its keys; reversing the join would create an independent demand plan + // that widens the correlated source back to a union of constraints. + if (mainSourceIsParentFiltered) { + return { activeSource: `main`, lazySource: rightCollection } + } // The smallest collection should be the active collection // and the biggest collection should be lazy return leftCollection.size < rightCollection.size diff --git a/packages/db/src/query/compiler/lazy-targets.ts b/packages/db/src/query/compiler/lazy-targets.ts index ee37c144f8..241ccd7764 100644 --- a/packages/db/src/query/compiler/lazy-targets.ts +++ b/packages/db/src/query/compiler/lazy-targets.ts @@ -1,4 +1,4 @@ -import { PropRef, followRef } from '../ir.js' +import { PropRef, followRef, getFromSources } from '../ir.js' import type { BasicExpression, CollectionRef, @@ -9,6 +9,7 @@ import type { import type { Collection } from '../../collection/index.js' export type LazyLoadTarget = { + sourceId: string alias: string collection: Collection path: Array @@ -49,9 +50,25 @@ export function getLazyLoadTargets( return [] } + const alias = followRefResult.alias || aliasRemapping[lazyAlias] || lazyAlias + const source = resolveLazySource(rawQuery, lazyFrom, { + alias, + collection: followRefResult.collection, + }) + if (!source) { + return [] + } + + // The subscription we drive lazy loading through must be the one for the + // collection the join key actually resolves to. When the key traces through a + // subquery's select into a *joined* source, that collection differs from the + // subquery's from clause (which is what `aliasRemapping[lazyAlias]` yields), + // so prefer the alias reported by `followRef`. Fall back to the from-clause + // remapping when the key resolves directly to the from source. return [ { - alias: aliasRemapping[lazyAlias] || lazyAlias, + sourceId: source.sourceId, + alias, collection: followRefResult.collection, path: followRefResult.path, }, @@ -143,7 +160,14 @@ function getTargetsFromPropRef( } if (source.type === `collectionRef`) { - return [{ alias: source.alias, collection: source.collection, path }] + return [ + { + sourceId: source.sourceId, + alias: source.alias, + collection: source.collection, + path, + }, + ] } if (source.query.limit || source.query.offset) { @@ -165,14 +189,63 @@ function getSourceFromAlias( } } - const from = query.from - const sources = - from.type === `unionFrom` - ? from.sources - : from.type === `unionAll` - ? [] - : [from] - return sources.find((source) => source.alias === alias) + return getFromSources(query.from).find((source) => source.alias === alias) +} + +function resolveLazySource( + query: QueryIR, + lazyFrom: From, + target: { alias: string; collection: Collection }, +): CollectionRef | undefined { + // Prefer the lexical source from the user's query. The optimizer may create + // an equivalent CollectionRef with a new source ID, but subscriptions and + // demand callbacks are owned by the original lexical source. + const source = findCollectionSource(query, target.alias, target.collection) + if (source) return source + + if ( + lazyFrom.type === `collectionRef` && + lazyFrom.collection === target.collection && + lazyFrom.alias === target.alias + ) { + return lazyFrom + } + + return undefined +} + +function findCollectionSource( + query: QueryIR, + alias: string, + collection: Collection, +): CollectionRef | undefined { + const sources = [ + ...getFromSources(query.from), + ...(query.join?.map((join) => join.from) ?? []), + ] + + for (const source of sources) { + if ( + source.type === `collectionRef` && + source.alias === alias && + source.collection === collection + ) { + return source + } + if (source.type === `queryRef`) { + const nested = findCollectionSource(source.query, alias, collection) + if (nested) return nested + } + } + + if (query.from.type === `unionAll`) { + for (const branch of query.from.queries) { + const nested = findCollectionSource(branch, alias, collection) + if (nested) return nested + } + } + + return undefined } function dedupeLazyLoadTargets( @@ -181,7 +254,7 @@ function dedupeLazyLoadTargets( const seen = new Set() const deduped: Array = [] for (const target of targets) { - const key = `${target.alias}:${target.path.join(`.`)}` + const key = `${target.sourceId}:${target.path.join(`.`)}` if (!seen.has(key)) { seen.add(key) deduped.push(target) diff --git a/packages/db/src/query/compiler/order-by.ts b/packages/db/src/query/compiler/order-by.ts index d9aede85e6..fb4843fbe0 100644 --- a/packages/db/src/query/compiler/order-by.ts +++ b/packages/db/src/query/compiler/order-by.ts @@ -3,10 +3,17 @@ import { orderByWithFractionalIndex, } from '@tanstack/db-ivm' import { defaultComparator, makeComparator } from '../../utils/comparison.js' -import { PropRef, followRef } from '../ir.js' +import { + PropRef, + collectCollectionSources, + followRef, + getWhereExpression, + isResidualWhere, +} from '../ir.js' import { ensureIndexForField } from '../../indexes/auto-index.js' import { findIndexForField } from '../../utils/index-optimization.js' import { compileExpression } from './evaluators.js' +import { getSourceAliasesFromExpression } from './expressions.js' import { replaceAggregatesByRefs } from './group-by.js' import type { CompareOptions } from '../builder/types.js' import type { WindowOptions } from './types.js' @@ -18,10 +25,11 @@ import type { NamespacedRow, } from '../../types.js' import type { IStreamBuilder, KeyValue } from '@tanstack/db-ivm' -import type { IndexInterface } from '../../indexes/base-index.js' +import type { IndexReader } from '../../indexes/base-index.js' import type { Collection } from '../../collection/index.js' export type OrderByOptimizationInfo = { + sourceId: string alias: string orderBy: OrderBy offset: number @@ -32,11 +40,13 @@ export type OrderByOptimizationInfo = { ) => number /** Extracts all orderBy column values from a raw row (array for multi-column) */ valueExtractorForRawRow: (row: Record) => unknown - /** Extracts only the first column value - used for index-based cursor */ - firstColumnValueExtractor: (row: Record) => unknown /** Index on the first orderBy column - used for lazy loading */ - index?: IndexInterface + index?: IndexReader dataNeeded?: () => number + /** Reads the source loader's synchronous request guard, when installed. */ + isRequesting?: () => boolean + /** Whether local operators can discard or reorder the provider's prefix. */ + requiresFullSource: boolean } /** @@ -69,7 +79,6 @@ export function processOrderBy( compareOptions: buildCompareOptions(clause, collection), } }) - // Create a value extractor function for the orderBy operator const valueExtractor = (row: NamespacedRow & { $selected?: any }) => { // The namespaced row contains: @@ -133,178 +142,142 @@ export function processOrderBy( // Skip this optimization when using grouped ordering (includes with limit), // because the limit is per-group, not global — the child collection needs all data loaded. if ( - limit && + limit !== undefined && !groupKeyFn && rawQuery.from.type !== `unionFrom` && rawQuery.from.type !== `unionAll` ) { - let index: IndexInterface | undefined + let index: IndexReader | undefined let followRefCollection: Collection | undefined - let firstColumnValueExtractor: CompiledSingleRowExpression | undefined let orderByAlias: string = rawQuery.from.alias + let orderBySourceId: string | undefined // Try to create/find an index on the FIRST orderBy column for lazy loading const firstClause = orderByClause[0]! const firstOrderByExpression = firstClause.expression - if (firstOrderByExpression.type === `ref`) { - const followRefResult = followRef( - rawQuery, - firstOrderByExpression, - collection, - ) - - if (followRefResult) { - followRefCollection = followRefResult.collection - const fieldName = followRefResult.path[0] - const compareOpts = buildCompareOptions( - firstClause, - followRefCollection, - ) - - if (fieldName) { - // Use a single-column comparator for the index, not the - // multi-column `compare` function. The multi-column comparator - // expects array values [col1, col2, ...] but the index stores - // individual field values. Passing `compare` here causes the - // BTree to treat all single values as equal (since number[0] - // === undefined for both sides of the comparison). - const firstColumnCompareFn = makeComparator(compareOpts) - ensureIndexForField( - fieldName, - followRefResult.path, - followRefCollection, - compareOpts, - firstColumnCompareFn, - ) - } - - // First column value extractor - used for index cursor - firstColumnValueExtractor = compileExpression( - new PropRef(followRefResult.path), - true, - ) as CompiledSingleRowExpression - - index = findIndexForField( - followRefCollection, + const followRefResult = + firstOrderByExpression.type === `ref` + ? followRef(rawQuery, firstOrderByExpression, collection) + : undefined + if (firstOrderByExpression.type === `ref` && followRefResult) { + followRefCollection = followRefResult.collection + orderBySourceId = followRefResult.sourceId + const fieldName = followRefResult.path[0] + // The query's first source defines implicit string collation for the + // whole order. Build the source index with that same resolved term so + // provider admission cannot disagree with emitted query order. + const compareOpts = buildCompareOptions(firstClause, collection) + + if (fieldName) { + // Use a single-column comparator for the index, not the + // multi-column `compare` function. The multi-column comparator + // expects array values [col1, col2, ...] but the index stores + // individual field values. Passing `compare` here causes the + // BTree to treat all single values as equal (since number[0] + // === undefined for both sides of the comparison). + const firstColumnCompareFn = makeComparator(compareOpts) + ensureIndexForField( + fieldName, followRefResult.path, + followRefCollection, compareOpts, + firstColumnCompareFn, ) + } - // Only use the index if it supports range queries - if (!index?.supports(`gt`)) { - index = undefined - } + index = findIndexForField( + followRefCollection, + followRefResult.path, + compareOpts, + ) - if (!index) { - const collectionId = followRefCollection.id - const fieldPath = followRefResult.path.join(`.`) - console.warn( - `[TanStack DB]${collectionId ? ` [${collectionId}]` : ``} orderBy with limit requires an index on "${fieldPath}" for efficient lazy loading. ` + - `Falling back to loading all data. ` + - `Consider creating an index on the collection with collection.createIndex((row) => row.${fieldPath}) ` + - `or enable auto-indexing with autoIndex: 'eager' and a defaultIndexType.`, - ) - } + // Only use the index if it supports range queries + if (!index?.supports(`gt`)) { + index = undefined + } - orderByAlias = - firstOrderByExpression.path.length > 1 - ? String(firstOrderByExpression.path[0]) - : rawQuery.from.alias + if (!index) { + const collectionId = followRefCollection.id + const fieldPath = followRefResult.path.join(`.`) + console.warn( + `[TanStack DB]${collectionId ? ` [${collectionId}]` : ``} orderBy with limit requires an index on "${fieldPath}" for efficient lazy loading. ` + + `Falling back to loading all data. ` + + `Consider creating an index on the collection with collection.createIndex((row) => row.${fieldPath}) ` + + `or enable auto-indexing with autoIndex: 'eager' and a defaultIndexType.`, + ) } + + orderByAlias = + firstOrderByExpression.path.length > 1 + ? String(firstOrderByExpression.path[0]) + : rawQuery.from.alias + orderBySourceId ??= collectCollectionSources(rawQuery).find( + (source) => + source.alias === orderByAlias && + source.collection === followRefCollection, + )?.sourceId } - // Only create comparator and value extractors if the first column is a ref expression - // For aggregate or computed expressions, we can't extract values from raw collection rows - if (!firstColumnValueExtractor) { - // Skip optimization for non-ref expressions (aggregates, computed values, etc.) - // The query will still work, but without lazy loading optimization - } else { - // Build value extractors for all columns (must all be ref expressions for multi-column) - // Check if all orderBy expressions are ref types (required for multi-column extraction) - const allColumnsAreRefs = orderByClause.every( - (clause) => clause.expression.type === `ref`, + if (orderBySourceId && followRefResult) { + const sourceOrderBy = resolveOrderBy( + orderByClause, + collection.compareOptions, ) - - // Create extractors for all columns if they're all refs - const allColumnExtractors: - | Array - | undefined = allColumnsAreRefs - ? orderByClause.map((clause) => { - // We know it's a ref since we checked allColumnsAreRefs - const refExpr = clause.expression as PropRef - const followResult = followRef(rawQuery, refExpr, collection) - if (followResult) { - return compileExpression( - new PropRef(followResult.path), - true, - ) as CompiledSingleRowExpression - } - // Fallback for refs that don't follow - return compileExpression( - clause.expression, - true, - ) as CompiledSingleRowExpression - }) - : undefined - - // Create a comparator for raw rows (used for tracking sent values) - // This compares ALL orderBy columns for proper ordering - const comparator = ( + const sourceOrderIsDirect = orderByClause.every(({ expression }) => { + if (expression.type !== `ref`) return false + return ( + followRef(rawQuery, expression, collection)?.sourceId === + orderBySourceId + ) + }) + const extract = compileExpression( + new PropRef(followRefResult.path), + true, + ) as CompiledSingleRowExpression + const compareTerm = makeComparator(sourceOrderBy[0]!.compareOptions) + const compareSourceRows = ( a: Record | null | undefined, b: Record | null | undefined, - ) => { - if (orderByClause.length === 1) { - // Single column: extract and compare - const extractedA = a ? firstColumnValueExtractor(a) : a - const extractedB = b ? firstColumnValueExtractor(b) : b - return compare(extractedA, extractedB) - } - if (allColumnExtractors) { - // Multi-column with all refs: extract all values and compare - const extractAll = ( - row: Record | null | undefined, - ) => { - if (!row) return row - return allColumnExtractors.map((extractor) => extractor(row)) - } - return compare(extractAll(a), extractAll(b)) - } - // Fallback: can't compare (shouldn't happen since we skip non-ref cases) - return 0 - } + ) => compareTerm(a ? extract(a) : a, b ? extract(b) : b) - // Create a value extractor for raw rows that extracts ALL orderBy column values - // This is used for tracking sent values and building composite cursors - const rawRowValueExtractor = (row: Record): unknown => { - if (orderByClause.length === 1) { - // Single column: return single value - return firstColumnValueExtractor(row) - } - if (allColumnExtractors) { - // Multi-column: return array of all values - return allColumnExtractors.map((extractor) => extractor(row)) - } - // Fallback (shouldn't happen) - return undefined - } - - orderByOptimizationInfo = { + const info: OrderByOptimizationInfo = { + sourceId: orderBySourceId, alias: orderByAlias, offset: offset ?? 0, limit, - comparator, - valueExtractorForRawRow: rawRowValueExtractor, - firstColumnValueExtractor: firstColumnValueExtractor, + comparator: compareSourceRows, + valueExtractorForRawRow: extract, index, - orderBy: orderByClause, + orderBy: sourceOrderBy, + requiresFullSource: + !sourceOrderIsDirect || + rawQuery.from.type !== `collectionRef` || + rawQuery.from.sourceId !== orderBySourceId || + (rawQuery.join?.some( + ({ type }) => type === `inner` || type === `right`, + ) ?? + false) || + (rawQuery.where?.some( + (where) => + isResidualWhere(where) || + [ + ...getSourceAliasesFromExpression(getWhereExpression(where)), + ].some((alias) => alias !== orderByAlias), + ) ?? + false) || + (rawQuery.fnWhere?.length ?? 0) > 0 || + rawQuery.groupBy !== undefined || + rawQuery.having !== undefined || + rawQuery.fnHaving !== undefined || + rawQuery.distinct === true, } + orderByOptimizationInfo = info - // Store the optimization info keyed by collection ID - // Use the followed collection if available, otherwise use the main collection - const targetCollectionId = followRefCollection?.id ?? collection.id - optimizableOrderByCollections[targetCollectionId] = - orderByOptimizationInfo + // Ordered loading is owned by one lexical source. A collection can occur + // more than once in a query tree, so collection ID and alias are not + // sufficient identities here. + optimizableOrderByCollections[orderBySourceId] = info // Set up lazy loading callback to track how much more data is needed // This is used by loadMoreIfNeeded to determine if more data should be loaded @@ -312,10 +285,10 @@ export function processOrderBy( // and all data is loaded eagerly via requestSnapshot instead. if (index) { setSizeCallback = (getSize: () => number) => { - optimizableOrderByCollections[targetCollectionId]![`dataNeeded`] = + optimizableOrderByCollections[orderBySourceId]![`dataNeeded`] = () => { const size = getSize() - return Math.max(0, orderByOptimizationInfo!.limit - size) + return Math.max(0, info.limit - size) } } } @@ -389,13 +362,28 @@ export function buildCompareOptions( clause: OrderByClause, collection: CollectionLike, ): CompareOptions { - if (clause.compareOptions.stringSort !== undefined) { - return clause.compareOptions - } + return resolveCompareOptions(clause, collection.compareOptions) +} - return { - ...collection.compareOptions, - direction: clause.compareOptions.direction, - nulls: clause.compareOptions.nulls, - } +function resolveOrderBy( + orderBy: OrderBy, + defaults: CollectionLike[`compareOptions`], +): OrderBy { + return orderBy.map((clause) => ({ + expression: clause.expression, + compareOptions: resolveCompareOptions(clause, defaults), + })) +} + +function resolveCompareOptions( + clause: OrderByClause, + defaults: CollectionLike[`compareOptions`], +): CompareOptions { + return clause.compareOptions.stringSort === undefined + ? { + ...defaults, + direction: clause.compareOptions.direction, + nulls: clause.compareOptions.nulls, + } + : clause.compareOptions } diff --git a/packages/db/src/query/compiler/parent-routes.ts b/packages/db/src/query/compiler/parent-routes.ts new file mode 100644 index 0000000000..b6eddb32f9 --- /dev/null +++ b/packages/db/src/query/compiler/parent-routes.ts @@ -0,0 +1,40 @@ +import { filter, join as joinOperator, map } from '@tanstack/db-ivm' +import type { KeyedStream } from '../../types.js' + +const PARENT_ROUTE_CROSS_KEY = `__tanstack_parent_route_cross__` + +export function crossJoinParentRoutes( + input: KeyedStream, + parentKeyStream: KeyedStream, + assemble: ( + rowKey: unknown, + row: unknown, + correlationKey: unknown, + parentContext: unknown, + ) => [unknown, unknown], +): KeyedStream { + // Recursive sources need their route before a correlation field is always + // available. The constant key intentionally creates one copy of each input + // row per active route; callers filter those copies once the field is visible. + const rows: any = input.pipe( + map(([rowKey, row]) => [PARENT_ROUTE_CROSS_KEY, [rowKey, row]]), + ) + const routes: any = parentKeyStream.pipe( + map(([correlationKey, parentContext]) => [ + PARENT_ROUTE_CROSS_KEY, + [correlationKey, parentContext], + ]), + ) + + return rows.pipe( + joinOperator(routes, `inner`), + filter(([, [rowSide, routeSide]]: any) => + Boolean(rowSide != null && routeSide != null), + ), + map(([, [rowSide, routeSide]]: any) => { + const [rowKey, row] = rowSide + const [correlationKey, parentContext] = routeSide + return assemble(rowKey, row, correlationKey, parentContext) + }), + ) as KeyedStream +} diff --git a/packages/db/src/query/compiler/route-metadata.ts b/packages/db/src/query/compiler/route-metadata.ts new file mode 100644 index 0000000000..9a5fad30d0 --- /dev/null +++ b/packages/db/src/query/compiler/route-metadata.ts @@ -0,0 +1,228 @@ +import { isPlainObject } from '../../utils/type-guards.js' + +const ROUTED_SCALAR_VALUE = Symbol(`tanstack_db_routed_scalar_value`) +const ROUTE_METADATA = Symbol(`tanstack_db_route_metadata`) +export const INCLUDES_PUBLIC_KEY = Symbol(`includesPublicKey`) +export const INCLUDES_ROUTING = Symbol(`includesRouting`) +const INTERNAL_ROUTE_KEYS = new Set([ + ROUTE_METADATA, + INCLUDES_PUBLIC_KEY, +]) +const INTERNAL_CALLBACK_KEYS = new Set([ + ...INTERNAL_ROUTE_KEYS, + INCLUDES_ROUTING, +]) + +type RoutedResult = { + [ROUTED_SCALAR_VALUE]: unknown + [ROUTE_METADATA]: RouteMetadata + [INCLUDES_PUBLIC_KEY]: unknown +} + +type PublicContainerProperty = { + descriptor: PropertyDescriptor + value?: { original: unknown; replacement: unknown } +} + +export type RouteMetadata = { + correlationKey: unknown + parentContext: unknown +} + +export type RoutedScalarMetadata = { + value: unknown + correlationKey: unknown + parentContext: unknown + publicKey: unknown +} + +export function attachRouteMetadata( + value: T, + correlationKey: unknown, + parentContext: unknown, +): T { + return Object.assign(value, { + [ROUTE_METADATA]: { correlationKey, parentContext } satisfies RouteMetadata, + }) +} + +export function getRouteMetadata(value: unknown): RouteMetadata | undefined { + if ( + value == null || + typeof value !== `object` || + !(ROUTE_METADATA in value) + ) { + return undefined + } + return (value as { [ROUTE_METADATA]: RouteMetadata })[ROUTE_METADATA] +} + +export function getNamespacedRouteMetadata( + row: unknown, + source: string, +): RouteMetadata | undefined { + return ( + getRouteMetadata(row) ?? + (row != null && typeof row === `object` + ? getRouteMetadata((row as Record)[source]) + : undefined) + ) +} + +export function stripRouteMetadata(value: T): T { + const result = { ...value } as T & Record + delete result[ROUTE_METADATA] + return result +} + +export function attachRouteMetadataToResult( + value: unknown, + correlationKey: unknown, + parentContext: unknown, + publicKey: unknown, +): unknown { + if ( + correlationKey === undefined && + parentContext === undefined && + publicKey === undefined + ) { + return value + } + + if (isPlainObject(value)) { + return { + ...value, + [ROUTE_METADATA]: { correlationKey, parentContext }, + [INCLUDES_PUBLIC_KEY]: publicKey, + } + } + + return { + [ROUTED_SCALAR_VALUE]: value, + [ROUTE_METADATA]: { correlationKey, parentContext }, + [INCLUDES_PUBLIC_KEY]: publicKey, + } satisfies RoutedResult +} + +export function getRoutedScalarMetadata( + value: unknown, +): RoutedScalarMetadata | undefined { + if ( + value == null || + typeof value !== `object` || + !(ROUTED_SCALAR_VALUE in value) + ) { + return undefined + } + + const routed = value as RoutedResult + const route = routed[ROUTE_METADATA] + return { + value: routed[ROUTED_SCALAR_VALUE], + correlationKey: route.correlationKey, + parentContext: route.parentContext, + publicKey: routed[INCLUDES_PUBLIC_KEY], + } +} + +/** Copy public containers while removing private route state at every depth. */ +export function stripInternalRouteMetadata(value: unknown): unknown { + return transformPublicContainers(value, (leaf) => leaf, INTERNAL_ROUTE_KEYS) +} + +/** Remove every compiler-owned key before invoking user code. */ +export function stripInternalCallbackMetadata(value: unknown): unknown { + return transformPublicContainers( + value, + (leaf) => leaf, + INTERNAL_CALLBACK_KEYS, + ) +} + +/** Copy only paths changed by a leaf transform or an omitted private key. */ +export function transformPublicContainers( + value: unknown, + transformLeaf: (value: unknown) => unknown, + omittedKeys: ReadonlySet, +): unknown { + const rootReplacement = transformLeaf(value) + if (!Object.is(rootReplacement, value)) return rootReplacement + if (!isPublicContainer(value)) return value + + const parents = new WeakMap>() + const properties = new WeakMap< + object, + Map + >() + const dirty = new Set() + const visit = (current: object): void => { + if (properties.has(current)) return + const currentProperties = new Map() + properties.set(current, currentProperties) + for (const key of Reflect.ownKeys(current)) { + if (omittedKeys.has(key)) { + dirty.add(current) + continue + } + const descriptor = Object.getOwnPropertyDescriptor(current, key) + if (!descriptor) continue + const property: PublicContainerProperty = { descriptor } + currentProperties.set(key, property) + if (!descriptor.enumerable || !(`value` in descriptor)) continue + const child = descriptor.value + const replacement = transformLeaf(child) + property.value = { original: child, replacement } + if (!Object.is(replacement, child)) { + dirty.add(current) + continue + } + if (!isPublicContainer(child)) continue + const childParents = parents.get(child) ?? new Set() + childParents.add(current) + parents.set(child, childParents) + visit(child) + } + } + visit(value) + + const queue = [...dirty] + for (const current of queue) { + for (const parent of parents.get(current) ?? []) { + if (dirty.has(parent)) continue + dirty.add(parent) + queue.push(parent) + } + } + if (!dirty.has(value)) return value + + const copies = new WeakMap() + const copy = (current: object): object => { + if (!dirty.has(current)) return current + const existing = copies.get(current) + if (existing) return existing + + const result = Array.isArray(current) + ? [] + : Object.create(Object.getPrototypeOf(current)) + copies.set(current, result) + for (const [key, property] of properties.get(current) ?? []) { + const descriptor = { ...property.descriptor } + if (property.value) { + const { original, replacement } = property.value + descriptor.value = !Object.is(replacement, original) + ? replacement + : isPublicContainer(original) + ? copy(original) + : original + } + Object.defineProperty(result, key, descriptor) + } + return result + } + + return copy(value) +} + +function isPublicContainer(value: unknown): value is object { + return Array.isArray(value) || isPlainObject(value) +} diff --git a/packages/db/src/query/compiler/select.ts b/packages/db/src/query/compiler/select.ts index e257b09ca4..8b7fb4e127 100644 --- a/packages/db/src/query/compiler/select.ts +++ b/packages/db/src/query/compiler/select.ts @@ -5,7 +5,7 @@ import { Value as ValClass, isExpressionLike, } from '../ir.js' -import { AggregateNotSupportedError } from '../../errors.js' +import { UnsafeAliasPathError } from '../../errors.js' import { compileExpression, isCaseWhenConditionTrue } from './evaluators.js' import { containsAggregate } from './group-by.js' import type { @@ -39,6 +39,16 @@ function unwrapVal(input: any): any { return input } +const UNSAFE_ALIAS_SEGMENTS = new Set([`__proto__`, `prototype`, `constructor`]) + +function assertSafeAliasSegments(segments: ReadonlyArray): void { + for (const seg of segments) { + if (UNSAFE_ALIAS_SEGMENTS.has(seg)) { + throw new UnsafeAliasPathError(seg) + } + } +} + /** * Processes a merge operation by merging source values into the target path */ @@ -47,6 +57,7 @@ function processMerge( namespacedRow: NamespacedRow, selectResults: Record, ): void { + assertSafeAliasSegments(op.targetPath) const value = op.source(namespacedRow) if (value && typeof value === `object`) { // Ensure target object exists @@ -89,6 +100,7 @@ function processNonMergeOp( ): void { // Support nested alias paths like "meta.author.name" const path = op.alias.split(`.`) + assertSafeAliasSegments(path) if (path.length === 1) { selectResults[op.alias] = op.compiled(namespacedRow) } else { @@ -141,6 +153,16 @@ export function processSelect( select: Select, _allInputs: Record, ): NamespacedAndKeyedStream { + if (!isNestedSelectObject(select)) { + const compiled = compileSelectValue(select as SelectValueExpression) + return pipeline.pipe( + map(([key, namespacedRow]) => [ + key, + { ...namespacedRow, $selected: compiled(namespacedRow) }, + ]), + ) as NamespacedAndKeyedStream + } + // Build ordered operations to preserve authoring order (spreads and fields) const ops: Array = [] @@ -241,24 +263,6 @@ function isAggregateExpression( return expr.type === `agg` } -/** - * Processes a single argument in a function context - */ -export function processArgument( - arg: BasicExpression | Aggregate, - namespacedRow: NamespacedRow, -): any { - if (isAggregateExpression(arg)) { - throw new AggregateNotSupportedError() - } - - // Pre-compile the expression and evaluate immediately - const compiledExpression = compileExpression(arg) - const value = compiledExpression(namespacedRow) - - return value -} - /** * Helper function to check if an object is a nested select object * @@ -283,6 +287,9 @@ function addFromObject( ops: Array, ) { for (const [key, value] of Object.entries(obj)) { + if (!key.startsWith(`__SPREAD_SENTINEL__`)) { + assertSafeAliasSegments(key.split(`.`)) + } if (key.startsWith(`__SPREAD_SENTINEL__`)) { const rest = key.slice(`__SPREAD_SENTINEL__`.length) const splitIndex = rest.lastIndexOf(`__`) diff --git a/packages/db/src/query/effect.ts b/packages/db/src/query/effect.ts index 0237519212..bc35dec6a3 100644 --- a/packages/db/src/query/effect.ts +++ b/packages/db/src/query/effect.ts @@ -1,22 +1,25 @@ import { D2, output } from '@tanstack/db-ivm' -import { transactionScopedScheduler } from '../scheduler.js' +import { createDeferred } from '../deferred.js' +import { + getActivePublicationContext, + transactionScopedScheduler, +} from '../scheduler.js' import { getActiveTransaction } from '../transactions.js' +import { runAllCallbacks } from '../utils/callbacks.js' +import { normalizeError } from '../utils/error.js' import { compileQuery } from './compiler/index.js' -import { - normalizeExpressionPaths, - normalizeOrderByPaths, -} from './compiler/expressions.js' +import { normalizeExpressionPaths } from './compiler/expressions.js' import { getCollectionBuilder } from './live/collection-registry.js' +import { SubsetDemandController } from './live/subset-demand-controller.js' +import { OrderedSourceLoader } from './live/ordered-source-loader.js' import { buildQueryFromConfig, - computeOrderedLoadCursor, computeSubscriptionOrderByHints, - extractCollectionAliases, + extractCollectionSources, extractCollectionsFromQuery, - filterDuplicateInserts, + reconcileChangesForD2, sendChangesToInput, splitUpdates, - trackBiggestSentValue, } from './live/utils.js' import type { RootStreamBuilder } from '@tanstack/db-ivm' import type { Collection } from '../collection/index.js' @@ -25,6 +28,10 @@ import type { InitialQueryBuilder, QueryBuilder } from './builder/index.js' import type { Context } from './builder/types.js' import type { BasicExpression, QueryIR } from './ir.js' import type { OrderByOptimizationInfo } from './compiler/order-by.js' +import type { + LazyCollectionCallbacks, + LazyDemandPlan, +} from './compiler/joins.js' import type { ChangeMessage, KeyedStream, ResultStream } from '../types.js' // --------------------------------------------------------------------------- @@ -132,7 +139,10 @@ export interface EffectConfig< /** Handle returned by createEffect */ export interface Effect { - /** Dispose the effect. Returns a promise that resolves when in-flight handlers complete. */ + /** + * Dispose the effect and await in-flight handlers. Calls during one cleanup + * attempt, including calls from abort/release callbacks, share its outcome. + */ dispose: () => Promise /** Whether this effect has been disposed */ readonly disposed: boolean @@ -242,20 +252,44 @@ export function createEffect< // The dispose function is referenced by both the returned Effect object // and the onSourceError callback, so we define it first. - const dispose = async () => { - if (disposed) return + let disposalPromise: Promise | undefined + const dispose = (): Promise => { + if (disposalPromise) return disposalPromise + // Abort and source-release callbacks may synchronously call dispose again. + // Publish the shared result before entering either user callback boundary. + const completion = createDeferred() + const attempt = completion.promise + disposalPromise = attempt disposed = true // Abort signal for in-flight handlers abortController.abort() - // Tear down the pipeline (unsubscribe from sources, etc.) - runner.dispose() + void (async () => { + // Tear down the pipeline (unsubscribe from sources, etc.) + let cleanupFailed = false + let cleanupError: unknown + try { + runner.dispose() + } catch (error) { + cleanupFailed = true + cleanupError = error + } - // Wait for any in-flight async handlers to settle - if (inFlightHandlers.size > 0) { - await Promise.allSettled([...inFlightHandlers]) - } + // Wait for any in-flight async handlers to settle + if (inFlightHandlers.size > 0) { + await Promise.allSettled([...inFlightHandlers]) + } + + if (cleanupFailed) throw cleanupError + })().then(completion.resolve, completion.reject) + void attempt.then( + () => {}, + () => { + if (disposalPromise === attempt) disposalPromise = undefined + }, + ) + return attempt } // Create and start the pipeline @@ -280,10 +314,27 @@ export function createEffect< } // Auto-dispose — the effect can no longer function - dispose() + void dispose().catch((cleanupError) => { + console.error( + `[Effect '${id}'] failed to dispose after a source error:`, + cleanupError, + ) + }) }, }) - runner.start() + try { + runner.start() + } catch (error) { + try { + runner.dispose() + } catch (cleanupError) { + console.error( + `[Effect '${id}'] failed to dispose after a startup error:`, + cleanupError, + ) + } + throw error + } return { dispose, @@ -314,27 +365,31 @@ interface EffectPipelineRunnerConfig< * Sets up the IVM graph, subscribes to source collections, runs the graph * when changes arrive, and classifies output multiplicities into DeltaEvents. * - * Unlike CollectionConfigBuilder, this does NOT: - * - Create or write to a collection (no materialisation) - * - Manage ordering, windowing, or lazy loading + * Unlike CollectionConfigBuilder, this does not publish results to a + * Collection. */ class EffectPipelineRunner { private readonly query: QueryIR private readonly collections: Record> - private readonly collectionByAlias: Record> + private readonly collectionSources: ReturnType< + typeof extractCollectionSources + > private graph: D2 | undefined private inputs: Record> | undefined private pipeline: ResultStream | undefined private sourceWhereClauses: Map> | undefined - private compiledAliasToCollectionId: Record = {} // Mutable objects passed to compileQuery by reference. // The join compiler captures these references and reads them later when // the graph runs, so they must be populated before the first graph run. private readonly subscriptions: Record = {} - private readonly lazySourcesCallbacks: Record = {} + private readonly lazySourcesCallbacks: Record< + string, + LazyCollectionCallbacks + > = {} private readonly lazySources = new Set() + private readonly demand = new SubsetDemandController() // OrderBy optimization info populated by the compiler when limit is present private readonly optimizableOrderByCollections: Record< string, @@ -342,14 +397,15 @@ class EffectPipelineRunner { > = {} // Ordered subscription state for cursor-based loading - private readonly biggestSentValue = new Map() - private readonly lastLoadRequestKey = new Map() - private pendingOrderedLoadPromise: Promise | undefined + private readonly orderedLoaders = new Map() // Subscription management private readonly unsubscribeCallbacks = new Set<() => void>() - // Duplicate insert prevention per alias - private readonly sentToD2KeysByAlias = new Map>() + // Exact row last contributed to D2 per lexical source key. + private readonly sentToD2RowsBySource = new Map< + string, + Map> + >() // Output accumulator private pendingChanges: Map> = new Map() @@ -361,13 +417,11 @@ class EffectPipelineRunner { // Scheduler integration private subscribedToAllCollections = false private readonly builderDependencies = new Set() - private readonly aliasDependencies: Record> = {} // Reentrance guard private isGraphRunning = false + private starting = false private disposed = false - // When dispose() is called mid-graph-run, defer heavy cleanup until the run completes - private deferredCleanup = false private readonly onBatchProcessed: ( events: Array>, @@ -384,17 +438,7 @@ class EffectPipelineRunner { // Extract source collections this.collections = extractCollectionsFromQuery(this.query) - const aliasesById = extractCollectionAliases(this.query) - - // Build alias → collection map - this.collectionByAlias = {} - for (const [collectionId, aliases] of aliasesById.entries()) { - const collection = this.collections[collectionId] - if (!collection) continue - for (const alias of aliases) { - this.collectionByAlias[alias] = collection - } - } + this.collectionSources = extractCollectionSources(this.query) // Compile the pipeline this.compilePipeline() @@ -404,8 +448,8 @@ class EffectPipelineRunner { private compilePipeline(): void { this.graph = new D2() this.inputs = Object.fromEntries( - Object.keys(this.collectionByAlias).map((alias) => [ - alias, + this.collectionSources.map((source) => [ + source.sourceId, this.graph!.newInput(), ]), ) @@ -426,7 +470,6 @@ class EffectPipelineRunner { this.pipeline = compilation.pipeline this.sourceWhereClauses = compilation.sourceWhereClauses - this.compiledAliasToCollectionId = compilation.aliasToCollectionId // Attach the output operator that accumulates changes this.pipeline.pipe( @@ -439,12 +482,16 @@ class EffectPipelineRunner { this.graph.finalize() } + private isDisposed(): boolean { + return this.disposed + } + /** Subscribe to source collections and start processing */ start(): void { - // Use compiled aliases as the source of truth - const compiledAliases = Object.entries(this.compiledAliasToCollectionId) - if (compiledAliases.length === 0) { + this.starting = true + if (this.collectionSources.length === 0) { // Nothing to subscribe to + this.starting = false return } @@ -467,90 +514,121 @@ class EffectPipelineRunner { Array>> >() - for (const [alias, collectionId] of compiledAliases) { - const collection = - this.collectionByAlias[alias] ?? this.collections[collectionId]! + for (const source of this.collectionSources) { + if (this.isDisposed()) { + this.starting = false + return + } - // Initialise per-alias duplicate tracking - this.sentToD2KeysByAlias.set(alias, new Set()) + const { sourceId, alias, collection } = source + const collectionId = collection.id + + this.sentToD2RowsBySource.set(sourceId, new Map()) // Discover dependencies: if source collection is itself a live query // collection, its builder must run first during transaction flushes. const dependencyBuilder = getCollectionBuilder(collection) if (dependencyBuilder) { - this.aliasDependencies[alias] = [dependencyBuilder] this.builderDependencies.add(dependencyBuilder) - } else { - this.aliasDependencies[alias] = [] } // Get where clause for this alias (for predicate push-down) - const whereClause = this.sourceWhereClauses?.get(alias) + const whereClause = this.sourceWhereClauses?.get(sourceId) const whereExpression = whereClause ? normalizeExpressionPaths(whereClause, alias) : undefined // Initialise buffer for this alias const buffer: Array>> = [] - pendingBuffers.set(alias, buffer) + pendingBuffers.set(sourceId, buffer) // Lazy aliases (marked by the join compiler) should NOT load initial state // eagerly — the join tap operator will load exactly the rows it needs on demand. // For on-demand collections, eager loading would trigger a full server fetch // for data that should be lazily loaded based on join keys. - const isLazy = this.lazySources.has(alias) + const isLazy = this.lazySources.has(sourceId) // Check if this alias has orderBy optimization (cursor-based loading) - const orderByInfo = this.getOrderByInfoForAlias(alias) + const orderByInfo = this.getOrderByInfoForSource(sourceId) // Build the change callback — for ordered aliases, split updates into - // delete+insert and track the biggest sent value for cursor positioning. + // delete+insert and invalidate loading state from changed contributions. const changeCallback = orderByInfo ? (changes: Array>) => { - if (pendingBuffers.has(alias)) { - pendingBuffers.get(alias)!.push(changes) + if (pendingBuffers.has(sourceId)) { + pendingBuffers.get(sourceId)!.push(changes) } else { - this.trackSentValues(alias, changes, orderByInfo.comparator) + this.orderedLoaders + .get(sourceId) + ?.onSourceChanges( + changes, + this.sentToD2RowsBySource.get(sourceId), + ) const split = [...splitUpdates(changes)] - this.handleSourceChanges(alias, split) + this.handleSourceChanges(sourceId, split) } } : (changes: Array>) => { - if (pendingBuffers.has(alias)) { - pendingBuffers.get(alias)!.push(changes) + if (pendingBuffers.has(sourceId)) { + pendingBuffers.get(sourceId)!.push(changes) } else { - this.handleSourceChanges(alias, changes) + this.handleSourceChanges(sourceId, changes) } } - // Determine subscription options based on ordered vs unordered path - const subscriptionOptions = this.buildSubscriptionOptions( - alias, - isLazy, - orderByInfo, - whereExpression, - ) - // Subscribe to source changes - const subscription = collection.subscribeChanges( - changeCallback, - subscriptionOptions, - ) + const subscription = collection.subscribeChanges(changeCallback, { + ...this.buildSubscriptionOptions( + alias, + isLazy, + orderByInfo, + whereExpression, + ), + onLoadSubsetError: ({ error }) => { + this.onSourceError(normalizeError(error)) + }, + }) // Store subscription immediately so the join compiler can find it - this.subscriptions[alias] = subscription + this.subscriptions[sourceId] = subscription + + const unsubscribe = () => { + delete this.subscriptions[sourceId] + subscription.unsubscribe() + } + + // subscribeChanges can synchronously report a source error and dispose + // the runner before returning the subscription. + if (this.isDisposed()) { + unsubscribe() + this.starting = false + return + } + + // Own the subscription before any ordered snapshot or lazy demand can + // throw. A partially started effect has no handle for its caller to + // dispose, so start() must be able to release every acquired source. + this.unsubscribeCallbacks.add(unsubscribe) + + const lazyCallbacks = this.lazySourcesCallbacks[sourceId] + if (lazyCallbacks) { + lazyCallbacks.setDemand = (plan: LazyDemandPlan, keys: Set) => + this.setDemand(subscription, plan, keys) + for (const plan of lazyCallbacks.plans ?? []) { + if (plan.initialKeys.size > 0) { + lazyCallbacks.setDemand(plan, plan.initialKeys) + } + } + } // For ordered aliases with an index, trigger the initial limited snapshot. // This loads only the top N rows rather than the entire collection. if (orderByInfo) { - this.requestInitialOrderedSnapshot(alias, orderByInfo, subscription) + const loader = new OrderedSourceLoader(orderByInfo, subscription, alias) + this.orderedLoaders.set(sourceId, loader) + loader.start() } - this.unsubscribeCallbacks.add(() => { - subscription.unsubscribe() - delete this.subscriptions[alias] - }) - // Listen for status changes on source collections const statusUnsubscribe = collection.on(`status:change`, (event) => { if (this.disposed) return @@ -600,22 +678,23 @@ class EffectPipelineRunner { // switches that alias to direct-processing mode. Any new callbacks that // fire during the drain (e.g. from requestLimitedSnapshot) will go // through handleSourceChanges directly instead of being lost. - for (const [alias] of pendingBuffers) { - const buffer = pendingBuffers.get(alias)! - pendingBuffers.delete(alias) - - const orderByInfo = this.getOrderByInfoForAlias(alias) + for (const [sourceId] of pendingBuffers) { + const buffer = pendingBuffers.get(sourceId)! + pendingBuffers.delete(sourceId) + const orderByInfo = this.getOrderByInfoForSource(sourceId) // Drain all buffered batches. Since we deleted the alias from // pendingBuffers above, any new changes arriving during drain go // through handleSourceChanges directly (not back into this buffer). for (const changes of buffer) { if (orderByInfo) { - this.trackSentValues(alias, changes, orderByInfo.comparator) + this.orderedLoaders + .get(sourceId) + ?.onSourceChanges(changes, this.sentToD2RowsBySource.get(sourceId)) const split = [...splitUpdates(changes)] - this.sendChangesToD2(alias, split) + this.sendChangesToD2(sourceId, split) } else { - this.sendChangesToD2(alias, changes) + this.sendChangesToD2(sourceId, changes) } } } @@ -631,15 +710,40 @@ class EffectPipelineRunner { this.initialLoadComplete = true } } + this.starting = false } /** Handle incoming changes from a source collection */ private handleSourceChanges( - alias: string, + sourceId: string, changes: Array>, ): void { - this.sendChangesToD2(alias, changes) - this.scheduleGraphRun(alias) + this.sendChangesToD2(sourceId, changes) + this.scheduleGraphRun() + } + + private setDemand( + subscription: CollectionSubscription, + plan: LazyDemandPlan, + keys: Set, + ): void { + let update + try { + update = this.demand.setDemand(subscription, plan, keys) + } catch (error) { + // The subscription error event already reports adapter failures and + // disposes this effect. Do not let that query-local failure escape the + // source commit, but keep unrelated graph errors visible. + if (!Object.is(subscription.lastError, error)) throw error + if (this.starting) throw error + return + } + if (update.ready instanceof Promise) { + // Each segment reports its own failure through the subscription. Consume + // the aggregate rejection so Promise.all does not create a second, + // detached error channel. + void update.ready.then(undefined, () => {}) + } } /** @@ -652,19 +756,12 @@ class EffectPipelineRunner { * Dependencies are discovered from source collections that are themselves * live query collections, ensuring parent queries run before effects. */ - private scheduleGraphRun(alias?: string): void { - const contextId = getActiveTransaction()?.id - - // Collect dependencies for this schedule call - const deps = new Set(this.builderDependencies) - if (alias) { - const aliasDeps = this.aliasDependencies[alias] - if (aliasDeps) { - for (const dep of aliasDeps) { - deps.add(dep) - } - } - } + private scheduleGraphRun(): void { + const contextId = + getActiveTransaction()?.id ?? getActivePublicationContext() + + // Snapshot before scheduling parents, which can reenter source setup. + const deps = [...this.builderDependencies] // Ensure dependent builders are scheduled in this context so that // dependency edges always point to a real job. @@ -699,23 +796,22 @@ class EffectPipelineRunner { } /** - * Send changes to the D2 input for the given alias. + * Send changes to the D2 input for the given lexical source. * Returns the number of multiset entries sent. */ private sendChangesToD2( - alias: string, + sourceId: string, changes: Array>, ): number { if (this.disposed || !this.inputs || !this.graph) return 0 - const input = this.inputs[alias] + const input = this.inputs[sourceId] if (!input) return 0 - // Filter duplicates per alias - const sentKeys = this.sentToD2KeysByAlias.get(alias)! - const filtered = filterDuplicateInserts(changes, sentKeys) + const sentRows = this.sentToD2RowsBySource.get(sourceId)! + const reconciled = reconcileChangesForD2(changes, sentRows) - return sendChangesToInput(input, filtered) + return sendChangesToInput(input, reconciled) } /** @@ -730,7 +826,8 @@ class EffectPipelineRunner { this.isGraphRunning = true try { - while (this.graph.pendingWork()) { + // Ordered refill can also dispose the runner between graph steps. + while (!this.isDisposed() && this.graph.pendingWork()) { this.graph.run() // A handler (via onBatchProcessed) or source error callback may have // called dispose() during graph.run(). Stop early to avoid operating @@ -747,13 +844,6 @@ class EffectPipelineRunner { this.flushPendingChanges() } finally { this.isGraphRunning = false - // If dispose() was called during this graph run, it deferred the heavy - // cleanup (clearing graph/inputs/pipeline) to avoid nulling references - // mid-loop. Complete that cleanup now. - if (this.deferredCleanup) { - this.deferredCleanup = false - this.finalCleanup() - } } } @@ -805,6 +895,10 @@ class EffectPipelineRunner { orderBy?: any limit?: number } { + if (this.query.limit === 0) { + return { includeInitialState: false, whereExpression } + } + // Ordered aliases explicitly disable initial state — data is loaded // via requestLimitedSnapshot/requestSnapshot after subscription setup. if (orderByInfo) { @@ -825,48 +919,12 @@ class EffectPipelineRunner { } } - /** - * Request the initial ordered snapshot for an alias. - * Uses requestLimitedSnapshot (index-based cursor) or requestSnapshot - * (full load with limit) depending on whether an index is available. - */ - private requestInitialOrderedSnapshot( - alias: string, - orderByInfo: OrderByOptimizationInfo, - subscription: CollectionSubscription, - ): void { - const { orderBy, offset, limit, index } = orderByInfo - const normalizedOrderBy = normalizeOrderByPaths(orderBy, alias) - - if (index) { - subscription.setOrderByIndex(index) - subscription.requestLimitedSnapshot({ - limit: offset + limit, - orderBy: normalizedOrderBy, - trackLoadSubsetPromise: false, - }) - } else { - subscription.requestSnapshot({ - orderBy: normalizedOrderBy, - limit: offset + limit, - trackLoadSubsetPromise: false, - }) - } - } - - /** - * Get orderBy optimization info for a given alias. - * Returns undefined if no optimization exists for this alias. - */ - private getOrderByInfoForAlias( - alias: string, + /** Get orderBy optimization info for one lexical source. */ + private getOrderByInfoForSource( + sourceId: string, ): OrderByOptimizationInfo | undefined { - // optimizableOrderByCollections is keyed by collection ID - const collectionId = this.compiledAliasToCollectionId[alias] - if (!collectionId) return undefined - - const info = this.optimizableOrderByCollections[collectionId] - if (info && info.alias === alias) { + const info = this.optimizableOrderByCollections[sourceId] + if (info?.sourceId === sourceId) { return info } return undefined @@ -877,123 +935,57 @@ class EffectPipelineRunner { * needs more data. If so, load more rows via requestLimitedSnapshot. */ private loadMoreIfNeeded(): void { - for (const [, orderByInfo] of Object.entries( - this.optimizableOrderByCollections, - )) { - if (!orderByInfo.dataNeeded || !orderByInfo.index) continue - - if (this.pendingOrderedLoadPromise) { - // Wait for in-flight loads to complete before requesting more - continue - } - - const n = orderByInfo.dataNeeded() - if (n > 0) { - this.loadNextItems(orderByInfo, n) + for (const loader of this.orderedLoaders.values()) { + try { + loader.loadMore() + } catch (error) { + if ( + !this.disposed && + !Object.values(this.subscriptions).some((subscription) => + Object.is(subscription.lastError, error), + ) + ) + throw error } } } - /** - * Load n more items from the source collection, starting from the cursor - * position (the biggest value sent so far). - */ - private loadNextItems(orderByInfo: OrderByOptimizationInfo, n: number): void { - const { alias } = orderByInfo - const subscription = this.subscriptions[alias] - if (!subscription) return - - const cursor = computeOrderedLoadCursor( - orderByInfo, - this.biggestSentValue.get(alias), - this.lastLoadRequestKey.get(alias), - alias, - n, - ) - if (!cursor) return // Duplicate request — skip - - this.lastLoadRequestKey.set(alias, cursor.loadRequestKey) - - subscription.requestLimitedSnapshot({ - orderBy: cursor.normalizedOrderBy, - limit: n, - minValues: cursor.minValues, - trackLoadSubsetPromise: false, - onLoadSubsetResult: (loadResult: Promise | true) => { - // Track in-flight load to prevent redundant concurrent requests - if (loadResult instanceof Promise) { - this.pendingOrderedLoadPromise = loadResult - loadResult.finally(() => { - if (this.pendingOrderedLoadPromise === loadResult) { - this.pendingOrderedLoadPromise = undefined - } - }) - } - }, - }) - } - - /** - * Track the biggest value sent for a given ordered alias. - * Used for cursor-based pagination in loadNextItems. - */ - private trackSentValues( - alias: string, - changes: Array>, - comparator: (a: any, b: any) => number, - ): void { - const sentKeys = this.sentToD2KeysByAlias.get(alias) ?? new Set() - const result = trackBiggestSentValue( - changes, - this.biggestSentValue.get(alias), - sentKeys, - comparator, - ) - this.biggestSentValue.set(alias, result.biggest) - if (result.shouldResetLoadKey) { - this.lastLoadRequestKey.delete(alias) - } - } - /** Tear down subscriptions and clear state */ dispose(): void { if (this.disposed) return this.disposed = true this.subscribedToAllCollections = false - // Immediately unsubscribe from sources and clear cheap state - this.unsubscribeCallbacks.forEach((fn) => fn()) - this.unsubscribeCallbacks.clear() - this.sentToD2KeysByAlias.clear() + // Release every source in one attempt; the first failure wins after the + // peers finish. A reentrant dispose returns at the guard above, so this + // call still owns each release exactly once. + try { + runAllCallbacks(this.unsubscribeCallbacks) + } finally { + this.unsubscribeCallbacks.clear() + this.clearPipelineState() + } + } + + private clearPipelineState(): void { + this.sentToD2RowsBySource.clear() this.pendingChanges.clear() this.lazySources.clear() + this.demand.clear() this.builderDependencies.clear() - this.biggestSentValue.clear() - this.lastLoadRequestKey.clear() - this.pendingOrderedLoadPromise = undefined + for (const loader of this.orderedLoaders.values()) loader.dispose() + this.orderedLoaders.clear() // Clear mutable objects for (const key of Object.keys(this.lazySourcesCallbacks)) { delete this.lazySourcesCallbacks[key] } - for (const key of Object.keys(this.aliasDependencies)) { - delete this.aliasDependencies[key] - } for (const key of Object.keys(this.optimizableOrderByCollections)) { delete this.optimizableOrderByCollections[key] } - // If the graph is currently running, defer clearing graph/inputs/pipeline - // until runGraph() completes — otherwise we'd null references mid-loop. - if (this.isGraphRunning) { - this.deferredCleanup = true - } else { - this.finalCleanup() - } - } - - /** Clear graph references — called after graph run completes or immediately from dispose */ - private finalCleanup(): void { + // graph.run() keeps its own stack reference. The disposed guard prevents + // another step or new input; clearing our references does not destroy it. this.graph = undefined this.inputs = undefined this.pipeline = undefined @@ -1090,9 +1082,10 @@ function trackPromise( inFlightHandlers: Set>, ): void { inFlightHandlers.add(promise) - promise.finally(() => { + const finish = () => { inFlightHandlers.delete(promise) - }) + } + void promise.then(finish, finish) } /** Report an error to the onError callback or console */ @@ -1101,7 +1094,7 @@ function reportError( event: DeltaEvent, onError?: (error: Error, event: DeltaEvent) => void, ): void { - const normalised = error instanceof Error ? error : new Error(String(error)) + const normalised = normalizeError(error) if (onError) { try { onError(normalised, event) diff --git a/packages/db/src/query/equality-value-identity.ts b/packages/db/src/query/equality-value-identity.ts new file mode 100644 index 0000000000..5e397f9e0f --- /dev/null +++ b/packages/db/src/query/equality-value-identity.ts @@ -0,0 +1,104 @@ +import { serializeValue } from '@tanstack/db-ivm' +import { normalizeValue } from '../utils/comparison.js' +import { + createRuntimeReferenceIdentityFactory, + getRuntimeReferenceIdentity, +} from './runtime-reference-identity.js' + +const PARENT_CONTEXT = Symbol(`tanstack_db_parent_context`) + +type ParentContext = { + [PARENT_CONTEXT]: true + value: Record + identity: unknown +} + +type ReferenceIdentity = typeof getRuntimeReferenceIdentity + +export type ValueIdentity = { + equality: (value: unknown) => unknown + exact: (value: unknown) => unknown + serializeEquality: (value: unknown) => string +} + +function equalityIdentity( + value: unknown, + referenceIdentity: ReferenceIdentity, +): unknown { + const normalized = normalizeValue(value) + if ( + (typeof normalized === `object` && normalized !== null) || + typeof normalized === `function` || + typeof normalized === `symbol` + ) { + return referenceIdentity(normalized as object | symbol) + } + return normalized +} + +function exactIdentity( + value: unknown, + referenceIdentity: ReferenceIdentity, +): unknown { + if ( + (typeof value === `object` && value !== null) || + typeof value === `function` || + typeof value === `symbol` + ) { + return referenceIdentity(value) + } + if (typeof value === `number`) { + if (Object.is(value, -0)) return [`number`, `-0`] + if (Number.isNaN(value)) return [`number`, `NaN`] + } + return value +} + +export function createValueIdentity(): ValueIdentity { + const referenceIdentity = createRuntimeReferenceIdentityFactory() + const equality = (value: unknown) => + equalityIdentity(value, referenceIdentity) + return { + equality, + exact: (value) => exactIdentity(value, referenceIdentity), + serializeEquality: (value) => serializeValue(equality(value)), + } +} + +/** Preserve the value relation used by equality predicates in keyed state. */ +export function getEqualityValueIdentity(value: unknown): unknown { + return equalityIdentity(value, getRuntimeReferenceIdentity) +} + +/** Keep compiler identity outside the namespace that holds user aliases. */ +export function createParentContext( + value: Record, + identity: unknown, +): ParentContext { + return { [PARENT_CONTEXT]: true, value, identity } +} + +function isParentContext(context: unknown): context is ParentContext { + return ( + typeof context === `object` && context !== null && PARENT_CONTEXT in context + ) +} + +export function getParentContextValue( + context: unknown, +): Record | undefined { + if (isParentContext(context)) return context.value + if (typeof context === `object` && context !== null) { + return context as Record + } + return undefined +} + +/** + * The envelope is structural D2 state, but its value keeps the user's alias + * namespace separate from compiler identity. A later insert or retract can + * therefore rebuild the same route without reserving a user-visible key. + */ +export function getParentContextIdentity(context: unknown): unknown { + return isParentContext(context) ? context.identity : context +} diff --git a/packages/db/src/query/index.ts b/packages/db/src/query/index.ts index e23475bc25..3c104b813c 100644 --- a/packages/db/src/query/index.ts +++ b/packages/db/src/query/index.ts @@ -59,6 +59,9 @@ export { coalesce, caseWhen, add, + subtract, + multiply, + divide, // Aggregates count, avg, @@ -92,16 +95,16 @@ export { queryOnce, type QueryOnceConfig } from './query-once.js' export { type LiveQueryCollectionConfig } from './live/types.js' export { type LiveQueryCollectionUtils } from './live/collection-config-builder.js' - -// Predicate utilities for predicate push-down export { - isWhereSubset, - unionWherePredicates, - minusWherePredicates, - isOrderBySubset, - isLimitSubset, - isOffsetLimitSubset, - isPredicateSubset, -} from './predicate-utils.js' + UnhashableQueryIRError, + canonicalizeQueryIR, + getLoadSubsetDemandKey, + getQueryIdentity, + getStableQueryBuilderHash, + getStableQueryIRHash, + getStableValueHash, + type DemandKey, + type QueryIdentity, +} from './ir-stable-identity.js' export { DeduplicatedLoadSubset } from './subset-dedupe.js' diff --git a/packages/db/src/query/ir-stable-identity.ts b/packages/db/src/query/ir-stable-identity.ts new file mode 100644 index 0000000000..7139942ce2 --- /dev/null +++ b/packages/db/src/query/ir-stable-identity.ts @@ -0,0 +1,1148 @@ +import { isPlainObject } from '../utils/type-guards.js' +import { normalizeValue } from '../utils/comparison.js' +import { isRefProxy, toExpression } from './builder/ref-proxy.js' +import { getQueryIR } from './builder/query-ir.js' +import { getRuntimeReferenceIdentity } from './runtime-reference-identity.js' +import type { + Aggregate, + BasicExpression, + ConditionalSelect, + From, + Having, + IncludesSubquery, + JoinClause, + OrderByClause, + QueryIR, + Select, + Where, +} from './ir.js' +import type { InitialQueryBuilder, QueryBuilder } from './builder/index.js' +import type { LoadSubsetOptions } from '../types.js' + +type StableIdentityValue = + | null + | boolean + | number + | string + | Array + | { [key: string]: StableIdentityValue } + +type ValueIdentityContext = + | `exact-output` + | `equality-operand` + | `ordering-operand` + +type AliasScope = { + bindings: ReadonlyMap + hasUnqualifiedOutput: boolean + parent: AliasScope | undefined +} + +declare const queryIdentityBrand: unique symbol +declare const demandKeyBrand: unique symbol + +/** Semantic identity for a query plan, independent of its runtime owners. */ +export type QueryIdentity = string & { + readonly [queryIdentityBrand]: true +} + +/** Exact identity for one loadSubset demand, including its requested window. */ +export type DemandKey = string & { + readonly [demandKeyBrand]: true +} + +export class UnhashableQueryIRError extends Error { + constructor( + public readonly path: string, + public readonly reason: string, + ) { + super(`Query IR is not stably hashable at ${path}: ${reason}`) + this.name = `UnhashableQueryIRError` + } +} + +export function getStableQueryIRHash(query: QueryIR): string { + return getQueryIdentity(query) +} + +export function getStableQueryBuilderHash( + query: InitialQueryBuilder | QueryBuilder, +): string { + return getStableQueryIRHash(getQueryIR(query)) +} + +export function getStableValueHash(value: unknown, path = `value`): string { + return JSON.stringify(canonicalizeRuntimeValue(value, path, new WeakSet())) +} + +/** + * Returns the semantic identity of a structured query. + * + * Logical conjunctions and disjunctions are associative, commutative, and + * idempotent. Equality operands are commutative, while reversed inequalities + * are normalized by inverting their operator. Order-sensitive clauses and + * function arguments retain their original order. + */ +export function getQueryIdentity(query: QueryIR): QueryIdentity { + return JSON.stringify(canonicalizeQueryIR(query)) as QueryIdentity +} + +/** + * Returns the exact semantic identity of a loadSubset request. + * + * Abort signals and subscriptions are owners of a request, not part of the + * requested data, and therefore do not affect the key. A demand generation + * scopes one asynchronous attempt rather than the data it requests. Code that + * rejects stale work compares this key alongside its generation; query-db uses + * the key alone so equivalent data demands can reuse one cache entry across + * generations. + */ +export function getLoadSubsetDemandKey( + options: LoadSubsetOptions, +): DemandKey | undefined { + if ( + options.where === undefined && + !options.orderBy?.length && + options.limit === undefined && + (options.offset === undefined || options.offset === 0) && + options.cursor === undefined + ) { + // Query-db uses its base query key for the one unconstrained demand. An + // owner-only option must not create another cache entry for the same data. + return undefined + } + + const seen = new WeakSet() + const result: Record = { + type: `loadSubsetDemand`, + query: canonicalizeLoadSubsetQuery(options, `loadSubset`, seen), + } + + if (options.limit !== undefined) { + result.limit = canonicalizeRuntimeValue( + options.limit, + `loadSubset.limit`, + seen, + ) + } + + if (options.offset !== undefined && options.offset !== 0) { + result.offset = canonicalizeRuntimeValue( + options.offset, + `loadSubset.offset`, + seen, + ) + } + + if (options.cursor !== undefined) { + const cursor: Record = { + whereFrom: canonicalizeExpression( + options.cursor.whereFrom, + `loadSubset.cursor.whereFrom`, + seen, + `exact-output`, + ), + whereCurrent: canonicalizeExpression( + options.cursor.whereCurrent, + `loadSubset.cursor.whereCurrent`, + seen, + `exact-output`, + ), + } + if (options.cursor.lastKey !== undefined) { + cursor.lastKey = canonicalizeRuntimeValue( + options.cursor.lastKey, + `loadSubset.cursor.lastKey`, + seen, + ) + } + result.cursor = cursor + } + + return JSON.stringify(result) as DemandKey +} + +export function canonicalizeQueryIR(query: QueryIR): StableIdentityValue { + return canonicalizeQuery(query, `query`, new WeakSet()) +} + +function createAliasScope( + query: QueryIR, + parent: AliasScope | undefined, +): AliasScope { + const bindings = new Map() + + const bindSource = (source: From): void => { + if (source.type === `unionFrom`) { + source.sources.forEach(bindSource) + return + } + if (source.type === `unionAll`) return + if (!bindings.has(source.alias)) { + bindings.set(source.alias, bindings.size) + } + } + + bindSource(query.from) + query.join?.forEach(({ from }) => bindSource(from)) + return { + bindings, + hasUnqualifiedOutput: query.from.type === `unionAll`, + parent, + } +} + +function resolveAliasBinding( + scope: AliasScope | undefined, + alias: string, +): readonly [number, number] | undefined { + let current = scope + let parentDistance = 0 + while (current) { + const binding = current.bindings.get(alias) + if (binding !== undefined) return [parentDistance, binding] + // A result-level union has no source alias. Every downstream ref starts at + // an output field, including nested paths such as profile.id, so it must + // not fall through and bind that field name to an enclosing query alias. + if (current.hasUnqualifiedOutput) return undefined + current = current.parent + parentDistance++ + } + return undefined +} + +function canonicalizeQuery( + query: QueryIR, + path: string, + seen: WeakSet, + parentScope?: AliasScope, +): StableIdentityValue { + return canonicalizeQueryInScope( + query, + path, + seen, + createAliasScope(query, parentScope), + ) +} + +function canonicalizeQueryInScope( + query: QueryIR, + path: string, + seen: WeakSet, + scope: AliasScope, +): StableIdentityValue { + if (query.fnSelect) { + throw new UnhashableQueryIRError(`${path}.fnSelect`, `function select`) + } + + if (query.fnWhere?.length) { + throw new UnhashableQueryIRError(`${path}.fnWhere`, `function where`) + } + + if (query.fnHaving?.length) { + throw new UnhashableQueryIRError(`${path}.fnHaving`, `function having`) + } + + const result: Record = { + type: `query`, + from: canonicalizeSource(query.from, `${path}.from`, seen, scope), + } + + if (query.select) { + result.select = canonicalizeSelect( + query.select, + `${path}.select`, + seen, + scope, + ) + } + + if ( + !query.select && + (query.from.type === `unionFrom` || + query.join !== undefined || + query.groupBy !== undefined) + ) { + // Without an explicit projection, these query shapes return a namespaced + // row. Its alias keys are public output and therefore part of identity. + result.implicitOutput = { + type: `namespaced`, + aliases: Array.from(scope.bindings.keys()), + } + } + + if (query.join) { + result.join = query.join.map((join, index) => + canonicalizeJoin(join, `${path}.join[${index}]`, seen, scope), + ) + } + + if (query.where) { + result.where = canonicalizeImplicitConjunction( + query.where, + `${path}.where`, + seen, + scope, + ) + } + + if (query.groupBy) { + result.groupBy = query.groupBy.map((expression, index) => + canonicalizeExpression( + expression, + `${path}.groupBy[${index}]`, + seen, + `exact-output`, + scope, + ), + ) + } + + if (query.having) { + result.having = canonicalizeImplicitConjunction( + query.having, + `${path}.having`, + seen, + scope, + ) + } + + if (query.orderBy) { + result.orderBy = query.orderBy.map((orderBy, index) => + canonicalizeOrderBy( + orderBy, + `${path}.orderBy[${index}]`, + seen, + `ordering-operand`, + scope, + ), + ) + } + + if (query.limit !== undefined) { + result.limit = canonicalizeRuntimeValue(query.limit, `${path}.limit`, seen) + } + + if (query.offset !== undefined && query.offset !== 0) { + result.offset = canonicalizeRuntimeValue( + query.offset, + `${path}.offset`, + seen, + ) + } + + if (query.distinct) { + result.distinct = true + } + + if (query.singleResult) { + result.singleResult = true + } + + return result +} + +function canonicalizeImplicitConjunction( + clauses: ReadonlyArray, + path: string, + seen: WeakSet, + scope: AliasScope, +): Array { + const canonical = clauses.map((clause, index) => + canonicalizeWhere(clause, `${path}[${index}]`, seen, scope), + ) + canonical.sort(compareStableIdentityValues) + + return canonical.filter( + (clause, index) => + index === 0 || + compareStableIdentityValues(clause, canonical[index - 1]!) !== 0, + ) +} + +function canonicalizeLoadSubsetQuery( + options: LoadSubsetOptions, + path: string, + seen: WeakSet, +): StableIdentityValue { + const result: Record = { + type: `loadSubsetQuery`, + } + + if (options.where !== undefined) { + result.where = canonicalizeExpression( + options.where, + `${path}.where`, + seen, + `exact-output`, + ) + } + + if (options.orderBy?.length) { + result.orderBy = options.orderBy.map((orderBy, index) => + canonicalizeOrderBy( + orderBy, + `${path}.orderBy[${index}]`, + seen, + `ordering-operand`, + ), + ) + } + + return result +} + +function canonicalizeJoin( + join: JoinClause, + path: string, + seen: WeakSet, + scope: AliasScope, +): StableIdentityValue { + return { + type: join.type, + from: canonicalizeSource(join.from, `${path}.from`, seen, scope), + left: canonicalizeExpression( + join.left, + `${path}.left`, + seen, + `equality-operand`, + scope, + ), + right: canonicalizeExpression( + join.right, + `${path}.right`, + seen, + `equality-operand`, + scope, + ), + } +} + +function canonicalizeSource( + source: From, + path: string, + seen: WeakSet, + scope: AliasScope, +): StableIdentityValue { + if (source.type === `collectionRef`) { + return { + type: `collectionRef`, + collectionId: canonicalizeRuntimeValue( + source.collection.id, + `${path}.collection.id`, + seen, + ), + } + } + + if (source.type === `unionFrom`) { + return { + type: `unionFrom`, + sources: source.sources.map((unionSource, index) => + canonicalizeSource( + unionSource, + `${path}.sources[${index}]`, + seen, + scope, + ), + ), + } + } + + if (source.type === `unionAll`) { + return { + type: `unionAll`, + queries: source.queries.map((query, index) => + // Branches are peers that may capture the union query's outer scope; + // they are not children of the union result row itself. + canonicalizeQuery( + query, + `${path}.queries[${index}]`, + seen, + scope.parent, + ), + ), + } + } + + return { + type: `queryRef`, + query: canonicalizeQuery(source.query, `${path}.query`, seen, scope), + } +} + +function canonicalizeSelect( + select: Select, + path: string, + seen: WeakSet, + scope?: AliasScope, +): StableIdentityValue { + return { + type: `select`, + fields: Object.keys(select) + .sort() + .map((key) => [ + key, + canonicalizeSelectValue(select[key]!, `${path}.${key}`, seen, scope), + ]), + } +} + +function canonicalizeSelectValue( + value: unknown, + path: string, + seen: WeakSet, + scope?: AliasScope, +): StableIdentityValue { + if (isRefProxy(value)) { + return canonicalizeExpression( + toExpression(value), + path, + seen, + `exact-output`, + scope, + ) + } + + if (isExpression(value)) { + return canonicalizeExpression(value, path, seen, `exact-output`, scope) + } + + if (isPlainObject(value)) { + return canonicalizeSelect(value as Select, path, seen, scope) + } + + return canonicalizeExactOutputRuntimeValue(value, path, seen) +} + +function canonicalizeWhere( + where: Where | Having, + path: string, + seen: WeakSet, + scope?: AliasScope, +): StableIdentityValue { + if (isWhereObject(where)) { + const result: Record = { + type: `where`, + expression: canonicalizeExpression( + where.expression, + `${path}.expression`, + seen, + `exact-output`, + scope, + ), + } + + if (where.residual === true) { + result.residual = true + } + + return result + } + + return canonicalizeExpression(where, path, seen, `exact-output`, scope) +} + +function canonicalizeOrderBy( + orderBy: OrderByClause, + path: string, + seen: WeakSet, + valueContext: ValueIdentityContext = `exact-output`, + scope?: AliasScope, +): StableIdentityValue { + return { + expression: canonicalizeExpression( + orderBy.expression, + `${path}.expression`, + seen, + valueContext, + scope, + ), + compareOptions: canonicalizeRuntimeValue( + orderBy.compareOptions, + `${path}.compareOptions`, + seen, + ), + } +} + +function canonicalizeExpression( + expression: + | BasicExpression + | Aggregate + | IncludesSubquery + | ConditionalSelect, + path: string, + seen: WeakSet, + valueContext: ValueIdentityContext = `exact-output`, + scope?: AliasScope, +): StableIdentityValue { + if (expression.type === `ref`) { + const binding = resolveAliasBinding(scope, expression.path[0] ?? ``) + return { + type: `ref`, + path: + binding === undefined + ? expression.path.map((segment, index) => + canonicalizeRuntimeValue(segment, `${path}.path[${index}]`, seen), + ) + : [ + [`binding`, ...binding], + ...expression.path + .slice(1) + .map((segment, index) => + canonicalizeRuntimeValue( + segment, + `${path}.path[${index + 1}]`, + seen, + ), + ), + ], + } + } + + if (expression.type === `val`) { + return { + type: `val`, + value: + valueContext === `equality-operand` + ? canonicalizeEqualityRuntimeValue( + expression.value, + `${path}.value`, + seen, + scope, + ) + : valueContext === `ordering-operand` + ? canonicalizeOrderingRuntimeValue( + expression.value, + `${path}.value`, + seen, + ) + : canonicalizeExactOutputRuntimeValue( + expression.value, + `${path}.value`, + seen, + ), + } + } + + if (expression.type === `func`) { + if ( + expression.name === `in` && + expression.args.length === 2 && + expression.args[1]?.type === `val` && + Array.isArray(expression.args[1].value) + ) { + const candidates = expression.args[1].value.map((value, index) => + canonicalizeEqualityRuntimeValue( + value, + `${path}.args[1].value[${index}]`, + seen, + scope, + ), + ) + return canonicalizeFunction(expression.name, [ + canonicalizeExpression( + expression.args[0]!, + `${path}.args[0]`, + seen, + `equality-operand`, + scope, + ), + { + type: `val`, + // IN tests membership. Candidate order and duplicates do not change + // its result, but each candidate keeps its own equality semantics. + value: [`set`, sortUniqueStableIdentityValues(candidates)], + }, + ]) + } + + const operandContext: ValueIdentityContext = + expression.name === `eq` + ? `equality-operand` + : expression.name === `gt` || + expression.name === `gte` || + expression.name === `lt` || + expression.name === `lte` + ? `ordering-operand` + : `exact-output` + const args = expression.args.map((arg, index) => + canonicalizeExpression( + arg, + `${path}.args[${index}]`, + seen, + operandContext, + scope, + ), + ) + return canonicalizeFunction(expression.name, args) + } + + if (expression.type === `agg`) { + return { + type: `agg`, + name: expression.name, + args: expression.args.map((arg, index) => + canonicalizeExpression( + arg, + `${path}.args[${index}]`, + seen, + `exact-output`, + scope, + ), + ), + } + } + + if (expression.type === `conditionalSelect`) { + const result: Record = { + type: `conditionalSelect`, + branches: expression.branches.map((branch, index) => ({ + condition: canonicalizeExpression( + branch.condition, + `${path}.branches[${index}].condition`, + seen, + `exact-output`, + scope, + ), + value: canonicalizeSelectValue( + branch.value, + `${path}.branches[${index}].value`, + seen, + scope, + ), + })), + } + + if (expression.defaultValue !== undefined) { + result.defaultValue = canonicalizeSelectValue( + expression.defaultValue, + `${path}.defaultValue`, + seen, + scope, + ) + } + + return result + } + + const childScope = createAliasScope(expression.query, scope) + const result: Record = { + type: `includesSubquery`, + query: canonicalizeQueryInScope( + expression.query, + `${path}.query`, + seen, + childScope, + ), + correlationField: canonicalizeExpression( + expression.correlationField, + `${path}.correlationField`, + seen, + `equality-operand`, + scope, + ), + childCorrelationField: canonicalizeExpression( + expression.childCorrelationField, + `${path}.childCorrelationField`, + seen, + `equality-operand`, + childScope, + ), + fieldName: expression.fieldName, + materialization: expression.materialization, + } + + if (expression.parentFilters) { + result.parentFilters = expression.parentFilters.map((where, index) => + canonicalizeWhere(where, `${path}.parentFilters[${index}]`, seen, scope), + ) + } + + if (expression.parentProjection) { + result.parentProjection = expression.parentProjection.map( + (projection, index) => + canonicalizeExpression( + projection, + `${path}.parentProjection[${index}]`, + seen, + `exact-output`, + scope, + ), + ) + } + + if (expression.scalarField !== undefined) { + result.scalarField = expression.scalarField + } + + return result +} + +function canonicalizeFunction( + name: string, + args: Array, +): StableIdentityValue { + if ((name === `and` || name === `or`) && args.length > 0) { + const flattened = args.flatMap((arg) => + isCanonicalFunction(arg, name) ? arg.args : [arg], + ) + const unique = sortUniqueStableIdentityValues(flattened) + + // The evaluator gives `and` and `or` boolean results even when their sole + // operand returns another truthy or falsy value. Keep that coercion in the + // identity because expression result types are erased at runtime. + return { type: `func`, name, args: unique } + } + + if (name === `eq` && args.length === 2) { + args.sort(compareStableIdentityValues) + return { type: `func`, name, args } + } + + if ( + (name === `gt` || name === `gte` || name === `lt` || name === `lte`) && + args.length === 2 && + compareStableIdentityValues(args[0]!, args[1]!) > 0 + ) { + return { + type: `func`, + name: invertComparison(name), + args: [args[1]!, args[0]!], + } + } + + return { type: `func`, name, args } +} + +function sortUniqueStableIdentityValues( + values: Array, +): Array { + const keyedValues = values.map((value) => ({ + key: JSON.stringify(value), + value, + })) + keyedValues.sort((a, b) => (a.key < b.key ? -1 : a.key > b.key ? 1 : 0)) + return keyedValues + .filter( + (entry, index) => + index === 0 || entry.key !== keyedValues[index - 1]!.key, + ) + .map((entry) => entry.value) +} + +function isCanonicalFunction( + value: StableIdentityValue, + name: string, +): value is { + type: string + name: string + args: Array +} { + return ( + value !== null && + typeof value === `object` && + !Array.isArray(value) && + value.type === `func` && + value.name === name && + Array.isArray(value.args) + ) +} + +function invertComparison( + name: `gt` | `gte` | `lt` | `lte`, +): `gt` | `gte` | `lt` | `lte` { + switch (name) { + case `gt`: + return `lt` + case `gte`: + return `lte` + case `lt`: + return `gt` + case `lte`: + return `gte` + } +} + +function canonicalizeRuntimeValue( + value: unknown, + path: string, + seen: WeakSet, +): StableIdentityValue { + if (value === null) return [`null`] + + if (typeof value === `string`) { + return [`string`, value] + } + + if (typeof value === `boolean`) { + return [`boolean`, value] + } + + if (typeof value === `number`) { + if (Number.isNaN(value)) { + return [`number`, `NaN`] + } + + if (value === Infinity) { + return [`number`, `Infinity`] + } + + if (value === -Infinity) { + return [`number`, `-Infinity`] + } + + if (Object.is(value, -0)) { + return [`number`, `-0`] + } + + return [`number`, value] + } + + if (typeof value === `undefined`) { + return [`undefined`] + } + + if (typeof value === `bigint`) { + return [`bigint`, value.toString()] + } + + if (typeof value === `function`) { + throw new UnhashableQueryIRError(path, `function value`) + } + + if (typeof value === `symbol`) { + throw new UnhashableQueryIRError(path, `symbol value`) + } + + if (isRefProxy(value)) { + return canonicalizeExpression(toExpression(value), path, seen) + } + + if (Array.isArray(value)) { + return withCircularGuard(value, path, seen, () => [ + `array`, + value.map((item, index) => + canonicalizeRuntimeValue(item, `${path}[${index}]`, seen), + ), + ]) + } + + if (value instanceof Date) { + const timestamp = value.getTime() + if (Number.isNaN(timestamp)) { + throw new UnhashableQueryIRError(path, `invalid Date`) + } + + return [`Date`, value.toISOString()] + } + + if (value instanceof ArrayBuffer) { + return [`binary`, `ArrayBuffer`, Array.from(new Uint8Array(value))] + } + + if (ArrayBuffer.isView(value)) { + return [ + `binary`, + value.constructor.name, + Array.from( + new Uint8Array(value.buffer, value.byteOffset, value.byteLength), + ), + ] + } + + if (value instanceof Map) { + return withCircularGuard(value, path, seen, () => { + const entries = Array.from( + value.entries(), + ([key, entryValue], index) => [ + canonicalizeRuntimeValue(key, `${path}.key[${index}]`, seen), + canonicalizeRuntimeValue(entryValue, `${path}.value[${index}]`, seen), + ], + ) + entries.sort(compareStableIdentityValues) + return [`Map`, entries] + }) + } + + if (value instanceof Set) { + return withCircularGuard(value, path, seen, () => { + const entries = Array.from(value, (entry, index) => + canonicalizeRuntimeValue(entry, `${path}[${index}]`, seen), + ) + entries.sort(compareStableIdentityValues) + return [`Set`, entries] + }) + } + + if (isPlainObject(value)) { + return canonicalizeObject(value, path, seen) + } + + throw new UnhashableQueryIRError(path, `non-plain object value`) +} + +function canonicalizeExactOutputRuntimeValue( + value: unknown, + path: string, + seen: WeakSet, +): StableIdentityValue { + if ( + (typeof value === `object` && value !== null) || + typeof value === `function` || + typeof value === `symbol` + ) { + return getRuntimeReferenceIdentity(value) + } + + return canonicalizeRuntimeValue(value, path, seen) +} + +function canonicalizeEqualityRuntimeValue( + value: unknown, + path: string, + seen: WeakSet, + scope?: AliasScope, +): StableIdentityValue { + if (isRefProxy(value)) { + return canonicalizeExpression( + toExpression(value), + path, + seen, + `equality-operand`, + scope, + ) + } + + if (typeof value === `number` && Object.is(value, -0)) { + return canonicalizeRuntimeValue(0, path, seen) + } + + // Equality compares Uint8Array and Buffer values by content, independent of + // their concrete constructor and size. + const isUint8Array = + (typeof Buffer !== `undefined` && value instanceof Buffer) || + value instanceof Uint8Array + if (isUint8Array) { + return [`binary`, `Uint8Array`, Array.from(value as Uint8Array)] + } + + const normalized = normalizeValue(value) + if (normalized !== value) { + return canonicalizeRuntimeValue(normalized, path, seen) + } + + if ( + (typeof value === `object` && value !== null) || + typeof value === `function` || + typeof value === `symbol` + ) { + return getRuntimeReferenceIdentity(value) + } + + return canonicalizeRuntimeValue(value, path, seen) +} + +function canonicalizeOrderingRuntimeValue( + value: unknown, + path: string, + seen: WeakSet, +): StableIdentityValue { + if (typeof value === `number` && Object.is(value, -0)) { + return canonicalizeRuntimeValue(0, path, seen) + } + + if (value instanceof Date && Number.isNaN(value.getTime())) { + return canonicalizeRuntimeValue(Number.NaN, path, seen) + } + + const normalized = normalizeValue(value) + if (normalized !== value && !(value instanceof Uint8Array)) { + return canonicalizeRuntimeValue(normalized, path, seen) + } + + try { + return canonicalizeRuntimeValue(value, path, seen) + } catch (error) { + if ( + error instanceof UnhashableQueryIRError && + typeof value === `object` && + value !== null + ) { + return getRuntimeReferenceIdentity(value) + } + throw error + } +} + +function compareStableIdentityValues( + left: StableIdentityValue, + right: StableIdentityValue, +): number { + const serializedLeft = JSON.stringify(left) + const serializedRight = JSON.stringify(right) + return serializedLeft < serializedRight + ? -1 + : serializedLeft > serializedRight + ? 1 + : 0 +} + +function canonicalizeObject( + value: Record, + path: string, + seen: WeakSet, +): StableIdentityValue { + return withCircularGuard(value, path, seen, () => [ + `object`, + Object.keys(value) + .sort() + .map((key) => [ + key, + canonicalizeRuntimeValue(value[key], `${path}.${key}`, seen), + ]), + ]) +} + +function withCircularGuard( + value: object, + path: string, + seen: WeakSet, + callback: () => T, +): T { + if (seen.has(value)) { + throw new UnhashableQueryIRError(path, `circular value`) + } + + seen.add(value) + try { + return callback() + } finally { + seen.delete(value) + } +} + +function isWhereObject( + where: Where | Having, +): where is { expression: BasicExpression; residual?: boolean } { + return `expression` in where +} + +function isExpression( + value: unknown, +): value is BasicExpression | Aggregate | IncludesSubquery { + if (value === null || typeof value !== `object`) { + return false + } + + const expressionType = (value as { type?: unknown }).type + return ( + expressionType === `agg` || + expressionType === `conditionalSelect` || + expressionType === `func` || + expressionType === `ref` || + expressionType === `val` || + expressionType === `includesSubquery` + ) +} diff --git a/packages/db/src/query/ir.ts b/packages/db/src/query/ir.ts index 48019a23b2..a04370a90a 100644 --- a/packages/db/src/query/ir.ts +++ b/packages/db/src/query/ir.ts @@ -74,6 +74,8 @@ export type Limit = number export type Offset = number +let nextCollectionSourceId = 0 + /* Expressions */ abstract class BaseExpression { @@ -84,11 +86,17 @@ abstract class BaseExpression { export class CollectionRef extends BaseExpression { public type = `collectionRef` as const + /** Opaque runtime identity; aliases are lexical names only. */ + public readonly sourceId!: string constructor( public collection: CollectionImpl, public alias: string, ) { super() + Object.defineProperty(this, `sourceId`, { + value: `source-${++nextCollectionSourceId}`, + enumerable: false, + }) } } @@ -181,7 +189,7 @@ export class IncludesSubquery extends BaseExpression { public childCorrelationField: PropRef, // Child-side ref (e.g., issue.projectId) public fieldName: string, // Result field name (e.g., "issues") public parentFilters?: Array, // WHERE clauses referencing parent aliases (applied post-join) - public parentProjection?: Array, // Parent field refs used by parentFilters + public parentProjection?: Array, // Parent field refs used anywhere in the child plan public materialization: IncludesMaterialization = `collection`, public scalarField?: string, ) { @@ -254,6 +262,55 @@ export function isExpressionLike(value: any): boolean { return false } +/** Returns each lexical Collection source in a query tree once. */ +export function collectCollectionSources(query: QueryIR): Array { + const sources: Array = [] + const seen = new Set() + + const visitSource = (source: QueryIR[`from`]): void => { + if (source.type === `collectionRef`) { + if (!seen.has(source.sourceId)) { + seen.add(source.sourceId) + sources.push(source) + } + } else if (source.type === `queryRef`) { + visitQuery(source.query) + } else if (source.type === `unionFrom`) { + source.sources.forEach(visitSource) + } else { + source.queries.forEach(visitQuery) + } + } + + const visitSelectValue = (value: any): void => { + if (value instanceof IncludesSubquery) { + visitQuery(value.query) + } else if (value instanceof ConditionalSelect) { + value.branches.forEach((branch) => visitSelectValue(branch.value)) + if (value.defaultValue !== undefined) { + visitSelectValue(value.defaultValue) + } + } else if ( + value !== null && + typeof value === `object` && + !Array.isArray(value) && + !isExpressionLike(value) && + value.__refProxy !== true + ) { + Object.values(value).forEach(visitSelectValue) + } + } + + const visitQuery = (current: QueryIR): void => { + visitSource(current.from) + current.join?.forEach(({ from }) => visitSource(from)) + if (current.select) Object.values(current.select).forEach(visitSelectValue) + } + + visitQuery(query) + return sources +} + /** * Helper functions for working with Where clauses */ @@ -299,18 +356,21 @@ export function createResidualWhere( return { expression, residual: true } } +/** Sources declared by a FROM clause. UnionAll branches own their sources. */ +export function getFromSources(from: From): Array { + if (from.type === `unionFrom`) return from.sources + if (from.type === `unionAll`) return [] + return [from] +} + function getRefFromAlias( query: QueryIR, alias: string, ): CollectionRef | QueryRef | void { - if (query.from.type === `unionFrom`) { - for (const source of query.from.sources) { - if (source.alias === alias) { - return source - } + for (const source of getFromSources(query.from)) { + if (source.alias === alias) { + return source } - } else if (query.from.type !== `unionAll` && query.from.alias === alias) { - return query.from } for (const join of query.join || []) { @@ -323,13 +383,22 @@ function getRefFromAlias( /** * Follows the given reference in a query * until its finds the root field the reference points to. - * @returns The collection, its alias, and the path to the root field in this collection + * @returns The collection, its alias, and the path to the root field in this collection. + * `alias` is the alias under which the resolved collection is referenced in the + * query it was reached from (when the ref crosses into a joined source). It is + * left undefined when the ref simply resolves to a field on the passed-in + * `collection`, in which case the caller already knows the alias. */ export function followRef( query: QueryIR, ref: PropRef, collection: Collection, -): { collection: Collection; path: Array } | void { +): { + collection: Collection + path: Array + alias?: string + sourceId?: string +} | void { if (ref.path.length === 0) { return } @@ -365,8 +434,15 @@ export function followRef( } else { // This is a reference to a collection // we can't follow it further - // so the field must be on the collection itself - return { collection: aliasRef.collection, path: rest } + // so the field must be on the collection itself. + // Report the alias too: when the ref crossed a join, this is the source + // that actually holds the field (which may differ from the from clause). + return { + collection: aliasRef.collection, + path: rest, + alias, + sourceId: aliasRef.sourceId, + } } } } diff --git a/packages/db/src/query/live-query-collection.ts b/packages/db/src/query/live-query-collection.ts index 8649bc0bc1..893c8f10a4 100644 --- a/packages/db/src/query/live-query-collection.ts +++ b/packages/db/src/query/live-query-collection.ts @@ -190,9 +190,13 @@ export function createLiveQueryCollection< // been validated by the public signatures, but the branch loses that precision. const options = liveQueryCollectionOptions(config as any) - // Merge custom utils if provided, preserving the getBuilder() method for dependency tracking + // Merge custom utils without evaluating internal getters such as + // lastSubsetError into stale data properties. if (config.utils) { - options.utils = { ...options.utils, ...config.utils } + Object.defineProperties( + options.utils, + Object.getOwnPropertyDescriptors(config.utils), + ) } return bridgeToCreateCollection(options) as CollectionForContext< diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md new file mode 100644 index 0000000000..c8b9896093 --- /dev/null +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -0,0 +1,1182 @@ +# Live-query materialization architecture + +This document defines the architecture for correlated live-query +materialization in `@tanstack/db`. It follows +[RFC #1658](https://github.com/TanStack/db/issues/1658). + +The central rule is simple: + +> Keep relation contents, routes, nested materialization, and propagation in +> one D2 graph. Use custom state only at asynchronous source and public +> Collection boundaries. + +The correlated-materialization oracle suites listed below are behavioral +contracts for this design. Functional projections accept inline include values, +not compiled Collection-valued inputs. Suites for +adjacent planner and query-db ownership boundaries may also contain exact +classifiers for defects outside this graph. + +## Scope + +This architecture covers: + +- compiled identities for sources, relations, and materialization edges; +- weighted contributions and public-key reduction; +- correlated routes and ordered bucket contents; +- nested inline and Collection-valued materialization; +- lazy and progressive source demand; +- coherent publication to public Collections; +- the boundaries with query-db ownership and physical query planning. + +The applied-settlement receipt described below is its only new public boundary +contract. Optimistic transactions are another source of weighted input changes; +they do not have a separate routing model. + +## One relational graph + +Correlated materialization is part of the compiled D2 graph, not a second +incremental engine around its output. + +```text +raw weighted query rows + | + v +public-key reduction + | + v +CanonicalRow(base row, order, outgoing parameters) + | + +------------------------------+ + | | + v | +Route(bucket, cell) | + | | + +--> distinct --> ActiveBucket-+--> async demand adapter + | | | + | v | + | child rows --> BucketValue + | | + +------------------------------+ + | + v + CellValue(cell, value) + | + v + CanonicalRow + outgoing CellValues + | + v + MaterializedRow + | + v + one normal root Collection transaction +``` + +Canonical base rows flow down to derive correlation routes and source demand. +Fully materialized child rows flow up into their parents. Because the include +graph is acyclic, these streams form one acyclic D2 graph even though demand +and results move in opposite conceptual directions. + +The graph owns the data plane. A small adapter owns asynchronous demand. The +normal Collection transaction boundary owns public publication. + +## Concrete implementation map + +The relation and identity names in this document describe the graph's logical +model. They are not a second set of runtime objects, nor does every name need a +matching TypeScript type. The implementation maps this model onto existing D2 +operators and a few boundary adapters: + +| Architectural role | Concrete implementation | +| ----------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| Compile relation IDs and demand plans | `packages/db/src/query/compiler/index.ts`, `packages/db/src/query/compiler/joins.ts` | +| Reduce public keys and build routes | `packages/db/src/query/live/materialized-pipeline.ts` | +| Run the graph and publish root rows | `packages/db/src/query/live/collection-config-builder.ts` | +| Publish Collection-valued buckets | `packages/db/src/query/live/bucket-facade-adapter.ts` | +| Start and release asynchronous demand | `packages/db/src/query/live/subset-demand-controller.ts`, `packages/db/src/collection/subscription.ts` | +| Ordered provider loading and continuation | `packages/db/src/query/live/ordered-source-loader.ts` | + +Queries without includes keep the original compiled pipeline and do not pay +for facade state. The one exception is a joined query with a custom public-key +function: its possible duplicate contributors still pass through the keyed +reduction that enforces public-key congruence and multiplicity. + +### Loading handoffs + +These owners cooperate; they are not phases of one exclusive state machine. +The detailed loading and publication laws below still apply. + +| Owner | Accepts / retires | Does not establish | +| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | +| Subscription acquisition | Retires the old physical lease before replay acquisition; installs tentative ownership before adapter callbacks; each lease gets one cleanup attempt | Replay completion or permission to publish | +| OrderedSourceLoader | Tracks request settlement, safe continuation and repair debt; reset discards the cursor, disposal ignores late settlement | Provider exhaustion or acceptance of an imperative window | +| Subscription replay | Counts setup and logical acquisition participants; checks completion after reentrant release callbacks; success releases the source replacement hold | Success of a previously failed window operation | +| Query builder | Tracks ordered publication participants in one sync session and accepts a window only for its operation generation | Physical adapter ownership or cancellation | +| D2 and public Collection boundary | D2 accumulates private result changes; the builder flushes root and child changes when the existing gates allow it | Source completeness merely because graph work drained | + +Session and participant checks precede changes to the builder's ordered failure +state, not just scheduling. An obsolete rejection cannot close a replacement +session's publication gate. Loader-local stale-result guards are separate. + +`hasPendingTruncateReplacement` means publication is still withheld, including +after replay failure. `pendingTruncateReplacement` exposes only an unsettled +completion promise. Neither is a general readiness flag. A direct subscriber +buffers and diffs its own replacement rows; a query subscription delegates +publication to the builder while the graph keeps its private contributions. + +## Identity + +Aliases are lexical query-language names rather than source runtime identities. +The query builder requires collection aliases to be unique within each lexical +scope and rejects nested queries that shadow an ancestor alias. Sibling include +scopes may reuse an alias because neither alias is visible to the other. +Compilation then assigns opaque IDs to the accepted plan: + +```ts +type SourceId = Brand +type RelationNodeId = Brand +type MaterializationEdgeId = Brand +``` + +An explicit projection can alpha-normalize aliases because its field names +define the public shape. Without a projection, joined and grouped queries return +a namespaced row whose keys are the lexical aliases. Those observable keys are +part of query identity. Alias text may otherwise remain as debug metadata +without becoming source identity. + +A `CanonicalCorrelationKey` is the canonical tuple of every evaluated +parent-dependent value that can affect the child plan. This includes values +used by filters, joins, grouping, aggregates, ordering, projections, limits, +and nullable predicates, not only the obvious foreign-key equality. + +A bucket key identifies one such correlated partition at one relation node: + +```ts +type BucketKey = readonly [ + relationNodeId: RelationNodeId, + correlationKey: CanonicalCorrelationKey, +] +``` + +Correlation equality must use the same value semantics as query predicates. +Implementations use canonical values, interned handles, or nested maps; they do +not reconstruct array or object keys and expect JavaScript `Map` identity to +match. Equality tokens collapse `-0` with `0`, compare Date, Temporal, and +binary values by the same normalized value as `eq`/`in`, and retain runtime +reference identity for other objects, functions, and symbols. These tokens are +valid only for equality-keyed routing, grouping, and demand. Output values and +arbitrary function arguments keep their exact runtime identity and value. +Tree indexes give symbols a stable runtime-local order because JavaScript +relational comparison throws for them; comparator equality still holds only +for the same symbol. That order is a physical index detail: symbol range +predicates fall back to the evaluator instead of treating it as query +semantics. Range predicates also fall back when the live indexed values do not +share the bound's relational domain. An index's advertised comparison options +also define its executable comparator; metadata cannot claim an order that the +index does not use. Explicit `undefined` range and cursor bounds denote the +indexed nullish comparator group, while an absent bound denotes the start or +end of the index. An ordered index groups exact value +buckets that compare at the same position and keeps a live representative for +each group, so range traversal and ordered limits cannot drop rows whose +distinct values are comparator-equal. +Compiler tokens belong to one compiled graph. This keeps every operator in the +graph on the same identity relation. Objects, functions, and local symbols are +weakly keyed where the runtime supports weak symbol keys. Older runtimes retain +local symbols strongly within the scope rather than collapse distinct symbols +and corrupt equality. Registered symbols use their registry key because the +runtime registry already retains them. A demand controller owns a separate +scope and discards it when the controller is cleared. Process-wide query +identity and opaque public group keys keep their own runtime scope because +equivalent query plans and retained public keys must survive graph replacement. +For grouping, the equality token is the D2 group key. The group retains a raw +value from a currently positive contributor only as the projected +representative. The representative is chosen by stable source-row identity, so +restoring the same source state restores the same value regardless of update +history. D2 sees only safe exact-value identity for that representative, not +the raw value itself. A separate public group key preserves primitive keys and +serializes opaque equality identity; graph-local identity tokens never cross +the Collection boundary. Compiler group fields use a query-local namespace +disjoint from every selected alias. Direct correlated joins canonicalize both +sides before the first D2 join; normalizing only the later group key is too +late. + +### Route-context transport + +A parent reference is a lexical dependency, even when it appears below the +immediate child query. For every parent reference that the builder can inspect, +the compiler must: + +1. discover it across nested includes, `QueryRef` sources, union branches, and + joined sources; +2. include its evaluated value in the route identity; +3. attach that route context before the first operator that evaluates it; and +4. preserve it through each later recursive source, join, grouping, and + materialization edge. + +The third rule fixes the evaluation order. A parent-dependent filter, join key, +aggregate wrapper, order, or window must run once per parent route. It cannot +run on a shared child relation first and receive a route after the fact. + +The route-context grammar crosses these dimensions: + +```text +lexical dependency scope + x recursive source boundary (nested include, QueryRef, union) + x recursive result shape (record, scalar, nullable scalar) + x evaluation phase (filter, join, group, aggregate, order, window) + x join side and correlation attachment point + x materialization form + x parent or child update +``` + +Adding one dimension to the query language requires checking its product with +the others. A passing one-level filter case does not prove a nested aggregate, +joined subquery, or union branch transports the same context. + +The executable oracle factors that product into valid compiler sub-grammars: + +- parent field projection by whole-row projection; +- unmatched correlation values by null correlation values; +- lexical scope, including nested outer and inner materialization forms; +- grouping mode by aggregate-expression placement; +- recursive source boundary by evaluation phase; and +- join-key side by correlation attachment point; +- union form and public-key identity; and +- derived-result boundary by selection mode and scalar nullability; and +- user namespace collision by parent alias and selected child field, crossed + with direct, `QueryRef`, join, and group boundaries; and +- public-surface shape across opaque atomic values, opaque wrappers, nested + reference identity, user symbol keys, adversarial property keys, functional + spreads, and implicit joins. + +Plain record results carry route metadata under a private symbol while the +compiler moves them through recursive sources. Primitives and opaque objects, +such as `Date`, use an internal envelope at those same edges. Namespacing and +join adapters unwrap the value and keep its route beside it. Every functional +callback whose source can carry route state receives a clean copy of only the +paths that contain private state; this includes recursive and union sources, +not only directly correlated child queries. The callback boundary removes all +compiler-owned fields before invoking user code. The publication boundary +applies the same copy-on-write walk while resolving facade references. Both +paths preserve property descriptors, clean nested references, cycles, +adversarial keys, and user-owned symbols. Discovery reads data descriptors +directly and never invokes an accessor merely to find private state. + +This walker also strips metadata from a correlated subquery's output before +its parent query consumes it. Clean object and array references at that internal +boundary are equality operands, not just render identities. Eagerly cloning +them can make a later `eq(projected.key, parent.key)` lose a matching row. +Relaxing cross-publication reference stability does not permit changing these +internal matches. The public-container copy matrix crosses reference-key type, +ordered and unordered subqueries, materialization form, and parent/child updates. + +D2 hashes enumerable symbol keys and uses exact local-symbol identity plus +registry keys for registered symbols. D2 rejects structural cycles with a clear +error, including cycles through arrays, Maps, Sets, and enumerable symbol keys. +Shared acyclic subtrees remain supported and are hashed once per traversal. Structural hashing +limits recursion depth and value visits; it rejects values that exceed these +limits instead of expanding a shared graph or overflowing the JavaScript stack. +This does not bound the cost of arbitrary user getters or key sorting. +A failed hash does not publish partial +structural cache entries, so retrying the same value cannot bypass a guard. +A graph-run failure marks the current live query as errored and preserves the +thrown error. It must not continue publishing from a partly advanced graph; +recovery requires a fresh query session. +Opaque reference-hashed leaves are resolved before structural recursion; their +own properties, including self-references, are not traversed. Hash inputs must +remain immutable once successfully cached, as with other retained D2 values. +Collections register as opaque handles at construction using the existing hash +cache. Their identity, not their mutable internal state, is visible to hashing +operators in a downstream query. This does not add child-row dependencies to a +functional projection that reads a Collection-valued field. +The descriptor-preserving boundary walkers may still encounter cycles, but that +does not make cyclic structural results valid input to a hashing operator. +Symbol-only changes cannot disappear before publication, and unsupported cycles +fail rather than silently merge. Neither +boundary mutates values retained by D2. Compiler-created +parent contexts use a separate internal +envelope that keeps projected user aliases apart from the equality identity +derived from their leaves. The whole parent-context envelope is structural D2 +state. This avoids reserving user aliases or selected field names while keeping +the context stable across D2 operators without collapsing two +reference-sensitive leaf values that happen to have the same object shape. + +A functional projection consumes fully materialized inline input before +downstream operators run. Compiled Collection-valued includes are not supported +as `fn.select()` inputs, including nested descendants. The compiler rejects +the plan before invoking the callback, even if the callback would ignore or +pass through the Collection. This keeps callbacks inside the ordinary D2 +pipeline without temporary Collection views or graph continuations. + +Use `toArray()` or `materialize()` in the upstream expression `select()` to +make child values available to a functional callback. Child changes then +update the inline value and rerun the projection. To keep live child +Collections, use expression projections, or do parent-only functional work +before adding the Collection-valued include. This restriction concerns compiled +include inputs; it does not inspect arbitrary source-row fields or captured +Collections. Reading an already published Collection from a callback does not +add a child-row dependency. + +Include paths describe a functional projection's input, not its arbitrary +output. A callback may drop or rename a field, or return a scalar. Its input +paths must not be attached to that output by a downstream QueryRef consumer. +The compiler consumes those descriptors through the existing D2 materializer; +downstream keys, distinct, ordering, and QueryRef consumers see the callback's +actual output. Queries without includes keep their original pipeline. + +The projection matrices keep Collection-input cases as rejection checks and +exercise supported inline forms across route changes, child updates, recursive +sources, unions, and chained callbacks. Expression controls retain ordinary +Collection reads, indexes, subscriptions, rollback, pending loads, and +cleanup/restart coverage. Work counters check repeated reads on public facades. +A manual forced-GC probe checks retained public handles and captured methods +after cleanup, with live facades as a positive retention control; it is not a +whole-application heap or throughput measurement. + +A materialization cell identifies one include field on one parent-row +occurrence: + +```ts +type MaterializationCellId = readonly [ + containingBucket: BucketKey | 'root', + parentPublicKey: PublicKey, + edgeId: MaterializationEdgeId, +] +``` + +The containing bucket prevents equal child keys in separate correlated +contexts from colliding. + +Only work that crosses an asynchronous boundary needs a generation token. A +live-query graph generation invalidates work from an old graph. A demand +generation invalidates an old load for the same bucket. Synchronous route rows +inside D2 do not need their own lifecycle objects or generations. + +## Weighted relations and public keys + +D2 multisets are the source of truth. A row with positive weight contributes; +a row with negative weight retracts the same contribution. + +Internal contribution identity is independent of the user-visible Collection +key. When several internal rows collapse to one public key, a keyed D2 +reduction retains all contributors and derives at most one canonical row: + +```ts +type CanonicalRow = { + publicKey: PublicKey + value: Row + order: OrderKey | undefined + outgoingParameters: ReadonlyMap< + MaterializationEdgeId, + CanonicalCorrelationKey + > +} +``` + +```text +raw weighted rows + -> reduce by [containing bucket, public key] + -> CanonicalRow + +-> derive route rows + +-> compose with materialized include values +``` + +For every affected public key, the reduction compares its complete before and +after state and emits no change, one replacement, or one removal. It does not +infer the previous state from `collection.has()`. + +Routes derive only from canonical rows. Raw contributors never create routes +that must later be reconciled. The same boundary applies recursively: roots +reduce by root public key, while child rows reduce by their containing bucket +and child public key. + +All positive contributors collapsed under one public key must be congruent on: + +- the visible value; +- the total order key; +- every outgoing correlation input. + +Query aggregation occurs upstream in the query graph. This reduction only +preserves multiplicity while collapsing congruent contributors under the +public Collection key. Incongruent contributors are a duplicate-key invariant +error; flush order never chooses a winner. A zero aggregate removes the public +row. A negative aggregate is an invariant violation. + +This is a specialized use of the existing D2 keyed reduction. It is not a +separate contribution-ledger subsystem. + +## Routes and buckets are relations + +For each materialization edge, the compiler produces these keyed relations: + +```ts +type RouteRow = readonly [bucketKey: BucketKey, cellId: MaterializationCellId] + +type ActiveBucket = readonly [bucketKey: BucketKey] + +type BucketRow = readonly [ + bucketKey: BucketKey, + child: readonly [publicKey: PublicKey, row: Row, order: OrderKey | undefined], +] + +type BucketValue = readonly [bucketKey: BucketKey, value: Value] + +type CellValue = readonly [cellId: MaterializationCellId, value: Value] +``` + +A route move is an ordinary weighted batch: + +```text +-1 [old bucket, cell] ++1 [new bucket, cell] +``` + +Distinct route keys produce `ActiveBucket`. For inline modes, child rows are +ordered and reduced once per active bucket into exactly one `BucketValue`. +Routes then join with bucket values to fan the same immutable logical value out +as a `CellValue`: + +```text +Route(bucket, cell) -> distinct -> ActiveBucket(bucket) + | +ActiveBucket + BucketRow -> reduce ------+-> BucketValue(bucket, value) + | +Route(bucket, cell) -------------------------------+ + v + CellValue(cell, value) +``` + +The bucket-value reduction belongs to the materialization edge because two +edges may apply different materialization modes to the same child relation. +Computing it before fan-out means ordering and materialization happen once per +unique bucket rather than once per parent. + +`ActiveBucket` also seeds the empty value. Every active inline bucket therefore +has exactly one value even when it has no child rows: + +- `array`: `[]`; +- `singleton`: `undefined`; +- `concat`: `""`. + +A null or otherwise unsatisfiable correlation may route to an active empty +bucket without creating source demand. This preserves the materialization +mode's empty value instead of relying on a placeholder or a missing join path. + +D2's retained join indexes provide the required lifecycle behavior: + +- adding a route joins it with the bucket's existing value; +- removing a route retracts only that cell's value; +- moving a route retracts the old rows and adds the new rows in one graph run; +- several cells may consume one bucket without recomputing its value; +- changing a bucket value reaches every current route; +- a departed route receives no later value changes. + +Root rows and nested rows use the same relation shape and operators. There is +no special root routing path. + +The implementation must not recreate these semantics with route registries, +reverse indexes, drained buffers, or per-depth snapshots outside D2. Existing +retained operator state is the first implementation choice. Add a reusable +arrangement only if counters show that the compiler duplicates indexes or +state; arrangements are a physical optimization, not part of correctness. + +The total-materialization law is: + +> Every active inline materialization cell has exactly one canonical value, +> including when its bucket contains no rows. + +## Nested materialization + +The compiler builds each include from the materialized output relation of its +child: + +```text +child base rows + + child include values + -> child materialized rows + -> rows in the parent's bucket relation + -> parent include value +``` + +A descendant update therefore becomes an ordinary change to the child's +materialized row and propagates through the same joins and reductions at every +depth. There are no depth-specific flush passes, dirty-cell registries, or +manual relation revisions. + +Inline modes are reductions over the rows in one active bucket: + +- `array`: total-order the rows and return their values; +- `singleton`: choose the first row under the total order; +- `concat`: total-order the rows and concatenate their scalar values. + +A total order is the query's order keys followed by a deterministic stable +tie-breaker, normally the child public key. An order-only change is a +bucket-value change for arrays, singletons, concatenation, and Collection +layout. + +A bare child query is a Collection-valued include. It exposes one stable public +Collection facade per active bucket in that edge: + +```text +ActiveBucket + BucketRow -> ActiveBucketRow -> BucketFacade(bucket, Collection) +Route + BucketFacade -> CellValue(cell, Collection) +``` + +Parents sharing a bucket share its facade. Child changes update that Collection +without re-emitting every parent, and moving a route changes the parent field to +the destination bucket's facade. A facade is never retargeted to another +bucket. The D2 join retains inactive bucket rows and emits their current +snapshot when the bucket becomes active; the facade adapter does not buffer +discarded deltas. The adapter retains a facade only while at least one parent +route uses its bucket. When the last route leaves, it retracts the facade's rows +and drops its strong reference. An external holder may keep that empty +Collection alive, but a later active interval gets a new facade. Inline modes +do not create child Collections. + +Composition is pure. It constructs a new result along changed paths and does +not mutate a previously published row or use public routing metadata: + +```ts +compose( + baseRow: BaseRow, + includeValues: ReadonlyMap, +): MaterializedRow +``` + +When a parent result changes, unchanged inline include arrays may receive new +object identities. Cross-publication `===` equality for those arrays is not a +contract. Their values and prior snapshots must remain correct; a downstream +query whose selected result is unchanged must not emit a spurious update. This +does not guarantee that a UI component using shallow prop comparison skips a +render, nor does it relax the stable public Collection facade contract above. + +## Demand plane + +### Demand grouping and ownership + +Demand is derived from data, but it performs asynchronous side effects outside +D2: + +```text +ActiveBucket(bucket, demand parameters) + -> group by [source, parameterized child plan] + -> current demanded parameter set + -> demand adapter + -> source loadSubset / release + -> source deltas return to D2 inputs +``` + +The adapter groups demand into shared source work, not one request per bucket: + +```ts +type DemandPlanId = Brand + +type DemandSet = readonly [ + planId: DemandPlanId, + parameters: CanonicalSet, +] +``` + +One request may serve many buckets according to the compiled demand plan. +This does not imply transport sharing between independent subscriptions. +The exact-request deduper reuses completed requests and shares in-flight work +only when callers supply no abort signal. Independently cancelable requests +use separate transports, trading duplicate concurrent fetches for simpler +ownership. An adapter may share its own resources, but releasing one owner +must not cancel work or remove rows still owned by another. + +Request data is immutable from submission onward, including the options, +expression trees, comparison options, and constant payloads such as Dates, +byte arrays, and membership arrays. Core and adapters retain that data without +cloning or freezing it. Changed demand needs new request data, not edits to an +old constant, even after its first load settles: deduplication and query state +may retain its identity. Request data uses stable data properties, not stateful +getters. The signal and subscription references do not change, but their +lifecycle remains live. Cancellation and release are not data mutations. +The immutable-demand boundary matrix checks direct and deferred sync startup, +adapter return and asynchronous settlement, cancellation, and release identity. + +A Collection subscription installs each logical subset owner before it calls +the source adapter. Reentrant release during `loadSubset` therefore retires the +logical owner at once, but physical release waits until the adapter returns and +proves that it established an acquisition. A synchronous `loadSubset` throw +rolls the tentative owner back without calling `unloadSubset`. Logical demand +retires even when `unloadSubset` fails. Each physical acquisition gets one release +attempt, marked before calling adapter or error-listener code. Reentrant and +repeated teardown cannot repeat it. Other acquisitions still receive cleanup, +and a cleanup failure cannot replace an earlier request failure. Core reports +the error but retains no retry debt: a broken adapter can leak external resources +if it throws before freeing them. Adapters must make their own cleanup reliable. +Replay replaces physical leases sequentially: detach and release the old lease, +then acquire a fresh one only if the logical demand and replay are still current. +A release failure fails that replay without starting a replacement. A load +throw leaves the logical demand detached; a later authoritative replay can +reacquire it. Neither path restores an already released lease. A sole adapter +resource may stop and restart in this gap; adapters must not tear down resources +held by another owner. The public replacement barrier remains closed throughout +the gap and through failed startup, so visible results do not flicker. Once a +new load returns successfully, its lease is active before status callbacks run. +Reentrant callbacks therefore see either detached demand, tentative startup, +or one active lease, not an old and new lease being transferred together. + +Request predicates describe acquisition, not row ownership. Releasing a demand +does not delete matching rows from either the public snapshot or an unfinished +replacement. The source controls retention through actual row writes; a +successful authoritative replacement reconciles the retained public snapshot. +This rule also applies when another demand overlaps the released predicate or +an independent source write happens to match it. Source deletions during replay +stay private until successful publication; failure preserves the last complete +snapshot. Query filters and routes, not request release, decide which retained +source rows belong in a query result. + +### Cleanup, restart, and detached waiters + +Restart is not allowed inside an active cleanup callback. `startSyncImmediate()` +throws `CollectionStateError` and `preload()` rejects with it before acquiring +new work. Nested cleanup does not open a new lifecycle turn. The Collection +holds this guard until sync, state, subscriptions, and indexes finish retiring; +it releases the guard even if teardown throws. Restart after cleanup completes, +including from its final `cleaned-up` status event, remains supported. This +avoids letting old teardown clear a replacement graph or its source ownership. + +Collection cleanup detaches surviving logical demand from the discarded sync +session. It aborts that session's physical work and rejects its replay barrier, +and rejects an unfinished initial preload with `AbortError`. Cleanup never +invokes first-ready callbacks; those callbacks belong to the discarded run. +Physical acquisitions belong to the sync session that created them; cleanup +retires them instead of sending an old release to a replacement adapter. +Unlike individual subset releases, a failed sync adapter cleanup callback +remains retryable only while that +retirement is current; it cannot replace a newer session's cleanup callback. +Demand requested while the Collection is cleaned up remains detached +rather than pretending that a physical acquisition succeeded. When the +Collection starts a new sync session, the subscription enters `loadingSubset` +before it queues reacquisition, then reacquires all detached demand through a +fresh private publication barrier. Settlements from the old session cannot +publish rows, report errors, or change readiness in the new session. + +This is the direct subscription's restart contract, not automatic recovery of +a dependent live query. Manually cleaning up a source puts its live queries in +a terminal error state. Restarting that source alone does not revive their +graphs or publish replacement results; callers must restart or recreate the +live query itself. This differs from a source truncate, which keeps the live +query active behind its replay publication barrier. + +An initial sync error also leaves newly requested demand detached, even when +the adapter has installed a loader. Same-session `markReady()` resumes that +demand; releasing it before recovery creates no physical acquisition or unload. +Queued reacquisition must not retry a failed attempt merely because both +loading and ready notifications scheduled it. + +Requests waiting for a loader report one pending promise synchronously through +`onLoadSubsetResult`, including requests made during initial error or after +cleanup. The callback is not delayed until acquisition: query callers capture +its result before the snapshot request returns. This promise waits for the +recovery's publication barrier, not just adapter return. Failure rejects it with +the replay error; release, external abort, unsubscribe, or another cleanup +rejects it with `AbortError`. Later transport settlement cannot change that +outcome. Cleanup may retain logical demand for the next session, but it does +not retain the old caller's unfinished wait. + +Eager collections have no subset reacquisition barrier. After cleanup, their +next public batch reconciles retained subscriber rows against the installed +state, including deletions for keys that do not return. An empty ready batch +also reconciles an empty replacement. On-demand sources cannot infer absence +from their partial installed state; their replay barrier owns replacement. + +### Source cancellation and applied settlement + +The initial-demand contract is: + +> Every active, satisfiable bucket must be served by a settled current demand +> request before initial preload completes. + +A request may remain in flight after some served buckets become inactive. +Those buckets no longer participate in readiness and cannot receive rows +through routes that no longer exist. Sharing source work never merges the route +rows themselves. + +The source contract stays abstract: a demand request eventually establishes +one coherent baseline and identifies when that baseline is complete. Each +request receives an `AbortSignal`. Cancellation is cooperative at this source +boundary. An obsolete request cannot satisfy current demand. Its settlement +may release a replay wait, but never substitutes for completion of the current +acquisition. A source that can cancel request-scoped work must honor the signal +before installing more rows. A source that cannot cancel an in-flight baseline +must settle that work; core keeps overlapping replay private until then. Core +cannot prevent an arbitrary adapter from writing after it ignores both parts +of that contract. Buffering, snapshot tokens, shape offsets, Collection +transactions, and local indexes are source-specific ways to satisfy it; they +are not materializer state. + +Every sync `commit()` returns an applied receipt: `true` when that +transaction's writes and events are already visible, or a promise when the +transaction is parked in the causal queue. The promise resolves only after the +writes and events become visible. It rejects with `AbortError` if request +cancellation or collection cleanup abandons the transaction first. An abort +after application has no effect. Application becomes irrevocable before change +events are emitted, so an abort raised by a publication observer is already +late. A successful `loadSubset` implementation must await or return every +receipt for the transactions that establish its result. A source must not add +priority merely to make a subset load settle. +Existing immediate bootstrap and persistence-hydration paths, plus truncate, +retain their queue-bypass contract; if one applies a parked subset transaction +as part of that prefix, the subset receipt settles only after the writes are +visible. Rejected acquisitions establish no result. Canceled or obsolete +acquisitions either stop before publishing more request-scoped rows or settle +behind the active replay barrier. + +### Ordered requests, continuation, and recovery + +Core constructs cursors only for one order column. A direct +`requestLimitedSnapshot()` call with a nonempty `minValues` must supply one +value and one order term; composite or partial-composite inputs throw before +local delivery or source acquisition. Multi-column queries remain supported +through the ordered loader's prefix-and-tie fallback. Its first-column equality +request closes a tie group; it is not a composite continuation cursor. + +Successful settlement proves only that the exact request finished and that its +writes were applied. It does not prove source exhaustion or broader coverage. +Ordered loading reaches a fixed point from public rows and exact request +identity; it must not invent source extent from a requested limit. A local row +seen before the first ordered source request proves neither a continuation +boundary nor a remote offset. This matters when a zero-sized window admits live +source changes before it opens: the first nonzero window must still request its +prefix from the start. Starting that request proves nothing until it succeeds; +if it rejects, an explicit retry must also start at offset zero without a +cursor. The same holds after any later ordered request fails or is canceled: +the adapter may already have written only part of its response, so those rows +cannot establish a continuation boundary. The next explicit retry starts from +the source as one authoritative filtered full-source request. Core cannot know +which rows a failed request wrote, and a successful limited request proves +neither how many authoritative rows it applied nor source exhaustion. Recovery +therefore does not infer a safe finite prefix from local row count or boundary +values. This rare error path trades bandwidth for a small, sound rule and keeps +the last settled public snapshot visible until recovery succeeds. It also lets +multi-column windows revalidate after a non-boundary row leaves. If the provider +predicate cannot express the local order relation, such as locale string order, +ordinary refinement likewise loads the full source instead of treating boundary +equality as an ordered continuation. An asynchronous failure of that +full-source acquisition does not start duplicate recovery work. It keeps the +logical demand so a later truncate replay can retry one authoritative +replacement, and clears the loader's completion marker so an explicit retry +of the window can issue the request again. That explicit retry retires and +releases the earlier failed acquisition before installing its replacement, so +a later truncate replays one logical demand rather than both attempts. A +successful authoritative replay clears its source-recovery gate, but it does +not clear an unrelated failed window operation. A later explicit window move +revalidates that physical window before publishing it. + +A finite page or tie-boundary request can remain in flight when full-source +repair starts. Its later success still settles its publication participant, +but cannot clear a recorded failure, change the repair's state, or start more +finite work. The full-source request owns that repair outcome. Explicit retry +releases a failed full-source acquisition once before replacing it. +If overlapping finite and full-source requests both fail, retry retires every +failed acquisition, even if one release throws. A successful full-source replay +repairs only that demand; obsolete failed finite demands still retire on retry. + +Successful authoritative full-source recovery also retires settled successful +page and tie demands. Later replay therefore reacquires the full source without +repeating those finite requests. Retirement waits for each original request to +settle and for any active replay to finish; it does not cancel unfinished work +merely because the full-source request finished first. Failed finite demands +still follow the explicit-retry rule above. Release callbacks retire ownership +before adapter code runs, and reentrant truncate or disposal stops the current +retirement pass. No copied rows or additional cursor history are retained. + +An ordered request cannot start another ordered request through its own +synchronous writes. If the adapter then throws, graph callbacks scheduled by +those writes still belong to the failed window operation and cannot retry it. +A public `setWindow()` call made from inside that synchronous operation throws +`SetWindowReentrancyError`; it must not claim that a nested window settled after +the loader suppressed its work. +The guard reads the loader's existing synchronous request state for initial +and later refinement requests, including requests after an asynchronous page. +It also rejects window changes during graph publication, before mutating top-K. +A synchronous result callback is provisional until the whole snapshot request +returns: a later local read or publication throw fails and retires that +acquisition instead of letting its queued success erase the failure. +A later explicit window operation has a new generation and may retry from the +safe source boundary. + +The ordered loader retains one settled loading boundary, independently of +live rows sent to D2. It derives invalidation from the existing contribution +rows rather than tracking a second largest-row cursor. New keys may reopen +refinement, while duplicate delivery and order-equal updates do not. After a +successful finite acquisition, it reads at +most the requested limit within that request's filtered, ordered range. That +range's last available row can advance the boundary; an unrelated live outlier +cannot advance it merely by entering D2. This relies on the adapter fulfilling +the exact ordered request, not just resolving after an arbitrary partial write. +An empty range does not invent a boundary or prove source exhaustion. + +For no-index and multi-column prefix loading, an unrelated new key does not +reacquire an already full window. An explicit window move, an underfilled +window, or a settled prefix smaller than a window widened during that request +still requires acquisition. A full local window alone does not prove that the +provider fulfilled a concurrent window change. + +A successful larger prefix retires settled smaller prefix acquisitions from +the same ordered source plan, after the replacement has applied. It does not +retire cursor suffixes, ties, unfinished work, or another subscription's leases. +Adapter eviction must still preserve rows owned by the replacement or peers. +If an older prefix's release throws, the successful replacement still finishes +its boundary and continuation bookkeeping before surfacing the cleanup error. +Cleanup failure does not turn the successful acquisition into a failed load. + +Automatic full-source repair after an established window fails can retry twice, +after 250 ms and 500 ms. Every retry releases failed acquisitions before starting +the replacement. It uses the same publication barrier; stale rows stay public +and the last error stays observable if the budget is exhausted. Initial loads +and explicit window failures do not auto-retry. Cleanup, truncate, and explicit +retry supersede queued repair work. A successful repair resets the budget. + +An explicit window move counts current rows at or before that boundary in the +requested prefix. It acquires only the missing portion, with both cursor and +offset derived from that confirmed range, not from all observed rows. These +reads reuse the Collection's indexed snapshot code; they retain no page list +or second row index. Transfer checks and local-read work are separate costs: +counting a long prefix can still revisit its rows. Boundary-read failures use +the same authoritative recovery path as failed acquisitions. Deletes and +source-order changes invalidate finite coverage as described below. Cleanup +and truncate discard the boundary; replay establishes an authoritative source +replacement instead of reviving a stale cursor. + +### Atomic window publication + +An initial ordered load or imperative window move includes every page, +tie-boundary request, and forward refill needed to reach its fixed point. Its +preload or window promise cannot settle before that chain, and a failure in any +required step belongs to the same operation. Rows may enter the private D2 +result while the chain runs, but the public Collection publishes the completed +window once. If refinement fails, the operation rejects and leaves the last +settled public snapshot visible. The private source and D2 state may already +have advanced, so core does not try to reconstruct the old window over that +new state. A later successful retry publishes the coherent replacement. A +superseding window also waits for older source work that still gates +publication; it does not report success until its own chosen window is visible. +Window controllers treat `getWindow()` as settled state, not the current lease +request. An overlapping preload joins its lease's pending window promise rather +than replacing it with the smaller committed page count. Lease release may also +settle asynchronously; completion, not the release call, establishes its window. +Partial window options inherit omitted fields from the active requested window, +or from the last settled window when no move is active. Collection cleanup +rejects a pending window operation with `AbortError`; it cannot report success +after discarding the graph and requested window. +That error belongs to the operation even if cleanup precedes registration of +its waiter. Cleanup does not retroactively cancel an already completed operation. +Window-operation generations stay monotonic across cleanup and restart, so a +late rejection from an abandoned session cannot reset the replacement +session's requested window. +A window move started during an active source replay waits for that replay and +applies only after its replacement is complete. A failed replay rejects the +move without advancing the reported window. Replay completion callbacks carry +their sync-session identity and become no-ops after cleanup or restart. +Cleanup rejects the replay barrier, and therefore every window move waiting on +it, with `AbortError`; no waiter may outlive the discarded subscription. +Subscription-owned Promise observers carry the Collection's load-session +generation. Cleanup invalidates that generation before adapter teardown, so an +obsolete replay cannot publish its private rows, report a late error, or emit a +late `ready` transition even when the transport ignores cancellation. +Ordinary source mutations stay synchronous except while an initial ordered +load, imperative window move, or asynchronous repair of invalid finite source +coverage owns this publication barrier. A visible delete or a change to a +visible row's source-order value can invalidate a provider prefix because a +hidden row may now belong in the window. That repair loads the authoritative +source and keeps the last complete public snapshot until it settles; an update +that compares equal under the source order does not broaden demand. Mutations +that arrive during a barrier join the private state and publish with the +completed replacement; a failed operation keeps them private until retry or +restart. Queued ordered-repair startup joins this barrier before invoking the +adapter, so a synchronous throw cannot publish a partial replacement merely +because it returned no acquisition promise. The queued task belongs to the +loader that scheduled it, not a replacement created after cleanup. +The loader tracks each sequential request as a bounded participant, +not every recursive suffix of a long refinement chain. + +### Replay participants and failure + +A truncate replay is one publication barrier. Every acquisition started while +that replay is active, including ordered full-source recovery, belongs to the +barrier. Success publishes only after all current acquisitions settle. A +released demand stops participating even if its canceled transport promise +never settles. A newer truncate aborts prior acquisitions, but publication +still waits for overlapping work that had already started because some sources +cannot cancel an in-flight snapshot. Such work must settle and must not install +rows after observing cancellation. Settled historical attempts are discarded. +Replacing an acquisition does not release its logical owner. A delayed +cancellation therefore remains pending; prompt cancellation settles that wait. +Releasing the owner removes both its current and older work from readiness. +An ordinary acquisition started before replay may still hold subscription +readiness after its replacement publishes. It is not a replay publication +participant: its canceled writes must stop at the source boundary. Work started +inside replay, including an older overlapping replay, does hold publication. +Core installs each tentative acquisition and binds it to the current replay +attempt before calling adapter code. A reentrant release or newer truncate can +therefore see and retire the exact work it supersedes; work returned after that +reentrancy cannot attach itself to a newer attempt. Once reentrancy supersedes +an attempt, core starts none of that attempt's remaining demands. A demand that +releases itself during adapter or status callbacks cannot join readiness or +poison the replay with a later synchronous failure. Successful replacement +publication happens before the subscription emits `ready`. Cleanup runs every +ownership step even when replacement publication throws. Subscriber errors +raised by an asynchronous replacement do not turn source success into replay +failure: core finishes its internal state and surfaces the exact callback error +in a host microtask. Status callbacks may synchronously change demand. Generic +and specific status delivery capture the transition revision and stop before a +later listener when reentry supersedes it, including an ABA transition back to +the same status label. Subscription teardown is a one-shot logical transition: +it stops the listener set already being walked, emits no later status, and +removes subscriber ownership once. A later `unsubscribe()` is a no-op, including +after a physical subset release failed. +Failure keeps the last complete result visible and partly replayed source state +private for both direct subscribers and query graphs. Ordinary source deltas or +snapshot requests do not reopen that gate because they cannot prove the source +complete; only a later successful truncate replay provides the authoritative +replacement. Replay failure is scoped to the logical demand that failed. If +that demand retires, its failure cannot poison a successful replacement for the +remaining demand. If the last logical demand retires, the now-unreachable source +replay rejects its completion with `AbortError` and stops gating the shared +graph; unrelated parent or sibling changes may then publish. If release +publication or adapter unload synchronously acquires new demand, core checks +completion after that callback: the new demand joins the private replacement, +while the retired transport can no longer gate it. A genuine replay failure is +normalized once by the subscription. +The `loadSubset:error` event, `lastSubsetError`, and any window move waiting on +that replay expose the same `Error` object. + +### Mutation boundaries and initial readiness + +A transaction `mutationFn` must not start or await collection or live-query +preloads. User persistence owns the causal queue while that function runs, so a +preload that waits for a queued sync commit can wait on the mutation that is +waiting on the preload. Use an adapter's documented mutation acknowledgement +helper instead; it must confirm the optimistic write without starting new +collection demand. + +This project uses a single graph-run order rather than multi-dimensional +timely-dataflow frontiers. Do not introduce a general timestamp or frontier +framework unless a source contract proves that the generation and up-to-date +protocol cannot express its ordering. + +**Initial readiness:** preload is complete when every demand currently +reachable from the initial query graph is covered by a settled request. Demand +that is no longer reachable does not block completion. An empty outer relation +has no child demand, but its root demand must still settle. Later readiness +transitions follow the existing Collection contract until an executable test +defines another public behavior. + +Pending demand does not hide the parent row. An active empty bucket gives it +the current canonical bucket value, and available partial source rows produce +the current partial materialization when the source supports progressive +delivery. Later source rows enter D2 as ordinary deltas and recompute the +parent. “Fully composed” means that every include field has its canonical value +for the graph's current input state; it does not mean that asynchronous demand +has settled. + +## Coherent publication + +D2 runs until the whole materialization graph has no pending synchronous work +for its currently available inputs. Only fully materialized canonical root +deltas cross into the public Collection. + +For each scheduled graph turn: + +1. enqueue all currently committed input deltas into their D2 inputs; +2. run D2 until it has no pending synchronous work; +3. consolidate the already canonical final-output deltas; +4. install child-facade state through normal Collection transactions while + deferring their subscriber delivery; +5. apply direct root insert, update, and delete writes through one normal + Collection transaction; +6. release the deferred child-facade events after every synchronous read can + see the complete root and facade state; +7. allow dependent live-query graphs to run through the existing + transaction-scoped scheduler. + +The Collection boundary performs no identity reconciliation, routing, +materialization, or multiplicity interpretation. The canonical root relation +has already done that work. + +The public Collection is an output, never scratch state. Placeholder rows, +in-place include repair, and forced secondary events are forbidden. + +Classify root deltas against authoritative membership, including earlier queued +sync writes, not the optimistic public view. An optimistic delete must not turn +a balanced graph update into an authoritative delete. This does not bypass the +normal sync queue or publish part of a graph-output transaction early. +Build queued membership lazily on the first balanced delta in an output flush, +preserving committed last-write and truncate semantics. Insert-only flushes do +not scan the queue, and balanced rows share that flush's lookup. + +At the Collection boundary, optimistic mutations own whole validated row +snapshots, including fields they did not change and insert schema defaults. +Do not merge newer synced fields into those snapshots: that could publish a +combination neither the mutation nor the server created. This applies to both +ordinary sync and truncate. The mutation payload stays unchanged as well. +Active snapshots are selected in transaction order. Completed snapshots remain +beneath active transactions under the existing retention policy until sync +retires them. A later snapshot may contain values seen from an earlier sibling; +rolling back that sibling does not rewrite the later snapshot. Sync publication +compares actual previous and next visible rows, not just mutation identities. +An update made over an unconfirmed insert retains that exact insert dependency, +not just its key. Insert success preserves the later completed snapshot; insert +failure removes the already-retained dependent row. An independently submitted +update accepted after that failure still retains its own snapshot. An +acknowledged insert or a later same-key +insertion is not the failed insertion. Truncate replay derives events and reads +from the same snapshot overlay, without merging in its new authoritative fields. + +Installed state, synchronous reads, change-event payloads, and downstream +queries must all observe the same fully materialized commit. The facade adapter +may defer event delivery across its Collection transactions, but it must not +defer state or index installation. Routing and identity remain inside D2. + +## External boundaries + +### Query-db ownership + +Row ownership in `@tanstack/query-db-collection` is separate. Eager retention, +active query acquisition, and persisted retention are distinct owner tokens. +The live-query graph publishes coherent rows but does not own query-db cache or +listener lifetime. + +### Physical planning and work + +Correct relation state does not prove efficient work. When an applicable index +exists, irrelevant correlated rows must not cause scans of unrelated rows or +activate unrelated downstream routes. Relation rows, indexed keys, active +demands, materialization cells, and public facades are the relevant space +units. Queries without includes retain their original pipeline unless a joined +custom-key query needs contributor reduction. Inline materialization must not +create recursive Collection machinery. + +## Normative laws + +1. **Alpha-renaming:** changing any accepted alias to another unused name cannot + change an explicitly projected result. An implicit namespaced result keeps + its aliases as public field names. Aliases must be unique within one lexical + scope and cannot shadow an ancestor alias. Sibling scopes may reuse aliases. +2. **Contribution conservation:** a public row exists exactly when its reduced + supporting weight and collision policy produce one. +3. **Batch partition:** equivalent valid split and atomic deliveries converge. +4. **Route relation:** current route rows joined with current bucket values + equal current materialization-cell values. +5. **Total materialization:** every active inline cell has exactly one value, + including its mode's empty value when its bucket has no rows. +6. **Stale demand:** an obsolete graph cannot settle current readiness, and an + obsolete acquisition cannot satisfy current demand. A conforming source + cannot publish its request-scoped rows after cancellation. +7. **Applied settlement:** a successful subset load settles only after its + establishing sync transactions are visible; a source must not add queue + priority merely to force the load to settle. Settlement proves no broader + source extent than the exact request. +8. **Nested propagation:** every materialized relation consumes the fully + materialized output relation of its children. +9. **Publication:** reads, events, and downstream queries observe the same + complete graph result. A truncate replacement stays private until all work + started by its active replay demands settles; failure keeps the prior public + result and later partial source changes private until an authoritative replay + succeeds. A failed replay with no remaining logical demand cannot gate other + graph work. +10. **Initial demand:** preload completes when every initially reachable demand + is covered; obsolete demand does not block it. +11. **Ownership:** a query-db row exists exactly while an explicit owner + remains. +12. **Work:** irrelevant rows do not cause unrelated scans or activate unrelated + routes when an applicable index exists. +13. **Space:** state scales with retained D2 relation/index rows, active demands, + materialization cells, visible rows, the current private replay state, and + required Collection facades—not with settled historical replay attempts or + raw delta history. + +## Glossary + +- **Relation:** an internal weighted multiset maintained by D2, not a public + TanStack Collection. +- **Weighted delta:** a positive or negative change to a relation row. +- **Data plane:** the D2 graph that joins, reduces, orders, and materializes + relations. +- **Demand plane:** the async adapter that starts and releases source loads. +- **Bucket key:** the canonical identity of one correlated child partition. +- **Bucket relation:** child rows partitioned by bucket key. +- **Active bucket:** a bucket referenced by at least one current route; it seeds + empty materialization values and contributes to source demand. +- **Bucket value:** the one inline value reduced from an active bucket's rows. +- **Route relation:** weighted links from bucket keys to materialization cells. +- **Materialization cell:** one include field on one parent-row occurrence. +- **Arrangement:** retained relation state indexed for efficient keyed access + and reuse. +- **Reduction:** deriving one visible value from weighted rows sharing a key. +- **Hydration:** establishing an initial snapshot before forwarding later + changes. +- **Generation:** a token that rejects obsolete asynchronous work. +- **Collection facade:** a stable public Collection view shared by the parents + routed to one active bucket. +- **Coherent commit:** one publication in which state, events, and consumers see + the same fully materialized result. + +## Executable contracts + +| Contract | Test suite | +| ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | +| State equivalence, route lifecycle, transition history, and batch partition | `packages/db/tests/query/includes-oracle.property.test.ts` | +| Joined multiplicity, alias identity, and null-key normalization | `packages/db/tests/query/includes-query-shape-oracle.test.ts` | +| Demand, cancellation, and progressive timing | `packages/db/tests/query/includes-temporal-oracle.test.ts` | +| Optimistic confirmation, rollback, and later reactivity | `packages/db/tests/query/includes-optimistic-oracle.property.test.ts` | +| Coherent layered publication | `packages/db/tests/query/includes-publication-oracle.test.ts` | +| Collection facades, event coherence, and route activation | `packages/db/tests/query/includes-collection-oracle.property.test.ts` | +| Correlated physical work | `packages/db/tests/query/includes-work-counter-oracle.test.ts` | +| Constructed and retained facades in a nested Collection tree | `packages/db/tests/query/includes-space-oracle.test.ts` | +| Route-context discovery and transport across recursive and join boundaries | `packages/db/tests/query/includes-context-transport-oracle.test.ts` | +| Functional projection input boundaries, timing, and output preservation | `packages/db/tests/query/includes-functional-projection-oracle.test.ts` | +| Functional input rejection and inline alternatives | `packages/db/tests/query/includes-functional-input-boundary.test.ts` | +| Public-container descriptors and reference-key matches across internal query stages | `packages/db/tests/query/public-container-copy.test.ts` | +| Cross-formulation equivalence and reference-sensitive route identity | `packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts` | +| Query-db ownership | `packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts` | +| Failed replay retention, peer isolation, and explicit consumer-only recovery | `packages/db/tests/query/replay-failure-boundary.test.ts` | +| Replay lease balance, reference-counted peers, and failed-start recovery | `packages/db/tests/replay-adapter-ownership.test.ts` | +| Reachable nested shape | `packages/query-db-collection/tests/includes-work-counter-oracle.test.ts` | + +Each oracle identifies the first divergent checkpoint and compares either the +whole result or one exact structural difference. Correlated-materialization +scenarios use direct assertions. A boundary suite may retain an exact +expected-failure guard for a planner or ownership defect that this graph does +not own. + +Run the DB oracle set with `pnpm test:oracles` from `packages/db`. Broad +properties use FastCheck's random seed, while structural matrices keep fixed +seeds so each run covers the same named cells. Increase both corpora with +`TANSTACK_DB_ORACLE_RUNS_MULTIPLIER=10 pnpm test:oracles`. Preserve FastCheck's +reported seed and shrink path while reducing a failure. Replay a broad +campaign with `TANSTACK_DB_ORACLE_SEED= pnpm test:oracles`, then add the +smallest case as a deterministic regression trace. + +The nested-space test inspects facade construction and retained entries through +test-only instrumentation. It adds no production metrics API. Run the related +diagnostic workload with `pnpm bench:nested-includes` from `packages/db`; +wall-clock timings are not a CI threshold. + +The broad relationship history changes correlation keys rather than freezing +them. Set `TANSTACK_DB_ORACLE_STATISTICS=1` to print its generated depth, +relationship-change, optimistic, and delete distribution. Collection-valued, +array, and materialized includes are checked together for every Collection +scenario instead of relying on a random mode sample. A separate metamorphic +oracle compares nested includes with a flat join, fresh per-parent queries, and +three-valued predicate partitioning. + +## Implementation discipline + +- Express relation state with existing D2 inputs, joins, reductions, grouping, + ordering, and consolidation before adding custom state. +- Keep route and bucket rows in the same graph as parent and child query rows. +- Add a reusable indexed D2 primitive only when existing operators cannot share + or expose required retained state. +- Keep asynchronous demand state outside D2 and make its generation boundary + explicit. +- Never use a public Collection, emitted event, or materialized row as internal + routing or contribution state. +- Add a reduced oracle trace before adding any special lifecycle branch. +- Measure retained relation rows, active demands, and public facades. Preserve + the no-includes fast path and verify any claimed space improvement with those + counters. diff --git a/packages/db/src/query/live/bucket-facade-adapter.ts b/packages/db/src/query/live/bucket-facade-adapter.ts new file mode 100644 index 0000000000..2648416809 --- /dev/null +++ b/packages/db/src/query/live/bucket-facade-adapter.ts @@ -0,0 +1,485 @@ +import { output, serializeValue } from '@tanstack/db-ivm' +import { isPlainObject } from '../../utils/type-guards.js' +import { getOrCreate } from '../../utils/get-or-create.js' +import { createCollection } from '../../collection/index.js' +import { + INCLUDES_ROUTING, + transformPublicContainers, +} from '../compiler/route-metadata.js' +import { BUCKET_FACADE_REF } from './materialized-pipeline.js' +import type { Collection } from '../../collection/index.js' +import type { SyncConfig } from '../../types.js' +import type { PublicationDeferral } from '../../collection/changes.js' +import type { + BucketFacadeCompilation, + BucketFacadeRef, + BucketRow, +} from './materialized-pipeline.js' + +const PRIVATE_RESULT_KEYS = new Set([INCLUDES_ROUTING]) + +type FacadeSync = Parameters[`sync`]>[0] + +type PendingRow = { + deletes: number + inserts: number + value: BucketRow +} + +type FacadeEntry = { + collection: Collection + sync: FacadeSync | undefined + keys: WeakMap + order: WeakMap + currentOrder: Map +} + +type FacadeSnapshot = { + activeBuckets: Map> + entries: Map> + rows: Map< + FacadeEntry, + Array<{ + key: string | number + value: object + order: string | undefined + }> + > +} + +export type FacadePublication = { + prepare: () => void + publish: () => void + rollback: () => void +} + +/** + * The only stateful boundary outside the materialization graph. It turns inert + * bucket references into stable public Collection facades and applies the + * graph's canonical bucket-row deltas to those facades. + */ +export class BucketFacadeAdapter { + private readonly pending = new Map< + string, + Map> + >() + private readonly pendingActivity = new Map>() + private readonly activeBuckets = new Map>() + private readonly entries = new Map>() + private readonly retiredEntries = new Map>() + private resolvedValues = new WeakMap() + + constructor( + private readonly parentId: string, + private readonly compilations: Array, + onMessages: (count: number) => void, + ) { + for (const compilation of compilations) { + compilation.rows.pipe( + output((data) => { + const messages = data.getInner() + onMessages(messages.length) + for (const [[bucketKey, row], multiplicity] of messages) { + this.accumulate(compilation.edgeId, bucketKey, row, multiplicity) + } + }), + ) + compilation.activeBuckets.pipe( + output((data) => { + const messages = data.getInner() + onMessages(messages.length) + for (const [[bucketKey], multiplicity] of messages) { + this.accumulateActivity(compilation.edgeId, bucketKey, multiplicity) + } + }), + ) + } + } + + hasPendingChanges(): boolean { + return this.pending.size > 0 || this.pendingActivity.size > 0 + } + + flush(): FacadePublication { + const snapshot = this.snapshot() + const deferredEntries = new Set() + const publications: Array = [] + const deferPublication = (entry: FacadeEntry) => { + if (deferredEntries.has(entry)) return + deferredEntries.add(entry) + publications.push(entry.collection._deferPublication()) + } + const newBaselines: Array = [] + + // Compilations are child-first, so nested facade references resolve before + // their containing rows are written to the next facade. + try { + for (const compilation of this.compilations) { + const activity = this.pendingActivity.get(compilation.edgeId) + const active = this.getActiveBuckets(compilation.edgeId) + for (const [bucketKey, multiplicity] of activity ?? []) { + if (multiplicity > 0 && !active.has(bucketKey)) { + active.add(bucketKey) + newBaselines.push(this.getEntry(compilation.edgeId, bucketKey)) + } + } + + const buckets = this.pending.get(compilation.edgeId) + for (const [bucketKey, changes] of buckets ?? []) { + const existing = this.entries.get(compilation.edgeId)?.get(bucketKey) + if (!active.has(bucketKey) && !existing) continue + const entry = this.getEntry(compilation.edgeId, bucketKey) + const sync = entry.sync + if (!sync || changes.size === 0) continue + + for (const change of changes.values()) { + this.prepareChange(entry, change) + } + deferPublication(entry) + sync.begin() + for (const change of changes.values()) { + this.applyChange(entry, sync, change, compilation.hasOrderBy) + } + sync.commit() + } + for (const [bucketKey, multiplicity] of activity ?? []) { + if (multiplicity >= 0) continue + active.delete(bucketKey) + this.retireEntry(compilation.edgeId, bucketKey, deferPublication) + } + } + } catch (error) { + this.restore(snapshot, deferredEntries) + this.retiredEntries.clear() + for (const publication of publications) publication.discard() + throw error + } + this.pending.clear() + this.pendingActivity.clear() + + let closed = false + let prepared = false + const prepare = () => { + if (closed || prepared) return + prepared = true + for (const entry of newBaselines) entry.sync?.markReady() + } + return { + prepare, + publish: () => { + if (closed) return + prepare() + closed = true + for (const publication of publications) publication.publish() + // Drop only the adapter's strong reference. External holders keep an + // empty, ready facade; a later active interval receives a new one. + this.retiredEntries.clear() + }, + rollback: () => { + if (closed) return + closed = true + this.restore(snapshot, deferredEntries) + this.retiredEntries.clear() + for (const publication of publications) publication.discard() + }, + } + } + + resolve(value: T): T { + return this.resolveValue(value) as T + } + + cleanup(): void { + for (const byBucket of this.entries.values()) { + for (const entry of byBucket.values()) { + void entry.collection.cleanup() + } + } + this.entries.clear() + this.cleanupRetiredEntries() + this.pending.clear() + this.pendingActivity.clear() + this.activeBuckets.clear() + } + + private accumulate( + edgeId: string, + bucketKey: string, + row: BucketRow, + multiplicity: number, + ): void { + const buckets = getOrCreate(this.pending, edgeId, () => new Map()) + const rows = getOrCreate(buckets, bucketKey, () => new Map()) + + const key = serializeValue(row.publicKey) + const change = rows.get(key) ?? { + deletes: 0, + inserts: 0, + value: row, + } + if (multiplicity < 0) { + change.deletes += -multiplicity + } else if (multiplicity > 0) { + change.inserts += multiplicity + change.value = row + } + rows.set(key, change) + } + + private snapshot(): FacadeSnapshot { + const rows = new Map< + FacadeEntry, + Array<{ + key: string | number + value: object + order: string | undefined + }> + >() + for (const byBucket of this.entries.values()) { + for (const entry of byBucket.values()) { + rows.set( + entry, + [...entry.collection._state.syncedData].map(([key, value]) => ({ + key, + value, + order: entry.currentOrder.get(key), + })), + ) + } + } + return { + activeBuckets: new Map( + [...this.activeBuckets].map(([edgeId, buckets]) => [ + edgeId, + new Set(buckets), + ]), + ), + entries: new Map( + [...this.entries].map(([edgeId, byBucket]) => [ + edgeId, + new Map(byBucket), + ]), + ), + rows, + } + } + + private restore( + snapshot: FacadeSnapshot, + changedEntries: Set, + ): void { + const previousEntries = new Set( + [...snapshot.entries.values()].flatMap((byBucket) => [ + ...byBucket.values(), + ]), + ) + const currentEntries = new Set( + [...this.entries.values()].flatMap((byBucket) => [...byBucket.values()]), + ) + + for (const entry of changedEntries) { + if (!previousEntries.has(entry)) continue + const sync = entry.sync + if (!sync) continue + const rows = snapshot.rows.get(entry) ?? [] + const restoredKeys = new Set(rows.map((row) => row.key)) + sync.begin() + for (const key of entry.collection.keys()) { + if (!restoredKeys.has(key)) sync.write({ type: `delete`, key }) + } + entry.currentOrder.clear() + for (const row of rows) { + entry.keys.set(row.value, row.key) + if (row.order !== undefined) entry.order.set(row.value, row.order) + entry.currentOrder.set(row.key, row.order) + sync.write({ + type: entry.collection.has(row.key) ? `update` : `insert`, + value: row.value, + }) + } + sync.commit() + } + + this.entries.clear() + for (const [edgeId, byBucket] of snapshot.entries) { + this.entries.set(edgeId, new Map(byBucket)) + } + this.activeBuckets.clear() + for (const [edgeId, buckets] of snapshot.activeBuckets) { + this.activeBuckets.set(edgeId, new Set(buckets)) + } + this.resolvedValues = new WeakMap() + + for (const entry of currentEntries) { + if (!previousEntries.has(entry)) void entry.collection.cleanup() + } + } + + private accumulateActivity( + edgeId: string, + bucketKey: string, + multiplicity: number, + ): void { + const activity = getOrCreate(this.pendingActivity, edgeId, () => new Map()) + activity.set(bucketKey, (activity.get(bucketKey) ?? 0) + multiplicity) + } + + private getActiveBuckets(edgeId: string): Set { + return getOrCreate(this.activeBuckets, edgeId, () => new Set()) + } + + private retireEntry( + edgeId: string, + bucketKey: string, + deferPublication: (entry: FacadeEntry) => void, + ): void { + const byBucket = this.entries.get(edgeId) + const entry = byBucket?.get(bucketKey) + if (!entry) return + + const sync = entry.sync + const keys = [...entry.collection.keys()] + if (sync && keys.length > 0) { + deferPublication(entry) + sync.begin() + for (const key of keys) sync.write({ type: `delete`, key }) + sync.commit() + } + byBucket!.delete(bucketKey) + if (byBucket!.size === 0) this.entries.delete(edgeId) + const retired = getOrCreate(this.retiredEntries, edgeId, () => new Map()) + retired.set(bucketKey, entry) + } + + private getEntry(edgeId: string, bucketKey: string): FacadeEntry { + const byBucket = getOrCreate(this.entries, edgeId, () => new Map()) + const existing = byBucket.get(bucketKey) + if (existing) return existing + + const keys = new WeakMap() + const order = new WeakMap() + let sync: FacadeSync | undefined + const collection = createCollection({ + id: `__bucket-facade:${this.parentId}:${edgeId}:${bucketKey}`, + getKey: (row) => { + const key = keys.get(row) ?? row?.$key + if (typeof key !== `string` && typeof key !== `number`) { + throw new Error(`Bucket facade row has no public key`) + } + return key + }, + compare: (left, right) => { + const leftOrder = order.get(left) + const rightOrder = order.get(right) + if (leftOrder === rightOrder) return 0 + if (leftOrder === undefined) return 1 + if (rightOrder === undefined) return -1 + return leftOrder < rightOrder ? -1 : 1 + }, + sync: { + rowUpdateMode: `full`, + sync: (methods) => { + sync = methods + return () => { + sync = undefined + } + }, + }, + startSync: true, + gcTime: 0, + }) + const entry: FacadeEntry = { + collection, + get sync() { + return sync + }, + keys, + order, + currentOrder: new Map(), + } + byBucket.set(bucketKey, entry) + return entry + } + + private applyChange( + entry: FacadeEntry, + sync: FacadeSync, + change: PendingRow, + hasOrderBy: boolean, + ): void { + const key = change.value.publicKey as string | number + const previousOrder = entry.currentOrder.get(key) + const nextOrder = change.value.order + const orderChanged = sync.collection.has(key) && previousOrder !== nextOrder + const resolvedRow = this.resolve(change.value.value) + const row = orderChanged ? { ...resolvedRow } : resolvedRow + entry.keys.set(row, key) + if (nextOrder !== undefined) { + entry.order.set(row, nextOrder) + } + + if (change.inserts > change.deletes) { + sync.write({ + type: sync.collection.has(key) ? `update` : `insert`, + value: row, + }) + } else if (change.inserts === change.deletes && sync.collection.has(key)) { + sync.write({ type: `update`, value: row }) + } else if (change.deletes > 0) { + sync.write({ type: `delete`, key }) + entry.currentOrder.delete(key) + return + } + + entry.currentOrder.set(key, nextOrder) + if (hasOrderBy && orderChanged) sync.collection._markLayoutChange() + } + + /** Resolve and validate every public key before opening a sync transaction. */ + private prepareChange(entry: FacadeEntry, change: PendingRow): void { + const key = change.value.publicKey as string | number + const row = this.resolve(change.value.value) + entry.keys.set(row, key) + entry.collection.getKeyFromItem(row) + } + + private resolveValue(value: unknown): unknown { + if (value === null || typeof value !== `object`) return value + const cached = this.resolvedValues.get(value) + if (cached !== undefined) return cached + if (isBucketFacadeRef(value)) { + const { edgeId, bucketKey } = value[BUCKET_FACADE_REF] + const facade = + this.entries.get(edgeId)?.get(bucketKey)?.collection ?? + this.retiredEntries.get(edgeId)?.get(bucketKey)?.collection ?? + this.getEntry(edgeId, bucketKey).collection + this.resolvedValues.set(value, facade) + return facade + } + if (Array.isArray(value) || isPlainObject(value)) { + const result = transformPublicContainers( + value, + (leaf) => (isBucketFacadeRef(leaf) ? this.resolveValue(leaf) : leaf), + PRIVATE_RESULT_KEYS, + ) + this.resolvedValues.set(value, result) + return result + } + return value + } + + private cleanupRetiredEntries(): void { + for (const byBucket of this.retiredEntries.values()) { + for (const entry of byBucket.values()) { + void entry.collection.cleanup() + } + } + this.retiredEntries.clear() + } +} + +function isBucketFacadeRef(value: unknown): value is BucketFacadeRef { + return ( + value !== null && typeof value === `object` && BUCKET_FACADE_REF in value + ) +} diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index b40e6e431a..dd29f95636 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -1,37 +1,38 @@ -import { D2, output, serializeValue } from '@tanstack/db-ivm' -import { - FN_SELECT_STATE, - INCLUDES_ROUTING, - compileQuery, -} from '../compiler/index.js' -import { createCollection } from '../../collection/index.js' +import { D2, output } from '@tanstack/db-ivm' +import { compileQuery } from '../compiler/index.js' import { MissingAliasInputsError, + SetWindowReentrancyError, SetWindowRequiresOrderByError, } from '../../errors.js' -import { transactionScopedScheduler } from '../../scheduler.js' +import { + getActivePublicationContext, + transactionScopedScheduler, + withPublicationContext, +} from '../../scheduler.js' import { getActiveTransaction } from '../../transactions.js' +import { deepEquals } from '../../utils.js' +import { runAllCallbacks } from '../../utils/callbacks.js' +import { normalizeError } from '../../utils/error.js' import { CollectionSubscriber } from './collection-subscriber.js' import { getCollectionBuilder } from './collection-registry.js' import { LIVE_QUERY_INTERNAL } from './internal.js' +import { materializeCompilation } from './materialized-pipeline.js' +import { BucketFacadeAdapter } from './bucket-facade-adapter.js' import { buildQueryFromConfig, - extractCollectionAliases, extractCollectionFromSource, + extractCollectionSources, extractCollectionsFromQuery, } from './utils.js' import type { LiveQueryInternalUtils } from './internal.js' -import type { - IncludesCompilationResult, - WindowOptions, -} from '../compiler/index.js' +import type { WindowOptions } from '../compiler/index.js' import type { SchedulerContextId } from '../../scheduler.js' import type { CollectionSubscription } from '../../collection/subscription.js' import type { RootStreamBuilder } from '@tanstack/db-ivm' import type { OrderByOptimizationInfo } from '../compiler/order-by.js' import type { Collection } from '../../collection/index.js' import type { - ChangeMessage, CollectionConfigSingleRowOption, KeyedStream, ResultStream, @@ -40,12 +41,7 @@ import type { UtilsRecord, } from '../../types.js' import type { Context, GetResult } from '../builder/types.js' -import type { - BasicExpression, - IncludesMaterialization, - PropRef, - QueryIR, -} from '../ir.js' +import type { BasicExpression, QueryIR } from '../ir.js' import type { LazyCollectionCallbacks } from '../compiler/joins.js' import type { Changes, @@ -56,7 +52,8 @@ import type { import type { AllCollectionEvents } from '../../collection/events.js' export type LiveQueryCollectionUtils = UtilsRecord & { - getRunCount: () => number + /** Most recent subset-load failure observed by this live query. */ + readonly lastSubsetError: unknown | undefined /** * Sets the offset and limit of an ordered query. * Is a no-op if the query is not ordered. @@ -74,7 +71,8 @@ export type LiveQueryCollectionUtils = UtilsRecord & { } type PendingGraphRun = { - loadCallbacks: Set<() => boolean> + syncSession: number + loadCallbacks: Set<() => void> } // Global counter for auto-generated collection IDs @@ -91,9 +89,9 @@ export class CollectionConfigBuilder< private readonly id: string readonly query: QueryIR private readonly collections: Record> - private readonly collectionByAlias: Record> - // Populated during compilation with all aliases (including subquery inner aliases) - private compiledAliasToCollectionId: Record = {} + private readonly collectionSources: ReturnType< + typeof extractCollectionSources + > // WeakMap to store the keys of the results // so that we can retrieve them in the getKey function @@ -106,7 +104,6 @@ export class CollectionConfigBuilder< private readonly compareOptions?: StringCollationConfig private isGraphRunning = false - private runCount = 0 // Current sync session state (set when sync starts, cleared when it stops) // Public for testing purposes (CollectionConfigBuilder is internal, not public API) @@ -117,20 +114,22 @@ export class CollectionConfigBuilder< // Error state tracking private isInErrorState = false + private fatalQueryError = false + private readonly erroredSourceIds = new Set() + private lastSubsetError: unknown | undefined // Reference to the live query collection for error state transitions public liveQueryCollection?: Collection private windowFn: ((options: WindowOptions) => void) | undefined + private readonly initialWindow: WindowOptions | undefined private currentWindow: WindowOptions | undefined + private settledWindow: WindowOptions | undefined + private activeWindowOperation: + | { generation: number; failed: boolean; error?: unknown } + | undefined private maybeRunGraphFn: (() => void) | undefined - - private readonly aliasDependencies: Record< - string, - Array> - > = {} - private readonly builderDependencies = new Set< CollectionConfigBuilder >() @@ -154,15 +153,31 @@ export class CollectionConfigBuilder< public sourceWhereClausesCache: | Map> | undefined - private includesCache: Array | undefined + private bucketFacadesCache: + | ReturnType[`facades`] + | undefined - // Map of source alias to subscription + // Map of opaque source ID to subscription readonly subscriptions: Record = {} - // Map of source aliases to functions that load keys for that lazy source + // Map of opaque source ID to demand callbacks for that lazy source lazySourcesCallbacks: Record = {} - // Set of source aliases that are lazy (don't load initial state) + // Set of opaque source IDs that are lazy (don't load initial state) readonly lazySources = new Set() - // Set of collection IDs that include an optimizable ORDER BY clause + private readonly activeDemands = new Map< + string, + { + generation: number + settled: boolean + } + >() + private readonly demandGenerations = new Map() + private readonly pendingOrderedLoads = new Set>() + private orderedLoadFailed = false + // Source replay cannot settle a failed imperative window operation. + private windowFailed = false + private syncSession = 0 + private windowOperationGeneration = 0 + // Map of lexical source IDs to optimizable ORDER BY state optimizableOrderByCollections: Record = {} constructor( @@ -175,20 +190,15 @@ export class CollectionConfigBuilder< query: config.query, requireObjectResult: true, }) + this.initialWindow = this.query.orderBy?.length + ? { + offset: this.query.offset ?? 0, + limit: this.query.limit ?? Infinity, + } + : undefined + this.settledWindow = this.initialWindow this.collections = extractCollectionsFromQuery(this.query) - const collectionAliasesById = extractCollectionAliases(this.query) - - // Build a reverse lookup map from alias to collection instance. - // This enables self-join support where the same collection can be referenced - // multiple times with different aliases (e.g., { employee: col, manager: col }) - this.collectionByAlias = {} - for (const [collectionId, aliases] of collectionAliasesById.entries()) { - const collection = this.collections[collectionId] - if (!collection) continue - for (const alias of aliases) { - this.collectionByAlias[alias] = collection - } - } + this.collectionSources = extractCollectionSources(this.query) // Create compare function for ordering if the query has orderBy if (this.query.orderBy && this.query.orderBy.length > 0) { @@ -239,6 +249,7 @@ export class CollectionConfigBuilder< getConfig(): CollectionConfigSingleRowOption & { utils: LiveQueryCollectionUtils } { + const builder = this return { id: this.id, getKey: @@ -248,7 +259,7 @@ export class CollectionConfigBuilder< sync: this.getSyncConfig(), compare: this.compare, defaultStringCollation: this.compareOptions, - gcTime: this.config.gcTime || 5000, // 5 seconds by default for live queries + gcTime: this.config.gcTime ?? 5000, // 5 seconds by default for live queries schema: this.config.schema, onInsert: this.config.onInsert, onUpdate: this.config.onUpdate, @@ -256,7 +267,9 @@ export class CollectionConfigBuilder< startSync: this.config.startSync, singleResult: this.query.singleResult, utils: { - getRunCount: this.getRunCount.bind(this), + get lastSubsetError() { + return builder.lastSubsetError + }, setWindow: this.setWindow.bind(this), getWindow: this.getWindow.bind(this), [LIVE_QUERY_INTERNAL]: { @@ -270,70 +283,251 @@ export class CollectionConfigBuilder< } setWindow(options: WindowOptions): true | Promise { - if (!this.windowFn) { + const windowFn = this.windowFn + if (!windowFn) { throw new SetWindowRequiresOrderByError() } + if ( + this.activeWindowOperation || + this.isGraphRunning || + Object.values(this.optimizableOrderByCollections).some((info) => + info.isRequesting?.(), + ) + ) { + throw new SetWindowReentrancyError() + } - this.currentWindow = options - this.windowFn(options) - this.maybeRunGraphFn?.() - - // Check if loading a subset was triggered - if (this.liveQueryCollection?.isLoadingSubset) { - // Loading was triggered, return a promise that resolves when it completes - return new Promise((resolve) => { - const unsubscribe = this.liveQueryCollection!.on( - `loadingSubset:change`, - (event) => { - if (!event.isLoadingSubset) { - unsubscribe() - resolve() - } - }, - ) + // Keep caller-owned objects out of the long-lived query state. A caller may + // reuse and mutate its options object after this operation settles. + const baseWindow = + this.currentWindow ?? this.settledWindow ?? this.initialWindow + const requestedWindow: WindowOptions = { + offset: options.offset ?? baseWindow?.offset, + limit: options.limit ?? baseWindow?.limit, + } + const sourceRecovery = this.pendingSourceRecovery() + if (sourceRecovery) { + return sourceRecovery.then(async () => { + const settlement = this.setWindow(requestedWindow) + if (settlement !== true) await settlement }) } + if (this.hasFailedSourceRecovery()) { + return Promise.reject( + this.lastSubsetError ?? new Error(`Source recovery failed`), + ) + } + const windowOperationGeneration = ++this.windowOperationGeneration + const loadOperation = + this.liveQueryCollection?._sync.beginLoadSubsetOperation() + const previousOperation = this.activeWindowOperation + const operation: { + generation: number + failed: boolean + error?: unknown + } = { generation: windowOperationGeneration, failed: false } + this.activeWindowOperation = operation + this.windowFailed = false + if (this.pendingOrderedLoads.size === 0) this.orderedLoadFailed = false + try { + // The window and all source work it causes form one synchronous + // publication. This makes operation tracking see requests scheduled by + // the graph rather than declaring the window settled too early. + this.currentWindow = requestedWindow + withPublicationContext(() => { + windowFn(requestedWindow) + this.maybeRunGraphFn?.() + }) + if (operation.failed) throw operation.error + } catch (error) { + if (windowOperationGeneration === this.windowOperationGeneration) { + this.windowFailed = true + this.currentWindow = this.settledWindow + } + loadOperation?.cancel() + throw error + } finally { + this.activeWindowOperation = previousOperation + } - // No loading was triggered - return true + const settlement = loadOperation?.wait() ?? true + if (settlement === true) { + this.settledWindow = requestedWindow + return true + } + return settlement.then( + () => { + if (windowOperationGeneration === this.windowOperationGeneration) { + this.settledWindow = requestedWindow + } + }, + (error) => { + if (windowOperationGeneration === this.windowOperationGeneration) { + this.windowFailed = true + this.currentWindow = this.settledWindow + } + throw error + }, + ) } getWindow(): { offset: number; limit: number } | undefined { // Only return window if this is a windowed query (has orderBy and windowFn) - if (!this.windowFn || !this.currentWindow) { + const window = this.settledWindow ?? this.initialWindow + if (!this.windowFn || !window) { return undefined } return { - offset: this.currentWindow.offset ?? 0, - limit: this.currentWindow.limit ?? 0, + offset: window.offset ?? 0, + limit: window.limit ?? 0, } } - /** - * Resolves a collection alias to its collection ID. - * - * Uses a two-tier lookup strategy: - * 1. First checks compiled aliases (includes subquery inner aliases) - * 2. Falls back to declared aliases from the query's from/join clauses - * - * @param alias - The alias to resolve (e.g., "employee", "manager") - * @returns The collection ID that the alias references - * @throws {Error} If the alias is not found in either lookup - */ - getCollectionIdForAlias(alias: string): string { - const compiled = this.compiledAliasToCollectionId[alias] - if (compiled) { - return compiled + isLazySource(sourceId: string): boolean { + return this.lazySources.has(sourceId) + } + + beginDemand(planId: string): number { + const generation = (this.demandGenerations.get(planId) ?? 0) + 1 + this.demandGenerations.set(planId, generation) + this.activeDemands.set(planId, { + generation, + settled: false, + }) + return generation + } + + settleDemand(planId: string, generation: number): void { + const demand = this.activeDemands.get(planId) + if (!demand || demand.generation !== generation || demand.settled) return + demand.settled = true + this.maybeRunGraphFn?.() + } + + failDemand(planId: string, generation: number, error: unknown): void { + const demand = this.activeDemands.get(planId) + if (!demand || demand.generation !== generation) return + const normalized = this.recordSubsetError(error) + this.transitionToError( + `Subset demand '${planId}' failed: ${normalized.message}`, + normalized, + ) + } + + recordSubsetError(error: unknown, fatalBeforeReady = false): Error { + const normalized = normalizeError(error) + this.lastSubsetError = normalized + if (this.activeWindowOperation) { + this.activeWindowOperation.failed = true + this.activeWindowOperation.error = normalized + // A synchronous adapter failure can arrive before it returns a promise + // for the ordered-load tracker. Keep any private graph changes hidden. + this.orderedLoadFailed = true } - const collection = this.collectionByAlias[alias] - if (collection) { - return collection.id + if (fatalBeforeReady) { + this.transitionToError( + `Initial subset load failed: ${normalized.message}`, + normalized, + ) } - throw new Error(`Unknown source alias "${alias}"`) + return normalized + } + + trackSubsetLoadPromise(promise: Promise): void { + this.liveQueryCollection!._sync.trackLoadPromise(promise) } - isLazyAlias(alias: string): boolean { - return this.lazySources.has(alias) + trackSubsetLoadOperationPromise(promise: Promise): void { + this.liveQueryCollection!._sync.trackLoadSubsetOperationPromise(promise) + } + + hasActiveWindowOperation(): boolean { + return this.activeWindowOperation !== undefined + } + + getActiveWindowOperationGeneration(): number | undefined { + return this.activeWindowOperation?.generation + } + + scheduleGraphRunForSession(syncSession: number): void { + if ( + syncSession !== this.syncSession || + !this.currentSyncConfig || + !this.currentSyncState + ) { + return + } + this.scheduleGraphRun() + } + + trackOrderedLoadPromise( + promise: Promise, + holdPublication = false, + ): void { + // Hold the last complete public snapshot during an initial load or an + // imperative window move. Source changes that arrive during the move join + // its private graph state and publish with the completed replacement. + if ( + !holdPublication && + !this.activeWindowOperation && + this.liveQueryCollection?.status !== `loading` && + this.pendingOrderedLoads.size === 0 + ) { + return + } + const syncSession = this.syncSession + if (this.pendingOrderedLoads.size === 0) this.orderedLoadFailed = false + this.pendingOrderedLoads.add(promise) + const finish = (succeeded: boolean) => { + // Admission precedes mutation: cleanup retires this session's participants. + if ( + syncSession !== this.syncSession || + !this.pendingOrderedLoads.delete(promise) + ) { + return + } + if (!succeeded) this.orderedLoadFailed = true + if (!this.orderedLoadFailed && this.pendingOrderedLoads.size === 0) { + // The ordered chain already drove its source graph to quiescence. + // Flush the retained result without invoking the source loaders again. + this.scheduleGraphRun() + } + } + void promise.then( + () => finish(true), + () => finish(false), + ) + } + + retireDemand(planId: string): void { + this.activeDemands.delete(planId) + } + + hasPendingSourceRecovery(): boolean { + return Object.values(this.subscriptions).some( + (subscription) => subscription.hasPendingTruncateReplacement, + ) + } + + private pendingSourceRecovery(): Promise | undefined { + const pending = Object.values(this.subscriptions).flatMap((subscription) => + subscription.pendingTruncateReplacement + ? [subscription.pendingTruncateReplacement] + : [], + ) + return pending.length > 0 + ? Promise.all(pending).then(() => undefined) + : undefined + } + + private hasFailedSourceRecovery(): boolean { + return Object.values(this.subscriptions).some( + (subscription) => subscription.hasFailedTruncateReplacement, + ) + } + + getSyncSession(): number { + return this.syncSession } // The callback function is called after the graph has run. @@ -343,8 +537,8 @@ export class CollectionConfigBuilder< // That can happen because even though we load N rows, the pipeline might filter some of these rows out // causing the orderBy operator to receive less than N rows or even no rows at all. // So this callback would notice that it doesn't have enough rows and load some more. - // The callback returns a boolean, when it's true it's done loading data and we can mark the collection as ready. - maybeRunGraph(callback?: () => boolean) { + // Readiness follows source/demand state, not the callback's return value. + maybeRunGraph(callback?: () => void) { if (this.isGraphRunning) { // no nested runs of the graph // which is possible if the `callback` @@ -362,8 +556,14 @@ export class CollectionConfigBuilder< this.isGraphRunning = true try { - const { begin, commit } = this.currentSyncConfig + const syncSession = this.syncSession + const config = this.currentSyncConfig + const { begin, commit } = config const syncState = this.currentSyncState + const isCurrentSession = () => + syncSession === this.syncSession && + this.currentSyncConfig === config && + this.currentSyncState === syncState // Don't run if the live query is in an error state if (this.isInErrorState) { @@ -373,23 +573,47 @@ export class CollectionConfigBuilder< // Always run the graph if subscribed (eager execution) if (syncState.subscribedToAllCollections) { let callbackCalled = false - while (syncState.graph.pendingWork()) { - syncState.graph.run() - // Flush accumulated changes after each graph step to commit them as one transaction. - // This ensures intermediate join states (like null on one side) don't cause - // duplicate key errors when the full join result arrives in the same step. - syncState.flushPendingChanges?.() - callback?.() - callbackCalled = true + const drainGraph = () => { + while (syncState.graph.pendingWork()) { + try { + syncState.graph.run() + } catch (error) { + if (isCurrentSession()) { + this.transitionToError(`Live query graph failed`, error) + } + throw error + } + if (!isCurrentSession()) return false + callback?.() + if (!isCurrentSession()) return false + callbackCalled = true + } + return true } + if (!drainGraph()) return + // Ensure the callback runs at least once even when the graph has no pending work. // This handles lazy loading scenarios where setWindow() increases the limit or // an async loadSubset completes and we need to re-check if more data is needed. + // drainGraph changes this flag inside its closure. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition if (!callbackCalled) { callback?.() + if (!isCurrentSession()) return } + // A synchronous loader can write while this graph run is active. Its + // nested schedule is intentionally coalesced, so drain that new input + // here before publishing the transaction. + if (!drainGraph()) return + + // Publish only after every operator has reached quiescence. A source + // change can reach sibling materializations in different graph steps; + // flushing between those steps would expose a mixed root snapshot. + syncState.flushPendingChanges?.() + if (!isCurrentSession()) return + // On the initial run, we may need to do an empty commit to ensure that // the collection is initialized if (syncState.messagesCount === 0) { @@ -402,7 +626,7 @@ export class CollectionConfigBuilder< // 1. All data has been processed through the graph // 2. All source collections have had a chance to send their initial data // This prevents marking ready before data is processed (fixes isReady=true with empty data) - this.updateLiveQueryStatus(this.currentSyncConfig) + this.updateLiveQueryStatus(config) } } finally { this.isGraphRunning = false @@ -420,45 +644,31 @@ export class CollectionConfigBuilder< * * Uses the current sync session's config and syncState from instance properties. * - * @param callback - Optional callback to load more data if needed (returns true when done) + * @param callback - Optional callback to load more data if needed * @param options - Optional scheduling configuration * @param options.contextId - Transaction ID to group work; defaults to active transaction * @param options.jobId - Unique identifier for this job; defaults to this builder instance - * @param options.alias - Source alias that triggered this schedule; adds alias-specific dependencies * @param options.dependencies - Explicit dependency list; overrides auto-discovered dependencies */ scheduleGraphRun( - callback?: () => boolean, + callback?: () => void, options?: { contextId?: SchedulerContextId jobId?: unknown - alias?: string dependencies?: Array> }, ) { - const contextId = options?.contextId ?? getActiveTransaction()?.id + const contextId = + options?.contextId ?? + getActiveTransaction()?.id ?? + getActivePublicationContext() // Use the builder instance as the job ID for deduplication. This is memory-safe // because the scheduler's context Map is deleted after flushing (no long-term retention). const jobId = options?.jobId ?? this - const dependentBuilders = (() => { - if (options?.dependencies) { - return options.dependencies - } - - const deps = new Set(this.builderDependencies) - if (options?.alias) { - const aliasDeps = this.aliasDependencies[options.alias] - if (aliasDeps) { - for (const dep of aliasDeps) { - deps.add(dep) - } - } - } - - deps.delete(this) - - return Array.from(deps) - })() + // Snapshot before scheduling parents, which can reenter source setup. + const dependentBuilders = options?.dependencies ?? [ + ...this.builderDependencies, + ] // Ensure dependent builders are actually scheduled in this context so that // dependency edges always point to a real job (or a deduped no-op if already scheduled). @@ -482,8 +692,9 @@ export class CollectionConfigBuilder< // Manage our own state - get or create pending callbacks for this context let pending = contextId ? this.pendingGraphRuns.get(contextId) : undefined - if (!pending) { + if (!pending || pending.syncSession !== this.syncSession) { pending = { + syncSession: this.syncSession, loadCallbacks: new Set(), } if (contextId) { @@ -551,31 +762,15 @@ export class CollectionConfigBuilder< } // If sync session has ended, don't execute (graph is finalized, subscriptions cleared) - if (!this.currentSyncConfig || !this.currentSyncState) { + if ( + pending.syncSession !== this.syncSession || + !this.currentSyncConfig || + !this.currentSyncState + ) { return } - this.incrementRunCount() - - const combinedLoader = () => { - let allDone = true - let firstError: unknown - pending.loadCallbacks.forEach((loader) => { - try { - allDone = loader() && allDone - } catch (error) { - allDone = false - firstError ??= error - } - }) - if (firstError) { - throw firstError - } - // Returning false signals that callers should schedule another pass. - return allDone - } - - this.maybeRunGraph(combinedLoader) + this.maybeRunGraph(() => runAllCallbacks(pending.loadCallbacks)) } private getSyncConfig(): SyncConfig { @@ -585,17 +780,15 @@ export class CollectionConfigBuilder< } } - incrementRunCount() { - this.runCount++ - } - - getRunCount() { - return this.runCount - } - private syncFn(config: SyncMethods) { + const syncSession = ++this.syncSession // Store reference to the live query collection for error state transitions this.liveQueryCollection = config.collection + // Reset error state from any previous sync session so a restarted sync can become ready again. + this.isInErrorState = false + this.fatalQueryError = false + this.erroredSourceIds.clear() + this.lastSubsetError = undefined // Store config and syncState as instance properties for the duration of this sync session this.currentSyncConfig = config @@ -605,83 +798,122 @@ export class CollectionConfigBuilder< unsubscribeCallbacks: new Set<() => void>(), } - // Extend the pipeline such that it applies the incoming changes to the collection - const fullSyncState = this.extendPipelineWithChangeProcessing( - config, - syncState, - ) - this.currentSyncState = fullSyncState - - // Listen for scheduler context clears to clean up our pending state - // Re-register on each sync start so the listener is active for the sync session's lifetime - this.unsubscribeFromSchedulerClears = transactionScopedScheduler.onClear( - (contextId) => { - this.clearPendingGraphRun(contextId) - }, - ) - - // Listen for loadingSubset changes on the live query collection BEFORE subscribing. - // This ensures we don't miss the event if subset loading completes synchronously. - // When isLoadingSubset becomes false, we may need to mark the collection as ready - // (if all source collections are already ready but we were waiting for subset load to complete) - const loadingSubsetUnsubscribe = config.collection.on( - `loadingSubset:change`, - (event) => { - if (!event.isLoadingSubset) { - // Subset loading finished, check if we can now mark ready - this.updateLiveQueryStatus(config) - } - }, - ) - syncState.unsubscribeCallbacks.add(loadingSubsetUnsubscribe) - - const loadSubsetDataCallbacks = this.subscribeToAllCollections( - config, - fullSyncState, - ) + let tornDown = false + const teardown = () => { + if (tornDown) return + tornDown = true + if (this.syncSession === syncSession) this.syncSession++ + + // Release every source in one attempt; the first failure wins after the + // peers finish. Each subscription release is itself one-shot, so the + // Collection's cleanup retry has nothing left to repeat here. + try { + runAllCallbacks(syncState.unsubscribeCallbacks) + } finally { + syncState.unsubscribeCallbacks.clear() + this.clearSyncSessionState() + } + } - this.maybeRunGraphFn = () => this.scheduleGraphRun(loadSubsetDataCallbacks) + try { + // Extend the pipeline such that it applies the incoming changes to the collection + const fullSyncState = this.extendPipelineWithChangeProcessing( + config, + syncState, + ) + this.currentSyncState = fullSyncState - // Initial run with callback to load more data if needed - this.scheduleGraphRun(loadSubsetDataCallbacks) + // Listen for scheduler context clears to clean up our pending state + // Re-register on each sync start so the listener is active for the sync session's lifetime + this.unsubscribeFromSchedulerClears = transactionScopedScheduler.onClear( + (contextId) => { + this.clearPendingGraphRun(contextId) + }, + ) - // Return the unsubscribe function - return () => { - syncState.unsubscribeCallbacks.forEach((unsubscribe) => unsubscribe()) + // Listen for loadingSubset changes on the live query collection BEFORE subscribing. + // This ensures we don't miss the event if subset loading completes synchronously. + // When isLoadingSubset becomes false, we may need to mark the collection as ready + // (if all source collections are already ready but we were waiting for subset load to complete) + const loadingSubsetUnsubscribe = config.collection.on( + `loadingSubset:change`, + (event) => { + if (!event.isLoadingSubset) { + // Subset loading finished, check if we can now mark ready + this.updateLiveQueryStatus(config) + if (this.hasPendingSourceRecovery()) this.maybeRunGraphFn?.() + } + }, + ) + syncState.unsubscribeCallbacks.add(loadingSubsetUnsubscribe) - // Clear current sync session state - this.currentSyncConfig = undefined - this.currentSyncState = undefined + const loadSubsetDataCallbacks = this.subscribeToAllCollections( + config, + fullSyncState, + ) - // Clear all pending graph runs to prevent memory leaks from in-flight transactions - // that may flush after the sync session ends - this.pendingGraphRuns.clear() + this.maybeRunGraphFn = () => + this.scheduleGraphRun(loadSubsetDataCallbacks) - // Reset caches so a fresh graph/pipeline is compiled on next start - // This avoids reusing a finalized D2 graph across GC restarts - this.graphCache = undefined - this.inputsCache = undefined - this.pipelineCache = undefined - this.sourceWhereClausesCache = undefined - this.includesCache = undefined + // Initial run with callback to load more data if needed + this.scheduleGraphRun(loadSubsetDataCallbacks) + } catch (error) { + try { + teardown() + } catch { + // Preserve the setup failure. It is the error the caller can act on. + } + throw error + } - // Reset lazy source alias state - this.lazySources.clear() - this.optimizableOrderByCollections = {} - this.lazySourcesCallbacks = {} + return teardown + } - // Clear subscription references to prevent memory leaks - // Note: Individual subscriptions are already unsubscribed via unsubscribeCallbacks - Object.keys(this.subscriptions).forEach( - (key) => delete this.subscriptions[key], - ) - this.compiledAliasToCollectionId = {} + private clearSyncSessionState(): void { + // Late window settlement belongs to the discarded graph, not its restart. + this.windowOperationGeneration++ + // Clear current sync session state + this.currentSyncConfig = undefined + this.currentSyncState = undefined + this.maybeRunGraphFn = undefined + this.currentWindow = undefined + this.settledWindow = this.initialWindow + this.isInErrorState = false + this.fatalQueryError = false + this.erroredSourceIds.clear() + + // Clear all pending graph runs to prevent memory leaks from in-flight transactions + // that may flush after the sync session ends + this.pendingGraphRuns.clear() + + // Reset caches so a fresh graph/pipeline is compiled on next start + // This avoids reusing a finalized D2 graph across GC restarts + this.graphCache = undefined + this.inputsCache = undefined + this.pipelineCache = undefined + this.sourceWhereClausesCache = undefined + this.bucketFacadesCache = undefined + + // Reset lazy source alias state + this.lazySources.clear() + this.demandGenerations.clear() + this.activeDemands.clear() + this.pendingOrderedLoads.clear() + this.orderedLoadFailed = false + this.windowFailed = false + this.optimizableOrderByCollections = {} + this.lazySourcesCallbacks = {} + + // Clear subscription references to prevent memory leaks + // Note: Individual subscriptions are already unsubscribed via unsubscribeCallbacks + Object.keys(this.subscriptions).forEach( + (key) => delete this.subscriptions[key], + ) - // Unregister from scheduler's onClear listener to prevent memory leaks - // The scheduler's listener Set would otherwise keep a strong reference to this builder - this.unsubscribeFromSchedulerClears?.() - this.unsubscribeFromSchedulerClears = undefined - } + // Unregister from scheduler's onClear listener to prevent memory leaks + // The scheduler's listener Set would otherwise keep a strong reference to this builder + this.unsubscribeFromSchedulerClears?.() + this.unsubscribeFromSchedulerClears = undefined } /** @@ -690,8 +922,8 @@ export class CollectionConfigBuilder< private compileBasePipeline() { this.graphCache = new D2() this.inputsCache = Object.fromEntries( - Object.keys(this.collectionByAlias).map((alias) => [ - alias, + this.collectionSources.map((source) => [ + source.sourceId, this.graphCache!.newInput(), ]), ) @@ -706,22 +938,29 @@ export class CollectionConfigBuilder< this.optimizableOrderByCollections, (windowFn: (options: WindowOptions) => void) => { this.windowFn = windowFn + // `setWindow` mutates the compiled top-K operator, which is replaced + // whenever a cleaned-up live query compiles a fresh pipeline. Keep the + // desired window on the builder and replay it into each new operator. + if (this.currentWindow) { + windowFn(this.currentWindow) + } }, ) - this.pipelineCache = compilation.pipeline - this.sourceWhereClausesCache = compilation.sourceWhereClauses - this.compiledAliasToCollectionId = compilation.aliasToCollectionId - this.includesCache = compilation.includes - - // Defensive check: verify all compiled aliases have corresponding inputs - // This should never happen since all aliases come from user declarations, - // but catch it early if the assumption is violated in the future. - const missingAliases = Object.keys(this.compiledAliasToCollectionId).filter( - (alias) => !Object.hasOwn(this.inputsCache!, alias), + const materialized = materializeCompilation( + compilation, + this.config.getKey, + this.hasJoins(this.query), ) - if (missingAliases.length > 0) { - throw new MissingAliasInputsError(missingAliases) + this.pipelineCache = materialized.pipeline + this.sourceWhereClausesCache = compilation.sourceWhereClauses + this.bucketFacadesCache = materialized.facades + + const missingSources = this.collectionSources + .map((source) => source.sourceId) + .filter((sourceId) => !Object.hasOwn(this.inputsCache!, sourceId)) + if (missingSources.length > 0) { + throw new MissingAliasInputsError(missingSources) } } @@ -760,68 +999,99 @@ export class CollectionConfigBuilder< }), ) - // Set up includes output routing and child collection lifecycle - const includesState = this.setupIncludesOutput( - this.includesCache, - syncState, + const bucketFacades = new BucketFacadeAdapter( + this.id, + this.bucketFacadesCache ?? [], + (count) => { + syncState.messagesCount += count + }, ) + syncState.unsubscribeCallbacks.add(() => bucketFacades.cleanup()) // Flush pending changes and reset the accumulator. // Called at the end of each graph run to commit all accumulated changes. syncState.flushPendingChanges = () => { const hasParentChanges = pendingChanges.size > 0 - const hasChildChanges = hasPendingIncludesChanges(includesState) + const hasChildChanges = bucketFacades.hasPendingChanges() if (!hasParentChanges && !hasChildChanges) { return } - let changesToApply = pendingChanges - - // When a custom getKey is provided, multiple D2 internal keys may map - // to the same user-visible key. Re-accumulate by custom key so that a - // retract + insert for the same logical row merges into an UPDATE - // instead of a separate DELETE and INSERT that can race. - if (this.config.getKey) { - const merged = new Map>() - for (const [, changes] of pendingChanges) { - const customKey = this.config.getKey(changes.value) - const existing = merged.get(customKey) - if (existing) { - existing.inserts += changes.inserts - existing.deletes += changes.deletes - // Keep the value from the insert side (the new value) - if (changes.inserts > 0) { - existing.value = changes.value - if (changes.orderByIndex !== undefined) { - existing.orderByIndex = changes.orderByIndex - } + if ( + this.windowFailed || + this.orderedLoadFailed || + this.hasPendingSourceRecovery() || + this.pendingOrderedLoads.size > 0 + ) { + return + } + + let facadePublication: + | ReturnType + | undefined + let rootPublication: + | ReturnType + | undefined + try { + facadePublication = bucketFacades.flush() + rootPublication = hasParentChanges + ? config.collection._deferPublication() + : undefined + const changesToApply: Map> = new Map( + [...pendingChanges].map(([key, changes]) => { + const resolved: Changes = { + ...changes, + value: bucketFacades.resolve(changes.value), } - } else { - merged.set(customKey, { ...changes }) + if (changes.previousValue !== undefined) { + resolved.previousValue = bucketFacades.resolve( + changes.previousValue, + ) + } + return [key, resolved] + }), + ) + // New facades are not reachable until their root row is installed, so + // make them ready first. A facade failure then leaves the root intact, + // and the root commit is the final state change before publication. + facadePublication.prepare() + if (hasParentChanges) { + begin() + let lookup: ((key: string | number) => boolean) | undefined + const hasSyncedKey = (key: string | number) => { + lookup ??= config.collection._state.createSyncedKeyLookup() + return lookup(key) + } + changesToApply.forEach( + this.applyChanges.bind(this, config, hasSyncedKey), + ) + if (hasOrderOnlyMove(changesToApply)) { + markLayoutChange(config.collection) } + commit() } - changesToApply = merged - } - - // 1. Flush parent changes - if (hasParentChanges) { - begin() - changesToApply.forEach(this.applyChanges.bind(this, config)) - commit() + } catch (error) { + rootPublication?.discard() + facadePublication?.rollback() + throw error } pendingChanges = new Map() - // 2. Process includes: create/dispose child Collections, route child changes - flushIncludesState( - includesState, - config.collection, - this.id, - hasParentChanges ? changesToApply : null, - config, - ) + let publicationError: unknown + for (const publish of [ + rootPublication?.publish, + facadePublication.publish, + ]) { + if (!publish) continue + try { + publish() + } catch (error) { + publicationError ??= error + } + } + if (publicationError !== undefined) throw publicationError } - graph.finalize() // Extend the sync state with the graph, inputs, and pipeline @@ -832,91 +1102,9 @@ export class CollectionConfigBuilder< return syncState as FullSyncState } - /** - * Sets up output callbacks for includes child pipelines. - * Each includes entry gets its own output callback that accumulates child changes, - * and a child registry that maps correlation key → child Collection. - */ - private setupIncludesOutput( - includesEntries: Array | undefined, - syncState: SyncState, - ): Array { - if (!includesEntries || includesEntries.length === 0) { - return [] - } - - return includesEntries.map((entry) => { - const state: IncludesOutputState = { - fieldName: entry.fieldName, - resultPath: entry.resultPath, - childCorrelationField: entry.childCorrelationField, - hasOrderBy: entry.hasOrderBy, - materialization: entry.materialization, - scalarField: entry.scalarField, - childRegistry: new Map(), - pendingChildChanges: new Map(), - correlationToParentKeys: new Map(), - } - - // Attach output callback on the child pipeline - entry.pipeline.pipe( - output((data) => { - const messages = data.getInner() - syncState.messagesCount += messages.length - - for (const [[childKey, tupleData], multiplicity] of messages) { - const [childResult, _orderByIndex, correlationKey, parentContext] = - tupleData as unknown as [ - any, - string | undefined, - unknown, - Record | null, - ] - - const routingKey = computeRoutingKey(correlationKey, parentContext) - - // Accumulate by [routingKey, childKey] - let byChild = state.pendingChildChanges.get(routingKey) - if (!byChild) { - byChild = new Map() - state.pendingChildChanges.set(routingKey, byChild) - } - - const existing = byChild.get(childKey) || { - deletes: 0, - inserts: 0, - value: childResult, - orderByIndex: _orderByIndex, - } - - if (multiplicity < 0) { - existing.deletes += Math.abs(multiplicity) - } else if (multiplicity > 0) { - existing.inserts += multiplicity - existing.value = childResult - } - - byChild.set(childKey, existing) - } - }), - ) - - // Set up shared buffers for nested includes (e.g., comments inside issues) - if (entry.childCompilationResult.includes) { - state.nestedSetups = setupNestedPipelines( - entry.childCompilationResult.includes, - syncState, - ) - state.nestedRoutingIndex = new Map() - state.nestedRoutingReverseIndex = new Map() - } - - return state - }) - } - private applyChanges( config: SyncMethods, + hasSyncedKey: (key: string | number) => boolean, changes: { deletes: number inserts: number @@ -946,9 +1134,9 @@ export class CollectionConfigBuilder< } else if ( // Insert & update(s) (updates are a delete & insert) inserts > deletes || - // Just update(s) but the item is already in the collection (so - // was inserted previously). - (inserts === deletes && collection.has(collection.getKeyFromItem(value))) + // A balanced delta updates an existing authoritative row, even if an + // optimistic delete hides it or its earlier insert is still queued. + (inserts === deletes && hasSyncedKey(collection.getKeyFromItem(value))) ) { write({ value, @@ -972,6 +1160,7 @@ export class CollectionConfigBuilder< */ private handleSourceStatusChange( config: SyncMethods, + sourceId: string, collectionId: string, event: AllCollectionEvents[`status:change`], ) { @@ -979,7 +1168,8 @@ export class CollectionConfigBuilder< // Handle error state - any source collection in error puts live query in error if (status === `error`) { - this.transitionToError( + this.erroredSourceIds.add(sourceId) + this.setErrorState( `Source collection '${collectionId}' entered error state`, ) return @@ -995,6 +1185,18 @@ export class CollectionConfigBuilder< return } + if (status === `ready`) { + const recovered = this.erroredSourceIds.delete(sourceId) + if ( + recovered && + !this.fatalQueryError && + this.erroredSourceIds.size === 0 + ) { + this.isInErrorState = false + this.maybeRunGraphFn?.() + } + } + // Update ready status based on all source collections this.updateLiveQueryStatus(config) } @@ -1011,15 +1213,19 @@ export class CollectionConfigBuilder< } const subscribedToAll = this.currentSyncState?.subscribedToAllCollections - const allReady = this.allCollectionsReady() + const allReady = this.allRequiredSourcesReady() + const allDemandsSettled = [...this.activeDemands.values()].every( + (demand) => demand.settled, + ) const isLoading = this.liveQueryCollection?.isLoadingSubset // Mark ready when: // 1. All subscriptions are set up (subscribedToAllCollections) // 2. All source collections are ready - // 3. The live query collection is not loading subset data + // 3. Every active route demand has settled + // 4. The live query collection is not loading subset data // This prevents marking the live query ready before its data is processed // (fixes issue where useLiveQuery returns isReady=true with empty data) - if (subscribedToAll && allReady && !isLoading) { + if (subscribedToAll && allReady && allDemandsSettled && !isLoading) { markReady() } } @@ -1027,74 +1233,96 @@ export class CollectionConfigBuilder< /** * Transition the live query to error state */ - private transitionToError(message: string) { + private transitionToError(message: string, error?: unknown) { + this.fatalQueryError = true + this.setErrorState(message, error) + } + + private setErrorState(message: string, error?: unknown) { this.isInErrorState = true // Log error to console for debugging console.error(`[Live Query Error] ${message}`) // Transition live query collection to error state - this.liveQueryCollection?._lifecycle.setStatus(`error`) + this.liveQueryCollection?._lifecycle.markError(error ?? new Error(message)) } - private allCollectionsReady() { - return Object.values(this.collections).every((collection) => - collection.isReady(), + private allRequiredSourcesReady() { + return this.collectionSources.every( + (source) => + // Only on-demand sources settle through route demand. Eager + // loadSubset calls return immediately, so they must reach ready. + (this.lazySources.has(source.sourceId) && + source.collection.config.syncMode === `on-demand`) || + source.collection.isReady(), ) } /** - * Creates per-alias subscriptions enabling self-join support. - * Each alias gets its own subscription with independent filters, even for the same collection. + * Creates one subscription per lexical collection source. + * Each source gets independent filters, even when aliases or collections repeat. * Example: `{ employee: col, manager: col }` creates two separate subscriptions. */ private subscribeToAllCollections( config: SyncMethods, syncState: FullSyncState, ) { - // Use compiled aliases as the source of truth - these include all aliases from the query - // including those from subqueries, which may not be in collectionByAlias - const compiledAliases = Object.entries(this.compiledAliasToCollectionId) - if (compiledAliases.length === 0) { + if (this.collectionSources.length === 0) { throw new Error( - `Compiler returned no alias metadata for query '${this.id}'. This should not happen; please report.`, + `Query '${this.id}' has no collection sources. This should not happen; please report.`, ) } - // Create a separate subscription for each alias, enabling self-joins where the same - // collection can be used multiple times with different filters and subscriptions - const loaders = compiledAliases.map(([alias, collectionId]) => { - // Try collectionByAlias first (for declared aliases), fall back to collections (for subquery aliases) - const collection = - this.collectionByAlias[alias] ?? this.collections[collectionId]! + const loaders = this.collectionSources.map((source) => { + const { sourceId, alias, collection } = source + const collectionId = collection.id const dependencyBuilder = getCollectionBuilder(collection) if (dependencyBuilder && dependencyBuilder !== this) { - this.aliasDependencies[alias] = [dependencyBuilder] this.builderDependencies.add(dependencyBuilder) - } else { - this.aliasDependencies[alias] = [] } // CollectionSubscriber handles the actual subscription to the source collection // and feeds data into the D2 graph inputs for this specific alias const collectionSubscriber = new CollectionSubscriber( + sourceId, alias, - collectionId, collection, this, ) // Subscribe to status changes for status flow const statusUnsubscribe = collection.on(`status:change`, (event) => { - this.handleSourceStatusChange(config, collectionId, event) + this.handleSourceStatusChange(config, sourceId, collectionId, event) }) syncState.unsubscribeCallbacks.add(statusUnsubscribe) + // The source may have failed before this live query subscribed. Register + // the listener first, then reconcile that current state so no transition + // can be missed between observation and subscription. + if (collection.status === `error`) { + this.handleSourceStatusChange(config, sourceId, collectionId, { + type: `status:change`, + collection, + status: `error`, + previousStatus: `error`, + }) + } + const subscription = collectionSubscriber.subscribe() - // Store subscription by alias (not collection ID) to support lazy loading - // which needs to look up subscriptions by their query alias - this.subscriptions[alias] = subscription + this.subscriptions[sourceId] = subscription + + const lazyCallbacks = this.lazySourcesCallbacks[sourceId] + if (lazyCallbacks) { + lazyCallbacks.setDemand = (plan, keys) => + collectionSubscriber.setDemand(subscription, plan, keys) + for (const plan of lazyCallbacks.plans ?? []) { + if (plan.initialKeys.size > 0) { + lazyCallbacks.setDemand(plan, plan.initialKeys) + } + } + } // Create a callback for loading more data if needed (used by OrderBy optimization) const loadMore = collectionSubscriber.loadMoreIfNeeded.bind( @@ -1105,14 +1333,6 @@ export class CollectionConfigBuilder< return loadMore }) - // Combine all loaders into a single callback that initiates loading more data - // from any source that needs it. Returns true once all loaders have been called, - // but the actual async loading may still be in progress. - const loadSubsetDataCallbacks = () => { - loaders.map((loader) => loader()) - return true - } - // Mark as subscribed so the graph can start running // (graph only runs when all collections are subscribed) syncState.subscribedToAllCollections = true @@ -1122,7 +1342,7 @@ export class CollectionConfigBuilder< // The canonical place to mark ready is after the graph processes data // in maybeRunGraph(), which ensures data has been processed first. - return loadSubsetDataCallbacks + return () => runAllCallbacks(loaders) } } @@ -1150,846 +1370,6 @@ function createOrderByComparator( } } -/** - * Shared buffer setup for a single nested includes level. - * Pipeline output writes into the buffer; during flush the buffer is drained - * into per-entry states via the routing index. - */ -type NestedIncludesSetup = { - compilationResult: IncludesCompilationResult - /** Shared buffer: nestedCorrelationKey → Map */ - buffer: Map>> - /** For 3+ levels of nesting */ - nestedSetups?: Array -} - -/** - * State tracked per includes entry for output routing and child lifecycle - */ -type IncludesOutputState = { - fieldName: string - resultPath: Array - childCorrelationField: PropRef - /** Whether the child query has an ORDER BY clause */ - hasOrderBy: boolean - /** How the child result is materialized on the parent row */ - materialization: IncludesMaterialization - /** Internal field used to unwrap scalar child selects */ - scalarField?: string - /** Maps correlation key value → child Collection entry */ - childRegistry: Map - /** Pending child changes: correlationKey → Map */ - pendingChildChanges: Map>> - /** Reverse index: correlation key → Set of parent collection keys */ - correlationToParentKeys: Map> - /** Shared nested pipeline setups (one per nested includes level) */ - nestedSetups?: Array - /** nestedCorrelationKey → parentCorrelationKey */ - nestedRoutingIndex?: Map - /** parentCorrelationKey → Set */ - nestedRoutingReverseIndex?: Map> -} - -type ChildCollectionEntry = { - collection: Collection - syncMethods: SyncMethods | null - resultKeys: WeakMap - orderByIndices: WeakMap | null - /** Per-entry nested includes states (one per nested includes level) */ - includesStates?: Array -} - -function materializesInline(state: IncludesOutputState): boolean { - return state.materialization !== `collection` -} - -function materializeIncludedValue( - state: IncludesOutputState, - entry: ChildCollectionEntry | undefined, -): unknown { - if (!entry) { - if (state.materialization === `array`) { - return [] - } - if (state.materialization === `concat`) { - return `` - } - // `singleton` and `collection` both fall through to undefined when no - // child entry exists for the parent's correlation key. - return undefined - } - - if (state.materialization === `collection`) { - return entry.collection - } - - const rows = [...entry.collection.toArray] - const values = state.scalarField - ? rows.map((row) => row?.[state.scalarField!]) - : rows - - if (state.materialization === `array`) { - return values - } - - if (state.materialization === `singleton`) { - // findOne() doesn't currently push LIMIT 1 to the IR, so the child - // Collection may hold more than one row; pick the first deterministically. - return values[0] - } - - return values.map((value) => String(value ?? ``)).join(``) -} - -/** - * Sets up shared buffers for nested includes pipelines. - * Instead of writing directly into a single shared IncludesOutputState, - * each nested pipeline writes into a buffer that is later drained per-entry. - */ -function setupNestedPipelines( - includes: Array, - syncState: SyncState, -): Array { - return includes.map((entry) => { - const buffer: Map>> = new Map() - - // Attach output callback that writes into the shared buffer - entry.pipeline.pipe( - output((data) => { - const messages = data.getInner() - syncState.messagesCount += messages.length - - for (const [[childKey, tupleData], multiplicity] of messages) { - const [childResult, _orderByIndex, correlationKey, parentContext] = - tupleData as unknown as [ - any, - string | undefined, - unknown, - Record | null, - ] - - const routingKey = computeRoutingKey(correlationKey, parentContext) - - let byChild = buffer.get(routingKey) - if (!byChild) { - byChild = new Map() - buffer.set(routingKey, byChild) - } - - const existing = byChild.get(childKey) || { - deletes: 0, - inserts: 0, - value: childResult, - orderByIndex: _orderByIndex, - } - - if (multiplicity < 0) { - existing.deletes += Math.abs(multiplicity) - } else if (multiplicity > 0) { - existing.inserts += multiplicity - existing.value = childResult - } - - byChild.set(childKey, existing) - } - }), - ) - - const setup: NestedIncludesSetup = { - compilationResult: entry, - buffer, - } - - // Recursively set up deeper levels - if (entry.childCompilationResult.includes) { - setup.nestedSetups = setupNestedPipelines( - entry.childCompilationResult.includes, - syncState, - ) - } - - return setup - }) -} - -/** - * Creates fresh per-entry IncludesOutputState array from NestedIncludesSetup array. - * Each entry gets its own isolated state for nested includes. - */ -function createPerEntryIncludesStates( - setups: Array, -): Array { - return setups.map((setup) => { - const state: IncludesOutputState = { - fieldName: setup.compilationResult.fieldName, - resultPath: setup.compilationResult.resultPath, - childCorrelationField: setup.compilationResult.childCorrelationField, - hasOrderBy: setup.compilationResult.hasOrderBy, - materialization: setup.compilationResult.materialization, - scalarField: setup.compilationResult.scalarField, - childRegistry: new Map(), - pendingChildChanges: new Map(), - correlationToParentKeys: new Map(), - } - - if (setup.nestedSetups) { - state.nestedSetups = setup.nestedSetups - state.nestedRoutingIndex = new Map() - state.nestedRoutingReverseIndex = new Map() - } - - return state - }) -} - -/** - * Drains shared buffers into per-entry states using the routing index. - * Returns the set of parent correlation keys that had changes routed to them. - */ -function drainNestedBuffers(state: IncludesOutputState): Set { - const dirtyCorrelationKeys = new Set() - - if (!state.nestedSetups) return dirtyCorrelationKeys - - for (let i = 0; i < state.nestedSetups.length; i++) { - const setup = state.nestedSetups[i]! - const toDelete: Array = [] - - for (const [nestedCorrelationKey, childChanges] of setup.buffer) { - const parentCorrelationKey = - state.nestedRoutingIndex!.get(nestedCorrelationKey) - if (parentCorrelationKey === undefined) { - // Unroutable — parent not yet seen; keep in buffer - continue - } - - const entry = state.childRegistry.get(parentCorrelationKey) - if (!entry || !entry.includesStates) { - continue - } - - // Route changes into this entry's per-entry state at position i - const entryState = entry.includesStates[i]! - for (const [childKey, changes] of childChanges) { - let byChild = entryState.pendingChildChanges.get(nestedCorrelationKey) - if (!byChild) { - byChild = new Map() - entryState.pendingChildChanges.set(nestedCorrelationKey, byChild) - } - const existing = byChild.get(childKey) - if (existing) { - existing.inserts += changes.inserts - existing.deletes += changes.deletes - if (changes.inserts > 0) { - existing.value = changes.value - if (changes.orderByIndex !== undefined) { - existing.orderByIndex = changes.orderByIndex - } - } - } else { - byChild.set(childKey, { ...changes }) - } - } - - dirtyCorrelationKeys.add(parentCorrelationKey) - toDelete.push(nestedCorrelationKey) - } - - for (const key of toDelete) { - setup.buffer.delete(key) - } - } - - return dirtyCorrelationKeys -} - -/** - * Updates the routing index after processing child changes. - * Maps nested correlation keys to parent correlation keys so that - * grandchild changes can be routed to the correct per-entry state. - */ -function updateRoutingIndex( - state: IncludesOutputState, - correlationKey: unknown, - childChanges: Map>, -): void { - if (!state.nestedSetups) return - - for (const setup of state.nestedSetups) { - for (const [, change] of childChanges) { - if (change.inserts > 0) { - // Read the nested routing key from the INCLUDES_ROUTING stamp. - // Must use the composite routing key (not raw correlationKey) to match - // how nested buffers are keyed by computeRoutingKey. - const nestedRouting = - change.value[INCLUDES_ROUTING]?.[setup.compilationResult.fieldName] - const nestedCorrelationKey = nestedRouting?.correlationKey - const nestedParentContext = nestedRouting?.parentContext ?? null - const nestedRoutingKey = computeRoutingKey( - nestedCorrelationKey, - nestedParentContext, - ) - - if (nestedCorrelationKey != null) { - state.nestedRoutingIndex!.set(nestedRoutingKey, correlationKey) - let reverseSet = state.nestedRoutingReverseIndex!.get(correlationKey) - if (!reverseSet) { - reverseSet = new Set() - state.nestedRoutingReverseIndex!.set(correlationKey, reverseSet) - } - reverseSet.add(nestedRoutingKey) - } - } else if (change.deletes > 0 && change.inserts === 0) { - // Remove from routing index - const nestedRouting2 = - change.value[INCLUDES_ROUTING]?.[setup.compilationResult.fieldName] - const nestedCorrelationKey = nestedRouting2?.correlationKey - const nestedParentContext2 = nestedRouting2?.parentContext ?? null - const nestedRoutingKey = computeRoutingKey( - nestedCorrelationKey, - nestedParentContext2, - ) - - if (nestedCorrelationKey != null) { - state.nestedRoutingIndex!.delete(nestedRoutingKey) - const reverseSet = - state.nestedRoutingReverseIndex!.get(correlationKey) - if (reverseSet) { - reverseSet.delete(nestedRoutingKey) - if (reverseSet.size === 0) { - state.nestedRoutingReverseIndex!.delete(correlationKey) - } - } - } - } - } - } -} - -/** - * Cleans routing index entries when a parent is deleted. - * Uses the reverse index to find and remove all nested routing entries. - */ -function cleanRoutingIndexOnDelete( - state: IncludesOutputState, - correlationKey: unknown, -): void { - if (!state.nestedRoutingReverseIndex) return - - const nestedKeys = state.nestedRoutingReverseIndex.get(correlationKey) - if (nestedKeys) { - for (const nestedKey of nestedKeys) { - state.nestedRoutingIndex!.delete(nestedKey) - } - state.nestedRoutingReverseIndex.delete(correlationKey) - } -} - -/** - * Recursively checks whether any nested buffer has pending changes. - */ -function hasNestedBufferChanges(setups: Array): boolean { - for (const setup of setups) { - if (setup.buffer.size > 0) return true - if (setup.nestedSetups && hasNestedBufferChanges(setup.nestedSetups)) - return true - } - return false -} - -/** - * Computes a composite routing key from correlation key and parent context. - * When parentContext is null (no parent filters), returns the raw correlationKey - * for zero behavioral change on existing queries. - */ -function computeRoutingKey( - correlationKey: unknown, - parentContext: Record | null, -): unknown { - if (parentContext == null) return correlationKey - return JSON.stringify([correlationKey, parentContext]) -} - -/** - * Creates a child Collection entry for includes subqueries. - * The child Collection is a full-fledged Collection instance that starts syncing immediately. - */ -function createChildCollectionEntry( - parentId: string, - fieldName: string, - correlationKey: unknown, - hasOrderBy: boolean, - nestedSetups?: Array, -): ChildCollectionEntry { - const resultKeys = new WeakMap() - const orderByIndices = hasOrderBy ? new WeakMap() : null - let syncMethods: SyncMethods | null = null - - const compare = orderByIndices - ? createOrderByComparator(orderByIndices) - : undefined - - const collection = createCollection({ - id: `__child-collection:${parentId}-${fieldName}-${serializeValue(correlationKey)}`, - getKey: (item: any) => resultKeys.get(item) as string | number, - compare, - sync: { - rowUpdateMode: `full`, - sync: (methods) => { - syncMethods = methods - return () => { - syncMethods = null - } - }, - }, - startSync: true, - gcTime: 0, - }) - - const entry: ChildCollectionEntry = { - collection, - get syncMethods() { - return syncMethods - }, - resultKeys, - orderByIndices, - } - - if (nestedSetups) { - entry.includesStates = createPerEntryIncludesStates(nestedSetups) - } - - return entry -} - -/** - * Flushes includes state using a bottom-up per-entry approach. - * Five phases ensure correct ordering: - * 1. Parent INSERTs — create child entries with per-entry nested states - * 2. Child changes — apply to child Collections, update routing index - * 3. Drain nested buffers — route buffered grandchild changes to per-entry states - * 4. Flush per-entry states — recursively flush nested includes on each entry - * 5. Parent DELETEs — clean up child entries and routing index - */ -function flushIncludesState( - includesState: Array, - parentCollection: Collection, - parentId: string, - parentChanges: Map> | null, - parentSyncMethods: SyncMethods | null, -): void { - for (const state of includesState) { - // Phase 1: Parent INSERTs — ensure a child Collection exists for every parent - if (parentChanges) { - for (const [parentKey, changes] of parentChanges) { - if (changes.inserts > 0) { - const parentResult = changes.value - // Extract routing info from INCLUDES_ROUTING symbol (set by compiler) - const routing = parentResult[INCLUDES_ROUTING]?.[state.fieldName] - const correlationKey = routing?.correlationKey - const parentContext = routing?.parentContext ?? null - const routingKey = computeRoutingKey(correlationKey, parentContext) - - if (correlationKey != null) { - // Ensure child Collection exists for this routing key - if (!state.childRegistry.has(routingKey)) { - const entry = createChildCollectionEntry( - parentId, - state.fieldName, - routingKey, - state.hasOrderBy, - state.nestedSetups, - ) - state.childRegistry.set(routingKey, entry) - } - // Update reverse index: routing key → parent keys - let parentKeys = state.correlationToParentKeys.get(routingKey) - if (!parentKeys) { - parentKeys = new Set() - state.correlationToParentKeys.set(routingKey, parentKeys) - } - parentKeys.add(parentKey) - - const childValue = materializeIncludedValue( - state, - state.childRegistry.get(routingKey), - ) - setIncludedValue(parentResult, state.resultPath, childValue) - - // Parent rows may already be materialized in the live collection by the - // time includes state is flushed, so update the stored row as well. - const storedParent = parentCollection.get(parentKey as any) - if (storedParent && storedParent !== parentResult) { - setIncludedValue(storedParent, state.resultPath, childValue) - } - } - } - } - } - - // Track affected correlation keys for inline materializations before clearing child changes. - const affectedCorrelationKeys = materializesInline(state) - ? new Set(state.pendingChildChanges.keys()) - : null - - // Phase 2: Child changes — apply to child Collections - // Track which entries had child changes and capture their childChanges maps - const entriesWithChildChanges = new Map< - unknown, - { entry: ChildCollectionEntry; childChanges: Map> } - >() - if (state.pendingChildChanges.size > 0) { - for (const [correlationKey, childChanges] of state.pendingChildChanges) { - // Ensure child Collection exists for this correlation key - let entry = state.childRegistry.get(correlationKey) - if (!entry) { - entry = createChildCollectionEntry( - parentId, - state.fieldName, - correlationKey, - state.hasOrderBy, - state.nestedSetups, - ) - state.childRegistry.set(correlationKey, entry) - } - - if (state.materialization === `collection`) { - attachChildCollectionToParent( - parentCollection, - state.resultPath, - correlationKey, - state.correlationToParentKeys, - entry.collection, - ) - } - - // Apply child changes to the child Collection - if (entry.syncMethods) { - entry.syncMethods.begin() - for (const [childKey, change] of childChanges) { - entry.resultKeys.set(change.value, childKey) - if (entry.orderByIndices && change.orderByIndex !== undefined) { - entry.orderByIndices.set(change.value, change.orderByIndex) - } - if (change.inserts > 0 && change.deletes === 0) { - entry.syncMethods.write({ value: change.value, type: `insert` }) - } else if ( - change.inserts > change.deletes || - (change.inserts === change.deletes && - entry.syncMethods.collection.has( - entry.syncMethods.collection.getKeyFromItem(change.value), - )) - ) { - entry.syncMethods.write({ value: change.value, type: `update` }) - } else if (change.deletes > 0) { - entry.syncMethods.write({ value: change.value, type: `delete` }) - } - } - entry.syncMethods.commit() - } - - // Update routing index for nested includes - updateRoutingIndex(state, correlationKey, childChanges) - - entriesWithChildChanges.set(correlationKey, { entry, childChanges }) - } - state.pendingChildChanges.clear() - } - - // Phase 3: Drain nested buffers — route buffered grandchild changes to per-entry states - const dirtyFromBuffers = drainNestedBuffers(state) - - // Phase 4: Flush per-entry states - // First: entries that had child changes in Phase 2 - for (const [, { entry, childChanges }] of entriesWithChildChanges) { - if (entry.includesStates) { - flushIncludesState( - entry.includesStates, - entry.collection, - entry.collection.id, - childChanges, - entry.syncMethods, - ) - } - } - // Then: entries that only had buffer-routed changes (no child changes at this level) - for (const correlationKey of dirtyFromBuffers) { - if (entriesWithChildChanges.has(correlationKey)) continue - const entry = state.childRegistry.get(correlationKey) - if (entry?.includesStates) { - flushIncludesState( - entry.includesStates, - entry.collection, - entry.collection.id, - null, - entry.syncMethods, - ) - } - } - // Finally: entries with deep nested buffer changes (grandchild-or-deeper buffers - // have pending data, but neither this level nor the immediate child level changed). - // Without this pass, changes at depth 3+ are stranded because drainNestedBuffers - // only drains one level and Phase 4 only flushes entries dirty from Phase 2/3. - const deepBufferDirty = new Set() - if (state.nestedSetups) { - for (const [correlationKey, entry] of state.childRegistry) { - if (entriesWithChildChanges.has(correlationKey)) continue - if (dirtyFromBuffers.has(correlationKey)) continue - if ( - entry.includesStates && - hasPendingIncludesChanges(entry.includesStates) - ) { - flushIncludesState( - entry.includesStates, - entry.collection, - entry.collection.id, - null, - entry.syncMethods, - ) - deepBufferDirty.add(correlationKey) - } - } - } - - // For inline materializations: re-emit affected parents with updated snapshots. - // We mutate items in-place (so collection.get() reflects changes immediately) - // and emit UPDATE events directly. We bypass the sync methods because - // commitPendingTransactions compares previous vs new visible state using - // deepEquals, but in-place mutation means both sides reference the same - // object, so the comparison always returns true and suppresses the event. - const inlineReEmitKeys = materializesInline(state) - ? new Set([ - ...(affectedCorrelationKeys || []), - ...dirtyFromBuffers, - ...deepBufferDirty, - ]) - : null - if (parentSyncMethods && inlineReEmitKeys && inlineReEmitKeys.size > 0) { - const events: Array> = [] - for (const correlationKey of inlineReEmitKeys) { - const parentKeys = state.correlationToParentKeys.get(correlationKey) - if (!parentKeys) continue - const entry = state.childRegistry.get(correlationKey) - for (const parentKey of parentKeys) { - const item = parentCollection.get(parentKey as any) - if (item) { - // Capture previous value before in-place mutation - const previousValue = cloneForIncludesUpdate(item, state.resultPath) - setIncludedValue( - item, - state.resultPath, - materializeIncludedValue(state, entry), - ) - const nextValue = cloneForIncludesUpdate(item, state.resultPath) - events.push({ - type: `update`, - key: parentKey as any, - value: nextValue, - previousValue, - }) - } - } - } - if (events.length > 0) { - // Emit directly — the in-place mutation already updated the data in - // syncedData, so we only need to notify subscribers. - const changesManager = (parentCollection as any)._changes as { - emitEvents: ( - changes: Array>, - forceEmit?: boolean, - ) => void - } - changesManager.emitEvents(events, true) - } - } - - // Phase 5: Parent DELETEs — dispose child Collections and clean up - if (parentChanges) { - for (const [parentKey, changes] of parentChanges) { - if (changes.deletes > 0 && changes.inserts === 0) { - const routing = changes.value[INCLUDES_ROUTING]?.[state.fieldName] - const correlationKey = routing?.correlationKey - const parentContext = routing?.parentContext ?? null - const routingKey = computeRoutingKey(correlationKey, parentContext) - if (correlationKey != null) { - // Clean up reverse index first, only delete child collection - // when the last parent referencing it is removed - const parentKeys = state.correlationToParentKeys.get(routingKey) - if (parentKeys) { - parentKeys.delete(parentKey) - if (parentKeys.size === 0) { - cleanRoutingIndexOnDelete(state, routingKey) - state.childRegistry.delete(routingKey) - state.correlationToParentKeys.delete(routingKey) - } - } - } - } - } - } - } - - // Clean up the internal routing stamp from parent/child results - if (parentChanges) { - for (const [, changes] of parentChanges) { - delete changes.value[INCLUDES_ROUTING] - } - } -} - -/** - * Checks whether any includes state has pending changes that need to be flushed. - * Checks direct pending child changes and shared nested buffers. - */ -function hasPendingIncludesChanges( - states: Array, -): boolean { - for (const state of states) { - if (state.pendingChildChanges.size > 0) return true - if (state.nestedSetups && hasNestedBufferChanges(state.nestedSetups)) - return true - } - return false -} - -/** - * Attaches a child Collection to parent rows that match a given correlation key. - * Uses the reverse index to look up parent keys directly instead of scanning. - */ -function attachChildCollectionToParent( - parentCollection: Collection, - resultPath: Array, - correlationKey: unknown, - correlationToParentKeys: Map>, - childCollection: Collection, -): void { - const parentKeys = correlationToParentKeys.get(correlationKey) - if (!parentKeys) return - - for (const parentKey of parentKeys) { - const item = parentCollection.get(parentKey as any) - if (item) { - setIncludedValue(item, resultPath, childCollection) - } - } -} - -function setIncludedValue( - target: Record, - path: Array, - value: unknown, -): void { - const state = getFnSelectState(target) - if (!state) { - setNestedValue(target, path, value) - return - } - - setNestedValue(state.sourceRow, path, value) - refreshFnSelectResult(target, state) -} - -function getFnSelectState(target: Record): - | { - sourceRow: Record - fnSelect: (row: Record) => any - } - | undefined { - return (target as Record)[FN_SELECT_STATE] as - | { - sourceRow: Record - fnSelect: (row: Record) => any - } - | undefined -} - -function refreshFnSelectResult( - target: Record, - state: { - sourceRow: Record - fnSelect: (row: Record) => any - }, -): void { - const targetRecord = target as Record - const sourceRecord = state.sourceRow as Record - const routing = - targetRecord[INCLUDES_ROUTING] ?? sourceRecord[INCLUDES_ROUTING] - const nextValue = state.fnSelect(state.sourceRow) - if (!nextValue || typeof nextValue !== `object`) { - return - } - - for (const key of Object.keys(target)) { - delete target[key] - } - Object.assign(target, nextValue) - - if (routing) { - targetRecord[INCLUDES_ROUTING] = routing - } - Object.defineProperty(target, FN_SELECT_STATE, { - value: state, - enumerable: true, - configurable: true, - }) -} - -function setNestedValue( - target: Record, - path: Array, - value: unknown, -): void { - if (path.length === 0) { - return - } - - let cursor = target - for (let i = 0; i < path.length - 1; i++) { - const segment = path[i]! - const next = cursor[segment] - if (next == null || typeof next !== `object`) { - cursor[segment] = {} - } - cursor = cursor[segment] - } - cursor[path[path.length - 1]!] = value -} - -function cloneForIncludesUpdate>( - target: T, - path: Array, -): T { - return getFnSelectState(target) - ? { ...target } - : clonePathForUpdate(target, path) -} - -function clonePathForUpdate>( - target: T, - path: Array, -): T { - const root = { ...target } - let sourceCursor: any = target - let cloneCursor: any = root - - for (let i = 0; i < path.length - 1; i++) { - const segment = path[i]! - const sourceValue = sourceCursor?.[segment] - if (sourceValue == null || typeof sourceValue !== `object`) { - return root - } - - const clonedValue = Array.isArray(sourceValue) - ? [...sourceValue] - : { ...sourceValue } - cloneCursor[segment] = clonedValue - sourceCursor = sourceValue - cloneCursor = clonedValue - } - - return root -} - function accumulateChanges( acc: Map>, [[key, tupleData], multiplicity]: [ @@ -2009,6 +1389,10 @@ function accumulateChanges( } if (multiplicity < 0) { changes.deletes += Math.abs(multiplicity) + // Remember the retracted (old) value + position so the flush can tell an + // order-only move apart from a real value change. + changes.previousValue = value + changes.previousOrderByIndex = orderByIndex } else if (multiplicity > 0) { changes.inserts += multiplicity // Update value to the latest version for this key @@ -2020,3 +1404,33 @@ function accumulateChanges( acc.set(key, changes) return acc } + +/** + * Decide whether a flush contains an order-only move. + * + * An "order-only move" — a row updated in place whose `orderByIndex` moved but + * whose projected value is deep-equal to before — is swallowed by the value-diff + * and needs an explicit layout notification. The collection coalesces that + * signal with any ordinary row publication per subscriber. + */ +function hasOrderOnlyMove( + changesToApply: Map>, +): boolean { + for (const changes of changesToApply.values()) { + const isUpdate = changes.inserts > 0 && changes.deletes > 0 + if ( + isUpdate && + changes.previousValue !== undefined && + deepEquals(changes.previousValue, changes.value) && + changes.orderByIndex !== changes.previousOrderByIndex + ) { + return true + } + } + return false +} + +/** Mark the collection's next commit as layout-changing. */ +function markLayoutChange(collection: { _markLayoutChange: () => void }): void { + collection._markLayoutChange() +} diff --git a/packages/db/src/query/live/collection-subscriber.ts b/packages/db/src/query/live/collection-subscriber.ts index 69c8220ad3..4b1bcb9c72 100644 --- a/packages/db/src/query/live/collection-subscriber.ts +++ b/packages/db/src/query/live/collection-subscriber.ts @@ -1,18 +1,18 @@ +import { normalizeExpressionPaths } from '../compiler/expressions.js' +import { OrderedSourceLoader } from './ordered-source-loader.js' import { - normalizeExpressionPaths, - normalizeOrderByPaths, -} from '../compiler/expressions.js' -import { - computeOrderedLoadCursor, computeSubscriptionOrderByHints, - filterDuplicateInserts, + reconcileChangesForD2, sendChangesToInput, splitUpdates, - trackBiggestSentValue, } from './utils.js' +import { SubsetDemandController } from './subset-demand-controller.js' import type { Collection } from '../../collection/index.js' import type { ChangeMessage, + LoadSubsetRequestResult, + SubscribeChangesOptions, + SubscriptionLoadSubsetErrorEvent, SubscriptionStatusChangeEvent, } from '../../types.js' import type { Context, GetResult } from '../builder/types.js' @@ -20,48 +20,43 @@ import type { BasicExpression } from '../ir.js' import type { OrderByOptimizationInfo } from '../compiler/order-by.js' import type { CollectionConfigBuilder } from './collection-config-builder.js' import type { CollectionSubscription } from '../../collection/subscription.js' +import type { LazyDemandPlan } from '../compiler/joins.js' const loadMoreCallbackSymbol = Symbol.for( `@tanstack/db.collection-config-builder`, ) +type TruncateReplayPublicationControl = NonNullable< + SubscribeChangesOptions[`truncateReplayPublication`] +> + export class CollectionSubscriber< TContext extends Context, TResult extends object = GetResult, > { - // Keep track of the biggest value we've sent so far (needed for orderBy optimization) - private biggest: any = undefined - - // Track the most recent ordered load request key (cursor + window). - // This avoids infinite loops from cached data re-writes while still allowing - // window moves or new keys at the same cursor value to trigger new requests. - private lastLoadRequestKey: string | undefined - // Track deferred promises for subscription loading states private subscriptionLoadingPromises = new Map< CollectionSubscription, { resolve: () => void } >() - // Track keys that have been sent to the D2 pipeline to prevent duplicate inserts - // This is necessary because different code paths (initial load, change events) - // can potentially send the same item to D2 multiple times. - private sentToD2Keys = new Set() + // Exact row last contributed to D2 for each source key. + private sentToD2Rows = new Map>() // Direct load tracking callback for ordered path (set during subscribeToOrderedChanges, // used by loadNextItems for subsequent requestLimitedSnapshot calls) - private orderedLoadSubsetResult?: (result: Promise | true) => void - private pendingOrderedLoadPromise: Promise | undefined + private orderedLoader: OrderedSourceLoader | undefined + private readonly demand = new SubsetDemandController() constructor( + private sourceId: string, private alias: string, - private collectionId: string, private collection: Collection, private collectionConfigBuilder: CollectionConfigBuilder, ) {} subscribe(): CollectionSubscription { - const whereClause = this.getWhereClauseForAlias() + const whereClause = this.getWhereClause() if (whereClause) { const whereExpression = normalizeExpressionPaths(whereClause, this.alias) @@ -77,11 +72,16 @@ export class CollectionSubscriber< // Direct load promise tracking: pipes loadSubset results straight to the // live query collection, avoiding the multi-hop deferred promise chain that // can break under microtask timing (e.g., queueMicrotask in TanStack Query). - const trackLoadResult = (result: Promise | true) => { + const trackLoadResult = (result: LoadSubsetRequestResult) => { if (result instanceof Promise) { - this.collectionConfigBuilder.liveQueryCollection!._sync.trackLoadPromise( - result, - ) + // Defer the tracked rejection by one microtask so the subscription's + // error event can put an initial live query in error before loading + // state would otherwise let it become ready. + const trackedResult = result.catch(async (error: unknown) => { + await Promise.resolve() + throw error + }) + this.collectionConfigBuilder.trackSubsetLoadPromise(trackedResult) } } @@ -90,6 +90,7 @@ export class CollectionSubscriber< // Used as a fallback for status transitions not covered by direct tracking // (e.g., truncate-triggered reloads that call trackLoadSubsetPromise directly). const onStatusChange = (event: SubscriptionStatusChangeEvent) => { + if (this.collectionConfigBuilder.isLazySource(this.sourceId)) return const subscription = event.subscription as CollectionSubscription if (event.status === `loadingSubset`) { this.ensureLoadingPromise(subscription) @@ -102,6 +103,16 @@ export class CollectionSubscriber< } } } + const onLoadSubsetError = (event: SubscriptionLoadSubsetErrorEvent) => { + this.collectionConfigBuilder.recordSubsetError( + event.error, + // Lazy demand owns its fatal-error path. For eager sources, one + // successful page does not finish initial ordered refinement. + !this.collectionConfigBuilder.isLazySource(this.sourceId) && + this.collectionConfigBuilder.liveQueryCollection?.status === + `loading`, + ) + } // Create subscription with onStatusChange - listener is registered before any async work let subscription: CollectionSubscription @@ -111,26 +122,40 @@ export class CollectionSubscriber< orderByInfo, onStatusChange, trackLoadResult, + onLoadSubsetError, ) } else { - // If the source alias is lazy then we should not include the initial state - const includeInitialState = !this.collectionConfigBuilder.isLazyAlias( - this.alias, - ) + // Lazy sources load only the subsets demanded by the compiled graph. + const includeInitialState = + (this.collection.config.syncMode !== `on-demand` || + this.collectionConfigBuilder.query.limit !== 0) && + !this.collectionConfigBuilder.isLazySource(this.sourceId) subscription = this.subscribeToMatchingChanges( whereExpression, includeInitialState, onStatusChange, + trackLoadResult, + onLoadSubsetError, ) + this.registerSubscriptionCleanup(subscription) } // Check current status after subscribing - if status is 'loadingSubset', track it. // The onStatusChange listener will catch the transition to 'ready'. - if (subscription.status === `loadingSubset`) { + if ( + !this.collectionConfigBuilder.isLazySource(this.sourceId) && + subscription.status === `loadingSubset` + ) { this.ensureLoadingPromise(subscription) } + return subscription + } + + private registerSubscriptionCleanup( + subscription: CollectionSubscription, + ): void { const unsubscribe = () => { // If subscription has a pending promise, resolve it before unsubscribing const deferred = this.subscriptionLoadingPromises.get(subscription) @@ -139,31 +164,74 @@ export class CollectionSubscriber< deferred.resolve() } - subscription.unsubscribe() + try { + this.demand.clear() + } finally { + subscription.unsubscribe() + } } // currentSyncState is always defined when subscribe() is called // (called during sync session setup) this.collectionConfigBuilder.currentSyncState!.unsubscribeCallbacks.add( unsubscribe, ) - return subscription + } + + setDemand( + subscription: CollectionSubscription, + plan: LazyDemandPlan, + keys: Set, + ): void { + let update + try { + update = this.demand.setDemand(subscription, plan, keys) + } catch (error) { + // CollectionSubscription reports adapter failures before rethrowing. + // Convert that synchronous form to the same query-local fatal demand + // state as a rejected load, without letting it escape the source commit. + // Preserve unrelated graph/programming errors as throws. + if (!Object.is(subscription.lastError, error)) throw error + const isInitialSync = + this.collectionConfigBuilder.liveQueryCollection?.status === `loading` + const generation = this.collectionConfigBuilder.beginDemand(plan.id) + this.collectionConfigBuilder.failDemand(plan.id, generation, error) + if (isInitialSync) throw error + return + } + if (!update.changed) return + + if (update.empty) { + this.collectionConfigBuilder.retireDemand(plan.id) + return + } + + const generation = this.collectionConfigBuilder.beginDemand(plan.id) + if (update.ready instanceof Promise) { + this.collectionConfigBuilder.trackSubsetLoadOperationPromise(update.ready) + void update.ready.then( + () => this.collectionConfigBuilder.settleDemand(plan.id, generation), + (error) => + this.collectionConfigBuilder.failDemand(plan.id, generation, error), + ) + } else { + this.collectionConfigBuilder.settleDemand(plan.id, generation) + } } private sendChangesToPipeline( changes: Iterable>, - callback?: () => boolean, + callback?: () => void, ) { const changesArray = Array.isArray(changes) ? changes : [...changes] - const filteredChanges = filterDuplicateInserts( + const reconciledChanges = reconcileChangesForD2( changesArray, - this.sentToD2Keys, + this.sentToD2Rows, ) - // currentSyncState and input are always defined when this method is called // (only called from active subscriptions during a sync session) const input = - this.collectionConfigBuilder.currentSyncState!.inputs[this.alias]! - const sentChanges = sendChangesToInput(input, filteredChanges) + this.collectionConfigBuilder.currentSyncState!.inputs[this.sourceId]! + const sentChanges = sendChangesToInput(input, reconciledChanges) // Do not provide the callback that loads more data // if there's no more data to load @@ -173,15 +241,15 @@ export class CollectionSubscriber< // We need to schedule a graph run even if there's no data to load // because we need to mark the collection as ready if it's not already // and that's only done in `scheduleGraphRun` - this.collectionConfigBuilder.scheduleGraphRun(dataLoader, { - alias: this.alias, - }) + this.collectionConfigBuilder.scheduleGraphRun(dataLoader) } private subscribeToMatchingChanges( whereExpression: BasicExpression | undefined, includeInitialState: boolean, onStatusChange: (event: SubscriptionStatusChangeEvent) => void, + onLoadSubsetResult: (result: LoadSubsetRequestResult) => void, + onLoadSubsetError: (event: SubscriptionLoadSubsetErrorEvent) => void, ): CollectionSubscription { const sendChanges = ( changes: Array>, @@ -198,23 +266,15 @@ export class CollectionSubscriber< // Track loading via the loadSubset promise directly. // requestSnapshot uses trackLoadSubsetPromise: false (needed for truncate handling), // so we use onLoadSubsetResult to get the promise and track it ourselves. - const onLoadSubsetResult = includeInitialState - ? (result: Promise | true) => { - if (result instanceof Promise) { - this.collectionConfigBuilder.liveQueryCollection!._sync.trackLoadPromise( - result, - ) - } - } - : undefined - const subscription = this.collection.subscribeChanges(sendChanges, { ...(includeInitialState && { includeInitialState }), whereExpression, onStatusChange, + onLoadSubsetError, + truncateReplayPublication: this.truncateReplayPublicationControl(), orderBy: hints.orderBy, limit: hints.limit, - onLoadSubsetResult, + onLoadSubsetResult: includeInitialState ? onLoadSubsetResult : undefined, }) return subscription @@ -224,43 +284,24 @@ export class CollectionSubscriber< whereExpression: BasicExpression | undefined, orderByInfo: OrderByOptimizationInfo, onStatusChange: (event: SubscriptionStatusChangeEvent) => void, - onLoadSubsetResult: (result: Promise | true) => void, + onLoadSubsetResult: (result: LoadSubsetRequestResult) => void, + onLoadSubsetError: (event: SubscriptionLoadSubsetErrorEvent) => void, ): CollectionSubscription { - const { orderBy, offset, limit, index } = orderByInfo - - // Store the callback so loadNextItems can also use direct tracking. - // Track in-flight ordered loads to avoid issuing redundant requests while - // a previous snapshot is still pending. - const handleLoadSubsetResult = (result: Promise | true) => { - if (result instanceof Promise) { - this.pendingOrderedLoadPromise = result - result.finally(() => { - if (this.pendingOrderedLoadPromise === result) { - this.pendingOrderedLoadPromise = undefined - } - }) - } - onLoadSubsetResult(result) - } - - this.orderedLoadSubsetResult = handleLoadSubsetResult - // Use a holder to forward-reference subscription in the callback const subscriptionHolder: { current?: CollectionSubscription } = {} const sendChangesInRange = ( changes: Iterable>, ) => { + const subscription = subscriptionHolder.current + if (!subscription) return const changesArray = Array.isArray(changes) ? changes : [...changes] - this.trackSentValues(changesArray, orderByInfo.comparator) + this.orderedLoader?.onSourceChanges(changesArray, this.sentToD2Rows) // Split live updates into a delete of the old value and an insert of the new value const splittedChanges = splitUpdates(changesArray) - this.sendChangesToPipelineWithTracking( - splittedChanges, - subscriptionHolder.current!, - ) + this.sendChangesToPipelineWithTracking(splittedChanges, subscription) } // Subscribe to changes with onStatusChange - listener is registered before any snapshot @@ -268,87 +309,106 @@ export class CollectionSubscriber< const subscription = this.collection.subscribeChanges(sendChangesInRange, { whereExpression, onStatusChange, + onLoadSubsetError, + truncateReplayPublication: this.truncateReplayPublicationControl(() => { + // Recovery favors a simple, authoritative rebuild over resuming a + // fragile cursor. The retained full-source demand is replayed on later + // truncates, so this adds at most one demand per subscription. + // Queue startup inside the publication barrier too: a synchronous + // throw establishes no acquisition for the replay to wait on. + const loader = this.orderedLoader + this.collectionConfigBuilder.trackOrderedLoadPromise( + Promise.resolve().then(() => loader?.loadFullSource()), + true, + ) + }), }) subscriptionHolder.current = subscription + this.registerSubscriptionCleanup(subscription) - // Listen for truncate events to reset cursor tracking state and sentToD2Keys - // This ensures that after a must-refetch/truncate, we don't use stale cursor data - // and allow re-inserts of previously sent keys + // Reset ordered-load state on truncate. Keep exact D2 rows until the + // replacement publication retracts or replaces them. const truncateUnsubscribe = this.collection.on(`truncate`, () => { - this.biggest = undefined - this.lastLoadRequestKey = undefined - this.pendingOrderedLoadPromise = undefined - this.sentToD2Keys.clear() + this.orderedLoader?.resetCursor() }) // Clean up truncate listener when subscription is unsubscribed subscription.on(`unsubscribed`, () => { truncateUnsubscribe() + subscriptionHolder.current = undefined + this.orderedLoader?.dispose() + this.orderedLoader = undefined }) - // Normalize the orderBy clauses such that the references are relative to the collection - const normalizedOrderBy = normalizeOrderByPaths(orderBy, this.alias) - - // Trigger the snapshot request — use direct load tracking (trackLoadSubsetPromise: false) - // to pipe the loadSubset result straight to the live query collection. This bypasses - // the subscription status → onStatusChange → deferred promise chain which is fragile - // under microtask timing (e.g., queueMicrotask delays in TanStack Query observers). - if (index) { - // We have an index on the first orderBy column - use lazy loading optimization - subscription.setOrderByIndex(index) - - subscription.requestLimitedSnapshot({ - limit: offset + limit, - orderBy: normalizedOrderBy, - trackLoadSubsetPromise: false, - onLoadSubsetResult: handleLoadSubsetResult, - }) - } else { - // No index available (e.g., non-ref expression): pass orderBy/limit to loadSubset - subscription.requestSnapshot({ - orderBy: normalizedOrderBy, - limit: offset + limit, - trackLoadSubsetPromise: false, - onLoadSubsetResult: handleLoadSubsetResult, - }) - } + this.orderedLoader = new OrderedSourceLoader( + orderByInfo, + subscription, + this.alias, + (result, holdPublication) => { + if (result instanceof Promise) { + this.collectionConfigBuilder.trackOrderedLoadPromise( + result, + holdPublication && !subscription.hasPendingTruncateReplacement, + ) + } + onLoadSubsetResult(result) + }, + () => + this.collectionConfigBuilder.liveQueryCollection?.status === `ready` && + !this.collectionConfigBuilder.hasActiveWindowOperation(), + ) + this.orderedLoader.start() return subscription } + private truncateReplayPublicationControl( + onStart?: () => void, + ): TruncateReplayPublicationControl { + const syncSession = this.collectionConfigBuilder.getSyncSession() + return { + start: () => { + onStart?.() + }, + succeed: () => { + if (syncSession !== this.collectionConfigBuilder.getSyncSession()) { + return + } + this.orderedLoader?.settleFullSourceReplay() + this.collectionConfigBuilder.scheduleGraphRunForSession(syncSession) + }, + } + } + // This function is called by maybeRunGraph // after each iteration of the query pipeline // to ensure that the orderBy operator has enough data to work with - loadMoreIfNeeded(subscription: CollectionSubscription) { + loadMoreIfNeeded(subscription: CollectionSubscription): void { + if ( + subscription.hasPendingTruncateReplacement && + !this.collectionConfigBuilder.hasActiveWindowOperation() + ) { + return + } + const orderByInfo = this.getOrderByInfo() if (!orderByInfo) { // This query has no orderBy operator // so there's no data to load - return true - } - - const { dataNeeded, index } = orderByInfo - - if (!dataNeeded || !index) { - // dataNeeded is not set when there's no index (e.g., non-ref expression - // or auto-indexing is disabled). Without an index, lazy loading can't work — - // all data was already loaded eagerly via requestSnapshot. - return true - } - - if (this.pendingOrderedLoadPromise) { - // Wait for in-flight ordered loads to resolve before issuing another request. - return true + return } - // `dataNeeded` probes the orderBy operator to see if it needs more data - // if it needs more data, it returns the number of items it needs - const n = dataNeeded() - if (n > 0) { - this.loadNextItems(n, subscription) + try { + const pending = this.orderedLoader?.loadMore( + this.collectionConfigBuilder.getActiveWindowOperationGeneration(), + ) + if (pending) { + this.collectionConfigBuilder.trackSubsetLoadOperationPromise(pending) + } + } catch (error) { + if (!Object.is(subscription.lastError, error)) throw error } - return true } private sendChangesToPipelineWithTracking( @@ -365,7 +425,7 @@ export class CollectionSubscriber< // This ensures we pass the same function instance to the scheduler each time, // allowing it to deduplicate callbacks when multiple changes arrive during a transaction. type SubscriptionWithLoader = CollectionSubscription & { - [loadMoreCallbackSymbol]?: () => boolean + [loadMoreCallbackSymbol]?: () => void } const subscriptionWithLoader = subscription as SubscriptionWithLoader @@ -379,73 +439,24 @@ export class CollectionSubscriber< ) } - // Loads the next `n` items from the collection - // starting from the biggest item it has sent - private loadNextItems(n: number, subscription: CollectionSubscription) { - const orderByInfo = this.getOrderByInfo() - if (!orderByInfo) { - return - } - - const cursor = computeOrderedLoadCursor( - orderByInfo, - this.biggest, - this.lastLoadRequestKey, - this.alias, - n, - ) - if (!cursor) return // Duplicate request — skip - - this.lastLoadRequestKey = cursor.loadRequestKey - - // Take the `n` items after the biggest sent value - // Omit offset so requestLimitedSnapshot can advance based on - // the number of rows already loaded (supports offset-based backends). - subscription.requestLimitedSnapshot({ - orderBy: cursor.normalizedOrderBy, - limit: n, - minValues: cursor.minValues, - trackLoadSubsetPromise: false, - onLoadSubsetResult: this.orderedLoadSubsetResult, - }) - } - - private getWhereClauseForAlias(): BasicExpression | undefined { + private getWhereClause(): BasicExpression | undefined { const sourceWhereClausesCache = this.collectionConfigBuilder.sourceWhereClausesCache if (!sourceWhereClausesCache) { return undefined } - return sourceWhereClausesCache.get(this.alias) + return sourceWhereClausesCache.get(this.sourceId) } private getOrderByInfo(): OrderByOptimizationInfo | undefined { const info = - this.collectionConfigBuilder.optimizableOrderByCollections[ - this.collectionId - ] - if (info && info.alias === this.alias) { + this.collectionConfigBuilder.optimizableOrderByCollections[this.sourceId] + if (info?.sourceId === this.sourceId) { return info } return undefined } - private trackSentValues( - changes: Array>, - comparator: (a: any, b: any) => number, - ): void { - const result = trackBiggestSentValue( - changes, - this.biggest, - this.sentToD2Keys, - comparator, - ) - this.biggest = result.biggest - if (result.shouldResetLoadKey) { - this.lastLoadRequestKey = undefined - } - } - private ensureLoadingPromise(subscription: CollectionSubscription) { if (this.subscriptionLoadingPromises.has(subscription)) { return @@ -459,8 +470,6 @@ export class CollectionSubscriber< this.subscriptionLoadingPromises.set(subscription, { resolve: resolve!, }) - this.collectionConfigBuilder.liveQueryCollection!._sync.trackLoadPromise( - promise, - ) + this.collectionConfigBuilder.trackSubsetLoadPromise(promise) } } diff --git a/packages/db/src/query/live/materialized-pipeline.ts b/packages/db/src/query/live/materialized-pipeline.ts new file mode 100644 index 0000000000..d7f76e79c5 --- /dev/null +++ b/packages/db/src/query/live/materialized-pipeline.ts @@ -0,0 +1,521 @@ +import { + compareKeys, + distinct, + filter, + join, + map, + reduce, + serializeValue, +} from '@tanstack/db-ivm' +import { deepEquals } from '../../utils.js' +import { getParentContextIdentity } from '../equality-value-identity.js' +import { INCLUDES_ROUTING } from '../compiler/route-metadata.js' +import type { ValueIdentity } from '../equality-value-identity.js' +import type { + CompilationResult, + IncludesCompilationResult, +} from '../compiler/index.js' +import type { IncludesMaterialization } from '../ir.js' +import type { IStreamBuilder } from '@tanstack/db-ivm' +import type { ResultStream } from '../../types.js' + +type ResultTuple = [ + value: Record, + order: string | undefined, + correlationKey?: unknown, + parentContext?: Record | null, + routing?: IncludesRouting, + publicKey?: unknown, +] + +type IncludesRouting = Record + +type IncludeRoute = { + active: boolean + correlationKey: unknown + parentContext: Record | null +} + +type CanonicalResult = { + publicKey: unknown + tuple: ResultTuple +} + +export type BucketRow = { + publicKey: unknown + value: Record + order: string | undefined +} + +export type BucketFacadeCompilation = { + edgeId: string + rows: IStreamBuilder<[string, BucketRow]> + activeBuckets: IStreamBuilder<[string, true]> + hasOrderBy: boolean +} + +export const BUCKET_FACADE_REF = Symbol(`bucketFacadeRef`) + +export type BucketFacadeRef = { + [BUCKET_FACADE_REF]: { + edgeId: string + bucketKey: string + } +} + +export type MaterializedCompilation = { + pipeline: ResultStream + facades: Array +} + +type RelationScope = `root` | `child` + +type BuiltRelations = WeakMap< + CompilationResult, + Partial> +> + +let nextBucketFacadeEdgeId = 0 + +/** + * Compiles inline includes into the same D2 graph as their parent relation. + * Collection-valued includes become inert bucket references. The public facade + * adapter resolves those references after the graph reaches quiescence. + */ +export function materializeCompilation( + compilation: CompilationResult, + getRootKey?: (row: any) => unknown, + reduceJoinedPublicKeys = false, +): MaterializedCompilation { + if ( + !compilation.includes?.length && + !(getRootKey && reduceJoinedPublicKeys) + ) { + return { pipeline: compilation.pipeline, facades: [] } + } + + const built: BuiltRelations = new WeakMap() + const materialized = materializeRelation( + compilation, + getRootKey, + built, + `root`, + ) + return { + ...materialized, + facades: dedupeFacades(materialized.facades), + } +} + +function materializeRelation( + compilation: CompilationResult, + getKey: ((row: any) => unknown) | undefined, + built: BuiltRelations, + scope: RelationScope, +): MaterializedCompilation { + const cached = built.get(compilation)?.[scope] + if (cached) return cached + + let pipeline = canonicalizeByPublicKey( + exposeRouting(compilation.pipeline), + getKey, + scope, + compilation.valueIdentity, + ) + const facades: Array = [] + + for (const include of compilation.includes ?? []) { + const child = materializeRelation( + include.childCompilationResult, + undefined, + built, + `child`, + ) + facades.push(...child.facades) + + const bucketRows = createBucketRows( + child.pipeline, + include.childCompilationResult.valueIdentity, + ) + if (include.materialization === `collection`) { + const edgeId = `bucket-facade-${++nextBucketFacadeEdgeId}` + const activeBuckets = createActiveBuckets( + pipeline, + include, + compilation.valueIdentity, + ) + const activeBucketRows = activeBuckets.pipe( + join(bucketRows), + map(([bucketKey, [, row]]) => [bucketKey, row]), + ) as IStreamBuilder<[string, BucketRow]> + facades.push({ + edgeId, + rows: activeBucketRows, + activeBuckets, + hasOrderBy: include.hasOrderBy, + }) + pipeline = attachCollectionInclude( + pipeline, + include, + edgeId, + scope, + compilation.valueIdentity, + ) + } else { + pipeline = attachInlineInclude( + pipeline, + bucketRows, + include, + scope, + compilation.valueIdentity, + ) + } + } + + const result = { pipeline, facades } + built.set(compilation, { ...built.get(compilation), [scope]: result }) + return result +} + +function dedupeFacades( + facades: Array, +): Array { + return [...new Map(facades.map((facade) => [facade.edgeId, facade])).values()] +} + +function exposeRouting(pipeline: ResultStream): ResultStream { + return pipeline.pipe( + map(([key, rawTuple]) => { + const tuple = rawTuple as ResultTuple + return [ + key, + [ + tuple[0], + tuple[1], + tuple[2], + tuple[3], + tuple[0][INCLUDES_ROUTING], + tuple[4], + ], + ] + }), + ) as unknown as ResultStream +} + +function canonicalizeByPublicKey( + pipeline: ResultStream, + getKey: ((row: any) => unknown) | undefined, + scope: RelationScope, + valueIdentity: ValueIdentity, +): ResultStream { + return pipeline.pipe( + map(([internalKey, rawTuple]) => { + const tuple = rawTuple as ResultTuple + const publicKey = getKey ? getKey(tuple[0]) : (tuple[5] ?? internalKey) + const relationKey = + scope === `root` + ? serializeValue([`root`, publicKey]) + : serializeValue([ + routeKey(tuple[2], tuple[3], valueIdentity), + publicKey, + ]) + return [relationKey, { publicKey, tuple }] as [string, CanonicalResult] + }), + reduce((values: Array<[CanonicalResult, number]>) => { + const totalMultiplicity = values.reduce( + (total, [, multiplicity]) => total + multiplicity, + 0, + ) + if (totalMultiplicity === 0) return [] + if (totalMultiplicity < 0) { + throw new Error(`Canonical query row has negative multiplicity`) + } + + const visible = values.find(([, multiplicity]) => multiplicity > 0)?.[0] + if (!visible) { + throw new Error(`Canonical query row has no positive contributor`) + } + + for (const [candidate, multiplicity] of values) { + if (multiplicity <= 0) continue + assertCongruentContributors(visible, candidate) + } + + return [[visible, 1]] + }), + map(([relationKey, { publicKey, tuple }]) => [ + scope === `root` ? publicKey : relationKey, + tuple, + ]), + ) as ResultStream +} + +function assertCongruentContributors( + left: CanonicalResult, + right: CanonicalResult, +): void { + const [leftValue, leftOrder, leftCorrelation, leftContext, leftRouting] = + left.tuple + const [rightValue, rightOrder, rightCorrelation, rightContext, rightRouting] = + right.tuple + + if ( + leftOrder !== rightOrder || + !deepEquals(leftValue, rightValue) || + !deepEquals(leftCorrelation, rightCorrelation) || + !deepEquals(leftContext, rightContext) || + !deepEquals(leftRouting, rightRouting) + ) { + throw new Error( + `Query contributors for public key ${serializeValue(left.publicKey)} are not congruent`, + ) + } +} + +function attachInlineInclude( + parentPipeline: ResultStream, + bucketRows: IStreamBuilder<[string, BucketRow]>, + include: IncludesCompilationResult, + scope: RelationScope, + valueIdentity: ValueIdentity, +): ResultStream { + const bucketValues = bucketRows.pipe( + reduce((values: Array<[BucketRow, number]>) => { + const rows: Array = [] + for (const [row, multiplicity] of values) { + if (multiplicity < 0) { + throw new Error( + `Materialization bucket row has negative multiplicity`, + ) + } + for (let index = 0; index < multiplicity; index++) rows.push(row) + } + if (rows.length === 0) return [] + + rows.sort(compareBucketRows) + return [[{ value: materializeRows(rows, include) }, 1]] + }), + ) + const routedParents = parentPipeline.pipe( + map(([parentKey, rawTuple]) => { + const tuple = rawTuple as ResultTuple + const routing = getIncludeRoute(tuple, include.fieldName) + return [ + routing?.active !== true + ? `inactive:${serializeValue(parentKey)}` + : routeKey( + routing.correlationKey, + routing.parentContext, + valueIdentity, + ), + { parentKey, tuple }, + ] as [string, { parentKey: unknown; tuple: ResultTuple }] + }), + join(bucketValues, `left`), + map(([_bucketKey, [parent, bucketValue]]) => { + const [value, order, correlationKey, parentContext, routing, publicKey] = + parent!.tuple + const edgeRouting = routing?.[include.fieldName] + if (edgeRouting?.active !== true) { + return [ + parent!.parentKey, + [value, order, correlationKey, parentContext, routing, publicKey], + ] + } + const materialized = + bucketValue === null + ? emptyMaterializedValue(include.materialization) + : bucketValue.value + return [ + parent!.parentKey, + [ + setNestedValue(value, include.resultPath, materialized), + order, + correlationKey, + parentContext, + routing, + publicKey, + ], + ] + }), + ) + // A route move can make the join emit matched and empty-bucket deltas for + // the same parent key in one graph turn. Reduce those deltas back to the one + // canonical parent row before the next include or the public output sees it. + return canonicalizeByPublicKey( + routedParents as ResultStream, + undefined, + scope, + valueIdentity, + ) +} + +function createBucketRows( + childPipeline: ResultStream, + valueIdentity: ValueIdentity, +): IStreamBuilder<[string, BucketRow]> { + return childPipeline.pipe( + map(([internalKey, rawTuple]) => { + const [value, order, correlationKey, parentContext, , publicKey] = + rawTuple as ResultTuple + return [ + routeKey(correlationKey, parentContext, valueIdentity), + { publicKey: publicKey ?? internalKey, value, order }, + ] as [string, BucketRow] + }), + ) +} + +function attachCollectionInclude( + parentPipeline: ResultStream, + include: IncludesCompilationResult, + edgeId: string, + scope: RelationScope, + valueIdentity: ValueIdentity, +): ResultStream { + const routedParents = parentPipeline.pipe( + map(([parentKey, rawTuple]) => { + const tuple = rawTuple as ResultTuple + const routing = getIncludeRoute(tuple, include.fieldName) + if (routing?.active !== true) return [parentKey, tuple] + const facade = createBucketFacadeRef( + edgeId, + routeKey(routing.correlationKey, routing.parentContext, valueIdentity), + ) + return [ + parentKey, + [ + setNestedValue(tuple[0], include.resultPath, facade), + tuple[1], + tuple[2], + tuple[3], + tuple[4], + tuple[5], + ], + ] + }), + ) + return canonicalizeByPublicKey( + routedParents as ResultStream, + undefined, + scope, + valueIdentity, + ) +} + +function createActiveBuckets( + parentPipeline: ResultStream, + include: IncludesCompilationResult, + valueIdentity: ValueIdentity, +): IStreamBuilder<[string, true]> { + return parentPipeline.pipe( + map(([parentKey, rawTuple]) => { + const tuple = rawTuple as ResultTuple + const routing = getIncludeRoute(tuple, include.fieldName) + const bucketKey = + routing?.active === true + ? routeKey( + routing.correlationKey, + routing.parentContext, + valueIdentity, + ) + : undefined + return [parentKey, bucketKey] as [unknown, string | undefined] + }), + filter(([, bucketKey]) => bucketKey !== undefined), + distinct(([, bucketKey]) => bucketKey), + map(([, bucketKey]) => [bucketKey!, true] as [string, true]), + ) +} + +function createBucketFacadeRef( + edgeId: string, + bucketKey: string, +): BucketFacadeRef { + return { [BUCKET_FACADE_REF]: { edgeId, bucketKey } } +} + +function getIncludeRoute( + tuple: ResultTuple, + fieldName: string, +): IncludeRoute | undefined { + return tuple[4]?.[fieldName] +} + +function routeKey( + correlationKey: unknown, + parentContext: Record | null | undefined, + valueIdentity: ValueIdentity, +): string { + return serializeValue([ + valueIdentity.equality(correlationKey ?? null), + getParentContextIdentity(parentContext ?? null), + ]) +} + +function compareBucketRows(left: BucketRow, right: BucketRow): number { + if (left.order !== right.order) { + if (left.order === undefined) return 1 + if (right.order === undefined) return -1 + return left.order < right.order ? -1 : 1 + } + + if ( + (typeof left.publicKey === `string` || + typeof left.publicKey === `number`) && + (typeof right.publicKey === `string` || typeof right.publicKey === `number`) + ) { + return compareKeys(left.publicKey, right.publicKey) + } + + const leftKey = serializeValue(left.publicKey) + const rightKey = serializeValue(right.publicKey) + return leftKey < rightKey ? -1 : leftKey > rightKey ? 1 : 0 +} + +function materializeRows( + rows: Array, + include: IncludesCompilationResult, +): unknown { + const scalarField = include.scalarField + const values = scalarField + ? rows.map(({ value }) => value[scalarField]) + : rows.map(({ value }) => value) + + if (include.materialization === `array`) return values + if (include.materialization === `singleton`) return values[0] + return values.map((value) => String(value ?? ``)).join(``) +} + +function emptyMaterializedValue( + materialization: IncludesMaterialization, +): unknown { + if (materialization === `array`) return [] + if (materialization === `concat`) return `` + if (materialization === `singleton`) return undefined + throw new Error(`Collection includes require a bucket facade`) +} + +function setNestedValue( + source: Record, + path: Array, + value: unknown, +): Record { + const root = { ...source } + let target = root + let current: Record | null | undefined = source + + for (let index = 0; index < path.length - 1; index++) { + const part = path[index]! + const currentChild: any = current?.[part] + const next = Array.isArray(currentChild) + ? [...currentChild] + : { ...(currentChild ?? {}) } + target[part] = next + target = next + current = currentChild + } + + target[path[path.length - 1]!] = value + return root +} diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts new file mode 100644 index 0000000000..af680ec560 --- /dev/null +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -0,0 +1,653 @@ +import { + buildCursorCurrent, + canExpressCursorOrder, +} from '../../utils/cursor.js' +import { normalizeError } from '../../utils/error.js' +import { runAllCallbacks } from '../../utils/callbacks.js' +import { normalizeOrderByPaths } from '../compiler/expressions.js' +import type { + CollectionSubscription, + ReleaseLoadSubset, +} from '../../collection/subscription.js' +import type { + ChangeMessage, + LoadSubsetOptions, + LoadSubsetRequestResult, +} from '../../types.js' +import type { OrderByOptimizationInfo } from '../compiler/order-by.js' + +type OrderedRequestKind = `ordered` | `boundary` | `full-source` + +/** Owns the conservative provider-loading policy for one ordered source. */ +export class OrderedSourceLoader { + private pending: Promise | undefined + // Exact request settlement is not provider extent. This latch only records + // that some request once completed; reset may discard the boundary, and an + // empty page retains it. A failure never reads it before a full-source + // completion sets it again, so it never needs clearing. + private hasSettledSourceRequest = false + private settledSourceBoundary: Record | undefined + // Independent of finite success: only full-source success repairs ordering. + private needsFullSourceRecovery = false + private requesting = false + // Retaining a demand does not prove it succeeded. Async failure retains it + // (`failed`) for replay; a synchronous startup failure retains nothing. + private fullSource: `none` | `held` | `complete` | `failed` = `none` + // Keep callbacks, not copied requests or rows. Successful full-source work + // subsumes these logical owners; unfinished transports remain observed. + private settledFiniteAcquisitions = new Map< + ReleaseLoadSubset, + number | undefined + >() + // The record's presence blocks automatic retry, including initial requests + // that have no explicit window-operation generation. + private failedRequest: + | { windowOperationGeneration: number | undefined } + | undefined + private failedAcquisitions = new Map() + private active = true + private generation = 0 + private lastPage: { count: number; boundary: unknown } | undefined + private lastPrefixCount: number | undefined + private lastBoundary: unknown + private repairRetries = 0 + private repairTimer: ReturnType | undefined + + constructor( + private readonly info: OrderByOptimizationInfo, + private readonly subscription: CollectionSubscription, + private readonly alias: string, + private readonly onResult: ( + result: LoadSubsetRequestResult, + holdPublication: boolean, + ) => void = () => {}, + private readonly canRetryRepair: () => boolean = () => false, + ) { + this.info.isRequesting = () => this.requesting + } + + /** Derive invalidation from actual contributions, not a second cursor. */ + onSourceChanges( + changes: Array, string | number>>, + sentRows: ReadonlyMap> | undefined, + ): void { + let hasNewRows = false + for (const change of changes) { + const previous = sentRows?.get(change.key) + if ( + change.type !== `insert` && + previous !== undefined && + (change.type === `delete` || + this.info.comparator(previous, change.value) !== 0) + ) { + this.invalidateSourceOrdering() + return + } + if (change.type !== `delete` && previous === undefined) hasNewRows = true + } + // New keys, including ties, may need another page. Duplicate delivery or + // an order-equal update cannot invalidate an already attempted request. + if (hasNewRows) this.invalidateCursor() + } + + start(): void { + const { index, limit, offset, orderBy, requiresFullSource } = this.info + if (index) this.subscription.setOrderByIndex(index) + if (limit === 0) return + if (requiresFullSource) { + this.loadFullSource() + return + } + if (!index || orderBy.length !== 1) { + this.loadPrefix(offset + limit) + return + } + this.loadPage(offset + limit) + } + + loadMore(windowOperationGeneration?: number): Promise | undefined { + if (!this.active || this.info.limit === 0 || this.requesting) return + const mayRetryFailure = + this.failedRequest === undefined || + (windowOperationGeneration !== undefined && + windowOperationGeneration !== + this.failedRequest.windowOperationGeneration) + if (!mayRetryFailure) return this.pending + if ( + (this.failedRequest || this.failedAcquisitions.size > 0) && + windowOperationGeneration !== undefined + ) { + this.cancelRepairRetry() + this.repairRetries = 0 + // Move ownership to the explicit replacement before releasing the old + // lease. Adapter cleanup may reenter the loader. + if (this.failedRequest) { + this.failedRequest.windowOperationGeneration = windowOperationGeneration + } + this.releaseFailedAcquisitions() + // Adapter cleanup can synchronously tear down this loader. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (!this.active) return + } + if (this.fullSource === `failed`) this.fullSource = `none` + else if (this.fullSource !== `none`) return this.pending + if (this.needsFullSourceRecovery || this.info.requiresFullSource) { + this.loadFullSource(windowOperationGeneration) + return this.pending + } + if (!this.info.index || this.info.orderBy.length !== 1) { + if ( + windowOperationGeneration !== undefined || + (this.info.dataNeeded?.() ?? 0) > 0 || + (this.lastPrefixCount !== undefined && + this.lastPrefixCount < this.info.offset + this.info.limit) + ) { + this.loadPrefix( + this.info.offset + this.info.limit, + windowOperationGeneration, + ) + } + return this.pending + } + if (!this.info.dataNeeded || this.pending) return this.pending + // A recorded failure always carries recovery debt, so it cannot reach this + // finite path; only the first request needs the whole prefix here. + let count = Math.max( + this.info.dataNeeded(), + this.hasSettledSourceRequest ? 0 : this.info.offset + this.info.limit, + ) + if ( + windowOperationGeneration !== undefined && + this.settledSourceBoundary !== undefined + ) { + const needed = this.info.offset + this.info.limit + count = Math.max(count, needed - this.countAcquiredRows()) + } + if (count > 0) { + this.loadPage(count, windowOperationGeneration) + } + return this.pending + } + + loadFullSource(windowOperationGeneration?: number): void { + if (!this.active || this.fullSource !== `none`) return + this.fullSource = `held` + this.requestAndObserve( + (onLoadSubsetResult) => { + this.subscription.requestSnapshot({ + trackLoadSubsetPromise: false, + onLoadSubsetResult, + }) + }, + `full-source`, + windowOperationGeneration, + ) + } + + private loadPrefix(count: number, windowOperationGeneration?: number): void { + if (!this.active || this.pending) return + if (this.lastPrefixCount === count) { + if ((this.info.dataNeeded?.() ?? 0) > 0) { + this.loadFullSource(windowOperationGeneration) + } + return + } + this.requestAndObserve( + (onLoadSubsetResult) => { + this.subscription.requestSnapshot({ + orderBy: normalizeOrderByPaths(this.info.orderBy, this.alias), + limit: count, + trackLoadSubsetPromise: false, + onLoadSubsetResult, + }) + }, + `ordered`, + windowOperationGeneration, + ) + this.lastPrefixCount = count + } + + resetCursor(): void { + this.cancelRepairRetry() + this.repairRetries = 0 + this.generation++ + if (this.fullSource === `complete`) this.fullSource = `held` + this.pending = undefined + this.lastBoundary = undefined + this.settledSourceBoundary = undefined + this.invalidateCursor() + } + + settleFullSourceReplay(): void { + // Replay repaired the retained logical acquisition. A later window retry + // must not release that now-successful source demand. A failed finite + // page is still obsolete and must be released by that retry. + if (this.fullSource === `failed`) { + for (const [release, kind] of this.failedAcquisitions) { + if (kind === `full-source`) this.failedAcquisitions.delete(release) + } + this.fullSource = `held` + } + if (this.fullSource !== `none`) { + this.fullSource = `complete` + this.retireSettledFiniteAcquisitions() + } + } + + private retireSettledFiniteAcquisitions(prefix?: { + release: ReleaseLoadSubset + count: number + }): void { + if ( + (this.fullSource !== `complete` && !prefix) || + this.subscription.hasPendingTruncateReplacement + ) + return + const generation = this.generation + runAllCallbacks( + Array.from(this.settledFiniteAcquisitions, ([release, count]) => () => { + if (!this.active || generation !== this.generation) return + if ( + this.fullSource !== `complete` && + (!prefix || + release === prefix.release || + count === undefined || + count > prefix.count) + ) + return + this.settledFiniteAcquisitions.delete(release) + release() + }), + ) + } + + invalidateCursor(): void { + this.lastPage = undefined + this.lastPrefixCount = undefined + } + + invalidateSourceOrdering(): void { + this.invalidateCursor() + this.requireFullSourceRecovery() + } + + dispose(): void { + this.active = false + this.resetCursor() + this.failedAcquisitions.clear() + this.settledFiniteAcquisitions.clear() + } + + private countAcquiredRows(): number { + return this.subscription + .readOrderedSnapshot({ + orderBy: normalizeOrderByPaths(this.info.orderBy, this.alias), + limit: this.info.offset + this.info.limit, + }) + .filter( + ({ value }) => + this.info.comparator(value, this.settledSourceBoundary) <= 0, + ).length + } + + private loadPage(count: number, windowOperationGeneration?: number): void { + if (!this.active || this.pending) return + // Rows observed before the first provider request do not prove ordered + // source coverage. In particular, a row inserted while limit is zero must + // not become the cursor when that window first opens. + const startsFromSourcePrefix = this.settledSourceBoundary === undefined + const biggest = this.settledSourceBoundary + let minValues: Array | undefined + if (biggest !== undefined) { + const value = this.info.valueExtractorForRawRow(biggest) + if (!canExpressCursorOrder(this.info.orderBy, [value])) { + this.loadPrefix( + this.info.offset + this.info.limit, + windowOperationGeneration, + ) + return + } + minValues = [value] + } + const boundary = minValues?.[0] + if ( + this.lastPage?.count === count && + Object.is(this.lastPage.boundary, boundary) + ) { + return + } + this.lastPage = { count, boundary } + this.requestAndObserve( + (onLoadSubsetResult) => { + this.subscription.requestLimitedSnapshot({ + orderBy: normalizeOrderByPaths(this.info.orderBy, this.alias), + limit: count, + minValues, + // Local rows seen before the first provider request prove neither + // a cursor nor a remote offset. Start the first acquisition at zero. + offset: startsFromSourcePrefix ? 0 : this.countAcquiredRows(), + trackLoadSubsetPromise: false, + onLoadSubsetResult, + }) + }, + `ordered`, + windowOperationGeneration, + ) + } + + private observe( + result: LoadSubsetRequestResult, + releaseAcquisition: ReleaseLoadSubset, + kind: OrderedRequestKind, + windowOperationGeneration?: number, + options?: LoadSubsetOptions, + ): Promise { + const isFullSource = kind === `full-source` + const retryRepair = + isFullSource && + this.hasSettledSourceRequest && + this.needsFullSourceRecovery && + windowOperationGeneration === undefined + const generation = this.generation + const complete = (): void => { + if (this.pending === tracked) this.pending = undefined + if (!this.active) return + // Retirement failure does not undo a successful acquisition. Finish its + // boundary and continuation, then report the first cleanup error. + runAllCallbacks([ + () => { + if (!isFullSource) { + // A replay can replace the physical lease while this older transport + // finishes. Retire its logical owner only outside the replay barrier. + const prefixCount = + options?.orderBy && !options.cursor ? options.limit : undefined + this.settledFiniteAcquisitions.set(releaseAcquisition, prefixCount) + this.retireSettledFiniteAcquisitions() + if (generation === this.generation && prefixCount !== undefined) { + this.retireSettledFiniteAcquisitions({ + release: releaseAcquisition, + count: prefixCount, + }) + } + } + }, + () => { + if (generation !== this.generation) return + // A finite request may finish behind an authoritative repair. It cannot + // clear that repair's failure or resume finite refinement around it. + if ( + !isFullSource && + (this.failedRequest || this.fullSource !== `none`) + ) + return + this.failedRequest = undefined + if (kind !== `boundary`) { + this.hasSettledSourceRequest = true + // Source delivery can invalidate the in-flight prefix marker. + if (options?.orderBy && !options.cursor) { + this.lastPrefixCount = options.limit + } + if (!isFullSource && options?.orderBy) { + try { + this.settledSourceBoundary = + this.subscription.readOrderedSnapshot(options).at(-1) + ?.value ?? this.settledSourceBoundary + } catch (error) { + fail(error) + } + } + } + if (isFullSource) { + this.cancelRepairRetry() + this.repairRetries = 0 + this.needsFullSourceRecovery = false + this.fullSource = `complete` + this.retireSettledFiniteAcquisitions() + } + if (kind === `ordered`) { + this.loadBoundary(windowOperationGeneration) + return + } + // A boundary request may add tied rows without filling the query's + // window. Resume forward loading once it settles. + this.loadMore() + }, + ]) + } + const settlesAsync = result instanceof Promise + const request = settlesAsync ? result : Promise.resolve() + const fail = (error: unknown) => { + this.settledFiniteAcquisitions.delete(releaseAcquisition) + if (this.pending === tracked) this.pending = undefined + if (!this.active) return + // A failed request may already have written only part of its result. + // None of those rows is a safe continuation boundary. + this.requireFullSourceRecovery() + if (generation !== this.generation) return + // A failed request proves no full-source coverage. An explicit window + // move or later replay may retry it, but an ordinary graph pass must + // not start an eager retry loop. + if (isFullSource) this.fullSource = `failed` + this.recordRequestFailure(windowOperationGeneration) + this.failedAcquisitions.set(releaseAcquisition, kind) + if (retryRepair) this.scheduleRepairRetry() + throw error + } + const tracked = request.then(complete, fail) + this.pending = tracked + void tracked.catch(() => {}) + // Register each request separately. The operation tracker observes the + // next request before this promise settles, so the logical chain remains + // pending without retaining every ancestor promise until the final page. + this.onResult( + tracked, + settlesAsync && isFullSource && this.needsFullSourceRecovery, + ) + return tracked + } + + private loadBoundary( + windowOperationGeneration?: number, + ): Promise | undefined { + const biggest = this.settledSourceBoundary + if (biggest === undefined) return + const value = this.info.valueExtractorForRawRow(biggest) + const orderBy = normalizeOrderByPaths(this.info.orderBy, this.alias) + if (!canExpressCursorOrder(orderBy.slice(0, 1), [value])) { + this.loadFullSource(windowOperationGeneration) + return this.pending + } + // Undefined is not an expressible cursor boundary, so it denotes that no + // tie request has been attempted. Other falsy values remain valid keys. + if (Object.is(this.lastBoundary, value)) { + return this.loadMore() + } + const where = buildCursorCurrent(orderBy, [value]) + if (!where) { + this.loadFullSource(windowOperationGeneration) + return this.pending + } + this.lastBoundary = value + return this.requestAndObserve( + (onLoadSubsetResult) => { + this.subscription.requestSnapshot({ + where, + trackLoadSubsetPromise: false, + onLoadSubsetResult, + }) + }, + `boundary`, + windowOperationGeneration, + ) + } + + private requireFullSourceRecovery(): void { + this.settledSourceBoundary = undefined + this.needsFullSourceRecovery = true + } + + private cancelRepairRetry(): void { + clearTimeout(this.repairTimer) + this.repairTimer = undefined + } + + private releaseFailedAcquisitions(): void { + const failed = this.failedAcquisitions + this.failedAcquisitions = new Map() + this.requesting = true + try { + runAllCallbacks(failed.keys()) + } finally { + this.requesting = false + } + } + + private scheduleRepairRetry(): void { + if ( + !this.active || + !this.canRetryRepair() || + this.repairTimer !== undefined || + this.repairRetries >= 2 + ) + return + const generation = this.generation + const failedRequest = this.failedRequest + this.repairTimer = setTimeout( + () => { + this.repairTimer = undefined + const retry = Promise.resolve().then(() => { + if ( + !this.active || + !this.canRetryRepair() || + generation !== this.generation || + this.failedRequest !== failedRequest || + this.failedRequest?.windowOperationGeneration !== undefined + ) + return + this.releaseFailedAcquisitions() + // Release may dispose or start a new replay; neither belongs to this retry. + if ( + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition -- release callbacks can dispose the loader + !this.active || + generation !== this.generation || + this.subscription.hasPendingTruncateReplacement + ) + return + this.failedRequest = undefined + this.fullSource = `none` + this.loadFullSource() + }) + void retry.catch(() => {}) + this.onResult(retry, true) + }, + 250 * 2 ** this.repairRetries++, + ) + } + + private failRequest( + observed: + | { + result: LoadSubsetRequestResult + options: LoadSubsetOptions + release: ReleaseLoadSubset + } + | undefined, + error: Error, + isFullSource: boolean, + windowOperationGeneration?: number, + cancelObservedSettlement = false, + ): Error { + if (cancelObservedSettlement) { + this.generation++ + this.pending = undefined + } + this.requireFullSourceRecovery() + this.recordRequestFailure(windowOperationGeneration) + if (isFullSource) this.fullSource = `none` + try { + observed?.release({ error }) + } catch { + // Cleanup is attempted once and must not replace the request failure. + } + if ( + isFullSource && + this.hasSettledSourceRequest && + windowOperationGeneration === undefined + ) { + this.scheduleRepairRetry() + } + return error + } + + /** A failed request blocks ordinary refinement until a new operation. */ + private recordRequestFailure(windowOperationGeneration?: number): void { + this.failedRequest = { windowOperationGeneration } + this.invalidateCursor() + this.lastBoundary = undefined + } + + /** Observe settlement only after all synchronous request work succeeds. */ + private requestAndObserve( + request: ( + onResult: ( + result: LoadSubsetRequestResult, + options: LoadSubsetOptions, + release: ReleaseLoadSubset, + ) => void, + ) => void, + kind: OrderedRequestKind, + windowOperationGeneration?: number, + ): Promise | undefined { + const isFullSource = kind === `full-source` + let observed: + | { + result: LoadSubsetRequestResult + options: LoadSubsetOptions + release: ReleaseLoadSubset + } + | undefined + this.requesting = true + let observing = false + try { + try { + request((result, options, release) => { + observed = { result, options, release } + }) + } finally { + this.requesting = false + } + if (!observed) return + observing = true + return this.observe( + observed.result, + observed.release, + kind, + windowOperationGeneration, + observed.options, + ) + } catch (error) { + // Both request and settlement callbacks may reenter through cleanup. + // Keep refinement blocked until failure and release finish unwinding. + this.requesting = true + try { + const failure = this.failRequest( + observed, + normalizeError(error), + isFullSource, + windowOperationGeneration, + observing, + ) + if (!observing && isFullSource && this.hasSettledSourceRequest) { + // A synchronous repair failure has no transport promise, but must + // still close publication before the triggering graph turn returns. + const rejected = Promise.reject(failure) + void rejected.catch(() => {}) + this.onResult(rejected, true) + } + throw failure + } finally { + this.requesting = false + } + } + } +} diff --git a/packages/db/src/query/live/subset-demand-controller.ts b/packages/db/src/query/live/subset-demand-controller.ts new file mode 100644 index 0000000000..af194a92ce --- /dev/null +++ b/packages/db/src/query/live/subset-demand-controller.ts @@ -0,0 +1,193 @@ +import { inArray } from '../builder/functions.js' +import { PropRef } from '../ir.js' +import { createValueIdentity } from '../equality-value-identity.js' +import type { ValueIdentity } from '../equality-value-identity.js' +import type { CollectionSubscription } from '../../collection/subscription.js' +import type { LazyDemandPlan } from '../compiler/joins.js' +import type { BasicExpression } from '../ir.js' +import type { LoadSubsetRequestResult } from '../../types.js' + +type DemandSegment = { + keys: Map + where: BasicExpression + abortController: AbortController + ready: LoadSubsetRequestResult + state: `pending` | `settled` | `failed` +} + +type DemandState = { + keys: Map + segments: Array +} + +export type DemandUpdate = { + changed: boolean + empty: boolean + ready: Promise> | true +} + +/** + * Keeps lazy subset requests aligned with the current relation of demanded + * keys. Additions load only new coverage. Removals release and rebuild only + * request segments that covered a removed key. + */ +export class SubsetDemandController { + private readonly states = new Map() + private readonly warnedPlans = new Set() + private valueIdentity = createValueIdentity() + + setDemand( + subscription: CollectionSubscription, + plan: LazyDemandPlan, + keys: Set, + ): DemandUpdate { + const nextKeys = canonicalizeKeys(keys, this.valueIdentity) + const previous = this.states.get(plan.id) + const hasFailedCoverage = previous?.segments.some( + (segment) => + segment.state === `failed` && intersects(segment.keys, nextKeys), + ) + if ( + previous && + equalKeySets(previous.keys, nextKeys) && + !hasFailedCoverage + ) { + return { changed: false, empty: nextKeys.size === 0, ready: true } + } + + const segments: Array = [] + + for (const segment of previous?.segments ?? []) { + if (segment.state !== `failed` && intersects(segment.keys, nextKeys)) { + segments.push(segment) + continue + } + + segment.abortController.abort() + try { + subscription.releaseSnapshot(segment.where) + } catch { + // The subscription reports adapter cleanup failures after its one + // release attempt. Demand changes must still reach the graph instead + // of escaping the source commit; adapters own any remote retry. + } + } + + const coveredKeys = new Set( + segments.flatMap((segment) => [...segment.keys.keys()]), + ) + const added = new Map( + [...nextKeys].filter(([key]) => !coveredKeys.has(key)), + ) + if (added.size > 0) { + segments.push( + requestSegment(subscription, plan, added, () => + this.warnUnoptimized(plan), + ), + ) + } + + if (nextKeys.size === 0) { + this.states.delete(plan.id) + } else { + this.states.set(plan.id, { keys: nextKeys, segments }) + } + + const activeSegments = segments.filter((segment) => + intersects(segment.keys, nextKeys), + ) + const pending = activeSegments + .map((segment) => segment.ready) + .filter((ready): ready is Promise => ready instanceof Promise) + return { + changed: true, + empty: nextKeys.size === 0, + ready: pending.length > 0 ? Promise.all(pending) : true, + } + } + + clear(): void { + for (const state of this.states.values()) { + for (const segment of state.segments) segment.abortController.abort() + } + this.states.clear() + this.warnedPlans.clear() + this.valueIdentity = createValueIdentity() + } + + private warnUnoptimized(plan: LazyDemandPlan): void { + if (this.warnedPlans.has(plan.id)) return + this.warnedPlans.add(plan.id) + const path = plan.path.join(`.`) + console.warn( + `[TanStack DB]${plan.collectionId ? ` [${plan.collectionId}]` : ``} Join requires an index on "${path}" for efficient loading. ` + + `Falling back to scanning local data. ` + + `Consider creating an index on the collection with collection.createIndex((row) => row.${path}) ` + + `or enable auto-indexing with autoIndex: 'eager' and a defaultIndexType.`, + ) + } +} + +function canonicalizeKeys( + keys: Set, + valueIdentity: ValueIdentity, +): Map { + return new Map( + [...keys].map((key) => [valueIdentity.serializeEquality(key), key]), + ) +} + +function equalKeySets( + left: Map, + right: Map, +): boolean { + return ( + left.size === right.size && [...left.keys()].every((key) => right.has(key)) + ) +} + +function intersects( + left: Map, + right: Map, +): boolean { + return [...left.keys()].some((key) => right.has(key)) +} + +function requestSegment( + subscription: CollectionSubscription, + plan: LazyDemandPlan, + keys: Map, + onUnoptimized: () => void, +): DemandSegment { + const where = inArray(new PropRef(plan.path), [...keys.values()]) + const abortController = new AbortController() + const load = { ready: true as LoadSubsetRequestResult } + subscription.requestSnapshot({ + where, + signal: abortController.signal, + trackLoadSubsetPromise: false, + onUnoptimized, + onLoadSubsetResult: (result) => { + load.ready = result + }, + }) + const ready = load.ready + const segment: DemandSegment = { + keys, + where, + abortController, + ready, + state: ready instanceof Promise ? `pending` : `settled`, + } + if (ready instanceof Promise) { + void ready.then( + () => { + segment.state = `settled` + }, + () => { + segment.state = `failed` + }, + ) + } + return segment +} diff --git a/packages/db/src/query/live/types.ts b/packages/db/src/query/live/types.ts index 118015bd6f..307397698e 100644 --- a/packages/db/src/query/live/types.ts +++ b/packages/db/src/query/live/types.ts @@ -16,6 +16,12 @@ export type Changes = { inserts: number value: T orderByIndex: string | undefined + // Captured from the retract side of a change so the flush can detect an + // "order-only move": a row whose projected value is unchanged but whose + // `orderByIndex` moved. Such a move is swallowed by the collection's + // value-diff, so it needs an explicit layout notification. + previousValue?: T + previousOrderByIndex?: string | undefined } export type SyncState = { diff --git a/packages/db/src/query/live/utils.ts b/packages/db/src/query/live/utils.ts index c7f701124f..a1823b7bf0 100644 --- a/packages/db/src/query/live/utils.ts +++ b/packages/db/src/query/live/utils.ts @@ -1,15 +1,14 @@ -import { MultiSet, serializeValue } from '@tanstack/db-ivm' +import { MultiSet } from '@tanstack/db-ivm' import { UnsupportedRootScalarSelectError } from '../../errors.js' import { normalizeOrderByPaths } from '../compiler/expressions.js' import { buildQuery, getQueryIR } from '../builder/index.js' -import { ConditionalSelect, IncludesSubquery, isExpressionLike } from '../ir.js' +import { collectCollectionSources, isExpressionLike } from '../ir.js' import type { MultiSetArray, RootStreamBuilder } from '@tanstack/db-ivm' import type { Collection } from '../../collection/index.js' import type { ChangeMessage } from '../../types.js' import type { InitialQueryBuilder, QueryBuilder } from '../builder/index.js' import type { Context } from '../builder/types.js' import type { OrderBy, QueryIR } from '../ir.js' -import type { OrderByOptimizationInfo } from '../compiler/order-by.js' /** * Helper function to extract collections from a compiled query. @@ -17,90 +16,17 @@ import type { OrderByOptimizationInfo } from '../compiler/order-by.js' * Maps collections by their ID (not alias) as expected by the compiler. */ export function extractCollectionsFromQuery( - query: any, + query: QueryIR, ): Record> { - const collections: Record = {} - - // Helper function to recursively extract collections from a query or source - function extractFromSource(source: any) { - if (source.type === `collectionRef`) { - collections[source.collection.id] = source.collection - } else if (source.type === `queryRef`) { - // Recursively extract from subquery - extractFromQuery(source.query) - } else if (source.type === `unionFrom`) { - for (const childSource of source.sources) { - extractFromSource(childSource) - } - } else if (source.type === `unionAll`) { - for (const branch of source.queries) { - extractFromQuery(branch) - } - } - } - - // Helper function to recursively extract collections from a query - function extractFromQuery(q: any) { - // Extract from FROM clause - if (q.from) { - extractFromSource(q.from) - } - - // Extract from JOIN clauses - if (q.join && Array.isArray(q.join)) { - for (const joinClause of q.join) { - if (joinClause.from) { - extractFromSource(joinClause.from) - } - } - } - - // Extract from SELECT (for IncludesSubquery) - if (q.select) { - extractFromSelect(q.select) - } + const collections: Record> = {} + for (const source of collectCollectionSources(query)) { + collections[source.collection.id] = source.collection } - - function extractFromSelect(select: any) { - for (const [key, value] of Object.entries(select)) { - if (typeof key === `string` && key.startsWith(`__SPREAD_SENTINEL__`)) { - continue - } - if (value instanceof IncludesSubquery) { - extractFromQuery(value.query) - } else if (value instanceof ConditionalSelect) { - extractFromConditionalSelect(value) - } else if (isNestedSelectObject(value)) { - extractFromSelect(value) - } - } - } - - function extractFromConditionalSelect(conditional: ConditionalSelect) { - for (const branch of conditional.branches) { - extractFromSelectValue(branch.value) - } - if (conditional.defaultValue !== undefined) { - extractFromSelectValue(conditional.defaultValue) - } - } - - function extractFromSelectValue(value: any) { - if (value instanceof IncludesSubquery) { - extractFromQuery(value.query) - } else if (value instanceof ConditionalSelect) { - extractFromConditionalSelect(value) - } else if (isNestedSelectObject(value)) { - extractFromSelect(value) - } - } - - // Start extraction from the root query - extractFromQuery(query) - return collections } +export { collectCollectionSources as extractCollectionSources } + /** * Helper function to extract the collection that is referenced in the query's FROM clause. * The FROM clause may refer directly to a collection or indirectly to a subquery. @@ -126,119 +52,11 @@ export function extractCollectionFromSource( ) } -/** - * Extracts all aliases used for each collection across the entire query tree. - * - * Traverses the QueryIR recursively to build a map from collection ID to all aliases - * that reference that collection. This is essential for self-join support, where the - * same collection may be referenced multiple times with different aliases. - * - * For example, given a query like: - * ```ts - * q.from({ employee: employeesCollection }) - * .join({ manager: employeesCollection }, ({ employee, manager }) => - * eq(employee.managerId, manager.id) - * ) - * ``` - * - * This function would return: - * ``` - * Map { "employees" => Set { "employee", "manager" } } - * ``` - * - * @param query - The query IR to extract aliases from - * @returns A map from collection ID to the set of all aliases referencing that collection - */ -export function extractCollectionAliases( - query: QueryIR, -): Map> { - const aliasesById = new Map>() - - function recordAlias(source: any) { - if (!source) return - - if (source.type === `collectionRef`) { - const { id } = source.collection - const existing = aliasesById.get(id) - if (existing) { - existing.add(source.alias) - } else { - aliasesById.set(id, new Set([source.alias])) - } - } else if (source.type === `queryRef`) { - traverse(source.query) - } else if (source.type === `unionFrom`) { - for (const childSource of source.sources) { - recordAlias(childSource) - } - } else if (source.type === `unionAll`) { - for (const branch of source.queries) { - traverse(branch) - } - } - } - - function traverseSelect(select: any) { - for (const [key, value] of Object.entries(select)) { - if (typeof key === `string` && key.startsWith(`__SPREAD_SENTINEL__`)) { - continue - } - if (value instanceof IncludesSubquery) { - traverse(value.query) - } else if (value instanceof ConditionalSelect) { - traverseConditionalSelect(value) - } else if (isNestedSelectObject(value)) { - traverseSelect(value) - } - } - } - - function traverseConditionalSelect(conditional: ConditionalSelect) { - for (const branch of conditional.branches) { - traverseSelectValue(branch.value) - } - if (conditional.defaultValue !== undefined) { - traverseSelectValue(conditional.defaultValue) - } - } - - function traverseSelectValue(value: any) { - if (value instanceof IncludesSubquery) { - traverse(value.query) - } else if (value instanceof ConditionalSelect) { - traverseConditionalSelect(value) - } else if (isNestedSelectObject(value)) { - traverseSelect(value) - } - } - - function traverse(q?: QueryIR) { - if (!q) return - - recordAlias(q.from) - - if (q.join) { - for (const joinClause of q.join) { - recordAlias(joinClause.from) - } - } - - if (q.select) { - traverseSelect(q.select) - } - } - - traverse(query) - - return aliasesById -} - /** * Check if a value is a nested select object (plain object, not an expression) */ function isNestedSelectObject(obj: any): boolean { if (obj === null || typeof obj !== `object`) return false - if (obj instanceof IncludesSubquery) return false if (isExpressionLike(obj)) return false // Ref proxies from spread operations if (obj.__refProxy) return false @@ -320,71 +138,35 @@ export function* splitUpdates< } } -/** - * Filter changes to prevent duplicate inserts to a D2 pipeline. - * Maintains D2 multiplicity at 1 for visible items so that deletes - * properly reduce multiplicity to 0. - * - * Mutates `sentKeys` in place: adds keys on insert, removes on delete. - */ -export function filterDuplicateInserts( - changes: Array>, - sentKeys: Set, -): Array> { - const filtered: Array> = [] +/** Keep each source key at one exact D2 contribution. */ +export function reconcileChangesForD2< + T extends object, + TKey extends string | number, +>( + changes: Array>, + sentRows: Map, +): Array> { + const reconciled: Array> = [] for (const change of changes) { + const previousValue = sentRows.get(change.key) if (change.type === `insert`) { - if (sentKeys.has(change.key)) { - continue // Skip duplicate - } - sentKeys.add(change.key) + if (previousValue !== undefined) continue + sentRows.set(change.key, change.value) + reconciled.push(change) } else if (change.type === `delete`) { - sentKeys.delete(change.key) - } - filtered.push(change) - } - return filtered -} - -/** - * Track the biggest value seen in a stream of changes, used for cursor-based - * pagination in ordered subscriptions. Returns whether the load request key - * should be reset (allowing another load). - * - * @param changes - changes to process (deletes are skipped) - * @param current - the current biggest value (or undefined if none) - * @param sentKeys - set of keys already sent to D2 (for new-key detection) - * @param comparator - orderBy comparator - * @returns `{ biggest, shouldResetLoadKey }` — the new biggest value and - * whether the caller should clear its last-load-request-key - */ -export function trackBiggestSentValue( - changes: Array>, - current: unknown | undefined, - sentKeys: Set, - comparator: (a: any, b: any) => number, -): { biggest: unknown; shouldResetLoadKey: boolean } { - let biggest = current - let shouldResetLoadKey = false - - for (const change of changes) { - if (change.type === `delete`) continue - - const isNewKey = !sentKeys.has(change.key) - - if (biggest === undefined) { - biggest = change.value - shouldResetLoadKey = true - } else if (comparator(biggest, change.value) < 0) { - biggest = change.value - shouldResetLoadKey = true - } else if (isNewKey) { - // New key at same sort position — allow another load if needed - shouldResetLoadKey = true + if (previousValue === undefined) continue + sentRows.delete(change.key) + reconciled.push({ ...change, value: previousValue }) + } else { + sentRows.set(change.key, change.value) + reconciled.push( + previousValue === undefined + ? { type: `insert`, key: change.key, value: change.value } + : { ...change, previousValue }, + ) } } - - return { biggest, shouldResetLoadKey } + return reconciled } /** @@ -420,58 +202,3 @@ export function computeSubscriptionOrderByHints( limit: canPassOrderBy ? effectiveLimit : undefined, } } - -/** - * Compute the cursor for loading the next batch of ordered data. - * Extracts values from the biggest sent row and builds the `minValues` - * array and a deduplication key. - * - * @returns `undefined` if the load should be skipped (duplicate request), - * otherwise `{ minValues, normalizedOrderBy, loadRequestKey }`. - */ -export function computeOrderedLoadCursor( - orderByInfo: Pick< - OrderByOptimizationInfo, - 'orderBy' | 'valueExtractorForRawRow' | 'offset' - >, - biggestSentRow: unknown | undefined, - lastLoadRequestKey: string | undefined, - alias: string, - limit: number, -): - | { - minValues: Array | undefined - normalizedOrderBy: OrderBy - loadRequestKey: string - } - | undefined { - const { orderBy, valueExtractorForRawRow, offset } = orderByInfo - - // Extract all orderBy column values from the biggest sent row - // For single-column: returns single value, for multi-column: returns array - const extractedValues = biggestSentRow - ? valueExtractorForRawRow(biggestSentRow as Record) - : undefined - - // Normalize to array format for minValues - let minValues: Array | undefined - if (extractedValues !== undefined) { - minValues = Array.isArray(extractedValues) - ? extractedValues - : [extractedValues] - } - - // Deduplicate: skip if we already issued an identical load request - const loadRequestKey = serializeValue({ - minValues: minValues ?? null, - offset, - limit, - }) - if (lastLoadRequestKey === loadRequestKey) { - return undefined - } - - const normalizedOrderBy = normalizeOrderByPaths(orderBy, alias) - - return { minValues, normalizedOrderBy, loadRequestKey } -} diff --git a/packages/db/src/query/optimizer.ts b/packages/db/src/query/optimizer.ts index fbc50661da..19ce43b40f 100644 --- a/packages/db/src/query/optimizer.ts +++ b/packages/db/src/query/optimizer.ts @@ -130,6 +130,7 @@ import { UnionAll as UnionAllClass, UnionFrom as UnionFromClass, createResidualWhere, + getFromSources, getWhereExpression, isResidualWhere, } from './ir.js' @@ -900,16 +901,6 @@ function optimizeNestedFrom(from: From): From { return from } -function getFromSources(from: From): Array { - if (from.type === `unionFrom`) { - return from.sources - } - if (from.type === `unionAll`) { - return [] - } - return [from] -} - function getFirstFromAlias(query: QueryIR): string | undefined { return getFromSources(query.from)[0]?.alias } diff --git a/packages/db/src/query/predicate-utils.ts b/packages/db/src/query/predicate-utils.ts deleted file mode 100644 index 4483d44ae2..0000000000 --- a/packages/db/src/query/predicate-utils.ts +++ /dev/null @@ -1,1516 +0,0 @@ -import { Func, Value } from './ir.js' -import type { BasicExpression, OrderBy, PropRef } from './ir.js' -import type { LoadSubsetOptions } from '../types.js' - -/** - * Check if one where clause is a logical subset of another. - * Returns true if the subset predicate is more restrictive than (or equal to) the superset predicate. - * - * @example - * // age > 20 is subset of age > 10 (more restrictive) - * isWhereSubset(gt(ref('age'), val(20)), gt(ref('age'), val(10))) // true - * - * @example - * // age > 10 AND name = 'X' is subset of age > 10 (more conditions) - * isWhereSubset(and(gt(ref('age'), val(10)), eq(ref('name'), val('X'))), gt(ref('age'), val(10))) // true - * - * @param subset - The potentially more restrictive predicate - * @param superset - The potentially less restrictive predicate - * @returns true if subset logically implies superset - */ -export function isWhereSubset( - subset: BasicExpression | undefined, - superset: BasicExpression | undefined, -): boolean { - // undefined/missing where clause means "no filter" (all data) - // Both undefined means subset relationship holds (all data ⊆ all data) - if (subset === undefined && superset === undefined) { - return true - } - - // If subset is undefined but superset is not, we're requesting ALL data - // but have only loaded SOME data - subset relationship does NOT hold - if (subset === undefined && superset !== undefined) { - return false - } - - // If superset is undefined (no filter = all data loaded), - // then any constrained subset is contained - if (superset === undefined && subset !== undefined) { - return true - } - - return isWhereSubsetInternal(subset!, superset!) -} - -function makeDisjunction( - preds: Array>, -): BasicExpression { - if (preds.length === 0) { - return new Value(false) - } - if (preds.length === 1) { - return preds[0]! - } - return new Func(`or`, preds) -} - -function convertInToOr(inField: InField) { - const equalities = inField.values.map( - (value) => new Func(`eq`, [inField.ref, new Value(value)]), - ) - return makeDisjunction(equalities) -} - -function isWhereSubsetInternal( - subset: BasicExpression, - superset: BasicExpression, -): boolean { - // If subset is false it is requesting no data, - // thus the result set is empty - // and the empty set is a subset of any set - if (subset.type === `val` && subset.value === false) { - return true - } - - // If expressions are structurally equal, subset relationship holds - if (areExpressionsEqual(subset, superset)) { - return true - } - - // Handle superset being an AND: subset must imply ALL conjuncts - // If superset is (A AND B), then subset ⊆ (A AND B) only if subset ⊆ A AND subset ⊆ B - // Example: (age > 20) ⊆ (age > 10 AND status = 'active') is false (doesn't imply status condition) - if (superset.type === `func` && superset.name === `and`) { - return superset.args.every((arg) => - isWhereSubsetInternal(subset, arg as BasicExpression), - ) - } - - // Handle OR in subset: (A OR B) ⊆ C only if both A ⊆ C and B ⊆ C. - // Must be checked before OR superset so that or(A, B) ⊆ or(C, D) - // decomposes the subset first: A ⊆ or(C, D) AND B ⊆ or(C, D). - if (subset.type === `func` && subset.name === `or`) { - return subset.args.every((arg) => - isWhereSubsetInternal(arg as BasicExpression, superset), - ) - } - - // Handle OR in superset: subset ⊆ (A OR B) if subset ⊆ A or subset ⊆ B. - // Must be checked before decomposing AND subsets so that and(A, B) can - // match a structurally equal disjunct via areExpressionsEqual. - if (superset.type === `func` && superset.name === `or`) { - return superset.args.some((arg) => - isWhereSubsetInternal(subset, arg as BasicExpression), - ) - } - - // Handle subset being an AND: (A AND B) implies both A and B - if (subset.type === `func` && subset.name === `and`) { - // For (A AND B) ⊆ C, since (A AND B) implies A, we check if any conjunct implies C - return subset.args.some((arg) => - isWhereSubsetInternal(arg as BasicExpression, superset), - ) - } - - // Turn x IN [A, B, C] into x = A OR x = B OR x = C - // for unified handling of IN and OR - if (subset.type === `func` && subset.name === `in`) { - const inField = extractInField(subset) - if (inField) { - return isWhereSubsetInternal(convertInToOr(inField), superset) - } - } - - if (superset.type === `func` && superset.name === `in`) { - const inField = extractInField(superset) - if (inField) { - return isWhereSubsetInternal(subset, convertInToOr(inField)) - } - } - - // Handle comparison operators on the same field - if (subset.type === `func` && superset.type === `func`) { - const subsetFunc = subset as Func - const supersetFunc = superset as Func - - // Check if both are comparisons on the same field - const subsetField = extractComparisonField(subsetFunc) - const supersetField = extractComparisonField(supersetFunc) - - if ( - subsetField && - supersetField && - areRefsEqual(subsetField.ref, supersetField.ref) - ) { - return isComparisonSubset( - subsetFunc, - subsetField.value, - supersetFunc, - supersetField.value, - ) - } - - /* - // Handle eq vs in - if (subsetFunc.name === `eq` && supersetFunc.name === `in`) { - const subsetFieldEq = extractEqualityField(subsetFunc) - const supersetFieldIn = extractInField(supersetFunc) - if ( - subsetFieldEq && - supersetFieldIn && - areRefsEqual(subsetFieldEq.ref, supersetFieldIn.ref) - ) { - // field = X is subset of field IN [X, Y, Z] if X is in the array - // Use cached primitive set and metadata from extraction - return arrayIncludesWithSet( - supersetFieldIn.values, - subsetFieldEq.value, - supersetFieldIn.primitiveSet ?? null, - supersetFieldIn.areAllPrimitives - ) - } - } - - // Handle in vs in - if (subsetFunc.name === `in` && supersetFunc.name === `in`) { - const subsetFieldIn = extractInField(subsetFunc) - const supersetFieldIn = extractInField(supersetFunc) - if ( - subsetFieldIn && - supersetFieldIn && - areRefsEqual(subsetFieldIn.ref, supersetFieldIn.ref) - ) { - // field IN [A, B] is subset of field IN [A, B, C] if all values in subset are in superset - // Use cached primitive set and metadata from extraction - return subsetFieldIn.values.every((subVal) => - arrayIncludesWithSet( - supersetFieldIn.values, - subVal, - supersetFieldIn.primitiveSet ?? null, - supersetFieldIn.areAllPrimitives - ) - ) - } - } - */ - } - - // Conservative: if we can't determine, return false - return false -} - -/** - * Helper to combine where predicates with common logic for AND/OR operations - */ -function combineWherePredicates( - predicates: Array>, - operation: `and` | `or`, - simplifyFn: ( - preds: Array>, - ) => BasicExpression | null, -): BasicExpression { - const emptyValue = operation === `and` ? true : false - const identityValue = operation === `and` ? true : false - - if (predicates.length === 0) { - return { type: `val`, value: emptyValue } as BasicExpression - } - - if (predicates.length === 1) { - return predicates[0]! - } - - // Flatten nested expressions of the same operation - const flatPredicates: Array> = [] - for (const pred of predicates) { - if (pred.type === `func` && pred.name === operation) { - flatPredicates.push(...pred.args) - } else { - flatPredicates.push(pred) - } - } - - // Group predicates by field for simplification - const grouped = groupPredicatesByField(flatPredicates) - - // Simplify each group - const simplified: Array> = [] - for (const [field, preds] of grouped.entries()) { - if (field === null) { - // Complex predicates that we can't group by field - simplified.push(...preds) - } else { - // Try to simplify same-field predicates - const result = simplifyFn(preds) - - // For intersection: check for empty set (contradiction) - if ( - operation === `and` && - result && - result.type === `val` && - result.value === false - ) { - // Intersection is empty (conflicting constraints) - entire AND is false - return { type: `val`, value: false } as BasicExpression - } - - // For union: result may be null if simplification failed - if (result) { - simplified.push(result) - } - } - } - - if (simplified.length === 0) { - return { type: `val`, value: identityValue } as BasicExpression - } - - if (simplified.length === 1) { - return simplified[0]! - } - - // Return combined predicate - return { - type: `func`, - name: operation, - args: simplified, - } as BasicExpression -} - -/** - * Combine multiple where predicates with OR logic (union). - * Returns a predicate that is satisfied when any input predicate is satisfied. - * Simplifies when possible (e.g., age > 10 OR age > 20 → age > 10). - * - * @example - * // Take least restrictive - * unionWherePredicates([gt(ref('age'), val(10)), gt(ref('age'), val(20))]) // age > 10 - * - * @example - * // Combine equals into IN - * unionWherePredicates([eq(ref('age'), val(5)), eq(ref('age'), val(10))]) // age IN [5, 10] - * - * @param predicates - Array of where predicates to union - * @returns Combined predicate representing the union - */ -export function unionWherePredicates( - predicates: Array>, -): BasicExpression { - return combineWherePredicates(predicates, `or`, unionSameFieldPredicates) -} - -/** - * Compute the difference between two where predicates: `fromPredicate AND NOT(subtractPredicate)`. - * Returns the simplified predicate, or null if the difference cannot be simplified - * (in which case the caller should fetch the full fromPredicate). - * - * @example - * // Range difference - * minusWherePredicates( - * gt(ref('age'), val(10)), // age > 10 - * gt(ref('age'), val(20)) // age > 20 - * ) // → age > 10 AND age <= 20 - * - * @example - * // Set difference - * minusWherePredicates( - * inOp(ref('status'), ['A', 'B', 'C', 'D']), // status IN ['A','B','C','D'] - * inOp(ref('status'), ['B', 'C']) // status IN ['B','C'] - * ) // → status IN ['A', 'D'] - * - * @example - * // Common conditions - * minusWherePredicates( - * and(gt(ref('age'), val(10)), eq(ref('status'), val('active'))), // age > 10 AND status = 'active' - * and(gt(ref('age'), val(20)), eq(ref('status'), val('active'))) // age > 20 AND status = 'active' - * ) // → age > 10 AND age <= 20 AND status = 'active' - * - * @example - * // Complete overlap - empty result - * minusWherePredicates( - * gt(ref('age'), val(20)), // age > 20 - * gt(ref('age'), val(10)) // age > 10 - * ) // → {type: 'val', value: false} (empty set) - * - * @param fromPredicate - The predicate to subtract from - * @param subtractPredicate - The predicate to subtract - * @returns The simplified difference, or null if cannot be simplified - */ -export function minusWherePredicates( - fromPredicate: BasicExpression | undefined, - subtractPredicate: BasicExpression | undefined, -): BasicExpression | null { - // If nothing to subtract, return the original - if (subtractPredicate === undefined) { - return ( - fromPredicate ?? - ({ type: `val`, value: true } as BasicExpression) - ) - } - - // If from is undefined then we are asking for all data - // so we need to load all data minus what we already loaded - // i.e. we need to load NOT(subtractPredicate) - if (fromPredicate === undefined) { - return { - type: `func`, - name: `not`, - args: [subtractPredicate], - } as BasicExpression - } - - // Check if fromPredicate is entirely contained in subtractPredicate - // In that case, fromPredicate AND NOT(subtractPredicate) = empty set - if (isWhereSubset(fromPredicate, subtractPredicate)) { - return { type: `val`, value: false } as BasicExpression - } - - // Try to detect and handle common conditions - const commonConditions = findCommonConditions( - fromPredicate, - subtractPredicate, - ) - if (commonConditions.length > 0) { - // Extract predicates without common conditions - const fromWithoutCommon = removeConditions(fromPredicate, commonConditions) - const subtractWithoutCommon = removeConditions( - subtractPredicate, - commonConditions, - ) - - // Recursively compute difference on simplified predicates - const simplifiedDifference = minusWherePredicates( - fromWithoutCommon, - subtractWithoutCommon, - ) - - if (simplifiedDifference !== null) { - // Combine the simplified difference with common conditions - return combineConditions([...commonConditions, simplifiedDifference]) - } - } - - // Check if they are on the same field - if so, we can try to simplify - if (fromPredicate.type === `func` && subtractPredicate.type === `func`) { - const result = minusSameFieldPredicates(fromPredicate, subtractPredicate) - if (result !== null) { - return result - } - } - - // Can't simplify - return null to indicate caller should fetch full fromPredicate - return null -} - -/** - * Helper function to compute difference for same-field predicates - */ -function minusSameFieldPredicates( - fromPred: Func, - subtractPred: Func, -): BasicExpression | null { - // Extract field information - const fromField = - extractComparisonField(fromPred) || - extractEqualityField(fromPred) || - extractInField(fromPred) - const subtractField = - extractComparisonField(subtractPred) || - extractEqualityField(subtractPred) || - extractInField(subtractPred) - - // Must be on the same field - if ( - !fromField || - !subtractField || - !areRefsEqual(fromField.ref, subtractField.ref) - ) { - return null - } - - // Handle IN minus IN: status IN [A,B,C,D] - status IN [B,C] = status IN [A,D] - if (fromPred.name === `in` && subtractPred.name === `in`) { - const fromInField = fromField as InField - const subtractInField = subtractField as InField - - // Filter out values that are in the subtract set - const remainingValues = fromInField.values.filter( - (v) => - !arrayIncludesWithSet( - subtractInField.values, - v, - subtractInField.primitiveSet ?? null, - subtractInField.areAllPrimitives, - ), - ) - - if (remainingValues.length === 0) { - return { type: `val`, value: false } as BasicExpression - } - - if (remainingValues.length === 1) { - return { - type: `func`, - name: `eq`, - args: [fromField.ref, { type: `val`, value: remainingValues[0] }], - } as BasicExpression - } - - return { - type: `func`, - name: `in`, - args: [fromField.ref, { type: `val`, value: remainingValues }], - } as BasicExpression - } - - // Handle IN minus equality: status IN [A,B,C] - status = B = status IN [A,C] - if (fromPred.name === `in` && subtractPred.name === `eq`) { - const fromInField = fromField as InField - const subtractValue = (subtractField as { ref: PropRef; value: any }).value - - const remainingValues = fromInField.values.filter( - (v) => !areValuesEqual(v, subtractValue), - ) - - if (remainingValues.length === 0) { - return { type: `val`, value: false } as BasicExpression - } - - if (remainingValues.length === 1) { - return { - type: `func`, - name: `eq`, - args: [fromField.ref, { type: `val`, value: remainingValues[0] }], - } as BasicExpression - } - - return { - type: `func`, - name: `in`, - args: [fromField.ref, { type: `val`, value: remainingValues }], - } as BasicExpression - } - - // Handle equality minus equality: age = 15 - age = 15 = empty, age = 15 - age = 20 = age = 15 - if (fromPred.name === `eq` && subtractPred.name === `eq`) { - const fromValue = (fromField as { ref: PropRef; value: any }).value - const subtractValue = (subtractField as { ref: PropRef; value: any }).value - - if (areValuesEqual(fromValue, subtractValue)) { - return { type: `val`, value: false } as BasicExpression - } - - // No overlap - return original - return fromPred as BasicExpression - } - - // Handle range minus range: age > 10 - age > 20 = age > 10 AND age <= 20 - const fromComp = extractComparisonField(fromPred) - const subtractComp = extractComparisonField(subtractPred) - - if ( - fromComp && - subtractComp && - areRefsEqual(fromComp.ref, subtractComp.ref) - ) { - // Try to compute the difference using range logic - const result = minusRangePredicates( - fromPred, - fromComp.value, - subtractPred, - subtractComp.value, - ) - return result - } - - // Can't simplify - return null -} - -/** - * Helper to compute difference between range predicates - */ -function minusRangePredicates( - fromFunc: Func, - fromValue: any, - subtractFunc: Func, - subtractValue: any, -): BasicExpression | null { - const fromOp = fromFunc.name as `gt` | `gte` | `lt` | `lte` | `eq` - const subtractOp = subtractFunc.name as `gt` | `gte` | `lt` | `lte` | `eq` - const ref = (extractComparisonField(fromFunc) || - extractEqualityField(fromFunc))!.ref - - // age > 10 - age > 20 = (age > 10 AND age <= 20) - if (fromOp === `gt` && subtractOp === `gt`) { - if (fromValue < subtractValue) { - // Result is: fromValue < field <= subtractValue - return { - type: `func`, - name: `and`, - args: [ - fromFunc as BasicExpression, - { - type: `func`, - name: `lte`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - ], - } as BasicExpression - } - // fromValue >= subtractValue means no overlap - return fromFunc as BasicExpression - } - - // age >= 10 - age >= 20 = (age >= 10 AND age < 20) - if (fromOp === `gte` && subtractOp === `gte`) { - if (fromValue < subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - fromFunc as BasicExpression, - { - type: `func`, - name: `lt`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age > 10 - age >= 20 = (age > 10 AND age < 20) - if (fromOp === `gt` && subtractOp === `gte`) { - if (fromValue < subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - fromFunc as BasicExpression, - { - type: `func`, - name: `lt`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age >= 10 - age > 20 = (age >= 10 AND age <= 20) - if (fromOp === `gte` && subtractOp === `gt`) { - if (fromValue <= subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - fromFunc as BasicExpression, - { - type: `func`, - name: `lte`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age < 30 - age < 20 = (age >= 20 AND age < 30) - if (fromOp === `lt` && subtractOp === `lt`) { - if (fromValue > subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - { - type: `func`, - name: `gte`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - fromFunc as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age <= 30 - age <= 20 = (age > 20 AND age <= 30) - if (fromOp === `lte` && subtractOp === `lte`) { - if (fromValue > subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - { - type: `func`, - name: `gt`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - fromFunc as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age < 30 - age <= 20 = (age > 20 AND age < 30) - if (fromOp === `lt` && subtractOp === `lte`) { - if (fromValue > subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - { - type: `func`, - name: `gt`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - fromFunc as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age <= 30 - age < 20 = (age >= 20 AND age <= 30) - if (fromOp === `lte` && subtractOp === `lt`) { - if (fromValue >= subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - { - type: `func`, - name: `gte`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - fromFunc as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // Can't simplify other combinations - return null -} - -/** - * Check if one orderBy clause is a subset of another. - * Returns true if the subset ordering requirements are satisfied by the superset ordering. - * - * @example - * // Subset is prefix of superset - * isOrderBySubset([{expr: age, asc}], [{expr: age, asc}, {expr: name, desc}]) // true - * - * @param subset - The ordering requirements to check - * @param superset - The ordering that might satisfy the requirements - * @returns true if subset is satisfied by superset - */ -export function isOrderBySubset( - subset: OrderBy | undefined, - superset: OrderBy | undefined, -): boolean { - // No ordering requirement is always satisfied - if (!subset || subset.length === 0) { - return true - } - - // If there's no superset ordering but subset requires ordering, not satisfied - if (!superset || superset.length === 0) { - return false - } - - // Check if subset is a prefix of superset with matching expressions and compare options - if (subset.length > superset.length) { - return false - } - - for (let i = 0; i < subset.length; i++) { - const subClause = subset[i]! - const superClause = superset[i]! - - // Check if expressions match - if (!areExpressionsEqual(subClause.expression, superClause.expression)) { - return false - } - - // Check if compare options match - if ( - !areCompareOptionsEqual( - subClause.compareOptions, - superClause.compareOptions, - ) - ) { - return false - } - } - - return true -} - -/** - * Check if one limit is a subset of another. - * Returns true if the subset limit requirements are satisfied by the superset limit. - * - * Note: This function does NOT consider offset. For offset-aware subset checking, - * use `isOffsetLimitSubset` instead. - * - * @example - * isLimitSubset(10, 20) // true (requesting 10 items when 20 are available) - * isLimitSubset(20, 10) // false (requesting 20 items when only 10 are available) - * isLimitSubset(10, undefined) // true (requesting 10 items when unlimited are available) - * - * @param subset - The limit requirement to check - * @param superset - The limit that might satisfy the requirement - * @returns true if subset is satisfied by superset - */ -export function isLimitSubset( - subset: number | undefined, - superset: number | undefined, -): boolean { - // Unlimited superset satisfies any limit requirement - if (superset === undefined) { - return true - } - - // If requesting all data (no limit), we need unlimited data to satisfy it - // But we know superset is not unlimited so we return false - if (subset === undefined) { - return false - } - - // Otherwise, subset must be less than or equal to superset - return subset <= superset -} - -/** - * Check if one offset+limit range is a subset of another. - * Returns true if the subset range is fully contained within the superset range. - * - * A query with `{limit: 10, offset: 0}` loads rows [0, 10). - * A query with `{limit: 10, offset: 20}` loads rows [20, 30). - * - * For subset to be satisfied by superset: - * - Superset must start at or before subset (superset.offset <= subset.offset) - * - Superset must end at or after subset (superset.offset + superset.limit >= subset.offset + subset.limit) - * - * @example - * isOffsetLimitSubset({ offset: 0, limit: 5 }, { offset: 0, limit: 10 }) // true - * isOffsetLimitSubset({ offset: 5, limit: 5 }, { offset: 0, limit: 10 }) // true (rows 5-9 within 0-9) - * isOffsetLimitSubset({ offset: 5, limit: 10 }, { offset: 0, limit: 10 }) // false (rows 5-14 exceed 0-9) - * isOffsetLimitSubset({ offset: 20, limit: 10 }, { offset: 0, limit: 10 }) // false (rows 20-29 outside 0-9) - * - * @param subset - The offset+limit requirements to check - * @param superset - The offset+limit that might satisfy the requirements - * @returns true if subset range is fully contained within superset range - */ -export function isOffsetLimitSubset( - subset: { offset?: number; limit?: number }, - superset: { offset?: number; limit?: number }, -): boolean { - const subsetOffset = subset.offset ?? 0 - const supersetOffset = superset.offset ?? 0 - - // Superset must start at or before subset - if (supersetOffset > subsetOffset) { - return false - } - - // If superset is unlimited, it covers everything from its offset onwards - if (superset.limit === undefined) { - return true - } - - // If subset is unlimited but superset has a limit, subset can't be satisfied - if (subset.limit === undefined) { - return false - } - - // Both have limits - check if subset range is within superset range - const subsetEnd = subsetOffset + subset.limit - const supersetEnd = supersetOffset + superset.limit - - return subsetEnd <= supersetEnd -} - -/** - * Check if one predicate (where + orderBy + limit + offset) is a subset of another. - * Returns true if all aspects of the subset predicate are satisfied by the superset. - * - * @example - * isPredicateSubset( - * { where: gt(ref('age'), val(20)), limit: 10 }, - * { where: gt(ref('age'), val(10)), limit: 20 } - * ) // true - * - * @param subset - The predicate requirements to check - * @param superset - The predicate that might satisfy the requirements - * @returns true if subset is satisfied by superset - */ -export function isPredicateSubset( - subset: LoadSubsetOptions, - superset: LoadSubsetOptions, -): boolean { - // When the superset has a limit, we can only determine subset relationship - // if the where clauses are equal (not just subset relationship). - // - // This is because a limited query only loads a portion of the matching rows. - // A more restrictive where clause might require rows outside that portion. - // - // Example: superset = {where: undefined, limit: 10, orderBy: desc} - // subset = {where: LIKE 'search%', limit: 10, orderBy: desc} - // The top 10 items matching 'search%' might include items outside the overall top 10. - // - // However, if the where clauses are equal, then the subset relationship can - // be determined by orderBy, limit, and offset: - // Example: superset = {where: status='active', limit: 10, offset: 0, orderBy: desc} - // subset = {where: status='active', limit: 5, offset: 0, orderBy: desc} - // The top 5 active items ARE contained in the top 10 active items. - if (superset.limit !== undefined) { - // For limited supersets, where clauses must be equal - if (!areWhereClausesEqual(subset.where, superset.where)) { - return false - } - return ( - isOrderBySubset(subset.orderBy, superset.orderBy) && - isOffsetLimitSubset(subset, superset) - ) - } - - // For unlimited supersets, use the normal subset logic - // Still need to consider offset - an unlimited query with offset only covers - // rows from that offset onwards - return ( - isWhereSubset(subset.where, superset.where) && - isOrderBySubset(subset.orderBy, superset.orderBy) && - isOffsetLimitSubset(subset, superset) - ) -} - -/** - * Check if two where clauses are structurally equal. - * Used for limited query subset checks where subset relationship isn't sufficient. - */ -function areWhereClausesEqual( - a: BasicExpression | undefined, - b: BasicExpression | undefined, -): boolean { - if (a === undefined && b === undefined) { - return true - } - if (a === undefined || b === undefined) { - return false - } - return areExpressionsEqual(a, b) -} - -// ============================================================================ -// Helper functions -// ============================================================================ - -/** - * Find common conditions between two predicates. - * Returns an array of conditions that appear in both predicates. - */ -function findCommonConditions( - predicate1: BasicExpression, - predicate2: BasicExpression, -): Array> { - const conditions1 = extractAllConditions(predicate1) - const conditions2 = extractAllConditions(predicate2) - - const common: Array> = [] - - for (const cond1 of conditions1) { - for (const cond2 of conditions2) { - if (areExpressionsEqual(cond1, cond2)) { - // Avoid duplicates - if (!common.some((c) => areExpressionsEqual(c, cond1))) { - common.push(cond1) - } - break - } - } - } - - return common -} - -/** - * Extract all individual conditions from a predicate, flattening AND operations. - */ -function extractAllConditions( - predicate: BasicExpression, -): Array> { - if (predicate.type === `func` && predicate.name === `and`) { - const conditions: Array> = [] - for (const arg of predicate.args) { - conditions.push(...extractAllConditions(arg as BasicExpression)) - } - return conditions - } - - return [predicate] -} - -/** - * Remove specified conditions from a predicate. - * Returns the predicate with the specified conditions removed, or undefined if all conditions are removed. - */ -function removeConditions( - predicate: BasicExpression, - conditionsToRemove: Array>, -): BasicExpression | undefined { - if (predicate.type === `func` && predicate.name === `and`) { - const remainingArgs = predicate.args.filter( - (arg) => - !conditionsToRemove.some((cond) => - areExpressionsEqual(arg as BasicExpression, cond), - ), - ) - - if (remainingArgs.length === 0) { - return undefined - } else if (remainingArgs.length === 1) { - return remainingArgs[0]! - } else { - return { - type: `func`, - name: `and`, - args: remainingArgs, - } as BasicExpression - } - } - - // For non-AND predicates, don't remove anything - return predicate -} - -/** - * Combine multiple conditions into a single predicate using AND logic. - * Flattens nested AND operations to avoid unnecessary nesting. - */ -function combineConditions( - conditions: Array>, -): BasicExpression { - if (conditions.length === 0) { - return { type: `val`, value: true } as BasicExpression - } else if (conditions.length === 1) { - return conditions[0]! - } else { - // Flatten all conditions, including those that are already AND operations - const flattenedConditions: Array> = [] - - for (const condition of conditions) { - if (condition.type === `func` && condition.name === `and`) { - // Flatten nested AND operations - flattenedConditions.push(...condition.args) - } else { - flattenedConditions.push(condition) - } - } - - if (flattenedConditions.length === 1) { - return flattenedConditions[0]! - } else { - return { - type: `func`, - name: `and`, - args: flattenedConditions, - } as BasicExpression - } - } -} - -/** - * Find a predicate with a specific operator and value - */ -function findPredicateWithOperator( - predicates: Array>, - operator: string, - value: any, -): BasicExpression | undefined { - return predicates.find((p) => { - if (p.type === `func`) { - const f = p as Func - const field = extractComparisonField(f) - return f.name === operator && field && areValuesEqual(field.value, value) - } - return false - }) -} - -function areExpressionsEqual(a: BasicExpression, b: BasicExpression): boolean { - if (a.type !== b.type) { - return false - } - - if (a.type === `val` && b.type === `val`) { - return areValuesEqual(a.value, b.value) - } - - if (a.type === `ref` && b.type === `ref`) { - return areRefsEqual(a, b) - } - - if (a.type === `func` && b.type === `func`) { - const aFunc = a - const bFunc = b - if (aFunc.name !== bFunc.name) { - return false - } - if (aFunc.args.length !== bFunc.args.length) { - return false - } - return aFunc.args.every((arg, i) => - areExpressionsEqual(arg, bFunc.args[i]!), - ) - } - - return false -} - -function areValuesEqual(a: any, b: any): boolean { - // Simple equality check - could be enhanced for deep object comparison - if (a === b) { - return true - } - - // Handle NaN - if (typeof a === `number` && typeof b === `number` && isNaN(a) && isNaN(b)) { - return true - } - - // Handle Date objects - if (a instanceof Date && b instanceof Date) { - return a.getTime() === b.getTime() - } - - // For arrays and objects, use reference equality - // (In practice, we don't need deep equality for these cases - - // same object reference means same value for our use case) - if ( - typeof a === `object` && - typeof b === `object` && - a !== null && - b !== null - ) { - return a === b - } - - return false -} - -function areRefsEqual(a: PropRef, b: PropRef): boolean { - if (a.path.length !== b.path.length) { - return false - } - return a.path.every((segment, i) => segment === b.path[i]) -} - -/** - * Check if a value is a primitive (string, number, boolean, null, undefined) - * Primitives can use Set for fast lookups - */ -function isPrimitive(value: any): boolean { - return ( - value === null || - value === undefined || - typeof value === `string` || - typeof value === `number` || - typeof value === `boolean` - ) -} - -/** - * Check if all values in an array are primitives - */ -function areAllPrimitives(values: Array): boolean { - return values.every(isPrimitive) -} - -/** - * Check if a value is in an array, with optional pre-built Set for optimization. - * The primitiveSet is cached in InField during extraction and reused for all lookups. - */ -function arrayIncludesWithSet( - array: Array, - value: any, - primitiveSet: Set | null, - arrayIsAllPrimitives?: boolean, -): boolean { - // Fast path: use pre-built Set for O(1) lookup - if (primitiveSet) { - // Skip isPrimitive check if we know the value must be primitive for a match - // (if array is all primitives, only primitives can match) - if (arrayIsAllPrimitives || isPrimitive(value)) { - return primitiveSet.has(value) - } - return false // Non-primitive can't be in primitive-only set - } - - // Fallback: use areValuesEqual for Dates and objects - return array.some((v) => areValuesEqual(v, value)) -} - -/** - * Get the maximum of two values, handling both numbers and Dates - */ -function maxValue(a: any, b: any): any { - if (a instanceof Date && b instanceof Date) { - return a.getTime() > b.getTime() ? a : b - } - return Math.max(a, b) -} - -/** - * Get the minimum of two values, handling both numbers and Dates - */ -function minValue(a: any, b: any): any { - if (a instanceof Date && b instanceof Date) { - return a.getTime() < b.getTime() ? a : b - } - return Math.min(a, b) -} - -function areCompareOptionsEqual( - a: { direction?: `asc` | `desc`; [key: string]: any }, - b: { direction?: `asc` | `desc`; [key: string]: any }, -): boolean { - // For now, just compare direction - could be enhanced for other options - return a.direction === b.direction -} - -interface ComparisonField { - ref: PropRef - value: any -} - -function extractComparisonField(func: Func): ComparisonField | null { - // Handle comparison operators: eq, gt, gte, lt, lte - if ([`eq`, `gt`, `gte`, `lt`, `lte`].includes(func.name)) { - // Assume first arg is ref, second is value - const firstArg = func.args[0] - const secondArg = func.args[1] - - if (firstArg?.type === `ref` && secondArg?.type === `val`) { - return { - ref: firstArg, - value: secondArg.value, - } - } - } - - return null -} - -function extractEqualityField(func: Func): ComparisonField | null { - if (func.name === `eq`) { - const firstArg = func.args[0] - const secondArg = func.args[1] - - if (firstArg?.type === `ref` && secondArg?.type === `val`) { - return { - ref: firstArg, - value: secondArg.value, - } - } - } - return null -} - -interface InField { - ref: PropRef - values: Array - // Cached optimization data (computed once, reused many times) - areAllPrimitives?: boolean - primitiveSet?: Set | null -} - -function extractInField(func: Func): InField | null { - if (func.name === `in`) { - const firstArg = func.args[0] - const secondArg = func.args[1] - - if ( - firstArg?.type === `ref` && - secondArg?.type === `val` && - Array.isArray(secondArg.value) - ) { - let values = secondArg.value - // Precompute optimization metadata once - const allPrimitives = areAllPrimitives(values) - let primitiveSet: Set | null = null - - if (allPrimitives && values.length > 10) { - // Build Set and dedupe values at the same time - primitiveSet = new Set(values) - // If we found duplicates, use the deduped array going forward - if (primitiveSet.size < values.length) { - values = Array.from(primitiveSet) - } - } - - return { - ref: firstArg, - values, - areAllPrimitives: allPrimitives, - primitiveSet, - } - } - } - return null -} - -function isComparisonSubset( - subsetFunc: Func, - subsetValue: any, - supersetFunc: Func, - supersetValue: any, -): boolean { - const subOp = subsetFunc.name - const superOp = supersetFunc.name - - // Handle same operator - if (subOp === superOp) { - if (subOp === `eq`) { - // field = X is subset of field = X only - // Fast path: primitives can use strict equality - if (isPrimitive(subsetValue) && isPrimitive(supersetValue)) { - return subsetValue === supersetValue - } - return areValuesEqual(subsetValue, supersetValue) - } else if (subOp === `gt`) { - // field > 20 is subset of field > 10 if 20 > 10 - return subsetValue >= supersetValue - } else if (subOp === `gte`) { - // field >= 20 is subset of field >= 10 if 20 >= 10 - return subsetValue >= supersetValue - } else if (subOp === `lt`) { - // field < 10 is subset of field < 20 if 10 <= 20 - return subsetValue <= supersetValue - } else if (subOp === `lte`) { - // field <= 10 is subset of field <= 20 if 10 <= 20 - return subsetValue <= supersetValue - } - } - - // Handle different operators on same field - // eq vs gt/gte: field = 15 is subset of field > 10 if 15 > 10 - if (subOp === `eq` && superOp === `gt`) { - return subsetValue > supersetValue - } - if (subOp === `eq` && superOp === `gte`) { - return subsetValue >= supersetValue - } - if (subOp === `eq` && superOp === `lt`) { - return subsetValue < supersetValue - } - if (subOp === `eq` && superOp === `lte`) { - return subsetValue <= supersetValue - } - - // gt/gte vs gte/gt - if (subOp === `gt` && superOp === `gte`) { - // field > 10 is subset of field >= 10 if 10 >= 10 (always true for same value) - return subsetValue >= supersetValue - } - if (subOp === `gte` && superOp === `gt`) { - // field >= 11 is subset of field > 10 if 11 > 10 - return subsetValue > supersetValue - } - - // lt/lte vs lte/lt - if (subOp === `lt` && superOp === `lte`) { - // field < 10 is subset of field <= 10 if 10 <= 10 - return subsetValue <= supersetValue - } - if (subOp === `lte` && superOp === `lt`) { - // field <= 9 is subset of field < 10 if 9 < 10 - return subsetValue < supersetValue - } - - return false -} - -function groupPredicatesByField( - predicates: Array>, -): Map>> { - const groups = new Map>>() - - for (const pred of predicates) { - let fieldKey: string | null = null - - if (pred.type === `func`) { - const func = pred as Func - const field = - extractComparisonField(func) || - extractEqualityField(func) || - extractInField(func) - if (field) { - fieldKey = field.ref.path.join(`.`) - } - } - - const group = groups.get(fieldKey) || [] - group.push(pred) - groups.set(fieldKey, group) - } - - return groups -} - -function unionSameFieldPredicates( - predicates: Array>, -): BasicExpression | null { - if (predicates.length === 1) { - return predicates[0]! - } - - // Try to extract range constraints - let maxGt: number | null = null - let maxGte: number | null = null - let minLt: number | null = null - let minLte: number | null = null - const eqValues: Set = new Set() - const inValues: Set = new Set() - const otherPredicates: Array> = [] - - for (const pred of predicates) { - if (pred.type === `func`) { - const func = pred as Func - const field = extractComparisonField(func) - - if (field) { - const value = field.value - if (func.name === `gt`) { - maxGt = maxGt === null ? value : minValue(maxGt, value) - } else if (func.name === `gte`) { - maxGte = maxGte === null ? value : minValue(maxGte, value) - } else if (func.name === `lt`) { - minLt = minLt === null ? value : maxValue(minLt, value) - } else if (func.name === `lte`) { - minLte = minLte === null ? value : maxValue(minLte, value) - } else if (func.name === `eq`) { - eqValues.add(value) - } else { - otherPredicates.push(pred) - } - } else { - const inField = extractInField(func) - if (inField) { - for (const val of inField.values) { - inValues.add(val) - } - } else { - otherPredicates.push(pred) - } - } - } else { - otherPredicates.push(pred) - } - } - - // If we have multiple equality values, combine into IN - if (eqValues.size > 1 || (eqValues.size > 0 && inValues.size > 0)) { - const allValues = [...eqValues, ...inValues] - const ref = predicates.find((p) => { - if (p.type === `func`) { - const field = - extractComparisonField(p as Func) || extractInField(p as Func) - return field !== null - } - return false - }) - - if (ref && ref.type === `func`) { - const field = - extractComparisonField(ref as Func) || extractInField(ref as Func) - if (field) { - return { - type: `func`, - name: `in`, - args: [ - field.ref, - { type: `val`, value: allValues } as BasicExpression, - ], - } as BasicExpression - } - } - } - - // Build the least restrictive range - const result: Array> = [] - - // Choose the least restrictive lower bound - if (maxGt !== null && maxGte !== null) { - // Take the smaller one (less restrictive) - const pred = - maxGte <= maxGt - ? findPredicateWithOperator(predicates, `gte`, maxGte) - : findPredicateWithOperator(predicates, `gt`, maxGt) - if (pred) result.push(pred) - } else if (maxGt !== null) { - const pred = findPredicateWithOperator(predicates, `gt`, maxGt) - if (pred) result.push(pred) - } else if (maxGte !== null) { - const pred = findPredicateWithOperator(predicates, `gte`, maxGte) - if (pred) result.push(pred) - } - - // Choose the least restrictive upper bound - if (minLt !== null && minLte !== null) { - const pred = - minLte >= minLt - ? findPredicateWithOperator(predicates, `lte`, minLte) - : findPredicateWithOperator(predicates, `lt`, minLt) - if (pred) result.push(pred) - } else if (minLt !== null) { - const pred = findPredicateWithOperator(predicates, `lt`, minLt) - if (pred) result.push(pred) - } else if (minLte !== null) { - const pred = findPredicateWithOperator(predicates, `lte`, minLte) - if (pred) result.push(pred) - } - - // Add single eq value - if (eqValues.size === 1 && inValues.size === 0) { - const pred = findPredicateWithOperator(predicates, `eq`, [...eqValues][0]) - if (pred) result.push(pred) - } - - // Add IN if only IN values - if (eqValues.size === 0 && inValues.size > 0) { - result.push( - predicates.find((p) => { - if (p.type === `func`) { - return (p as Func).name === `in` - } - return false - })!, - ) - } - - // Add other predicates - result.push(...otherPredicates) - - if (result.length === 0) { - return { type: `val`, value: true } as BasicExpression - } - - if (result.length === 1) { - return result[0]! - } - - return { - type: `func`, - name: `or`, - args: result, - } as BasicExpression -} diff --git a/packages/db/src/query/runtime-reference-identity.ts b/packages/db/src/query/runtime-reference-identity.ts new file mode 100644 index 0000000000..03162dab29 --- /dev/null +++ b/packages/db/src/query/runtime-reference-identity.ts @@ -0,0 +1,96 @@ +export type RuntimeReferenceIdentity = [ + `runtimeReference`, + namespace: string, + sequence: number, +] + +type ReferenceIdStore = { + get: (key: TKey) => number | undefined + set: (key: TKey, value: number) => unknown +} + +export function createRuntimeReferenceIdentityFactory(): ( + value: object | symbol, +) => RuntimeReferenceIdentity { + const referenceIds = new WeakMap() + let localSymbolIds: ReferenceIdStore | undefined + let registeredSymbolIds: Map | undefined + let namespace: string | undefined + let sequence = 0 + + const getReferenceId = ( + ids: ReferenceIdStore, + key: TKey, + ): number => { + let referenceId = ids.get(key) + if (referenceId === undefined) { + referenceId = ++sequence + ids.set(key, referenceId) + } + return referenceId + } + + return (value) => { + namespace ??= createRuntimeReferenceNamespace() + let referenceId: number + if (typeof value === `symbol`) { + const registeredKey = Symbol.keyFor(value) + if (registeredKey === undefined) { + localSymbolIds ??= createLocalSymbolIdStore() + referenceId = getReferenceId(localSymbolIds, value) + } else { + registeredSymbolIds ??= new Map() + referenceId = getReferenceId(registeredSymbolIds, registeredKey) + } + } else { + referenceId = getReferenceId(referenceIds, value) + } + return [`runtimeReference`, namespace, referenceId] + } +} + +function createLocalSymbolIdStore(): ReferenceIdStore { + const weakIds = new WeakMap< + object, + number + >() as unknown as ReferenceIdStore + const probe = Symbol() + + try { + weakIds.set(probe, 0) + if (weakIds.get(probe) === 0) return weakIds + } catch { + // Older runtimes reject symbols as weak keys. Retain them rather than + // collapse distinct symbols and corrupt equality. + } + + return new Map() +} + +let runtimeReferenceIdentityFactory: + | ReturnType + | undefined + +export function getRuntimeReferenceIdentity( + value: object | symbol, +): RuntimeReferenceIdentity { + runtimeReferenceIdentityFactory ??= createRuntimeReferenceIdentityFactory() + + return runtimeReferenceIdentityFactory(value) +} + +function createRuntimeReferenceNamespace(): string { + const randomValues = new Uint32Array(4) + const runtimeCrypto = Reflect.get(globalThis, `crypto`) as + | { getRandomValues?: (values: Uint32Array) => Uint32Array } + | undefined + if (typeof runtimeCrypto?.getRandomValues === `function`) { + runtimeCrypto.getRandomValues(randomValues) + return Array.from(randomValues, (value) => value.toString(36)).join(`-`) + } + + // Reference equality cannot survive a runtime boundary. A per-runtime nonce + // prevents a persisted key from matching an unrelated reference after a + // reload, even on platforms without Web Crypto. + return `${Date.now().toString(36)}-${Math.random().toString(36).slice(2)}-${Math.random().toString(36).slice(2)}` +} diff --git a/packages/db/src/query/subset-dedupe.ts b/packages/db/src/query/subset-dedupe.ts index 3dc5cc2bc8..8b819888c5 100644 --- a/packages/db/src/query/subset-dedupe.ts +++ b/packages/db/src/query/subset-dedupe.ts @@ -1,246 +1,69 @@ -import { - isPredicateSubset, - isWhereSubset, - minusWherePredicates, - unionWherePredicates, -} from './predicate-utils.js' -import type { BasicExpression } from './ir.js' -import type { LoadSubsetOptions } from '../types.js' +import { getLoadSubsetDemandKey } from './ir-stable-identity.js' +import type { LoadSubsetFn, LoadSubsetOptions } from '../types.js' /** - * Deduplicated wrapper for a loadSubset function. - * Tracks what data has been loaded and avoids redundant calls by applying - * subset logic to predicates. - * - * @param opts - The options for the DeduplicatedLoadSubset - * @param opts.loadSubset - The underlying loadSubset function to wrap - * @param opts.onDeduplicate - An optional callback function that is invoked when a loadSubset call is deduplicated. - * If the call is deduplicated because the requested data is being loaded by an inflight request, - * then this callback is invoked when the inflight request completes successfully and the data is fully loaded. - * This callback is useful if you need to track rows per query, in which case you can't ignore deduplicated calls - * because you need to know which rows were loaded for each query. - * @example - * const dedupe = new DeduplicatedLoadSubset({ loadSubset: myLoadSubset, onDeduplicate: (opts) => console.log(`Call was deduplicated:`, opts) }) - * - * // First call - fetches data - * await dedupe.loadSubset({ where: gt(ref('age'), val(10)) }) - * - * // Second call - subset of first, returns true immediately - * await dedupe.loadSubset({ where: gt(ref('age'), val(20)) }) - * - * // Clear state to start fresh - * dedupe.reset() + * Deduplicates exact canonical demands without inferring broader coverage. + * Requests follow the immutable LoadSubsetOptions contract; no copies are made. */ export class DeduplicatedLoadSubset { - // The underlying loadSubset function to wrap - private readonly _loadSubset: ( - options: LoadSubsetOptions, - ) => true | Promise - - // An optional callback function that is invoked when a loadSubset call is deduplicated. - private readonly onDeduplicate: - | ((options: LoadSubsetOptions) => void) - | undefined - - // Combined where predicate for all unlimited calls (no limit) - private unlimitedWhere: BasicExpression | undefined = undefined - - // Flag to track if we've loaded all data (unlimited call with no where clause) - private hasLoadedAllData = false - - // List of all limited calls (with limit, possibly with orderBy) - // We clone options before storing to prevent mutation of stored predicates - private limitedCalls: Array = [] - - // Track in-flight calls to prevent concurrent duplicate requests - // We store both the options and the promise so we can apply subset logic - private inflightCalls: Array<{ - options: LoadSubsetOptions - promise: Promise - }> = [] - - // Generation counter to invalidate in-flight requests after reset() - // When reset() is called, this increments, and any in-flight completion handlers - // check if their captured generation matches before updating tracking state + private readonly completed = new Set() + private readonly inflight = new Map>() private generation = 0 - constructor(opts: { - loadSubset: (options: LoadSubsetOptions) => true | Promise - onDeduplicate?: (options: LoadSubsetOptions) => void - }) { - this._loadSubset = opts.loadSubset - this.onDeduplicate = opts.onDeduplicate - } + constructor( + private readonly options: { + loadSubset: LoadSubsetFn + onDeduplicate?: (options: LoadSubsetOptions) => void + }, + ) {} - /** - * Load a subset of data, with automatic deduplication based on previously - * loaded predicates and in-flight requests. - * - * This method is auto-bound, so it can be safely passed as a callback without - * losing its `this` context (e.g., `loadSubset: dedupe.loadSubset` in a sync config). - * - * @param options - The predicate options (where, orderBy, limit) - * @returns true if data is already loaded, or a Promise that resolves when data is loaded - */ loadSubset = (options: LoadSubsetOptions): true | Promise => { - // If we've loaded all data, everything is covered - if (this.hasLoadedAllData) { - this.onDeduplicate?.(options) + const key = getLoadSubsetDemandKey(options) + if (this.completed.has(key)) { + this.options.onDeduplicate?.(options) return true } - // Check against unlimited combined predicate - // If we've loaded all data matching a where clause, we don't need to refetch subsets - if (this.unlimitedWhere !== undefined && options.where !== undefined) { - if (isWhereSubset(options.where, this.unlimitedWhere)) { - this.onDeduplicate?.(options) - return true // Data already loaded via unlimited call - } + // Requests with independent cancellation own independent transports. + // Unabortable requests can share without an ownership protocol. + const existing = options.signal ? undefined : this.inflight.get(key) + if (existing) { + // Observer failures must not reject a detached promise after success. + void existing + .then(() => this.options.onDeduplicate?.(options)) + .catch(() => {}) + return existing } - // Check against limited calls - if (options.limit !== undefined) { - const alreadyLoaded = this.limitedCalls.some((loaded) => - isPredicateSubset(options, loaded), - ) + const generation = this.generation + const result = this.options.loadSubset(options) - if (alreadyLoaded) { - this.onDeduplicate?.(options) - return true // Already loaded + if (result === true) { + if (generation === this.generation && !options.signal?.aborted) { + this.completed.add(key) } - } - - // Check against in-flight calls using the same subset logic as resolved calls - // This prevents duplicate requests when concurrent calls have subset relationships - const matchingInflight = this.inflightCalls.find((inflight) => - isPredicateSubset(options, inflight.options), - ) - - if (matchingInflight !== undefined) { - // An in-flight call will load data that covers this request - // Return the same promise so this caller waits for the data to load - // The in-flight promise already handles tracking updates when it completes - const prom = matchingInflight.promise - // Call `onDeduplicate` when the inflight request has loaded the data - prom.then(() => this.onDeduplicate?.(options)).catch() // ignore errors - return prom - } - - // Preserve the original request for tracking and in-flight dedupe, but allow - // the backend request to be narrowed to only the missing subset. - const trackingOptions = cloneOptions(options) - const loadOptions = cloneOptions(options) - if (this.unlimitedWhere !== undefined && options.limit === undefined) { - // Compute difference to get only the missing data - // We can only do this for unlimited queries - // and we can only remove data that was loaded from unlimited queries - // because with limited queries we have no way to express that we already loaded part of the matching data - loadOptions.where = - minusWherePredicates(loadOptions.where, this.unlimitedWhere) ?? - loadOptions.where - } - - // Call underlying loadSubset to load the missing data - const resultPromise = this._loadSubset(loadOptions) - - // Handle both sync (true) and async (Promise) return values - if (resultPromise === true) { - this.updateTracking(trackingOptions) return true - } else { - // Async return - track the promise and update tracking after it resolves - - // Capture the current generation - this lets us detect if reset() was called - // while this request was in-flight, so we can skip updating tracking state - const capturedGeneration = this.generation - - // We need to create a reference to the in-flight entry so we can remove it later - const inflightEntry = { - options: trackingOptions, - promise: resultPromise - .then((result) => { - // Only update tracking if this request is still from the current generation - // If reset() was called, the generation will have incremented and we should - // not repopulate the state that was just cleared - if (capturedGeneration === this.generation) { - this.updateTracking(trackingOptions) - } - return result - }) - .finally(() => { - // Always remove from in-flight array on completion OR rejection - // This ensures failed requests can be retried instead of being cached forever - const index = this.inflightCalls.indexOf(inflightEntry) - if (index !== -1) { - this.inflightCalls.splice(index, 1) - } - }), - } + } - // Store the in-flight entry so concurrent subset calls can wait for it - this.inflightCalls.push(inflightEntry) - return inflightEntry.promise + const promise = result + .then((value) => { + if (generation === this.generation && !options.signal?.aborted) { + this.completed.add(key) + } + return value + }) + .finally(() => { + if (this.inflight.get(key) === promise) this.inflight.delete(key) + }) + if (!options.signal && generation === this.generation) { + this.inflight.set(key, promise) } + return promise } - /** - * Reset all tracking state. - * Clears the history of loaded predicates and in-flight calls. - * Use this when you want to start fresh, for example after clearing the underlying data store. - * - * Note: Any in-flight requests will still complete, but they will not update the tracking - * state after the reset. This prevents old requests from repopulating cleared state. - */ reset(): void { - this.unlimitedWhere = undefined - this.hasLoadedAllData = false - this.limitedCalls = [] - this.inflightCalls = [] - // Increment generation to invalidate any in-flight completion handlers - // This ensures requests that were started before reset() don't repopulate the state + this.completed.clear() + this.inflight.clear() this.generation++ } - - private updateTracking(options: LoadSubsetOptions): void { - // Update tracking based on whether this was a limited or unlimited call - if (options.limit === undefined) { - // Unlimited call - update combined where predicate - // We ignore orderBy for unlimited calls as mentioned in requirements - if (options.where === undefined) { - // No where clause = all data loaded - this.hasLoadedAllData = true - this.unlimitedWhere = undefined - this.limitedCalls = [] - this.inflightCalls = [] - } else if (this.unlimitedWhere === undefined) { - this.unlimitedWhere = options.where - } else { - this.unlimitedWhere = unionWherePredicates([ - this.unlimitedWhere, - options.where, - ]) - } - } else { - // Limited call - add to list for future subset checks - // Options are already cloned by caller to prevent mutation issues - this.limitedCalls.push(options) - } - } -} - -/** - * Clones a LoadSubsetOptions object to prevent mutation of stored predicates. - * This is crucial because callers often reuse the same options object and mutate - * properties like limit or where between calls. Without cloning, our stored history - * would reflect the mutated values rather than what was actually loaded. - */ -export function cloneOptions(options: LoadSubsetOptions): LoadSubsetOptions { - return { - ...options, - orderBy: options.orderBy?.map((clause) => ({ - ...clause, - compareOptions: { ...clause.compareOptions }, - })), - cursor: options.cursor ? { ...options.cursor } : undefined, - } } diff --git a/packages/db/src/scheduler.ts b/packages/db/src/scheduler.ts index 8ffe8b768c..d5faa684e5 100644 --- a/packages/db/src/scheduler.ts +++ b/packages/db/src/scheduler.ts @@ -1,3 +1,5 @@ +import { runAllCallbacks } from './utils/callbacks.js' + /** * Identifier used to scope scheduled work. Maps to a transaction id for live queries. */ @@ -16,13 +18,13 @@ interface ScheduleOptions { /** * State per context. Queue preserves order, jobs hold run functions, dependencies track - * prerequisites, and completed records which jobs have run during the current flush. + * prerequisites. A job leaves the pending map before its callback runs, so work + * queued by that callback is a new pending dependency. */ interface SchedulerContextState { queue: Array jobs: Map void> dependencies: Map> - completed: Set } interface PendingAwareJob { @@ -62,7 +64,6 @@ export class Scheduler { queue: [], jobs: new Map(), dependencies: new Map(), - completed: new Set(), } this.contexts.set(contextId, context) } @@ -98,9 +99,6 @@ export class Scheduler { } else if (!context.dependencies.has(jobId)) { context.dependencies.set(jobId, new Set()) } - - // Clear completion status since we're rescheduling - context.completed.delete(jobId) } /** @@ -111,7 +109,7 @@ export class Scheduler { const context = this.contexts.get(contextId) if (!context) return - const { queue, jobs, dependencies, completed } = context + const { queue, jobs, dependencies } = context while (queue.length > 0) { let ranThisPass = false @@ -122,7 +120,6 @@ export class Scheduler { const run = jobs.get(jobId) if (!run) { dependencies.delete(jobId) - completed.delete(jobId) continue } @@ -137,13 +134,10 @@ export class Scheduler { isPendingAwareJob(dep) && dep.hasPendingGraphRun(contextId) // Treat dependencies as blocking if the dep has a pending run in this - // context or if it's enqueued and not yet complete. If the dep is + // context or if it's enqueued. If the dep is // neither pending nor enqueued, consider it satisfied to avoid deadlocks // on lazy sources that never schedule work. - if ( - (jobs.has(dep) && !completed.has(dep)) || - (!jobs.has(dep) && depHasPending) - ) { + if (jobs.has(dep) || depHasPending) { ready = false break } @@ -153,10 +147,9 @@ export class Scheduler { if (ready) { jobs.delete(jobId) dependencies.delete(jobId) - // Run the job. If it throws, we don't mark it complete, allowing the - // error to propagate while maintaining scheduler state consistency. + // A reentrant schedule now owns a fresh pending job; finishing this + // callback must not mark that replacement as complete. run() - completed.add(jobId) ranThisPass = true } else { queue.push(jobId) @@ -175,20 +168,12 @@ export class Scheduler { this.contexts.delete(contextId) } - /** - * Flush all contexts with pending work. Useful during tear-down. - */ - flushAll(): void { - for (const contextId of Array.from(this.contexts.keys())) { - this.flush(contextId) - } - } - /** Clear all scheduled jobs for a context. */ clear(contextId: SchedulerContextId): void { this.contexts.delete(contextId) - // Notify listeners that this context was cleared - this.clearListeners.forEach((listener) => listener(contextId)) + runAllCallbacks( + [...this.clearListeners].map((listener) => () => listener(contextId)), + ) } /** Register a listener to be notified when a context is cleared. */ @@ -196,27 +181,65 @@ export class Scheduler { this.clearListeners.add(listener) return () => this.clearListeners.delete(listener) } +} - /** Check if a context has pending jobs. */ - hasPendingJobs(contextId: SchedulerContextId): boolean { - const context = this.contexts.get(contextId) - return !!context && context.jobs.size > 0 - } +export const transactionScopedScheduler = new Scheduler() - /** Remove a single job from a context and clean up its dependencies. */ - clearJob(contextId: SchedulerContextId, jobId: unknown): void { - const context = this.contexts.get(contextId) - if (!context) return +let activePublicationContext: SchedulerContextId | undefined +let activePublicationFailure: { error: unknown } | undefined - context.jobs.delete(jobId) - context.dependencies.delete(jobId) - context.completed.delete(jobId) - context.queue = context.queue.filter((id) => id !== jobId) +function getActivePublicationFailure(): { error: unknown } | undefined { + return activePublicationFailure +} + +/** + * Returns the Collection publication that currently owns synchronous change + * delivery. Live-query jobs use it to coalesce all source subscriptions that + * observe one committed batch. + */ +export function getActivePublicationContext(): SchedulerContextId | undefined { + return activePublicationContext +} + +/** Report a listener failure after the whole publication graph has drained. */ +export function recordPublicationError(error: unknown): void { + if (activePublicationContext === undefined) throw error + activePublicationFailure ??= { error } +} - if (context.jobs.size === 0) { - this.contexts.delete(contextId) +/** + * Runs one synchronous Collection publication inside a scheduler context. + * Nested publications share the outer context, so downstream live queries run + * only after every subscriber to the original committed batch has observed it. + */ +export function withPublicationContext(publish: () => T): T { + if (activePublicationContext !== undefined) return publish() + + const contextId = Symbol(`collection-publication`) + activePublicationContext = contextId + activePublicationFailure = undefined + let result!: T + let listenerFailure: { error: unknown } | undefined + try { + result = publish() + transactionScopedScheduler.flush(contextId) + listenerFailure = getActivePublicationFailure() + } catch (error) { + try { + transactionScopedScheduler.clear(contextId) + } catch { + // Keep the earlier publication or graph failure. } + // Keep the first reported failure, including one from an earlier listener. + const publicationFailure = getActivePublicationFailure() + if (publicationFailure) { + throw publicationFailure.error + } + throw error + } finally { + activePublicationContext = undefined + activePublicationFailure = undefined } + if (listenerFailure) throw listenerFailure.error + return result } - -export const transactionScopedScheduler = new Scheduler() diff --git a/packages/db/src/transactions.ts b/packages/db/src/transactions.ts index 84e2bb0d5d..738ec3cedb 100644 --- a/packages/db/src/transactions.ts +++ b/packages/db/src/transactions.ts @@ -1,4 +1,6 @@ import { createDeferred } from './deferred' +import { safeRandomUUID } from './utils/uuid' +import { normalizeError } from './utils/error.js' import './duplicate-instance-check' import { MissingMutationFunctionError, @@ -16,10 +18,136 @@ import type { TransactionWithMutations, } from './types' -const transactions: Array> = [] -let transactionStack: Array> = [] +export class TransactionScope { + private transactions: Array> = [] + private transactionStack: Array> = [] + private sequenceNumber = 0 + + createTransaction>( + config: TransactionConfig, + ): Transaction { + const transaction = new Transaction(config, this, this.sequenceNumber++) + this.transactions.push(transaction) + return transaction + } + + getActiveTransaction(): Transaction | undefined { + return this.transactionStack.at(-1) + } + + getActiveTransactionForCollection(): Transaction | undefined { + const activeTransaction = this.getActiveTransaction() + if (activeTransaction) { + return activeTransaction + } + + if (this === defaultTransactionScope) { + return undefined + } + + return defaultTransactionScope.claimActiveTransaction(this) + } + + private claimActiveTransaction( + targetScope: TransactionScope, + ): Transaction | undefined { + const transaction = this.getActiveTransaction() + if (!transaction) { + return undefined + } + + const owner = getTransactionScope(transaction) + if (owner === targetScope) { + return transaction + } + if (owner !== this) { + throw new Error( + `A transaction created with createTransaction() cannot mutate collections from multiple DbClient instances. Use dbClient.createTransaction() for explicit client scope.`, + ) + } + + this.removeTransaction(transaction) + targetScope.transactions.push(transaction) + targetScope.transactionStack.push(transaction) + transaction.sequenceNumber = targetScope.sequenceNumber++ + transactionScopes.set(transaction, targetScope) + return transaction + } + + registerTransaction(transaction: Transaction): void { + // Clear stale work left by an aborted mutate scope before reusing the id. + transactionScopedScheduler.clear(transaction.id) + this.transactionStack.push(transaction) + } + + unregisterTransaction(transaction: Transaction): void { + try { + transactionScopedScheduler.flush(transaction.id) + } finally { + this.transactionStack = this.transactionStack.filter( + (candidate) => candidate.id !== transaction.id, + ) + } + } + + removeTransaction(transaction: Transaction): void { + const index = this.transactions.findIndex( + (candidate) => candidate.id === transaction.id, + ) + if (index !== -1) { + this.transactions.splice(index, 1) + } + } + + rollbackConflictingTransactions( + transaction: Transaction, + mutationIds: Set, + ): void { + for (const candidate of [...this.transactions]) { + if ( + candidate !== transaction && + candidate.state === `pending` && + candidate.mutations.some((mutation) => + mutationIds.has(mutation.globalKey), + ) + ) { + candidate.rollback({ isSecondaryRollback: true }) + } + } + } + + clear(): void { + const transactionIds = new Set([ + ...this.transactions.map((transaction) => transaction.id), + ...this.transactionStack.map((transaction) => transaction.id), + ]) + for (const transactionId of transactionIds) { + transactionScopedScheduler.clear(transactionId) + } + this.transactions = [] + this.transactionStack = [] + } +} -let sequenceNumber = 0 +const defaultTransactionScope = new TransactionScope() +const transactionScopes = new WeakMap() +const transactionAmbientScopes = new WeakMap() + +function getTransactionScope(transaction: object): TransactionScope { + const scope = transactionScopes.get(transaction) + if (!scope) { + throw new Error(`Transaction is not associated with a TransactionScope.`) + } + return scope +} + +function getTransactionAmbientScope(transaction: object): TransactionScope { + const scope = transactionAmbientScopes.get(transaction) + if (!scope) { + throw new Error(`Transaction is not associated with an ambient scope.`) + } + return scope +} /** * Merges two pending mutations for the same item within a transaction @@ -156,9 +284,7 @@ function mergePendingMutations( export function createTransaction>( config: TransactionConfig, ): Transaction { - const newTransaction = new Transaction(config) - transactions.push(newTransaction) - return newTransaction + return defaultTransactionScope.createTransaction(config) } /** @@ -173,36 +299,7 @@ export function createTransaction>( * } */ export function getActiveTransaction(): Transaction | undefined { - if (transactionStack.length > 0) { - return transactionStack.slice(-1)[0] - } else { - return undefined - } -} - -function registerTransaction(tx: Transaction) { - // Clear any stale work that may have been left behind if a previous mutate - // scope aborted before we could flush. - transactionScopedScheduler.clear(tx.id) - transactionStack.push(tx) -} - -function unregisterTransaction(tx: Transaction) { - // Always flush pending work for this transaction before removing it from - // the ambient stack – this runs even if the mutate callback throws. - // If flush throws (e.g., due to a job error), we still clean up the stack. - try { - transactionScopedScheduler.flush(tx.id) - } finally { - transactionStack = transactionStack.filter((t) => t.id !== tx.id) - } -} - -function removeFromPendingList(tx: Transaction) { - const index = transactions.findIndex((t) => t.id === tx.id) - if (index !== -1) { - transactions.splice(index, 1) - } + return defaultTransactionScope.getActiveTransaction() } class Transaction> { @@ -210,6 +307,18 @@ class Transaction> { public state: TransactionState public mutationFn: MutationFn public mutations: Array> + /** + * Deferred that settles when this transaction settles. + * + * Await `isPersisted.promise`, not `isPersisted` itself. The promise resolves + * when the transaction completes successfully and rejects if the transaction + * fails or is rolled back. + * + * For non-empty commits, the mutation function is the normal settlement + * boundary. This does not inherently prove that a backend has uploaded, + * confirmed, or read back the write unless the mutation function waits for + * that backend observation before returning. + */ public isPersisted: Deferred> public autoCommit: boolean public createdAt: Date @@ -220,26 +329,32 @@ class Transaction> { error: Error } - constructor(config: TransactionConfig) { + constructor( + config: TransactionConfig, + scope: TransactionScope, + sequenceNumber: number, + ) { if (typeof config.mutationFn === `undefined`) { throw new MissingMutationFunctionError() } - this.id = config.id ?? crypto.randomUUID() + this.id = config.id ?? safeRandomUUID() this.mutationFn = config.mutationFn this.state = `pending` this.mutations = [] this.isPersisted = createDeferred>() this.autoCommit = config.autoCommit ?? true this.createdAt = new Date() - this.sequenceNumber = sequenceNumber++ + this.sequenceNumber = sequenceNumber this.metadata = config.metadata ?? {} + transactionScopes.set(this, scope) + transactionAmbientScopes.set(this, scope) } setState(newState: TransactionState) { this.state = newState if (newState === `completed` || newState === `failed`) { - removeFromPendingList(this) + getTransactionScope(this).removeTransaction(this) } } @@ -297,12 +412,22 @@ class Transaction> { throw new TransactionNotPendingMutateError() } - registerTransaction(this) + const initialScope = getTransactionScope(this) + const registeredScopes = new Set([ + initialScope, + getTransactionAmbientScope(this), + ]) + for (const scope of registeredScopes) { + scope.registerTransaction(this) + } try { callback() } finally { - unregisterTransaction(this) + registeredScopes.add(getTransactionScope(this)) + for (const scope of registeredScopes) { + scope.unregisterTransaction(this) + } } if (this.autoCommit) { @@ -333,27 +458,39 @@ class Transaction> { * @param mutations - Array of new mutations to apply */ applyMutations(mutations: Array>): void { + // Merge via a globalKey-keyed map rather than a findIndex scan per + // mutation, which is O(n²) for bulk operations (e.g. inserting many rows + // in one call). Map preserves insertion order, matching the previous + // replace-in-place / remove / append semantics. + const merged = new Map>() + for (const mutation of this.mutations) { + merged.set(mutation.globalKey, mutation) + } + for (const newMutation of mutations) { - const existingIndex = this.mutations.findIndex( - (m) => m.globalKey === newMutation.globalKey, - ) + const existingMutation = merged.get(newMutation.globalKey) - if (existingIndex >= 0) { - const existingMutation = this.mutations[existingIndex]! + if (existingMutation) { const mergeResult = mergePendingMutations(existingMutation, newMutation) if (mergeResult === null) { // Remove the mutation (e.g., delete after insert cancels both) - this.mutations.splice(existingIndex, 1) + merged.delete(newMutation.globalKey) } else { // Replace with merged mutation - this.mutations[existingIndex] = mergeResult + merged.set(newMutation.globalKey, mergeResult) } } else { // Insert new mutation - this.mutations.push(newMutation) + merged.set(newMutation.globalKey, newMutation) } } + + // Rebuild in place to preserve the array's identity for external holders + this.mutations.length = 0 + for (const mutation of merged.values()) { + this.mutations.push(mutation) + } } /** @@ -399,19 +536,20 @@ class Transaction> { if (this.state === `completed`) { throw new TransactionAlreadyCompletedRollbackError() } + if (this.state === `failed`) return this this.setState(`failed`) // See if there's any other transactions w/ mutations on the same ids // and roll them back as well. if (!isSecondaryRollback) { - const mutationIds = new Set() - this.mutations.forEach((m) => mutationIds.add(m.globalKey)) - for (const t of transactions) { - t.state === `pending` && - t.mutations.some((m) => mutationIds.has(m.globalKey)) && - t.rollback({ isSecondaryRollback: true }) - } + const mutationIds = new Set( + this.mutations.map((mutation) => mutation.globalKey), + ) + getTransactionScope(this).rollbackConflictingTransactions( + this, + mutationIds, + ) } // Reject the promise @@ -499,15 +637,11 @@ class Transaction> { await this.mutationFn({ transaction: this as unknown as TransactionWithMutations, }) - - this.setState(`completed`) - this.touchCollection() - - this.isPersisted.resolve(this) } catch (error) { + if ((this.state as TransactionState) !== `persisting`) return this + // Preserve the original error for rethrowing - const originalError = - error instanceof Error ? error : new Error(String(error)) + const originalError = normalizeError(error) // Update transaction with error information this.error = { @@ -522,6 +656,17 @@ class Transaction> { throw originalError } + if ((this.state as TransactionState) !== `persisting`) return this + + this.setState(`completed`) + // Publication errors cannot undo persistence or leave its receipt pending. + // Keep normal publication queued before callers resume from the receipt. + try { + this.touchCollection() + } finally { + this.isPersisted.resolve(this) + } + return this } diff --git a/packages/db/src/types.ts b/packages/db/src/types.ts index 6087e234ec..46d304e910 100644 --- a/packages/db/src/types.ts +++ b/packages/db/src/types.ts @@ -126,6 +126,12 @@ export type MutationFnParams> = { transaction: TransactionWithMutations } +/** + * Persists an optimistic transaction. Do not start or await collection or + * live-query preloads here. Sync commits queue behind this function, so waiting + * for preload work that needs one of those commits can deadlock the mutation. + * Use the collection adapter's mutation acknowledgement helper instead. + */ export type MutationFn> = ( params: MutationFnParams, ) => Promise @@ -229,6 +235,14 @@ export interface SubscriptionStatusEvent { status: T } +/** Event emitted when a subset requested by this subscription fails to load. */ +export interface SubscriptionLoadSubsetErrorEvent { + type: `loadSubset:error` + subscription: Subscription + options: LoadSubsetOptions + error: unknown +} + /** * Event emitted when subscription is unsubscribed */ @@ -244,6 +258,7 @@ export type SubscriptionEvents = { 'status:change': SubscriptionStatusChangeEvent 'status:ready': SubscriptionStatusEvent<`ready`> 'status:loadingSubset': SubscriptionStatusEvent<`loadingSubset`> + 'loadSubset:error': SubscriptionLoadSubsetErrorEvent unsubscribed: SubscriptionUnsubscribedEvent } @@ -254,6 +269,8 @@ export type SubscriptionEvents = { export interface Subscription extends EventEmitter { /** Current status of the subscription */ readonly status: SubscriptionStatus + /** Most recent subset-load failure observed by this subscription. */ + readonly lastError: unknown | undefined } /** @@ -266,9 +283,8 @@ export interface Subscription extends EventEmitter { export type CursorExpressions = { /** * Expression for rows greater than (after) the cursor value. - * For multi-column orderBy, this is a composite cursor using OR of conditions. - * Example for [col1 ASC, col2 DESC] with values [v1, v2]: - * or(gt(col1, v1), and(eq(col1, v1), lt(col2, v2))) + * Core emits cursors for a single order column. Multi-column queries use + * prefix-and-tie loading instead of constructing a composite cursor. */ whereFrom: BasicExpression /** @@ -284,6 +300,15 @@ export type CursorExpressions = { lastKey?: string | number } +/** + * Immutable request data. From submission onward, callers and adapters must + * not mutate these options, their expression trees, comparison options, or + * constant payloads (including Dates, byte arrays, and membership arrays). + * Create new request data to change a demand; core does not clone or freeze it. + * Use stable data properties, not stateful getters, for request data. + * Signal and subscription references stay fixed, but their lifecycle remains + * live: aborting the signal or releasing the subscription is supported. + */ export type LoadSubsetOptions = { /** The where expression to filter the data (does NOT include cursor expressions) */ where?: BasicExpression @@ -302,6 +327,14 @@ export type LoadSubsetOptions = { * The sync layer can use this instead of `cursor` if it prefers offset-based pagination. */ offset?: number + /** + * Aborted when this exact subset request is no longer current. Cancellation + * is cooperative: async adapters should stop before installing more + * request-scoped rows. If an in-flight baseline cannot be canceled, the + * returned load promise must settle after those writes become visible so + * core can keep overlapping replay private until then. + */ + signal?: AbortSignal /** * The subscription that triggered the load. * Advanced sync implementations can use this for: @@ -313,8 +346,35 @@ export type LoadSubsetOptions = { subscription?: Subscription } +/** @internal Result returned by the collection's normalized subset boundary. */ +export type LoadSubsetRequestResult = true | Promise + +/** + * Loads one subset and transfers its ongoing resource ownership only after + * returning `true` or a promise. An implementation that throws synchronously + * must release any partially acquired resource before throwing. A successful + * implementation must await or return every applied receipt from the sync + * `commit()` calls that establish the loaded subset. A result describes only + * the exact `options` passed to this call. + */ export type LoadSubsetFn = (options: LoadSubsetOptions) => true | Promise +/** + * Confirms whether a committed sync transaction is visible or is waiting for + * its turn in the collection's causal queue. A pending receipt rejects with an + * error named `AbortError` if cancellation wins before application. Once the + * writes are visible, later cancellation has no effect. + */ +export type SyncAppliedReceipt = true | Promise + +/** + * Releases the exact acquisition created for `options`. + * + * Implementations must be idempotent and must not throw. An adapter owns any + * remote unsubscribe retry needed to make release reliable. Core attempts + * each acquisition's release once, reports failures, and continues retiring + * other acquisitions. It does not retry a failed subset release. + */ export type UnloadSubsetFn = (options: LoadSubsetOptions) => void export type CleanupFn = () => void @@ -337,8 +397,23 @@ export interface SyncConfig< */ begin: (options?: { immediate?: boolean }) => void write: (message: ChangeMessageOrDeleteKeyMessage) => void - commit: () => void + /** + * Commit the active sync transaction in FIFO order. + * Returns `true` when the writes and events are already visible. Otherwise + * returns a receipt that resolves after they become visible. If collection + * cleanup or an optional request abort abandons the transaction first, the + * receipt rejects with an error named `AbortError`. + * Pass a signal only for request-scoped work that must not publish after + * cancellation. Aborting after application has no effect. + */ + commit: (signal?: AbortSignal) => SyncAppliedReceipt + /** Signal that a usable initial or recovered snapshot is available. */ markReady: () => void + /** + * Signal that initial sync failed before producing a usable snapshot. + * When supplied, `error` is preserved as the rejection reason from `preload()`. + */ + markError: (error?: unknown) => void truncate: () => void metadata?: SyncMetadataApi }) => void | CleanupFn | SyncConfigRes @@ -349,6 +424,22 @@ export interface SyncConfig< */ getSyncMetadata?: () => Record + /** + * Export adapter-specific metadata that lets hydration/persistence resume sync. + * The payload shape is owned by the adapter. + */ + exportSyncMeta?: () => unknown + + /** + * Import adapter-specific metadata produced by exportSyncMeta. + */ + importSyncMeta?: (meta: unknown) => void + + /** + * Merge two adapter-specific metadata payloads during hydration. + */ + mergeSyncMeta?: (current: unknown, incoming: unknown) => unknown + /** * The row update mode used to sync to the collection. * @default `partial` @@ -502,7 +593,8 @@ export type DeleteMutationFn< * @example * // Status transitions * // idle → loading → ready (when markReady() is called) - * // Any status can transition to → error or cleaned-up + * // Any active status can transition to → error or cleaned-up + * // error → ready after a successful sync recovery */ export type CollectionStatus = /** Collection is created but sync hasn't started yet (when startSync config is false) */ @@ -546,6 +638,10 @@ export interface BaseCollectionConfig< /** * Time in milliseconds after which the collection will be garbage collected * when it has no active subscribers. Defaults to 5 minutes (300000ms). + * Sync started without subscribers gets a minimum 50ms grace period. + * Pending preloads retain the collection until they settle. Preloading ready + * data refreshes the retention period. A non-positive or non-finite value + * disables automatic garbage collection. */ gcTime?: number /** @@ -863,7 +959,14 @@ export interface SubscribeChangesOptions< * Allows the caller to directly track the loading promise for isReady status. * @internal */ - onLoadSubsetResult?: (result: Promise | true) => void + onLoadSubsetResult?: (result: LoadSubsetRequestResult) => void + /** Receives subset-load failures scoped to this subscription. @internal */ + onLoadSubsetError?: (event: SubscriptionLoadSubsetErrorEvent) => void + /** Lets a live-query graph retain its last publication during replay. @internal */ + truncateReplayPublication?: { + readonly start: () => void + readonly succeed: () => void + } } export interface SubscribeChangesSnapshotOptions< diff --git a/packages/db/src/utils.ts b/packages/db/src/utils.ts index e652087419..d54c6e3e5c 100644 --- a/packages/db/src/utils.ts +++ b/packages/db/src/utils.ts @@ -30,10 +30,19 @@ export function deepEquals(a: any, b: any): boolean { return deepEqualsInternal(a, b, new Map()) } +function enumerableOwnKeys(value: object): Array { + const keys: Array = Object.keys(value) + for (const key of Object.getOwnPropertySymbols(value)) { + if (Object.prototype.propertyIsEnumerable.call(value, key)) keys.push(key) + } + return keys +} + /** - * Internal implementation with cycle detection to prevent infinite recursion + * Internal implementation with cycle detection to prevent infinite recursion. + * Internal callers can seed already-paired roots when comparing their children. */ -function deepEqualsInternal( +export function deepEqualsInternal( a: any, b: any, visited: Map, @@ -188,9 +197,10 @@ function deepEqualsInternal( } visited.set(a, b) - // Get all keys from both objects - const keysA = Object.keys(a) - const keysB = Object.keys(b) + // Compare enumerable symbol keys as well as string keys. Query results may + // use user-owned symbols, and a symbol-only update is still a value change. + const keysA = enumerableOwnKeys(a) + const keysB = enumerableOwnKeys(b) // Check if they have the same number of keys if (keysA.length !== keysB.length) { @@ -200,7 +210,9 @@ function deepEqualsInternal( // Check if all keys exist in both objects and their values are equal const result = keysA.every( - (key) => key in b && deepEqualsInternal(a[key], b[key], visited), + (key) => + Object.prototype.propertyIsEnumerable.call(b, key) && + deepEqualsInternal(a[key], b[key], visited), ) visited.delete(a) diff --git a/packages/db/src/utils/array-utils.ts b/packages/db/src/utils/array-utils.ts index 47569cacb7..67d505d4e6 100644 --- a/packages/db/src/utils/array-utils.ts +++ b/packages/db/src/utils/array-utils.ts @@ -1,3 +1,13 @@ +import { compareKeys } from '@tanstack/db-ivm' + +/** Key order for descending pages, so no page needs a separate reverse pass. */ +export function compareKeysReversed( + a: string | number, + b: string | number, +): number { + return compareKeys(b, a) +} + /** * Finds the correct insert position for a value in a sorted array using binary search * @param sortedArray The sorted array to search in @@ -26,52 +36,3 @@ export function findInsertPositionInArray( return left } - -/** - * Finds the correct insert position for a value in a sorted tuple array using binary search - * @param sortedArray The sorted tuple array to search in - * @param value The value to find the position for - * @param compareFn Comparison function to use for ordering - * @returns The index where the value should be inserted to maintain order - */ -export function findInsertPosition( - sortedArray: Array<[T, any]>, - value: T, - compareFn: (a: T, b: T) => number, -): number { - let left = 0 - let right = sortedArray.length - - while (left < right) { - const mid = Math.floor((left + right) / 2) - const comparison = compareFn(sortedArray[mid]![0], value) - - if (comparison < 0) { - left = mid + 1 - } else { - right = mid - } - } - - return left -} - -/** - * Deletes a value from a sorted array while maintaining sort order - * @param sortedArray The sorted array to delete from - * @param value The value to delete - * @param compareFn Comparison function to use for ordering - * @returns True if the value was found and deleted, false otherwise - */ -export function deleteInSortedArray( - sortedArray: Array, - value: T, - compareFn: (a: T, b: T) => number, -): boolean { - const idx = findInsertPositionInArray(sortedArray, value, compareFn) - if (idx < sortedArray.length && compareFn(sortedArray[idx]!, value) === 0) { - sortedArray.splice(idx, 1) - return true - } - return false -} diff --git a/packages/db/src/utils/btree.ts b/packages/db/src/utils/btree.ts index 0d35cf7a5b..39a7a0a38c 100644 --- a/packages/db/src/utils/btree.ts +++ b/packages/db/src/utils/btree.ts @@ -32,71 +32,14 @@ type index = number // - V8 source (NewElementsCapacity in src/objects.h): arrays grow by 50% + 16 elements /** - * A reasonably fast collection of key-value pairs with a powerful API. - * Largely compatible with the standard Map. BTree is a B+ tree data structure, - * so the collection is sorted by key. - * - * B+ trees tend to use memory more efficiently than hashtables such as the - * standard Map, especially when the collection contains a large number of - * items. However, maintaining the sort order makes them modestly slower: - * O(log size) rather than O(1). This B+ tree implementation supports O(1) - * fast cloning. It also supports freeze(), which can be used to ensure that - * a BTree is not changed accidentally. - * - * Confusingly, the ES6 Map.forEach(c) method calls c(value,key) instead of - * c(key,value), in contrast to other methods such as set() and entries() - * which put the key first. I can only assume that the order was reversed on - * the theory that users would usually want to examine values and ignore keys. - * BTree's forEach() therefore works the same way, but a second method - * `.forEachPair((key,value)=>{...})` is provided which sends you the key - * first and the value second; this method is slightly faster because it is - * the "native" for-each method for this class. - * - * Out of the box, BTree supports keys that are numbers, strings, arrays of - * numbers/strings, Date, and objects that have a valueOf() method returning a - * number or string. Other data types, such as arrays of Date or custom - * objects, require a custom comparator, which you must pass as the second - * argument to the constructor (the first argument is an optional list of - * initial items). Symbols cannot be used as keys because they are unordered - * (one Symbol is never "greater" or "less" than another). - * - * @example - * Given a {name: string, age: number} object, you can create a tree sorted by - * name and then by age like this: - * - * var tree = new BTree(undefined, (a, b) => { - * if (a.name > b.name) - * return 1; // Return a number >0 when a > b - * else if (a.name < b.name) - * return -1; // Return a number <0 when a < b - * else // names are equal (or incomparable) - * return a.age - b.age; // Return >0 when a.age > b.age - * }); - * - * tree.set({name:"Bill", age:17}, "happy"); - * tree.set({name:"Fran", age:40}, "busy & stressed"); - * tree.set({name:"Bill", age:55}, "recently laid off"); - * tree.forEachPair((k, v) => { - * console.log(`Name: ${k.name} Age: ${k.age} Status: ${v}`); - * }); - * - * @description - * The "range" methods (`forEach, forRange, editRange`) will return the number - * of elements that were scanned. In addition, the callback can return {break:R} - * to stop early and return R from the outer function. - * - * - TODO: Test performance of preallocating values array at max size - * - TODO: Add fast initialization when a sorted array is provided to constructor - * - * For more documentation see https://github.com/qwertie/btree-typescript - * - * Are you a C# developer? You might like the similar data structures I made for C#: - * BDictionary, BList, etc. See http://core.loyc.net/collections/ - * + * Mutable B+ tree used by BTreeIndex for sorted value buckets. Keys use the + * supplied comparator; point operations cost O(log size). This local fork has + * no copy-on-write sharing, cloning, or optional-value storage. + * Range callbacks may return { break: result } to stop traversal early. * @author David Piepgrass */ export class BTree { - private _root: BNode = EmptyLeaf as BNode + private _root: BNode = new BNode() _size = 0 _maxNodeSize: number @@ -109,19 +52,12 @@ export class BTree { /** * Initializes an empty B+ tree. * @param compare Custom function to compare pairs of elements in the tree. - * If not specified, defaultComparator will be used which is valid as long as K extends DefaultComparable. - * @param entries A set of key-value pairs to initialize the tree * @param maxNodeSize Branching factor (maximum items or children per node) * Must be in range 4..256. If undefined or <4 then default is used; if >256 then 256. */ - public constructor( - compare: (a: K, b: K) => number, - entries?: Array<[K, V]>, - maxNodeSize?: number, - ) { + public constructor(compare: (a: K, b: K) => number, maxNodeSize?: number) { this._maxNodeSize = maxNodeSize! >= 4 ? Math.min(maxNodeSize!, 256) : 32 this._compare = compare - if (entries) this.setPairs(entries) } // /////////////////////////////////////////////////////////////////////////// @@ -131,18 +67,10 @@ export class BTree { get size() { return this._size } - /** Gets the number of key-value pairs in the tree. */ - get length() { - return this._size - } - /** Returns true iff the tree contains no key-value pairs. */ - get isEmpty() { - return this._size === 0 - } /** Releases the tree so that its size is 0. */ clear() { - this._root = EmptyLeaf as BNode + this._root = new BNode() this._size = 0 } @@ -160,7 +88,7 @@ export class BTree { * Adds or overwrites a key-value pair in the B+ tree. * @param key the key is used to determine the sort order of * data in the tree. - * @param value data to associate with the key (optional) + * @param value data to associate with the key * @param overwrite Whether to overwrite an existing key-value pair * (default: true). If this is false and there is an existing * key-value pair then this method has no effect. @@ -171,7 +99,6 @@ export class BTree { * has data that does not affect its sort order. */ set(key: K, value: V, overwrite?: boolean): boolean { - if (this._root.isShared) this._root = this._root.clone() const result = this._root.set(key, value, overwrite, this) if (result === true || result === false) return result // Root node has split, so create a new root node. @@ -203,11 +130,6 @@ export class BTree { // /////////////////////////////////////////////////////////////////////////// // Additional methods /////////////////////////////////////////////////////// - /** Returns the maximum number of children/values before nodes will split. */ - get maxNodeSize() { - return this._maxNodeSize - } - /** Gets the lowest key in the tree. Complexity: O(log size) */ minKey(): K | undefined { return this._root.minKey() @@ -218,23 +140,6 @@ export class BTree { return this._root.maxKey() } - /** Gets an array of all keys, sorted */ - keysArray() { - const results: Array = [] - this._root.forRange( - this.minKey()!, - this.maxKey()!, - true, - false, - this, - 0, - (k, _v) => { - results.push(k) - }, - ) - return results - } - /** Returns the next pair whose key is larger than the specified key (or undefined if there is none). * If key === undefined, this function returns the lowest pair. * @param key The key to search for. @@ -254,14 +159,6 @@ export class BTree { ) } - /** Returns the next key larger than the specified key, or undefined if there is none. - * Also, nextHigherKey(undefined) returns the lowest key. - */ - nextHigherKey(key: K | undefined): K | undefined { - const p = this.nextHigherPair(key, ReusedArray as [K, V]) - return p && p[0] - } - /** Returns the next pair whose key is smaller than the specified key (or undefined if there is none). * If key === undefined, this function returns the highest pair. * @param key The key to search for. @@ -276,31 +173,6 @@ export class BTree { return this._root.getPairOrNextLower(key, this._compare, false, reusedArray) } - /** Returns the next key smaller than the specified key, or undefined if there is none. - * Also, nextLowerKey(undefined) returns the highest key. - */ - nextLowerKey(key: K | undefined): K | undefined { - const p = this.nextLowerPair(key, ReusedArray as [K, V]) - return p && p[0] - } - - /** Adds all pairs from a list of key-value pairs. - * @param pairs Pairs to add to this tree. If there are duplicate keys, - * later pairs currently overwrite earlier ones (e.g. [[0,1],[0,7]] - * associates 0 with 7.) - * @param overwrite Whether to overwrite pairs that already exist (if false, - * pairs[i] is ignored when the key pairs[i][0] already exists.) - * @returns The number of pairs added to the collection. - * @description Computational complexity: O(pairs.length * log(size + pairs.length)) - */ - setPairs(pairs: Array<[K, V]>, overwrite?: boolean): number { - let added = 0 - for (const pair of pairs) { - if (this.set(pair[0], pair[1], overwrite)) added++ - } - return added - } - forRange( low: K, high: K, @@ -348,12 +220,10 @@ export class BTree { /** * Scans and potentially modifies values for a subsequence of keys. * Note: the callback `onFound` should ideally be a pure function. - * Specfically, it must not insert items, call clone(), or change - * the collection except via return value; out-of-band editing may - * cause an exception or may cause incorrect data to be sent to - * the callback (duplicate or missed items). It must not cause a - * clone() of the collection, otherwise the clone could be modified - * by changes requested by the callback. + * Specfically, it must not insert items or change the collection + * except via return value; out-of-band editing may cause an + * exception or may cause incorrect data to be sent to the callback + * (duplicate or missed items). * @param low The first key scanned will be greater than or equal to `low`. * @param high Scanning stops when a key larger than this is reached. * @param includeHigh If the `high` key is present, `onFound` is called for @@ -370,9 +240,6 @@ export class BTree { * `{break:R}` to stop early. * @description * Computational complexity: O(number of items scanned + log size) - * Note: if the tree has been cloned with clone(), any shared - * nodes are copied before `onFound` is called. This takes O(n) time - * where n is proportional to the amount of shared data scanned. */ editRange( low: K, @@ -382,7 +249,6 @@ export class BTree { initialCounter?: number, ): R | number { let root = this._root - if (root.isShared) this._root = root = root.clone() try { const r = root.forRange( low, @@ -395,18 +261,12 @@ export class BTree { ) return typeof r === `number` ? r : r.break! } finally { - let isShared while (root.keys.length <= 1 && !root.isLeaf) { - isShared ||= root.isShared this._root = root = root.keys.length === 0 - ? EmptyLeaf + ? new BNode() : (root as any as BNodeInternal).children[0]! } - // If any ancestor of the new root was shared, the new root must also be shared - if (isShared) { - root.isShared = true - } } } } @@ -416,19 +276,13 @@ class BNode { // If this is an internal node, _keys[i] is the highest key in children[i]. keys: Array values: Array - // True if this node might be within multiple `BTree`s (or have multiple parents). - // If so, it must be cloned before being mutated to avoid changing an unrelated tree. - // This is transitive: if it's true, children are also shared even if `isShared!=true` - // in those children. (Certain operations will propagate isShared=true to children.) - isShared: true | undefined get isLeaf() { return (this as any).children === undefined } - constructor(keys: Array = [], values?: Array) { + constructor(keys: Array = [], values: Array = []) { this.keys = keys - this.values = values || undefVals - this.isShared = undefined + this.values = values } // ///////////////////////////////////////////////////////////////////////// @@ -486,11 +340,6 @@ class BNode { return reusedArray } - clone(): BNode { - const v = this.values - return new BNode(this.keys.slice(0), v === undefVals ? v : v.slice(0)) - } - get(key: K, defaultValue: V | undefined, tree: BTree): V | undefined { const i = this.indexOf(key, -1, tree._compare) return i < 0 ? defaultValue : this.values[i] @@ -545,7 +394,7 @@ class BNode { tree._size++ if (this.keys.length < tree._maxNodeSize) { - return this.insertInLeaf(i, key, value, tree) + return this.insertInLeaf(i, key, value) } else { // This leaf node is full and must split const newRightSibling = this.splitOffRightSide() @@ -554,13 +403,12 @@ class BNode { i -= this.keys.length target = newRightSibling } - target.insertInLeaf(i, key, value, tree) + target.insertInLeaf(i, key, value) return newRightSibling } } else { // Key already exists if (overwrite !== false) { - if (value !== undefined) this.reifyValues() // usually this is a no-op, but some users may wish to edit the key this.keys[i] = key this.values[i] = value @@ -569,61 +417,30 @@ class BNode { } } - reifyValues() { - if (this.values === undefVals) - return (this.values = this.values.slice(0, this.keys.length)) - return this.values - } - - insertInLeaf(i: index, key: K, value: V, tree: BTree) { + insertInLeaf(i: index, key: K, value: V) { this.keys.splice(i, 0, key) - if (this.values === undefVals) { - while (undefVals.length < tree._maxNodeSize) undefVals.push(undefined) - if (value === undefined) { - return true - } else { - this.values = undefVals.slice(0, this.keys.length - 1) - } - } this.values.splice(i, 0, value) return true } takeFromRight(rhs: BNode) { // Reminder: parent node must update its copy of key for this node - // assert: neither node is shared // assert rhs.keys.length > (maxNodeSize/2 && this.keys.length) { // Reminder: parent node must update its copy of key for this node - // assert: neither node is shared // assert rhs.keys.length > (maxNodeSize/2 && this.keys.length { // Reminder: parent node must update its copy of key for this node - const half = this.keys.length >> 1, - keys = this.keys.splice(half) - const values = - this.values === undefVals ? undefVals : this.values.splice(half) - return new BNode(keys, values) + const half = this.keys.length >> 1 + return new BNode(this.keys.splice(half), this.values.splice(half)) } // /////////////////////////////////////////////////////////////////////////// @@ -658,11 +475,11 @@ class BNode { const result = onFound(key, values[i]!, count++) if (result !== undefined) { if (editMode === true) { - if (key !== keys[i] || this.isShared === true) - throw new Error(`BTree illegally changed or cloned in editRange`) + if (key !== keys[i]) + throw new Error(`BTree illegally changed in editRange`) if (result.delete) { this.keys.splice(i, 1) - if (this.values !== undefVals) this.values.splice(i, 1) + this.values.splice(i, 1) tree._size-- i-- iHigh-- @@ -680,11 +497,7 @@ class BNode { /** Adds entire contents of right-hand sibling (rhs is left unchanged) */ mergeSibling(rhs: BNode, _: number) { this.keys.push.apply(this.keys, rhs.keys) - if (this.values === undefVals) { - if (rhs.values === undefVals) return - this.values = this.values.slice(0, this.keys.length) - } - this.values.push.apply(this.values, rhs.reifyValues()) + this.values.push.apply(this.values, rhs.values) } } @@ -695,10 +508,6 @@ class BNodeInternal extends BNode { // keys[i] caches the value of children[i].maxKey(). children: Array> - /** - * This does not mark `children` as shared, so it is the responsibility of the caller - * to ensure children are either marked shared, or aren't included in another tree. - */ constructor(children: Array>, keys?: Array) { if (!keys) { keys = [] @@ -783,10 +592,9 @@ class BNodeInternal extends BNode { const c = this.children, max = tree._maxNodeSize, cmp = tree._compare - let i = Math.min(this.indexOf(key, 0, cmp), c.length - 1), - child = c[i]! + let i = Math.min(this.indexOf(key, 0, cmp), c.length - 1) + const child = c[i]! - if (child.isShared) c[i] = child = child.clone() if (child.keys.length >= max) { // child is full; inserting anything else will cause a split. // Shifting an item to the left or right sibling may avoid a split. @@ -798,7 +606,6 @@ class BNodeInternal extends BNode { (other = c[i - 1]!).keys.length < max && cmp(child.keys[0]!, key) < 0 ) { - if (other.isShared) c[i - 1] = other = other.clone() other.takeFromRight(child) this.keys[i - 1] = other.maxKey()! } else if ( @@ -806,7 +613,6 @@ class BNodeInternal extends BNode { other.keys.length < max && cmp(child.maxKey()!, key) < 0 ) { - if (other.isShared) c[i + 1] = other = other.clone() other.takeFromLeft(child) this.keys[i] = c[i]!.maxKey()! } @@ -835,11 +641,7 @@ class BNodeInternal extends BNode { } } - /** - * Inserts `child` at index `i`. - * This does not mark `child` as shared, so it is the responsibility of the caller - * to ensure that either child is marked shared, or it is not included in another tree. - */ + /** Inserts `child` at index `i`. */ insert(i: index, child: BNode) { this.children.splice(i, 0, child) this.keys.splice(i, 0, child.maxKey()!) @@ -850,7 +652,6 @@ class BNodeInternal extends BNode { * Modifies this to remove the second half of the items, returning a separate node containing them. */ splitOffRightSide() { - // assert !this.isShared; const half = this.children.length >> 1 return new BNodeInternal( this.children.splice(half), @@ -860,7 +661,6 @@ class BNodeInternal extends BNode { takeFromRight(rhs: BNode) { // Reminder: parent node must update its copy of key for this node - // assert: neither node is shared // assert rhs.keys.length > (maxNodeSize/2 && this.keys.length).children.shift()!) @@ -868,7 +668,6 @@ class BNodeInternal extends BNode { takeFromLeft(lhs: BNode) { // Reminder: parent node must update its copy of key for this node - // assert: neither node is shared // assert rhs.keys.length > (maxNodeSize/2 && this.keys.length).children.pop()!) @@ -916,7 +715,6 @@ class BNodeInternal extends BNode { } else if (i <= iHigh) { try { for (; i <= iHigh; i++) { - if (children[i]!.isShared) children[i] = children[i]!.clone() const result = children[i]!.forRange( low, high, @@ -959,9 +757,6 @@ class BNodeInternal extends BNode { const children = this.children if (i >= 0 && i + 1 < children.length) { if (children[i]!.keys.length + children[i + 1]!.keys.length <= maxSize) { - if (children[i]!.isShared) - // cloned already UNLESS i is outside scan range - children[i] = children[i]!.clone() children[i]!.mergeSibling(children[i + 1]!, maxSize) children.splice(i + 1, 1) this.keys.splice(i + 1, 1) @@ -974,22 +769,14 @@ class BNodeInternal extends BNode { /** * Move children from `rhs` into this. - * `rhs` must be part of this tree, and be removed from it after this call - * (otherwise isShared for its children could be incorrect). + * `rhs` must be part of this tree, and be removed from it after this call. */ mergeSibling(rhs: BNode, maxNodeSize: number) { - // assert !this.isShared; const oldLength = this.keys.length this.keys.push.apply(this.keys, rhs.keys) const rhsChildren = (rhs as any as BNodeInternal).children this.children.push.apply(this.children, rhsChildren) - if (rhs.isShared && !this.isShared) { - // All children of a shared node are implicitly shared, and since their new - // parent is not shared, they must now be explicitly marked as shared. - for (const child of rhsChildren) child.isShared = true - } - // If our children are themselves almost empty due to a mass-delete, // they may need to be merged too (but only the oldLength-1 and its // right sibling should need this). @@ -997,27 +784,8 @@ class BNodeInternal extends BNode { } } -// Optimization: this array of `undefined`s is used instead of a normal -// array of values in nodes where `undefined` is the only value. -// Its length is extended to max node size on first use; since it can -// be shared between trees with different maximums, its length can only -// increase, never decrease. Its type should be undefined[] but strangely -// TypeScript won't allow the comparison V[] === undefined[]. To prevent -// users from making this array too large, BTree has a maximum node size. -// -// FAQ: undefVals[i] is already undefined, so why increase the array size? -// Reading outside the bounds of an array is relatively slow because it -// has the side effect of scanning the prototype chain. -const undefVals: Array = [] - const Delete = { delete: true }, DeleteRange = () => Delete -const EmptyLeaf = (function () { - const n = new BNode() - n.isShared = true - return n -})() -const ReusedArray: Array = [] // assumed thread-local function check(fact: boolean, ...args: Array) { if (!fact) { diff --git a/packages/db/src/utils/callbacks.ts b/packages/db/src/utils/callbacks.ts new file mode 100644 index 0000000000..1b0aba4859 --- /dev/null +++ b/packages/db/src/utils/callbacks.ts @@ -0,0 +1,12 @@ +/** Attempt every callback, then rethrow the first exact failure value. */ +export function runAllCallbacks(callbacks: Iterable<() => void>): void { + let firstFailure: { error: unknown } | undefined + for (const callback of callbacks) { + try { + callback() + } catch (error) { + firstFailure ??= { error } + } + } + if (firstFailure) throw firstFailure.error +} diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index bf5ac1a913..4c01a49873 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -1,4 +1,5 @@ import { isTemporal } from '../utils' +import { getRuntimeReferenceIdentity } from '../query/runtime-reference-identity' import type { CompareOptions } from '../query/builder/types' // WeakMap to store stable IDs for objects @@ -17,6 +18,21 @@ function getObjectId(obj: object): number { return id } +/** + * Whether a value has no IEEE-754 natural order: `NaN`, or an invalid Date + * (whose timestamp is `NaN`). The query engine follows PostgreSQL float + * semantics for these values — they are all equal to one another and greater + * than every other (non-null) value — so the comparator and the WHERE + * evaluator treat them explicitly instead of letting `NaN` compare unequal to + * everything (which has no consistent order and cannot be indexed or sorted). + */ +export function isUnorderable(value: any): boolean { + return ( + (typeof value === `number` && Number.isNaN(value)) || + (value instanceof Date && Number.isNaN(value.getTime())) + ) +} + /** * Universal comparison function for all data types * Handles null/undefined, strings, arrays, dates, objects, and primitives @@ -30,6 +46,16 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { if (a == null) return nulls === `first` ? -1 : 1 if (b == null) return nulls === `first` ? 1 : -1 + // Handle NaN / invalid Dates. Following PostgreSQL float semantics, they are + // all equal and sort greater than every other non-null value. This keeps the + // order total (NaN would otherwise compare equal to everything), so such + // values can be sorted and stored in tree-based indexes. + const aUnordered = isUnorderable(a) + const bUnordered = isUnorderable(b) + if (aUnordered && bUnordered) return 0 + if (aUnordered) return 1 + if (bUnordered) return -1 + // if a and b are both strings, compare them based on locale if (typeof a === `string` && typeof b === `string`) { if (opts.stringSort === `locale`) { @@ -55,14 +81,22 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { return a.getTime() - b.getTime() } - // If both are Temporal objects of the same type, compare by string representation + // If both are Temporal objects, use compareTemporalValues for correct semantic ordering if (isTemporal(a) && isTemporal(b)) { - const aStr = a.toString() - const bStr = b.toString() - if (aStr < bStr) return -1 - if (aStr > bStr) return 1 - return 0 + return compareTemporalValues(a, b) + } + + // Symbols have identity but no built-in order: relational comparison throws. + // A stable runtime ID gives tree indexes a total order while preserving + // equality only for the same symbol. + const aIsSymbol = typeof a === `symbol` + const bIsSymbol = typeof b === `symbol` + if (aIsSymbol && bIsSymbol) { + if (a === b) return 0 + return getRuntimeReferenceIdentity(a)[2] - getRuntimeReferenceIdentity(b)[2] } + if (aIsSymbol) return 1 + if (bIsSymbol) return -1 // If at least one of the values is an object, use stable IDs for comparison const aIsObject = typeof a === `object` @@ -121,9 +155,15 @@ export const defaultComparator = makeComparator({ stringSort: `locale`, }) -/** - * Compare two Uint8Arrays for content equality - */ +/** Include host Buffers when the current realm has a different Uint8Array. */ +export function isUint8Array(value: unknown): value is Uint8Array { + return ( + value instanceof Uint8Array || + (typeof Buffer !== `undefined` && value instanceof Buffer) + ) +} + +/** Compare two Uint8Arrays for content equality. */ function areUint8ArraysEqual(a: Uint8Array, b: Uint8Array): boolean { if (a.byteLength !== b.byteLength) { return false @@ -136,20 +176,26 @@ function areUint8ArraysEqual(a: Uint8Array, b: Uint8Array): boolean { return true } -/** - * Threshold for normalizing Uint8Arrays to string representations. - * Arrays larger than this will use reference equality to avoid memory overhead. - * 128 bytes is enough for common ID formats (ULIDs are 16 bytes, UUIDs are 16 bytes) - * while avoiding excessive string allocation for large binary data. - */ -const UINT8ARRAY_NORMALIZE_THRESHOLD = 128 +const NORMALIZED_KEY_PREFIX = `\u0000tanstack-db:` + +function normalizedKey(kind: string, value: string): string { + return `${NORMALIZED_KEY_PREFIX}${kind}:${value}` +} + +function normalizeBinary(value: Uint8Array): string { + let bytes = `` + for (let index = 0; index < value.byteLength; index++) { + bytes += String.fromCharCode(value[index]!) + } + return normalizedKey(`binary`, bytes) +} /** * Sentinel value representing undefined in normalized form. * This allows distinguishing between "start from beginning" (undefined parameter) * and "start from the key undefined" (actual undefined value in the tree). */ -export const UNDEFINED_SENTINEL = `__TS_DB_BTREE_UNDEFINED_VALUE__` +export const UNDEFINED_SENTINEL = normalizedKey(`undefined`, ``) /** * Normalize a value for comparison and Map key usage @@ -160,29 +206,31 @@ export const UNDEFINED_SENTINEL = `__TS_DB_BTREE_UNDEFINED_VALUE__` * for BTree index operations that need to distinguish undefined values. */ export function normalizeValue(value: any): any { + if (typeof value === `string`) { + return value.startsWith(NORMALIZED_KEY_PREFIX) + ? normalizedKey(`string`, value) + : value + } + + if (typeof value !== `object` || value === null) { + return value + } + if (value instanceof Date) { return value.getTime() } if (isTemporal(value)) { - return `__temporal__${value[Symbol.toStringTag]}__${value.toString()}` + return normalizedKey( + `temporal`, + `${value[Symbol.toStringTag]}:${value.toString()}`, + ) } // Normalize Uint8Arrays/Buffers to a string representation for Map key usage // This enables content-based equality for binary data like ULIDs - const isUint8Array = - (typeof Buffer !== `undefined` && value instanceof Buffer) || - value instanceof Uint8Array - - if (isUint8Array) { - // Only normalize small arrays to avoid memory overhead for large binary data - if (value.byteLength <= UINT8ARRAY_NORMALIZE_THRESHOLD) { - // Convert to a string representation that can be used as a Map key - // Use a special prefix to avoid collisions with user strings - return `__u8__${Array.from(value).join(`,`)}` - } - // For large arrays, fall back to reference equality - // Users working with large binary data should use a derived key if needed + if (isUint8Array(value)) { + return normalizeBinary(value) } return value @@ -201,6 +249,13 @@ export function normalizeForBTree(value: any): any { return normalizeValue(value) } +/** + * Compare values using the equality semantics used by Map keys. + */ +export function areSameValueZeroEqual(a: unknown, b: unknown): boolean { + return a === b || (Number.isNaN(a) && Number.isNaN(b)) +} + /** * Converts the `UNDEFINED_SENTINEL` back to `undefined`. * Needed such that the sentinel is converted back to `undefined` before comparison. @@ -212,6 +267,68 @@ export function denormalizeUndefined(value: any): any { return value } +// Cached map from Symbol.toStringTag → static compare function (null = none defined). +// Populated lazily on first encounter of each Temporal type so we never access +// `.constructor` more than once per type, and dispatch is keyed on the already- +// computed brand tag rather than on the constructor itself. +const temporalCompareByTag = new Map< + string, + ((a: unknown, b: unknown) => number) | null +>() + +/** + * Compare two Temporal values of the same type, returning -1, 0, or 1. + * + * Dispatch is keyed on `Symbol.toStringTag` (the brand already checked by + * `isTemporal`) rather than `a.constructor`, making it robust across realms + * and resistant to a shadowed `constructor` property. Types without a static + * `.compare` (e.g. `PlainMonthDay`) throw rather than fall back to string + * comparison, matching Temporal's design intent. + * + * Callers must ensure both arguments are Temporal objects; mixed types throw. + */ +export function compareTemporalValues(a: unknown, b: unknown): number { + const aTag = (a as Record)[Symbol.toStringTag] as string + const bTag = (b as Record)[Symbol.toStringTag] as string + if (aTag !== bTag) { + throw new TypeError( + `Cannot order Temporal values of different types: ${aTag} vs ${bTag}`, + ) + } + let compare = temporalCompareByTag.get(aTag) + if (compare === undefined) { + const fn = ( + (a as { constructor: unknown }).constructor as { + compare?: (x: unknown, y: unknown) => number + } + ).compare + compare = typeof fn === `function` ? fn : null + temporalCompareByTag.set(aTag, compare) + } + if (compare === null) { + throw new TypeError(`${aTag} has no defined ordering`) + } + return compare(a, b) +} + +/** + * Order two non-null values, returning -1, 0, or 1. + * + * Temporal types intentionally throw from `valueOf` to prevent silent + * miscomparison via the native relational operators — delegate to + * `compareTemporalValues` for them. For everything else (numbers, strings, + * Dates via `valueOf`, etc.) the native operators do the right thing. + * + * Callers must handle null/undefined themselves — this helper assumes both + * arguments are non-null. + */ +export function compareValues(a: unknown, b: unknown): number { + if (isTemporal(a) && isTemporal(b)) { + return compareTemporalValues(a, b) + } + return (a as any) < (b as any) ? -1 : (a as any) > (b as any) ? 1 : 0 +} + /** * Compare two values for equality, with special handling for Uint8Arrays and Buffers */ @@ -222,15 +339,8 @@ export function areValuesEqual(a: any, b: any): boolean { } // Check for Uint8Array/Buffer comparison - const aIsUint8Array = - (typeof Buffer !== `undefined` && a instanceof Buffer) || - a instanceof Uint8Array - const bIsUint8Array = - (typeof Buffer !== `undefined` && b instanceof Buffer) || - b instanceof Uint8Array - // If both are Uint8Arrays, compare by content - if (aIsUint8Array && bIsUint8Array) { + if (isUint8Array(a) && isUint8Array(b)) { return areUint8ArraysEqual(a, b) } diff --git a/packages/db/src/utils/cursor.ts b/packages/db/src/utils/cursor.ts index 322a374703..b2aca0a994 100644 --- a/packages/db/src/utils/cursor.ts +++ b/packages/db/src/utils/cursor.ts @@ -1,78 +1,92 @@ -import { and, eq, gt, lt, or } from '../query/builder/functions.js' +import { + and, + eq, + gt, + gte, + isNull, + isUndefined, + lt, + not, + or, +} from '../query/builder/functions.js' import { Value } from '../query/ir.js' -import type { BasicExpression, OrderBy } from '../query/ir.js' +import type { BasicExpression, OrderBy, OrderByClause } from '../query/ir.js' -/** - * Builds a cursor expression for paginating through ordered results. - * For multi-column orderBy, creates a composite cursor that respects all columns. - * - * For [col1 ASC, col2 DESC] with values [v1, v2], produces: - * or( - * gt(col1, v1), // col1 > v1 - * and(eq(col1, v1), lt(col2, v2)) // col1 = v1 AND col2 < v2 (DESC) - * ) - * - * This creates a precise cursor that works with composite indexes on the backend. - * - * @param orderBy - The order-by clauses defining sort columns and directions - * @param values - The cursor values corresponding to each order-by column - * @returns A filter expression for rows after the cursor position, or undefined if empty - */ +function isNullish( + expression: OrderByClause[`expression`], +): BasicExpression { + return or(isNull(expression), isUndefined(expression)) +} + +function followsBoundary( + clause: OrderByClause, + value: unknown, +): BasicExpression { + const nullish = isNullish(clause.expression) + if (value == null) { + return clause.compareOptions.nulls === `first` + ? not(nullish) + : new Value(false) + } + + const operator = clause.compareOptions.direction === `asc` ? gt : lt + const comparison = operator(clause.expression, new Value(value)) + return clause.compareOptions.nulls === `last` + ? or(comparison, nullish) + : comparison +} + +/** Build a single-column cursor; multi-column queries use prefix loading. */ export function buildCursor( orderBy: OrderBy, values: Array, ): BasicExpression | undefined { - if (values.length === 0 || orderBy.length === 0) { - return undefined + if (values.length === 0) return undefined + if (orderBy.length !== 1 || values.length !== 1) { + throw new Error(`Only single-column cursors are supported`) } + return followsBoundary(orderBy[0]!, values[0]) +} - // For single column, just use simple gt/lt - if (orderBy.length === 1) { - const { expression, compareOptions } = orderBy[0]! - const operator = compareOptions.direction === `asc` ? gt : lt - return operator(expression, new Value(values[0])) - } - - // For multi-column, build the composite cursor: - // or( - // gt(col1, v1), - // and(eq(col1, v1), gt(col2, v2)), - // and(eq(col1, v1), eq(col2, v2), gt(col3, v3)), - // ... - // ) - const clauses: Array> = [] - - for (let i = 0; i < orderBy.length && i < values.length; i++) { - const clause = orderBy[i]! - const value = values[i] - - // Build equality conditions for all previous columns - const eqConditions: Array> = [] - for (let j = 0; j < i; j++) { - const prevClause = orderBy[j]! - const prevValue = values[j] - eqConditions.push(eq(prevClause.expression, new Value(prevValue))) - } - - // Add the comparison for the current column (respecting direction) - const operator = clause.compareOptions.direction === `asc` ? gt : lt - const comparison = operator(clause.expression, new Value(value)) - - if (eqConditions.length === 0) { - // First column: just the comparison - clauses.push(comparison) - } else { - // Subsequent columns: and(eq(prev...), comparison) - // We need to spread into and() which expects at least 2 args - const allConditions = [...eqConditions, comparison] - clauses.push(allConditions.reduce((acc, cond) => and(acc, cond))) - } +/** Build the equality range that closes the first ordered boundary term. */ +export function buildCursorCurrent( + orderBy: OrderBy, + values: ReadonlyArray, +): BasicExpression | undefined { + const { expression } = orderBy[0] ?? {} + if (!expression || values.length === 0) return undefined + const value = values[0] + if (value == null) return isNullish(expression) + if (value instanceof Date) { + if (!Number.isFinite(value.getTime())) return undefined + return and( + gte(expression, new Value(value)), + lt(expression, new Value(new Date(value.getTime() + 1))), + ) } + if (typeof value === `object`) return undefined + return eq(expression, new Value(value)) +} - // Combine all clauses with OR - if (clauses.length === 1) { - return clauses[0]! +/** + * Whether the public predicate IR can express this boundary's comparison. + * Unsupported values must use an unbounded fetch rather than a provider order + * that may differ from the local comparator. + */ +export function canExpressCursorOrder( + orderBy: OrderBy, + values: ReadonlyArray, +): boolean { + if (orderBy.length !== 1 || values.length !== 1) return false + const value = values[0] + if (value == null) return false + if (value instanceof Date) return Number.isFinite(value.getTime()) + if (typeof value === `string`) { + return orderBy[0]!.compareOptions.stringSort === `lexical` } - // Use reduce to combine with or() which expects exactly 2 args - return clauses.reduce((acc, clause) => or(acc, clause)) + return ( + (typeof value === `number` && Number.isFinite(value)) || + typeof value === `bigint` || + typeof value === `boolean` + ) } diff --git a/packages/db/src/utils/error.ts b/packages/db/src/utils/error.ts new file mode 100644 index 0000000000..17842241a5 --- /dev/null +++ b/packages/db/src/utils/error.ts @@ -0,0 +1,8 @@ +export const normalizeError = (error: unknown): Error => { + try { + if (error instanceof Error) return error + return new Error(String(error)) + } catch { + return new Error(`Unknown error`) + } +} diff --git a/packages/db/src/utils/get-or-create.ts b/packages/db/src/utils/get-or-create.ts new file mode 100644 index 0000000000..313a2683ba --- /dev/null +++ b/packages/db/src/utils/get-or-create.ts @@ -0,0 +1,16 @@ +/** Lazily initialize a map entry; undefined denotes an absent value. */ +export function getOrCreate( + entries: { + get: (key: K) => V | undefined + set: (key: K, value: V) => unknown + }, + key: K, + create: () => V, +): V { + let value = entries.get(key) + if (value === undefined) { + value = create() + entries.set(key, value) + } + return value +} diff --git a/packages/db/src/utils/index-optimization.ts b/packages/db/src/utils/index-optimization.ts index 81b111af56..92eee5a376 100644 --- a/packages/db/src/utils/index-optimization.ts +++ b/packages/db/src/utils/index-optimization.ts @@ -18,8 +18,9 @@ import { DEFAULT_COMPARE_OPTIONS } from '../utils.js' import { ReverseIndex } from '../indexes/reverse-index.js' import { hasVirtualPropPath } from '../virtual-props.js' +import { makeComparator } from './comparison.js' import type { CompareOptions } from '../query/builder/types.js' -import type { IndexInterface, IndexOperation } from '../indexes/base-index.js' +import type { IndexOperation, IndexReader } from '../indexes/base-index.js' import type { BasicExpression } from '../query/ir.js' import type { CollectionLike } from '../types.js' @@ -29,6 +30,13 @@ import type { CollectionLike } from '../types.js' export interface OptimizationResult { canOptimize: boolean matchingKeys: Set + /** + * Whether `matchingKeys` is exactly the set of keys matching the expression. + * When `false`, the keys are a superset of the true result (some conditions + * could not be served by an index) and each row must be re-checked against + * the full expression before being included in the result. + */ + isExact: boolean } /** @@ -38,7 +46,7 @@ export function findIndexForField( collection: CollectionLike, fieldPath: Array, compareOptions?: CompareOptions, -): IndexInterface | undefined { +): IndexReader | undefined { if (hasVirtualPropPath(fieldPath)) { return undefined } @@ -94,6 +102,97 @@ export function unionSets(sets: Array>): Set { return result } +/** + * Whether a value can be matched exactly by an index lookup, i.e. the index + * result for it is not a superset that the caller must re-filter. + * + * Only `null`/`undefined` are inexact: the WHERE evaluator's three-valued logic + * makes any comparison against them UNKNOWN, yet a BTree index stores and + * returns rows with nullish keys (they sort to the nulls end), so a result that + * could include such rows must be re-filtered. + * + * `NaN` and invalid Dates are exact: under the engine's PostgreSQL float + * semantics they are equal to themselves and ordered (greatest non-null value), + * so the evaluator and the index agree on them and no re-filtering is needed. + */ +function isExactComparisonValue(value: unknown): boolean { + return value != null +} + +/** + * Whether the collection orders strings using locale collation. + * + * Under `stringSort: 'locale'` a BTree string index orders values with + * `localeCompare`, but the WHERE evaluator compares strings with JS relational + * operators (code-point order). For range predicates these orders disagree + * (e.g. `'ö' > 'z'` is true in JS but `'ö'` sorts before `'z'` under locale + * `en`), so an index range lookup can omit matching rows. Such omissions cannot + * be recovered by re-filtering, so locale-backed string range predicates must + * not be index-optimized. + */ +function usesLocaleStringSort(collection: CollectionLike): boolean { + const opts = { ...DEFAULT_COMPARE_OPTIONS, ...collection.compareOptions } + return opts.stringSort === `locale` +} + +/** + * Whether a range predicate on this operand would use an index ordering that + * differs from the WHERE evaluator's relational operators, so an index range + * lookup could omit genuine matches that re-filtering cannot recover. + * + * The evaluator compares with JS relational operators (extended with + * PostgreSQL float semantics for `NaN`/invalid Dates). That order matches the + * index comparator for numbers, booleans, bigints, lexically-sorted strings, + * Dates (valid, ordered by time; invalid, ordered as the greatest value) and + * `NaN`. It diverges for locale-sorted strings (localeCompare vs code-point + * order) and for arrays, plain objects, Temporal values and typed arrays + * (recursive/identity ordering vs string coercion). + * + * Note: `null`/`undefined` operands are not handled here — those are superset + * cases handled by re-filtering ({@link isExactComparisonValue}). + */ +function isRangeOrderingDivergent( + value: unknown, + collection: CollectionLike, +): boolean { + switch (typeof value) { + case `number`: + case `bigint`: + case `boolean`: + return false + case `string`: + return usesLocaleStringSort(collection) + case `symbol`: + return true + case `object`: { + if (value === null) return false + // Dates order consistently with the evaluator: valid Dates by time, and + // invalid Dates as the greatest value under PostgreSQL float semantics. + return !(value instanceof Date) + } + default: + return false + } +} + +/** + * Whether a range predicate (gt/gte/lt/lte) on this operand can be safely + * served by the given index: the operand's domain must order the same way the + * index does, and the index itself must support trustworthy range traversal + * (no custom comparator). + */ +function canRangeOptimize( + value: unknown, + index: IndexReader, + collection: CollectionLike, +): boolean { + return ( + !isRangeOrderingDivergent(value, collection) && + index.supportsRangeOptimization && + (index.canOptimizeRangeFor?.(value) ?? true) + ) +} + /** * Optimizes a query expression using available indexes to find matching keys */ @@ -134,7 +233,7 @@ function optimizeQueryRecursive( } } - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } /** @@ -167,6 +266,14 @@ export function canOptimizeExpression< return false } +/** + * Result of compound range optimization, including which AND arguments + * were covered by the range query so the caller can process the rest. + */ +interface CompoundRangeResult extends OptimizationResult { + coveredArgIndices: Set +} + /** * Optimizes compound range queries on the same field * Example: WHERE age > 5 AND age < 10 @@ -177,9 +284,14 @@ function optimizeCompoundRangeQuery< >( expression: BasicExpression, collection: CollectionLike, -): OptimizationResult { +): CompoundRangeResult { if (expression.type !== `func` || expression.args.length < 2) { - return { canOptimize: false, matchingKeys: new Set() } + return { + canOptimize: false, + matchingKeys: new Set(), + isExact: false, + coveredArgIndices: new Set(), + } } // Group range operations by field @@ -188,11 +300,12 @@ function optimizeCompoundRangeQuery< Array<{ operation: `gt` | `gte` | `lt` | `lte` value: any + argIndex: number }> >() // Collect all range operations from AND arguments - for (const arg of expression.args) { + for (const [argIndex, arg] of expression.args.entries()) { if (arg.type === `func` && [`gt`, `gte`, `lt`, `lte`].includes(arg.name)) { const rangeOp = arg as any if (rangeOp.args.length === 2) { @@ -238,7 +351,7 @@ function optimizeCompoundRangeQuery< if (!fieldOperations.has(fieldKey)) { fieldOperations.set(fieldKey, []) } - fieldOperations.get(fieldKey)!.push({ operation, value }) + fieldOperations.get(fieldKey)!.push({ operation, value, argIndex }) } } } @@ -250,55 +363,114 @@ function optimizeCompoundRangeQuery< const fieldPath = fieldKey.split(`.`) const index = findIndexForField(collection, fieldPath) + // Only collapse this field into a range query when every bound's domain + // orders the same way the index does and the index supports trustworthy + // range traversal. Otherwise the index may omit matching rows that + // re-filtering cannot recover, so leave the field for a full scan. + if ( + index && + operations.some((op) => !canRangeOptimize(op.value, index, collection)) + ) { + continue + } + if (index && index.supports(`gt`) && index.supports(`lt`)) { - // Build range query options + // Compare values with the same semantics the index uses (dates, + // locale strings, ...), in ascending order since bounds are about + // value order regardless of the index direction + const compare = makeComparator({ + ...DEFAULT_COMPARE_OPTIONS, + ...collection.compareOptions, + direction: `asc`, + }) + + // Build range query options, keeping the strictest bound on each + // side: a larger lower bound (or smaller upper bound) wins, and at + // equal values the exclusive operation wins over the inclusive one. + // `hasFromBound`/`hasToBound` track whether a bound was selected, + // separately from the bound value (which may legitimately be falsy). let from: any = undefined let to: any = undefined + let hasFromBound = false + let hasToBound = false let fromInclusive = true let toInclusive = true + // A comparison against null/undefined is never true, but in an index + // nullish values sort to the nulls end, so a range query cannot + // represent such a bound. Track it and force a re-filter instead of + // claiming the result is exact. (NaN/invalid Dates are ordered and + // comparable under PostgreSQL semantics, so they are real bounds.) + let hasNonComparableBound = false for (const { operation, value } of operations) { + if (!isExactComparisonValue(value)) { + hasNonComparableBound = true + continue + } switch (operation) { case `gt`: - if (from === undefined || value > from) { + case `gte`: { + const cmp = hasFromBound ? compare(value, from) : 1 + if (cmp > 0) { from = value + hasFromBound = true + fromInclusive = operation === `gte` + } else if (cmp === 0 && operation === `gt`) { fromInclusive = false } break - case `gte`: - if (from === undefined || value > from) { - from = value - fromInclusive = true - } - break + } case `lt`: - if (to === undefined || value < to) { + case `lte`: { + const cmp = hasToBound ? compare(value, to) : -1 + if (cmp < 0) { to = value + hasToBound = true + toInclusive = operation === `lte` + } else if (cmp === 0 && operation === `lt`) { toInclusive = false } break - case `lte`: - if (to === undefined || value < to) { - to = value - toInclusive = true - } - break + } } } - const matchingKeys = (index as any).rangeQuery({ - from, - to, - fromInclusive, - toInclusive, - }) - - return { canOptimize: true, matchingKeys } + // Only pass the bounds that were selected: rangeQuery distinguishes + // an absent bound (open-ended) from an explicitly provided one + const rangeOptions: Record = {} + if (hasFromBound) { + rangeOptions.from = from + rangeOptions.fromInclusive = fromInclusive + } + if (hasToBound) { + rangeOptions.to = to + rangeOptions.toInclusive = toInclusive + } + const matchingKeys = (index as any).rangeQuery(rangeOptions) + + return { + canOptimize: true, + matchingKeys, + // The range result is exact only when it cannot include rows with a + // nullish indexed value (which a comparison would reject but the + // index returns, as they sort as the smallest key). That requires a + // non-nullish lower bound to exclude them: without `hasFromBound` + // the range is open at the bottom and captures those rows, and a + // non-comparable bound value (`hasNonComparableBound`) can never + // bound them out. + isExact: hasFromBound && !hasNonComparableBound, + coveredArgIndices: new Set(operations.map((op) => op.argIndex)), + } } } } - return { canOptimize: false, matchingKeys: new Set() } + return { + canOptimize: false, + matchingKeys: new Set(), + isExact: false, + coveredArgIndices: new Set(), + } } /** @@ -312,7 +484,7 @@ function optimizeSimpleComparison< collection: CollectionLike, ): OptimizationResult { if (expression.type !== `func` || expression.args.length !== 2) { - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } const leftArg = expression.args[0]! @@ -362,15 +534,46 @@ function optimizeSimpleComparison< // Check if the index supports this operation if (!index.supports(indexOperation)) { - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } + } + + // A range op can only use the index when the operand's domain orders the + // same way the index does and the index supports trustworthy traversal. + // Otherwise the index may omit matching rows, which re-filtering cannot + // recover, so fall back to a full scan. + if ( + (operation === `gt` || + operation === `gte` || + operation === `lt` || + operation === `lte`) && + !canRangeOptimize(queryValue, index, collection) + ) { + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } const matchingKeys = index.lookup(indexOperation, queryValue) - return { canOptimize: true, matchingKeys } + + // A comparison against a nullish value is never true, but BTree indexes + // store and return rows with nullish keys (they sort to the nulls end). + // Determine whether the index result is exact or a superset that the + // caller must re-filter: + // - eq/gt/gte: a nullish query value matches nothing while the index + // still returns nullish-keyed rows -> inexact. A non-nullish lower + // bound (gt/gte) excludes those bottom-sorted rows, so they stay exact. + // - lt/lte: the open lower bound always includes nullish-keyed rows, + // so the result is conservatively inexact. + // NaN/invalid Dates are exact here: under PostgreSQL float semantics the + // evaluator and the index agree on them (equal to self, greatest). + const isExact = + operation === `lt` || operation === `lte` + ? false + : isExactComparisonValue(queryValue) + + return { canOptimize: true, matchingKeys, isExact } } } - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } /** @@ -412,22 +615,41 @@ function optimizeAndExpression( collection: CollectionLike, ): OptimizationResult { if (expression.type !== `func` || expression.args.length < 2) { - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } // First, try to optimize compound range queries on the same field + // (e.g. age > 5 AND age < 10 becomes a single range query) const compoundRangeResult = optimizeCompoundRangeQuery(expression, collection) - if (compoundRangeResult.canOptimize) { - return compoundRangeResult - } + const coveredArgIndices = compoundRangeResult.canOptimize + ? compoundRangeResult.coveredArgIndices + : new Set() const results: Array> = [] + if (compoundRangeResult.canOptimize) { + results.push(compoundRangeResult) + } - // Try to optimize each part, keep the optimizable ones - for (const arg of expression.args) { + // Try to optimize the remaining conjuncts, keep the optimizable ones. + // Conjuncts that cannot use an index make the result inexact: the + // intersection is then a superset of the true result and must be + // re-filtered against the full expression by the caller. The compound + // range result may itself be inexact (e.g. a null/undefined bound). + let allConjunctsExact = !compoundRangeResult.canOptimize + ? true + : compoundRangeResult.isExact + for (const [argIndex, arg] of expression.args.entries()) { + if (coveredArgIndices.has(argIndex)) { + continue + } const result = optimizeQueryRecursive(arg, collection) if (result.canOptimize) { results.push(result) + if (!result.isExact) { + allConjunctsExact = false + } + } else { + allConjunctsExact = false } } @@ -435,10 +657,14 @@ function optimizeAndExpression( // Use intersectSets utility for AND logic const allMatchingSets = results.map((r) => r.matchingKeys) const intersectedKeys = intersectSets(allMatchingSets) - return { canOptimize: true, matchingKeys: intersectedKeys } + return { + canOptimize: true, + matchingKeys: intersectedKeys, + isExact: allConjunctsExact, + } } - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } /** @@ -464,27 +690,31 @@ function optimizeOrExpression( collection: CollectionLike, ): OptimizationResult { if (expression.type !== `func` || expression.args.length < 2) { - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } const results: Array> = [] - // Try to optimize each part, keep the optimizable ones + // Every disjunct must be optimizable: rows matched only by a disjunct + // that cannot use an index would be missing from the union, and no + // post-filtering can recover them. In that case fall back to a full scan. for (const arg of expression.args) { const result = optimizeQueryRecursive(arg, collection) - if (result.canOptimize) { - results.push(result) + if (!result.canOptimize) { + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } + results.push(result) } - if (results.length > 0) { - // Use unionSets utility for OR logic - const allMatchingSets = results.map((r) => r.matchingKeys) - const unionedKeys = unionSets(allMatchingSets) - return { canOptimize: true, matchingKeys: unionedKeys } + // Use unionSets utility for OR logic + const allMatchingSets = results.map((r) => r.matchingKeys) + const unionedKeys = unionSets(allMatchingSets) + return { + canOptimize: true, + matchingKeys: unionedKeys, + // An inexact (superset) disjunct makes the union a superset as well + isExact: results.every((r) => r.isExact), } - - return { canOptimize: false, matchingKeys: new Set() } } /** @@ -498,8 +728,9 @@ function canOptimizeOrExpression< return false } - // If any argument can be optimized, we can gain some speedup - return expression.args.some((arg) => canOptimizeExpression(arg, collection)) + // Every disjunct must be optimizable, otherwise the union would miss + // rows matched only by the non-optimizable disjuncts + return expression.args.every((arg) => canOptimizeExpression(arg, collection)) } /** @@ -513,7 +744,7 @@ function optimizeInArrayExpression< collection: CollectionLike, ): OptimizationResult { if (expression.type !== `func` || expression.args.length !== 2) { - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } const fieldArg = expression.args[0]! @@ -528,11 +759,17 @@ function optimizeInArrayExpression< const values = (arrayArg as any).value const index = findIndexForField(collection, fieldPath) + // A nullish or NaN member can never be matched by `IN` (a comparison + // against null/undefined/NaN is never true), but the index would still + // return rows with such an indexed value. When the list contains one of + // those the result is a superset that the caller must re-filter. + const isExact = values.every((value: any) => isExactComparisonValue(value)) + if (index) { // Check if the index supports IN operation if (index.supports(`in`)) { const matchingKeys = index.lookup(`in`, values) - return { canOptimize: true, matchingKeys } + return { canOptimize: true, matchingKeys, isExact } } else if (index.supports(`eq`)) { // Fallback to multiple equality lookups const matchingKeys = new Set() @@ -542,12 +779,12 @@ function optimizeInArrayExpression< matchingKeys.add(key) } } - return { canOptimize: true, matchingKeys } + return { canOptimize: true, matchingKeys, isExact } } } } - return { canOptimize: false, matchingKeys: new Set() } + return { canOptimize: false, matchingKeys: new Set(), isExact: false } } /** diff --git a/packages/db/src/utils/type-guards.ts b/packages/db/src/utils/type-guards.ts index 4c54d80773..a6cc0d803a 100644 --- a/packages/db/src/utils/type-guards.ts +++ b/packages/db/src/utils/type-guards.ts @@ -1,3 +1,11 @@ +export function isPlainObject( + value: unknown, +): value is Record { + if (value === null || typeof value !== `object`) return false + const prototype = Object.getPrototypeOf(value) + return prototype === Object.prototype || prototype === null +} + /** * Type guard to check if a value is promise-like (has a `.then` method) * @param value - The value to check diff --git a/packages/db/src/utils/uuid.ts b/packages/db/src/utils/uuid.ts new file mode 100644 index 0000000000..45875a459f --- /dev/null +++ b/packages/db/src/utils/uuid.ts @@ -0,0 +1,45 @@ +/** + * Returns a RFC 4122 version 4 UUID. + * + * Prefers `crypto.randomUUID()` when available. In non-secure browser contexts + * (e.g. a dev server accessed via a LAN IP over HTTP) `crypto.randomUUID` is + * `undefined`, so this falls back to building a UUIDv4 from + * `crypto.getRandomValues`. Throws if neither API is available. + * + * See https://github.com/TanStack/db/issues/1541. + */ +export function safeRandomUUID(): string { + const c: Crypto | undefined = + typeof globalThis !== `undefined` ? (globalThis as any).crypto : undefined + + if (c && typeof c.randomUUID === `function`) { + return c.randomUUID() + } + + if (c && typeof c.getRandomValues === `function`) { + const bytes = c.getRandomValues(new Uint8Array(16)) + // Per RFC 4122 §4.4: set version (4) and variant (10xx) bits. + bytes[6] = (bytes[6]! & 0x0f) | 0x40 + bytes[8] = (bytes[8]! & 0x3f) | 0x80 + + const hex: Array = [] + for (let i = 0; i < 16; i++) { + hex.push(bytes[i]!.toString(16).padStart(2, `0`)) + } + return ( + hex.slice(0, 4).join(``) + + `-` + + hex.slice(4, 6).join(``) + + `-` + + hex.slice(6, 8).join(``) + + `-` + + hex.slice(8, 10).join(``) + + `-` + + hex.slice(10, 16).join(``) + ) + } + + throw new Error( + `No secure random number generator available: neither crypto.randomUUID nor crypto.getRandomValues is defined in this environment.`, + ) +} diff --git a/packages/db/src/virtual-props.ts b/packages/db/src/virtual-props.ts index 3205d31c2d..3f600a5008 100644 --- a/packages/db/src/virtual-props.ts +++ b/packages/db/src/virtual-props.ts @@ -37,7 +37,7 @@ export type VirtualOrigin = 'local' | 'remote' * // Accessing virtual properties on a row * const user = collection.get('user-1') * if (user.$synced) { - * console.log('Confirmed by backend') + * console.log('No pending local optimistic writes for this row') * } * if (user.$origin === 'local') { * console.log('Created/modified locally') @@ -47,7 +47,7 @@ export type VirtualOrigin = 'local' | 'remote' * @example * ```typescript * // Using virtual properties in queries - * const confirmedOrders = createLiveQueryCollection({ + * const ordersWithoutLocalWrites = createLiveQueryCollection({ * query: (q) => q * .from({ order: orders }) * .where(({ order }) => eq(order.$synced, true)) @@ -58,10 +58,15 @@ export interface VirtualRowProps< TKey extends string | number = string | number, > { /** - * Whether this row reflects confirmed state from the backend. + * Whether this row currently has no pending local optimistic writes. * - * - `true`: Row is confirmed by the backend (no pending optimistic mutations) - * - `false`: Row has pending optimistic mutations that haven't been confirmed + * - `true`: No pending local optimistic mutation currently affects this row + * - `false`: One or more pending local optimistic mutations currently affect this row + * + * This is local mutation status. It does not prove that a backend has uploaded, + * confirmed, or read back the row. If you need backend-confirmed status, keep + * your mutation function pending until that backend observation has happened, + * or expose adapter-specific status. * * For local-only collections (no sync), this is always `true`. * For live query collections, this is passed through from the source collection. @@ -152,34 +157,6 @@ export function hasVirtualProps( ) } -/** - * Creates virtual properties for a row in a source collection. - * - * This is the internal function used by collections to add virtual properties - * to rows when emitting change messages. - * - * @param key - The row's key - * @param collectionId - The collection's ID - * @param isSynced - Whether the row is synced (not optimistic) - * @param origin - Whether the change was local or remote - * @returns Virtual properties object to merge with the row - * - * @internal - */ -export function createVirtualProps( - key: TKey, - collectionId: string, - isSynced: boolean, - origin: VirtualOrigin, -): VirtualRowProps { - return { - $synced: isSynced, - $origin: origin, - $key: key, - $collectionId: collectionId, - } -} - /** * Enriches a row with virtual properties using the "add-if-missing" pattern. * @@ -221,39 +198,6 @@ export function enrichRowWithVirtualProps< } as WithVirtualProps } -/** - * Computes aggregate virtual properties for a group of rows. - * - * For aggregates: - * - `$synced`: true if ALL rows in the group are synced; false if ANY row is optimistic - * - `$origin`: 'local' if ANY row in the group is local; otherwise 'remote' - * - * @param rows - The rows in the group - * @param groupKey - The group key - * @param collectionId - The collection ID - * @returns Virtual properties for the aggregate row - * - * @internal - */ -export function computeAggregateVirtualProps( - rows: Array>>, - groupKey: TKey, - collectionId: string, -): VirtualRowProps { - // $synced = true only if ALL rows are synced (false if ANY is optimistic) - const allSynced = rows.every((row) => row.$synced ?? true) - - // $origin = 'local' if ANY row is local (consistent with "local influence" semantics) - const hasLocal = rows.some((row) => row.$origin === 'local') - - return { - $synced: allSynced, - $origin: hasLocal ? 'local' : 'remote', - $key: groupKey, - $collectionId: collectionId, - } -} - /** * List of virtual property names for iteration and checking. * @internal diff --git a/packages/db/tests/basic-index-work.test.ts b/packages/db/tests/basic-index-work.test.ts new file mode 100644 index 0000000000..438a9bfede --- /dev/null +++ b/packages/db/tests/basic-index-work.test.ts @@ -0,0 +1,97 @@ +import { describe, expect, it, vi } from 'vitest' +import { BasicIndex } from '../src/indexes/basic-index' +import { PropRef } from '../src/query/ir' + +describe(`BasicIndex removal work`, () => { + it.each([1, 4])( + `searches only the comparator group of size %s`, + (groupSize) => { + const size = 1024 + let comparisons = 0 + let scanned = 0 + const index = new BasicIndex( + 1, + new PropRef([`value`]), + undefined, + { + compareFn: (a: number, b: number) => { + comparisons++ + return Math.floor(a / groupSize) - Math.floor(b / groupSize) + }, + }, + ) + for (let value = 0; value < size; value++) index.add(value, { value }) + const target = size - groupSize + const findIndex = Array.prototype.findIndex + const spy = vi + .spyOn(Array.prototype, `findIndex`) + .mockImplementation(function ( + this: Array, + predicate, + thisArg, + ) { + return findIndex.call(this, (value, position, array) => { + scanned++ + return predicate.call(thisArg, value, position, array) + }) + }) + comparisons = 0 + try { + index.remove(target, { value: target }) + } finally { + spy.mockRestore() + } + expect(scanned + comparisons).toBeLessThanOrEqual( + Math.ceil(Math.log2(size)) + groupSize + 1, + ) + expect(index.lookup(`eq`, target).size).toBe(0) + for (let value = target + 1; value < size; value++) { + expect(index.lookup(`eq`, value)).toEqual(new Set([value])) + } + }, + ) +}) + +describe(`BasicIndex page filtering work`, () => { + it.each( + [30, 3000, 100000].flatMap((size) => + [false, true].flatMap((reverse) => + [1, 3].map((stride) => ({ size, reverse, stride })), + ), + ), + )( + `filters only visited keys: $size rows, reverse=$reverse, stride=$stride`, + ({ size, reverse, stride }) => { + const index = new BasicIndex(1, new PropRef([`value`])) + const rows = Array.from({ length: size }, (_, id) => ({ + id, + value: id % 3, + })) + // Deliberately insert backwards; insertion order is not key order. + for (const row of [...rows].reverse()) index.add(row.id, row) + const ordered = rows + .slice() + .sort((a, b) => a.value - b.value || a.id - b.id) + if (reverse) ordered.reverse() + let calls = 0 + const accept = (key: number) => Math.floor(key / 3) % stride === 0 + const expected = ordered + .filter((row) => accept(row.id)) + .slice(0, 10) + .map((row) => row.id) + const filter = (key: number) => { + calls++ + return accept(key) + } + const actual = reverse + ? index.takeReversedFromEnd(10, filter) + : index.takeFromStart(10, filter) + expect(actual).toEqual(expected) + const visits = + expected.length === 10 + ? ordered.findIndex((row) => row.id === expected[9]) + 1 + : size + expect(calls).toBe(visits) + }, + ) +}) diff --git a/packages/db/tests/btree-index-undefined-values.test.ts b/packages/db/tests/btree-index-undefined-values.test.ts index 11e29690c6..252737e0ab 100644 --- a/packages/db/tests/btree-index-undefined-values.test.ts +++ b/packages/db/tests/btree-index-undefined-values.test.ts @@ -16,6 +16,7 @@ import { createLiveQueryCollection } from '../src/query/live-query-collection.js import { eq } from '../src/query/builder/functions.js' import { BTreeIndex } from '../src/indexes/btree-index.js' import { PropRef } from '../src/query/ir.js' +import { orderedEntriesArray, valueMapData } from './utils' import type { Collection } from '../src/collection/index.js' interface TaskItem { @@ -195,7 +196,7 @@ describe(`BTreeIndex - undefined value handling`, () => { index.add(`num2`, { value: 2 }) index.add(`num0`, { value: 0 }) - const ordered = index.orderedEntriesArray + const ordered = orderedEntriesArray(index) expect(ordered[0]![0]).toBe(undefined) expect(ordered[0]![1]).toContain(`undef`) }) @@ -206,7 +207,7 @@ describe(`BTreeIndex - undefined value handling`, () => { index.add(`undef`, { name: undefined }) index.add(`str2`, { name: `banana` }) - expect(index.orderedEntriesArray[0]![0]).toBe(undefined) + expect(orderedEntriesArray(index)[0]![0]).toBe(undefined) }) it(`should handle mixed undefined and null values`, () => { @@ -248,6 +249,24 @@ describe(`BTreeIndex - undefined value handling`, () => { expect(withoutFrom.size).toBe(3) }) + it(`should not drop the minimum key when an upper-only range is exclusive on the (absent) lower bound`, () => { + // When no `from` bound is provided, `fromInclusive` must not cause the + // smallest key to be excluded: there is no lower bound to exclude + // against. Only an explicitly provided exclusive lower bound should + // drop its boundary value. + const index = createIndex(`value`) + index.add(`a`, { value: 1 }) + index.add(`b`, { value: 5 }) + index.add(`c`, { value: 10 }) + + const result = index.rangeQuery({ to: 10, fromInclusive: false }) + + expect(result.size).toBe(3) + expect(result).toContain(`a`) + expect(result).toContain(`b`) + expect(result).toContain(`c`) + }) + it(`should handle range query from undefined to undefined`, () => { const index = createIndex(`value`) index.add(`a`, { value: undefined }) @@ -294,7 +313,7 @@ describe(`BTreeIndex - undefined value handling`, () => { ) expect(undefinedComparisons.length).toBeGreaterThan(0) - const ordered = index.orderedEntriesArray + const ordered = orderedEntriesArray(index) expect(ordered[0]![0]).toBe(1) expect(ordered[1]![0]).toBe(undefined) }) @@ -390,7 +409,7 @@ describe(`BTreeIndex - undefined value handling`, () => { index.add(`a`, { value: undefined }) index.add(`b`, { value: 1 }) - const mapData = index.valueMapData + const mapData = valueMapData(index) expect(mapData.has(undefined)).toBe(true) expect(mapData.has(`__TS_DB_BTREE_UNDEFINED_VALUE__`)).toBe(false) diff --git a/packages/db/tests/btree-index-work.test.ts b/packages/db/tests/btree-index-work.test.ts new file mode 100644 index 0000000000..d907df5694 --- /dev/null +++ b/packages/db/tests/btree-index-work.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from 'vitest' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { PropRef } from '../src/query/ir.js' + +describe(`BTree exact-bucket ownership work`, () => { + it.each( + [300, 100000].flatMap((size) => + [1, 2].map((groupSize) => ({ size, groupSize })), + ), + )( + `reuses comparator buckets for $size keys with $groupSize exact values per position`, + ({ size, groupSize }) => { + let comparisons = 0 + const index = new BTreeIndex( + 1, + new PropRef([`value`]), + undefined, + { + compareFn: (a: number, b: number) => { + comparisons++ + return Math.floor(a / groupSize) - Math.floor(b / groupSize) + }, + }, + ) + const distinct = 3 * groupSize + // Keep one owner of every exact value throughout the measured batch. + for (let value = 0; value < distinct; value++) + index.add(-value - 1, { value }) + comparisons = 0 + for (let key = 0; key < size; key++) + index.add(key, { value: key % distinct }) + const insertComparisons = comparisons + comparisons = 0 + for (let key = 0; key < size; key++) + index.remove(key, { value: key % distinct }) + const removeComparisons = comparisons + expect(index.keyCount).toBe(distinct) + for (let value = 0; value < distinct; value++) { + expect(index.lookup(`eq`, value)).toEqual(new Set([-value - 1])) + } + expect(insertComparisons).toBe(0) + expect(removeComparisons).toBe(0) + }, + ) +}) diff --git a/packages/db/tests/btree-map-oracle.test.ts b/packages/db/tests/btree-map-oracle.test.ts new file mode 100644 index 0000000000..351b23f491 --- /dev/null +++ b/packages/db/tests/btree-map-oracle.test.ts @@ -0,0 +1,63 @@ +import { describe, expect, it } from 'vitest' +import { BTree } from '../src/utils/btree.js' + +describe(`BTree Map oracle`, () => { + it(`matches a Map oracle under random insert/delete/overwrite with small nodes`, () => { + let seed = 12345 + const rnd = () => + (seed = (seed * 1103515245 + 12345) & 0x7fffffff) / 0x7fffffff + for (let round = 0; round < 40; round++) { + const tree = new BTree( + (a, b) => a - b, + 4 + Math.floor(rnd() * 5), + ) + const oracle = new Map() + for (let step = 0; step < 3000; step++) { + const key = Math.floor(rnd() * 200) + const op = rnd() + if (op < 0.5) { + const val = { v: step } + const added = tree.set(key, val) + expect(added).toBe(!oracle.has(key)) + oracle.set(key, val) + } else if (op < 0.85) { + const deleted = tree.delete(key) + expect(deleted).toBe(oracle.delete(key)) + } else if (op < 0.9) { + tree.clear() + oracle.clear() + } else { + expect(tree.get(key)).toBe(oracle.get(key)) + expect(tree.has(key)).toBe(oracle.has(key)) + } + if (step % 97 === 0) { + const sorted = [...oracle.keys()].sort((a, b) => a - b) + expect(tree.size).toBe(oracle.size) + expect(tree.minKey()).toBe(sorted[0]) + expect(tree.maxKey()).toBe(sorted[sorted.length - 1]) + const seen: Array = [] + if (sorted.length) + tree.forRange( + sorted[0]!, + sorted[sorted.length - 1]!, + true, + (k, v) => { + seen.push(k) + expect(v).toBe(oracle.get(k)) + }, + ) + expect(seen).toEqual(sorted) + const probe = Math.floor(rnd() * 200) + const higher = sorted.find((k) => k > probe) + const lower = [...sorted].reverse().find((k) => k < probe) + expect(tree.nextHigherPair(probe)?.[0]).toBe(higher) + expect(tree.nextLowerPair(probe)?.[0]).toBe(lower) + expect(tree.nextHigherPair(undefined)?.[0]).toBe(sorted[0]) + expect(tree.nextLowerPair(undefined)?.[0]).toBe( + sorted[sorted.length - 1], + ) + } + } + } + }) +}) diff --git a/packages/db/tests/cleanup-queue.test.ts b/packages/db/tests/cleanup-queue.test.ts index d95ddc9f77..8fb2698b54 100644 --- a/packages/db/tests/cleanup-queue.test.ts +++ b/packages/db/tests/cleanup-queue.test.ts @@ -1,15 +1,17 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { CleanupQueue } from '../src/collection/cleanup-queue' +import { resetCleanupQueue } from './utils' describe('CleanupQueue', () => { beforeEach(() => { vi.useFakeTimers() - CleanupQueue.resetInstance() + resetCleanupQueue() }) afterEach(() => { + resetCleanupQueue() + vi.restoreAllMocks() vi.useRealTimers() - CleanupQueue.resetInstance() }) it('batches setTimeout creations across multiple synchronous schedules', async () => { @@ -58,6 +60,8 @@ describe('CleanupQueue', () => { queue.cancel('key1') + expect(vi.getTimerCount()).toBe(0) + vi.advanceTimersByTime(1000) expect(cb1).not.toHaveBeenCalled() }) diff --git a/packages/db/tests/collection-auto-index.test.ts b/packages/db/tests/collection-auto-index.test.ts index 4fdaac0127..7d004d4946 100644 --- a/packages/db/tests/collection-auto-index.test.ts +++ b/packages/db/tests/collection-auto-index.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' import { CollectionConfigurationError } from '../src/errors' import { createCollection } from '../src/collection/index.js' import { @@ -197,7 +197,7 @@ describe(`Collection Auto-Indexing`, () => { await collection.stateWhenReady() - expect(() => collection.createIndex((row) => row.age)).toThrow( + expect(() => collection.createIndex((item) => item.age)).toThrow( CollectionConfigurationError, ) }) @@ -250,13 +250,60 @@ describe(`Collection Auto-Indexing`, () => { subscription.unsubscribe() }) + it(`indexes symbol-valued equality fields without falling back to a scan`, async () => { + type SymbolItem = { id: string; group: symbol } + const firstGroup = Symbol(`first`) + const secondGroup = Symbol(`second`) + const symbolRow = createSingleRowRefProxy() + const warning = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const collection = createCollection({ + getKey: (item) => item.id, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `one`, group: firstGroup } }) + write({ type: `insert`, value: { id: `two`, group: secondGroup } }) + commit() + markReady() + }, + }, + }) + + try { + await collection.stateWhenReady() + const changes: Array = [] + const subscription = collection.subscribeChanges( + (items) => changes.push(...items), + { + includeInitialState: true, + whereExpression: eq(symbolRow.group, firstGroup), + }, + ) + + expect(collection.indexes.size).toBe(1) + expect(changes.map(({ value }) => value.id)).toEqual([`one`]) + expect(warning).not.toHaveBeenCalled() + subscription.unsubscribe() + } finally { + warning.mockRestore() + await collection.cleanup() + } + }) + it(`should create auto-indexes for transformed fields of subqueries when autoIndex is "eager"`, async () => {}) - it(`should not create duplicate auto-indexes for the same field`, async () => { + it(`should not create duplicate auto-indexes when locale options are omitted`, async () => { const autoIndexCollection = createCollection({ getKey: (item) => item.id, autoIndex: `eager`, defaultIndexType: BTreeIndex, + defaultStringCollation: { + stringSort: `locale`, + localeOptions: { sensitivity: undefined }, + }, startSync: true, sync: { sync: ({ begin, write, commit, markReady }) => { diff --git a/packages/db/tests/collection-change-events.test.ts b/packages/db/tests/collection-change-events.test.ts index 085af31f08..dee88c6d59 100644 --- a/packages/db/tests/collection-change-events.test.ts +++ b/packages/db/tests/collection-change-events.test.ts @@ -1,6 +1,9 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' -import { currentStateAsChanges } from '../src/collection/change-events.js' +import { + createFilterFunctionFromExpression, + currentStateAsChanges, +} from '../src/collection/change-events.js' import { Func, PropRef, Value } from '../src/query/ir.js' import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' import { BTreeIndex } from '../src/indexes/btree-index.js' @@ -13,6 +16,26 @@ interface TestUser { status: `active` | `inactive` } +it(`treats predicate evaluation failures as nonmatches`, () => { + const filter = createFilterFunctionFromExpression( + new Func(`eq`, [new PropRef([`status`]), new Value(`active`)]), + ) + const row = { + id: `1`, + name: `Ada`, + age: 36, + score: 100, + status: `active`, + } as TestUser + Object.defineProperty(row, `status`, { + get: () => { + throw new Error(`predicate evaluation failed`) + }, + }) + + expect(filter(row)).toBe(false) +}) + describe(`currentStateAsChanges`, () => { let mockSync: ReturnType diff --git a/packages/db/tests/collection-cleanup-restart-oracle.test.ts b/packages/db/tests/collection-cleanup-restart-oracle.test.ts new file mode 100644 index 0000000000..b131d67c0f --- /dev/null +++ b/packages/db/tests/collection-cleanup-restart-oracle.test.ts @@ -0,0 +1,208 @@ +import { describe, expect, it } from 'vitest' +import { createCollection, createLiveQueryCollection } from '../src' +import type { SyncConfig } from '../src/types' + +type Row = { id: number; rank: number } +const cleanupError = { + name: `CollectionStateError`, + message: expect.stringContaining(`after cleanup() completes`), +} + +const scenarios = ([`abort`, `release`] as const).flatMap((boundary) => + [false, true].flatMap((nestedCleanup) => + [1, 2].map((attempts) => ({ boundary, nestedCleanup, attempts })), + ), +) + +describe(`Collection cleanup admission oracle`, () => { + it.each(scenarios)( + `rejects restart without creating replacement ownership: %j`, + async ({ boundary, nestedCleanup, attempts }) => { + let ops!: Parameters[`sync`]>[0] + let loads = 0 + let releases = 0 + let armed = false + const errors: Array = [] + const cleanups: Array> = [] + const reenter = () => { + if (!armed) return + armed = false + for (let i = 0; i < attempts; i++) { + if (nestedCleanup) cleanups.push(live.cleanup()) + try { + live.startSyncImmediate() + errors.push(undefined) + } catch (error) { + errors.push(error) + } + } + } + const source = createCollection({ + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (methods) => { + ops = methods + methods.begin() + methods.write({ type: `insert`, value: { id: 1, rank: 1 } }) + methods.commit() + methods.markReady() + return { + loadSubset: ({ signal }) => { + loads++ + signal?.addEventListener(`abort`, () => { + if (boundary === `abort`) reenter() + }) + return true + }, + unloadSubset: () => { + releases++ + if (boundary === `release`) reenter() + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => q.from({ row: source })) + try { + await live.preload() + armed = true + await live.cleanup() + await Promise.all(cleanups) + expect(armed).toBe(false) + expect(errors).toHaveLength(attempts) + for (const error of errors) expect(error).toMatchObject(cleanupError) + expect(loads).toBe(1) + expect(releases).toBe(1) + expect(source.subscriberCount).toBe(0) + expect(live.status).toBe(`cleaned-up`) + + // The rejected calls must not poison a later, ordinary restart. + await live.preload() + expect(loads).toBe(2) + expect(source.subscriberCount).toBe(1) + ops.begin() + ops.write({ type: `update`, value: { id: 1, rank: 2 } }) + ops.commit() + expect(live.status).toBe(`ready`) + expect(live.get(1)?.rank).toBe(2) + } finally { + armed = false + await live.cleanup() + await source.cleanup() + } + }, + ) + + it.each([`start`, `preload`] as const)( + `rejects %s from adapter cleanup but allows another collection to start`, + async (method) => { + let starts = 0 + let armed = false + let observed: Promise | undefined + const peer = createCollection({ + getKey: (row) => row.id, + sync: { sync: ({ markReady }) => markReady() }, + }) + const source = createCollection({ + getKey: (row) => row.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + starts++ + begin() + write({ type: `insert`, value: { id: 1, rank: starts } }) + commit() + markReady() + return () => { + if (!armed) return + armed = false + void source.cleanup() + try { + const result = + method === `start` + ? source.startSyncImmediate() + : source.preload() + observed = Promise.resolve(result).then( + () => undefined, + (error: unknown) => error, + ) + } catch (error) { + observed = Promise.resolve(error) + } + peer.startSyncImmediate() + } + }, + }, + }) + try { + await source.preload() + armed = true + await source.cleanup() + expect(await observed).toMatchObject(cleanupError) + expect(starts).toBe(1) + expect(peer.status).toBe(`ready`) + await source.preload() + expect(starts).toBe(2) + expect(source.get(1)?.rank).toBe(2) + } finally { + armed = false + await source.cleanup() + await peer.cleanup() + } + }, + ) + + it.each( + ([`event`, `await`] as const).flatMap((boundary) => + [false, true].map((liveQuery) => ({ boundary, liveQuery })), + ), + )( + `admits restart at the completed cleanup boundary: %j`, + async ({ boundary, liveQuery }) => { + let starts = 0 + let armed = false + let ops!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: (row) => row.id, + sync: { + sync: (methods) => { + ops = methods + const { begin, write, commit, markReady } = methods + starts++ + begin() + write({ type: `insert`, value: { id: 1, rank: starts } }) + commit() + markReady() + }, + }, + }) + const collection = liveQuery + ? createLiveQueryCollection((q) => q.from({ row: source })) + : source + const off = collection.on(`status:change`, ({ status }) => { + if (armed && boundary === `event` && status === `cleaned-up`) { + armed = false + collection.startSyncImmediate() + } + }) + try { + await collection.preload() + armed = true + await collection.cleanup() + if (boundary === `await`) collection.startSyncImmediate() + expect(starts).toBe(liveQuery ? 1 : 2) + expect(collection.status).toBe(`ready`) + expect(collection.get(1)?.rank).toBe(liveQuery ? 1 : 2) + ops.begin() + ops.write({ type: `update`, value: { id: 1, rank: 3 } }) + ops.commit() + expect(collection.get(1)?.rank).toBe(3) + } finally { + armed = false + off() + if (liveQuery) await collection.cleanup() + await source.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/collection-errors.test.ts b/packages/db/tests/collection-errors.test.ts index f4bdae52d8..1422e89f9b 100644 --- a/packages/db/tests/collection-errors.test.ts +++ b/packages/db/tests/collection-errors.test.ts @@ -2,9 +2,11 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' import { CollectionInErrorStateError, + CollectionIsInErrorStateError, InvalidCollectionStatusTransitionError, SyncCleanupError, } from '../src/errors' +import type { SyncConfig } from '../src/types' describe(`Collection Error Handling`, () => { let originalQueueMicrotask: typeof queueMicrotask @@ -26,6 +28,107 @@ describe(`Collection Error Handling`, () => { }) describe(`Cleanup Error Handling`, () => { + it.each([false, true])( + `finishes adapter resource cleanup after a failure, already released=%s`, + async (releaseBeforeThrow) => { + const resources = new Set() + const failure = new Error(`adapter cleanup interrupted`) + let attempts = 0 + const collection = createCollection<{ id: string }>({ + getKey: (row) => row.id, + sync: { + sync: ({ markReady }) => { + const resource = {} + resources.add(resource) + markReady() + return () => { + attempts++ + if (attempts === 1) { + if (releaseBeforeThrow) resources.delete(resource) + throw failure + } + resources.delete(resource) + } + }, + }, + }) + collection.startSyncImmediate() + try { + expect(resources.size).toBe(1) + await collection.cleanup() + expect(collection.status).toBe(`cleaned-up`) + expect(resources.size).toBe(releaseBeforeThrow ? 0 : 1) + expect(mockQueueMicrotask).toHaveBeenCalledTimes(1) + expect(() => mockQueueMicrotask.mock.calls[0]![0]()).toThrow( + SyncCleanupError, + ) + + // The Collection's public status alone does not prove resource release. + await collection.cleanup() + expect(resources.size).toBe(0) + expect(attempts).toBe(2) + expect(mockQueueMicrotask).toHaveBeenCalledTimes(1) + } finally { + await collection.cleanup() + } + }, + ) + + it.each([false, true])( + `retries failed cleanup only before replacement, restart after rejection=%s`, + async (restart) => { + const failure = new Error(`cleanup failed`) + const cleanups: Array = [] + let session = 0 + const collection = createCollection<{ id: string }>({ + id: `failed-cleanup-session-${restart}`, + getKey: ({ id }) => id, + sync: { + sync: ({ markReady }) => { + const currentSession = session++ + markReady() + return () => { + cleanups.push(currentSession) + if (cleanups.length !== 1) return + if (restart) { + void collection.cleanup() + expect(() => collection.startSyncImmediate()).toThrow( + `after cleanup() completes`, + ) + } + throw failure + } + }, + }, + }) + + collection.startSyncImmediate() + try { + await collection.cleanup() + expect(cleanups).toEqual([0]) + expect(mockQueueMicrotask).toHaveBeenCalledTimes(1) + let reportedError: unknown + try { + mockQueueMicrotask.mock.calls[0]![0]() + } catch (error) { + reportedError = error + } + expect(reportedError).toBeInstanceOf(SyncCleanupError) + expect((reportedError as Error).cause).toBe(failure) + + expect(session).toBe(1) + if (restart) collection.startSyncImmediate() + await collection.cleanup() + expect(cleanups).toEqual(restart ? [0, 1] : [0, 0]) + await collection.cleanup() + expect(cleanups).toHaveLength(2) + expect(mockQueueMicrotask).toHaveBeenCalledTimes(1) + } finally { + await collection.cleanup() + } + }, + ) + it(`should complete cleanup successfully even when sync cleanup function throws an Error`, async () => { const collection = createCollection<{ id: string; name: string }>({ id: `error-test-collection`, @@ -246,7 +349,186 @@ describe(`Collection Error Handling`, () => { }) }) + describe(`Sync Session Isolation`, () => { + it(`preserves an asynchronous sync error and removes its first-ready waiter`, async () => { + let markError: (error?: unknown) => void = () => { + throw new Error(`Sync has not started`) + } + const collection = createCollection<{ id: string }>({ + id: `rejected-preload-waiter`, + getKey: (item) => item.id, + startSync: false, + sync: { + sync: (sync) => { + markError = sync.markError + }, + }, + }) + + const preload = collection.preload() + const stateWhenReady = collection.stateWhenReady() + const arrayWhenReady = collection.toArrayWhenReady() + expect(collection._lifecycle.onFirstReadyCallbacks).toHaveLength(1) + + const syncError = new Error(`Asynchronous sync failed exactly`) + markError(syncError) + await expect(preload).rejects.toBe(syncError) + await expect(stateWhenReady).rejects.toBe(syncError) + await expect(arrayWhenReady).rejects.toBe(syncError) + await expect(collection.preload()).rejects.toBe(syncError) + expect(collection._lifecycle.onFirstReadyCallbacks).toHaveLength(0) + + await collection.cleanup() + }) + + it(`uses the generic state error when asynchronous sync supplies no cause`, async () => { + let markError: (error?: unknown) => void = () => { + throw new Error(`Sync has not started`) + } + const collection = createCollection<{ id: string }>({ + id: `generic-asynchronous-sync-error`, + getKey: (item) => item.id, + startSync: false, + sync: { + sync: (sync) => { + markError = sync.markError + }, + }, + }) + + const preload = collection.preload() + markError() + + await expect(preload).rejects.toBeInstanceOf( + CollectionIsInErrorStateError, + ) + await collection.cleanup() + }) + + it(`ignores an error callback retained after cleanup`, async () => { + let markError: () => void = () => { + throw new Error(`Sync has not started`) + } + const collection = createCollection<{ id: string }>({ + id: `stale-error-after-cleanup`, + getKey: (item) => item.id, + startSync: false, + sync: { + sync: (sync) => { + markError = sync.markError + }, + }, + }) + const preload = collection.preload() + const cancelled = expect(preload).rejects.toMatchObject({ + name: `AbortError`, + }) + await collection.cleanup() + await cancelled + markError() + + expect(collection.status).toBe(`cleaned-up`) + }) + + it(`ignores an error callback retained by an earlier sync session`, async () => { + const sessions: Array<{ + markError: () => void + markReady: () => void + }> = [] + const collection = createCollection<{ id: string }>({ + id: `stale-error-after-restart`, + getKey: (item) => item.id, + startSync: false, + sync: { + sync: ({ markError, markReady }) => { + sessions.push({ markError, markReady }) + }, + }, + }) + + await collection.cleanup() + const preload = collection.preload() + expect(sessions).toHaveLength(1) + const first = sessions[0]! + const cancelled = expect(preload).rejects.toMatchObject({ + name: `AbortError`, + }) + await collection.cleanup() + await cancelled + const restartedPreload = collection.preload() + expect(sessions).toHaveLength(2) + const second = sessions[1]! + + first.markError() + expect(collection.status).toBe(`loading`) + + second.markReady() + await restartedPreload + expect(collection.status).toBe(`ready`) + }) + + it(`ignores transaction callbacks retained by an earlier sync session`, async () => { + type Item = { id: string } + type SyncMethods = Parameters[`sync`]>[0] + const sessions: Array = [] + const collection = createCollection({ + id: `stale-transaction-after-restart`, + getKey: (item) => item.id, + startSync: false, + sync: { + sync: (sync) => { + sessions.push(sync) + }, + }, + }) + + const firstPreload = collection.preload() + const first = sessions[0]! + const cancelled = expect(firstPreload).rejects.toMatchObject({ + name: `AbortError`, + }) + await collection.cleanup() + await cancelled + + const secondPreload = collection.preload() + const second = sessions[1]! + first.begin() + first.write({ type: `insert`, value: { id: `stale` } }) + first.commit() + first.markReady() + + expect(collection.status).toBe(`loading`) + expect(collection.get(`stale`)).toBeUndefined() + + second.begin() + second.write({ type: `insert`, value: { id: `current` } }) + second.commit() + second.markReady() + await secondPreload + + expect(collection.status).toBe(`ready`) + expect(collection.get(`current`)).toMatchObject({ id: `current` }) + }) + }) + describe(`Operation Validation Errors`, () => { + it(`preserves a synchronous sync startup error`, async () => { + const startupError = new Error(`Sync initialization failed exactly`) + const collection = createCollection<{ id: string }>({ + id: `exact-startup-error`, + getKey: (item) => item.id, + startSync: false, + sync: { + sync: () => { + throw startupError + }, + }, + }) + + await expect(collection.preload()).rejects.toBe(startupError) + expect(collection.status).toBe(`error`) + }) + it(`should throw helpful errors when trying to use operations on error status collection`, async () => { const collection = createCollection<{ id: string; name: string }>({ id: `error-status-test`, @@ -446,6 +728,9 @@ describe(`Collection Error Handling`, () => { expect(() => collectionImpl._lifecycle.validateStatusTransition(`error`, `idle`), ).not.toThrow() + expect(() => + collectionImpl._lifecycle.validateStatusTransition(`error`, `ready`), + ).not.toThrow() // Valid transitions from cleaned-up (allow restart) expect(() => diff --git a/packages/db/tests/collection-events.test.ts b/packages/db/tests/collection-events.test.ts index 3e3221b43a..4f55482e2e 100644 --- a/packages/db/tests/collection-events.test.ts +++ b/packages/db/tests/collection-events.test.ts @@ -1,8 +1,19 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' +import { EventEmitter } from '../src/event-emitter.js' import { BTreeIndex } from '../src/indexes/btree-index.js' import type { Collection } from '../src/collection/index.js' +class TestEventEmitter extends EventEmitter<{ event: { id: number } }> { + emit(id: number): void { + this.emitInner(`event`, { id }) + } + + clear(): void { + this.clearListeners() + } +} + describe(`Collection Events System`, () => { let collection: Collection let mockSync: ReturnType @@ -47,6 +58,88 @@ describe(`Collection Events System`, () => { status: `loading`, }) }) + + it(`stops an obsolete status event after a listener changes status`, () => { + const genericEvents: Array<{ + previousStatus: string + status: string + current: string + }> = [] + const loadingEvents: Array = [] + collection.on(`status:change`, ({ status }) => { + if (status === `loading`) collection._lifecycle.markReady() + }) + collection.on(`status:change`, ({ previousStatus, status }) => { + genericEvents.push({ + previousStatus, + status, + current: collection.status, + }) + }) + collection.on(`status:loading`, ({ status }) => { + loadingEvents.push(status) + }) + + collection.startSyncImmediate() + + expect(genericEvents).toEqual([ + { + previousStatus: `loading`, + status: `ready`, + current: `ready`, + }, + ]) + expect(loadingEvents).toEqual([]) + }) + + it.each([`generic`, `specific`] as const)( + `keeps cross-channel order under %s-listener ABA reentry`, + (reentryEvent) => { + const trace: Array = [] + let reentered = false + const reenter = () => { + if (reentered) return + reentered = true + collection._lifecycle.setStatus(`error`) + collection._lifecycle.setStatus(`idle`) + collection._lifecycle.setStatus(`loading`) + } + if (reentryEvent === `generic`) { + collection.on(`status:change`, ({ status }) => { + if (status === `loading`) reenter() + }) + } else { + collection.on(`status:loading`, reenter) + } + collection.on(`status:change`, ({ previousStatus, status }) => { + trace.push( + `generic:${previousStatus}->${status}:${collection.status}`, + ) + }) + collection.on(`status:loading`, () => { + trace.push(`specific:loading:${collection.status}`) + }) + + collection.startSyncImmediate() + + expect(trace).toEqual( + reentryEvent === `generic` + ? [ + `generic:loading->error:error`, + `generic:error->idle:idle`, + `generic:idle->loading:loading`, + `specific:loading:loading`, + ] + : [ + `generic:idle->loading:loading`, + `generic:loading->error:error`, + `generic:error->idle:idle`, + `generic:idle->loading:loading`, + `specific:loading:loading`, + ], + ) + }, + ) }) describe(`Subscriber Count Change Events`, () => { @@ -256,6 +349,169 @@ describe(`Collection Events System`, () => { unsubscribe() }) + + it(`removes a once listener before invoking a throwing callback`, () => { + const emitter = new TestEventEmitter() + const failure = new Error(`once listener failed`) + const deferredMicrotasks: Array = [] + const queueMicrotaskSpy = vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => deferredMicrotasks.push(callback)) + const listener = vi.fn(() => { + throw failure + }) + + try { + emitter.once(`event`, listener) + emitter.emit(1) + emitter.emit(2) + + expect(listener).toHaveBeenCalledTimes(1) + expect(deferredMicrotasks).toHaveLength(1) + expect(() => deferredMicrotasks[0]!()).toThrow(failure) + } finally { + queueMicrotaskSpy.mockRestore() + } + }) + + it(`removes a pending once listener through off`, () => { + const emitter = new TestEventEmitter() + const calls: Array = [] + const onceListener = vi.fn(() => calls.push(`once`)) + emitter.on(`event`, () => { + calls.push(`off`) + emitter.off(`event`, onceListener) + }) + emitter.once(`event`, onceListener) + + emitter.emit(1) + + expect(calls).toEqual([`off`]) + expect(onceListener).not.toHaveBeenCalled() + }) + + it(`removes a pending once listener through its returned unsubscribe`, () => { + const emitter = new TestEventEmitter() + const onceListener = vi.fn() + const unsubscribe = emitter.once(`event`, onceListener) + + unsubscribe() + emitter.emit(1) + + expect(onceListener).not.toHaveBeenCalled() + }) + + it(`removes every pending once registration for the same callback`, () => { + const emitter = new TestEventEmitter() + const onceListener = vi.fn() + emitter.once(`event`, onceListener) + emitter.once(`event`, onceListener) + + emitter.off(`event`, onceListener) + emitter.emit(1) + + expect(onceListener).not.toHaveBeenCalled() + }) + + it(`does not treat an ordinary callback property as a once registration`, () => { + const emitter = new TestEventEmitter() + const claimedOnceCallback = vi.fn() + const ordinaryListener = Object.assign(vi.fn(), { + onceCallback: claimedOnceCallback, + }) + emitter.on(`event`, ordinaryListener) + + emitter.off(`event`, claimedOnceCallback) + emitter.emit(1) + + expect(ordinaryListener).toHaveBeenCalledOnce() + }) + + it(`removes a once listener before a reentrant emission`, () => { + const emitter = new TestEventEmitter() + const observed: Array = [] + emitter.once(`event`, ({ id }) => { + observed.push(id) + emitter.emit(2) + }) + + emitter.emit(1) + + expect(observed).toEqual([1]) + }) + + it(`visits a listener once when it removes and re-adds itself`, () => { + const emitter = new TestEventEmitter() + const observed: Array = [] + let readded = false + let unsubscribe = () => {} + const listener = ({ id }: { id: number }) => { + observed.push(id) + unsubscribe() + if (!readded) { + readded = true + unsubscribe = emitter.on(`event`, listener) + } + } + unsubscribe = emitter.on(`event`, listener) + + emitter.emit(1) + + expect(observed).toEqual([1]) + }) + + it(`defers a pending listener that is removed and re-added`, () => { + const emitter = new TestEventEmitter() + const observed: Array = [] + let replaced = false + const pending = ({ id }: { id: number }) => { + observed.push(`pending:${id}`) + } + let unsubscribePending = () => {} + emitter.on(`event`, ({ id }) => { + observed.push(`first:${id}`) + if (replaced) return + replaced = true + unsubscribePending() + unsubscribePending = emitter.on(`event`, pending) + }) + unsubscribePending = emitter.on(`event`, pending) + + emitter.emit(1) + expect(observed).toEqual([`first:1`]) + + emitter.emit(2) + expect(observed).toEqual([`first:1`, `first:2`, `pending:2`]) + }) + + it(`clears ordinary and once listeners together`, () => { + const emitter = new TestEventEmitter() + const ordinaryListener = vi.fn() + const onceListener = vi.fn() + emitter.on(`event`, ordinaryListener) + emitter.once(`event`, onceListener) + + emitter.clear() + emitter.emit(1) + + expect(ordinaryListener).not.toHaveBeenCalled() + expect(onceListener).not.toHaveBeenCalled() + }) + + it(`exposes the same pending-once removal law through Collection`, () => { + const calls: Array = [] + const onceListener = vi.fn(() => calls.push(`once`)) + collection.on(`status:change`, () => { + calls.push(`off`) + collection.off(`status:change`, onceListener) + }) + collection.once(`status:change`, onceListener) + + collection.startSyncImmediate() + + expect(calls).toEqual([`off`]) + expect(onceListener).not.toHaveBeenCalled() + }) }) describe(`Event Structure`, () => { diff --git a/packages/db/tests/collection-indexes.test.ts b/packages/db/tests/collection-indexes.test.ts index a441a5520d..0d8062d628 100644 --- a/packages/db/tests/collection-indexes.test.ts +++ b/packages/db/tests/collection-indexes.test.ts @@ -14,8 +14,20 @@ import { or, } from '../src/query/builder/functions' import { PropRef } from '../src/query/ir' +import { BasicIndex } from '../src/indexes/basic-index.js' import { BTreeIndex } from '../src/indexes/btree-index.js' -import { expectIndexUsage, stripVirtualProps, withIndexTracking } from './utils' +import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' +import { findIndexForField } from '../src/utils/index-optimization.js' +import { makeComparator } from '../src/utils/comparison.js' +import { + expectIndexUsage, + indexedKeysSet, + orderedEntriesArray, + orderedEntriesArrayReversed, + stripVirtualProps, + valueMapData, + withIndexTracking, +} from './utils' import type { Collection } from '../src/collection/index.js' import type { MutationFn, PendingMutation } from '../src/types' @@ -149,7 +161,7 @@ describe(`Collection Indexes`, () => { expect(index.id).toBeGreaterThan(0) expect(index.name).toBeUndefined() expect(index.expression.type).toBe(`ref`) - expect(index.indexedKeysSet.size).toBe(5) + expect(indexedKeysSet(index).size).toBe(5) }) it(`should create a named index`, () => { @@ -158,7 +170,75 @@ describe(`Collection Indexes`, () => { }) expect(index.name).toBe(`ageIndex`) - expect(index.indexedKeysSet.size).toBe(5) + expect(indexedKeysSet(index).size).toBe(5) + }) + + it(`should match compare options by collation semantics`, () => { + const index = collection.createIndex((row) => row.status) + + expect( + index.matchesCompareOptions({ + ...DEFAULT_COMPARE_OPTIONS, + locale: undefined, + localeOptions: undefined, + }), + ).toBe(true) + expect( + index.matchesCompareOptions({ + ...DEFAULT_COMPARE_OPTIONS, + localeOptions: { sensitivity: undefined }, + }), + ).toBe(true) + expect( + index.matchesCompareOptions({ + ...DEFAULT_COMPARE_OPTIONS, + direction: `desc`, + }), + ).toBe(true) + expect( + index.matchesCompareOptions({ + ...DEFAULT_COMPARE_OPTIONS, + locale: `de-DE`, + }), + ).toBe(false) + expect( + index.matchesCompareOptions({ + ...DEFAULT_COMPARE_OPTIONS, + localeOptions: { sensitivity: `base` }, + }), + ).toBe(false) + expect( + index.matchesCompareOptions({ + ...DEFAULT_COMPARE_OPTIONS, + nulls: `last`, + }), + ).toBe(false) + expect( + index.matchesCompareOptions({ + ...DEFAULT_COMPARE_OPTIONS, + stringSort: `lexical`, + }), + ).toBe(false) + }) + + it(`should reuse an index for equivalent locale identifiers`, () => { + const indexCompareOptions = { + ...DEFAULT_COMPARE_OPTIONS, + locale: `en-us`, + } + const index = collection.createIndex((row) => row.name, { + options: { + compareOptions: indexCompareOptions, + compareFn: makeComparator(indexCompareOptions), + }, + }) + + expect( + findIndexForField(collection, [`name`], { + ...DEFAULT_COMPARE_OPTIONS, + locale: `en-US`, + }), + ).toBe(index) }) it(`should create multiple indexes`, () => { @@ -166,15 +246,15 @@ describe(`Collection Indexes`, () => { const ageIndex = collection.createIndex((row) => row.age) expect(statusIndex.id).not.toBe(ageIndex.id) - expect(statusIndex.indexedKeysSet.size).toBe(5) - expect(ageIndex.indexedKeysSet.size).toBe(5) + expect(indexedKeysSet(statusIndex).size).toBe(5) + expect(indexedKeysSet(ageIndex).size).toBe(5) }) it(`should maintain ordered entries`, () => { const ageIndex = collection.createIndex((row) => row.age) // Ages should be ordered: 22, 25, 28, 30, 35 - const orderedAges = ageIndex.orderedEntriesArray.map(([age]) => age) + const orderedAges = orderedEntriesArray(ageIndex).map(([age]) => age) expect(orderedAges).toEqual([22, 25, 28, 30, 35]) }) @@ -182,10 +262,10 @@ describe(`Collection Indexes`, () => { const statusIndex = collection.createIndex((row) => row.status) // Should have 3 unique status values - expect(statusIndex.orderedEntriesArray.length).toBe(3) + expect(orderedEntriesArray(statusIndex).length).toBe(3) // "active" status should have 3 items - const activeKeys = statusIndex.valueMapData.get(`active`) + const activeKeys = valueMapData(statusIndex).get(`active`) expect(activeKeys?.size).toBe(3) }) @@ -193,10 +273,10 @@ describe(`Collection Indexes`, () => { const scoreIndex = collection.createIndex((row) => row.score) // Should include the item with undefined score - expect(scoreIndex.indexedKeysSet.size).toBe(5) + expect(indexedKeysSet(scoreIndex).size).toBe(5) // undefined should be first in ordered entries - const firstValue = scoreIndex.orderedEntriesArray[0]?.[0] + const firstValue = orderedEntriesArray(scoreIndex)[0]?.[0] expect(firstValue).toBeUndefined() }) }) @@ -625,6 +705,19 @@ describe(`Collection Indexes`, () => { }) }) + it(`should exclude the boundary value from greater than queries on dates`, () => { + // gt must be strict for date fields: Bob was created exactly on + // 2023-01-02, so only rows created strictly later may be returned. + collection.createIndex((row) => row.createdAt) + + const result = collection.currentStateAsChanges({ + where: gt(new PropRef([`createdAt`]), new Date(`2023-01-02`)), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Charlie`, `Diana`, `Eve`]) + }) + it(`should perform greater than or equal queries`, () => { withIndexTracking(collection, (tracker) => { const result = collection.currentStateAsChanges({ @@ -1179,6 +1272,675 @@ describe(`Collection Indexes`, () => { }) }) }) + + it(`should include rows matched by any OR condition when conditions mix indexed and non-indexed expressions`, () => { + // An OR query must return the union of rows matching each condition: + // eq(age, 25) matches Alice (age 25) + // gt(length(name), 6) matches Charlie (name length 7) + // `age` has an index while `length(name)` is a computed expression + // without one, but the chosen execution strategy must not change the + // result: both Alice and Charlie satisfy the OR and must be returned. + const result = collection.currentStateAsChanges({ + where: or( + eq(new PropRef([`age`]), 25), + gt(length(new PropRef([`name`])), 6), + ), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Alice`, `Charlie`]) + }) + + it(`should only return rows matching every AND condition when conditions mix indexed and non-indexed expressions`, () => { + // An AND query must return only the rows matching all conditions: + // eq(status, 'active') matches Alice, Charlie and Eve + // gt(length(name), 6) matches only Charlie (name length 7) + // `status` has an index while `length(name)` is a computed expression + // without one, but every condition must still be enforced: only + // Charlie satisfies both. + const result = collection.currentStateAsChanges({ + where: and( + eq(new PropRef([`status`]), `active`), + gt(length(new PropRef([`name`])), 6), + ), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Charlie`]) + }) + + it(`should apply the strictest lower bound when range conditions share the same value`, () => { + // gte(age, 25) AND gt(age, 25) reduces to age > 25: the strict + // comparison wins at the shared boundary, so Alice (age 25) must be + // excluded regardless of the order the conditions appear in. + const result = collection.currentStateAsChanges({ + where: and(gte(new PropRef([`age`]), 25), gt(new PropRef([`age`]), 25)), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Bob`, `Charlie`, `Diana`]) + }) + + it(`should apply the strictest upper bound when range conditions share the same value`, () => { + // lte(age, 30) AND lt(age, 30) reduces to age < 30: the strict + // comparison wins at the shared boundary, so Bob (age 30) must be + // excluded. + const result = collection.currentStateAsChanges({ + where: and(lte(new PropRef([`age`]), 30), lt(new PropRef([`age`]), 30)), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Alice`, `Diana`, `Eve`]) + }) + + it(`should apply the strictest bound for date ranges sharing the same value`, () => { + // Distinct Date instances representing the same point in time must be + // treated as equal values: gte(createdAt, jan2) AND gt(createdAt, jan2) + // reduces to createdAt > jan2, so Bob (created 2023-01-02) must be + // excluded. + collection.createIndex((row) => row.createdAt) + + const result = collection.currentStateAsChanges({ + where: and( + gte(new PropRef([`createdAt`]), new Date(`2023-01-02`)), + gt(new PropRef([`createdAt`]), new Date(`2023-01-02`)), + ), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Charlie`, `Diana`, `Eve`]) + }) + + it(`should enforce every AND condition when a range on one field is combined with conditions on other fields`, () => { + // An AND query that contains a compound range on one field plus a + // condition on another field must enforce all of them: + // gt(age, 24) AND lt(age, 36) matches Alice (25), Bob (30), + // Charlie (35) and Diana (28) + // eq(status, 'active') matches Alice, Charlie and Eve + // Only Alice and Charlie satisfy the full conjunction. + const result = collection.currentStateAsChanges({ + where: and( + gt(new PropRef([`age`]), 24), + lt(new PropRef([`age`]), 36), + eq(new PropRef([`status`]), `active`), + ), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Alice`, `Charlie`]) + }) + + it(`should match a full scan when a range condition uses an undefined bound`, () => { + // A comparison against `undefined` matches no rows (a comparison with + // null/undefined is never true), so `gt(score, undefined)` excludes + // every row and the whole AND must return nothing. The index-optimized + // path must agree with a plain full scan and not leak rows. + collection.createIndex((row) => row.score) + + const result = collection.currentStateAsChanges({ + where: and( + gt(new PropRef([`score`]), undefined), + lt(new PropRef([`score`]), 90), + ), + })! + + expect(result).toEqual([]) + }) + + it(`should not match rows with a missing value for an equality on undefined`, () => { + // An equality comparison against `undefined` is never true, so + // `eq(score, undefined)` must return no rows even though Eve has an + // undefined score. The index-optimized path must agree with a full + // predicate scan. + collection.createIndex((row) => row.score) + + const result = collection.currentStateAsChanges({ + where: eq(new PropRef([`score`]), undefined), + })! + + expect(result).toEqual([]) + }) + + it(`should ignore an undefined member when matching an IN list`, () => { + // A row only matches `IN` when its value equals one of the listed + // values; a comparison with `undefined` is never true. So + // `inArray(score, [undefined, 80])` must match only Bob (score 80) + // and must not match Eve (undefined score). + collection.createIndex((row) => row.score) + + const result = collection.currentStateAsChanges({ + where: inArray(new PropRef([`score`]), [undefined, 80]), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Bob`]) + }) + + it(`should not match rows with a missing value for a range comparison`, () => { + // A range comparison against a row with an undefined value is never + // true, so `lt(score, 85)` must match only Bob (score 80) and must + // not match Eve (undefined score). + collection.createIndex((row) => row.score) + + const result = collection.currentStateAsChanges({ + where: lt(new PropRef([`score`]), 85), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Bob`]) + }) + + it(`should not match rows with a missing value for an upper-bounded compound range`, () => { + // A compound range with only upper bounds (e.g. score <= 90) must not + // match a row with an undefined value, since a comparison against + // undefined is never true. Only Bob (80), Charlie (90) and Diana (85) + // satisfy `score <= 90`; Eve (undefined) must be excluded. + collection.createIndex((row) => row.score) + + const result = collection.currentStateAsChanges({ + where: and( + lte(new PropRef([`score`]), 90), + lte(new PropRef([`score`]), 95), + ), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`Bob`, `Charlie`, `Diana`]) + }) + + it(`should match a string range predicate using the same ordering as a full scan`, async () => { + // String comparisons in the WHERE evaluator use JS relational operators + // (code-point order), where `'ö' > 'z'` is true. A row named `ö` must + // therefore be returned by `name > 'z'`, even though a locale-collated + // index orders `ö` before `z`. The index-optimized result must agree + // with a full predicate scan. + const stringCollection = createCollection< + { id: string; name: string }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `1`, name: `apple` } }) + write({ type: `insert`, value: { id: `2`, name: `ö` } }) + commit() + markReady() + }, + }, + }) + await stringCollection.stateWhenReady() + stringCollection.createIndex((row) => row.name) + + const result = stringCollection.currentStateAsChanges({ + where: gt(new PropRef([`name`]), `z`), + })! + + const names = result.map((r) => r.value.name).sort() + expect(names).toEqual([`ö`]) + }) + + it(`should match a row with a NaN value for an equality on NaN`, async () => { + // Under PostgreSQL float semantics NaN is equal to itself, so + // `eq(score, NaN)` matches the NaN-valued row (and the index, which + // stores and returns it, agrees with a full scan). + const nanCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `1`, score: 5 } }) + write({ type: `insert`, value: { id: `2`, score: NaN } }) + commit() + markReady() + }, + }, + }) + await nanCollection.stateWhenReady() + nanCollection.createIndex((row) => row.score) + + const result = nanCollection.currentStateAsChanges({ + where: eq(new PropRef([`score`]), NaN), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`2`]) + }) + + it(`should match a row with a NaN value for an IN list containing NaN`, async () => { + // A row matches `IN` when its value equals a listed value. Under + // PostgreSQL float semantics NaN is equal to itself, so + // `inArray(score, [NaN, 5])` matches both the score-5 row and the + // NaN-valued row. + const nanCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `1`, score: 5 } }) + write({ type: `insert`, value: { id: `2`, score: NaN } }) + commit() + markReady() + }, + }, + }) + await nanCollection.stateWhenReady() + nanCollection.createIndex((row) => row.score) + + const result = nanCollection.currentStateAsChanges({ + where: inArray(new PropRef([`score`]), [NaN, 5]), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`1`, `2`]) + }) + + it(`should return array-valued rows for a range predicate consistently with a full scan`, async () => { + // Range predicates are evaluated with standard relational comparison, + // under which `[2] > [10]` is true (arrays compare as their string + // form). An index on an array-valued field must return the same rows as + // a full scan and must not drop this match. + const arrayCollection = createCollection< + { id: string; value: Array }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `1`, value: [2] } }) + commit() + markReady() + }, + }, + }) + await arrayCollection.stateWhenReady() + arrayCollection.createIndex((row) => row.value) + + const result = arrayCollection.currentStateAsChanges({ + where: gt(new PropRef([`value`]), [10]), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`1`]) + }) + + it(`should match symbol range predicates consistently with a full scan`, async () => { + const boundary = Symbol(`boundary`) + const symbolCollection = createCollection< + { id: string; group: symbol }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { id: `1`, group: Symbol(`first`) }, + }) + write({ + type: `insert`, + value: { id: `2`, group: Symbol(`second`) }, + }) + commit() + markReady() + }, + }, + }) + await symbolCollection.stateWhenReady() + + const where = gt(new PropRef([`group`]), boundary) + const scanned = symbolCollection.currentStateAsChanges({ where })! + + symbolCollection.createIndex((row) => row.group) + withIndexTracking(symbolCollection, (tracker) => { + const indexed = symbolCollection.currentStateAsChanges({ where })! + + expect(indexed.map((change) => change.key).sort()).toEqual( + scanned.map((change) => change.key).sort(), + ) + expectIndexUsage(tracker.stats, { + shouldUseIndex: false, + shouldUseFullScan: true, + }) + }) + }) + + it.each( + [BasicIndex, BTreeIndex].flatMap((IndexType) => [ + { + name: `a symbol row under a numeric lower bound`, + IndexType, + rows: [ + { id: `number`, value: 1 as unknown }, + { id: `other`, value: Symbol(`other`) as unknown }, + ], + where: gt(new PropRef([`value`]), 0), + }, + { + name: `an array row under a numeric upper bound`, + IndexType, + rows: [ + { id: `number`, value: 50 as unknown }, + { id: `other`, value: [20] as unknown }, + ], + where: lt(new PropRef([`value`]), 100), + }, + ]), + )( + `should scan mixed domains for $name with $IndexType.name`, + async ({ rows, where, IndexType }) => { + const mixedCollection = createCollection< + { id: string; value: unknown }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: IndexType, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const value of rows) write({ type: `insert`, value }) + commit() + markReady() + }, + }, + }) + await mixedCollection.stateWhenReady() + + const scanned = mixedCollection.currentStateAsChanges({ where })! + mixedCollection.createIndex((row) => row.value) + + withIndexTracking(mixedCollection, (tracker) => { + const indexed = mixedCollection.currentStateAsChanges({ where })! + expect(indexed.map((change) => change.key).sort()).toEqual( + scanned.map((change) => change.key).sort(), + ) + expectIndexUsage(tracker.stats, { + shouldUseIndex: false, + shouldUseFullScan: true, + }) + }) + }, + ) + + it(`should retain every row whose index values share one comparator position`, async () => { + const shared = Symbol(`shared`) + const groupedCollection = createCollection< + { id: string; value: Array }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `first`, value: [shared] } }) + write({ type: `insert`, value: { id: `second`, value: [shared] } }) + commit() + markReady() + }, + }, + }) + await groupedCollection.stateWhenReady() + + const index = groupedCollection.createIndex((row) => row.value) + + expect(index.takeFromStart(2)).toEqual([`first`, `second`]) + expect(orderedEntriesArray(index)[0]?.[1]).toEqual( + new Set([`first`, `second`]), + ) + expect(orderedEntriesArrayReversed(index)[0]?.[1]).toEqual( + new Set([`first`, `second`]), + ) + }) + + it(`should return all matching rows for a range predicate on a custom-comparator index`, async () => { + // A range predicate must return every row that satisfies it regardless + // of the comparator the index was created with. With scores 5 and 20, + // `score > 10` matches only the row with score 20. + const customCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `low`, score: 5 } }) + write({ type: `insert`, value: { id: `high`, score: 20 } }) + commit() + markReady() + }, + }, + }) + await customCollection.stateWhenReady() + customCollection.createIndex((row) => row.score, { + options: { compareFn: (a: number, b: number) => b - a }, + }) + + const result = customCollection.currentStateAsChanges({ + where: gt(new PropRef([`score`]), 10), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`high`]) + }) + + it(`should return all matching rows for a range predicate when the field also contains NaN`, async () => { + // A range predicate must return every matching row even when other rows + // hold a NaN value for the field. Under PostgreSQL float semantics NaN is + // the greatest value, so with scores NaN, 1, 3, 5 and 7, `score > 2` + // matches the rows with scores 3, 5, 7 and NaN. + const nanCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `nan`, score: NaN } }) + write({ type: `insert`, value: { id: `one`, score: 1 } }) + write({ type: `insert`, value: { id: `three`, score: 3 } }) + write({ type: `insert`, value: { id: `five`, score: 5 } }) + write({ type: `insert`, value: { id: `seven`, score: 7 } }) + commit() + markReady() + }, + }, + }) + await nanCollection.stateWhenReady() + nanCollection.createIndex((row) => row.score) + + const result = nanCollection.currentStateAsChanges({ + where: gt(new PropRef([`score`]), 2), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`five`, `nan`, `seven`, `three`]) + }) + + it(`should use the index for a range query on a field that also contains NaN`, async () => { + // A NaN value has a well-defined sort position (greatest, under + // PostgreSQL float semantics), so a range query on the field can still be + // served by the index and does not need to fall back to a full scan. + const nanCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `nan`, score: NaN } }) + write({ type: `insert`, value: { id: `one`, score: 1 } }) + write({ type: `insert`, value: { id: `three`, score: 3 } }) + write({ type: `insert`, value: { id: `five`, score: 5 } }) + write({ type: `insert`, value: { id: `seven`, score: 7 } }) + commit() + markReady() + }, + }, + }) + await nanCollection.stateWhenReady() + nanCollection.createIndex((row) => row.score) + + withIndexTracking(nanCollection, (tracker) => { + const result = nanCollection.currentStateAsChanges({ + where: gt(new PropRef([`score`]), 2), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`five`, `nan`, `seven`, `three`]) + + expectIndexUsage(tracker.stats, { + shouldUseIndex: true, + shouldUseFullScan: false, + }) + }) + }) + + it(`should exclude NaN from a less-than range query`, async () => { + // Under PostgreSQL float semantics NaN is the greatest value, so + // `score < 4` matches the rows with scores 1 and 3 but never the + // NaN-valued row. + const nanCollection = createCollection< + { id: string; score: number }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `nan`, score: NaN } }) + write({ type: `insert`, value: { id: `one`, score: 1 } }) + write({ type: `insert`, value: { id: `three`, score: 3 } }) + write({ type: `insert`, value: { id: `five`, score: 5 } }) + write({ type: `insert`, value: { id: `seven`, score: 7 } }) + commit() + markReady() + }, + }, + }) + await nanCollection.stateWhenReady() + nanCollection.createIndex((row) => row.score) + + const result = nanCollection.currentStateAsChanges({ + where: lt(new PropRef([`score`]), 4), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`one`, `three`]) + }) + + // Invalid Dates have a NaN timestamp, so they follow the same PostgreSQL + // float semantics as NaN: equal to one another and greater than every valid + // Date. The index-served and full-scan results must agree. + const makeInvalidDateCollection = async () => { + const dateCollection = createCollection< + { id: string; createdAt: Date }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { id: `invalid`, createdAt: new Date(`not a date`) }, + }) + write({ + type: `insert`, + value: { id: `valid`, createdAt: new Date(`2023-01-01`) }, + }) + commit() + markReady() + }, + }, + }) + await dateCollection.stateWhenReady() + dateCollection.createIndex((row) => row.createdAt) + return dateCollection + } + + it(`should match an invalid-Date row for an equality on an invalid Date`, async () => { + const dateCollection = await makeInvalidDateCollection() + + const result = dateCollection.currentStateAsChanges({ + where: eq(new PropRef([`createdAt`]), new Date(`not a date`)), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`invalid`]) + }) + + it(`should match an invalid-Date member of an IN list`, async () => { + const dateCollection = await makeInvalidDateCollection() + + const result = dateCollection.currentStateAsChanges({ + where: inArray(new PropRef([`createdAt`]), [ + new Date(`not a date`), + new Date(`2023-01-01`), + ]), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`invalid`, `valid`]) + }) + + it(`should treat an invalid Date as greater than valid Dates in a range query`, async () => { + // `createdAt > 2022` matches the valid Date and the invalid Date (which + // is the greatest value under PostgreSQL float semantics). + const dateCollection = await makeInvalidDateCollection() + + const result = dateCollection.currentStateAsChanges({ + where: gt(new PropRef([`createdAt`]), new Date(`2022-01-01`)), + })! + + const ids = result.map((r) => r.value.id).sort() + expect(ids).toEqual([`invalid`, `valid`]) + }) }) describe(`Index Usage Verification`, () => { @@ -1487,11 +2249,11 @@ describe(`Collection Indexes`, () => { const ageIndex = specialCollection.createIndex((row) => row.age) // Verify index contains all items including special values - expect(ageIndex.indexedKeysSet.size).toBe(8) // Original 5 + 3 special - expect(ageIndex.orderedEntriesArray).toHaveLength(8) // 8 unique age values (including null) + expect(indexedKeysSet(ageIndex).size).toBe(8) // Original 5 + 3 special + expect(orderedEntriesArray(ageIndex)).toHaveLength(8) // 8 unique age values (including null) // Null/undefined should be ordered first - const firstValue = ageIndex.orderedEntriesArray[0]?.[0] + const firstValue = orderedEntriesArray(ageIndex)[0]?.[0] expect(firstValue == null).toBe(true) // Test that queries with special values use indexes correctly @@ -1535,17 +2297,17 @@ describe(`Collection Indexes`, () => { const index = emptyCollection.createIndex((row) => row.age) - expect(index.indexedKeysSet.size).toBe(0) - expect(index.orderedEntriesArray).toHaveLength(0) - expect(index.valueMapData.size).toBe(0) + expect(indexedKeysSet(index).size).toBe(0) + expect(orderedEntriesArray(index)).toHaveLength(0) + expect(valueMapData(index).size).toBe(0) }) it(`should handle index updates when data changes through sync`, async () => { const ageIndex = collection.createIndex((row) => row.age) // Original index should have 5 items - expect(ageIndex.indexedKeysSet.size).toBe(5) - expect(ageIndex.orderedEntriesArray).toHaveLength(5) + expect(indexedKeysSet(ageIndex).size).toBe(5) + expect(orderedEntriesArray(ageIndex)).toHaveLength(5) // Perform mutations that will sync back and update indexes const tx1 = createTransaction({ mutationFn }) @@ -1579,7 +2341,7 @@ describe(`Collection Indexes`, () => { await new Promise((resolve) => setTimeout(resolve, 10)) // Verify that indexes are updated after sync - expect(ageIndex.indexedKeysSet.size).toBe(5) // 5 original - 1 deleted + 1 inserted + expect(indexedKeysSet(ageIndex).size).toBe(5) // 5 original - 1 deleted + 1 inserted // Test that index-optimized queries work with the updated data withIndexTracking(collection, (tracker) => { diff --git a/packages/db/tests/collection-lifecycle.test.ts b/packages/db/tests/collection-lifecycle.test.ts index a5cf03f19e..7779c08e75 100644 --- a/packages/db/tests/collection-lifecycle.test.ts +++ b/packages/db/tests/collection-lifecycle.test.ts @@ -1,12 +1,151 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' import { CleanupQueue } from '../src/collection/cleanup-queue.js' +import { InvalidCollectionStatusTransitionError } from '../src/errors.js' +import { + getActivePublicationContext, + transactionScopedScheduler, + withPublicationContext, +} from '../src/scheduler.js' +import { resetCleanupQueue } from './utils' // Mock setTimeout and clearTimeout for testing GC behavior const originalSetTimeout = global.setTimeout const originalClearTimeout = global.clearTimeout +function getChangesManager(collection: object): { + emitEmptyReadyEvent: () => void +} { + return ( + collection as unknown as { + _changes: { emitEmptyReadyEvent: () => void } + } + )._changes +} + describe(`Collection Lifecycle Management`, () => { + it.each( + ([`same`, `missing`, `changed`, `empty`] as const).flatMap((shape) => + ([`atomic`, `split`] as const).map((delivery) => ({ shape, delivery })), + ), + )( + `keeps eager restart messages coherent for $shape keys with $delivery commits`, + async ({ shape, delivery }) => { + type Row = { id: string; version: number } + let rows: Array = [ + { id: `a`, version: 1 }, + { id: `b`, version: 1 }, + ] + const collection = createCollection({ + getKey: ({ id }) => id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + const batches = + delivery === `atomic` ? [rows] : rows.map((row) => [row]) + for (const batch of batches) { + begin() + for (const value of batch) write({ type: `insert`, value }) + commit() + } + markReady() + }, + }, + }) + await collection.preload() + const delivered = new Map() + const read = () => + collection.toArray.map(({ id, version }) => ({ id, version })) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) { + expect(delivered.get(change.key)).toEqual({ + id: change.value.id, + version: change.value.version, + }) + delivered.delete(change.key) + } else { + if (change.type === `insert`) + expect(delivered.has(change.key)).toBe(false) + else + expect(change.previousValue).toMatchObject( + delivered.get(change.key)!, + ) + delivered.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + expect([...delivered.values()]).toEqual(read()) + }, + { includeInitialState: true }, + ) + try { + expect([...delivered.values()]).toEqual(rows) + await collection.cleanup() + rows = + shape === `empty` + ? [] + : shape === `changed` + ? [ + { id: `c`, version: 2 }, + { id: `d`, version: 2 }, + ] + : shape === `missing` + ? [{ id: `a`, version: 2 }] + : [ + { id: `a`, version: 2 }, + { id: `b`, version: 2 }, + ] + await collection.preload() + expect(collection.status).toBe(`ready`) + expect([...delivered.values()]).toEqual(rows) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each([`pending`, `starting`, `ready`, `failed`] as const)( + `cleanup settles a %s preload without inventing first readiness`, + async (phase) => { + let starts = 0 + const failure = new Error(`initial failure`) + const ready = vi.fn() + const collection = createCollection<{ id: string }>({ + getKey: ({ id }) => id, + sync: { + sync: ({ collection: source, markReady, markError }) => { + starts++ + if (starts > 1 || phase === `ready`) markReady() + else if (phase === `failed`) markError(failure) + else if (phase === `starting`) void source.cleanup() + }, + }, + }) + collection.onFirstReady(ready) + const preload = collection.preload().then( + () => undefined, + (error: unknown) => error, + ) + if (phase !== `starting`) await collection.cleanup() + const result = await preload + if (phase === `ready`) expect(result).toBeUndefined() + else if (phase === `failed`) expect(result).toBe(failure) + else expect(result).toMatchObject({ name: `AbortError` }) + expect(ready).toHaveBeenCalledTimes(phase === `ready` ? 1 : 0) + const restartedReady = vi.fn() + collection.onFirstReady(restartedReady) + await collection.preload() + expect(starts).toBe(2) + expect(restartedReady).toHaveBeenCalledOnce() + expect(ready).toHaveBeenCalledTimes(phase === `ready` ? 1 : 0) + await collection.cleanup() + }, + ) + let mockSetTimeout: ReturnType let mockClearTimeout: ReturnType let timeoutCallbacks: Map void> @@ -47,7 +186,7 @@ describe(`Collection Lifecycle Management`, () => { global.setTimeout = originalSetTimeout global.clearTimeout = originalClearTimeout vi.clearAllMocks() - CleanupQueue.resetInstance() + resetCleanupQueue() }) const triggerAllTimeouts = () => { @@ -133,6 +272,38 @@ describe(`Collection Lifecycle Management`, () => { expect(collection.status).toBe(`cleaned-up`) }) + it(`clears terminal state without publishing one delete per row`, async () => { + const collection = createCollection<{ id: number; name: string }>({ + id: `cleanup-without-row-publication`, + getKey: (item) => item.id, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (let id = 0; id < 100; id++) { + write({ type: `insert`, value: { id, name: `row-${id}` } }) + } + commit() + markReady() + }, + }, + }) + const onChanges = vi.fn() + const subscription = collection.subscribeChanges(onChanges, { + includeInitialState: false, + }) + + try { + await collection.cleanup() + + expect(collection.toArray).toEqual([]) + expect(onChanges).not.toHaveBeenCalled() + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + it(`should transition when subscribing to changes`, () => { let beginCallback: (() => void) | undefined let commitCallback: (() => void) | undefined @@ -511,6 +682,919 @@ describe(`Collection Lifecycle Management`, () => { subscription.unsubscribe() }) + it(`freezes first-ready callback membership before delivery`, async () => { + let markReadyCallback: (() => void) | undefined + const calls: Array = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `first-ready-membership-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}) + let removeLater = () => {} + + collection.onFirstReady(() => { + calls.push(`first`) + removeLater() + collection.onFirstReady(() => calls.push(`nested`)) + }) + removeLater = collection.onFirstReady(() => calls.push(`later`)) + + try { + markReadyCallback!() + + expect(calls).toEqual([`first`, `nested`, `later`]) + + collection.onFirstReady(() => calls.push(`after`)) + expect(calls).toEqual([`first`, `nested`, `later`, `after`]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + expect(calls).toEqual([`first`, `nested`, `later`, `after`]) + } + }) + + it.each([ + { + from: `ready`, + expectedStatus: `ready`, + expectedFirstReadyCalls: 1, + invalid: false, + }, + { + from: `error`, + expectedStatus: `ready`, + expectedFirstReadyCalls: 1, + invalid: false, + }, + { + from: `idle`, + expectedStatus: `idle`, + expectedFirstReadyCalls: 0, + invalid: true, + }, + { + from: `cleaned-up`, + expectedStatus: `cleaned-up`, + expectedFirstReadyCalls: 0, + invalid: true, + }, + ] as const)( + `defines the $from -> ready transition`, + async ({ from, expectedStatus, expectedFirstReadyCalls, invalid }) => { + const syncFailure = new Error(`sync failed before recovery`) + let firstReadyCalls = 0 + let recoveryFirstReadyCalls = 0 + const collection = createCollection<{ id: string; name: string }>({ + id: `mark-ready-from-${from}`, + getKey: (item) => item.id, + startSync: false, + sync: { sync: () => {} }, + }) + collection.onFirstReady(() => { + firstReadyCalls++ + }) + + if (from === `ready` || from === `error`) { + collection._lifecycle.setStatus(`loading`) + collection._lifecycle.markReady() + } + if (from === `error`) { + collection._lifecycle.markError(syncFailure) + expect(collection._lifecycle.getSyncError()).toBe(syncFailure) + } else if (from === `cleaned-up`) { + collection._lifecycle.setStatus(`cleaned-up`) + } + expect(collection.status).toBe(from) + + if (from === `error`) { + collection.onFirstReady(() => { + recoveryFirstReadyCalls++ + }) + expect(recoveryFirstReadyCalls).toBe(1) + } + + const transitionTrace: Array< + | { + kind: `status` + previousStatus: string + status: string + syncError: unknown + } + | { + kind: `dependent-ready` + status: string + syncError: unknown + } + > = [] + collection.on(`status:change`, ({ previousStatus, status }) => { + transitionTrace.push({ + kind: `status`, + previousStatus, + status, + syncError: collection._lifecycle.getSyncError(), + }) + }) + const changes = getChangesManager(collection) + const originalEmitEmptyReadyEvent = + changes.emitEmptyReadyEvent.bind(changes) + vi.spyOn(changes, `emitEmptyReadyEvent`).mockImplementation(() => { + transitionTrace.push({ + kind: `dependent-ready`, + status: collection.status, + syncError: collection._lifecycle.getSyncError(), + }) + originalEmitEmptyReadyEvent() + }) + + let didThrow = false + let thrown: unknown + try { + collection._lifecycle.markReady() + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(invalid) + if (invalid) { + expect(thrown).toBeInstanceOf(InvalidCollectionStatusTransitionError) + expect((thrown as Error).message).toBe( + `Invalid collection status transition from "${from}" to "ready" for collection "mark-ready-from-${from}"`, + ) + } + expect(collection.status).toBe(expectedStatus) + expect(firstReadyCalls).toBe(expectedFirstReadyCalls) + expect(recoveryFirstReadyCalls).toBe(from === `error` ? 1 : 0) + expect(transitionTrace).toEqual( + from === `error` + ? [ + { + kind: `status`, + previousStatus: `error`, + status: `ready`, + syncError: undefined, + }, + { + kind: `dependent-ready`, + status: `ready`, + syncError: undefined, + }, + ] + : [], + ) + expect(collection._lifecycle.getSyncError()).toBeUndefined() + + await collection.cleanup() + }, + ) + + it(`does not resume ready effects after a status listener cleans up`, () => { + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-listener-cleanup-test`, + getKey: (item) => item.id, + startSync: false, + sync: { sync: () => {} }, + }) + const readyEvent = vi.spyOn( + getChangesManager(collection), + `emitEmptyReadyEvent`, + ) + const firstReadyStatuses: Array = [] + collection.onFirstReady(() => { + firstReadyStatuses.push(collection.status) + }) + collection.on(`status:ready`, () => { + void collection.cleanup() + }) + collection._lifecycle.setStatus(`loading`) + + collection._lifecycle.markReady() + + expect(collection.status).toBe(`cleaned-up`) + expect(collection._lifecycle.hasBeenReady).toBe(false) + expect(firstReadyStatuses).toEqual([]) + expect(readyEvent).not.toHaveBeenCalled() + + const laterFirstReady = vi.fn() + const removeLater = collection.onFirstReady(laterFirstReady) + expect(laterFirstReady).not.toHaveBeenCalled() + removeLater() + }) + + it(`does not resume ready effects after a status listener enters error`, async () => { + const failure = new Error(`ready listener failed the sync`) + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-listener-error-test`, + getKey: (item) => item.id, + startSync: false, + sync: { sync: () => {} }, + }) + const readyEvent = vi.spyOn( + getChangesManager(collection), + `emitEmptyReadyEvent`, + ) + const firstReady = vi.fn() + collection.onFirstReady(firstReady) + collection.on(`status:ready`, () => { + collection._lifecycle.markError(failure) + }) + collection._lifecycle.setStatus(`loading`) + + collection._lifecycle.markReady() + + expect(collection.status).toBe(`error`) + expect(collection._lifecycle.getSyncError()).toBe(failure) + expect(collection._lifecycle.hasBeenReady).toBe(false) + expect(firstReady).not.toHaveBeenCalled() + expect(readyEvent).not.toHaveBeenCalled() + await collection.cleanup() + }) + + it(`does not resume an outer ready transition after a synchronous restart`, async () => { + let syncStarts = 0 + let restartedPreload: Promise | undefined + let restartOnce = true + let lateSubscription: { unsubscribe: () => void } | undefined + const lateReadyBatches: Array> = [] + const firstReadyStatuses: Array = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-listener-aba-test`, + getKey: (item) => item.id, + startSync: false, + sync: { + sync: ({ markReady }) => { + syncStarts++ + markReady() + }, + }, + }) + const readyEvent = vi.spyOn( + getChangesManager(collection), + `emitEmptyReadyEvent`, + ) + collection.onFirstReady(() => { + firstReadyStatuses.push(collection.status) + }) + collection.on(`status:ready`, () => { + if (!restartOnce) return + restartOnce = false + void collection.cleanup() + restartedPreload = collection.preload() + lateSubscription = collection.subscribeChanges((batch) => { + lateReadyBatches.push(batch) + }) + }) + collection._lifecycle.setStatus(`loading`) + + collection._lifecycle.markReady() + await restartedPreload + + expect(syncStarts).toBe(1) + expect(collection.status).toBe(`ready`) + expect(firstReadyStatuses).toEqual([]) + expect(lateReadyBatches).toEqual([]) + expect(readyEvent).toHaveBeenCalledOnce() + lateSubscription!.unsubscribe() + await collection.cleanup() + }) + + it(`starts a fresh first-ready cycle after cleanup of a failed ready effect`, async () => { + const readyCallbacks: Array<() => void> = [] + const firstFailure = new Error(`first ready cycle failed exactly`) + const trace: Array = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-effect-restart-test`, + getKey: (item) => item.id, + startSync: false, + sync: { + sync: ({ markReady }) => { + readyCallbacks.push(markReady) + }, + }, + }) + const readyEvent = vi.spyOn( + getChangesManager(collection), + `emitEmptyReadyEvent`, + ) + collection.onFirstReady(() => { + trace.push(`first failure:${collection.status}`) + throw firstFailure + }) + collection.onFirstReady(() => { + trace.push(`first later:${collection.status}`) + }) + const firstPreload = collection.preload() + let firstPreloadSettled = false + void firstPreload.then(() => { + firstPreloadSettled = true + }) + await Promise.resolve() + expect(collection.status).toBe(`loading`) + expect(firstPreloadSettled).toBe(false) + + let thrown: unknown + try { + readyCallbacks[0]!() + } catch (error) { + thrown = error + } + expect(thrown).toBe(firstFailure) + await expect(firstPreload).resolves.toBeUndefined() + expect(trace).toEqual([`first failure:ready`, `first later:ready`]) + expect(readyEvent).toHaveBeenCalledOnce() + + await collection.cleanup() + expect(collection.status).toBe(`cleaned-up`) + expect(collection._lifecycle.hasBeenReady).toBe(false) + + collection.onFirstReady(() => { + trace.push(`second:${collection.status}`) + }) + expect(trace).toEqual([`first failure:ready`, `first later:ready`]) + + const secondPreload = collection.preload() + let secondPreloadSettled = false + void secondPreload.then(() => { + secondPreloadSettled = true + }) + expect(secondPreload).not.toBe(firstPreload) + expect(readyCallbacks).toHaveLength(2) + await Promise.resolve() + expect(collection.status).toBe(`loading`) + expect(secondPreloadSettled).toBe(false) + + readyCallbacks[0]!() + await Promise.resolve() + expect(collection.status).toBe(`loading`) + expect(secondPreloadSettled).toBe(false) + expect(trace).toEqual([`first failure:ready`, `first later:ready`]) + expect(readyEvent).toHaveBeenCalledOnce() + + readyCallbacks[1]!() + await expect(secondPreload).resolves.toBeUndefined() + expect(secondPreloadSettled).toBe(true) + + expect(trace).toEqual([ + `first failure:ready`, + `first later:ready`, + `second:ready`, + ]) + expect(readyEvent).toHaveBeenCalledTimes(2) + + await collection.cleanup() + }) + + it(`attempts every first-ready effect before rethrowing the first failure`, async () => { + let markReadyCallback: (() => void) | undefined + const readyBatches: Array> = [] + const readyTrace: Array = [] + const laterFailure = new Error(`later first-ready failure`) + const laterCallback = vi.fn(() => { + readyTrace.push(`later:${collection.status}`) + throw laterFailure + }) + let preloadSettled = false + + const collection = createCollection<{ id: string; name: string }>({ + id: `first-ready-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady as () => void + }, + }, + }) + const subscription = collection.subscribeChanges((batch) => { + readyTrace.push(`dependent:${collection.status}`) + readyBatches.push(batch) + }) + collection.onFirstReady(() => { + readyTrace.push(`first:${collection.status}`) + throw undefined + }) + collection.onFirstReady(laterCallback) + void collection.preload().then(() => { + preloadSettled = true + }) + + try { + let didThrow = false + let thrown: unknown + try { + markReadyCallback!() + } catch (error) { + didThrow = true + thrown = error + } + await Promise.resolve() + + expect(didThrow).toBe(true) + expect(thrown).toBeUndefined() + expect(laterCallback).toHaveBeenCalledOnce() + expect(preloadSettled).toBe(true) + expect(readyBatches).toEqual([[]]) + expect(readyTrace).toEqual([ + `first:ready`, + `later:ready`, + `dependent:ready`, + ]) + expect(collection.status).toBe(`ready`) + + expect(() => markReadyCallback!()).not.toThrow() + expect(laterCallback).toHaveBeenCalledOnce() + expect(readyBatches).toEqual([[]]) + expect(readyTrace).toEqual([ + `first:ready`, + `later:ready`, + `dependent:ready`, + ]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not classify synchronous first-ready callback failures as sync failures`, async () => { + const laterFailure = new Error(`later synchronous first-ready failure`) + const callbackTrace: Array = [] + let syncContinued = false + + const collection = createCollection<{ id: string; name: string }>({ + id: `synchronous-first-ready-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReady() + syncContinued = true + }, + }, + }) + collection.onFirstReady(() => { + callbackTrace.push(`first`) + throw undefined + }) + collection.onFirstReady(() => { + callbackTrace.push(`later`) + throw laterFailure + }) + + try { + let didThrow = false + let thrown: unknown + try { + collection._sync.startSync() + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(thrown).toBeUndefined() + expect(syncContinued).toBe(true) + expect(callbackTrace).toEqual([`first`, `later`]) + expect(collection.status).toBe(`ready`) + await expect(collection.preload()).resolves.toBeUndefined() + } finally { + await collection.cleanup() + } + }) + + it(`rejects a pending preload when the adapter fails after marking ready`, async () => { + const adapterFailure = new Error(`adapter failed after ready`) + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-then-adapter-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReady() + throw adapterFailure + }, + }, + }) + collection.onFirstReady(() => { + throw undefined + }) + + try { + await expect(collection.preload()).rejects.toBe(adapterFailure) + expect(collection.status).toBe(`error`) + } finally { + await collection.cleanup() + } + }) + + it(`ends the synchronous sync-entry boundary after an adapter failure`, async () => { + const adapterFailure = new Error(`adapter entry failed`) + let markReadyCallback: (() => void) | undefined + const collection = createCollection<{ id: string; name: string }>({ + id: `failed-sync-entry-boundary-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + throw adapterFailure + }, + }, + }) + collection.onFirstReady(() => { + throw undefined + }) + + try { + expect(() => collection._sync.startSync()).toThrow(adapterFailure) + expect(collection.status).toBe(`error`) + + let didThrow = false + let thrown: unknown + try { + markReadyCallback!() + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(thrown).toBeUndefined() + expect(collection.status).toBe(`ready`) + } finally { + await collection.cleanup() + } + }) + + it(`attempts every dependent ready listener before rethrowing`, async () => { + let markReadyCallback: (() => void) | undefined + const firstFailure = new Error(`first dependent failed`) + const firstBatches: Array> = [] + const secondBatches: Array> = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady as () => void + }, + }, + }) + const first = collection.subscribeChanges((batch) => { + firstBatches.push(batch) + throw firstFailure + }) + const second = collection.subscribeChanges((batch) => { + secondBatches.push(batch) + }) + + try { + let thrown: unknown + try { + markReadyCallback!() + } catch (error) { + thrown = error + } + + expect(thrown).toBe(firstFailure) + expect(firstBatches).toEqual([[]]) + expect(secondBatches).toEqual([[]]) + expect(collection.status).toBe(`ready`) + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`flushes work queued by a ready listener when a sibling throws`, async () => { + let markReadyCallback: (() => void) | undefined + const firstFailure = new Error(`dependent failed after sibling queued`) + const scheduledJob = vi.fn() + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-scheduler-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + const contextId = getActivePublicationContext() + expect(contextId).toBeDefined() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + const second = collection.subscribeChanges(() => { + throw firstFailure + }) + + try { + expect(() => markReadyCallback!()).toThrow(firstFailure) + expect(scheduledJob).toHaveBeenCalledOnce() + expect(collection.status).toBe(`ready`) + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`flushes ready work before rethrowing at an outer publication boundary`, async () => { + let markReadyCallback: (() => void) | undefined + const listenerFailure = new Error(`nested dependent failed`) + const scheduledJob = vi.fn() + const collection = createCollection<{ id: string; name: string }>({ + id: `nested-dependent-ready-scheduler-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + const second = collection.subscribeChanges(() => { + throw listenerFailure + }) + + try { + expect(() => + withPublicationContext(() => markReadyCallback!()), + ).toThrow(listenerFailure) + expect(scheduledJob).toHaveBeenCalledOnce() + expect(collection.status).toBe(`ready`) + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`preserves a falsy ready failure through a nested publication`, async () => { + let markReadyCallback: (() => void) | undefined + const scheduledJob = vi.fn() + const collection = createCollection<{ id: string; name: string }>({ + id: `nested-falsy-ready-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + const second = collection.subscribeChanges(() => { + throw undefined + }) + + try { + let didThrow = false + let thrown: unknown + try { + withPublicationContext(() => markReadyCallback!()) + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(thrown).toBeUndefined() + expect(scheduledJob).toHaveBeenCalledOnce() + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`surfaces a ready graph failure after running its job`, async () => { + let markReadyCallback: (() => void) | undefined + const graphFailure = new Error(`ready graph failed`) + const scheduledJob = vi.fn(() => { + throw graphFailure + }) + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-graph-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const subscription = collection.subscribeChanges(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + + try { + expect(() => markReadyCallback!()).toThrow(graphFailure) + expect(scheduledJob).toHaveBeenCalledOnce() + expect(collection.status).toBe(`ready`) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`keeps the ready listener failure when its queued graph job also fails`, async () => { + let markReadyCallback: (() => void) | undefined + const listenerFailure = new Error(`ready listener failed first`) + const graphFailure = new Error(`ready graph also failed`) + const scheduledJob = vi.fn(() => { + throw graphFailure + }) + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-failure-priority-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + const second = collection.subscribeChanges(() => { + throw listenerFailure + }) + + try { + expect(() => markReadyCallback!()).toThrow(listenerFailure) + expect(scheduledJob).toHaveBeenCalledOnce() + expect(collection.status).toBe(`ready`) + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`resolves a pending preload after a ready callback failure alone`, async () => { + let syncContinued = false + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-callback-preload-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReady() + syncContinued = true + }, + }, + }) + collection.onFirstReady(() => { + throw undefined + }) + + try { + await expect(collection.preload()).resolves.toBeUndefined() + expect(syncContinued).toBe(true) + expect(collection.status).toBe(`ready`) + } finally { + await collection.cleanup() + } + }) + + it(`skips a ready listener unsubscribed during the same delivery`, async () => { + let markReadyCallback: (() => void) | undefined + const calls: Array = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-membership-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + calls.push(`first`) + second.unsubscribe() + }) + const second = collection.subscribeChanges(() => { + calls.push(`second`) + }) + + try { + markReadyCallback!() + expect(calls).toEqual([`first`]) + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`excludes a dependent added during ready delivery until the next batch`, async () => { + let beginCallback: (() => void) | undefined + let writeCallback: + | ((message: { + type: `insert` + value: { id: string; name: string } + }) => void) + | undefined + let commitCallback: (() => void) | undefined + let markReadyCallback: (() => void) | undefined + let added: { unsubscribe: () => void } | undefined + const calls: Array = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-addition-test`, + getKey: (item) => item.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + beginCallback = begin + writeCallback = write + commitCallback = () => { + commit() + } + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + calls.push(`first`) + added ??= collection.subscribeChanges(() => calls.push(`added`)) + }) + const second = collection.subscribeChanges(() => calls.push(`second`)) + + try { + markReadyCallback!() + expect(calls).toEqual([`first`, `second`]) + + beginCallback!() + writeCallback!({ + type: `insert`, + value: { id: `one`, name: `One` }, + }) + commitCallback!() + expect(calls).toEqual([`first`, `second`, `first`, `second`, `added`]) + } finally { + first.unsubscribe() + second.unsubscribe() + added?.unsubscribe() + await collection.cleanup() + } + }) + + it(`notifies a dependent added during the first-ready fan-out`, async () => { + let markReadyCallback: (() => void) | undefined + let dependent: { unsubscribe: () => void } | undefined + const readyBatches: Array> = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `nested-dependent-ready-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady as () => void + }, + }, + }) + collection.onFirstReady(() => { + dependent = collection.subscribeChanges((batch) => { + readyBatches.push(batch) + }) + }) + const preload = collection.preload() + + try { + markReadyCallback!() + await preload + expect(readyBatches).toEqual([[]]) + } finally { + dependent?.unsubscribe() + await collection.cleanup() + } + }) + it(`should fire status:change event with 'cleaned-up' status before clearing event handlers`, () => { const collection = createCollection<{ id: string; name: string }>({ id: `cleanup-event-test`, diff --git a/packages/db/tests/collection-metadata-publication-oracle.property.test.ts b/packages/db/tests/collection-metadata-publication-oracle.property.test.ts new file mode 100644 index 0000000000..2da3765008 --- /dev/null +++ b/packages/db/tests/collection-metadata-publication-oracle.property.test.ts @@ -0,0 +1,535 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { SyncTransactionAbortedError } from '../src/errors.js' +import { createLiveQueryCollection } from '../src/query/index.js' +import { createTransaction } from '../src/transactions.js' +import { oraclePropertyOptions } from './oracle-config.js' +import type { Collection } from '../src/collection/index.js' +import type { ChangeMessage, SyncConfig } from '../src/types.js' + +type PublicationRow = { + id: number + position: number +} + +type SyncActions = Parameters[`sync`]>[0] + +type MetadataOperation = { type: `set`; value: unknown } | { type: `delete` } + +type MetadataEntryState = { present: false } | { present: true; value: unknown } + +type MetadataWrite = { key: number } & MetadataOperation + +type PublicationRound = { + key: number + delta: number + metadata: ReadonlyArray + outcome: `commit` | `abort` +} + +type ReadablePublicationCollection = { + values: () => IterableIterator + cleanup: () => Promise +} + +type PublicationHarness = { + rows: Collection + liveRows: ReadablePublicationCollection + batches: Array>> + unsubscribe: () => void + getSync: () => SyncActions +} + +type PublishedPublicationRow = PublicationRow & { + $collectionId: string + $key: number + $origin: `local` | `remote` + $synced: boolean +} + +const metadataValueArbitrary = fc.oneof( + fc.constant(undefined), + fc.constant(null), + fc.constant(false), + fc.constant(true), + fc.constant(0), + fc.constant(Number.NaN), + fc.constant(``), + fc.integer(), + fc.string(), + fc.record({ nested: fc.integer() }), +) + +const metadataOperationArbitrary: fc.Arbitrary = fc.oneof( + metadataValueArbitrary.map((value) => ({ type: `set` as const, value })), + fc.constant({ type: `delete` as const }), +) + +const metadataEntryStateArbitrary: fc.Arbitrary = fc.oneof( + fc.constant({ present: false as const }), + metadataValueArbitrary.map((value) => ({ + present: true as const, + value, + })), +) + +const metadataWriteArbitrary = fc + .tuple(fc.integer({ min: 0, max: 2 }), metadataOperationArbitrary) + .map(([key, operation]) => ({ key, ...operation })) + +const publicationRoundArbitrary: fc.Arbitrary = fc + .record({ + key: fc.integer({ min: 0, max: 2 }), + delta: fc.constantFrom(-2, -1, 1, 2), + extraMetadata: fc.array(metadataWriteArbitrary, { maxLength: 2 }), + outcome: fc.constantFrom(`commit` as const, `abort` as const), + primaryMetadata: metadataOperationArbitrary, + }) + .map(({ key, delta, extraMetadata, outcome, primaryMetadata }) => ({ + key, + delta, + outcome, + metadata: [{ key, ...primaryMetadata }, ...extraMetadata], + })) + +const metadataCancellationArbitrary = fc.record({ + canceledKeys: fc.uniqueArray(fc.integer({ min: 0, max: 2 }), { + minLength: 1, + maxLength: 3, + }), + retainedKeys: fc.uniqueArray(fc.integer({ min: 0, max: 2 }), { + minLength: 1, + maxLength: 3, + }), + canceledOperation: metadataOperationArbitrary, + retainedOperation: metadataOperationArbitrary, + canceledFirst: fc.boolean(), + initialMetadata: fc.array(metadataEntryStateArbitrary, { + minLength: 3, + maxLength: 3, + }), +}) + +async function createPublicationHarness(): Promise { + let sync!: SyncActions + const rows = createCollection({ + id: `metadata-publication-source`, + getKey: (row) => row.id, + startSync: true, + sync: { + sync: (actions) => { + sync = actions + actions.begin() + for (let id = 0; id < 3; id++) { + actions.write({ type: `insert`, value: { id, position: id } }) + } + actions.commit() + actions.markReady() + }, + }, + }) + const liveRows = createLiveQueryCollection((query) => + query.from({ row: rows }), + ) + await liveRows.preload() + + const batches: Array>> = + [] + const subscription = rows.subscribeChanges((changes) => { + batches.push(changes) + }) + return { + rows, + liveRows, + batches, + unsubscribe: () => subscription.unsubscribe(), + getSync: () => sync, + } +} + +function expectUniqueBatchKeys( + batches: ReadonlyArray< + ReadonlyArray> + >, +): void { + for (const batch of batches) { + const keys = batch.map((change) => change.key) + expect(keys).toEqual([...new Set(keys)]) + } +} + +function selectPublishedRow( + row: PublicationRow | undefined, +): PublishedPublicationRow | undefined { + if (row === undefined) return undefined + const published = row as PublishedPublicationRow + return { + id: published.id, + position: published.position, + $collectionId: published.$collectionId, + $key: published.$key, + $origin: published.$origin, + $synced: published.$synced, + } +} + +function selectPublishedChange( + change: ChangeMessage, +) { + return { + type: change.type, + key: change.key, + value: selectPublishedRow(change.value), + previousValue: selectPublishedRow(change.previousValue), + } +} + +function expectPublishedRows( + harness: PublicationHarness, + model: ReadonlyMap, +): void { + const expected = [...model.values()].sort((a, b) => a.id - b.id) + const selectBaseRows = (collection: ReadablePublicationCollection) => + [...collection.values()] + .map((row) => ({ id: row.id, position: row.position })) + .sort((a, b) => a.id - b.id) + + expect(selectBaseRows(harness.rows)).toEqual(expected) + expect(selectBaseRows(harness.liveRows)).toEqual(expected) +} + +function readMetadata( + harness: PublicationHarness, + keys: Iterable, +): Map { + const metadata = harness.getSync().metadata!.row + return new Map([...keys].map((key) => [key, metadata.get(key)])) +} + +function observableMetadata( + model: ReadonlyMap, + keys: Iterable, +): Map { + return new Map([...keys].map((key) => [key, model.get(key)])) +} + +async function applyRound( + harness: PublicationHarness, + round: PublicationRound, + model: Map, + metadataModel: Map, +): Promise { + const previous = model.get(round.key)! + const next = { ...previous, position: previous.position + round.delta } + const batchCountBefore = harness.batches.length + const keyWasPreviouslyPublished = harness.batches.some((batch) => + batch.some((change) => change.key === round.key), + ) + const sync = harness.getSync() + const transaction = createTransaction({ + mutationFn: async () => { + sync.begin({ immediate: true }) + sync.write({ type: `update`, value: next }) + sync.commit() + + sync.begin() + for (const write of round.metadata) { + if (write.type === `set`) { + sync.metadata!.row.set(write.key, write.value) + } else { + sync.metadata!.row.delete(write.key) + } + } + if (round.outcome === `commit`) { + sync.commit() + } else { + const controller = new AbortController() + const receipt = sync.commit(controller.signal) + controller.abort() + if (receipt !== true) { + await receipt.catch((error: unknown) => { + if (!(error instanceof SyncTransactionAbortedError)) throw error + }) + } + } + }, + }) + transaction.mutate(() => { + harness.rows.update(round.key, (draft) => { + draft.position = next.position + }) + }) + await transaction.isPersisted.promise + + model.set(round.key, next) + if (round.outcome === `commit`) { + for (const write of round.metadata) { + if (write.type === `set`) { + metadataModel.set(write.key, write.value) + } else { + metadataModel.delete(write.key) + } + } + } + await Promise.resolve() + const virtualRow = ( + row: PublicationRow, + synced: boolean, + ): PublishedPublicationRow => ({ + ...row, + $collectionId: harness.rows.id, + $key: row.id, + $origin: `local`, + $synced: synced, + }) + const expectedOptimisticChange = keyWasPreviouslyPublished + ? { + type: `update`, + key: round.key, + value: virtualRow(next, false), + previousValue: virtualRow(previous, true), + } + : { + type: `insert`, + key: round.key, + value: virtualRow(next, false), + previousValue: undefined, + } + expect( + harness.batches + .slice(batchCountBefore) + .map((batch) => batch.map(selectPublishedChange)), + ).toEqual([ + [expectedOptimisticChange], + [ + { + type: `update`, + key: round.key, + value: virtualRow(next, true), + previousValue: virtualRow(next, false), + }, + ], + ]) + expectUniqueBatchKeys(harness.batches) + expectPublishedRows(harness, model) + expect(readMetadata(harness, [0, 1, 2])).toEqual( + observableMetadata(metadataModel, [0, 1, 2]), + ) + expect(harness.rows._state.preSyncVisibleState.size).toBe(0) + expect(harness.rows._state.preSyncVirtualState.size).toBe(0) + expect(harness.rows._state.recentlySyncedKeys.size).toBe(0) +} + +async function runPublicationHistory( + rounds: ReadonlyArray, +): Promise { + const harness = await createPublicationHarness() + const model = new Map( + [0, 1, 2].map((id) => [id, { id, position: id }] as const), + ) + const metadataModel = new Map() + try { + for (const round of rounds) { + await applyRound(harness, round, model, metadataModel) + } + } finally { + harness.unsubscribe() + await Promise.all([harness.liveRows.cleanup(), harness.rows.cleanup()]) + } +} + +async function expectMetadataCancellationOwnership( + canceledKeys: ReadonlyArray, + retainedKeys: ReadonlyArray, + canceledOperation: MetadataOperation, + retainedOperation: MetadataOperation, + canceledFirst: boolean, + initialMetadataState: ReadonlyArray, +): Promise { + const harness = await createPublicationHarness() + const initialMetadata = new Map() + for (const [key, state] of initialMetadataState.entries()) { + if (state.present) initialMetadata.set(key, state.value) + } + const initialSync = harness.getSync() + initialSync.begin() + for (const [key, value] of initialMetadata) { + initialSync.metadata!.row.set(key, value) + } + initialSync.commit() + await Promise.resolve() + + const persistence = createDeferred() + const heldTransaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + heldTransaction.mutate(() => { + harness.rows.insert({ id: 99, position: 99 }) + }) + expect(heldTransaction.state).toBe(`persisting`) + + const stageMetadata = ( + keys: ReadonlyArray, + operation: MetadataOperation, + signal?: AbortSignal, + ) => { + const sync = harness.getSync() + sync.begin() + for (const key of keys) { + if (operation.type === `set`) { + sync.metadata!.row.set(key, operation.value) + } else { + sync.metadata!.row.delete(key) + } + } + const receipt = sync.commit(signal) + if (receipt === true) { + throw new Error(`Persisting optimistic work did not hold metadata sync`) + } + void receipt.catch(() => undefined) + return receipt + } + + const canceledController = new AbortController() + const first = canceledFirst + ? stageMetadata(canceledKeys, canceledOperation, canceledController.signal) + : stageMetadata(retainedKeys, retainedOperation) + const second = canceledFirst + ? stageMetadata(retainedKeys, retainedOperation) + : stageMetadata(canceledKeys, canceledOperation, canceledController.signal) + const canceled = canceledFirst ? first : second + const retained = canceledFirst ? second : first + const expectedMetadata = new Map(initialMetadata) + for (const key of retainedKeys) { + if (retainedOperation.type === `set`) { + expectedMetadata.set(key, retainedOperation.value) + } else { + expectedMetadata.delete(key) + } + } + + try { + const batchCountBefore = harness.batches.length + const rowsBefore = [...harness.rows.values()] + + canceledController.abort() + + await expect(canceled).rejects.toBeInstanceOf(SyncTransactionAbortedError) + expect(harness.batches).toHaveLength(batchCountBefore) + expect([...harness.rows.values()]).toEqual(rowsBefore) + expect(readMetadata(harness, [0, 1, 2])).toEqual( + observableMetadata(expectedMetadata, [0, 1, 2]), + ) + + persistence.resolve() + await heldTransaction.isPersisted.promise + await expect(retained).resolves.toBeUndefined() + expect(readMetadata(harness, [0, 1, 2])).toEqual( + observableMetadata(expectedMetadata, [0, 1, 2]), + ) + expectPublishedRows( + harness, + new Map([0, 1, 2].map((id) => [id, { id, position: id }] as const)), + ) + } finally { + persistence.resolve() + await heldTransaction.isPersisted.promise.catch(() => undefined) + await Promise.all([ + canceled.catch(() => undefined), + retained.catch(() => undefined), + ]) + harness.unsubscribe() + await Promise.all([harness.liveRows.cleanup(), harness.rows.cleanup()]) + } +} + +it(`publishes one event per key when metadata-only sync retires optimistic work`, async () => { + await runPublicationHistory([ + { + key: 1, + delta: 1, + metadata: [{ key: 1, type: `set`, value: false }], + outcome: `commit`, + }, + { + key: 1, + delta: 1, + metadata: [{ key: 1, type: `delete` }], + outcome: `commit`, + }, + ]) +}) + +it(`releases only canceled metadata keys while another sync remains pending`, async () => { + await expectMetadataCancellationOwnership( + [0, 1], + [1, 2], + { type: `delete` }, + { type: `set`, value: false }, + true, + [ + { present: true, value: undefined }, + { present: true, value: false }, + { present: true, value: null }, + ], + ) +}) + +it(`does not apply canceled metadata to an absent base key`, async () => { + await expectMetadataCancellationOwnership( + [0, 1], + [1, 2], + { type: `set`, value: `canceled` }, + { type: `set`, value: `retained` }, + true, + [{ present: false }, { present: false }, { present: false }], + ) +}) + +it(`settles an older metadata owner after canceling the newer owner`, async () => { + await expectMetadataCancellationOwnership( + [0, 1], + [1, 2], + { type: `delete` }, + { type: `set`, value: `retained` }, + false, + [ + { present: true, value: undefined }, + { present: false }, + { present: true, value: false }, + ], + ) +}) + +fcTest.prop( + [fc.array(publicationRoundArbitrary, { minLength: 1, maxLength: 8 })], + oraclePropertyOptions(50, `collection-publication.metadata-only`), +)( + `keeps metadata-only optimistic settlement a valid keyed diff across histories`, + runPublicationHistory, +) + +fcTest.prop( + [metadataCancellationArbitrary], + oraclePropertyOptions(50, `collection-publication.metadata-cancellation`), +)( + `keeps metadata suppression owned by the remaining pending transactions`, + ({ + canceledKeys, + retainedKeys, + canceledOperation, + retainedOperation, + canceledFirst, + initialMetadata, + }) => + expectMetadataCancellationOwnership( + canceledKeys, + retainedKeys, + canceledOperation, + retainedOperation, + canceledFirst, + initialMetadata, + ), +) diff --git a/packages/db/tests/collection-query-publication-boundaries.test.ts b/packages/db/tests/collection-query-publication-boundaries.test.ts new file mode 100644 index 0000000000..f5197fc4e1 --- /dev/null +++ b/packages/db/tests/collection-query-publication-boundaries.test.ts @@ -0,0 +1,198 @@ +import { expect, it } from 'vitest' +import { MultiSet } from '@tanstack/db-ivm' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { createLiveQueryCollection, eq } from '../src/query/index.js' +import { createTransaction } from '../src/transactions.js' +import { flushPromises } from './utils.js' +import type { SyncConfig } from '../src/types.js' + +type Row = { id: number; value: number } +type Actions = Parameters[`sync`]>[0] + +it(`consolidates Collection handles by instance, not their id or mutable state`, async () => { + const makeCollection = () => + createCollection({ + id: `shared-definition-id`, + getKey: (row) => row.id, + sync: { + sync: ({ markReady }) => { + markReady() + }, + }, + }) + const first = makeCollection() + const second = makeCollection() + try { + await Promise.all([first.preload(), second.preload()]) + const before = new MultiSet([[{ handle: first }, 1]]) + expect( + new MultiSet([ + [{ handle: first }, 1], + [{ handle: second }, -1], + ]) + .consolidate() + .getInner(), + ).toHaveLength(2) + await first.cleanup() + expect( + before + .concat(new MultiSet([[{ handle: first }, -1]])) + .consolidate() + .getInner(), + ).toEqual([]) + } finally { + await Promise.all([first.cleanup(), second.cleanup()]) + } +}) + +it.each([0, 2])(`opens an inner-join window from limit %s`, async (limit) => { + const makeSource = (collectionId: string) => + createCollection({ + id: collectionId, + getKey: (row) => row.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (let id = 1; id <= 5; id++) + write({ type: `insert`, value: { id, value: id } }) + commit() + markReady() + }, + }, + }) + const parent = makeSource(`window-parent-${limit}`) + const child = makeSource(`window-child-${limit}`) + const live = createLiveQueryCollection({ + query: (q) => + q + .from({ a: parent }) + .join({ b: child }, ({ a, b }) => eq(a.id, b.id), `inner`) + .orderBy(({ a }) => a.value) + .limit(limit) + .select(({ a }) => ({ id: a.id, value: a.value })), + }) + try { + await live.preload() + expect(live.size).toBe(limit) + await live.utils.setWindow({ limit: 10 }) + expect(live.toArray.map((row) => row.id)).toEqual([1, 2, 3, 4, 5]) + } finally { + await live.cleanup() + await parent.cleanup() + await child.cleanup() + } +}) + +it.each([1, 99])( + `publishes server echo value %s after two optimistic inserts hold a sync batch`, + async (echoValue) => { + let sync!: Actions + const source = createCollection({ + getKey: (row) => row.id, + sync: { + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + const observed = new Map() + const subscription = source.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) observed.delete(change.key) + else observed.set(change.key, change.value.value) + } + }, + { includeInitialState: true }, + ) + const live = createLiveQueryCollection({ query: (q) => q.from({ source }) }) + await live.preload() + const first = createDeferred() + const second = createDeferred() + const tx1 = createTransaction({ mutationFn: () => first.promise }) + const tx2 = createTransaction({ mutationFn: () => second.promise }) + try { + tx1.mutate(() => source.insert({ id: 1, value: 1 })) + tx2.mutate(() => source.insert({ id: 2, value: 2 })) + sync.begin() + sync.write({ type: `insert`, value: { id: 3, value: 3 } }) + const blocked = sync.commit() + first.resolve() + await tx1.isPersisted.promise + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: echoValue } }) + const echo = sync.commit() + second.resolve() + await tx2.isPersisted.promise + await blocked + await echo + await flushPromises() + expect(source.get(1)?.value).toBe(echoValue) + expect(observed.get(1)).toBe(echoValue) + expect(live.toArray.find((row) => row.id === 1)?.value).toBe(echoValue) + } finally { + first.resolve() + second.resolve() + subscription.unsubscribe() + await live.cleanup() + await source.cleanup() + } + }, +) + +it.each([`depth`, `cycle`] as const)( + `makes a graph hashing $failure failure visible without publishing it`, + async (failure) => { + type DeepRow = { id: number; nested: object } + let sync!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: (row) => row.id, + sync: { + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + const live = createLiveQueryCollection({ + query: (q) => + q + .from({ source }) + .select(({ source: row }) => ({ id: row.id, nested: row.nested })) + .distinct(), + }) + try { + await live.preload() + sync.begin() + sync.write({ type: `insert`, value: { id: 0, nested: { safe: true } } }) + sync.commit() + const before = [...live.toArray] + expect(before).toHaveLength(1) + let nested: object = {} + if (failure === `depth`) { + for (let depth = 0; depth < 800; depth++) nested = { child: nested } + } else { + Object.assign(nested, { self: nested }) + } + sync.begin() + sync.write({ type: `insert`, value: { id: 1, nested } }) + expect(() => sync.commit()).toThrow( + failure === `depth` + ? RangeError + : `Cannot hash cyclic structural values`, + ) + expect(source.has(1)).toBe(true) + expect(live.status).toBe(`error`) + sync.begin() + sync.write({ type: `insert`, value: { id: 2, nested: {} } }) + sync.commit() + expect(live.status).toBe(`error`) + expect(live.toArray).toEqual(before) + } finally { + await live.cleanup() + await source.cleanup() + } + }, +) diff --git a/packages/db/tests/collection-state-retention-oracle.property.test.ts b/packages/db/tests/collection-state-retention-oracle.property.test.ts new file mode 100644 index 0000000000..cd091d6b21 --- /dev/null +++ b/packages/db/tests/collection-state-retention-oracle.property.test.ts @@ -0,0 +1,847 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { DuplicateKeySyncError } from '../src/errors.js' +import { createTransaction } from '../src/transactions.js' +import { oraclePropertyOptions, oracleRuns } from './oracle-config.js' +import { runOptimisticHistory } from './optimistic-history-oracle.js' +import type { OptimisticStep } from './optimistic-history-oracle.js' +import type { Collection } from '../src/collection/index.js' +import type { SyncConfig, TransactionState } from '../src/types.js' + +type RetainedRow = { + id: number + value: number +} + +type SyncActions = Parameters[`sync`]>[0] + +type RetentionAction = + | { type: `insert`; row: RetainedRow } + | { type: `update`; row: RetainedRow } + | { type: `delete`; key: number } + | { type: `replace`; rows: ReadonlyArray } + | { type: `restart` } + | { + type: `reentrantRestart` + row: RetainedRow + commitPhase: `insideListener` | `afterOldReturn` + } + +type RetentionHarness = { + collection: Collection + sync: SyncActions +} + +const retainedRowArbitrary = fc.record({ + id: fc.integer({ min: 0, max: 3 }), + value: fc.integer({ min: -2, max: 2 }), +}) + +function snapshotRetainedRow(row: RetainedRow): RetainedRow { + return { id: row.id, value: row.value } +} + +const retentionActionArbitrary: fc.Arbitrary = fc.oneof( + { + weight: 4, + arbitrary: retainedRowArbitrary.map((row) => ({ + type: `insert` as const, + row, + })), + }, + { + weight: 4, + arbitrary: retainedRowArbitrary.map((row) => ({ + type: `update` as const, + row, + })), + }, + { + weight: 4, + arbitrary: fc + .integer({ min: 0, max: 3 }) + .map((key) => ({ type: `delete` as const, key })), + }, + { + weight: 2, + arbitrary: fc + .uniqueArray(retainedRowArbitrary, { + selector: (row) => row.id, + maxLength: 4, + }) + .map((rows) => ({ type: `replace` as const, rows })), + }, + { weight: 1, arbitrary: fc.constant({ type: `restart` as const }) }, + { + // Keep each phase at least as likely as the original unsplit restart arm. + weight: 3, + arbitrary: fc + .tuple( + retainedRowArbitrary, + fc.constantFrom(`insideListener` as const, `afterOldReturn` as const), + ) + .map(([row, commitPhase]) => ({ + type: `reentrantRestart` as const, + row, + commitPhase, + })), + }, +) + +function createRetentionHarness(): RetentionHarness { + let sync!: SyncActions + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + rowUpdateMode: `full`, + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + return { + collection, + get sync() { + return sync + }, + } +} + +function applyAction( + action: RetentionAction, + model: Map, + sync: SyncActions, +): void { + sync.begin() + switch (action.type) { + case `insert`: { + const previous = model.get(action.row.id) + if (previous !== undefined && previous.value !== action.row.value) { + expect(() => + sync.write({ + type: `insert`, + value: snapshotRetainedRow(action.row), + }), + ).toThrow(DuplicateKeySyncError) + break + } + const expectedRow = snapshotRetainedRow(action.row) + sync.write({ type: `insert`, value: snapshotRetainedRow(action.row) }) + model.set(expectedRow.id, expectedRow) + break + } + case `update`: { + const expectedRow = snapshotRetainedRow(action.row) + sync.write({ type: action.type, value: snapshotRetainedRow(action.row) }) + model.set(expectedRow.id, expectedRow) + break + } + case `delete`: + sync.write({ type: `delete`, key: action.key }) + model.delete(action.key) + break + case `replace`: + sync.truncate() + model.clear() + for (const row of action.rows) { + const expectedRow = snapshotRetainedRow(row) + sync.write({ type: `insert`, value: snapshotRetainedRow(row) }) + model.set(expectedRow.id, expectedRow) + } + break + case `restart`: + case `reentrantRestart`: + throw new Error(`Restart actions require the lifecycle driver`) + } + expect(sync.commit()).toBe(true) +} + +function expectRetainedState( + collection: Collection, + model: ReadonlyMap, +): void { + const expectedRows = [...model.entries()].sort(([a], [b]) => a - b) + const retainedRows = [...collection._state.syncedData.entries()].sort( + ([a], [b]) => a - b, + ) + + expect(retainedRows).toEqual(expectedRows) + expect( + [...collection._state.rowOrigins.keys()] + .filter((key) => !model.has(key)) + .sort((a, b) => a - b), + ).toEqual([]) + expect( + [...collection.state.entries()] + .map(([key, row]) => [key, { id: row.id, value: row.value }] as const) + .sort(([a], [b]) => a - b), + ).toEqual(expectedRows) +} + +async function runRetentionHistory( + actions: ReadonlyArray, +): Promise { + const harness = createRetentionHarness() + const { collection } = harness + const model = new Map() + try { + expectRetainedState(collection, model) + for (const action of actions) { + if (action.type === `restart`) { + await collection.cleanup() + collection.startSyncImmediate() + model.clear() + } else if (action.type === `reentrantRestart`) { + const oldSync = harness.sync + const triggerType = model.has(action.row.id) ? `update` : `insert` + const triggerRow = { + id: action.row.id, + value: (model.get(action.row.id)?.value ?? action.row.value) + 1, + } + const expectedTriggerRow = snapshotRetainedRow(triggerRow) + const restartedRow = { + id: (action.row.id + 1) % 4, + value: action.row.value + 1, + } + const expectedRestartedRow = snapshotRetainedRow(restartedRow) + let cleanup: Promise | undefined + let restarted = false + let restartedSync: SyncActions | undefined + let restartedReceipt: true | Promise | undefined + const batches: Array<{ + changes: Array<{ + type: string + key: string | number + row: RetainedRow + previousRow: RetainedRow | undefined + }> + rows: Array + }> = [] + const subscription = collection.subscribeChanges( + (changes) => { + batches.push({ + changes: changes.map(({ type, key, value, previousValue }) => ({ + type, + key, + row: { id: value.id, value: value.value }, + previousRow: + previousValue === undefined + ? undefined + : { + id: previousValue.id, + value: previousValue.value, + }, + })), + rows: [...collection.values()] + .map(({ id, value }) => ({ id, value })) + .sort((left, right) => left.id - right.id), + }) + if (restarted) return + restarted = true + cleanup = collection.cleanup() + collection.startSyncImmediate() + restartedSync = harness.sync + restartedSync.begin() + restartedSync.write({ + type: `insert`, + value: snapshotRetainedRow(restartedRow), + }) + if (action.commitPhase === `insideListener`) { + restartedReceipt = restartedSync.commit() + } + }, + { includeInitialState: false }, + ) + + oldSync.begin() + oldSync.write({ + type: `update`, + value: snapshotRetainedRow(triggerRow), + }) + expect(oldSync.commit()).toBe(true) + expect(restarted).toBe(true) + expect(restartedSync).toBeDefined() + if (restartedSync === undefined) { + throw new Error(`restarted sync session was not captured`) + } + if (action.commitPhase === `insideListener`) { + expect(restartedReceipt).toBeDefined() + if (restartedReceipt !== true) await restartedReceipt + } else { + expect(restartedSync.commit()).toBe(true) + } + const triggerRows = new Map(model) + triggerRows.set(expectedTriggerRow.id, expectedTriggerRow) + expect(batches).toEqual([ + { + changes: [ + { + type: triggerType, + key: expectedTriggerRow.id, + row: expectedTriggerRow, + previousRow: model.get(expectedTriggerRow.id), + }, + ], + rows: [...triggerRows.values()].sort( + (left, right) => left.id - right.id, + ), + }, + { + // This subscriber observed the trigger, but did not request the + // earlier initial state. Restart retracts its known old-session row. + changes: [ + { + type: `delete`, + key: expectedTriggerRow.id, + row: expectedTriggerRow, + previousRow: undefined, + }, + ], + rows: [], + }, + { + changes: [ + { + type: `insert`, + key: expectedRestartedRow.id, + row: expectedRestartedRow, + previousRow: undefined, + }, + ], + rows: [expectedRestartedRow], + }, + ]) + subscription.unsubscribe() + + await cleanup + model.clear() + model.set(expectedRestartedRow.id, expectedRestartedRow) + } else { + applyAction(action, model, harness.sync) + } + expectRetainedState(collection, model) + } + } finally { + await collection.cleanup() + } +} + +it(`retains only keys in the authoritative synced state`, async () => { + await runRetentionHistory([ + { type: `insert`, row: { id: 1, value: 1 } }, + { type: `insert`, row: { id: 2, value: 2 } }, + { type: `delete`, key: 1 }, + { type: `update`, row: { id: 1, value: -1 } }, + { type: `replace`, rows: [{ id: 3, value: 0 }] }, + { type: `delete`, key: 3 }, + ]) +}) + +it(`retains a missing row introduced by a sync update`, async () => { + await runRetentionHistory([{ type: `update`, row: { id: 1, value: 1 } }]) +}) + +it.each( + ([`insert`, `update`] as const).flatMap((triggerType) => + ([`insideListener`, `afterOldReturn`] as const).map( + (commitPhase) => [triggerType, commitPhase] as const, + ), + ), +)( + `retains an old-session %s and a restarted row committed %s`, + async (triggerType, commitPhase) => { + await runRetentionHistory([ + ...(triggerType === `update` + ? ([{ type: `insert`, row: { id: 1, value: 1 } }] as const) + : []), + { + type: `reentrantRestart`, + row: { id: 1, value: 1 }, + commitPhase, + }, + ]) + }, +) + +it(`releases retained keys after long unique-key churn`, async () => { + const keyCount = 1_000 + const actions: Array = [] + for (let key = 0; key < keyCount; key++) { + actions.push({ type: `insert`, row: { id: key, value: key } }) + actions.push({ type: `delete`, key }) + } + + await runRetentionHistory(actions) +}) + +it(`starts a new sync session without retained publication state`, async () => { + let sync!: SyncActions + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + rowUpdateMode: `full`, + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + const events: Array<{ type: string; key: string | number }> = [] + let subscription: ReturnType | undefined + + try { + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: 1 } }) + expect(sync.commit()).toBe(true) + + subscription = collection.subscribeChanges( + (changes) => { + events.push( + ...changes.map((change) => ({ + type: change.type, + key: change.key, + })), + ) + }, + { includeInitialState: false }, + ) + + sync.begin() + sync.write({ type: `update`, value: { id: 1, value: 2 } }) + collection._state.capturePreSyncVisibleState() + expect(collection._state.preSyncVisibleState.size).toBe(1) + expect(collection._state.recentlySyncedKeys).toEqual(new Set([1])) + + const cleanup = collection.cleanup() + const retainedAfterCleanup = { + visibleRows: collection._state.preSyncVisibleState.size, + virtualRows: collection._state.preSyncVirtualState.size, + recentKeys: collection._state.recentlySyncedKeys.size, + } + await cleanup + + events.length = 0 + collection.startSyncImmediate() + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: 3 } }) + expect(sync.commit()).toBe(true) + + expect({ retainedAfterCleanup, events }).toEqual({ + retainedAfterCleanup: { visibleRows: 0, virtualRows: 0, recentKeys: 0 }, + events: [{ type: `insert`, key: 1 }], + }) + } finally { + subscription?.unsubscribe() + await collection.cleanup() + } +}) + +it(`keeps a restarted session's publication state after the old listener returns`, async () => { + let sync!: SyncActions + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + rowUpdateMode: `full`, + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + let cleanup: Promise | undefined + let restarted = false + const subscription = collection.subscribeChanges( + () => { + if (restarted) return + restarted = true + cleanup = collection.cleanup() + collection.startSyncImmediate() + collection._state.preSyncVisibleState.set(2, { id: 2, value: 2 }) + collection._state.recentlySyncedKeys.add(2) + }, + { includeInitialState: false }, + ) + + try { + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: 1 } }) + expect(sync.commit()).toBe(true) + + expect(restarted).toBe(true) + expect(collection._state.preSyncVisibleState).toEqual( + new Map([[2, { id: 2, value: 2 }]]), + ) + expect(collection._state.recentlySyncedKeys).toEqual(new Set([2])) + expect(collection._state.hasReceivedFirstCommit).toBe(false) + + sync.begin() + sync.write({ type: `insert`, value: { id: 3, value: 3 } }) + expect(sync.commit()).toBe(true) + expect(collection._state.preSyncVisibleState.size).toBe(0) + expect(collection._state.preSyncVirtualState.size).toBe(0) + expect(collection._state.hasReceivedFirstCommit).toBe(true) + await Promise.resolve() + expect(collection._state.recentlySyncedKeys.size).toBe(0) + await cleanup + } finally { + subscription.unsubscribe() + await collection.cleanup() + } +}) + +it(`does not let an old publication microtask clear restarted sync state`, async () => { + let sync!: SyncActions + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + rowUpdateMode: `full`, + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + + try { + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: 1 } }) + expect(sync.commit()).toBe(true) + + const cleanup = collection.cleanup() + collection.startSyncImmediate() + sync.begin() + sync.write({ type: `insert`, value: { id: 2, value: 2 } }) + collection._state.capturePreSyncVisibleState() + expect(collection._state.recentlySyncedKeys).toEqual(new Set([2])) + + await Promise.resolve() + + expect(collection._state.recentlySyncedKeys).toEqual(new Set([2])) + + expect(sync.commit()).toBe(true) + expect(collection._state.hasReceivedFirstCommit).toBe(true) + await Promise.resolve() + expect(collection._state.preSyncVisibleState.size).toBe(0) + expect(collection._state.preSyncVirtualState.size).toBe(0) + expect(collection._state.recentlySyncedKeys.size).toBe(0) + await cleanup + } finally { + await collection.cleanup() + } +}) + +it(`publishes a virtual-state update when a restarted optimistic row is confirmed`, async () => { + let sync!: SyncActions + let syncSession = 0 + let releaseMutation!: () => void + const mutationHold = new Promise((resolve) => { + releaseMutation = resolve + }) + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + rowUpdateMode: `full`, + sync: (actions) => { + sync = actions + syncSession++ + if (syncSession === 1) actions.markReady() + }, + }, + }) + type ObservedRow = RetainedRow & { + $collectionId: string + $key: number + $origin: `local` | `remote` + $synced: boolean + } + type ObservedChange = { + type: string + key: string | number + value: ObservedRow + previousValue?: ObservedRow + } + const snapshotRow = (row: ObservedRow): ObservedRow => ({ + id: row.id, + value: row.value, + $collectionId: row.$collectionId, + $key: row.$key, + $origin: row.$origin, + $synced: row.$synced, + }) + const publications: Array<{ + changes: Array + rows: Array + }> = [] + const restartStatuses: Array = [] + const settlementTimeline: Array<`publication` | `receipt`> = [] + let restarted = false + let readMutationState: (() => TransactionState) | undefined + let rollbackMutation: (() => void) | undefined + let mutationCommit: Promise | undefined + let syncReceipt: ReturnType | undefined + let syncReceiptOutcome: Promise | undefined + let syncReceiptSettled = false + const subscription = collection.subscribeChanges( + (changes) => { + publications.push({ + changes: changes.map(({ type, key, value, previousValue }) => ({ + type, + key, + value: snapshotRow(value), + ...(previousValue === undefined + ? {} + : { previousValue: snapshotRow(previousValue) }), + })), + rows: [...collection.state.values()].map(snapshotRow), + }) + if (changes.some(({ type, key }) => type === `update` && key === 2)) { + queueMicrotask(() => settlementTimeline.push(`publication`)) + } + if (restarted || !changes.some(({ key }) => key === 1)) return + + restarted = true + restartStatuses.push(collection.status) + void collection.cleanup() + restartStatuses.push(collection.status) + collection.startSyncImmediate() + restartStatuses.push(collection.status) + sync.markReady() + restartStatuses.push(collection.status) + + const transaction = createTransaction({ + autoCommit: false, + mutationFn: () => mutationHold, + }) + readMutationState = () => transaction.state + rollbackMutation = () => transaction.rollback() + void transaction.isPersisted.promise.catch(() => undefined) + transaction.mutate(() => collection.insert({ id: 2, value: 2 })) + mutationCommit = transaction.commit() + + sync.begin() + sync.write({ type: `insert`, value: { id: 2, value: 2 } }) + syncReceipt = sync.commit() + if (syncReceipt !== true) { + syncReceiptOutcome = syncReceipt.then((value) => { + settlementTimeline.push(`receipt`) + syncReceiptSettled = true + return value + }) + } + }, + { includeInitialState: false }, + ) + + try { + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: 1 } }) + expect(sync.commit()).toBe(true) + + const remoteRow = (id: number): ObservedRow => ({ + id, + value: id, + $collectionId: collection.id, + $key: id, + $origin: `remote`, + $synced: true, + }) + const localRow = (id: number): ObservedRow => ({ + id, + value: id, + $collectionId: collection.id, + $key: id, + $origin: `local`, + $synced: false, + }) + const expectedPublications = [ + { + changes: [{ type: `insert`, key: 1, value: remoteRow(1) }], + rows: [remoteRow(1)], + }, + { changes: [{ type: `delete`, key: 1, value: remoteRow(1) }], rows: [] }, + { + changes: [{ type: `insert`, key: 2, value: localRow(2) }], + rows: [localRow(2)], + }, + { + changes: [ + { + type: `update`, + key: 2, + value: remoteRow(2), + previousValue: localRow(2), + }, + ], + rows: [remoteRow(2)], + }, + ] + expect(publications).toEqual(expectedPublications.slice(0, 3)) + expect([...collection.state.keys()]).toEqual([2]) + expect(restartStatuses).toEqual([`ready`, `cleaned-up`, `loading`, `ready`]) + expect(collection.status).toBe(`ready`) + + expect(syncReceipt).toBeDefined() + expect(syncReceipt).not.toBe(true) + expect(syncReceiptSettled).toBe(false) + if (syncReceipt === undefined || syncReceipt === true) { + throw new Error(`restarted sync receipt was not parked`) + } + expect(syncReceipt).toBeInstanceOf(Promise) + expect(syncReceiptOutcome).toBeDefined() + expect(rollbackMutation).toBeDefined() + await Promise.resolve() + expect(syncReceiptSettled).toBe(false) + expect(settlementTimeline).toEqual([]) + + rollbackMutation?.() + expect(publications).toEqual(expectedPublications) + expect(syncReceiptSettled).toBe(false) + await expect(syncReceiptOutcome).resolves.toBeUndefined() + expect(syncReceiptSettled).toBe(true) + expect(settlementTimeline).toEqual([`publication`, `receipt`]) + expect(publications).toEqual(expectedPublications) + expect([...collection.state.values()].map(snapshotRow)).toEqual([ + remoteRow(2), + ]) + + releaseMutation() + await mutationCommit + expect(readMutationState?.()).toBe(`failed`) + expect(publications).toEqual(expectedPublications) + expect([...collection.state.values()].map(snapshotRow)).toEqual([ + remoteRow(2), + ]) + } finally { + releaseMutation() + await mutationCommit + subscription.unsubscribe() + await collection.cleanup() + } +}) + +fcTest.prop( + [fc.array(retentionActionArbitrary, { minLength: 1, maxLength: 20 })], + oraclePropertyOptions(100, `collection-state.retention`), +)( + `matches retained authoritative state without optimistic overlays after every committed sync history`, + async (actions) => { + await runRetentionHistory(actions) + }, +) + +const historyRow = fc.record({ + id: fc.integer({ min: 1, max: 3 }), + a: fc.integer({ min: -2, max: 2 }), + b: fc.integer({ min: -2, max: 2 }), + c: fc.integer({ min: -2, max: 2 }), +}) +const optimisticStep: fc.Arbitrary = fc.oneof( + { + weight: 4, + arbitrary: fc.record({ + type: fc.constant(`edit` as const), + key: fc.integer({ min: 1, max: 3 }), + fields: fc + .record( + { + a: fc.integer({ min: -2, max: 2 }), + b: fc.integer({ min: -2, max: 2 }), + c: fc.integer({ min: -2, max: 2 }), + }, + { requiredKeys: [] }, + ) + .filter((fields) => Object.keys(fields).length > 0), + optimistic: fc.boolean(), + }), + }, + { + weight: 4, + arbitrary: fc.record({ + type: fc.constant(`settle` as const), + slot: fc.nat(5), + success: fc.boolean(), + cascade: fc.boolean(), + }), + }, + { + weight: 3, + arbitrary: fc.record({ + type: fc.constant(`sync` as const), + rows: fc.uniqueArray(historyRow, { + selector: (row) => row.id, + maxLength: 3, + }), + truncate: fc.boolean(), + immediate: fc.boolean(), + copies: fc.integer({ min: 1, max: 2 }), + }), + }, +) +const optimisticHistory = fc.record({ + initial: fc.uniqueArray(historyRow, { + selector: (row) => row.id, + maxLength: 3, + }), + steps: fc.array(optimisticStep, { minLength: 2, maxLength: 24 }), +}) + +// These are replay programs for the same model and driver as randomized runs, +// not separate assertions that only know the reported final state. +const insertionPrefix: Array = [ + { type: `edit`, key: 1, fields: { a: 1 }, optimistic: true }, + { type: `edit`, key: 1, fields: { b: 2 }, optimistic: true }, + { type: `settle`, slot: 1, success: true, cascade: false }, +] +it.each([true, false])( + `replays insert dependency settlement, accepted=%s`, + async (success) => { + await runOptimisticHistory( + [], + [ + ...insertionPrefix, + { type: `settle`, slot: 0, success, cascade: false }, + ], + ) + }, +) +it.each([true, false])( + `preserves a whole-row mutation snapshot across sync, truncate=%s`, + async (truncate) => { + await runOptimisticHistory( + [{ id: 1, a: 0, b: 0, c: 0 }], + [ + { type: `edit`, key: 1, fields: { a: 1 }, optimistic: true }, + { + type: `sync`, + rows: [{ id: 1, a: 0, b: 2, c: 3 }], + immediate: !truncate, + truncate, + copies: 1, + }, + { type: `settle`, slot: 0, success: true, cascade: false }, + ], + ) + }, +) +fcTest.prop([optimisticHistory], { numRuns: oracleRuns(60), seed: 86103 })( + `matches optimistic ownership and publication histories with a fixed seed`, + async ({ initial, steps }) => { + await runOptimisticHistory(initial, steps) + }, +) +fcTest.prop( + [optimisticHistory], + oraclePropertyOptions(100, `collection-state.optimistic-history`), +)( + `matches optimistic ownership and publication histories with a random or replayed seed`, + async ({ initial, steps }) => { + await runOptimisticHistory(initial, steps) + }, +) diff --git a/packages/db/tests/collection-subscribe-changes.test.ts b/packages/db/tests/collection-subscribe-changes.test.ts index 4f851f08a7..269f726714 100644 --- a/packages/db/tests/collection-subscribe-changes.test.ts +++ b/packages/db/tests/collection-subscribe-changes.test.ts @@ -695,8 +695,9 @@ describe(`Collection.subscribeChanges`, () => { expect(callback).not.toHaveBeenCalled() }) - it(`should correctly handle filtered updates that transition between filter states`, () => { + it(`should correctly handle filtered updates that transition between filter states`, async () => { const callback = vi.fn() + const emitter = mitt() // Create collection with items that have a status field const collection = createCollection<{ @@ -708,6 +709,21 @@ describe(`Collection.subscribeChanges`, () => { getKey: (item) => item.id, sync: { sync: ({ begin, write, commit }) => { + // Feed persisted mutations back through the real sync transaction + // path so this test also observes applied-receipt failures. + // @ts-expect-error don't trust Mitt's typing and this works. + emitter.on(`*`, (_, changes: Array) => { + begin() + changes.forEach((change) => { + write({ + type: change.type, + // @ts-expect-error TODO type changes + value: change.modified, + }) + }) + commit() + }) + // Start with some initial data begin() write({ @@ -723,38 +739,8 @@ describe(`Collection.subscribeChanges`, () => { }, }) - const mutationFn: MutationFn = async () => { - // Simulate sync by writing the mutations back - const syncCollection = collection as any - syncCollection.config.sync.sync({ - collection: syncCollection, - begin: () => { - syncCollection._state.pendingSyncedTransactions.push({ - committed: false, - operations: [], - }) - }, - write: (messageWithoutKey: any) => { - const pendingTransaction = - syncCollection._state.pendingSyncedTransactions[ - syncCollection._state.pendingSyncedTransactions.length - 1 - ] - const key = syncCollection.getKeyFromItem(messageWithoutKey.value) - const message = { ...messageWithoutKey, key } - pendingTransaction.operations.push(message) - }, - commit: () => { - const pendingTransaction = - syncCollection._state.pendingSyncedTransactions[ - syncCollection._state.pendingSyncedTransactions.length - 1 - ] - pendingTransaction.committed = true - syncCollection.commitPendingTransactions() - }, - markReady: () => { - syncCollection.markReady() - }, - }) + const mutationFn: MutationFn = ({ transaction }) => { + emitter.emit(`sync`, transaction.mutations) return Promise.resolve() } @@ -858,6 +844,15 @@ describe(`Collection.subscribeChanges`, () => { // Should not emit any events for inactive items expect(callback).not.toHaveBeenCalled() + // Keep the immediate optimistic assertions isolated above, then prove that + // every auto-commit also completes through applied-receipt settlement. + await Promise.all([ + tx1.isPersisted.promise, + tx2.isPersisted.promise, + tx3.isPersisted.promise, + tx4.isPersisted.promise, + ]) + // Clean up subscription.unsubscribe() }) @@ -2151,6 +2146,74 @@ describe(`Collection.subscribeChanges`, () => { whereExpression: eq(new PropRef([`status`]), `active`), }) }).toThrow(`Cannot specify both 'where' and 'whereExpression' options`) + expect(collection.subscriberCount).toBe(0) + }) + + it(`releases subscriber ownership when a where callback throws`, () => { + const failure = new Error(`where callback failed`) + const collection = createCollection<{ id: number; status: string }>({ + id: `where-callback-error-test`, + getKey: (item) => item.id, + sync: { sync: () => {} }, + }) + + expect(() => + collection.subscribeChanges(() => {}, { + where: () => { + throw failure + }, + }), + ).toThrow(failure) + expect(collection.subscriberCount).toBe(0) + }) + + it(`rolls back subscriber ownership when starting sync throws`, () => { + const failure = new Error(`sync setup failed`) + const collection = createCollection<{ id: number }>({ + id: `subscriber-start-sync-error-test`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: () => { + throw failure + }, + }, + }) + + expect(() => collection.subscribeChanges(() => {})).toThrow(failure) + expect(collection.subscriberCount).toBe(0) + expect(collection.status).toBe(`error`) + }) + + it(`preserves setup failure when subscription cleanup also throws`, () => { + const loadFailure = new Error(`initial subset failed`) + const unloadFailure = new Error(`subset cleanup failed`) + const collection = createCollection<{ id: number }>({ + id: `subscriber-load-and-unload-error-test`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + throw unloadFailure + }, + } + }, + }, + }) + + expect(() => + collection.subscribeChanges(() => {}, { + includeInitialState: true, + onLoadSubsetResult: () => { + throw loadFailure + }, + }), + ).toThrow(loadFailure) + expect(collection.subscriberCount).toBe(0) }) }) @@ -2608,6 +2671,123 @@ describe(`Virtual properties`, () => { expect(collection.state.get(`row-1`)?.$origin).toBe(`remote`) }) + it.each([false, true])( + `keeps a completed reinsert visible before its sync echo (delete echoed: %s)`, + async (deleteEchoed) => { + let echoDelete!: () => void + const collection = createCollection< + { id: string; value: string }, + string + >({ + getKey: (row) => row.id, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row`, value: `original` } }) + commit() + markReady() + echoDelete = () => { + begin() + write({ type: `delete`, value: { id: `row`, value: `original` } }) + commit() + } + }, + }, + onDelete: () => Promise.resolve(), + onInsert: () => Promise.resolve(), + }) + try { + await collection.delete(`row`).isPersisted.promise + expect(collection.has(`row`)).toBe(false) + if (deleteEchoed) echoDelete() + await collection.insert({ id: `row`, value: `replacement` }).isPersisted + .promise + expect(collection.get(`row`)?.value).toBe(`replacement`) + } finally { + await collection.cleanup() + } + }, + ) + + it.each([`before`, `after`] as const)( + `replaces a direct mutation settling %s truncate with its authoritative row`, + async (settlement) => { + let finishMutation!: () => void + const mutation = new Promise((resolve) => { + finishMutation = resolve + }) + let syncFns: + | { + begin: () => void + write: (change: { + type: `insert` + value: { id: string; value: string } + }) => void + commit: () => true | Promise + truncate: () => void + } + | undefined + + const collection = createCollection< + { id: string; value: string }, + string + >({ + id: `truncate-replaces-completed-direct-mutation`, + getKey: (item) => item.id, + startSync: true, + sync: { + sync: ({ begin, write, commit, truncate, markReady }) => { + syncFns = { begin, write, commit, truncate } + markReady() + }, + }, + onInsert: () => mutation, + }) + + await collection.stateWhenReady() + const transaction = collection.insert({ id: `row-1`, value: `client` }) + if (settlement === `before`) { + finishMutation() + await transaction.isPersisted.promise + } + expect(collection.get(`row-1`)).toMatchObject({ + id: `row-1`, + value: `client`, + }) + + if (!syncFns) throw new Error(`Sync not ready`) + syncFns.begin() + syncFns.truncate() + syncFns.write({ + type: `insert`, + value: { id: `row-1`, value: `server` }, + }) + const applied = syncFns.commit() + if (settlement === `after`) { + // An unrelated mutation recomputes the optimistic overlay before the + // acknowledged insertion completes; it must not erase that evidence. + const peer = collection.insert({ id: `other`, value: `peer` }) + finishMutation() + await transaction.isPersisted.promise + await peer.isPersisted.promise + } + if (applied !== true) await applied + await waitForChanges() + + expect(collection.get(`row-1`)).toMatchObject({ + id: `row-1`, + value: `server`, + }) + expect(collection.state.get(`row-1`)?.$synced).toBe(true) + // A same-key sync during the active mutation is a local acknowledgement; + // after completion, truncate is an independent remote replacement. + expect(collection.state.get(`row-1`)?.$origin).toBe( + settlement === `after` ? `local` : `remote`, + ) + }, + ) + it(`should preserve local origin for rows confirmed in the same truncate batch`, async () => { let syncFns: | { @@ -2650,6 +2830,7 @@ describe(`Virtual properties`, () => { }) }) syncFns.commit() + return Promise.resolve() }, }) diff --git a/packages/db/tests/collection-subscriber-duplicate-inserts.test.ts b/packages/db/tests/collection-subscriber-duplicate-inserts.test.ts index 8b9d6be57c..b53f2bcf34 100644 --- a/packages/db/tests/collection-subscriber-duplicate-inserts.test.ts +++ b/packages/db/tests/collection-subscriber-duplicate-inserts.test.ts @@ -15,8 +15,9 @@ import type { ChangeMessage } from '../src/types.js' * If duplicate inserts reach D2, multiplicity becomes > 1, and deletes won't * properly remove items (multiplicity goes from 2 to 1, not triggering removal). * - * The fix: CollectionSubscriber tracks keys sent to D2 (sentToD2Keys) and - * filters out duplicate inserts before they reach the pipeline. + * The source boundary tracks the exact row sent for each key. It filters + * duplicate inserts and uses the stored row for later D2 retractions. The + * generated reconciliation oracle covers that stateful boundary directly. * * Additionally, for JOIN queries with lazy sources: * - The includeInitialState fix ensures internal lazy-loading subscriptions diff --git a/packages/db/tests/collection-subscription-lifecycle-grammar.ts b/packages/db/tests/collection-subscription-lifecycle-grammar.ts new file mode 100644 index 0000000000..ae3d4312ef --- /dev/null +++ b/packages/db/tests/collection-subscription-lifecycle-grammar.ts @@ -0,0 +1,697 @@ +import { fc } from '@fast-check/vitest' + +export type DemandName = `a` | `b` +export type AttemptScope = `current` | `obsolete` +export type AttemptAge = `oldest` | `newest` +export type LifecycleCommand = + | { type: `request`; demand: DemandName } + | { type: `abort`; demand: DemandName } + | { type: `release`; demand: DemandName } + | { + type: `settle` + demand: DemandName + scope: AttemptScope + age: AttemptAge + outcome: `resolve` | `reject` + } + | { type: `truncate` } + | { type: `cleanup` } + | { type: `restart` } + | { type: `unsubscribe` } + +export const lifecycleCommandArbitrary: fc.Arbitrary = + fc.oneof( + fc.record({ + type: fc.constant(`request` as const), + demand: fc.constantFrom(`a` as const, `b` as const), + }), + fc.record({ + type: fc.constant(`abort` as const), + demand: fc.constantFrom(`a` as const, `b` as const), + }), + fc.record({ + type: fc.constant(`release` as const), + demand: fc.constantFrom(`a` as const, `b` as const), + }), + fc.record({ + type: fc.constant(`settle` as const), + demand: fc.constantFrom(`a` as const, `b` as const), + scope: fc.constantFrom(`current` as const, `obsolete` as const), + age: fc.constantFrom(`oldest` as const, `newest` as const), + outcome: fc.constantFrom(`resolve` as const, `reject` as const), + }), + fc.constant({ type: `truncate` as const }), + fc.constant({ type: `cleanup` as const }), + fc.constant({ type: `restart` as const }), + fc.constant({ type: `unsubscribe` as const }), + ) + +export type LifecycleOwner = { + id: number + demand: DemandName + aborted: boolean + attemptId?: number +} + +export type LifecycleAttempt = { + id: number + ownerId: number + demand: DemandName + session: number + replay: number + settled: boolean + outcome?: `resolve` | `reject` + gating: boolean + inReplacement: boolean + reportable: boolean + aborted: boolean + failure: Error +} + +export type LifecycleLoadEvent = Pick< + LifecycleAttempt, + `id` | `demand` | `session` | `replay` +> +export type LifecycleUnloadEvent = { + attemptId: number + handlerSession: number +} +export type LifecycleErrorEvent = { attemptId: number; error: Error } +export type LifecycleResultKind = `promise` | `true` +export type LifecycleResultEvent = { + attemptId: number | `unacquired` + resultKind: LifecycleResultKind +} +export type LifecycleTraceEvent = + | ({ type: `load` } & LifecycleLoadEvent) + | ({ type: `unload` } & LifecycleUnloadEvent) + | { type: `error`; attemptId: number } + | ({ type: `result` } & LifecycleResultEvent) + | { type: `status`; status: string } + | { type: `publication` } + +export type LifecycleModel = { + acquisitionMode: `async-pending` | `sync-success` + cancellation: `manual` | `reject` + active: boolean + unsubscribed: boolean + session: number + replay: number + publicationBarrierOpen: boolean + nextOwnerId: number + nextAttemptId: number + owners: Array + attempts: Array + loads: Array + unloads: Array + errors: Array + results: Array + publications: number + statuses: Array + status: string + collectionStatus: `ready` | `cleaned-up` + lastError?: Error + failureForAttempt: (attemptId: number) => Error + reach: Set + trace: Array +} + +export type LifecycleEffect = { + ownerId?: number + attemptId?: number + requestResult?: boolean +} + +export function createLifecycleModel( + acquisitionMode: LifecycleModel[`acquisitionMode`] = `async-pending`, + failureForAttempt: (attemptId: number) => Error = (attemptId) => + new Error(`attempt ${attemptId} failed`), + cancellation: LifecycleModel[`cancellation`] = `manual`, +): LifecycleModel { + return { + acquisitionMode, + cancellation, + active: true, + unsubscribed: false, + session: 0, + replay: 0, + publicationBarrierOpen: false, + nextOwnerId: 0, + nextAttemptId: 0, + owners: [], + attempts: [], + loads: [], + unloads: [], + errors: [], + results: [], + publications: 0, + statuses: [], + status: `ready`, + collectionStatus: `ready`, + reach: new Set(), + trace: [], + failureForAttempt, + } +} + +function setStatus(model: LifecycleModel, queuedReplay = false): void { + if (model.unsubscribed) return + const status = + model.active && + (queuedReplay || model.attempts.some(({ gating }) => gating)) + ? `loadingSubset` + : `ready` + if (status !== model.status) { + model.status = status + model.statuses.push(status) + model.trace.push({ type: `status`, status }) + } +} + +// Settled failure is not an authoritative replacement. Keep subsequent reads +// private until the failed owner retires or a new replay succeeds. +function replacementSucceeded(model: LifecycleModel): boolean { + return ( + !model.attempts.some( + ({ gating, inReplacement }) => gating && inReplacement, + ) && + model.owners.every( + ({ attemptId }) => + attemptId === undefined || + model.attempts[attemptId]!.outcome === `resolve`, + ) + ) +} + +function startAttempt( + model: LifecycleModel, + owner: LifecycleOwner, + trace = true, +): LifecycleAttempt { + const id = model.nextAttemptId++ + const attempt: LifecycleAttempt = { + id, + ownerId: owner.id, + demand: owner.demand, + session: model.session, + replay: model.replay, + settled: model.acquisitionMode === `sync-success`, + ...(model.acquisitionMode === `sync-success` + ? { outcome: `resolve` as const } + : {}), + gating: model.acquisitionMode === `async-pending`, + // Initial/progressive acquisition can hold readiness without joining the + // authoritative replacement's publication boundary. + inReplacement: model.publicationBarrierOpen, + reportable: true, + aborted: false, + failure: model.failureForAttempt(id), + } + model.attempts.push(attempt) + model.reach.add( + `attempt-session:${attempt.session === 0 ? `initial` : `restarted`}`, + ) + model.reach.add( + `attempt-replay:${attempt.replay === 0 ? `initial` : `replayed`}`, + ) + model.reach.add( + `attempt-location:${attempt.session === 0 ? `initial` : `restarted`}:${attempt.replay === 0 ? `initial` : `replayed`}`, + ) + model.loads.push({ + id, + demand: attempt.demand, + session: attempt.session, + replay: attempt.replay, + }) + if (trace) { + model.trace.push({ + type: `load`, + id, + demand: attempt.demand, + session: attempt.session, + replay: attempt.replay, + }) + } + owner.attemptId = id + return attempt +} + +function retireAttempt( + model: LifecycleModel, + owner: LifecycleOwner, + options: { unload: boolean; trace?: boolean; keepPending?: boolean }, +): void { + if (owner.attemptId === undefined) return + const attempt = model.attempts[owner.attemptId] + owner.attemptId = undefined + if (!attempt) throw new Error(`model lost attempt`) + attempt.gating = options.keepPending === true && !attempt.settled + attempt.reportable = false + abortAttempt(model, attempt) + if (options.unload) { + model.unloads.push({ attemptId: attempt.id, handlerSession: model.session }) + if (options.trace !== false) { + model.trace.push({ + type: `unload`, + attemptId: attempt.id, + handlerSession: model.session, + }) + } + } +} + +function abortAttempt(model: LifecycleModel, attempt: LifecycleAttempt): void { + attempt.aborted = true + if (model.cancellation === `reject` && !attempt.settled) { + attempt.settled = true + attempt.outcome = `reject` + attempt.gating = false + } +} + +function selectAttempt( + model: LifecycleModel, + command: Extract, +): LifecycleAttempt | undefined { + const currentAttemptIds = new Set( + model.owners.flatMap(({ attemptId }) => + attemptId === undefined ? [] : [attemptId], + ), + ) + const candidates = model.attempts.filter( + (attempt) => + !attempt.settled && + attempt.demand === command.demand && + (command.scope === `current` + ? currentAttemptIds.has(attempt.id) + : !currentAttemptIds.has(attempt.id)), + ) + return command.age === `oldest` ? candidates[0] : candidates.at(-1) +} + +/** Pure reference transition. It never reads adapter callbacks or SUT state. */ +export function reduceLifecycle( + model: LifecycleModel, + command: LifecycleCommand, +): LifecycleEffect { + model.reach.add(`command:${command.type}`) + if (model.unsubscribed) { + if (command.type === `cleanup` && model.active) { + model.reach.add(`effective:cleanup`) + model.active = false + model.collectionStatus = `cleaned-up` + } else if (command.type === `restart` && !model.active) { + model.reach.add(`effective:restart`) + model.active = true + model.session++ + model.replay = 0 + model.publicationBarrierOpen = false + model.collectionStatus = `ready` + } else { + model.reach.add(`noop:${command.type}`) + } + return { requestResult: false } + } + + if (command.type === `request`) { + model.reach.add(`effective:request`) + if (model.owners.some(({ demand }) => demand === command.demand)) { + model.reach.add(`duplicate-owner`) + } + if (!model.active) model.reach.add(`request-while-cleaned`) + const owner: LifecycleOwner = { + id: model.nextOwnerId++, + demand: command.demand, + aborted: false, + } + model.owners.push(owner) + if (model.active) { + const attemptId = startAttempt(model, owner).id + const result = { + attemptId, + resultKind: + model.acquisitionMode === `async-pending` + ? (`promise` as const) + : (`true` as const), + } + model.results.push(result) + model.trace.push({ type: `result`, ...result }) + } else { + // A waiting owner gets a promise now, without claiming an acquisition. + const result = { attemptId: `unacquired`, resultKind: `promise` } as const + model.results.push(result) + model.trace.push({ type: `result`, ...result }) + } + setStatus(model) + if (!model.publicationBarrierOpen) { + model.publications++ + model.trace.push({ type: `publication` }) + } + return { ownerId: owner.id, requestResult: true } + } + + if (command.type === `abort`) { + const owner = model.owners.find( + ({ demand, aborted }) => demand === command.demand && !aborted, + ) + if (!owner) { + model.reach.add(`noop:abort`) + return {} + } + model.reach.add(`effective:abort`) + owner.aborted = true + if (owner.attemptId !== undefined) { + const attempt = model.attempts[owner.attemptId]! + abortAttempt(model, attempt) + attempt.reportable = false + } + setStatus(model) + return { ownerId: owner.id } + } + + if (command.type === `release`) { + const index = model.owners.findIndex( + ({ demand }) => demand === command.demand, + ) + if (index === -1) { + model.reach.add(`noop:release`) + return {} + } + model.reach.add(`effective:release`) + const [owner] = model.owners.splice(index, 1) + retireAttempt(model, owner!, { unload: true }) + // Retirement removes the logical owner, including its older transports. + for (const attempt of model.attempts) { + if (attempt.ownerId === owner!.id) attempt.gating = false + } + if (model.publicationBarrierOpen && replacementSucceeded(model)) { + model.publicationBarrierOpen = false + } + setStatus(model) + return { ownerId: owner!.id } + } + + if (command.type === `settle`) { + const attempt = selectAttempt(model, command) + if (!attempt) { + model.reach.add(`noop:settle`) + return {} + } + model.reach.add(`effective:settle`) + model.reach.add(`settle-scope:${command.scope}`) + model.reach.add(`settle-age:${command.age}`) + model.reach.add(`settle-outcome:${command.outcome}`) + model.reach.add(`settle:${command.scope}:${command.age}:${command.outcome}`) + attempt.settled = true + attempt.outcome = command.outcome + attempt.gating = false + if ( + command.outcome === `reject` && + attempt.reportable && + !attempt.aborted + ) { + model.lastError = attempt.failure + model.errors.push({ attemptId: attempt.id, error: attempt.failure }) + model.trace.push({ type: `error`, attemptId: attempt.id }) + } + if (model.publicationBarrierOpen && replacementSucceeded(model)) { + model.publicationBarrierOpen = false + } + setStatus(model) + return { attemptId: attempt.id } + } + + if (command.type === `truncate`) { + if (!model.active) { + model.reach.add(`noop:truncate`) + return {} + } + model.reach.add(`effective:truncate`) + if ( + model.replay > 0 && + model.attempts.some( + ({ session, settled }) => session === model.session && !settled, + ) + ) { + model.reach.add(`overlapping-replay`) + } + model.replay++ + // Replay setup is asynchronous even when every acquisition is synchronous + // or canceled. Logical owners queue setup; live owners start acquisitions. + setStatus(model, model.owners.length > 0) + // A canceled-only truncate starts no new work, but cannot end a prior + // replay's publication wait while that owner still owes settlement. + model.publicationBarrierOpen = + model.owners.some(({ aborted }) => !aborted) || + model.attempts.some( + ({ gating, inReplacement }) => gating && inReplacement, + ) + for (const owner of model.owners) { + // Replacing an acquisition is not releasing its logical owner. Delayed + // cancellation still holds readiness; replay work also holds publication. + retireAttempt(model, owner, { + unload: true, + keepPending: true, + }) + if (!owner.aborted) { + startAttempt(model, owner) + } + } + if (model.publicationBarrierOpen && replacementSucceeded(model)) { + model.publicationBarrierOpen = false + } + setStatus(model) + return {} + } + + if (command.type === `cleanup`) { + if (!model.active) { + model.reach.add(`noop:cleanup`) + return {} + } + model.reach.add(`effective:cleanup`) + const current = model.attempts.filter( + ({ session }) => session === model.session, + ) + if ( + current.some(({ settled }) => settled) && + current.some(({ settled }) => !settled) + ) { + model.reach.add(`partial-generation-supersession`) + } + for (const owner of model.owners) + retireAttempt(model, owner, { unload: false }) + for (const attempt of model.attempts) attempt.gating = false + model.active = false + model.publicationBarrierOpen = false + model.collectionStatus = `cleaned-up` + setStatus(model) + return {} + } + + if (command.type === `restart`) { + if (model.active) { + model.reach.add(`noop:restart`) + return {} + } + model.reach.add(`effective:restart`) + model.active = true + model.session++ + model.replay = 0 + model.publicationBarrierOpen = model.owners.some(({ aborted }) => !aborted) + model.collectionStatus = `ready` + setStatus(model, model.owners.length > 0) + const replayLoads: Array = [] + for (const owner of model.owners) { + if (!owner.aborted) replayLoads.push(startAttempt(model, owner, false)) + } + if (model.publicationBarrierOpen && replacementSucceeded(model)) { + model.publicationBarrierOpen = false + } + model.publications++ + model.trace.push({ type: `publication` }) + for (const load of replayLoads) { + model.trace.push({ + type: `load`, + id: load.id, + demand: load.demand, + session: load.session, + replay: load.replay, + }) + } + setStatus(model) + return {} + } + + model.reach.add(`effective:unsubscribe`) + for (const owner of model.owners) + retireAttempt(model, owner, { unload: true }) + model.owners.length = 0 + model.unsubscribed = true + return {} +} + +export const greenLifecycleHistoryArbitrary = fc.array( + lifecycleCommandArbitrary, + { minLength: 1, maxLength: 20 }, +) + +export const syncLifecycleHistoryArbitrary = fc.array( + lifecycleCommandArbitrary, + { minLength: 1, maxLength: 20 }, +) + +export const settle = ( + demand: DemandName, + scope: AttemptScope, + age: AttemptAge, + outcome: `resolve` | `reject`, +): LifecycleCommand => ({ type: `settle`, demand, scope, age, outcome }) + +const compoundSettlementHistories = ([`current`, `obsolete`] as const).flatMap( + (scope) => + ([`oldest`, `newest`] as const).flatMap((age) => + ([`resolve`, `reject`] as const).map((outcome) => [ + { type: `request`, demand: `a` } as const, + { type: `request`, demand: `a` } as const, + ...(scope === `obsolete` + ? ([ + { type: `release`, demand: `a` }, + { type: `release`, demand: `a` }, + ] as const) + : []), + settle(`a`, scope, age, outcome), + ]), + ), +) + +export const compoundLifecycleCoverageHistories: ReadonlyArray< + ReadonlyArray +> = [ + ...compoundSettlementHistories, + [ + { type: `request`, demand: `a` }, + settle(`a`, `current`, `oldest`, `resolve`), + { type: `truncate` }, + settle(`a`, `current`, `oldest`, `resolve`), + ], +] + +export const greenLifecycleHistories: ReadonlyArray< + ReadonlyArray +> = [ + [ + { type: `request`, demand: `a` }, + { type: `abort`, demand: `a` }, + settle(`a`, `current`, `oldest`, `reject`), + { type: `release`, demand: `a` }, + ], + [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `a` }, + settle(`a`, `current`, `newest`, `resolve`), + settle(`a`, `current`, `oldest`, `resolve`), + { type: `release`, demand: `a` }, + { type: `release`, demand: `a` }, + ], + [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `b` }, + settle(`a`, `current`, `oldest`, `resolve`), + { type: `cleanup` }, + { type: `restart` }, + settle(`b`, `obsolete`, `oldest`, `reject`), + settle(`a`, `current`, `oldest`, `resolve`), + settle(`b`, `current`, `oldest`, `reject`), + ], + [ + { type: `cleanup` }, + { type: `request`, demand: `a` }, + { type: `restart` }, + settle(`a`, `current`, `oldest`, `resolve`), + { type: `truncate` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ], + [ + { type: `request`, demand: `a` }, + settle(`a`, `current`, `oldest`, `reject`), + { type: `cleanup` }, + { type: `restart` }, + settle(`a`, `current`, `oldest`, `resolve`), + ], + [ + { type: `abort`, demand: `a` }, + { type: `release`, demand: `a` }, + settle(`a`, `current`, `oldest`, `resolve`), + { type: `truncate` }, + { type: `cleanup` }, + { type: `truncate` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `restart` }, + { type: `unsubscribe` }, + { type: `request`, demand: `b` }, + { type: `unsubscribe` }, + ], +] + +export const syncLifecycleHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `a` }, + { type: `release`, demand: `a` }, + { type: `request`, demand: `b` }, + { type: `truncate` }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, +] + +export const pendingSupersessionHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `a` }, + { type: `truncate` }, + { type: `truncate` }, + settle(`a`, `current`, `newest`, `resolve`), + settle(`a`, `current`, `oldest`, `reject`), + { type: `release`, demand: `a` }, +] + +export const abortReplayHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `abort`, demand: `a` }, + settle(`a`, `current`, `oldest`, `reject`), + { type: `truncate` }, + { type: `release`, demand: `a` }, +] + +export const abortedRestartHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `abort`, demand: `a` }, + { type: `restart` }, + { type: `release`, demand: `a` }, +] + +export const mixedAbortedRestartHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `b` }, + { type: `abort`, demand: `a` }, + settle(`a`, `current`, `oldest`, `reject`), + { type: `cleanup` }, + { type: `restart` }, + settle(`b`, `current`, `oldest`, `resolve`), + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, +] + +export const releasedObsoleteResolveHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `release`, demand: `a` }, + settle(`a`, `obsolete`, `oldest`, `resolve`), +] diff --git a/packages/db/tests/collection-subscription-lifecycle-history.property.test.ts b/packages/db/tests/collection-subscription-lifecycle-history.property.test.ts new file mode 100644 index 0000000000..55f316b094 --- /dev/null +++ b/packages/db/tests/collection-subscription-lifecycle-history.property.test.ts @@ -0,0 +1,798 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { + abortReplayHistory, + abortedRestartHistory, + compoundLifecycleCoverageHistories, + createLifecycleModel, + greenLifecycleHistories, + greenLifecycleHistoryArbitrary, + mixedAbortedRestartHistory, + pendingSupersessionHistory, + reduceLifecycle, + syncLifecycleHistory, + syncLifecycleHistoryArbitrary, +} from './collection-subscription-lifecycle-grammar.js' +import { flushPromises } from './utils.js' +import { + oraclePropertyOptions, + oracleRandomParameters, + readOracleRunConfig, +} from './oracle-config.js' +import type { LoadSubsetOptions, SyncConfig } from '../src/types.js' +import type { + DemandName, + LifecycleCommand, + LifecycleLoadEvent, + LifecycleResultEvent, + LifecycleTraceEvent, + LifecycleUnloadEvent, +} from './collection-subscription-lifecycle-grammar.js' + +type RuntimeAttempt = { + id: number + ownerId: number + demand: DemandName + options: LoadSubsetOptions + deferred?: ReturnType> + failure: Error + settled: boolean + current: boolean +} +type RuntimeOwner = { + id: number + demand: DemandName + controller: AbortController + aborted: boolean + attemptId?: number +} + +async function runHistory( + history: ReadonlyArray, + runOptions: { + acquisitionMode?: `async-pending` | `sync-success` + cancellation?: `manual` | `reject` + continueAfterMismatch?: boolean + } = {}, +): Promise> { + const acquisitionMode = runOptions.acquisitionMode ?? `async-pending` + const check = runOptions.continueAfterMismatch ? expect.soft : expect + const failures = new Map() + const failureForAttempt = (attemptId: number): Error => { + const existing = failures.get(attemptId) + if (existing) return existing + const failure = new Error(`attempt ${attemptId} failed`) + failures.set(attemptId, failure) + return failure + } + const cancellation = runOptions.cancellation ?? `manual` + const model = createLifecycleModel( + acquisitionMode, + failureForAttempt, + cancellation, + ) + const where = { + a: new Func(`eq`, [new PropRef([`id`]), new Value(`a`)]), + b: new Func(`eq`, [new PropRef([`id`]), new Value(`b`)]), + } + const demandForWhere = new Map([ + [where.a, `a`], + [where.b, `b`], + ]) + const runtimeAttempts = new Map() + const runtimeOwners: Array = [] + const attemptByOptions = new Map() + const observedLoads: Array = [] + const observedUnloads: Array< + LifecycleUnloadEvent | { attemptId: `unacquired`; handlerSession: number } + > = [] + const observedErrors: Array<{ + attemptId: number | `unacquired` + error: unknown + }> = [] + const observedResults: Array< + LifecycleResultEvent | { attemptId: `unacquired`; resultKind: string } + > = [] + const observedStatuses: Array = [] + const observedTrace: Array< + | LifecycleTraceEvent + | { type: `unload`; attemptId: `unacquired`; handlerSession: number } + | { type: `error`; attemptId: `unacquired` } + | { type: `result`; attemptId: `unacquired`; resultKind: string } + > = [] + let nextObservedAttemptId = 0 + let nextObservedOwnerId = 0 + let observedReplay = 0 + let observedSession = -1 + let observedActive = true + let observedUnsubscribed = false + let syncOps: + | Parameters[`sync`]>[0] + | undefined + + const collection = createCollection<{ id: string }, string>({ + id: `generated-async-demand-lifecycle`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + const handlerSession = ++observedSession + syncOps = operations + operations.markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`adapter load lost its demand`) + const owner = runtimeOwners.find( + (candidate) => + candidate.demand === demand && + !candidate.aborted && + candidate.attemptId === undefined, + ) + if (!owner) throw new Error(`adapter load has no runtime owner`) + const observed: LifecycleLoadEvent = { + id: nextObservedAttemptId++, + demand, + session: handlerSession, + replay: observedReplay, + } + const deferred = + acquisitionMode === `async-pending` + ? createDeferred() + : undefined + void deferred?.promise.catch(() => undefined) + runtimeAttempts.set(observed.id, { + id: observed.id, + ownerId: owner.id, + demand, + options, + deferred, + failure: failureForAttempt(observed.id), + settled: acquisitionMode === `sync-success`, + current: true, + }) + if (deferred && cancellation === `reject`) { + options.signal?.addEventListener( + `abort`, + () => { + const attempt = runtimeAttempts.get(observed.id)! + if (attempt.settled) return + attempt.settled = true + deferred.reject( + new DOMException(`acquisition aborted`, `AbortError`), + ) + }, + { once: true }, + ) + } + owner.attemptId = observed.id + attemptByOptions.set(options, observed.id) + observedLoads.push(observed) + observedTrace.push({ type: `load`, ...observed }) + return deferred?.promise ?? true + }, + unloadSubset: (options) => { + const unload = { + attemptId: attemptByOptions.get(options) ?? `unacquired`, + handlerSession, + } as const + observedUnloads.push(unload) + observedTrace.push({ type: `unload`, ...unload }) + }, + } + }, + }, + }) + const publications: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + publications.push(changes) + observedTrace.push({ type: `publication` }) + }, + { includeInitialState: false }, + ) + subscription.on(`status:change`, ({ status }) => { + observedStatuses.push(status) + observedTrace.push({ type: `status`, status }) + }) + subscription.on(`loadSubset:error`, ({ options, error }) => { + const attemptId = attemptByOptions.get(options) ?? `unacquired` + observedErrors.push({ attemptId, error }) + observedTrace.push({ type: `error`, attemptId }) + }) + + const assertState = (command: LifecycleCommand) => { + const context = JSON.stringify({ + history, + command, + observedTrace, + expectedTrace: model.trace, + }) + check(observedLoads, context).toEqual(model.loads) + check(observedUnloads, context).toEqual(model.unloads) + check( + observedErrors.map(({ attemptId }) => attemptId), + context, + ).toEqual(model.errors.map(({ attemptId }) => attemptId)) + for (const [index, { error }] of observedErrors.entries()) { + check(error, context).toBe(model.errors[index]?.error) + } + check(observedResults, context).toEqual(model.results) + check(observedStatuses, context).toEqual(model.statuses) + check(subscription.status, context).toBe(model.status) + check(subscription.lastError, context).toBe(model.lastError) + check(collection.status, context).toBe(model.collectionStatus) + check(publications, context).toEqual( + Array.from({ length: model.publications }, () => []), + ) + check(observedTrace, context).toEqual(model.trace) + for (const attempt of model.attempts) { + check( + runtimeAttempts.get(attempt.id)?.options.signal?.aborted, + context, + ).toBe(attempt.aborted) + } + } + + const selectRuntimeAttempt = ( + command: Extract, + ): RuntimeAttempt | undefined => { + if (observedUnsubscribed) return undefined + const candidates = [...runtimeAttempts.values()].filter( + (attempt) => + !attempt.settled && + attempt.demand === command.demand && + attempt.current === (command.scope === `current`), + ) + return command.age === `oldest` ? candidates[0] : candidates.at(-1) + } + + try { + for (const command of history) { + const runtimeOwner = + command.type === `request` + ? { + id: nextObservedOwnerId++, + demand: command.demand, + controller: new AbortController(), + aborted: false, + } + : command.type === `abort` + ? runtimeOwners.find( + ({ demand, aborted }) => demand === command.demand && !aborted, + ) + : command.type === `release` + ? runtimeOwners.find(({ demand }) => demand === command.demand) + : undefined + if (command.type === `request` && !model.unsubscribed) { + runtimeOwners.push(runtimeOwner!) + } + const runtimeAttempt = + command.type === `settle` ? selectRuntimeAttempt(command) : undefined + const effect = reduceLifecycle(model, command) + if (command.type === `request`) { + check(effect.ownerId).toBe( + model.unsubscribed ? undefined : runtimeOwner?.id, + ) + const result = subscription.requestSnapshot({ + where: where[command.demand], + signal: runtimeOwner?.controller.signal, + onLoadSubsetResult: (loadResult, requestOptions) => { + const attemptId = + attemptByOptions.get(requestOptions) ?? `unacquired` + const resultKind = loadResult === true ? `true` : `promise` + observedResults.push({ attemptId, resultKind }) + observedTrace.push({ type: `result`, attemptId, resultKind }) + }, + }) + check(result).toBe(effect.requestResult) + } else if (command.type === `abort`) { + check(effect.ownerId).toBe(runtimeOwner?.id) + if (runtimeOwner) { + runtimeOwner.aborted = true + runtimeOwner.controller.abort() + } + } else if (command.type === `release`) { + check(effect.ownerId).toBe(runtimeOwner?.id) + if (runtimeOwner) { + if (runtimeOwner.attemptId !== undefined) { + runtimeAttempts.get(runtimeOwner.attemptId)!.current = false + } + runtimeOwners.splice(runtimeOwners.indexOf(runtimeOwner), 1) + } + subscription.releaseSnapshot(where[command.demand]) + } else if (command.type === `settle`) { + check(effect.attemptId).toBe(runtimeAttempt?.id) + if (effect.attemptId === undefined) { + // Neither model found an effective settlement. + } else if (!runtimeAttempt?.deferred) { + throw new Error(`model selected an already settled acquisition`) + } else { + runtimeAttempt.settled = true + if (command.outcome === `resolve`) runtimeAttempt.deferred.resolve() + else runtimeAttempt.deferred.reject(runtimeAttempt.failure) + } + } else if (command.type === `truncate`) { + if (observedActive) { + observedReplay++ + for (const owner of runtimeOwners) { + if (owner.attemptId !== undefined) { + runtimeAttempts.get(owner.attemptId)!.current = false + } + owner.attemptId = undefined + } + } + syncOps?.begin() + syncOps?.truncate() + const receipt = syncOps?.commit() + if (observedActive && !observedUnsubscribed && model.owners.length) { + check(subscription.status).toBe(`loadingSubset`) + } + if (receipt !== true) await receipt + } else if (command.type === `cleanup`) { + if (observedActive) { + for (const owner of runtimeOwners) { + if (owner.attemptId !== undefined) { + runtimeAttempts.get(owner.attemptId)!.current = false + } + owner.attemptId = undefined + } + } + await collection.cleanup() + observedActive = false + } else if (command.type === `restart`) { + const queuesReplay = + !observedActive && !observedUnsubscribed && model.owners.length > 0 + if (!observedActive) { + observedReplay = 0 + observedActive = true + } + collection.startSyncImmediate() + if (queuesReplay) check(subscription.status).toBe(`loadingSubset`) + } else { + for (const attempt of runtimeAttempts.values()) attempt.current = false + subscription.unsubscribe() + observedUnsubscribed = true + runtimeOwners.length = 0 + } + await flushPromises() + assertState(command) + } + } finally { + for (const { deferred } of runtimeAttempts.values()) deferred?.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + return model.reach +} + +if (process.env.TANSTACK_DB_ORACLE_STATISTICS === `1`) { + fc.statistics( + greenLifecycleHistoryArbitrary, + (history) => { + const model = createLifecycleModel() + for (const command of history) reduceLifecycle(model, command) + return [...model.reach] + }, + oraclePropertyOptions(1_000, `subscription-lifecycle.history-statistics`), + ) +} + +describe(`CollectionSubscription async lifecycle history oracle`, () => { + it(`covers every required command and cross-phase transition`, async () => { + const reach = new Set() + for (const history of [ + ...greenLifecycleHistories, + ...compoundLifecycleCoverageHistories, + ]) { + for (const label of await runHistory(history)) reach.add(label) + } + const commands = [ + `request`, + `abort`, + `release`, + `settle`, + `truncate`, + `cleanup`, + `restart`, + `unsubscribe`, + ] + const required = new Set([ + ...commands.map((type) => `command:${type}`), + ...commands.map((type) => `effective:${type}`), + ...commands.map((type) => `noop:${type}`), + `settle-scope:current`, + `settle-scope:obsolete`, + `settle-age:oldest`, + `settle-age:newest`, + `settle-outcome:resolve`, + `settle-outcome:reject`, + ...([`current`, `obsolete`] as const).flatMap((scope) => + ([`oldest`, `newest`] as const).flatMap((age) => + ([`resolve`, `reject`] as const).map( + (outcome) => `settle:${scope}:${age}:${outcome}`, + ), + ), + ), + `attempt-session:initial`, + `attempt-session:restarted`, + `attempt-replay:initial`, + `attempt-replay:replayed`, + `attempt-location:initial:initial`, + `attempt-location:initial:replayed`, + `attempt-location:restarted:initial`, + `attempt-location:restarted:replayed`, + `duplicate-owner`, + `request-while-cleaned`, + `partial-generation-supersession`, + ]) + expect([...required].filter((label) => !reach.has(label))).toEqual([]) + }) + + it(`names the overlapping replay transition in the model`, () => { + const model = createLifecycleModel() + for (const command of pendingSupersessionHistory) { + reduceLifecycle(model, command) + } + expect(model.reach).toContain(`overlapping-replay`) + }) + + it.each([ + { + name: `one demand after one replay`, + history: [ + { type: `request`, demand: `a` }, + { type: `truncate` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + ] satisfies Array, + }, + { + name: `duplicate owners after overlapping replay`, + history: pendingSupersessionHistory, + }, + ])( + `waits for delayed cancellation settlement for $name`, + async ({ history }) => { + await runHistory(history) + }, + ) + + it(`releases exact ownership while older replay work is pending`, async () => { + await runHistory( + [ + ...pendingSupersessionHistory, + { type: `cleanup` }, + { type: `restart` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it.each( + ([`manual`, `reject`] as const).flatMap((cancellation) => + ([1, 2] as const).flatMap((replays) => + ([`resolve`, `reject`] as const).map((outcome) => ({ + cancellation, + replays, + outcome, + })), + ), + ), + )( + `tracks $cancellation cancellation across $replays replay(s) ending in $outcome`, + async ({ cancellation, replays, outcome }) => { + await runHistory( + [ + { type: `request`, demand: `a` }, + ...Array.from( + { length: replays }, + (): LifecycleCommand => ({ type: `truncate` }), + ), + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome, + }, + ...Array.from( + { length: replays }, + (_, index): LifecycleCommand => ({ + type: `settle`, + demand: `a`, + scope: `obsolete`, + age: `oldest`, + outcome: index % 2 === 0 ? `reject` : `resolve`, + }), + ), + { type: `release`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { cancellation }, + ) + }, + ) + + it.each([ + { name: `truncate replay`, history: abortReplayHistory }, + { name: `cleanup restart`, history: abortedRestartHistory }, + ])( + `queues replay without reacquiring an aborted demand on $name`, + async ({ history }) => { + await runHistory(history) + }, + ) + + it(`does not release an unacquired replacement after an aborted demand replays`, async () => { + await runHistory( + [ + ...abortReplayHistory, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`replays a live peer without reacquiring an aborted demand`, async () => { + await runHistory([ + { type: `request`, demand: `a` }, + { type: `request`, demand: `b` }, + { type: `abort`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `truncate` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + ]) + }) + + it(`restarts a live peer without reacquiring an aborted demand and completes teardown`, async () => { + await runHistory(mixedAbortedRestartHistory, { + continueAfterMismatch: true, + }) + }) + + const syncReplayScenarios = ([`truncate`, `restart`] as const).flatMap( + (transition) => + ([1, 2] as const).flatMap((ownerCount) => + ([false, true] as const).map((abortFirst) => { + const history: Array = [ + { type: `request`, demand: `a` }, + ...(ownerCount === 2 + ? ([{ type: `request`, demand: `b` }] as const) + : []), + ...(abortFirst ? ([{ type: `abort`, demand: `a` }] as const) : []), + ...(transition === `truncate` + ? ([{ type: `truncate` }] as const) + : ([{ type: `cleanup` }, { type: `restart` }] as const)), + ] + return { transition, ownerCount, abortFirst, history } + }), + ), + ) + + it.each(syncReplayScenarios)( + `settles queued synchronous $transition with $ownerCount owner(s), abort=$abortFirst`, + async ({ history }) => { + await runHistory(history, { acquisitionMode: `sync-success` }) + }, + ) + + it(`preserves physical ownership across queued synchronous replay`, async () => { + await runHistory(syncLifecycleHistory, { + acquisitionMode: `sync-success`, + continueAfterMismatch: true, + }) + }) + + it.each([ + { + name: `same-key owners across truncate`, + history: [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `a` }, + { type: `truncate` }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + }, + { + name: `same-key owners across restart`, + history: [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ], + }, + { + name: `aborted owner across repeated truncate`, + history: [ + { type: `request`, demand: `a` }, + { type: `abort`, demand: `a` }, + { type: `truncate` }, + { type: `truncate` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ], + }, + { + name: `detached last-owner abort across restart`, + history: [ + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `abort`, demand: `a` }, + { type: `restart` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ], + }, + ] satisfies ReadonlyArray<{ + name: string + history: ReadonlyArray + }>)( + `continues through the full synchronous replay suffix for $name`, + async ({ history }) => { + await runHistory(history, { + acquisitionMode: `sync-success`, + continueAfterMismatch: true, + }) + }, + ) + + it(`publishes a new snapshot while canceled initial work still holds readiness`, async () => { + // Seed 1413322355, path 757:13:15:15:9:9:9. This checks an empty snapshot + // notification: initial readiness is not a replacement publication gate. + await runHistory( + [ + { type: `request`, demand: `b` }, + { type: `truncate` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { + type: `settle`, + demand: `b`, + scope: `obsolete`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`keeps a new snapshot private after failed restart`, async () => { + // Minimized from seed 317005625 at 100×. The mismatch is an empty + // notification, not lost rows: failed replacement must keep reads private. + await runHistory( + [ + { type: `request`, demand: `b` }, + { type: `cleanup` }, + { type: `restart` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + const { multiplier, ...replay } = readOracleRunConfig() + const runs = 80 * multiplier + const cancellationArbitrary = fc.constantFrom( + `manual` as const, + `reject` as const, + ) + + fcTest.prop([greenLifecycleHistoryArbitrary, cancellationArbitrary], { + numRuns: runs, + seed: 1_657_003, + })( + `matches the pure lifecycle model for a fixed seed`, + async (history, cancellation) => { + await runHistory(history, { cancellation }) + }, + 120_000, + ) + fcTest.prop( + [greenLifecycleHistoryArbitrary, cancellationArbitrary], + oracleRandomParameters( + runs, + replay, + `subscription-lifecycle.async-history`, + ), + )( + `matches the pure lifecycle model for a random or replayed seed`, + async (history, cancellation) => { + await runHistory(history, { cancellation }) + }, + 120_000, + ) + fcTest.prop([syncLifecycleHistoryArbitrary], { + numRuns: runs, + seed: 1_657_004, + })( + `matches synchronous success histories for a fixed seed`, + async (history) => { + await runHistory(history, { acquisitionMode: `sync-success` }) + }, + 120_000, + ) + fcTest.prop( + [syncLifecycleHistoryArbitrary], + oracleRandomParameters(runs, replay, `subscription-lifecycle.sync-history`), + )( + `matches synchronous success histories for a random or replayed seed`, + async (history) => { + await runHistory(history, { acquisitionMode: `sync-success` }) + }, + 120_000, + ) +}) diff --git a/packages/db/tests/collection-subscription-lifecycle-oracle.test.ts b/packages/db/tests/collection-subscription-lifecycle-oracle.test.ts new file mode 100644 index 0000000000..31eb18be51 --- /dev/null +++ b/packages/db/tests/collection-subscription-lifecycle-oracle.test.ts @@ -0,0 +1,4111 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { afterAll, describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { createOnDemandCollection, flushPromises } from './utils.js' +import { + oraclePropertyOptions, + oracleRandomParameters, + readOracleRunConfig, +} from './oracle-config.js' +import type { CollectionSubscription } from '../src/collection/subscription.js' +import type { LoadSubsetOptions, SyncConfig } from '../src/types.js' + +type StartOutcome = `return` | `throw` | `resolve` | `reject` +type StartReentry = + | `none` + | `abort-self` + | `truncate` + | `release-self` + | `release-peer` + | `unsubscribe` + | `cleanup` +type RestartReentry = + | `none` + | `release-self` + | `release-peer` + | `unsubscribe` + | `cleanup` + +const acquisitionPhases = [ + `deferred`, + `starting`, + `on-demand`, + `eager`, + `retiring`, + `unavailable`, +] as const +const acquisitionEntries = [ + `request`, + `resume`, + `markReady`, + `markError`, + `syncReturn`, +] as const +type AcquisitionPhase = (typeof acquisitionPhases)[number] +type AcquisitionEntry = (typeof acquisitionEntries)[number] +type AcquisitionCell = `${AcquisitionPhase}:${AcquisitionEntry}` + +type AcquisitionCellDefinition = + | { kind: `covered` } + | { kind: `excluded`; reason: string } + +const acquisitionCellDefinitions = { + 'deferred:request': { kind: `covered` }, + 'deferred:resume': { kind: `covered` }, + 'deferred:markReady': { + kind: `excluded`, + reason: `the sync callback has not started and cannot mark ready`, + }, + 'deferred:markError': { + kind: `excluded`, + reason: `the sync callback has not started and cannot mark error`, + }, + 'deferred:syncReturn': { + kind: `excluded`, + reason: `the deferred sync callback has no result to return`, + }, + 'starting:request': { kind: `covered` }, + 'starting:resume': { + kind: `excluded`, + reason: `resuming the deferred gate enters this phase only once`, + }, + 'starting:markReady': { kind: `covered` }, + 'starting:markError': { kind: `covered` }, + 'starting:syncReturn': { kind: `covered` }, + 'on-demand:request': { kind: `covered` }, + 'on-demand:resume': { + kind: `excluded`, + reason: `an installed loader is no longer behind the deferred gate`, + }, + 'on-demand:markReady': { kind: `covered` }, + 'on-demand:markError': { kind: `covered` }, + 'on-demand:syncReturn': { + kind: `excluded`, + reason: `the sync callback already returned the installed loader`, + }, + 'eager:request': { kind: `covered` }, + 'eager:resume': { + kind: `excluded`, + reason: `eager sync is not a deferred subset acquisition`, + }, + 'eager:markReady': { + kind: `excluded`, + reason: `eager readiness does not install a subset loader`, + }, + 'eager:markError': { + kind: `excluded`, + reason: `eager errors do not change subset acquisition availability`, + }, + 'eager:syncReturn': { + kind: `excluded`, + reason: `eager sync results own no subset loader contract`, + }, + 'retiring:request': { kind: `covered` }, + 'retiring:resume': { + kind: `excluded`, + reason: `retirement is outside the deferred-start gate`, + }, + 'retiring:markReady': { + kind: `excluded`, + reason: `callbacks from a retiring session cannot restore availability`, + }, + 'retiring:markError': { + kind: `excluded`, + reason: `callbacks from a retiring session are obsolete`, + }, + 'retiring:syncReturn': { + kind: `excluded`, + reason: `obsolete returned resources use the resource-installation axis`, + }, + 'unavailable:request': { kind: `covered` }, + 'unavailable:resume': { + kind: `excluded`, + reason: `same-session recovery uses markReady rather than defer resume`, + }, + 'unavailable:markReady': { kind: `covered` }, + 'unavailable:markError': { + kind: `excluded`, + reason: `a repeated error leaves acquisition unavailable`, + }, + 'unavailable:syncReturn': { + kind: `excluded`, + reason: `handler-less return is the transition into unavailable`, + }, +} satisfies Record + +const legalAcquisitionCells = new Set( + Object.entries(acquisitionCellDefinitions) + .filter(([, definition]) => definition.kind === `covered`) + .map(([cell]) => cell as AcquisitionCell), +) +const excludedAcquisitionCells = new Map( + Object.entries(acquisitionCellDefinitions).flatMap(([cell, definition]) => + definition.kind === `excluded` + ? [[cell as AcquisitionCell, definition.reason]] + : [], + ), +) +const observedAcquisitionCells = new Set() + +function acquisitionCase( + cells: ReadonlyArray, + name: string, + run: (reach: (cell: AcquisitionCell) => void) => void | Promise, +): void { + const declaredCells = new Set(cells) + it(name, () => + run((cell) => { + if (!declaredCells.has(cell)) { + throw new Error(`${name} reached undeclared acquisition cell ${cell}`) + } + observedAcquisitionCells.add(cell) + }), + ) +} + +const physicalAcquisitionStates = [ + `none`, + `starting`, + `active`, + `obsolete`, + `failed-release`, +] as const +const physicalInteractionCauses = [ + `release`, + `abort`, + `truncate`, + `cleanup`, + `unsubscribe`, +] as const +type PhysicalAcquisitionState = (typeof physicalAcquisitionStates)[number] +type PhysicalInteractionCause = (typeof physicalInteractionCauses)[number] +type PhysicalInteractionCell = + `${PhysicalAcquisitionState}:${PhysicalInteractionCause}` +type PhysicalInteraction = + | `no-acquisition` + | `abort-only` + | `retire` + | `no-repeat-on-truncate` + | `no-repeat-on-cleanup` + | `no-repeat-on-unsubscribe` +type PhysicalInteractionCellDefinition = + | { kind: `covered`; interaction: PhysicalInteraction } + | { kind: `excluded`; reason: string } + +const physicalInteractionCellDefinitions = { + 'none:release': { + kind: `covered`, + interaction: `no-acquisition`, + }, + 'none:abort': { + kind: `covered`, + interaction: `no-acquisition`, + }, + 'none:truncate': { + kind: `covered`, + interaction: `no-acquisition`, + }, + 'none:cleanup': { + kind: `covered`, + interaction: `no-acquisition`, + }, + 'none:unsubscribe': { + kind: `covered`, + interaction: `no-acquisition`, + }, + 'starting:release': { kind: `covered`, interaction: `retire` }, + 'starting:abort': { kind: `covered`, interaction: `abort-only` }, + 'starting:truncate': { kind: `covered`, interaction: `retire` }, + 'starting:cleanup': { kind: `covered`, interaction: `retire` }, + 'starting:unsubscribe': { kind: `covered`, interaction: `retire` }, + 'active:release': { kind: `covered`, interaction: `retire` }, + 'active:abort': { + kind: `covered`, + interaction: `abort-only`, + }, + 'active:truncate': { kind: `covered`, interaction: `retire` }, + 'active:cleanup': { kind: `covered`, interaction: `retire` }, + 'active:unsubscribe': { kind: `covered`, interaction: `retire` }, + 'obsolete:release': { + kind: `excluded`, + reason: `the replacement owns later release; obsolete work was retired once`, + }, + 'obsolete:abort': { + kind: `excluded`, + reason: `obsolete work was already signaled and retired`, + }, + 'obsolete:truncate': { + kind: `excluded`, + reason: `another truncate retires the current replacement, not already-obsolete work`, + }, + 'obsolete:cleanup': { + kind: `excluded`, + reason: `source cleanup retires the current session; obsolete work was retired once`, + }, + 'obsolete:unsubscribe': { + kind: `excluded`, + reason: `unsubscribe retires current ownership; obsolete work was retired once`, + }, + 'failed-release:release': { + kind: `excluded`, + reason: `logical release already happened; the physical attempt is final`, + }, + 'failed-release:abort': { + kind: `excluded`, + reason: `the failed physical release is already aborted`, + }, + 'failed-release:truncate': { + kind: `covered`, + interaction: `no-repeat-on-truncate`, + }, + 'failed-release:cleanup': { + kind: `covered`, + interaction: `no-repeat-on-cleanup`, + }, + 'failed-release:unsubscribe': { + kind: `covered`, + interaction: `no-repeat-on-unsubscribe`, + }, +} satisfies Record + +const requiredPhysicalInteractions = new Map< + PhysicalInteractionCell, + PhysicalInteraction +>( + Object.entries(physicalInteractionCellDefinitions).flatMap( + ([cell, definition]) => + definition.kind === `covered` + ? [[cell as PhysicalInteractionCell, definition.interaction]] + : [], + ), +) +const observedPhysicalInteractions = new Map< + PhysicalInteractionCell, + PhysicalInteraction +>() + +function observePhysicalInteraction( + cell: PhysicalInteractionCell, + interaction: PhysicalInteraction, +): void { + observedPhysicalInteractions.set(cell, interaction) +} + +const requiredSourceSessionBoundaries = new Set([ + `active-cleanup`, + `restart-installed`, + `cleanup-callback-reentry`, + `obsolete-resource-return`, +] as const) +type SourceSessionBoundary = + typeof requiredSourceSessionBoundaries extends Set ? T : never +const observedSourceSessionBoundaries = new Set() + +function observeSourceSessionBoundary(boundary: SourceSessionBoundary): void { + observedSourceSessionBoundaries.add(boundary) +} + +const startOutcomes = [`return`, `throw`, `resolve`, `reject`] as const +const startReentries = [ + `none`, + `abort-self`, + `truncate`, + `release-self`, + `release-peer`, + `unsubscribe`, + `cleanup`, +] as const + +type StartScenario = { + outcome: StartOutcome + reentry: StartReentry +} + +const startScenarios: ReadonlyArray = startOutcomes.flatMap( + (outcome) => startReentries.map((reentry) => ({ outcome, reentry })), +) + +const failureScenarios = ([`throw`, `reject`] as const).flatMap((outcome) => + startReentries.map((reentry) => ({ outcome, reentry })), +) +type FailureDeliverySuffix = `${`throw` | `reject`}:${StartReentry}` +const requiredFailureDeliverySuffixes = new Set( + failureScenarios.map( + ({ outcome, reentry }) => `${outcome}:${reentry}` as const, + ), +) +const observedFailureDeliverySuffixes = new Set() + +const releaseScenarios = ([`return`, `throw`] as const).flatMap((outcome) => + ([`none`, `reacquire-self`, `release-peer`, `unsubscribe`] as const).map( + (reentry) => ({ outcome, reentry }), + ), +) + +const restartScenarios = startOutcomes.flatMap((outcome) => + ( + [`none`, `release-self`, `release-peer`, `unsubscribe`, `cleanup`] as const + ).map((reentry: RestartReentry) => ({ outcome, reentry })), +) + +const threeGenerationScenarios = ([`resolve`, `reject`] as const).flatMap( + (obsoleteOutcome) => + ([`resolve`, `reject`] as const).flatMap((currentOutcome) => + ([`obsolete-first`, `current-first`] as const).map((settlementOrder) => ({ + obsoleteOutcome, + currentOutcome, + settlementOrder, + })), + ), +) + +type AsyncRestartScenario = { + demands: ReadonlyArray<`a` | `b`> + generationOutcomes: ReadonlyArray> + settlementOrder: `obsolete-first` | `current-first` | `interleaved` +} + +const asyncRestartCoverageScenarios = [ + { + demands: [`a`], + generationOutcomes: [[`reject`]], + settlementOrder: `current-first`, + }, + { + demands: [`a`], + generationOutcomes: [[`resolve`], [`resolve`]], + settlementOrder: `obsolete-first`, + }, + { + demands: [`a`, `b`], + generationOutcomes: [ + [`reject`, `resolve`], + [`resolve`, `reject`], + ], + settlementOrder: `current-first`, + }, + { + demands: [`a`, `b`], + generationOutcomes: [ + [`resolve`, `resolve`], + [`reject`, `reject`], + [`resolve`, `resolve`], + ], + settlementOrder: `interleaved`, + }, +] as const satisfies ReadonlyArray + +const asyncRestartScenarioArbitrary: fc.Arbitrary = fc + .uniqueArray(fc.constantFrom(`a` as const, `b` as const), { + minLength: 1, + maxLength: 2, + }) + .chain((demands) => + fc.record({ + demands: fc.constant(demands), + generationOutcomes: fc.array( + fc.array(fc.constantFrom(`resolve` as const, `reject` as const), { + minLength: demands.length, + maxLength: demands.length, + }), + { minLength: 1, maxLength: 3 }, + ), + settlementOrder: fc.constantFrom( + `obsolete-first` as const, + `current-first` as const, + `interleaved` as const, + ), + }), + ) + +if (process.env.TANSTACK_DB_ORACLE_STATISTICS === `1`) { + fc.statistics( + asyncRestartScenarioArbitrary, + ({ demands, generationOutcomes, settlementOrder }) => { + const realizesInterleaving = + settlementOrder === `interleaved` && + demands.length > 1 && + generationOutcomes.length > 1 + return [ + `demands=${demands.length}`, + `generations=${generationOutcomes.length + 1}`, + `current=${generationOutcomes.at(-1)?.join(`+`)}`, + `mixed-current=${new Set(generationOutcomes.at(-1)).size > 1}`, + `obsolete-reject=${generationOutcomes + .slice(0, -1) + .some((outcomes) => outcomes.includes(`reject`))}`, + `requested-order=${ + settlementOrder === `interleaved` && !realizesInterleaving + ? `degenerate-interleaved` + : settlementOrder + }`, + ] + }, + oraclePropertyOptions(1_000, `subscription-lifecycle.async-statistics`), + ) +} + +async function runAsyncRestartScenario( + scenario: AsyncRestartScenario, +): Promise> { + type DemandName = `a` | `b` + type Row = { id: DemandName; version: number } + type Attempt = { + session: number + demand: DemandName + options: LoadSubsetOptions + deferred: ReturnType> + } + type SettlementEvent = { + attempt: Attempt + session: number + demand: DemandName + outcome: `resolve` | `reject` + activeSession: number + } + const where = { + a: new Func(`eq`, [new PropRef([`id`]), new Value(`a`)]), + b: new Func(`eq`, [new PropRef([`id`]), new Value(`b`)]), + } + const demandForWhere = new Map([ + [where.a, `a`], + [where.b, `b`], + ]) + const attempts: Array = [] + const errors: Array<{ demand: DemandName; error: unknown }> = [] + const publications: Array> = [] + const statuses: Array = [] + const visible = new Map() + const unloads: Array<{ session: number; demand: DemandName }> = [] + const settlements: Array = [] + const settledAttempts = new Set() + const publishedBeforeRetirement = new Set() + const failures = scenario.generationOutcomes.map((_, generation) => + scenario.demands.map( + (demand) => new Error(`session ${generation + 1} ${demand} failed`), + ), + ) + let session = -1 + + const outcomeFor = (attempt: Attempt) => + scenario.generationOutcomes[attempt.session - 1]![ + scenario.demands.indexOf(attempt.demand) + ]! + const failureFor = (attempt: Attempt) => + failures[attempt.session - 1]![scenario.demands.indexOf(attempt.demand)]! + + const settleAttempt = async (attempt: Attempt): Promise => { + const outcome = outcomeFor(attempt) + if (outcome === `resolve`) attempt.deferred.resolve() + else attempt.deferred.reject(failureFor(attempt)) + await flushPromises() + settlements.push({ + attempt, + session: attempt.session, + demand: attempt.demand, + outcome, + activeSession: session, + }) + settledAttempts.add(attempt) + } + + const collection = createOnDemandCollection({ + id: `async-restart-lifecycle`, + sync: { + sync: (operations) => { + session++ + const ownSession = session + operations.markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown async demand`) + const deferred = createDeferred() + void deferred.promise.catch(() => {}) + attempts.push({ + session: ownSession, + demand, + options, + deferred, + }) + return deferred.promise.then(() => { + operations.begin() + operations.write({ + type: `insert`, + value: { id: demand, version: ownSession + 1 }, + }) + const receipt = operations.commit() + if (receipt !== true) return receipt + return undefined + }) + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown async demand`) + unloads.push({ session: ownSession, demand }) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + publications.push( + [...visible.values()].sort((a, b) => a.id.localeCompare(b.id)), + ) + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ options, error }) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown errored demand`) + errors.push({ demand, error }) + }) + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + + try { + for (const demand of scenario.demands) { + subscription.requestSnapshot({ where: where[demand] }) + } + for (const attempt of attempts.filter( + ({ session: value }) => value === 0, + )) { + attempt.deferred.resolve() + } + await flushPromises() + expect( + [...visible.values()].sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual( + [...scenario.demands] + .sort((a, b) => a.localeCompare(b)) + .map((id) => ({ id, version: 1 })), + ) + + for ( + let generation = 0; + generation < scenario.generationOutcomes.length; + generation++ + ) { + const discardedSession = session + await collection.cleanup() + for (const attempt of attempts.filter( + ({ session: attemptSession }) => attemptSession === discardedSession, + )) { + expect(attempt.options.signal?.aborted).toBe(true) + } + collection.startSyncImmediate() + await flushPromises() + const expectedSession = generation + 1 + expect( + attempts + .filter( + ({ session: attemptSession }) => attemptSession <= expectedSession, + ) + .map(({ session: attemptSession, demand }) => ({ + session: attemptSession, + demand, + })), + ).toEqual( + Array.from({ length: expectedSession + 1 }, (_, attemptSession) => + scenario.demands.map((demand) => ({ + session: attemptSession, + demand, + })), + ).flat(), + ) + + const publishesBeforeLaterRestart = + scenario.settlementOrder === `interleaved` && + generation === 0 && + scenario.generationOutcomes.length > 1 && + scenario.generationOutcomes[generation]!.every( + (outcome) => outcome === `resolve`, + ) + if (publishesBeforeLaterRestart) { + const publicationCount = publications.length + for (const attempt of attempts.filter( + ({ session: attemptSession }) => attemptSession === expectedSession, + )) { + await settleAttempt(attempt) + } + const expectedRows = [...scenario.demands] + .sort((a, b) => a.localeCompare(b)) + .map((id) => ({ id, version: expectedSession + 1 })) + expect( + [...visible.values()].sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual(expectedRows) + expect(publications.slice(publicationCount)).toEqual([expectedRows]) + publishedBeforeRetirement.add(expectedSession) + } + } + + const currentSession = scenario.generationOutcomes.length + expect( + attempts.map(({ session: attemptSession, demand }) => ({ + session: attemptSession, + demand, + })), + ).toEqual( + Array.from({ length: currentSession + 1 }, (_, attemptSession) => + scenario.demands.map((demand) => ({ + session: attemptSession, + demand, + })), + ).flat(), + ) + const obsolete = attempts.filter( + (attempt) => + attempt.session > 0 && + attempt.session < currentSession && + !settledAttempts.has(attempt), + ) + const current = attempts.filter( + ({ session: value }) => value === currentSession, + ) + const orderedAttempts = + scenario.settlementOrder === `obsolete-first` + ? [...obsolete, ...current] + : scenario.settlementOrder === `current-first` + ? [...current, ...obsolete] + : attempts + .filter(({ session: value }) => value > 0) + .filter((attempt) => !settledAttempts.has(attempt)) + .sort((left, right) => + left.demand === right.demand + ? right.session - left.session + : left.demand.localeCompare(right.demand), + ) + + const settledCurrent: Array = [] + const publicationTraceStart = publications.length + const statusTraceStart = statuses.length + const retainedVersion = publishedBeforeRetirement.size + ? Math.max(...publishedBeforeRetirement) + 1 + : 1 + const assertObservableState = () => { + const currentComplete = settledCurrent.length === current.length + const currentSucceeded = current.every( + (attempt) => outcomeFor(attempt) === `resolve`, + ) + const visibleVersion = + currentComplete && currentSucceeded + ? currentSession + 1 + : retainedVersion + const expectedRows = [...scenario.demands] + .sort((a, b) => a.localeCompare(b)) + .map((id) => ({ id, version: visibleVersion })) + expect( + [...visible.values()].sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual(expectedRows) + const expectedFailedAttempts = settledCurrent.filter( + (attempt) => outcomeFor(attempt) === `reject`, + ) + expect(errors.map(({ demand }) => demand)).toEqual( + expectedFailedAttempts.map(({ demand }) => demand), + ) + for (const [index, { error }] of errors.entries()) { + expect(error).toBe(failureFor(expectedFailedAttempts[index]!)) + } + expect(subscription.lastError).toBe( + expectedFailedAttempts.length + ? failureFor(expectedFailedAttempts.at(-1)!) + : undefined, + ) + expect(subscription.status).toBe( + currentComplete ? `ready` : `loadingSubset`, + ) + expect(publications.slice(publicationTraceStart)).toEqual( + currentComplete && currentSucceeded ? [expectedRows] : [], + ) + expect(statuses.slice(statusTraceStart)).toEqual( + currentComplete ? [`ready`] : [], + ) + } + + for (const attempt of orderedAttempts) { + await settleAttempt(attempt) + if (attempt.session === currentSession) settledCurrent.push(attempt) + assertObservableState() + } + + const currentSucceeded = current.every( + (attempt) => outcomeFor(attempt) === `resolve`, + ) + const expectedVersion = currentSucceeded + ? currentSession + 1 + : retainedVersion + expect( + [...visible.values()].sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual( + [...scenario.demands] + .sort((a, b) => a.localeCompare(b)) + .map((id) => ({ id, version: expectedVersion })), + ) + if (currentSucceeded) { + expect(errors).toEqual([]) + expect(subscription.lastError).toBeUndefined() + } else { + const expectedFailedAttempts = settledCurrent.filter( + (attempt) => outcomeFor(attempt) === `reject`, + ) + expect(errors.map(({ demand }) => demand)).toEqual( + expectedFailedAttempts.map(({ demand }) => demand), + ) + for (const [index, { error }] of errors.entries()) { + expect(error).toBe(failureFor(expectedFailedAttempts[index]!)) + } + expect(subscription.lastError).toBe( + failureFor(expectedFailedAttempts.at(-1)!), + ) + } + expect(subscription.status).toBe(`ready`) + for (const attempt of current) { + expect(attempt.options.signal?.aborted).toBe(false) + } + + const finalScopes = settlements.map(({ session: attemptSession }) => + attemptSession === currentSession ? `current` : `obsolete`, + ) + const firstCurrent = finalScopes.indexOf(`current`) + const lastCurrent = finalScopes.lastIndexOf(`current`) + const firstObsolete = finalScopes.indexOf(`obsolete`) + const lastObsolete = finalScopes.lastIndexOf(`obsolete`) + const observedOrder = + firstCurrent === -1 || firstObsolete === -1 + ? undefined + : lastObsolete < firstCurrent + ? `obsolete-first` + : lastCurrent < firstObsolete + ? `current-first` + : `interleaved` + const currentOutcomes = settlements + .filter( + ({ session: attemptSession }) => attemptSession === currentSession, + ) + .map(({ outcome }) => outcome) + expect(new Set(settlements.map(({ attempt }) => attempt)).size).toBe( + settlements.length, + ) + expect(settlements).toHaveLength( + attempts.filter(({ session: attemptSession }) => attemptSession > 0) + .length, + ) + const reach = new Set([ + `demands:${new Set(attempts.map(({ demand }) => demand)).size}`, + `sessions:${new Set(attempts.map(({ session: attemptSession }) => attemptSession)).size}`, + ...[...new Set(currentOutcomes)].map((outcome) => `current:${outcome}`), + `mixed-current:${new Set(currentOutcomes).size > 1}`, + `obsolete-reject:${settlements.some( + ({ session: attemptSession, outcome }) => + attemptSession < currentSession && outcome === `reject`, + )}`, + ...(observedOrder ? [`order:${observedOrder}`] : []), + `real-interleaving:${settlements.some( + ({ session: attemptSession, activeSession }) => + attemptSession < currentSession && + activeSession === attemptSession && + publishedBeforeRetirement.has(attemptSession), + )}`, + ]) + + subscription.unsubscribe() + for (const attempt of current) { + expect(attempt.options.signal?.aborted).toBe(true) + } + expect(unloads).toEqual( + scenario.demands.map((demand) => ({ + session: currentSession, + demand, + })), + ) + return reach + } finally { + subscription.unsubscribe() + await collection.cleanup() + } +} + +/** + * Exhaust the synchronous adapter-start boundary before adding more runtime + * special cases. Logical demand is visible during this callback, but a + * physical lease exists only if the callback returns. + */ +describe(`CollectionSubscription demand lifecycle oracle`, () => { + it(`executes every required async restart regime`, async () => { + const reach = new Set() + for (const scenario of asyncRestartCoverageScenarios) { + for (const label of await runAsyncRestartScenario(scenario)) { + reach.add(label) + } + } + const required = [ + `demands:1`, + `demands:2`, + `sessions:2`, + `sessions:3`, + `sessions:4`, + `current:resolve`, + `current:reject`, + `mixed-current:true`, + `obsolete-reject:true`, + `order:obsolete-first`, + `order:current-first`, + `order:interleaved`, + `real-interleaving:true`, + ] + expect(required.filter((label) => !reach.has(label))).toEqual([]) + }) + + it(`covers every finite start, failure-delivery, and release cell`, () => { + expect( + new Set( + startScenarios.map(({ outcome, reentry }) => `${outcome}:${reentry}`), + ), + ).toHaveLength(startOutcomes.length * startReentries.length) + expect( + new Set( + failureScenarios.map(({ outcome, reentry }) => `${outcome}:${reentry}`), + ), + ).toHaveLength(2 * startReentries.length) + expect( + new Set( + releaseScenarios.map(({ outcome, reentry }) => `${outcome}:${reentry}`), + ), + ).toHaveLength(2 * 4) + expect( + new Set( + restartScenarios.map(({ outcome, reentry }) => `${outcome}:${reentry}`), + ), + ).toHaveLength(4 * 5) + }) + + it(`accounts for every acquisition phase and entry pair`, () => { + const allCells = new Set( + acquisitionPhases.flatMap((phase) => + acquisitionEntries.map((entry) => `${phase}:${entry}` as const), + ), + ) + expect( + new Set([...legalAcquisitionCells, ...excludedAcquisitionCells.keys()]), + ).toEqual(allCells) + }) + + it(`accounts for every physical acquisition state and interaction cause`, () => { + const allCells = new Set( + physicalAcquisitionStates.flatMap((state) => + physicalInteractionCauses.map((cause) => `${state}:${cause}` as const), + ), + ) + expect(new Set(Object.keys(physicalInteractionCellDefinitions))).toEqual( + allCells, + ) + }) + + afterAll(() => { + expect(observedAcquisitionCells).toEqual(legalAcquisitionCells) + expect(observedPhysicalInteractions).toEqual(requiredPhysicalInteractions) + expect(observedSourceSessionBoundaries).toEqual( + requiredSourceSessionBoundaries, + ) + expect(observedFailureDeliverySuffixes).toEqual( + requiredFailureDeliverySuffixes, + ) + }) + + it.each(startScenarios)( + `keeps logical and physical ownership aligned for $outcome × $reentry`, + async ({ outcome, reentry }) => { + const targetWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`target`), + ]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`peer`)]) + const failure = new Error(`target load failed`) + const pending = createDeferred() + // A reentrant release can make the subscription stop observing the + // adapter Promise. Keep the test process deterministic while separately + // asserting the subscription's public error trace below. + void pending.promise.catch(() => {}) + const loads: Array = [] + const unloads: Array = [] + const errors: Array = [] + const statuses: Array = [] + const controller = new AbortController() + let didReenter = false + let statusAtTruncate: string | undefined + let truncate!: () => void + let runReentry = () => {} + + const collection = createOnDemandCollection<{ id: string }>({ + id: `demand-start-${outcome}-${reentry}`, + sync: { + sync: (operations) => { + const { markReady } = operations + truncate = () => { + operations.begin() + operations.truncate() + operations.commit() + } + markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (options.where === peerWhere) return true + if (didReenter) return true + didReenter = true + runReentry() + if (outcome === `throw`) throw failure + if (outcome === `return`) return true + return pending.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + + if (reentry === `release-peer`) { + subscription.requestSnapshot({ where: peerWhere }) + } + runReentry = () => { + if (reentry === `abort-self`) { + controller.abort() + } else if (reentry === `truncate`) { + truncate() + statusAtTruncate = subscription.status + } else if (reentry === `release-self`) { + subscription.releaseSnapshot(targetWhere) + } else if (reentry === `release-peer`) { + subscription.releaseSnapshot(peerWhere) + } else if (reentry === `unsubscribe`) { + subscription.unsubscribe() + } else if (reentry === `cleanup`) { + void collection.cleanup() + } + } + + let thrown: unknown + try { + subscription.requestSnapshot({ + where: targetWhere, + signal: controller.signal, + }) + } catch (error) { + thrown = error + } + + if (reentry === `truncate`) { + // The old acquisition is obsolete, but the queued replacement still + // owns a loading interval until its setup and work finish. + expect(statusAtTruncate).toBe(`loadingSubset`) + expect(subscription.status).toBe(`loadingSubset`) + expect(loads).toHaveLength(1) + } + const targetLoad = loads.find(({ where }) => where === targetWhere)! + const peerLoad = loads.find(({ where }) => where === peerWhere) + const targetWasReleased = + reentry === `release-self` || + reentry === `truncate` || + reentry === `unsubscribe` || + reentry === `cleanup` + const targetStarted = outcome !== `throw` && reentry !== `cleanup` + + if (outcome === `resolve`) pending.resolve() + if (outcome === `reject`) pending.reject(failure) + await flushPromises() + + const interaction = + reentry === `abort-self` + ? `starting:abort` + : reentry === `truncate` + ? `starting:truncate` + : reentry === `release-self` + ? `starting:release` + : reentry === `release-peer` + ? `active:release` + : reentry === `unsubscribe` + ? `starting:unsubscribe` + : reentry === `cleanup` + ? `starting:cleanup` + : undefined + if (interaction) { + observePhysicalInteraction( + interaction, + reentry === `abort-self` ? `abort-only` : `retire`, + ) + } + + expect(thrown).toBe(outcome === `throw` ? failure : undefined) + expect(targetLoad.signal?.aborted).toBe( + outcome === `throw` || targetWasReleased || reentry === `abort-self`, + ) + expect(unloads.filter((options) => options === targetLoad)).toHaveLength( + Number(targetStarted && targetWasReleased), + ) + expect(unloads.filter((options) => options === peerLoad)).toHaveLength( + Number(reentry === `release-peer`), + ) + expect(errors).toEqual( + (outcome === `throw` || outcome === `reject`) && + !targetWasReleased && + reentry !== `abort-self` + ? [failure] + : [], + ) + expect(statuses).toEqual( + reentry === `truncate` || + ((outcome === `resolve` || outcome === `reject`) && + !targetWasReleased) + ? [`loadingSubset`, `ready`] + : [], + ) + if (reentry === `truncate`) { + // A synchronous throw never acquired an owner to replay. Returned work + // retains logical demand, even when its first transport later rejects. + expect(loads).toHaveLength(outcome === `throw` ? 1 : 2) + if (outcome !== `throw`) { + expect(loads[1]).not.toBe(targetLoad) + expect(loads[1]?.where).toBe(targetWhere) + expect(loads[1]?.signal?.aborted).toBe(false) + } + } + + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + it.each(failureScenarios)( + `keeps a $outcome failure primary during $reentry error delivery`, + async ({ outcome, reentry }) => { + const targetWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`target`), + ]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`peer`)]) + const failure = new Error(`target load failed`) + const pending = createDeferred() + const loads: Array = [] + const attempts: Array<{ + session: number + options: LoadSubsetOptions + result: `peer-return` | `throw` | `pending` | `replay-return` + }> = [] + const unloads: Array = [] + const sourceCleanupSessions: Array = [] + const errors: Array = [] + const statuses: Array = [] + const controller = new AbortController() + let truncateCount = 0 + let truncate = () => {} + let targetLoadCount = 0 + + const collection = createOnDemandCollection<{ id: string }>({ + id: `demand-failure-${outcome}-${reentry}`, + sync: { + sync: (operations) => { + truncate = () => { + truncateCount++ + operations.begin() + operations.truncate() + operations.commit() + } + const { markReady } = operations + markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (options.where === peerWhere) { + attempts.push({ + session: 0, + options, + result: `peer-return`, + }) + return true + } + targetLoadCount++ + if (targetLoadCount > 1) { + attempts.push({ + session: 0, + options, + result: `replay-return`, + }) + return true + } + if (outcome === `throw`) { + attempts.push({ session: 0, options, result: `throw` }) + throw failure + } + attempts.push({ session: 0, options, result: `pending` }) + return pending.promise + }, + unloadSubset: (options) => unloads.push(options), + cleanup: () => sourceCleanupSessions.push(0), + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + subscription.on(`loadSubset:error`, ({ error }) => { + errors.push(error) + if (reentry === `abort-self`) { + controller.abort() + } else if (reentry === `truncate`) { + truncate() + } else if (reentry === `release-self`) { + subscription.releaseSnapshot(targetWhere) + } else if (reentry === `release-peer`) { + subscription.releaseSnapshot(peerWhere) + } else if (reentry === `unsubscribe`) { + subscription.unsubscribe() + } else if (reentry === `cleanup`) { + void collection.cleanup() + } + }) + + subscription.requestSnapshot({ where: peerWhere }) + let thrown: unknown + try { + subscription.requestSnapshot({ + where: targetWhere, + signal: controller.signal, + }) + } catch (error) { + thrown = error + } + if (outcome === `reject`) { + pending.reject(failure) + } + await flushPromises() + + const targetLoad = loads.find(({ where }) => where === targetWhere)! + const peerLoad = loads.find(({ where }) => where === peerWhere)! + const targetAttempts = attempts.filter( + ({ options }) => options.where === targetWhere, + ) + const peerAttempts = attempts.filter( + ({ options }) => options.where === peerWhere, + ) + const tearsDownTarget = + reentry === `release-self` || reentry === `unsubscribe` + + expect.soft(thrown).toBe(outcome === `throw` ? failure : undefined) + expect.soft(errors).toEqual([failure]) + expect.soft(subscription.lastError).toBe(failure) + expect.soft(targetAttempts[0]?.options).toBe(targetLoad) + expect.soft(targetAttempts[0]?.session).toBe(0) + expect + .soft(targetAttempts[0]?.result) + .toBe(outcome === `throw` ? `throw` : `pending`) + expect.soft(controller.signal.aborted).toBe(reentry === `abort-self`) + expect.soft(truncateCount).toBe(Number(reentry === `truncate`)) + expect + .soft(unloads.filter((options) => options === targetLoad)) + .toHaveLength( + Number( + outcome === `reject` && (tearsDownTarget || reentry === `truncate`), + ), + ) + expect + .soft(unloads.filter((options) => options === peerLoad)) + .toHaveLength( + Number( + reentry === `release-peer` || + reentry === `unsubscribe` || + reentry === `truncate`, + ), + ) + expect + .soft(statuses) + .toEqual( + reentry === `truncate` + ? [`loadingSubset`, `ready`] + : outcome === `reject` + ? reentry === `unsubscribe` + ? [`loadingSubset`] + : [`loadingSubset`, `ready`] + : [], + ) + if (reentry === `abort-self`) { + expect.soft(targetLoad.signal?.aborted).toBe(true) + expect.soft(targetAttempts).toHaveLength(1) + expect.soft(peerAttempts).toHaveLength(1) + } + const replacement = targetAttempts[1]?.options + const peerReplacement = peerAttempts[1]?.options + if (reentry === `truncate`) { + expect.soft(targetLoad.signal?.aborted).toBe(true) + // Error delivery may request replay, but it cannot turn a failed + // synchronous start into an owned acquisition. Only the peer survives. + expect.soft(targetAttempts).toHaveLength(outcome === `reject` ? 2 : 1) + if (outcome === `reject`) { + expect.soft(replacement).not.toBe(targetLoad) + expect.soft(replacement?.where).toBe(targetWhere) + expect.soft(targetAttempts[1]?.session).toBe(0) + expect.soft(targetAttempts[1]?.result).toBe(`replay-return`) + } else { + expect.soft(replacement).toBeUndefined() + } + expect.soft(peerLoad.signal?.aborted).toBe(true) + expect.soft(peerAttempts).toHaveLength(2) + expect.soft(peerReplacement).not.toBe(peerLoad) + expect.soft(peerReplacement?.where).toBe(peerWhere) + expect.soft(peerAttempts[1]?.session).toBe(0) + expect.soft(peerAttempts[1]?.result).toBe(`peer-return`) + expect.soft(unloads).toHaveLength(outcome === `reject` ? 2 : 1) + } + if (reentry === `cleanup`) { + expect.soft(collection.status).toBe(`cleaned-up`) + expect.soft(peerLoad.signal?.aborted).toBe(true) + expect.soft(targetLoad.signal?.aborted).toBe(true) + expect.soft(subscription.status).toBe(`ready`) + } + + subscription.unsubscribe() + const replays = reentry === `truncate` + expect + .soft(targetAttempts) + .toHaveLength(replays && outcome === `reject` ? 2 : 1) + expect.soft(peerAttempts).toHaveLength(replays ? 2 : 1) + expect + .soft(unloads.filter((options) => options === targetLoad)) + .toHaveLength(Number(outcome === `reject` && reentry !== `cleanup`)) + expect + .soft(unloads.filter((options) => options === peerLoad)) + .toHaveLength(Number(reentry !== `cleanup`)) + if (replacement) { + expect + .soft(unloads.filter((options) => options === replacement)) + .toHaveLength(Number(replays)) + expect.soft(replacement.signal?.aborted).toBe(true) + } + if (peerReplacement) { + expect + .soft(unloads.filter((options) => options === peerReplacement)) + .toHaveLength(Number(replays)) + expect.soft(peerReplacement.signal?.aborted).toBe(true) + } + const expectedUnloads = + reentry === `cleanup` + ? 0 + : (outcome === `reject` ? 2 : 1) * (replays ? 2 : 1) + expect.soft(unloads).toHaveLength(expectedUnloads) + expect.soft(peerLoad.signal?.aborted).toBe(true) + expect.soft(targetLoad.signal?.aborted).toBe(true) + const terminalAttempts = [...attempts] + const terminalUnloads = [...unloads] + const terminalStatuses = [...statuses] + await collection.cleanup() + expect.soft(sourceCleanupSessions).toEqual([0]) + expect.soft(collection.status).toBe(`cleaned-up`) + expect.soft(errors).toEqual([failure]) + expect.soft(subscription.lastError).toBe(failure) + expect.soft(attempts).toEqual(terminalAttempts) + expect.soft(unloads).toEqual(terminalUnloads) + expect.soft(statuses).toEqual(terminalStatuses) + observedFailureDeliverySuffixes.add(`${outcome}:${reentry}`) + }, + ) + + it.each(releaseScenarios)( + `retires logical ownership once for unload $outcome × $reentry`, + async ({ outcome, reentry }) => { + const targetWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`target`), + ]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`peer`)]) + const releaseFailure = new Error(`target release failed`) + const loads: Array = [] + const unloads: Array = [] + const errors: Array = [] + let allowRelease = outcome === `return` + let runReentry = () => {} + + const collection = createOnDemandCollection<{ id: string }>({ + id: `demand-release-${outcome}-${reentry}`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[1]) { + runReentry() + if (!allowRelease) throw releaseFailure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.requestSnapshot({ where: peerWhere }) + subscription.requestSnapshot({ where: targetWhere }) + const peerLoad = loads[0]! + const oldTargetLoad = loads[1]! + runReentry = () => { + runReentry = () => {} + if (reentry === `reacquire-self`) { + subscription.requestSnapshot({ where: targetWhere }) + } else if (reentry === `release-peer`) { + subscription.releaseSnapshot(peerWhere) + } else if (reentry === `unsubscribe`) { + subscription.unsubscribe() + } + } + + let thrown: unknown + try { + subscription.releaseSnapshot(targetWhere) + } catch (error) { + thrown = error + } + observePhysicalInteraction(`active:release`, `retire`) + if (reentry === `unsubscribe`) { + observePhysicalInteraction(`active:unsubscribe`, `retire`) + } + + expect(thrown).toBe(outcome === `throw` ? releaseFailure : undefined) + expect(oldTargetLoad.signal?.aborted).toBe(true) + expect( + unloads.filter((options) => options === oldTargetLoad), + ).toHaveLength(1) + expect(unloads.filter((options) => options === peerLoad)).toHaveLength( + Number(reentry === `release-peer` || reentry === `unsubscribe`), + ) + expect(errors).toEqual( + outcome === `throw` && reentry !== `unsubscribe` + ? [releaseFailure] + : [], + ) + expect(subscription.lastError).toBe( + outcome === `throw` ? releaseFailure : undefined, + ) + + allowRelease = true + subscription.unsubscribe() + expect( + unloads.filter((options) => options === oldTargetLoad), + ).toHaveLength(1) + const replacement = loads[2] + expect( + replacement === undefined + ? [] + : unloads.filter((options) => options === replacement), + ).toHaveLength(Number(reentry === `reacquire-self`)) + expect(unloads.filter((options) => options === peerLoad)).toHaveLength(1) + if (outcome === `throw` && reentry === `unsubscribe`) { + observePhysicalInteraction( + `failed-release:unsubscribe`, + `no-repeat-on-unsubscribe`, + ) + } + await collection.cleanup() + }, + ) + + it.each([`resolve`, `reject`] as const)( + `retires a pending replay on cleanup before an obsolete %s`, + async (outcome) => { + type Row = { id: string; version: number } + const replay = createDeferred() + const replayFailure = new Error(`obsolete replay failed`) + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let syncSession = 0 + let loadCount = 0 + const visible = new Map() + const errors: Array = [] + const statuses: Array = [] + + const collection = createOnDemandCollection({ + id: `cleanup-pending-replay-${outcome}`, + sync: { + sync: (operations) => { + syncSession++ + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + if (syncSession > 1) { + begin() + write({ type: `insert`, value: { id: `row`, version: 3 } }) + commit() + } + operations.markReady() + return { + loadSubset: () => { + loadCount++ + begin() + write({ + type: `insert`, + value: { id: `row`, version: loadCount }, + }) + commit() + return loadCount === 1 || syncSession > 1 + ? true + : replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + + subscription.requestSnapshot() + expect([...visible.values()]).toEqual([{ id: `row`, version: 1 }]) + begin() + truncate() + commit() + await flushPromises() + expect(subscription.status).toBe(`loadingSubset`) + + await collection.cleanup() + collection.startSyncImmediate() + expect(syncSession).toBe(2) + expect([...visible.values()]).toEqual([{ id: `row`, version: 3 }]) + expect(subscription.status).toBe(`loadingSubset`) + await flushPromises() + expect(subscription.status).toBe(`ready`) + + if (outcome === `resolve`) replay.resolve() + else replay.reject(replayFailure) + await flushPromises() + + expect([...visible.values()]).toEqual([{ id: `row`, version: 3 }]) + expect(errors).toEqual([]) + expect(subscription.lastError).toBeUndefined() + expect(statuses.at(-1)).toBe(`ready`) + + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + it(`reacquires surviving on-demand demand after collection restart`, async () => { + type Row = { id: string; version: number } + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let syncSession = 0 + let loadCount = 0 + const loads: Array = [] + const unloads: Array = [] + const visible = new Map() + const collection = createOnDemandCollection({ + id: `restart-surviving-demand`, + sync: { + sync: (operations) => { + syncSession++ + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + loadCount++ + begin() + write({ + type: `insert`, + value: { id: `row`, version: loadCount }, + }) + commit() + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + }, + { includeInitialState: false }, + ) + subscription.requestSnapshot() + expect([...visible.values()]).toEqual([{ id: `row`, version: 1 }]) + + await collection.cleanup() + collection.startSyncImmediate() + expect(subscription.status).toBe(`loadingSubset`) + expect(loads).toHaveLength(1) + await flushPromises() + + expect(syncSession).toBe(2) + expect(loads).toHaveLength(2) + expect(unloads).toEqual([]) + expect([...visible.values()]).toEqual([{ id: `row`, version: 2 }]) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + expect(unloads).toEqual([loads[1]]) + await collection.cleanup() + }) + + it(`reacquires demand requested while the collection is cleaned up`, async () => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + let syncSession = 0 + const loads: Array<{ session: number; options: LoadSubsetOptions }> = [] + const unloads: Array<{ session: number; options: LoadSubsetOptions }> = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `request-while-cleaned-up`, + sync: { + sync: ({ markReady }) => { + const session = syncSession++ + markReady() + return { + loadSubset: (options) => { + loads.push({ session, options }) + return true + }, + unloadSubset: (options) => unloads.push({ session, options }), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where: oldWhere }) + + await collection.cleanup() + subscription.requestSnapshot({ where: newWhere }) + collection.startSyncImmediate() + await flushPromises() + + expect(loads.map(({ session }) => session)).toEqual([0, 1, 1]) + expect(loads.slice(1).map(({ options }) => options.where)).toEqual([ + oldWhere, + newWhere, + ]) + + subscription.unsubscribe() + expect(unloads.map(({ session }) => session)).toEqual([1, 1]) + expect(unloads.map(({ options }) => options.where)).toEqual([ + oldWhere, + newWhere, + ]) + await collection.cleanup() + }) + + it(`does not report a detached demand as physically settled`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const observed: Array = [] + let loads = 0 + const collection = createOnDemandCollection<{ id: string }>({ + id: `detached-demand-settlement`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + await collection.cleanup() + subscription.requestSnapshot({ + where, + onLoadSubsetResult: (result) => observed.push(result), + }) + + expect(loads).toBe(0) + expect(observed).toEqual([expect.any(Promise)]) + + collection.startSyncImmediate() + await flushPromises() + expect(loads).toBe(1) + expect(observed).toEqual([expect.any(Promise)]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + acquisitionCase( + [`starting:request`], + `includes demand created by the synchronous restart status callback`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const demandForWhere = new Map([ + [oldWhere, `old`], + [newWhere, `new`], + ]) + const loads: Array<{ session: number; demand: `old` | `new` }> = [] + const unloads: Array<{ session: number; demand: `old` | `new` }> = [] + let session = -1 + let requestOnRestart = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `restart-status-reentry`, + sync: { + sync: ({ markReady }) => { + session++ + const adapterSession = session + markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown restart demand`) + loads.push({ session: adapterSession, demand }) + return true + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown restart demand`) + unloads.push({ session: adapterSession, demand }) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`status:change`, ({ status }) => { + if (!requestOnRestart || status !== `loadingSubset`) return + requestOnRestart = false + subscription.requestSnapshot({ where: newWhere }) + reach(`starting:request`) + }) + subscription.requestSnapshot({ where: oldWhere }) + + await collection.cleanup() + requestOnRestart = true + collection.startSyncImmediate() + await flushPromises() + + expect(loads).toEqual([ + { session: 0, demand: `old` }, + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + expect(unloads).toEqual([ + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + await collection.cleanup() + }, + ) + + acquisitionCase( + [`starting:markReady`], + `does not settle demand reentered before the restart loader is installed`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const demandForWhere = new Map([ + [oldWhere, `old`], + [newWhere, `new`], + ]) + const loads: Array<{ session: number; demand: `old` | `new` }> = [] + const unloads: Array<{ session: number; demand: `old` | `new` }> = [] + const observed: Array = [] + let session = -1 + let requestOnReady = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `restart-ready-reentry`, + sync: { + sync: ({ markReady }) => { + session++ + const adapterSession = session + markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown ready demand`) + loads.push({ session: adapterSession, demand }) + return true + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown ready unload`) + unloads.push({ session: adapterSession, demand }) + }, + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + const removeReadyListener = collection.on(`status:ready`, () => { + if (!requestOnReady) return + requestOnReady = false + subscription.requestSnapshot({ + where: newWhere, + onLoadSubsetResult: (result) => observed.push(result), + }) + reach(`starting:markReady`) + }) + subscription.requestSnapshot({ where: oldWhere }) + + await collection.cleanup() + requestOnReady = true + collection.startSyncImmediate() + await flushPromises() + + expect(loads).toEqual([ + { session: 0, demand: `old` }, + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + expect(observed).toEqual([expect.any(Promise)]) + expect(subscription.status).toBe(`ready`) + + removeReadyListener() + subscription.unsubscribe() + expect(unloads).toEqual([ + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + await collection.cleanup() + }, + ) + + acquisitionCase( + [`unavailable:request`], + `does not settle demand reentered before a failed restart installs a loader`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const syncFailure = new Error(`replacement sync failed`) + const demandForWhere = new Map([ + [oldWhere, `old`], + [newWhere, `new`], + ]) + const loads: Array<{ session: number; demand: `old` | `new` }> = [] + const unloads: Array<{ session: number; demand: `old` | `new` }> = [] + const observed: Array = [] + let session = -1 + let requestOnError = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `restart-error-reentry`, + sync: { + sync: ({ markReady }) => { + session++ + const adapterSession = session + if (session === 1) throw syncFailure + markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown error demand`) + loads.push({ session: adapterSession, demand }) + return true + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown error unload`) + unloads.push({ session: adapterSession, demand }) + }, + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + const removeErrorListener = collection.on(`status:error`, () => { + if (!requestOnError) return + requestOnError = false + subscription.requestSnapshot({ + where: newWhere, + onLoadSubsetResult: (result) => observed.push(result), + }) + reach(`unavailable:request`) + }) + subscription.requestSnapshot({ where: oldWhere }) + + await collection.cleanup() + requestOnError = true + expect(() => collection.startSyncImmediate()).toThrow(syncFailure) + expect(observed).toEqual([expect.any(Promise)]) + expect(collection.status).toBe(`error`) + expect(loads).toEqual([{ session: 0, demand: `old` }]) + + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + expect(loads).toEqual([ + { session: 0, demand: `old` }, + { session: 2, demand: `old` }, + { session: 2, demand: `new` }, + ]) + expect(subscription.status).toBe(`ready`) + + removeErrorListener() + subscription.unsubscribe() + expect(unloads).toEqual([ + { session: 2, demand: `old` }, + { session: 2, demand: `new` }, + ]) + await collection.cleanup() + }, + ) + + acquisitionCase( + [`retiring:request`], + `does not acquire through a retiring adapter cleanup callback`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const demandForWhere = new Map([ + [oldWhere, `old`], + [newWhere, `new`], + ]) + const loads: Array<{ session: number; demand: `old` | `new` }> = [] + const unloads: Array<{ session: number; demand: `old` | `new` }> = [] + const observed: Array = [] + let session = -1 + let requestDuringCleanup = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `adapter-cleanup-reentry`, + sync: { + sync: ({ markReady }) => { + session++ + const adapterSession = session + markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown cleanup demand`) + loads.push({ session: adapterSession, demand }) + return true + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown cleanup unload`) + unloads.push({ session: adapterSession, demand }) + }, + cleanup: () => { + if (!requestDuringCleanup) return + requestDuringCleanup = false + subscription.requestSnapshot({ + where: newWhere, + onLoadSubsetResult: (result) => observed.push(result), + }) + reach(`retiring:request`) + observeSourceSessionBoundary(`cleanup-callback-reentry`) + }, + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + subscription.requestSnapshot({ where: oldWhere }) + + requestDuringCleanup = true + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + + expect(loads).toEqual([ + { session: 0, demand: `old` }, + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + expect(observed).toEqual([expect.any(Promise)]) + + subscription.unsubscribe() + expect(unloads).toEqual([ + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + await collection.cleanup() + }, + ) + + acquisitionCase( + [`eager:request`], + `does not release a physical subset acquisition in eager mode`, + async (reach) => { + let loads = 0 + let unloads = 0 + const collection = createCollection<{ id: string }>({ + id: `eager-subset-ownership`, + getKey: ({ id }) => id, + syncMode: `eager`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + unloadSubset: () => { + unloads++ + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + subscription.requestSnapshot() + reach(`eager:request`) + subscription.unsubscribe() + observePhysicalInteraction(`none:unsubscribe`, `no-acquisition`) + + expect(loads).toBe(0) + expect(unloads).toBe(0) + await collection.cleanup() + }, + ) + + it(`directly releases eager demand without a physical acquisition`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + let loads = 0 + let unloads = 0 + const collection = createCollection<{ id: string }>({ + id: `eager-direct-release`, + getKey: ({ id }) => id, + syncMode: `eager`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + unloadSubset: () => { + unloads++ + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + subscription.requestSnapshot({ where }) + subscription.releaseSnapshot(where) + observePhysicalInteraction(`none:release`, `no-acquisition`) + + expect(loads).toBe(0) + expect(unloads).toBe(0) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`truncates eager demand without creating or releasing a physical acquisition`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const loads: Array = [] + const unloads: Array = [] + const collection = createCollection<{ id: string }>({ + id: `eager-truncate-without-acquisition`, + getKey: ({ id }) => id, + syncMode: `eager`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + + begin() + truncate() + commit() + await flushPromises() + observePhysicalInteraction(`none:truncate`, `no-acquisition`) + + expect.soft(loads).toEqual([]) + expect.soft(unloads).toEqual([]) + + subscription.unsubscribe() + expect(unloads).toEqual([]) + await collection.cleanup() + }) + + acquisitionCase( + [`on-demand:request`], + `does not release a subset request aborted before adapter acquisition`, + async (reach) => { + const controller = new AbortController() + controller.abort() + let loads = 0 + let unloads = 0 + const errors: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `pre-aborted-subset-ownership`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + unloadSubset: () => { + unloads++ + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + + subscription.requestSnapshot({ signal: controller.signal }) + await flushPromises() + reach(`on-demand:request`) + subscription.unsubscribe() + observePhysicalInteraction(`none:unsubscribe`, `no-acquisition`) + + expect(loads).toBe(0) + expect(unloads).toBe(0) + expect(errors).toEqual([]) + await collection.cleanup() + }, + ) + + it(`directly releases pre-aborted demand without a physical acquisition`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const controller = new AbortController() + controller.abort() + let loads = 0 + let unloads = 0 + const errors: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `pre-aborted-direct-release`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + unloadSubset: () => { + unloads++ + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + + subscription.requestSnapshot({ + where, + signal: controller.signal, + }) + await flushPromises() + subscription.releaseSnapshot(where) + observePhysicalInteraction(`none:release`, `no-acquisition`) + + expect(loads).toBe(0) + expect(unloads).toBe(0) + expect(errors).toEqual([]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it.each([false, true])( + `ignores a pre-aborted snapshot without changing an existing demand: %s`, + async (existingDemand) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const loads: Array = [] + const unloads: Array = [] + let publications = 0 + let results = 0 + const collection = createOnDemandCollection<{ id: string }>({ + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row` } }) + commit() + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => publications++, { + includeInitialState: false, + }) + try { + if (existingDemand) subscription.requestSnapshot({ where }) + const previousPublications = publications + const previousLoads = [...loads] + const controller = new AbortController() + controller.abort() + + expect( + subscription.requestSnapshot({ + where, + signal: controller.signal, + onLoadSubsetResult: () => results++, + }), + ).toBe(false) + await flushPromises() + expect(publications).toBe(previousPublications) + expect(results).toBe(0) + expect(loads).toEqual(previousLoads) + expect(unloads).toEqual([]) + if (existingDemand) expect(loads[0]!.signal?.aborted).toBe(false) + + subscription.releaseSnapshot(where) + expect(unloads).toEqual(previousLoads) + subscription.unsubscribe() + expect(unloads).toEqual(previousLoads) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`aborts detached demand without creating a physical acquisition`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const controller = new AbortController() + const loads: Array = [] + const unloads: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `detached-abort-without-acquisition`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + await collection.cleanup() + subscription.requestSnapshot({ where, signal: controller.signal }) + controller.abort() + await flushPromises() + observePhysicalInteraction(`none:abort`, `no-acquisition`) + + expect(loads).toEqual([]) + expect(unloads).toEqual([]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`keeps an aborted active acquisition until its owner retires`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const controller = new AbortController() + const pending = createDeferred() + const loads: Array = [] + const unloads: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `active-abort-before-release`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return pending.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + subscription.requestSnapshot({ where, signal: controller.signal }) + const acquisition = loads[0]! + controller.abort() + await flushPromises() + observePhysicalInteraction(`active:abort`, `abort-only`) + + expect(loads).toEqual([acquisition]) + expect(acquisition.signal?.aborted).toBe(true) + expect(unloads).toEqual([]) + + pending.resolve() + await flushPromises() + subscription.unsubscribe() + expect(unloads).toEqual([acquisition]) + await collection.cleanup() + }) + + acquisitionCase( + [`on-demand:request`, `on-demand:markReady`], + `acquires before and after ready once the on-demand loader is installed`, + async (reach) => { + const beforeReady = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`before`), + ]) + const afterReady = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`after`), + ]) + const loads: Array = [] + const unloads: Array = [] + let markReady!: () => void + const collection = createOnDemandCollection<{ id: string }>({ + id: `installed-loader-before-ready`, + startSync: false, + sync: { + sync: (operations) => { + markReady = operations.markReady + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + collection.startSyncImmediate() + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const removeReadyListener = collection.on(`status:ready`, () => { + subscription.requestSnapshot({ where: afterReady }) + reach(`on-demand:markReady`) + }) + + subscription.requestSnapshot({ where: beforeReady }) + reach(`on-demand:request`) + expect(collection.status).toBe(`loading`) + expect(loads.map(({ where }) => where)).toEqual([beforeReady]) + + markReady() + await flushPromises() + expect(loads.map(({ where }) => where)).toEqual([beforeReady, afterReady]) + + removeReadyListener() + subscription.unsubscribe() + expect(unloads).toEqual(loads) + await collection.cleanup() + }, + ) + + acquisitionCase( + [`deferred:request`, `deferred:resume`], + `owns deferred-start acquisition only when it reaches the adapter`, + async (reach) => { + for (const action of [`resume`, `release-before-resume`] as const) { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const loads: Array = [] + const unloads: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `deferred-start-${action}`, + startSync: false, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + reach(`deferred:request`) + expect(loads).toEqual([]) + + if (action === `release-before-resume`) { + subscription.releaseSnapshot(where) + } + collection._resumeSyncStart() + await flushPromises() + if (action === `resume`) reach(`deferred:resume`) + + expect(loads).toHaveLength(action === `resume` ? 1 : 0) + subscription.unsubscribe() + expect(unloads).toHaveLength(action === `resume` ? 1 : 0) + if (action === `resume`) expect(unloads).toEqual(loads) + await collection.cleanup() + } + }, + ) + + it.each([`cleanup`, `release`, `unsubscribe`, `resume`] as const)( + `settles queued demand according to whether it starts: %s`, + async (action) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const observed: Array> = [] + let loads = 0 + const collection = createOnDemandCollection<{ id: string }>({ + id: `deferred-start-cleanup-before-resume`, + startSync: false, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ + where, + onLoadSubsetResult: (result) => observed.push(result), + }) + + if (action === `cleanup`) await collection.cleanup() + else if (action === `release`) subscription.releaseSnapshot(where) + else if (action === `unsubscribe`) subscription.unsubscribe() + else collection._resumeSyncStart() + await flushPromises() + if (action === `cleanup`) { + observePhysicalInteraction(`none:cleanup`, `no-acquisition`) + } + + expect(loads).toBe(action === `resume` ? 1 : 0) + expect(observed).toHaveLength(1) + const deferredResult = observed[0] + expect(deferredResult).toBeInstanceOf(Promise) + if (!(deferredResult instanceof Promise)) { + throw new Error(`deferred acquisition did not return a promise`) + } + if (action === `resume`) + await expect(deferredResult).resolves.toBeUndefined() + else + await expect(deferredResult).rejects.toMatchObject({ + name: `AbortError`, + }) + + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + acquisitionCase( + [`starting:syncReturn`], + `does not settle ready-callback demand when on-demand sync returns no loader`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const loads: Array = [] + const observed: Array = [] + let session = 0 + let requestOnReady = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `ready-before-invalid-on-demand-return`, + sync: { + sync: ({ markReady }) => { + const ownSession = session++ + markReady() + if (ownSession === 1) return + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + const removeReadyListener = collection.on(`status:ready`, () => { + if (!requestOnReady) return + requestOnReady = false + subscription.requestSnapshot({ + where: newWhere, + onLoadSubsetResult: (result) => observed.push(result), + }) + }) + subscription.requestSnapshot({ where: oldWhere }) + + await collection.cleanup() + requestOnReady = true + expect(() => collection.startSyncImmediate()).toThrow( + /did not return a loadSubset handler/, + ) + reach(`starting:syncReturn`) + + expect(observed).toEqual([expect.any(Promise)]) + expect(collection.status).toBe(`error`) + expect(loads.map(({ where }) => where)).toEqual([oldWhere]) + + removeReadyListener() + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + it(`retires resources returned after ready-callback cleanup invalidates sync`, async () => { + const cleanupSessions: Array = [] + let session = 0 + let cleanOnReady = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `obsolete-sync-return`, + sync: { + sync: ({ markReady }) => { + const ownSession = session++ + markReady() + return { + loadSubset: () => true, + unloadSubset: () => {}, + cleanup: () => cleanupSessions.push(ownSession), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const removeReadyListener = collection.on(`status:ready`, () => { + if (!cleanOnReady) return + cleanOnReady = false + void collection.cleanup() + }) + + await collection.cleanup() + cleanOnReady = true + collection.startSyncImmediate() + observeSourceSessionBoundary(`obsolete-resource-return`) + + expect(collection.status).toBe(`cleaned-up`) + expect(cleanupSessions).toEqual([0, 1]) + + removeReadyListener() + subscription.unsubscribe() + await collection.cleanup() + }) + + it.each( + ([false, true] as const).flatMap((restart) => + ( + [`loading`, `ready`, `adapter-throw`, `ready-effect-throw`] as const + ).map((entry) => ({ restart, entry })), + ), + )( + `retires startup at $entry with nested restart=$restart`, + async ({ entry, restart }) => { + const failure = new Error(`obsolete startup failed`) + const cleanups: Array = [] + const loads: Array = [] + const unloads: Array = [] + let session = -1 + let retire = false + const collection = createOnDemandCollection<{ id: string }>({ + sync: { + sync: ({ markReady }) => { + const ownSession = ++session + markReady() + if (ownSession === 1 && entry === `adapter-throw`) throw failure + return { + loadSubset: () => { + loads.push(ownSession) + return true + }, + unloadSubset: () => unloads.push(ownSession), + cleanup: () => cleanups.push(ownSession), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + let removeListener = () => {} + try { + await collection.cleanup() + const retireSession = () => { + if (!retire) return + retire = false + void collection.cleanup() + if (restart) collection.startSyncImmediate() + if (entry === `ready-effect-throw`) throw failure + } + removeListener = + entry === `ready-effect-throw` + ? collection.onFirstReady(retireSession) + : collection.on( + entry === `loading` ? `status:loading` : `status:ready`, + retireSession, + ) + retire = true + if (entry === `adapter-throw` || entry === `ready-effect-throw`) { + expect(() => collection.startSyncImmediate()).toThrow(failure) + } else { + collection.startSyncImmediate() + } + await flushPromises() + + expect(collection.status).toBe(restart ? `ready` : `cleaned-up`) + const returnsObsoleteCleanup = + entry === `ready` || entry === `ready-effect-throw` + expect(cleanups).toEqual(returnsObsoleteCleanup ? [0, 1] : [0]) + expect(session).toBe( + entry === `loading` ? (restart ? 1 : 0) : restart ? 2 : 1, + ) + + if (restart) { + subscription.requestSnapshot({ + where: new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]), + }) + expect(loads).toEqual([session]) + subscription.unsubscribe() + expect(unloads).toEqual([session]) + } else { + expect(loads).toEqual([]) + expect(unloads).toEqual([]) + } + } finally { + removeListener() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + acquisitionCase( + [`starting:markError`, `unavailable:markReady`], + `retains demand requested during initial error for same-session recovery`, + async (reach) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const loads: Array = [] + const observed: Array = [] + let syncSession = 0 + let recover!: () => void + const collection = createOnDemandCollection<{ id: string }>({ + id: `sync-entry-error-ready-recovery`, + startSync: false, + sync: { + sync: ({ markError, markReady }) => { + if (syncSession++ === 0) { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: () => {}, + } + } + recover = markReady + markError(new Error(`initial sync failed`)) + reach(`starting:markError`) + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + await collection.cleanup() + const removeErrorListener = collection.on(`status:error`, () => { + subscription.requestSnapshot({ + where, + onLoadSubsetResult: (result) => observed.push(result), + }) + }) + + collection.startSyncImmediate() + await flushPromises() + expect.soft(collection.status).toBe(`error`) + expect.soft(loads).toEqual([]) + expect.soft(observed).toEqual([expect.any(Promise)]) + + recover() + reach(`unavailable:markReady`) + await flushPromises() + + expect(collection.status).toBe(`ready`) + expect(loads.map(({ where: loadedWhere }) => loadedWhere)).toEqual([ + where, + ]) + expect(observed).toEqual([expect.any(Promise)]) + await expect(observed[0]).resolves.toBeUndefined() + + removeErrorListener() + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + it.each( + ([`error`, `cleaned-up`] as const).flatMap((unavailable) => + ( + [ + `return`, + `resolve`, + `reject`, + `throw`, + `release`, + `unsubscribe`, + `cleanup`, + `abort`, + ] as const + ).flatMap((outcome) => + (outcome === `release` || + outcome === `unsubscribe` || + outcome === `cleanup` || + outcome === `abort` + ? ([`before`, `during`] as const) + : ([`during`] as const) + ).flatMap((phase) => + // Ordered snapshots do not accept an external AbortSignal. + (outcome === `abort` + ? ([`snapshot`] as const) + : ([`snapshot`, `limited`] as const) + ).map((entry) => ({ unavailable, outcome, phase, entry })), + ), + ), + ), + )( + `observes unavailable demand synchronously: $entry / $unavailable / $outcome / $phase`, + async ({ unavailable, outcome, phase, entry }) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const transport = createDeferred() + const failure = new Error(`recovery acquisition failed`) + const signal = new AbortController() + const loads: Array = [] + const unloads: Array = [] + const rows = new Map() + let operations!: Parameters[`sync`]>[0] + const collection = createOnDemandCollection<{ id: string }>({ + sync: { + sync: (next) => { + operations = next + return { + loadSubset: (options) => { + loads.push(options) + if (outcome === `throw`) throw failure + operations.begin() + operations.write({ type: `insert`, value: { id: `row` } }) + operations.commit() + return outcome === `return` ? true : transport.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) rows.delete(change.key) + else rows.set(change.key, change.value) + } + }, + { includeInitialState: false }, + ) + let result: true | Promise | undefined + let release: (() => void) | undefined + const settlements: Array = [] + const visibleOnSuccess: Array> = [] + let callbacks = 0 + try { + if (entry === `limited`) { + subscription.setOrderByIndex( + collection.createIndex((row) => row.id, { indexType: BTreeIndex }), + ) + } + if (unavailable === `error`) + operations.markError(new Error(`initial error`)) + else await collection.cleanup() + const onLoadSubsetResult = ( + value: true | Promise, + _options: LoadSubsetOptions, + releaseDemand?: () => void, + ) => { + callbacks++ + result = value + release = releaseDemand + if (value instanceof Promise) + void value.then( + () => { + visibleOnSuccess.push([...rows.values()].map(({ id }) => id)) + settlements.push(`success`) + }, + (error: unknown) => settlements.push(error), + ) + } + if (entry === `snapshot`) { + subscription.requestSnapshot({ + where, + signal: signal.signal, + onLoadSubsetResult, + }) + } else { + subscription.requestLimitedSnapshot({ + orderBy: [ + { + expression: new PropRef([`id`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + onLoadSubsetResult, + }) + } + // Production callers copy the result as soon as requestSnapshot returns. + expect(callbacks).toBe(1) + expect(result).toBeInstanceOf(Promise) + await flushPromises() + expect(settlements).toEqual([]) + expect(loads).toEqual([]) + const recover = () => { + if (unavailable === `cleaned-up`) collection.startSyncImmediate() + operations.markReady() + } + if (phase === `during`) { + recover() + await flushPromises() + expect(loads).toHaveLength(1) + if (outcome !== `return` && outcome !== `throw`) { + expect(settlements).toEqual([]) + expect([...rows.values()]).toEqual([]) + } + } + if (outcome === `release`) release!() + else if (outcome === `unsubscribe`) subscription.unsubscribe() + else if (outcome === `cleanup`) await collection.cleanup() + else if (outcome === `abort`) signal.abort() + else if (outcome === `reject`) transport.reject(failure) + else if (outcome === `resolve`) transport.resolve() + await flushPromises() + if (outcome === `return` || outcome === `resolve`) { + expect(settlements).toEqual([`success`]) + expect(visibleOnSuccess).toEqual([[`row`]]) + expect([...rows.values()].map(({ id }) => id)).toEqual([`row`]) + } else if (outcome === `throw` || outcome === `reject`) { + expect(settlements).toHaveLength(1) + expect(settlements[0]).toBe(failure) + expect([...rows.values()]).toEqual([]) + } else { + expect(settlements).toEqual([ + expect.objectContaining({ name: `AbortError` }), + ]) + expect(unloads).toEqual( + phase === `during` && + (outcome === `release` || outcome === `unsubscribe`) + ? loads + : [], + ) + if (phase === `before` && outcome !== `cleanup`) { + recover() + await flushPromises() + expect(loads).toEqual([]) + } + if (outcome === `cleanup`) { + // Cleanup ends this wait, not the surviving subscription's demand. + collection.startSyncImmediate() + operations.markReady() + await flushPromises() + expect(loads).toHaveLength(phase === `before` ? 1 : 2) + } + // Non-cooperative late settlement cannot rewrite the observed outcome. + transport.resolve() + await flushPromises() + expect(settlements).toEqual([ + expect.objectContaining({ name: `AbortError` }), + ]) + } + expect(callbacks).toBe(1) + } finally { + transport.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`re-enables an installed loader after same-session initial recovery`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const loads: Array = [] + let markError!: (error: unknown) => void + let markReady!: () => void + const collection = createOnDemandCollection<{ id: string }>({ + id: `installed-loader-error-ready-recovery`, + startSync: false, + sync: { + sync: (operations) => { + markError = operations.markError + markReady = operations.markReady + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + collection.startSyncImmediate() + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + markError(new Error(`initial sync failed`)) + markReady() + subscription.requestSnapshot({ where }) + + expect(collection.status).toBe(`ready`) + expect(loads.map(({ where: loadedWhere }) => loadedWhere)).toEqual([where]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`releases unavailable demand without creating a physical acquisition`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const loads: Array = [] + const unloads: Array = [] + let markError!: (error: unknown) => void + let markReady!: () => void + const collection = createOnDemandCollection<{ id: string }>({ + id: `release-unavailable-demand`, + startSync: false, + sync: { + sync: (operations) => { + markError = operations.markError + markReady = operations.markReady + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + collection.startSyncImmediate() + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + markError(new Error(`initial sync failed`)) + subscription.requestSnapshot({ where }) + subscription.releaseSnapshot(where) + markReady() + await flushPromises() + + expect(loads).toHaveLength(0) + expect(unloads).toHaveLength(0) + + subscription.unsubscribe() + await collection.cleanup() + }) + + acquisitionCase( + [`on-demand:markError`], + `defers demand while an installed loader is in initial error`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const loads: Array = [] + let markError!: (error: unknown) => void + let markReady!: () => void + const collection = createOnDemandCollection<{ id: string }>({ + id: `installed-loader-initial-error`, + startSync: false, + sync: { + sync: (operations) => { + markError = operations.markError + markReady = operations.markReady + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + collection.startSyncImmediate() + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where: oldWhere }) + const removeErrorListener = collection.on(`status:error`, () => { + subscription.requestSnapshot({ where: newWhere }) + reach(`on-demand:markError`) + }) + + markError(new Error(`initial sync failed`)) + + await flushPromises() + expect.soft(collection.status).toBe(`error`) + expect.soft(loads.map(({ where }) => where)).toEqual([oldWhere]) + + markReady() + await flushPromises() + expect(loads.map(({ where }) => where)).toEqual([oldWhere, newWhere]) + + removeErrorListener() + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + it.each( + ([`loading`, `ready`] as const).flatMap((phase) => + ([`cleanup`, `release`, `unsubscribe`] as const).map((action) => ({ + phase, + action, + })), + ), + )( + `cancels queued acquisition during $phase via $action`, + async ({ phase, action }) => { + const loads: Array = [] + const unloads: Array = [] + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + let cancelOnEntry = false + const observed: Array> = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `deferred-resume-cleanup`, + startSync: false, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const removeReadyListener = collection.on(`status:${phase}`, () => { + if (!cancelOnEntry) return + cancelOnEntry = false + if (action === `cleanup`) void collection.cleanup() + else if (action === `release`) subscription.releaseSnapshot(where) + else subscription.unsubscribe() + }) + subscription.requestSnapshot({ + where, + onLoadSubsetResult: (result) => observed.push(result), + }) + + try { + cancelOnEntry = true + collection._resumeSyncStart() + await flushPromises() + + expect(collection.status).toBe( + action === `cleanup` ? `cleaned-up` : `ready`, + ) + expect(loads).toHaveLength(0) + expect(unloads).toHaveLength(0) + expect(observed).toHaveLength(1) + await expect(observed[0]).rejects.toMatchObject({ name: `AbortError` }) + } finally { + removeReadyListener() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`keeps an eager subscription ready after collection restart`, async () => { + const collection = createCollection<{ id: string }>({ + id: `eager-subscription-restart`, + getKey: ({ id }) => id, + syncMode: `eager`, + sync: { sync: ({ markReady }) => markReady() }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot() + + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + + expect(collection.status).toBe(`ready`) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`retires restart loading when the replacement sync fails`, async () => { + const syncFailure = new Error(`replacement sync failed`) + let session = 0 + const collection = createOnDemandCollection<{ id: string }>({ + id: `failed-sync-restart`, + sync: { + sync: ({ markReady }) => { + if (session++ > 0) throw syncFailure + markReady() + return { loadSubset: () => true } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot() + + await collection.cleanup() + expect(() => collection.startSyncImmediate()).toThrow(syncFailure) + await flushPromises() + + expect(collection.status).toBe(`error`) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`retires failed physical release with its source session cleanup`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const releaseFailure = new Error(`release failed`) + let unloads = 0 + let sourceCleanups = 0 + const collection = createOnDemandCollection<{ id: string }>({ + id: `cleanup-failed-release`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloads++ + if (unloads === 1) throw releaseFailure + }, + cleanup: () => { + sourceCleanups++ + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + expect(() => subscription.releaseSnapshot(where)).toThrow(releaseFailure) + + await collection.cleanup() + expect(unloads).toBe(1) + expect(sourceCleanups).toBe(1) + observeSourceSessionBoundary(`active-cleanup`) + + await collection.cleanup() + expect(unloads).toBe(1) + expect(sourceCleanups).toBe(1) + observePhysicalInteraction(`failed-release:cleanup`, `no-repeat-on-cleanup`) + subscription.unsubscribe() + }) + + it.each( + ([`return`, `throw`] as const).flatMap((outcome) => + [false, true].map((releaseSelf) => ({ outcome, releaseSelf })), + ), + )( + `retires only the acquired lease for aborted replay with unload=$outcome, releaseSelf=$releaseSelf`, + async ({ outcome, releaseSelf }) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const controller = new AbortController() + const failure = new Error(`release failed`) + const loads: Array = [] + const unloads: Array = [] + const errors: Array = [] + let releaseOwner = () => {} + let operations!: Parameters[`sync`]>[0] + const collection = createCollection<{ id: string }, string>({ + id: `aborted-replay-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (nextOperations) => { + operations = nextOperations + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (unloads.length === 1) { + if (releaseSelf) releaseOwner() + if (outcome === `throw`) throw failure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + releaseOwner = () => subscription.releaseSnapshot(where) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + try { + subscription.requestSnapshot({ where, signal: controller.signal }) + controller.abort() + operations.begin() + operations.truncate() + const receipt = operations.commit() + if (receipt !== true) await receipt + await flushPromises() + + expect(loads).toHaveLength(1) + expect(unloads).toHaveLength(1) + expect(unloads[0]).toBe(loads[0]) + expect(loads[0]!.signal!.aborted).toBe(true) + expect(subscription.status).toBe(`ready`) + expect(errors).toHaveLength(outcome === `throw` ? 1 : 0) + if (outcome === `throw`) expect(errors[0]).toBe(failure) + + releaseOwner() + expect(unloads).toHaveLength(1) + subscription.unsubscribe() + expect(unloads).toHaveLength(1) + for (const options of unloads) expect(options).toBe(loads[0]) + expect(loads).toHaveLength(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each( + ([`adapter`, `error-listener`] as const).flatMap((reentry) => + [1, 2].map((failures) => ({ reentry, failures })), + ), + )( + `attempts release once across $reentry reentry with $failures configured failures`, + async ({ reentry, failures }) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const failure = new Error(`physical release failed`) + const loads: Array = [] + const unloads: Array = [] + const errors: Array = [] + const nestedFailures: Array = [] + let releaseOwner = () => {} + const collection = createOnDemandCollection<{ id: string }>({ + id: `release-reentry-${reentry}-${failures}`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (unloads.length === 1 && reentry === `adapter`) { + releaseOwner() + } + if (unloads.length <= failures) throw failure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + releaseOwner = () => { + try { + subscription.unsubscribe() + } catch (error) { + nestedFailures.push(error) + } + } + subscription.on(`loadSubset:error`, ({ error }) => { + errors.push(error) + if (errors.length === 1 && reentry === `error-listener`) releaseOwner() + }) + try { + subscription.requestSnapshot({ where }) + let releaseError: unknown + try { + subscription.releaseSnapshot(where) + } catch (error) { + releaseError = error + } + expect(releaseError).toBe(failure) + // Both callback paths see a retired acquisition, including after throw. + expect(unloads).toHaveLength(1) + expect(collection.subscriberCount).toBe(0) + if (reentry === `error-listener`) expect(errors[0]).toBe(failure) + expect(nestedFailures).toEqual([]) + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads).toHaveLength(1) + subscription.unsubscribe() + expect(unloads).toHaveLength(1) + expect(loads).toHaveLength(1) + for (const options of unloads) expect(options).toBe(loads[0]) + } finally { + await collection.cleanup() + subscription.unsubscribe() + } + }, + ) + + it(`keeps failed releases out of truncate replay`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const releaseFailure = new Error(`release failed`) + let unloads = 0 + let operations!: Parameters[`sync`]>[0] + const collection = createCollection<{ id: string }, string>({ + id: `truncate-failed-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (nextOperations) => { + operations = nextOperations + operations.markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloads++ + if (unloads === 1) throw releaseFailure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + expect(() => subscription.releaseSnapshot(where)).toThrow(releaseFailure) + + operations.begin() + operations.truncate() + const receipt = operations.commit() + if (receipt !== true) await receipt + expect(unloads).toBe(1) + + subscription.unsubscribe() + expect(unloads).toBe(1) + observePhysicalInteraction( + `failed-release:truncate`, + `no-repeat-on-truncate`, + ) + await collection.cleanup() + }) + + it(`does not repeat failed cleanup through a replacement adapter session`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + let syncSession = 0 + const unloadSessions: Array = [] + const releaseFailure = new Error(`old session release failed`) + const collection = createOnDemandCollection<{ id: string }>({ + id: `cleanup-debt-session`, + sync: { + sync: ({ markReady }) => { + const session = syncSession++ + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloadSessions.push(session) + if (session === 0) throw releaseFailure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + expect(() => subscription.releaseSnapshot(where)).toThrow(releaseFailure) + + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + subscription.unsubscribe() + + expect(unloadSessions).toEqual([0]) + await collection.cleanup() + }) + + it.each(restartScenarios)( + `keeps restart ownership aligned for $outcome × $reentry`, + async ({ outcome, reentry }) => { + type DemandName = `target` | `peer` + const targetWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`target`), + ]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`peer`)]) + const demandForWhere = new Map([ + [targetWhere, `target`], + [peerWhere, `peer`], + ]) + const failure = new Error(`restart acquisition failed`) + const pending = createDeferred() + void pending.promise.catch(() => {}) + const loads: Array<{ session: number; demand: DemandName }> = [] + const unloads: Array<{ session: number; demand: DemandName }> = [] + const sourceCleanups: Array = [] + const errors: Array = [] + let session = -1 + let ranReentry = false + + const collection = createOnDemandCollection<{ id: string }>({ + id: `restart-${outcome}-${reentry}`, + sync: { + sync: ({ markReady }) => { + session++ + const adapterSession = session + markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown restart demand`) + loads.push({ session: adapterSession, demand }) + if (adapterSession === 0 || demand === `peer`) return true + if (!ranReentry) { + ranReentry = true + if (reentry === `release-self`) { + subscription.releaseSnapshot(targetWhere) + } else if (reentry === `release-peer`) { + subscription.releaseSnapshot(peerWhere) + } else if (reentry === `unsubscribe`) { + subscription.unsubscribe() + } else if (reentry === `cleanup`) { + void collection.cleanup() + } + } + if (outcome === `throw`) throw failure + if (outcome === `return`) return true + return pending.promise + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown restart demand`) + unloads.push({ session: adapterSession, demand }) + }, + cleanup: () => sourceCleanups.push(adapterSession), + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.requestSnapshot({ where: targetWhere }) + subscription.requestSnapshot({ where: peerWhere }) + + await collection.cleanup() + expect(sourceCleanups).toEqual([0]) + observePhysicalInteraction(`active:cleanup`, `retire`) + collection.startSyncImmediate() + observeSourceSessionBoundary(`restart-installed`) + await flushPromises() + if (outcome === `resolve`) pending.resolve() + if (outcome === `reject`) pending.reject(failure) + await flushPromises() + + const targetEstablished = outcome !== `throw` && reentry !== `cleanup` + const targetSurvives = reentry === `none` || reentry === `release-peer` + const peerStarts = + reentry !== `release-peer` && + reentry !== `unsubscribe` && + reentry !== `cleanup` + expect(loads).toEqual([ + { session: 0, demand: `target` }, + { session: 0, demand: `peer` }, + { session: 1, demand: `target` }, + ...(peerStarts ? [{ session: 1, demand: `peer` as const }] : []), + ]) + expect(errors).toEqual( + (outcome === `throw` || outcome === `reject`) && targetSurvives + ? [failure] + : [], + ) + + if (reentry !== `unsubscribe`) subscription.unsubscribe() + expect(unloads).toEqual([ + ...(targetEstablished && !targetSurvives + ? [{ session: 1, demand: `target` as const }] + : []), + ...(targetEstablished && targetSurvives + ? [{ session: 1, demand: `target` as const }] + : []), + ...(peerStarts ? [{ session: 1, demand: `peer` as const }] : []), + ]) + await collection.cleanup() + expect(sourceCleanups).toEqual([0, 1]) + }, + ) + + it.each( + ([`initial`, `replay`] as const).flatMap((origin) => + ([`resolve`, `reject`] as const).flatMap((oldOutcome) => + ([`old-first`, `current-first`] as const).map((order) => ({ + origin, + oldOutcome, + order, + })), + ), + ), + )( + `separates publication from readiness for pending $origin work, $oldOutcome, $order`, + async ({ origin, oldOutcome, order }) => { + type Row = { id: string; version: number } + const where = { + a: new Func(`eq`, [new PropRef([`id`]), new Value(`a`)]), + b: new Func(`eq`, [new PropRef([`id`]), new Value(`b`)]), + } + const loads: Array<{ + options: LoadSubsetOptions + deferred: ReturnType> + }> = [] + const unloads: Array = [] + const errors: Array = [] + const visible = new Map() + let emptyBatches = 0 + let replacementChanges = 0 + let operations!: Parameters[`sync`]>[0] + const collection = createCollection({ + id: `publication-readiness-boundary`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (nextOperations) => { + operations = nextOperations + operations.markReady() + return { + loadSubset: (options) => { + const deferred = createDeferred() + void deferred.promise.catch(() => {}) + loads.push({ options, deferred }) + return deferred.promise + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + if (changes.length === 0) emptyBatches++ + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + if (change.value.id === `b` && change.value.version === 2) + replacementChanges++ + } + } + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + const write = async (id: string, version: number) => { + operations.begin() + operations.write({ + type: collection.has(id) ? `update` : `insert`, + value: { id, version }, + }) + const receipt = operations.commit() + if (receipt !== true) await receipt + } + const truncate = async () => { + operations.begin() + operations.truncate() + const receipt = operations.commit() + if (receipt !== true) await receipt + await flushPromises() + } + const settle = async ( + attempt: (typeof loads)[number], + outcome: `resolve` | `reject`, + id = `b`, + ) => { + // The source cannot cancel transport promptly, but must suppress its + // canceled writes. Late settlement does not install an obsolete row. + if (outcome === `resolve`) { + if (!attempt.options.signal?.aborted) await write(id, 2) + attempt.deferred.resolve() + } else attempt.deferred.reject(new Error(`obsolete source failed`)) + await flushPromises() + } + try { + subscription.requestSnapshot({ where: where.b }) + await write(`b`, 0) + if (origin === `replay`) { + loads[0]!.deferred.resolve() + await flushPromises() + await truncate() + } + const old = loads.at(-1)! + await truncate() + const current = loads.at(-1)! + expect(old.options.signal?.aborted).toBe(true) + expect([...visible.values()]).toEqual([{ id: `b`, version: 0 }]) + if (order === `old-first`) { + await settle(old, oldOutcome) + expect(subscription.status).toBe(`loadingSubset`) + expect([...visible.values()]).toEqual([{ id: `b`, version: 0 }]) + } + await settle(current, `resolve`) + const privateReplay = origin === `replay` && order === `current-first` + expect([...visible.values()]).toEqual([ + { id: `b`, version: privateReplay ? 0 : 2 }, + ]) + expect(subscription.status).toBe( + order === `old-first` ? `ready` : `loadingSubset`, + ) + const emptyBeforeRequest = emptyBatches + subscription.requestSnapshot({ where: where.a }) + expect(emptyBatches - emptyBeforeRequest).toBe(privateReplay ? 0 : 1) + await settle(loads.at(-1)!, `resolve`, `a`) + expect([...visible.values()]).toEqual( + privateReplay + ? [{ id: `b`, version: 0 }] + : [ + { id: `b`, version: 2 }, + { id: `a`, version: 2 }, + ], + ) + if (order === `current-first`) await settle(old, oldOutcome) + expect([...visible.values()]).toEqual([ + { id: `b`, version: 2 }, + { id: `a`, version: 2 }, + ]) + expect(subscription.status).toBe(`ready`) + expect(errors).toEqual([]) + expect(replacementChanges).toBe(1) + subscription.unsubscribe() + expect(unloads).toHaveLength(loads.length) + for (const { options } of loads) + expect(unloads.filter((value) => value === options)).toHaveLength(1) + } finally { + for (const { deferred } of loads) deferred.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each(threeGenerationScenarios)( + `fences three generations for $obsoleteOutcome/$currentOutcome settled $settlementOrder`, + async ({ obsoleteOutcome, currentOutcome, settlementOrder }) => { + type Row = { id: string; version: number } + const obsolete = createDeferred() + const current = createDeferred() + void obsolete.promise.catch(() => {}) + void current.promise.catch(() => {}) + const obsoleteFailure = new Error(`obsolete generation failed`) + const currentFailure = new Error(`current generation failed`) + const visible = new Map() + const errors: Array = [] + const unloadSessions: Array = [] + let session = -1 + + const collection = createOnDemandCollection({ + id: `three-generation-${obsoleteOutcome}-${currentOutcome}-${settlementOrder}`, + sync: { + sync: (operations) => { + session++ + const ownSession = session + operations.markReady() + return { + loadSubset: (options) => { + if (ownSession === 0) { + operations.begin() + operations.write({ + type: `insert`, + value: { id: `row`, version: 1 }, + }) + operations.commit(options.signal) + return true + } + const gate = ownSession === 1 ? obsolete : current + const outcome = + ownSession === 1 ? obsoleteOutcome : currentOutcome + const failure = + ownSession === 1 ? obsoleteFailure : currentFailure + return gate.promise.then(() => { + if (outcome === `reject`) throw failure + operations.begin() + operations.write({ + type: `insert`, + value: { id: `row`, version: ownSession + 1 }, + }) + const receipt = operations.commit(options.signal) + if (receipt !== true) return receipt + return undefined + }) + }, + unloadSubset: () => unloadSessions.push(ownSession), + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.requestSnapshot() + expect([...visible.values()]).toEqual([{ id: `row`, version: 1 }]) + + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + expect(subscription.status).toBe(`loadingSubset`) + + const settleObsolete = () => + obsoleteOutcome === `resolve` + ? obsolete.resolve() + : obsolete.reject(obsoleteFailure) + const settleCurrent = () => + currentOutcome === `resolve` + ? current.resolve() + : current.reject(currentFailure) + if (settlementOrder === `obsolete-first`) { + settleObsolete() + await flushPromises() + expect(subscription.status).toBe(`loadingSubset`) + settleCurrent() + } else { + settleCurrent() + await flushPromises() + settleObsolete() + } + await flushPromises() + + expect([...visible.values()]).toEqual([ + currentOutcome === `resolve` + ? { id: `row`, version: 3 } + : { id: `row`, version: 1 }, + ]) + expect(errors).toEqual( + currentOutcome === `reject` ? [currentFailure] : [], + ) + expect(subscription.lastError).toBe( + currentOutcome === `reject` ? currentFailure : undefined, + ) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + expect(unloadSessions).toEqual([2]) + await collection.cleanup() + }, + ) + + it(`treats an externally aborted replay as failed without publishing partial rows`, async () => { + type Row = { id: string; value: string } + const abort = new AbortController() + const replay = createDeferred() + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const visible = new Map() + const collection = createOnDemandCollection({ + id: `externally-aborted-replay`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount++ + begin() + write({ + type: `insert`, + value: { + id: `row`, + value: loadCount === 1 ? `old` : `partial`, + }, + }) + commit() + return loadCount === 1 ? true : replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + value: change.value.value, + }) + } + } + }, + { includeInitialState: false }, + ) + + subscription.requestSnapshot({ signal: abort.signal }) + expect([...visible.values()]).toEqual([{ id: `row`, value: `old` }]) + begin() + truncate() + commit() + await flushPromises() + abort.abort() + replay.reject(new DOMException(`aborted`, `AbortError`)) + await flushPromises() + + expect([...visible.values()]).toEqual([{ id: `row`, value: `old` }]) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`enters loading status when a truncate queues replay work`, async () => { + const replay = createDeferred() + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const loads: Array = [] + const unloads: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `queued-replay-status`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return loads.length === 1 ? true : replay.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + const original = loads[0]! + + begin() + truncate() + commit() + observePhysicalInteraction(`active:truncate`, `retire`) + + expect(loads).toHaveLength(1) + expect(subscription.status).toBe(`loadingSubset`) + + await flushPromises() + const replacement = loads[1]! + expect(loads).toHaveLength(2) + expect(original.signal?.aborted).toBe(true) + expect(unloads).toEqual([original]) + expect(replacement).not.toBe(original) + expect(replacement.where).toBe(where) + expect(replacement.signal?.aborted).toBe(false) + replay.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + expect(unloads).toEqual([original, replacement]) + await collection.cleanup() + }) + + it.each(startOutcomes)( + `retires replay setup when its adapter cleans up before %s`, + async (outcome) => { + type Row = { id: string; version: number } + const pending = createDeferred() + void pending.promise.catch(() => {}) + const failure = new Error(`obsolete replay failed`) + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const visible = new Map() + const errors: Array = [] + const statuses: Array = [] + + const collection = createOnDemandCollection({ + id: `reentrant-cleanup-${outcome}`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount++ + begin() + write({ + type: `insert`, + value: { id: `row`, version: loadCount }, + }) + commit() + if (loadCount === 1) return true + void collection.cleanup() + if (outcome === `throw`) throw failure + if (outcome === `return`) return true + return pending.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + subscription.requestSnapshot() + + begin() + truncate() + commit() + await flushPromises() + if (outcome === `resolve`) pending.resolve() + if (outcome === `reject`) pending.reject(failure) + await flushPromises() + + expect(collection.status).toBe(`cleaned-up`) + expect([...visible.values()]).toEqual([{ id: `row`, version: 1 }]) + expect(errors).toEqual([]) + expect(subscription.lastError).toBeUndefined() + expect(subscription.status).toBe(`ready`) + expect(statuses.at(-1)).not.toBe(`loadingSubset`) + + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + const { multiplier, ...replay } = readOracleRunConfig() + + fcTest.prop([asyncRestartScenarioArbitrary], { + numRuns: 30 * multiplier, + seed: 1_657_002, + })( + `fences async demand settlements across restart generations for a fixed seed`, + async (scenario) => { + await runAsyncRestartScenario(scenario) + }, + 120_000, + ) + + fcTest.prop( + [asyncRestartScenarioArbitrary], + oracleRandomParameters( + 30 * multiplier, + replay, + `subscription-lifecycle.async-restart`, + ), + )( + `fences async demand settlements across restart generations for a random or replayed seed`, + async (scenario) => { + await runAsyncRestartScenario(scenario) + }, + 120_000, + ) +}) diff --git a/packages/db/tests/collection-subscription-lifecycle-publication.property.test.ts b/packages/db/tests/collection-subscription-lifecycle-publication.property.test.ts new file mode 100644 index 0000000000..c2c2aebfad --- /dev/null +++ b/packages/db/tests/collection-subscription-lifecycle-publication.property.test.ts @@ -0,0 +1,1878 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { + createLifecycleModel, + greenLifecycleHistories, + greenLifecycleHistoryArbitrary, + reduceLifecycle, +} from './collection-subscription-lifecycle-grammar.js' +import { oracleRandomParameters, readOracleRunConfig } from './oracle-config.js' +import { flushPromises } from './utils.js' +import type { SyncConfig } from '../src/types.js' +import type { + DemandName, + LifecycleAttempt, + LifecycleCommand, + LifecycleEffect, + LifecycleModel, +} from './collection-subscription-lifecycle-grammar.js' + +type RowKey = DemandName | `c` | `d` +type Row = { id: RowKey; value: number } +type PublicationChange = { + type: `insert` | `update` | `delete` + key: RowKey + value: Row + previousValue?: Row +} +type SourceMutation = { + type: `source` + key: RowKey + action: `upsert` | `delete` + value: number +} +type PublicationCommand = + | Exclude + | { type: `truncate`; replacement?: Row } + | SourceMutation +type SyncOperations = Parameters[`sync`]>[0] +type RuntimeAttempt = { + id: number + ownerId: number + demand: DemandName + session: number + operations: SyncOperations + deferred: ReturnType> + signal: AbortSignal | undefined + settled: boolean + current: boolean +} +type RuntimeOwner = { + id: number + demand: DemandName + controller: AbortController + aborted: boolean + attemptId?: number +} +type Replacement = { + session: number + replay: number + rows: Map + failed: boolean +} +type PublicationModel = { + source: Map + visible: Map + // Public rows carried across a discarded source/replay, not yet refreshed. + retainedKeys: Set + replacement?: Replacement + batches: Array> + sentKeys: Set +} +type PublicationPhase = + | `public` + | `private-pending` + | `private-settling` + | `private-failed` +type SourceEffect = `insert` | `update` | `delete` +type PublicationObservation = { + index: number + command: PublicationCommand[`type`] + phaseBefore: PublicationPhase + pendingAttemptsBefore: number + executed: boolean + sourceEffect?: SourceEffect + settlement?: `resolve` | `reject` + publications: number + unloads: number + sessions: number + collectionStatus: string +} +type PublicationMismatch = { + history: string + commandIndex: number + command: PublicationCommand + expected: Array> + observed: Array> +} +type PublicationRunOptions = { + withoutLoader?: boolean + continueAfterMismatch?: boolean + historyName?: string + mismatches?: Array +} + +function recordSourceWrite(publication: PublicationModel, row: Row): void { + // This driver requests raw future changes (includeInitialState: false). + // After retiring private work, an unseen source row can still be updated. + // Retained public rows take precedence when reconciling a stale snapshot. + const previous = + publication.visible.get(row.id) ?? publication.source.get(row.id) + publication.source.set(row.id, cloneRow(row)) + publication.retainedKeys.delete(row.id) + if (previous?.value === row.value) return + publication.visible.set(row.id, cloneRow(row)) + publication.sentKeys.add(row.id) + publication.batches.push([ + previous + ? { + type: `update`, + key: row.id, + value: cloneRow(row), + previousValue: cloneRow(previous), + } + : { type: `insert`, key: row.id, value: cloneRow(row) }, + ]) +} + +const mapsEqual = ( + left: ReadonlyMap, + right: ReadonlyMap, +): boolean => + left.size === right.size && + [...left].every(([id, row]) => right.get(id)?.value === row.value) + +function cloneRow(row: Row): Row { + return { id: row.id, value: row.value } +} + +function clonePublicationBatches( + batches: ReadonlyArray>, +): Array> { + return batches.map((batch) => + batch.map((change) => ({ + ...change, + value: cloneRow(change.value), + ...(change.previousValue + ? { previousValue: cloneRow(change.previousValue) } + : {}), + })), + ) +} + +function normalizePublicationOrder( + batches: ReadonlyArray>, +): Array> { + // Distinct keys have no canonical delivery order within one callback. + // Keep callback boundaries and stable order among changes to the same key. + return clonePublicationBatches(batches).map((batch) => + batch.sort((left, right) => left.key.localeCompare(right.key)), + ) +} + +function publicationPhase( + publication: PublicationModel, + lifecycle: LifecycleModel, +): PublicationPhase { + if (!publication.replacement) return `public` + if (publication.replacement.failed) return `private-failed` + const currentAttemptIds = new Set( + lifecycle.owners.flatMap(({ aborted, attemptId }) => + aborted || attemptId === undefined ? [] : [attemptId], + ), + ) + return lifecycle.attempts.some( + ({ id, settled }) => currentAttemptIds.has(id) && settled, + ) + ? `private-settling` + : `private-pending` +} + +function publicationDiff( + previous: ReadonlyMap, + next: ReadonlyMap, +): Array { + const changes: Array = [] + for (const [key, previousValue] of [...previous].sort(([left], [right]) => + left.localeCompare(right), + )) { + const value = next.get(key) + if (!value) { + changes.push({ type: `delete`, key, value: cloneRow(previousValue) }) + } else if (value.value !== previousValue.value) { + changes.push({ + type: `update`, + key, + value: cloneRow(value), + previousValue: cloneRow(previousValue), + }) + } + } + for (const [key, value] of [...next].sort(([left], [right]) => + left.localeCompare(right), + )) { + if (!previous.has(key)) { + changes.push({ type: `insert`, key, value: cloneRow(value) }) + } + } + return changes +} + +function publishIfChanged( + publication: PublicationModel, + next: Map, +): void { + if (mapsEqual(publication.visible, next)) return + publication.batches.push(publicationDiff(publication.visible, next)) + publication.visible = next + publication.sentKeys = new Set(next.keys()) +} + +function finishReplacement( + publication: PublicationModel, + lifecycle: LifecycleModel, +): void { + const replacement = publication.replacement + if (!replacement) return + const currentAttempts = lifecycle.owners.flatMap(({ aborted, attemptId }) => + aborted || attemptId === undefined ? [] : [lifecycle.attempts[attemptId]!], + ) + replacement.failed = currentAttempts.some( + ({ outcome }) => outcome === `reject`, + ) + if (lifecycle.publicationBarrierOpen) return + if (replacement.failed) { + replacement.failed = true + if (currentAttempts.length === 0) { + publication.replacement = undefined + } + } else if ( + // A canceled-only reset can establish an empty replacement once older + // transports settle. Releasing every owner instead retires the work. + lifecycle.owners.length > 0 && + currentAttempts.every(({ outcome }) => outcome === `resolve`) + ) { + publishIfChanged(publication, new Map(replacement.rows)) + publication.retainedKeys.clear() + publication.replacement = undefined + } else { + // Retire private publication work, not the source's independently applied state. + publication.replacement = undefined + publication.retainedKeys = new Set(publication.visible.keys()) + } +} + +function projectPublication( + publication: PublicationModel, + lifecycle: LifecycleModel, + command: PublicationCommand, + effect: LifecycleEffect, + priorPublicationCount: number, + eagerRestart: boolean, +): void { + if ( + command.type === `truncate` && + lifecycle.active && + !lifecycle.unsubscribed + ) { + const previousSource = new Map(publication.source) + publication.source.clear() + if (command.replacement) { + publication.source.set( + command.replacement.id, + cloneRow(command.replacement), + ) + } + if (lifecycle.publicationBarrierOpen) { + publication.replacement = { + session: lifecycle.session, + replay: lifecycle.replay, + rows: new Map(publication.source), + failed: false, + } + } else { + publication.replacement = undefined + if ( + publication.retainedKeys.size === 0 && + (eagerRestart || lifecycle.owners.length === 0) + ) { + // Without held publications or replay demand this is a raw source + // transaction: all old source rows are deleted, even if never shown. + // A same-key replacement keeps its delete/insert pair in one callback. + const changes: Array = [ + ...[...previousSource].map(([key, value]) => ({ + type: `delete` as const, + key, + value: cloneRow(value), + })), + ...[...publication.source].map(([key, value]) => ({ + type: `insert` as const, + key, + value: cloneRow(value), + })), + ] + if (changes.length > 0) publication.batches.push(changes) + publication.visible = new Map(publication.source) + publication.sentKeys = new Set(publication.source.keys()) + } else { + // Held publications need a replacement diff, not raw source deletes. + publishIfChanged(publication, new Map(publication.source)) + } + publication.retainedKeys.clear() + } + } else if ( + command.type === `restart` && + lifecycle.publications > priorPublicationCount && + lifecycle.publicationBarrierOpen + ) { + publication.replacement = { + session: lifecycle.session, + replay: lifecycle.replay, + rows: new Map(), + failed: false, + } + } else if ( + command.type === `source` && + lifecycle.active && + !lifecycle.unsubscribed + ) { + const previousValue = publication.source.get(command.key) + if (command.action === `delete`) { + publication.source.delete(command.key) + publication.replacement?.rows.delete(command.key) + if (!publication.replacement && previousValue) { + const deletedValue = publication.retainedKeys.has(command.key) + ? (publication.visible.get(command.key) ?? previousValue) + : previousValue + publication.visible.delete(command.key) + publication.retainedKeys.delete(command.key) + publication.sentKeys.delete(command.key) + publication.batches.push([ + { + type: `delete`, + key: command.key, + value: cloneRow(deletedValue), + }, + ]) + } + } else { + const row = { + id: command.key, + value: command.value, + } + if (publication.replacement) { + publication.source.set(command.key, row) + publication.replacement.rows.set(command.key, row) + } else { + recordSourceWrite(publication, row) + } + } + } else if (command.type === `cleanup`) { + publication.retainedKeys = new Set(publication.visible.keys()) + publication.source.clear() + publication.replacement = undefined + } else if (command.type === `release`) { + // Release changes demand, not source retention. Only source writes or a + // successful replacement can change the subscriber's rows. + finishReplacement(publication, lifecycle) + } else if (command.type === `settle` && effect.attemptId !== undefined) { + const attempt = lifecycle.attempts[effect.attemptId]! + const isCurrent = lifecycle.owners.some( + ({ attemptId }) => attemptId === attempt.id, + ) + if (command.outcome === `resolve` && isCurrent && !attempt.aborted) { + const row = { id: attempt.demand, value: attempt.id } + const replacement = publication.replacement + if ( + replacement && + replacement.session === attempt.session && + replacement.replay === attempt.replay + ) { + publication.source.set(row.id, row) + replacement.rows.set(row.id, row) + } else { + recordSourceWrite(publication, row) + } + } + finishReplacement(publication, lifecycle) + } + + // This fixture marks an eager restart ready with its complete (empty) source. + // Unlike on-demand restart, that is authority to retire the retained snapshot. + if ( + eagerRestart && + command.type === `restart` && + lifecycle.publications > priorPublicationCount && + !mapsEqual(publication.visible, publication.source) + ) { + publishIfChanged(publication, new Map(publication.source)) + publication.retainedKeys.clear() + priorPublicationCount++ + } + + for ( + let index = priorPublicationCount; + index < lifecycle.publications; + index++ + ) { + const row = + command.type === `request` && + lifecycle.active && + lifecycle.owners.filter(({ demand }) => demand === command.demand) + .length === 1 && + !publication.sentKeys.has(command.demand) + ? publication.source.get(command.demand) + : undefined + publication.batches.push( + row + ? [ + { + type: `insert`, + key: row.id, + value: cloneRow(row), + }, + ] + : [], + ) + if (row) { + publication.visible.set(row.id, cloneRow(row)) + publication.retainedKeys.delete(row.id) + publication.sentKeys.add(row.id) + } + } +} + +const sourceMutationArbitrary: fc.Arbitrary = fc.record({ + type: fc.constant(`source` as const), + key: fc.constantFrom(`a` as const, `b` as const, `c` as const, `d` as const), + action: fc.constantFrom(`upsert` as const, `delete` as const), + value: fc.integer({ min: 0, max: 5 }), +}) + +const publicationCommandHistoryArbitrary: fc.Arbitrary< + Array +> = greenLifecycleHistoryArbitrary.chain((history) => + fc + .array( + fc.record({ + position: fc.integer({ min: 0, max: history.length }), + command: sourceMutationArbitrary, + }), + { minLength: 1, maxLength: 5 }, + ) + .map((insertions) => { + const commands: Array = [...history] + for (const { position, command } of insertions.sort( + (left, right) => right.position - left.position, + )) { + commands.splice(position, 0, command) + } + return commands + }), +) + +async function runPublicationHistory( + history: ReadonlyArray, + runOptions: PublicationRunOptions = {}, +): Promise> { + const check = runOptions.continueAfterMismatch ? expect.soft : expect + const observations: Array = [] + const lifecycle = createLifecycleModel() + const publication: PublicationModel = { + source: new Map(), + visible: new Map(), + retainedKeys: new Set(), + batches: [], + sentKeys: new Set(), + } + const where = { + a: new Func(`eq`, [new PropRef([`id`]), new Value(`a`)]), + b: new Func(`eq`, [new PropRef([`id`]), new Value(`b`)]), + } + const demandForWhere = new Map([ + [where.a, `a`], + [where.b, `b`], + ]) + const attempts = new Map() + const attemptForOptions = new Map() + const unloads: Array = [] + const owners: Array = [] + const sourceRows = new Map>() + const operationsBySession = new Map() + let nextAttemptId = 0 + let nextOwnerId = 0 + let session = -1 + let active = true + let unsubscribed = false + + const collection = createCollection({ + id: `generated-lifecycle-publication`, + getKey: ({ id }) => id, + syncMode: runOptions.withoutLoader ? `eager` : `on-demand`, + sync: { + sync: (operations) => { + const ownSession = ++session + operationsBySession.set(ownSession, operations) + sourceRows.set(ownSession, new Map()) + operations.markReady() + if (runOptions.withoutLoader) return + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`publication load lost its demand`) + const owner = owners.find( + (candidate) => + candidate.demand === demand && + !candidate.aborted && + candidate.attemptId === undefined, + ) + if (!owner) throw new Error(`publication load has no runtime owner`) + const id = nextAttemptId++ + const deferred = createDeferred() + void deferred.promise.catch(() => undefined) + attempts.set(id, { + id, + ownerId: owner.id, + demand, + session: ownSession, + operations, + deferred, + signal: options.signal, + settled: false, + current: true, + }) + attemptForOptions.set(options, id) + owner.attemptId = id + return deferred.promise + }, + unloadSubset: (options) => { + const attemptId = attemptForOptions.get(options) + if (attemptId === undefined) { + throw new Error(`publication unload lost its acquisition`) + } + unloads.push(attemptId) + }, + } + }, + }, + }) + + const visible = new Map() + const observedBatches: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => { + const batch = changes.map((change): PublicationChange => { + const key = change.key + if (key !== `a` && key !== `b` && key !== `c` && key !== `d`) { + throw new Error(`publication used an unknown row key`) + } + return { + type: change.type, + key, + value: cloneRow(change.value), + ...(change.previousValue === undefined + ? {} + : { previousValue: cloneRow(change.previousValue) }), + } + }) + for (const change of batch) { + const id = change.key + if (change.type === `delete`) visible.delete(id) + else visible.set(id, { id, value: change.value.value }) + } + observedBatches.push(batch) + }, + { includeInitialState: false }, + ) + + const writeAttempt = async (attempt: RuntimeAttempt): Promise => { + // Cancellation fences request-scoped writes at the adapter boundary. + // Transport may settle later; it must not publish canceled snapshot rows. + if (attempt.signal?.aborted) return + const rows = sourceRows.get(attempt.session) + const previous = rows?.get(attempt.demand) + const value = { id: attempt.demand, value: attempt.id } + attempt.operations.begin() + attempt.operations.write({ + type: previous ? `update` : `insert`, + value, + ...(previous ? { previousValue: previous } : {}), + }) + const receipt = attempt.operations.commit() + if (receipt !== true) await receipt + rows?.set(attempt.demand, value) + } + + const assertPublications = ( + command: PublicationCommand, + commandIndex: number, + expectedStart: number, + observedStart: number, + ): void => { + const expectedBatches = publication.batches.slice(expectedStart) + const observed = observedBatches.slice(observedStart) + const expected = normalizePublicationOrder(expectedBatches) + const normalizedObserved = normalizePublicationOrder(observed) + const context = JSON.stringify({ + history, + command, + commandIndex, + observed, + expected: expectedBatches, + }) + if ( + runOptions.mismatches && + JSON.stringify(normalizedObserved) !== JSON.stringify(expected) + ) { + const historyName = runOptions.historyName ?? JSON.stringify(history) + runOptions.mismatches.push({ + history: historyName, + commandIndex, + command, + expected: clonePublicationBatches(expectedBatches), + observed: clonePublicationBatches(observed), + }) + return + } + check(normalizedObserved, context).toEqual(expected) + } + + const selectRuntimeAttempt = ( + command: Extract, + ): RuntimeAttempt | undefined => { + if (unsubscribed) return undefined + const candidates = [...attempts.values()].filter( + (attempt) => + !attempt.settled && + attempt.demand === command.demand && + attempt.current === (command.scope === `current`), + ) + return command.age === `oldest` ? candidates[0] : candidates.at(-1) + } + + try { + for (const [index, command] of history.entries()) { + const priorPublicationCount = lifecycle.publications + const phaseBefore = publicationPhase(publication, lifecycle) + const pendingAttemptsBefore = [...attempts.values()].filter( + ({ current, settled }) => current && !settled, + ).length + const observedPublicationCount = observedBatches.length + const expectedPublicationCount = publication.batches.length + const unloadCount = unloads.length + const sessionCount = operationsBySession.size + let executed = false + let sourceEffect: SourceEffect | undefined + let settlement: `resolve` | `reject` | undefined + const runtimeOwner = + command.type === `request` + ? { + id: nextOwnerId++, + demand: command.demand, + controller: new AbortController(), + aborted: false, + } + : command.type === `abort` + ? owners.find( + ({ demand, aborted }) => demand === command.demand && !aborted, + ) + : command.type === `release` + ? owners.find(({ demand }) => demand === command.demand) + : undefined + if (command.type === `request` && !unsubscribed) + owners.push(runtimeOwner!) + const runtimeAttempt = + command.type === `settle` ? selectRuntimeAttempt(command) : undefined + const effect = + command.type === `source` + ? ({} satisfies LifecycleEffect) + : reduceLifecycle(lifecycle, command) + + if (command.type === `source` && active) { + const operations = operationsBySession.get(session) + const rows = sourceRows.get(session) + const previous = rows?.get(command.key) + executed = operations !== undefined && rows !== undefined + sourceEffect = + command.action === `delete` + ? previous + ? `delete` + : undefined + : previous + ? `update` + : `insert` + operations?.begin() + if (command.action === `delete`) { + operations?.write({ type: `delete`, key: command.key }) + rows?.delete(command.key) + } else { + const value = { id: command.key, value: command.value } + operations?.write({ + type: previous ? `update` : `insert`, + value, + ...(previous ? { previousValue: previous } : {}), + }) + rows?.set(command.key, value) + } + const receipt = operations?.commit() + if (receipt !== true) await receipt + } else if (command.type === `request`) { + check(effect.ownerId).toBe(unsubscribed ? undefined : runtimeOwner?.id) + executed = runtimeOwner !== undefined && !unsubscribed + subscription.requestSnapshot({ + where: where[command.demand], + signal: runtimeOwner?.controller.signal, + }) + } else if (command.type === `abort`) { + check(effect.ownerId).toBe(runtimeOwner?.id) + executed = runtimeOwner !== undefined + if (runtimeOwner) { + runtimeOwner.aborted = true + runtimeOwner.controller.abort() + } + } else if (command.type === `release`) { + check(effect.ownerId).toBe(runtimeOwner?.id) + executed = runtimeOwner !== undefined + if (runtimeOwner) { + if (runtimeOwner.attemptId !== undefined) { + attempts.get(runtimeOwner.attemptId)!.current = false + } + owners.splice(owners.indexOf(runtimeOwner), 1) + } + subscription.releaseSnapshot(where[command.demand]) + } else if (command.type === `settle`) { + check(effect.attemptId).toBe(runtimeAttempt?.id) + executed = runtimeAttempt !== undefined + if (runtimeAttempt) settlement = command.outcome + if (effect.attemptId !== undefined && runtimeAttempt) { + runtimeAttempt.settled = true + const expected = lifecycle.attempts[ + effect.attemptId + ] as LifecycleAttempt + if (command.outcome === `resolve`) { + await writeAttempt(runtimeAttempt) + runtimeAttempt.deferred.resolve() + } else { + runtimeAttempt.deferred.reject(expected.failure) + } + } + } else if (command.type === `truncate` && active) { + executed = true + for (const owner of owners) { + if (owner.attemptId !== undefined) { + attempts.get(owner.attemptId)!.current = false + } + owner.attemptId = undefined + } + const operations = operationsBySession.get(session) + operations?.begin() + operations?.truncate() + if (command.replacement) { + operations?.write({ + type: `insert`, + value: cloneRow(command.replacement), + }) + } + const receipt = operations?.commit() + if (receipt !== true) await receipt + sourceRows.get(session)?.clear() + if (command.replacement) { + sourceRows + .get(session) + ?.set(command.replacement.id, cloneRow(command.replacement)) + } + } else if (command.type === `cleanup` && active) { + executed = true + for (const owner of owners) { + if (owner.attemptId !== undefined) { + attempts.get(owner.attemptId)!.current = false + } + owner.attemptId = undefined + } + await collection.cleanup() + active = false + } else if (command.type === `restart` && !active) { + executed = true + collection.startSyncImmediate() + active = true + } else if (command.type === `unsubscribe`) { + executed = !unsubscribed + for (const attempt of attempts.values()) attempt.current = false + subscription.unsubscribe() + unsubscribed = true + owners.length = 0 + } + + await flushPromises() + projectPublication( + publication, + lifecycle, + command, + effect, + priorPublicationCount, + runOptions.withoutLoader ?? false, + ) + // Public retention never rewrites the independently installed source. + // The publication model stops tracking source commands after unsubscribe; + // its callback-silence assertions below still cover that suffix. + if (!lifecycle.unsubscribed) + check( + [...(active ? collection.values() : [])] + .map(cloneRow) + .sort((left, right) => left.id.localeCompare(right.id)), + `source state after command ${index}: ${JSON.stringify(command)}`, + ).toEqual( + [...publication.source.values()].sort((left, right) => + left.id.localeCompare(right.id), + ), + ) + assertPublications( + command, + index, + expectedPublicationCount, + observedPublicationCount, + ) + check( + [...visible.values()].sort((left, right) => + left.id.localeCompare(right.id), + ), + `consumer state after command ${index}: ${JSON.stringify(command)}`, + ).toEqual( + [...publication.visible.values()].sort((left, right) => + left.id.localeCompare(right.id), + ), + ) + observations.push({ + index, + command: command.type, + phaseBefore, + pendingAttemptsBefore, + executed, + ...(sourceEffect ? { sourceEffect } : {}), + ...(settlement ? { settlement } : {}), + publications: observedBatches.length - observedPublicationCount, + unloads: unloads.length - unloadCount, + sessions: operationsBySession.size - sessionCount, + collectionStatus: collection.status, + }) + } + } finally { + for (const attempt of attempts.values()) attempt.deferred.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + return observations +} + +type ProductSettlement = `none` | `resolve` | `reject` +type ProductSuffix = `release` | `cleanup` | `restart` | `unsubscribe` +type PriorIndependentRow = `absent` | `present` +type PublicationProductCase = { + name: string + phase: PublicationPhase + sourceEffect: SourceEffect + settlement: ProductSettlement + suffix: ProductSuffix + priorIndependentRow: PriorIndependentRow + history: Array + focalSourceIndex: number + settlementIndex?: number + pendingProbeIndex: number + suffixIndex: number + postUnsubscribeProbeIndex: number +} + +const settleCurrent = ( + demand: DemandName, + outcome: `resolve` | `reject`, +): LifecycleCommand => ({ + type: `settle`, + demand, + scope: `current`, + age: `oldest`, + outcome, +}) + +function createPublicationProductCase( + phase: PublicationPhase, + sourceEffect: SourceEffect, + settlement: ProductSettlement, + suffix: ProductSuffix, + priorIndependentRow: PriorIndependentRow, +): PublicationProductCase { + const history: Array = [] + const push = (command: PublicationCommand): number => + history.push(command) - 1 + const hasPeer = phase === `private-settling` || phase === `private-failed` + + if (priorIndependentRow === `present`) { + push({ type: `source`, key: `d`, action: `upsert`, value: 30 }) + } + push({ type: `request`, demand: `a` }) + if (hasPeer) push({ type: `request`, demand: `b` }) + if (phase !== `public`) { + push(settleCurrent(`a`, `resolve`)) + if (hasPeer) push(settleCurrent(`b`, `resolve`)) + push({ type: `truncate` }) + if (phase === `private-settling`) { + push(settleCurrent(`b`, `resolve`)) + } else if (phase === `private-failed`) { + push(settleCurrent(`b`, `reject`)) + } + } + + if (sourceEffect !== `insert`) { + push({ type: `source`, key: `c`, action: `upsert`, value: 40 }) + } + const focalSourceIndex = push({ + type: `source`, + key: `c`, + action: sourceEffect === `delete` ? `delete` : `upsert`, + value: 41, + }) + const settlementIndex = + settlement === `none` ? undefined : push(settleCurrent(`a`, settlement)) + const pendingProbeIndex = history.length + + let suffixIndex: number + if (suffix === `release`) { + suffixIndex = push({ type: `release`, demand: `a` }) + if (hasPeer) suffixIndex = push({ type: `release`, demand: `b` }) + push({ type: `unsubscribe` }) + } else if (suffix === `cleanup`) { + suffixIndex = push({ type: `cleanup` }) + push({ type: `unsubscribe` }) + } else if (suffix === `restart`) { + push({ type: `cleanup` }) + suffixIndex = push({ type: `restart` }) + push({ type: `unsubscribe` }) + } else { + suffixIndex = push({ type: `unsubscribe` }) + } + if (suffix === `cleanup`) push({ type: `restart` }) + const postUnsubscribeProbeIndex = push({ + type: `source`, + key: `d`, + action: `upsert`, + value: 99, + }) + + return { + name: `${phase}:${sourceEffect}:${settlement}:${suffix}:prior-${priorIndependentRow}`, + phase, + sourceEffect, + settlement, + suffix, + priorIndependentRow, + history, + focalSourceIndex, + ...(settlementIndex === undefined ? {} : { settlementIndex }), + pendingProbeIndex, + suffixIndex, + postUnsubscribeProbeIndex, + } +} + +const publicationPhases = [ + `public`, + `private-pending`, + `private-settling`, + `private-failed`, +] as const +const sourceEffects = [`insert`, `update`, `delete`] as const +const productSettlements = [`none`, `resolve`, `reject`] as const +const productSuffixes = [ + `release`, + `cleanup`, + `restart`, + `unsubscribe`, +] as const +const priorIndependentRows = [`absent`, `present`] as const + +const publicationProductCases = publicationPhases.flatMap((phase) => + sourceEffects.flatMap((sourceEffect) => + productSettlements.flatMap((settlement) => + productSuffixes.flatMap((suffix) => + priorIndependentRows.map((priorIndependentRow) => + createPublicationProductCase( + phase, + sourceEffect, + settlement, + suffix, + priorIndependentRow, + ), + ), + ), + ), + ), +) + +const successfulReplacementCases = publicationProductCases.filter( + ({ phase, sourceEffect, settlement }) => + (phase === `private-pending` || phase === `private-settling`) && + sourceEffect !== `delete` && + settlement === `resolve`, +) +const replacementOrderingCases = publicationProductCases.filter( + ({ phase, sourceEffect, settlement, suffix, priorIndependentRow }) => + (phase === `private-pending` || phase === `private-settling`) && + sourceEffect === `delete` && + settlement === `resolve` && + suffix !== `release` && + priorIndependentRow === `present`, +) +const replacementRetirementCases = publicationProductCases.filter( + (scenario) => + scenario.phase !== `public` && + scenario.suffix === `release` && + !successfulReplacementCases.includes(scenario) && + !replacementOrderingCases.includes(scenario), +) +const publicationControlCases = publicationProductCases.filter( + (scenario) => + !successfulReplacementCases.includes(scenario) && + !replacementOrderingCases.includes(scenario) && + !replacementRetirementCases.includes(scenario), +) + +async function runPublicationProduct( + scenarios: ReadonlyArray, +): Promise> { + const mismatches: Array = [] + const reached = new Set() + + for (const scenario of scenarios) { + const observations = await runPublicationHistory(scenario.history, { + historyName: scenario.name, + mismatches, + }) + expect(observations).toHaveLength(scenario.history.length) + expect(observations[scenario.focalSourceIndex]).toMatchObject({ + command: `source`, + phaseBefore: scenario.phase, + executed: true, + sourceEffect: scenario.sourceEffect, + }) + if (scenario.settlementIndex === undefined) { + expect( + observations[scenario.pendingProbeIndex]!.pendingAttemptsBefore, + scenario.name, + ).toBe(1) + } else { + expect(observations[scenario.settlementIndex]).toMatchObject({ + command: `settle`, + executed: true, + settlement: scenario.settlement, + publications: + scenario.settlement === `resolve` && + scenario.phase !== `private-failed` + ? 1 + : 0, + }) + } + + const suffix = observations[scenario.suffixIndex]! + expect(suffix).toMatchObject({ + command: scenario.suffix, + executed: true, + }) + if (scenario.suffix === `release`) { + expect(suffix.unloads, scenario.name).toBe(1) + } else if (scenario.suffix === `cleanup`) { + expect(suffix.collectionStatus, scenario.name).toBe(`cleaned-up`) + } else if (scenario.suffix === `restart`) { + expect(suffix.sessions, scenario.name).toBe(1) + } else { + const ownerCount = + scenario.phase === `private-settling` || + scenario.phase === `private-failed` + ? 2 + : 1 + expect(suffix.unloads, scenario.name).toBe(ownerCount) + } + + expect(observations[scenario.postUnsubscribeProbeIndex]).toMatchObject({ + command: `source`, + executed: true, + publications: 0, + }) + reached.add(scenario.name) + } + + expect(reached).toEqual(new Set(scenarios.map(({ name }) => name))) + return mismatches +} + +function expectNoPublicationMismatches( + mismatches: ReadonlyArray, +): void { + const summary = mismatches.map( + ({ history, commandIndex, command, expected, observed }) => ({ + history, + commandIndex, + command, + expected, + observed, + }), + ) + expect( + summary, + `publication product mismatches: ${JSON.stringify(summary)}`, + ).toEqual([]) +} + +describe(`CollectionSubscription lifecycle publication oracle`, () => { + it.each( + ([undefined, false] as const).flatMap((includeInitialState) => + ([`update`, `delete`, `truncate`] as const).map((operation) => ({ + includeInitialState, + operation, + })), + ), + )( + `distinguishes unseen-row $operation with includeInitialState=$includeInitialState`, + async ({ includeInitialState, operation }) => { + let operations!: SyncOperations + const collection = createCollection({ + getKey: ({ id }) => id, + startSync: true, + sync: { + sync: (sync) => { + operations = sync + sync.begin() + sync.write({ type: `insert`, value: { id: `d`, value: 0 } }) + sync.commit() + sync.markReady() + }, + }, + }) + const changes: Array = [] + const subscription = collection.subscribeChanges( + (batch) => { + for (const change of batch) { + changes.push({ + type: change.type, + key: change.value.id, + value: cloneRow(change.value), + ...(change.previousValue + ? { previousValue: cloneRow(change.previousValue) } + : {}), + }) + } + }, + { includeInitialState }, + ) + try { + expect(changes).toEqual([]) + operations.begin() + if (operation === `truncate`) operations.truncate() + else if (operation === `delete`) + operations.write({ type: `delete`, key: `d` }) + else operations.write({ type: `update`, value: { id: `d`, value: 4 } }) + await operations.commit() + expect(changes).toEqual( + operation === `update` + ? [ + { + type: includeInitialState === false ? `update` : `insert`, + key: `d`, + value: { id: `d`, value: 4 }, + ...(includeInitialState === false + ? { previousValue: { id: `d`, value: 0 } } + : {}), + }, + ] + : includeInitialState === false + ? [{ type: `delete`, key: `d`, value: { id: `d`, value: 0 } }] + : [], + ) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`keeps raw source updates after retiring the final replay owner`, async () => { + await runPublicationHistory([ + { type: `cleanup` }, + { type: `release`, demand: `b` }, + { type: `request`, demand: `b` }, + { type: `restart` }, + { type: `source`, key: `d`, action: `upsert`, value: 0 }, + { type: `abort`, demand: `b` }, + { type: `release`, demand: `b` }, + { type: `abort`, demand: `b` }, + { type: `source`, key: `d`, action: `upsert`, value: 4 }, + { type: `release`, demand: `b` }, + { type: `restart` }, + { type: `unsubscribe` }, + ]) + }) + + it(`keeps raw truncate deletes after retiring the final replay owner`, async () => { + await runPublicationHistory([ + { + type: `settle`, + demand: `a`, + scope: `obsolete`, + age: `oldest`, + outcome: `reject`, + }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `source`, key: `a`, action: `upsert`, value: 5 }, + { type: `release`, demand: `a` }, + { type: `restart` }, + { type: `truncate` }, + ]) + }) + + it(`records requested snapshot rows before a canceled reset`, async () => { + await runPublicationHistory([ + { type: `request`, demand: `b` }, + { type: `truncate` }, + { type: `source`, key: `b`, action: `upsert`, value: 0 }, + { type: `release`, demand: `b` }, + { type: `request`, demand: `b` }, + { type: `restart` }, + { type: `abort`, demand: `a` }, + { type: `abort`, demand: `b` }, + { type: `restart` }, + { type: `truncate` }, + { type: `restart` }, + { type: `unsubscribe` }, + { type: `release`, demand: `b` }, + ]) + }) + + it(`does not invent a publication for an unchanged private row`, async () => { + await runPublicationHistory([ + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `source`, key: `a`, action: `upsert`, value: 5 }, + { type: `release`, demand: `a` }, + { type: `source`, key: `a`, action: `upsert`, value: 5 }, + { type: `unsubscribe` }, + ]) + }) + + it.each( + ([`none`, `retained`, `refreshed`, `new`] as const).flatMap((baseline) => + ([`delete`, `empty`, `same`, `other`] as const).map((reset) => ({ + baseline, + reset, + })), + ), + )( + `distinguishes raw source resets from retained replacements: $baseline/$reset`, + async ({ baseline, reset }) => { + await runPublicationHistory([ + ...(baseline === `retained` || baseline === `refreshed` + ? [{ type: `source`, key: `d`, action: `upsert`, value: 0 } as const] + : []), + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `source`, key: `a`, action: `upsert`, value: 5 }, + { type: `release`, demand: `a` }, + ...(baseline === `refreshed` || baseline === `new` + ? [{ type: `source`, key: `d`, action: `upsert`, value: 0 } as const] + : []), + reset === `delete` + ? { type: `source`, key: `a`, action: `delete`, value: 0 } + : { + type: `truncate`, + ...(reset === `same` + ? { replacement: { id: `a`, value: 5 } as const } + : reset === `other` + ? { replacement: { id: `c`, value: 6 } as const } + : {}), + }, + { type: `unsubscribe` }, + ]) + }, + ) + + it(`reconciles a repeated reset after the last replay owner aborts`, async () => { + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `request`, demand: `a` }, + { type: `truncate` }, + { type: `abort`, demand: `a` }, + { type: `truncate` }, + { + type: `settle`, + demand: `a`, + scope: `obsolete`, + age: `newest`, + outcome: `resolve`, + }, + { type: `cleanup` }, + ]) + }) + + it(`keeps independent source rows when a replay settles after redundant restart calls`, async () => { + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `truncate` }, + { type: `source`, key: `b`, action: `upsert`, value: 1 }, + { type: `restart` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `newest`, + outcome: `reject`, + }, + { type: `restart` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `truncate` }, + { type: `request`, demand: `b` }, + { type: `cleanup` }, + ]) + }) + + it.each([ + `none`, + `missing`, + `duplicate`, + `value`, + `previous-value`, + `split`, + `merge`, + `same-key-order`, + ] as const)( + `normalizes only independent change order with corruption: %s`, + (corruption) => { + const baseline: Array> = [ + [ + { + type: `update`, + key: `a`, + value: { id: `a`, value: 1 }, + previousValue: { id: `a`, value: 0 }, + }, + { type: `delete`, key: `b`, value: { id: `b`, value: 0 } }, + { type: `insert`, key: `c`, value: { id: `c`, value: 1 } }, + ], + [ + { + type: `update`, + key: `a`, + value: { id: `a`, value: 2 }, + previousValue: { id: `a`, value: 1 }, + }, + { type: `delete`, key: `a`, value: { id: `a`, value: 2 } }, + ], + ] + const permutations = [ + [0, 1, 2], + [0, 2, 1], + [1, 0, 2], + [1, 2, 0], + [2, 0, 1], + [2, 1, 0], + ] + for (const permutation of permutations) { + const candidate = clonePublicationBatches(baseline) + const changes = candidate[0]! + if (corruption === `value`) changes[0]!.value.value++ + if (corruption === `previous-value`) changes[0]!.previousValue!.value++ + candidate[0] = permutation.map((index) => changes[index]!) + if (corruption === `missing`) candidate[0].pop() + if (corruption === `duplicate`) candidate[0].push(changes[0]!) + if (corruption === `split`) + candidate.splice(1, 0, candidate[0].splice(1)) + if (corruption === `merge`) candidate.splice(0, 2, candidate.flat()) + if (corruption === `same-key-order`) candidate[1]!.reverse() + const actual = normalizePublicationOrder(candidate) + const expected = normalizePublicationOrder(baseline) + if (corruption === `none`) expect(actual).toEqual(expected) + else expect(actual).not.toEqual(expected) + } + }, + ) + + it(`defines all 288 unique row-publication lifecycle cells`, () => { + expect(publicationProductCases).toHaveLength(288) + expect(new Set(publicationProductCases.map(({ name }) => name)).size).toBe( + 288, + ) + expect(successfulReplacementCases).toHaveLength(32) + expect(replacementOrderingCases).toHaveLength(6) + expect(replacementRetirementCases).toHaveLength(46) + expect(publicationControlCases).toHaveLength(204) + }) + + it(`matches row publications for lifecycle control cells`, async () => { + expectNoPublicationMismatches( + await runPublicationProduct(publicationControlCases), + ) + }) + + it(`preserves independent source work when a successful replay publishes`, async () => { + expectNoPublicationMismatches( + await runPublicationProduct(successfulReplacementCases), + ) + }) + + it(`publishes complete replacement batches regardless of independent key order`, async () => { + expectNoPublicationMismatches( + await runPublicationProduct(replacementOrderingCases), + ) + }) + + it(`retires incomplete or failed replacement without changing independent public rows`, async () => { + expectNoPublicationMismatches( + await runPublicationProduct(replacementRetirementCases), + ) + }) + + it(`maps every canonical green lifecycle history to public rows`, async () => { + for (const history of greenLifecycleHistories) { + await runPublicationHistory(history) + } + }) + + it(`suppresses canceled source writes when a released acquisition settles`, async () => { + await runPublicationHistory( + [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `b` }, + { type: `release`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `obsolete`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `b` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`publishes an authoritative truncate after the final demand is released`, async () => { + await runPublicationHistory([ + { type: `request`, demand: `b` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `b` }, + { type: `truncate` }, + { type: `unsubscribe` }, + ]) + }) + + it(`publishes independent source changes with a successful replay`, async () => { + await runPublicationHistory( + [ + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `truncate` }, + { type: `source`, key: `b`, action: `upsert`, value: 50 }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`does not republish a row already delivered by a live change`, async () => { + await runPublicationHistory( + [ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `request`, demand: `b` }, + { type: `source`, key: `b`, action: `upsert`, value: 1 }, + { type: `request`, demand: `b` }, + { type: `release`, demand: `b` }, + { type: `release`, demand: `b` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`suppresses canceled source writes when an aborted acquisition settles`, async () => { + await runPublicationHistory( + [ + { type: `request`, demand: `a` }, + { type: `abort`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`keeps later source changes private after a failed replay`, async () => { + await runPublicationHistory([ + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `truncate` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `source`, key: `b`, action: `upsert`, value: 51 }, + ]) + }) + + it(`retires failed private replacement rows when its final owner releases`, async () => { + await runPublicationHistory( + [ + { type: `source`, key: `b`, action: `upsert`, value: 7 }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `truncate` }, + { type: `source`, key: `b`, action: `upsert`, value: 51 }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `release`, demand: `a` }, + { type: `source`, key: `b`, action: `upsert`, value: 8 }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`does not delete an independent public row when releasing a restarted demand`, async () => { + await runPublicationHistory( + [ + { type: `source`, key: `b`, action: `upsert`, value: 0 }, + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + { type: `source`, key: `b`, action: `upsert`, value: 1 }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`keeps a restarted snapshot private while canceled replay work settles`, async () => { + // Seed 2018803696, path 65:10:2:10:13:12:12:0:0:0. A canceled-only + // truncate does not discharge the earlier replay's publication wait. + await runPublicationHistory( + [ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `cleanup` }, + { type: `request`, demand: `b` }, + { type: `restart` }, + { type: `abort`, demand: `b` }, + { type: `truncate` }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { + type: `settle`, + demand: `b`, + scope: `obsolete`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`matches retained rows across an empty restart and canceled-only truncates`, async () => { + // Seed 2018803696, path 65:29:0:0:0 exposed the model's missing retained-row + // deletion: an authoritative empty reset is not limited to resident rows. + await runPublicationHistory( + [ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `release`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `request`, demand: `b` }, + { type: `restart` }, + { type: `abort`, demand: `b` }, + { type: `abort`, demand: `b` }, + { type: `truncate` }, + { type: `truncate` }, + { type: `request`, demand: `b` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { + type: `settle`, + demand: `b`, + scope: `obsolete`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `b` }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it.each( + [false, true].flatMap((restart) => + [false, true].map((canceledOwner) => ({ restart, canceledOwner })), + ), + )( + `publishes an authoritative empty reset: %j`, + async ({ restart, canceledOwner }) => { + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + ...(restart + ? [{ type: `cleanup` } as const, { type: `restart` } as const] + : []), + ...(canceledOwner + ? [ + { type: `request`, demand: `b` } as const, + { type: `abort`, demand: `b` } as const, + ] + : []), + { type: `truncate` }, + { type: `truncate` }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, + ]) + }, + ) + + it.each( + [false, true].flatMap((canceledOwner) => + ( + [ + { id: `a`, value: 0 }, + { id: `a`, value: 1 }, + { id: `c`, value: 2 }, + ] satisfies Array + ).map((replacement) => ({ canceledOwner, replacement })), + ), + )( + `installs a retained-row replacement atomically: %j`, + async ({ canceledOwner, replacement }) => { + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `cleanup` }, + { type: `restart` }, + ...(canceledOwner + ? [ + { type: `request`, demand: `b` } as const, + { type: `abort`, demand: `b` } as const, + ] + : []), + { type: `truncate`, replacement }, + { type: `source`, key: replacement.id, action: `upsert`, value: 3 }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, + ]) + }, + ) + + it.each([ + undefined, + { id: `a`, value: 0 }, + { id: `a`, value: 1 }, + { id: `c`, value: 2 }, + ] satisfies Array)( + `publishes eager restart before a later source reset: %j`, + async (replacement) => { + await runPublicationHistory( + [ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `cleanup` }, + { type: `restart` }, + { type: `truncate`, replacement }, + { type: `source`, key: `a`, action: `upsert`, value: 3 }, + { type: `unsubscribe` }, + ], + { withoutLoader: true }, + ) + }, + ) + + it(`resets retained rows after no-op cleanup and release commands`, async () => { + // Seed 1657005, path 164:18 after removing the visible-row request omission. + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 1 }, + { type: `cleanup` }, + { type: `cleanup` }, + { type: `release`, demand: `b` }, + { type: `restart` }, + { type: `truncate` }, + { type: `request`, demand: `a` }, + { type: `release`, demand: `a` }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ]) + }) + + it(`resets retained rows after the last replay owner retires`, async () => { + // Seed 333468655, path 59:13:0:0:0. + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `request`, demand: `b` }, + { type: `truncate` }, + { type: `release`, demand: `b` }, + { type: `truncate` }, + { type: `release`, demand: `a` }, + { type: `truncate` }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ]) + }) + + it(`does not restore source rows when the final replay owner retires`, async () => { + // Seed 1337491191, path 591:20:1:8:8:8:7:7. + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `restart` }, + { type: `truncate` }, + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `request`, demand: `b` }, + { type: `truncate` }, + { type: `release`, demand: `b` }, + { type: `source`, key: `a`, action: `delete`, value: 0 }, + { type: `source`, key: `a`, action: `upsert`, value: 1 }, + { type: `unsubscribe` }, + ]) + }) + + const { multiplier, ...replay } = readOracleRunConfig() + const runs = 60 * multiplier + + fcTest.prop([publicationCommandHistoryArbitrary], { + numRuns: runs, + seed: 1_657_005, + })( + `matches row publications for a fixed seed`, + async (history) => { + await runPublicationHistory(history) + }, + 120_000, + ) + fcTest.prop( + [publicationCommandHistoryArbitrary], + oracleRandomParameters( + runs, + replay, + `subscription-lifecycle.publication-history`, + ), + )( + `matches row publications for a random or replayed seed`, + async (history) => { + await runPublicationHistory(history) + }, + 120_000, + ) +}) diff --git a/packages/db/tests/collection-subscription-replay-oracle.property.test.ts b/packages/db/tests/collection-subscription-replay-oracle.property.test.ts new file mode 100644 index 0000000000..ac09a533bd --- /dev/null +++ b/packages/db/tests/collection-subscription-replay-oracle.property.test.ts @@ -0,0 +1,4571 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { ReverseIndex } from '../src/indexes/reverse-index.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { DeduplicatedLoadSubset } from '../src/query/subset-dedupe.js' +import { createTransaction } from '../src/transactions.js' +import { oracleRandomParameters, readOracleRunConfig } from './oracle-config.js' +import { flushPromises } from './utils.js' +import type { Collection } from '../src/collection/index.js' +import type { CollectionSubscription } from '../src/collection/subscription.js' +import type { OrderBy } from '../src/query/ir.js' +import type { + ChangeMessageOrDeleteKeyMessage, + LoadSubsetOptions, + SyncConfig, +} from '../src/types.js' +import type { Scheduler } from 'fast-check' + +type ReplayRow = { + id: `one` | `two` + value: number +} + +type ReplayDemandId = ReplayRow[`id`] + +type ReplayLoad = { + demandId: ReplayDemandId + rows: ReadonlyArray + outcome: `resolve` | `reject` + writeBeforeSettlement?: boolean +} + +type ReplayAttempt = { + loads: ReadonlyArray +} + +type SourceAction = + | { type: `put`; row: ReplayRow } + | { type: `delete`; id: ReplayRow[`id`] } + | { type: `request`; demandId: ReplayDemandId } + +type SourceWriteOrigin = + | { type: `initial`; demandId: ReplayDemandId } + | { type: `replay`; demandId: ReplayDemandId; attemptIndex: number } + | { type: `ordinary` } + +type SourceWrite = { + origin: SourceWriteOrigin + installed: boolean + rows: ReadonlyArray +} + +type ReplayChange = { + type: `insert` | `update` | `delete` + key: string | number + value: ReplayRow + previousValue?: ReplayRow +} + +type ReplayScenario = { + initialRows: ReadonlyArray + demandIds: ReadonlyArray + attempts: ReadonlyArray + settlementOrder: ReadonlyArray + settlementPhases: ReadonlyArray + releaseOnLastAttempt?: ReplayDemandId + afterSettlement: ReadonlyArray +} + +type SequentialReplayLoad = { + rows: ReadonlyArray + outcome: `return` | `throw` | `resolve` | `reject` +} + +type SequentialReplayScenario = { + initialRows: ReadonlyArray + loads: ReadonlyArray +} + +type CleanupRestartScenario = { + oldOutcome: `resolve` | `reject` + newOutcome: `resolve` | `reject` + settleOldFirst: boolean +} + +type SharedSubscriptionScenario = { + outcome: `resolve` | `reject` + releaseCountBeforeSettlement: 0 | 1 | 2 +} + +type OptimisticReplayScenario = { + operation: `insert` | `update` | `delete` + outcome: `resolve` | `reject` + serverRetainsTarget: boolean + initialValue: number + optimisticValue: number + serverValue: number +} + +type PendingReplay = { + attemptIndex: number + load: ReplayLoad + signal: AbortSignal | undefined + deferred: ReturnType> + error: Error + wroteRows: boolean + settled: boolean +} + +const rowArbitrary: fc.Arbitrary = fc.record({ + id: fc.constantFrom(`one` as const, `two` as const), + value: fc.integer({ min: -2, max: 2 }), +}) + +const rowsArbitrary = fc.uniqueArray(rowArbitrary, { + minLength: 0, + maxLength: 2, + selector: ({ id }) => id, +}) + +function replayLoadArbitrary( + demandId: ReplayDemandId, +): fc.Arbitrary { + return fc.record({ + demandId: fc.constant(demandId), + rows: fc + .option(fc.integer({ min: -2, max: 2 }), { nil: undefined }) + .map((value) => (value === undefined ? [] : [{ id: demandId, value }])), + outcome: fc.constantFrom(`resolve` as const, `reject` as const), + writeBeforeSettlement: fc.boolean(), + }) +} + +function sourceActionArbitrary( + demandIds: ReadonlyArray, +): fc.Arbitrary { + return fc.oneof( + fc + .tuple(fc.constantFrom(...demandIds), fc.integer({ min: -2, max: 2 })) + .map(([id, value]) => ({ type: `put` as const, row: { id, value } })), + fc + .constantFrom(...demandIds) + .map((id) => ({ type: `delete` as const, id })), + ) +} + +const replayScenarioArbitrary: fc.Arbitrary = fc + .uniqueArray(fc.constantFrom(`one`, `two`), { + minLength: 1, + maxLength: 2, + }) + .chain((demandIds) => + fc + .record({ + initialRows: rowsArbitrary, + attempts: fc.array( + fc + .tuple( + ...demandIds.map((demandId) => replayLoadArbitrary(demandId)), + ) + .map((loads) => ({ loads })), + { minLength: 1, maxLength: 3 }, + ), + releaseOnLastAttempt: fc.option(fc.constantFrom(...demandIds), { + nil: undefined, + }), + }) + .chain(({ initialRows, attempts, releaseOnLastAttempt }) => { + const replayCount = attempts.length * demandIds.length + const lastAttemptIndex = attempts.length - 1 + return fc + .record({ + settlementOrder: fc.shuffledSubarray( + Array.from({ length: replayCount }, (_, index) => index), + { minLength: replayCount, maxLength: replayCount }, + ), + rawSettlementPhases: fc.array( + fc.integer({ min: 0, max: lastAttemptIndex }), + { minLength: replayCount, maxLength: replayCount }, + ), + afterSettlement: + releaseOnLastAttempt === undefined + ? fc.array(sourceActionArbitrary(demandIds), { + minLength: 0, + maxLength: 3, + }) + : fc + .tuple( + fc.constant({ + type: `request`, + demandId: releaseOnLastAttempt, + }), + fc.array(sourceActionArbitrary(demandIds), { + minLength: 0, + maxLength: 2, + }), + ) + .map(([request, actions]) => [request, ...actions]), + }) + .map(({ settlementOrder, rawSettlementPhases, afterSettlement }) => ({ + initialRows, + demandIds, + attempts, + settlementOrder, + settlementPhases: rawSettlementPhases.map((phase, replayIndex) => + Math.max(phase, Math.floor(replayIndex / demandIds.length)), + ), + releaseOnLastAttempt, + afterSettlement, + })) + }), + ) + +const sequentialReplayScenarioArbitrary: fc.Arbitrary = + fc.record({ + initialRows: rowsArbitrary, + loads: fc.array( + fc.record({ + rows: rowsArbitrary, + outcome: fc.constantFrom( + `return` as const, + `throw` as const, + `resolve` as const, + `reject` as const, + ), + }), + { minLength: 1, maxLength: 3 }, + ), + }) + +const cleanupRestartScenarioArbitrary: fc.Arbitrary = + fc.record({ + oldOutcome: fc.constantFrom(`resolve` as const, `reject` as const), + newOutcome: fc.constantFrom(`resolve` as const, `reject` as const), + settleOldFirst: fc.boolean(), + }) + +const sharedSubscriptionScenarioArbitrary: fc.Arbitrary = + fc.record({ + outcome: fc.constantFrom(`resolve` as const, `reject` as const), + releaseCountBeforeSettlement: fc.constantFrom( + 0 as const, + 1 as const, + 2 as const, + ), + }) + +const optimisticReplayScenarioArbitrary: fc.Arbitrary = + fc + .record({ + operation: fc.constantFrom( + `insert` as const, + `update` as const, + `delete` as const, + ), + outcome: fc.constantFrom(`resolve` as const, `reject` as const), + serverRetainsTarget: fc.boolean(), + values: fc.uniqueArray(fc.integer({ min: -3, max: 3 }), { + minLength: 3, + maxLength: 3, + }), + }) + .map(({ operation, outcome, serverRetainsTarget, values }) => ({ + operation, + outcome, + serverRetainsTarget, + initialValue: values[0]!, + optimisticValue: values[1]!, + serverValue: values[2]!, + })) + +function rowsById( + rows: ReadonlyArray, +): Map { + return new Map(rows.map((row) => [row.id, { ...row }])) +} + +function sortedRows( + rows: ReadonlyMap, +): Array { + return [...rows.values()].sort((left, right) => + left.id.localeCompare(right.id), + ) +} + +function publicationDiff( + baseline: ReadonlyMap, + finalRows: ReadonlyMap, +): Array { + const changes: Array = [] + for (const [key, previousValue] of baseline) { + const value = finalRows.get(key) + if (!value) { + changes.push({ + type: `delete`, + key, + value: { ...previousValue }, + }) + } else if (value.value !== previousValue.value) { + changes.push({ + type: `update`, + key, + value: { ...value }, + previousValue: { ...previousValue }, + }) + } + } + for (const [key, value] of finalRows) { + if (!baseline.has(key)) { + changes.push({ type: `insert`, key, value: { ...value } }) + } + } + return changes +} + +function sortedChanges( + changes: ReadonlyArray, +): Array { + return [...changes].sort((left, right) => + String(left.key).localeCompare(String(right.key)), + ) +} + +function recordPublishedChanges( + visible: Map, + changes: ReadonlyArray, +): Array { + const recorded = changes.map((change) => ({ + type: change.type, + key: change.key, + value: { id: change.value.id, value: change.value.value }, + ...(change.previousValue === undefined + ? {} + : { + previousValue: { + id: change.previousValue.id, + value: change.previousValue.value, + }, + }), + })) + for (const change of recorded) { + if (change.type === `delete`) visible.delete(change.key) + else visible.set(change.key, { ...change.value }) + } + return recorded +} + +function expectSameSubsetRequest( + actual: LoadSubsetOptions, + expected: LoadSubsetOptions, +): void { + expect(actual.where).toBe(expected.where) + expect(actual.orderBy).toBe(expected.orderBy) + expect(actual.limit).toBe(expected.limit) + expect(actual.cursor).toEqual(expected.cursor) + expect(actual.offset).toBe(expected.offset) +} + +async function runReplayScenario(scenario: ReplayScenario): Promise { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + let unloadCount = 0 + const leases = new Map< + LoadSubsetOptions, + { acquisitions: number; releases: number } + >() + const queuedLoads: Array<{ attemptIndex: number; load: ReplayLoad }> = [] + const queuedReacquisitions = new Set() + const pendingReplays: Array = [] + const sourceRows = new Map() + const sourceWrites: Array = [] + const expectedSourceWrites: Array = [] + const demandWheres = new Map( + scenario.demandIds.map((demandId) => [ + demandId, + new Func(`eq`, [new PropRef([`id`]), new Value(demandId)]), + ]), + ) + const demandIdByWhere = new Map< + NonNullable, + ReplayDemandId + >([...demandWheres].map(([demandId, where]) => [where, demandId])) + const requestByDemand = new Map() + const activeDemandIds = new Set(scenario.demandIds) + + const recordExpectedSourceWrite = ( + rows: ReadonlyArray, + origin: SourceWriteOrigin, + installed: boolean, + ) => { + expectedSourceWrites.push({ + origin, + installed, + rows: rows.map((row) => ({ ...row })), + }) + } + + const assertSourceWrites = () => { + expect(sourceWrites).toEqual(expectedSourceWrites) + } + + const applyRows = ( + rows: ReadonlyArray, + origin: SourceWriteOrigin, + signal?: AbortSignal, + ): boolean => { + const installed = !signal?.aborted + sourceWrites.push({ + origin, + installed, + rows: rows.map((row) => ({ ...row })), + }) + if (!installed || rows.length === 0) return installed + begin() + for (const row of rows) { + write({ + type: sourceRows.has(row.id) ? `update` : `insert`, + value: { ...row }, + }) + } + commit() + for (const row of rows) sourceRows.set(row.id, { ...row }) + return true + } + + const collection: Collection = + createCollection({ + id: `subscription-replay-oracle`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: (options) => { + loadCount++ + const lease = leases.get(options) ?? { + acquisitions: 0, + releases: 0, + } + lease.acquisitions++ + leases.set(options, lease) + const demandId = + options.where === undefined + ? undefined + : demandIdByWhere.get(options.where) + if (demandId === undefined) { + throw new Error(`Subset request did not preserve its demand`) + } + + if (!requestByDemand.has(demandId)) { + requestByDemand.set(demandId, options) + applyRows( + scenario.initialRows.filter(({ id }) => id === demandId), + { type: `initial`, demandId }, + ) + return true + } + + const queuedIndex = queuedLoads.findIndex( + ({ load }) => load.demandId === demandId, + ) + if (queuedIndex === -1) { + if (queuedReacquisitions.delete(demandId)) { + expectSameSubsetRequest( + options, + requestByDemand.get(demandId)!, + ) + return true + } + throw new Error(`Replay load was not queued for ${demandId}`) + } + const [queued] = queuedLoads.splice(queuedIndex, 1) + if (!queued) throw new Error(`Replay queue changed unexpectedly`) + expectSameSubsetRequest(options, requestByDemand.get(demandId)!) + const pending: PendingReplay = { + attemptIndex: queued.attemptIndex, + load: queued.load, + signal: options.signal, + deferred: createDeferred(), + error: new Error(`Replay rejected`), + wroteRows: false, + settled: false, + } + pendingReplays.push(pending) + return pending.deferred.promise + }, + unloadSubset: (options) => { + unloadCount++ + const lease = leases.get(options) ?? { + acquisitions: 0, + releases: 0, + } + lease.releases++ + leases.set(options, lease) + }, + } + }, + }, + }) + + const visible = new Map() + let publicationCount = 0 + const publicationBatches: Array> = [] + const subscription = collection.subscribeChanges((changes) => { + publicationCount++ + publicationBatches.push(recordPublishedChanges(visible, changes)) + }) + const reportedErrors: Array = [] + subscription.on(`loadSubset:error`, ({ error }) => reportedErrors.push(error)) + let unsubscribed = false + + const assertPublished = ( + expected: ReadonlyMap, + ) => { + expect(sortedRows(visible)).toEqual(sortedRows(expected)) + } + + const assertSource = () => { + const actual = rowsById( + collection.toArray.map(({ id, value }) => ({ id, value })), + ) + expect(sortedRows(actual)).toEqual(sortedRows(sourceRows)) + } + + const applySourceAction = (action: SourceAction): boolean => { + if (action.type === `request`) return false + if (action.type === `delete`) { + const previous = sourceRows.get(action.id) + if (!previous) return false + begin() + write({ type: `delete`, key: action.id }) + commit() + sourceRows.delete(action.id) + return true + } + + const previous = sourceRows.get(action.row.id) + if (previous?.value === action.row.value) return false + applyRows([action.row], { type: `ordinary` }) + return true + } + + try { + for (const demandId of scenario.demandIds) { + subscription.requestSnapshot({ + optimizedOnly: false, + where: demandWheres.get(demandId), + }) + recordExpectedSourceWrite( + scenario.initialRows.filter(({ id }) => id === demandId), + { type: `initial`, demandId }, + true, + ) + assertSourceWrites() + } + const expectedPublished = rowsById( + scenario.initialRows.filter(({ id }) => activeDemandIds.has(id)), + ) + assertPublished(expectedPublished) + assertSource() + let expectedPublicationCount = publicationCount + let lastReportedError: Error | undefined + let modelSession: + | { + baseline: Map + pending: Set + currentAttemptIndex: number + publicationCount: number + } + | undefined + + const writeReplayRows = ( + pending: PendingReplay, + isCurrent: boolean, + ): void => { + if (pending.wroteRows) return + const load = pending.load + recordExpectedSourceWrite( + load.rows, + { + type: `replay`, + demandId: load.demandId, + attemptIndex: pending.attemptIndex, + }, + isCurrent, + ) + const installed = applyRows( + load.rows, + { + type: `replay`, + demandId: load.demandId, + attemptIndex: pending.attemptIndex, + }, + pending.signal, + ) + pending.wroteRows = true + expect(installed).toBe(isCurrent) + assertSourceWrites() + } + + const settleReplay = async (replayIndex: number) => { + const pending = pendingReplays[replayIndex]! + const session = modelSession + const load = pending.load + const isCurrent = + session !== undefined && + pending.attemptIndex === session.currentAttemptIndex && + activeDemandIds.has(load.demandId) + pending.settled = true + if (load.outcome === `resolve`) { + writeReplayRows(pending, isCurrent) + pending.deferred.resolve() + } else { + if (isCurrent) { + lastReportedError = pending.error + } else { + expect(pending.signal?.aborted).toBe(true) + } + pending.deferred.reject(pending.error) + } + session?.pending.delete(replayIndex) + await flushPromises() + assertSource() + + if (!session) { + expect(subscription.status).toBe(`ready`) + assertPublished(expectedPublished) + expect(subscription.lastError).toBe(lastReportedError) + return + } + + const hasPendingReplay = session.pending.size > 0 + expect(subscription.status).toBe( + hasPendingReplay ? `loadingSubset` : `ready`, + ) + + if (session.pending.size === 0) { + const currentAttempt = scenario.attempts[session.currentAttemptIndex]! + const currentAttemptSucceeds = currentAttempt.loads.every( + ({ demandId, outcome }) => + !activeDemandIds.has(demandId) || outcome === `resolve`, + ) + const previousPublication = new Map(expectedPublished) + expectedPublished.clear() + const nextRows = currentAttemptSucceeds ? sourceRows : session.baseline + for (const [id, row] of nextRows) { + expectedPublished.set(id, { ...row }) + } + + if (currentAttemptSucceeds) { + const expectedBatch = publicationDiff( + previousPublication, + expectedPublished, + ) + expect(publicationCount - session.publicationCount).toBe( + Number(expectedBatch.length > 0), + ) + if (expectedBatch.length > 0) { + expect(sortedChanges(publicationBatches.at(-1)!)).toEqual( + sortedChanges(expectedBatch), + ) + } + modelSession = undefined + } else { + expect(publicationCount).toBe(session.publicationCount) + } + expectedPublicationCount = publicationCount + } else { + expect(publicationCount).toBe(session.publicationCount) + } + + assertPublished(expectedPublished) + expect(subscription.lastError).toBe(lastReportedError) + expect(reportedErrors.at(-1)).toBe(lastReportedError) + } + + for (const [attemptIndex, attempt] of scenario.attempts.entries()) { + modelSession ??= { + baseline: new Map(expectedPublished), + pending: new Set(), + currentAttemptIndex: attemptIndex, + publicationCount: expectedPublicationCount, + } + modelSession.currentAttemptIndex = attemptIndex + + for (const load of attempt.loads) { + queuedLoads.push({ attemptIndex, load }) + } + const firstReplayIndex = pendingReplays.length + begin() + truncate() + commit() + sourceRows.clear() + await flushPromises() + for ( + let replayIndex = firstReplayIndex; + replayIndex < pendingReplays.length; + replayIndex++ + ) { + modelSession.pending.add(replayIndex) + const pending = pendingReplays[replayIndex]! + if (pending.load.writeBeforeSettlement) { + writeReplayRows(pending, true) + } + } + if ( + attemptIndex === scenario.attempts.length - 1 && + scenario.releaseOnLastAttempt !== undefined + ) { + const releasedDemand = scenario.releaseOnLastAttempt + subscription.releaseSnapshot(demandWheres.get(releasedDemand)!) + activeDemandIds.delete(releasedDemand) + for (const replayIndex of modelSession.pending) { + if (pendingReplays[replayIndex]?.load.demandId === releasedDemand) { + modelSession.pending.delete(replayIndex) + } + } + // A released request does not retract rows already applied by the + // source, nor change the retained baseline of an unfinished replay. + if (modelSession.pending.size === 0 && activeDemandIds.size === 0) { + expectedPublicationCount = publicationCount + modelSession = undefined + } + } + assertSource() + assertPublished(expectedPublished) + expect(publicationCount).toBe( + modelSession?.publicationCount ?? expectedPublicationCount, + ) + expect(subscription.lastError).toBe(lastReportedError) + expect(subscription.status).toBe( + modelSession && modelSession.pending.size > 0 + ? `loadingSubset` + : `ready`, + ) + + for (const replayIndex of scenario.settlementOrder) { + const replay = pendingReplays[replayIndex] + if ( + replay && + !replay.settled && + scenario.settlementPhases[replayIndex] === attemptIndex + ) { + await settleReplay(replayIndex) + } + } + } + + expect(modelSession?.pending.size ?? 0).toBe(0) + + for (const action of scenario.afterSettlement) { + const countBeforeAction = publicationCount + const previousPublication = new Map(expectedPublished) + if (action.type === `request`) { + queuedReacquisitions.add(action.demandId) + activeDemandIds.add(action.demandId) + subscription.requestSnapshot({ + optimizedOnly: false, + where: demandWheres.get(action.demandId), + }) + const row = sourceRows.get(action.demandId) + if (!modelSession && row) { + expectedPublished.set(action.demandId, { ...row }) + } + } + const applied = applySourceAction(action) + if (applied && action.type === `delete`) { + if (!modelSession) expectedPublished.delete(action.id) + } else if (applied && action.type === `put`) { + recordExpectedSourceWrite([action.row], { type: `ordinary` }, true) + assertSourceWrites() + if (!modelSession) { + expectedPublished.set(action.row.id, { ...action.row }) + } + } + assertSource() + assertPublished(expectedPublished) + const expectedBatch = publicationDiff( + previousPublication, + expectedPublished, + ) + const expectsPublication = + !modelSession && (action.type === `request` || expectedBatch.length > 0) + expect(publicationCount).toBe( + countBeforeAction + Number(expectsPublication), + ) + if (expectsPublication) { + expect(sortedChanges(publicationBatches.at(-1)!)).toEqual( + sortedChanges(expectedBatch), + ) + } + } + + subscription.unsubscribe() + unsubscribed = true + expect(unloadCount).toBe(loadCount) + for (const lease of leases.values()) { + expect(lease).toEqual({ acquisitions: 1, releases: 1 }) + } + assertSourceWrites() + } finally { + for (const replay of pendingReplays) { + if (!replay.settled) replay.deferred.resolve() + } + await flushPromises() + if (!unsubscribed) subscription.unsubscribe() + await collection.cleanup() + } +} + +async function runSequentialReplayScenario( + scenario: SequentialReplayScenario, +): Promise { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + let nextLoad: SequentialReplayLoad | undefined + let nextError: Error | undefined + let initialLoad = true + const sourceRows = new Map() + const leases = new Map< + LoadSubsetOptions, + { acquisitions: number; releases: number } + >() + const pending: Array<{ + load: SequentialReplayLoad + deferred: ReturnType> + error: Error + }> = [] + + const applyRows = (rows: ReadonlyArray) => { + if (rows.length === 0) return + begin() + for (const row of rows) { + write({ + type: sourceRows.has(row.id) ? `update` : `insert`, + value: { ...row }, + }) + } + commit() + for (const row of rows) sourceRows.set(row.id, { ...row }) + } + + const collection = createCollection({ + id: `sequential-replay-oracle`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: (options) => { + if (initialLoad) { + initialLoad = false + applyRows(scenario.initialRows) + leases.set(options, { acquisitions: 1, releases: 0 }) + return true + } + + const load = nextLoad + if (!load) throw new Error(`Sequential replay was not queued`) + const error = nextError + if (!error) + throw new Error(`Sequential replay error was not queued`) + nextLoad = undefined + nextError = undefined + applyRows(load.rows) + if (load.outcome === `throw`) { + throw error + } + + leases.set(options, { acquisitions: 1, releases: 0 }) + if (load.outcome === `return`) return true + const deferred = createDeferred() + pending.push({ load, deferred, error }) + return deferred.promise + }, + unloadSubset: (options) => { + const lease = leases.get(options) + if (!lease) { + throw new Error(`Released an acquisition that never returned`) + } + lease.releases++ + }, + } + }, + }, + }) + const visible = new Map() + let publicationCount = 0 + const publicationBatches: Array> = [] + const subscription = collection.subscribeChanges((changes) => { + publicationCount++ + publicationBatches.push(recordPublishedChanges(visible, changes)) + }) + const reportedErrors: Array = [] + subscription.on(`loadSubset:error`, ({ error }) => reportedErrors.push(error)) + let unsubscribed = false + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + const expectedPublished = rowsById(scenario.initialRows) + let expectedLastError: unknown + + for (const load of scenario.loads) { + const baseline = new Map(expectedPublished) + const publicationBefore = publicationCount + const pendingBefore = pending.length + const expectedError = new Error( + load.outcome === `throw` + ? `Synchronous replay failure` + : `Asynchronous replay failure`, + ) + nextLoad = load + nextError = expectedError + begin() + truncate() + commit() + sourceRows.clear() + await flushPromises() + + const pendingLoad = pending[pendingBefore] + if (load.outcome === `resolve`) pendingLoad?.deferred.resolve() + if (load.outcome === `reject`) { + pendingLoad?.deferred.reject(pendingLoad.error) + } + await flushPromises() + + const succeeded = load.outcome === `return` || load.outcome === `resolve` + if (succeeded) { + expectedPublished.clear() + for (const [id, row] of sourceRows) { + expectedPublished.set(id, { ...row }) + } + } else { + expectedPublished.clear() + for (const [id, row] of baseline) expectedPublished.set(id, { ...row }) + expectedLastError = expectedError + } + + const expectedBatch = succeeded + ? publicationDiff(baseline, expectedPublished) + : [] + expect(publicationCount - publicationBefore).toBe( + Number(expectedBatch.length > 0), + ) + if (expectedBatch.length > 0) { + expect(sortedChanges(publicationBatches.at(-1)!)).toEqual( + sortedChanges(expectedBatch), + ) + } + expect(sortedRows(visible)).toEqual(sortedRows(expectedPublished)) + expect( + sortedRows( + rowsById(collection.toArray.map(({ id, value }) => ({ id, value }))), + ), + ).toEqual(sortedRows(sourceRows)) + expect(subscription.status).toBe(`ready`) + expect(subscription.lastError).toBe(expectedLastError) + expect(reportedErrors.at(-1)).toBe(expectedLastError) + } + + subscription.unsubscribe() + unsubscribed = true + for (const lease of leases.values()) { + expect(lease).toEqual({ acquisitions: 1, releases: 1 }) + } + } finally { + for (const load of pending) load.deferred.resolve() + await flushPromises() + if (!unsubscribed) subscription.unsubscribe() + await collection.cleanup() + } +} + +async function runCleanupRestartScenario( + scenario: CleanupRestartScenario, +): Promise { + const sessions: Array<{ + begin: () => void + write: (message: ChangeMessageOrDeleteKeyMessage) => void + commit: () => void + }> = [] + const loads: Array<{ + session: number + deferred: ReturnType> + }> = [] + let session = 0 + const collection = createCollection({ + id: `cleanup-restart-oracle`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + const currentSession = session++ + sessions.push({ begin, write, commit }) + markReady() + return { + loadSubset: () => { + const deferred = createDeferred() + loads.push({ session: currentSession, deferred }) + return deferred.promise + }, + } + }, + }, + }) + + const settle = async (loadIndex: number, outcome: `resolve` | `reject`) => { + const load = loads[loadIndex]! + if (outcome === `resolve`) load.deferred.resolve() + else load.deferred.reject(new Error(`session ${load.session} failed`)) + await flushPromises() + } + + try { + const oldResult = collection._sync.loadSubset({}) + expect(oldResult).toBeInstanceOf(Promise) + if (oldResult instanceof Promise) void oldResult.catch(() => {}) + expect(collection.isLoadingSubset).toBe(true) + + await collection.cleanup() + expect(collection.isLoadingSubset).toBe(false) + + collection.startSyncImmediate() + const newResult = collection._sync.loadSubset({}) + expect(newResult).toBeInstanceOf(Promise) + if (newResult instanceof Promise) void newResult.catch(() => {}) + expect(loads.map(({ session: loadSession }) => loadSession)).toEqual([0, 1]) + expect(collection.isLoadingSubset).toBe(true) + + const oldSession = sessions[0]! + oldSession.begin() + oldSession.write({ type: `insert`, value: { id: `one`, value: 1 } }) + oldSession.commit() + expect(collection.toArray).toEqual([]) + + const currentSession = sessions[1]! + currentSession.begin() + currentSession.write({ type: `insert`, value: { id: `two`, value: 2 } }) + currentSession.commit() + expect(collection.toArray.map(({ id, value }) => ({ id, value }))).toEqual([ + { id: `two`, value: 2 }, + ]) + + const settlementOrder = scenario.settleOldFirst ? [0, 1] : [1, 0] + const outcomes = [scenario.oldOutcome, scenario.newOutcome] as const + let newSettled = false + for (const loadIndex of settlementOrder) { + await settle(loadIndex, outcomes[loadIndex]!) + if (loadIndex === 1) newSettled = true + expect(collection.isLoadingSubset).toBe(!newSettled) + expect( + collection.toArray.map(({ id, value }) => ({ id, value })), + ).toEqual([{ id: `two`, value: 2 }]) + } + } finally { + for (const { deferred } of loads) deferred.resolve() + await flushPromises() + await collection.cleanup() + } +} + +async function expectScheduledReplaySettlementIsGenerationSafe( + scheduler: Scheduler, +): Promise { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const loads: Array<{ + signal: AbortSignal | undefined + outcome: Promise + }> = [] + const collection = createCollection({ + id: `scheduled-replay-settlement`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (actions) => { + begin = actions.begin + write = actions.write + commit = actions.commit + truncate = actions.truncate + actions.markReady() + return { + loadSubset: ({ signal }) => { + const generation = loads.length + 1 + const outcome = scheduler + .schedule(Promise.resolve(), `generation-${generation}`) + .then(() => { + if (signal?.aborted) return + begin() + write({ + type: `insert`, + value: { id: `one`, value: generation }, + }) + commit() + }) + loads.push({ signal, outcome }) + return outcome + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes) + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + + expect(loads).toHaveLength(2) + expect(loads[0]!.signal?.aborted).toBe(true) + + await scheduler.waitAll() + await Promise.all(loads.map(({ outcome }) => outcome)) + await flushPromises() + + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + expect(subscription.status).toBe(`ready`) + expect(subscription.lastError).toBeUndefined() + } finally { + if (scheduler.count() > 0) await scheduler.waitAll() + await Promise.allSettled(loads.map(({ outcome }) => outcome)) + subscription.unsubscribe() + await collection.cleanup() + } +} + +async function runSharedSubscriptionScenario( + scenario: SharedSubscriptionScenario, +): Promise { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + const transports = [createDeferred(), createDeferred()] as const + const transportOptions: Array = [] + const unloads: Array = [] + const dedupe = new DeduplicatedLoadSubset({ + loadSubset: (options) => { + const transport = transports[transportOptions.length] + if (!transport) throw new Error(`unexpected transport`) + transportOptions.push(options) + return transport.promise + }, + }) + const collection = createCollection({ + id: `shared-subscription-oracle`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: dedupe.loadSubset, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const visible = [ + new Map(), + new Map(), + ] as const + const subscribe = (rows: Map) => + collection.subscribeChanges((changes) => { + for (const change of changes) { + if (change.type === `delete`) rows.delete(change.key) + else { + rows.set(change.key, { + id: change.value.id, + value: change.value.value, + }) + } + } + }) + const subscriptions = [subscribe(visible[0]), subscribe(visible[1])] as const + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + let firstUnsubscribed = false + let secondUnsubscribed = false + + try { + subscriptions[0].requestSnapshot({ where }) + subscriptions[1].requestSnapshot({ where }) + expect(transportOptions).toHaveLength(2) + expect(subscriptions[0].status).toBe(`loadingSubset`) + expect(subscriptions[1].status).toBe(`loadingSubset`) + + if (scenario.releaseCountBeforeSettlement >= 1) { + subscriptions[0].unsubscribe() + firstUnsubscribed = true + expect(transportOptions[0]?.signal?.aborted).toBe(true) + expect(transportOptions[1]?.signal?.aborted).toBe(false) + } + if (scenario.releaseCountBeforeSettlement === 2) { + subscriptions[1].unsubscribe() + secondUnsubscribed = true + expect(transportOptions[1]?.signal?.aborted).toBe(true) + } + + const failure = new Error(`shared transport failed`) + if (scenario.outcome === `resolve`) { + if (transportOptions.some(({ signal }) => !signal?.aborted)) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + } + for (const transport of transports) transport.resolve() + } else { + transports.forEach((transport, index) => + transport.reject( + transportOptions[index]?.signal?.aborted + ? new DOMException(`obsolete`, `AbortError`) + : failure, + ), + ) + } + await flushPromises() + + if (!secondUnsubscribed) { + expect(subscriptions[1].status).toBe(`ready`) + expect(subscriptions[1].lastError).toBe( + scenario.outcome === `reject` ? failure : undefined, + ) + expect([...visible[1].values()]).toEqual( + scenario.outcome === `resolve` ? [{ id: `one`, value: 1 }] : [], + ) + } else { + expect(subscriptions[1].lastError).toBeUndefined() + expect([...visible[1].values()]).toEqual([]) + } + if (firstUnsubscribed) { + expect(subscriptions[0].lastError).toBeUndefined() + } else { + expect(subscriptions[0].lastError).toBe( + scenario.outcome === `reject` ? failure : undefined, + ) + } + + if (!firstUnsubscribed) { + subscriptions[0].unsubscribe() + firstUnsubscribed = true + } + if (!secondUnsubscribed) { + subscriptions[1].unsubscribe() + secondUnsubscribed = true + } + expect(unloads).toHaveLength(2) + expect(new Set(unloads).size).toBe(2) + } finally { + for (const transport of transports) transport.resolve() + await flushPromises() + if (!firstUnsubscribed) subscriptions[0].unsubscribe() + if (!secondUnsubscribed) subscriptions[1].unsubscribe() + await collection.cleanup() + } +} + +function applyOptimisticOperation( + source: ReadonlyMap, + scenario: OptimisticReplayScenario, +): Map { + const result = new Map( + [...source].map(([key, row]) => [key, { ...row }] as const), + ) + if (scenario.operation === `insert`) { + result.set(`two`, { id: `two`, value: scenario.optimisticValue }) + } else if (scenario.operation === `update`) { + result.set(`one`, { id: `one`, value: scenario.optimisticValue }) + } else { + result.delete(`one`) + } + return result +} + +async function runOptimisticReplayScenario( + scenario: OptimisticReplayScenario, +): Promise { + let begin!: (options?: { immediate?: boolean }) => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const replay = createDeferred() + const replayFailure = new Error(`optimistic replay failed`) + const mutation = createDeferred() + const initialSource = rowsById([{ id: `one`, value: scenario.initialValue }]) + const replayRows = + scenario.operation === `insert` + ? [ + { id: `one` as const, value: scenario.serverValue }, + ...(scenario.serverRetainsTarget + ? [{ id: `two` as const, value: scenario.serverValue }] + : []), + ] + : scenario.serverRetainsTarget + ? [{ id: `one` as const, value: scenario.serverValue }] + : [] + const replaySource = rowsById(replayRows) + const collection = createCollection({ + id: `optimistic-replay-${scenario.operation}-${scenario.outcome}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount === 1) { + begin() + for (const row of initialSource.values()) { + write({ type: `insert`, value: { ...row } }) + } + commit() + return true + } + return replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const batches: Array> = [] + const subscription = collection.subscribeChanges((changes) => { + batches.push(recordPublishedChanges(visible, changes)) + }) + const transaction = createTransaction({ + mutationFn: () => mutation.promise, + }) + void transaction.isPersisted.promise.catch(() => {}) + let unsubscribed = false + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + expect(sortedRows(visible)).toEqual(sortedRows(initialSource)) + + transaction.mutate(() => { + if (scenario.operation === `insert`) { + collection.insert({ id: `two`, value: scenario.optimisticValue }) + } else if (scenario.operation === `update`) { + collection.update(`one`, (draft) => { + draft.value = scenario.optimisticValue + }) + } else { + collection.delete(`one`) + } + }) + const optimisticBaseline = applyOptimisticOperation(initialSource, scenario) + expect(sortedRows(visible)).toEqual(sortedRows(optimisticBaseline)) + expect( + sortedRows( + rowsById(collection.toArray.map(({ id, value }) => ({ id, value }))), + ), + ).toEqual(sortedRows(optimisticBaseline)) + batches.length = 0 + + begin() + truncate() + commit() + await flushPromises() + // A loadSubset adapter must install its request-scoped rows before its + // promise settles, even while a user mutation is still persisting. + begin({ immediate: true }) + for (const row of replayRows) { + write({ type: `insert`, value: { ...row } }) + } + commit() + if (scenario.outcome === `resolve`) replay.resolve() + else replay.reject(replayFailure) + await flushPromises() + + const expected = applyOptimisticOperation( + scenario.outcome === `resolve` ? replaySource : initialSource, + scenario, + ) + const expectedBatch = + scenario.outcome === `resolve` + ? publicationDiff(optimisticBaseline, expected) + : [] + expect(sortedRows(visible)).toEqual(sortedRows(expected)) + expect(batches.map(sortedChanges)).toEqual( + expectedBatch.length > 0 ? [sortedChanges(expectedBatch)] : [], + ) + expect(subscription.lastError).toBe( + scenario.outcome === `reject` ? replayFailure : undefined, + ) + + subscription.unsubscribe() + unsubscribed = true + } finally { + replay.resolve() + mutation.resolve() + await flushPromises() + if (!unsubscribed) subscription.unsubscribe() + await collection.cleanup() + } +} + +const { multiplier, ...replayConfig } = readOracleRunConfig() +const generatedRuns = 30 * multiplier + +describe(`CollectionSubscription replay oracle`, () => { + it.each([`resolve`, `reject`] as const)( + `starts a replacement that lets canceled replay %s`, + async (outcome) => { + const oldReplay = createDeferred() + const newReplay = createDeferred() + const aborted = new DOMException(`superseded`, `AbortError`) + const loads: Array = [] + const unloads: Array = [] + const events: Array = [] + let operations!: Parameters[`sync`]>[0] + const collection = createCollection({ + id: `replacement-start-dependency-${outcome}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (sync) => { + operations = sync + sync.markReady() + return { + loadSubset: (options) => { + loads.push(options) + events.push(`load:${loads.length}`) + if (loads.length === 1) { + sync.begin() + sync.write({ type: `insert`, value: { id: `one`, value: 0 } }) + sync.commit() + return true + } + if (loads.length === 2) return oldReplay.promise + // This provider has stopped old request-scoped writes on abort. + // Its shared refresh protocol completes the old waiter only + // when a replacement acquisition registers. Completion does + // not require new result publication or a callback from core. + expect(loads[1]?.signal?.aborted).toBe(true) + if (outcome === `resolve`) oldReplay.resolve() + else oldReplay.reject(aborted) + events.push(`old:settled`) + return newReplay.promise + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + const { id, value } = change.value + visible.set(change.key, { id, value }) + } + } + }) + const truncate = () => { + operations.begin() + operations.truncate() + operations.commit() + } + try { + subscription.requestSnapshot({ optimizedOnly: false }) + expect([...visible.values()]).toEqual([{ id: `one`, value: 0 }]) + truncate() + const completion = subscription.pendingTruncateReplacement + expect(completion).toBeDefined() + let completed = false + const completionErrors: Array = [] + void completion?.then( + () => { + completed = true + }, + (error: unknown) => completionErrors.push(error), + ) + await flushPromises() + expect(loads).toHaveLength(2) + expect(oldReplay.isPending()).toBe(true) + expect([...visible.values()]).toEqual([{ id: `one`, value: 0 }]) + + truncate() + await flushPromises() + expect(events).toEqual([`load:1`, `load:2`, `load:3`, `old:settled`]) + expect(oldReplay.isPending()).toBe(false) + expect([...visible.values()]).toEqual([{ id: `one`, value: 0 }]) + expect(subscription.status).toBe(`loadingSubset`) + expect(subscription.lastError).toBeUndefined() + expect(completed).toBe(false) + expect(completionErrors).toEqual([]) + + operations.begin() + operations.write({ type: `insert`, value: { id: `one`, value: 2 } }) + await operations.commit() + newReplay.resolve() + await flushPromises() + expect([...visible.values()]).toEqual([{ id: `one`, value: 2 }]) + expect(subscription.status).toBe(`ready`) + expect(subscription.lastError).toBeUndefined() + expect(completed).toBe(true) + expect(completionErrors).toEqual([]) + } finally { + oldReplay.resolve() + newReplay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(loads.length) + for (const load of loads) { + expect(unloads.filter((unload) => unload === load)).toHaveLength(1) + } + }, + ) + + it(`generates shared, failed, stale, released, and post-replay histories`, () => { + const scenarios = fc.sample(replayScenarioArbitrary, { + seed: 1755, + numRuns: 300, + }) + + expect(scenarios.some(({ demandIds }) => demandIds.length > 1)).toBe(true) + expect(scenarios.some(({ attempts }) => attempts.length > 1)).toBe(true) + expect( + scenarios.some(({ attempts }) => + attempts.some(({ loads }) => + loads.some(({ outcome }) => outcome === `reject`), + ), + ), + ).toBe(true) + expect( + scenarios.some(({ attempts }) => + attempts.some(({ loads }) => + loads.some(({ writeBeforeSettlement }) => writeBeforeSettlement), + ), + ), + ).toBe(true) + expect( + scenarios.some(({ settlementOrder }) => + settlementOrder.some((value, index) => value !== index), + ), + ).toBe(true) + expect( + scenarios.some(({ releaseOnLastAttempt }) => + Boolean(releaseOnLastAttempt), + ), + ).toBe(true) + expect( + scenarios.some(({ afterSettlement }) => afterSettlement.length > 0), + ).toBe(true) + expect( + scenarios.some(({ afterSettlement }) => + afterSettlement.some(({ type }) => type === `request`), + ), + ).toBe(true) + }) + + it(`aborts an in-flight initial acquisition before its replay replaces it`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const loads: Array<{ + options: LoadSubsetOptions + deferred: ReturnType> + }> = [] + const collection = createCollection({ + id: `initial-acquisition-replay`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: (options) => { + const deferred = createDeferred() + loads.push({ options, deferred }) + return deferred.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + value: change.value.value, + }) + } + } + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + expect(loads[0]?.options.signal?.aborted).toBe(true) + + begin() + write({ type: `insert`, value: { id: `two`, value: 2 } }) + commit() + loads[1]?.deferred.resolve() + await flushPromises() + + if (!loads[0]?.options.signal?.aborted) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + } + loads[0]?.deferred.resolve() + await flushPromises() + + expect(sortedRows(visible)).toEqual([{ id: `two`, value: 2 }]) + expect(subscription.status).toBe(`ready`) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`keeps the published replacement after a reentrant replay fails`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const replayLoads: Array>> = [] + const collection = createCollection({ + id: `reentrant-replay`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + return true + } + + const deferred = createDeferred() + replayLoads.push(deferred) + return deferred.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + let startedNestedReplay = false + const subscription = collection.subscribeChanges((changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + value: change.value.value, + }) + } + } + + if (!startedNestedReplay && visible.get(`one`)?.value === 2) { + startedNestedReplay = true + begin() + truncate() + commit() + } + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + begin() + write({ type: `insert`, value: { id: `one`, value: 2 } }) + commit() + replayLoads[0]?.resolve() + await flushPromises() + expect(startedNestedReplay).toBe(true) + + replayLoads[1]?.reject(new Error(`nested replay failed`)) + await flushPromises() + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`ignores an aborted released demand while publishing the remaining replay`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const replays: Array<{ + options: { signal?: AbortSignal } + deferred: ReturnType> + }> = [] + let loadCount = 0 + const collection = createCollection({ + id: `released-demand-replay`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: (options) => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 0 } }) + write({ type: `insert`, value: { id: `two`, value: 0 } }) + commit() + return true + } + if (loadCount === 2) return true + + const deferred = createDeferred() + replays.push({ options, deferred }) + return deferred.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + value: change.value.value, + }) + } + } + }) + const demandOne = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const demandTwo = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + + try { + subscription.requestSnapshot({ where: demandOne }) + subscription.requestSnapshot({ where: demandTwo }) + begin() + truncate() + commit() + await flushPromises() + + subscription.releaseSnapshot(demandOne) + expect(replays[0]?.options.signal?.aborted).toBe(true) + begin() + write({ type: `insert`, value: { id: `two`, value: 2 } }) + commit() + replays[1]?.deferred.resolve() + await flushPromises() + + expect(sortedRows(visible)).toEqual([{ id: `two`, value: 2 }]) + expect(subscription.status).toBe(`ready`) + expect(subscription.lastError).toBeUndefined() + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + const orderedReplayCases = ([`asc`, `desc`] as const).flatMap((direction) => [ + ...([`return`, `resolve`] as const).flatMap((delivery) => + ([`same`, `changed`] as const).map((identity) => ({ + name: `${direction} ${delivery} with ${identity} keys`, + direction, + delivery, + identity, + })), + ), + ...([`throw`, `reject`] as const).map((delivery) => ({ + name: `${direction} ${delivery}`, + direction, + delivery, + identity: `none` as const, + })), + ]) + + it.each(orderedReplayCases)( + `restores ordered offset and cursor state after replay: $name`, + async ({ direction, delivery, identity }) => { + type OrderedReplayRow = { + id: `one` | `two` | `three` | `four` + value: number + } + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const loadOptions: Array = [] + const replayLoads: Array>> = [] + const replayRows: ReadonlyArray = + identity === `same` + ? [ + { id: `one`, value: 1 }, + { id: `two`, value: 2 }, + ] + : [ + { id: `three`, value: 1 }, + { id: `four`, value: 2 }, + ] + let replayRowsInstalled = false + const installReplayRows = () => { + if (replayRowsInstalled || identity === `none`) return + replayRowsInstalled = true + begin() + for (const row of replayRows) { + write({ type: `insert`, value: row }) + } + commit() + } + const collection = createCollection({ + id: `ordered-replay-${direction}-${delivery}-${identity}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + write({ type: `insert`, value: { id: `two`, value: 2 } }) + commit() + params.markReady() + return { + loadSubset: (options) => { + loadCount++ + loadOptions.push(options) + if (loadCount <= 2) return true + if (loadCount > 4) return true + + if (delivery === `return`) { + installReplayRows() + return true + } + if (delivery === `throw`) { + if (loadCount === 3) { + throw new Error(`ordered replay failed`) + } + return true + } + + const deferred = createDeferred() + replayLoads.push(deferred) + return deferred.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const index = collection.createIndex((row) => row.value, { + indexType: BTreeIndex, + }) + const orderedIndex = direction === `asc` ? index : new ReverseIndex(index) + const orderBy: OrderBy = [ + { + expression: new PropRef([`value`]), + compareOptions: { direction, nulls: `first` }, + }, + ] + const batches: Array> = [] + const subscription = collection.subscribeChanges((changes) => { + batches.push(changes.map(({ value }) => value.id)) + }) + subscription.setOrderByIndex(orderedIndex) + + const initialIds = + direction === `asc` + ? ([`one`, `two`] as const) + : ([`two`, `one`] as const) + const replacementIds = + direction === `asc` + ? ([`three`, `four`] as const) + : ([`four`, `three`] as const) + const succeeds = delivery === `return` || delivery === `resolve` + const expectedIds = identity === `changed` ? replacementIds : initialIds + + try { + subscription.requestLimitedSnapshot({ orderBy, limit: 1 }) + expect(batches).toEqual([[initialIds[0]]]) + subscription.requestLimitedSnapshot({ + orderBy, + limit: 1, + minValues: [direction === `asc` ? 1 : 2], + }) + expect(loadOptions[1]).toMatchObject({ + offset: 1, + cursor: { lastKey: initialIds[1] }, + }) + + begin() + truncate() + commit() + await flushPromises() + expectSameSubsetRequest(loadOptions[2]!, loadOptions[0]!) + expectSameSubsetRequest(loadOptions[3]!, loadOptions[1]!) + + if (delivery === `resolve`) { + expect(replayLoads).toHaveLength(2) + installReplayRows() + replayLoads[0]?.resolve() + replayLoads[1]?.resolve() + } else if (delivery === `reject`) { + expect(replayLoads).toHaveLength(2) + replayLoads[0]?.reject(new Error(`ordered replay failed`)) + replayLoads[1]?.resolve() + } else { + expect(replayLoads).toEqual([]) + } + await flushPromises() + expect(collection.toArray.map(({ id }) => id).sort()).toEqual( + succeeds ? [...expectedIds].sort() : [], + ) + + const batchCount = batches.length + subscription.requestLimitedSnapshot({ + orderBy, + limit: 1, + minValues: [direction === `asc` ? 2 : 1], + }) + expect(loadOptions[4]).toMatchObject({ + offset: 2, + cursor: { + lastKey: succeeds ? expectedIds[1] : initialIds[1], + }, + }) + if (succeeds) expect(batches.at(-1)).toEqual([]) + else expect(batches).toHaveLength(batchCount) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`keeps private row tracking through consecutive failed replays`, async () => { + await runReplayScenario({ + initialRows: [ + { id: `one`, value: -2 }, + { id: `two`, value: 0 }, + ], + demandIds: [`two`], + attempts: [ + { + loads: [ + { + demandId: `two`, + rows: [], + outcome: `reject`, + writeBeforeSettlement: true, + }, + ], + }, + { + loads: [ + { + demandId: `two`, + rows: [], + outcome: `reject`, + writeBeforeSettlement: true, + }, + ], + }, + { + loads: [ + { + demandId: `two`, + rows: [{ id: `two`, value: -2 }], + outcome: `resolve`, + writeBeforeSettlement: true, + }, + ], + }, + ], + settlementOrder: [0, 2, 1], + settlementPhases: [0, 1, 2], + afterSettlement: [], + }) + }) + + it(`retains a successful retry after retiring its failed peer`, async () => { + await runReplayScenario({ + initialRows: [{ id: `two`, value: 0 }], + demandIds: [`one`, `two`], + attempts: [ + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 1 }], + outcome: `reject`, + writeBeforeSettlement: false, + }, + { + demandId: `two`, + rows: [{ id: `two`, value: -1 }], + outcome: `reject`, + writeBeforeSettlement: false, + }, + ], + }, + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 2 }], + outcome: `reject`, + writeBeforeSettlement: true, + }, + { + demandId: `two`, + rows: [{ id: `two`, value: 1 }], + outcome: `resolve`, + writeBeforeSettlement: false, + }, + ], + }, + ], + settlementOrder: [3, 1, 0, 2], + settlementPhases: [0, 0, 1, 1], + releaseOnLastAttempt: `one`, + afterSettlement: [{ type: `request`, demandId: `one` }], + }) + }) + + it(`does not republish an identical snapshot after a synchronous replay failure`, async () => { + await runSequentialReplayScenario({ + initialRows: [{ id: `one`, value: 0 }], + loads: [ + { rows: [], outcome: `throw` }, + { rows: [{ id: `one`, value: 0 }], outcome: `return` }, + ], + }) + }) + + it(`keeps a same-key source replacement private after a failed replay`, async () => { + await runReplayScenario({ + initialRows: [{ id: `one`, value: 1 }], + demandIds: [`one`], + attempts: [ + { + loads: [{ demandId: `one`, rows: [], outcome: `reject` }], + }, + ], + settlementOrder: [0], + settlementPhases: [0], + afterSettlement: [{ type: `put`, row: { id: `one`, value: 2 } }], + }) + }) + + it(`does not let an unpublished truncate delete suppress a later insert`, async () => { + await runReplayScenario({ + initialRows: [], + demandIds: [`two`, `one`], + attempts: [ + { + loads: [ + { + demandId: `two`, + rows: [{ id: `two`, value: -1 }], + outcome: `resolve`, + }, + { demandId: `one`, rows: [], outcome: `reject` }, + ], + }, + { + loads: [ + { demandId: `two`, rows: [], outcome: `resolve` }, + { + demandId: `one`, + rows: [{ id: `one`, value: -1 }], + outcome: `resolve`, + }, + ], + }, + ], + settlementOrder: [0, 1, 2, 3], + settlementPhases: [0, 0, 1, 1], + afterSettlement: [{ type: `put`, row: { id: `two`, value: 1 } }], + }) + }) + + it(`lets the newest successful replay replace an older failed replay`, async () => { + await runReplayScenario({ + initialRows: [{ id: `one`, value: 1 }], + demandIds: [`one`], + attempts: [ + { + loads: [{ demandId: `one`, rows: [], outcome: `reject` }], + }, + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 2 }], + outcome: `resolve`, + }, + ], + }, + ], + settlementOrder: [1, 0], + settlementPhases: [1, 1], + afterSettlement: [], + }) + }) + + it(`ignores an obsolete replay that settles after the newest replay`, async () => { + await runReplayScenario({ + initialRows: [{ id: `one`, value: 0 }], + demandIds: [`one`], + attempts: [ + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 1 }], + outcome: `resolve`, + }, + ], + }, + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 2 }], + outcome: `resolve`, + }, + ], + }, + ], + settlementOrder: [1, 0], + settlementPhases: [1, 1], + afterSettlement: [], + }) + }) + + it(`releases every successful overlapping replay acquisition`, async () => { + await runReplayScenario({ + initialRows: [], + demandIds: [`one`], + attempts: [ + { + loads: [{ demandId: `one`, rows: [], outcome: `resolve` }], + }, + { + loads: [{ demandId: `one`, rows: [], outcome: `resolve` }], + }, + ], + settlementOrder: [1, 0], + settlementPhases: [1, 1], + afterSettlement: [], + }) + }) + + it(`uses the newest complete multi-demand replay`, async () => { + await runReplayScenario({ + initialRows: [{ id: `one`, value: 1 }], + demandIds: [`one`, `two`], + attempts: [ + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 2 }], + outcome: `resolve`, + }, + { demandId: `two`, rows: [], outcome: `reject` }, + ], + }, + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 3 }], + outcome: `resolve`, + }, + { + demandId: `two`, + rows: [{ id: `two`, value: 4 }], + outcome: `resolve`, + }, + ], + }, + ], + settlementOrder: [2, 3, 0, 1], + settlementPhases: [1, 1, 1, 1], + afterSettlement: [], + }) + }) + + it(`retains applied rows after their replay demand is released`, async () => { + await runReplayScenario({ + initialRows: [{ id: `two`, value: 0 }], + demandIds: [`one`, `two`], + attempts: [ + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 1 }], + outcome: `reject`, + writeBeforeSettlement: true, + }, + { + demandId: `two`, + rows: [{ id: `two`, value: 2 }], + outcome: `resolve`, + }, + ], + }, + ], + settlementOrder: [0, 1], + settlementPhases: [0, 0], + releaseOnLastAttempt: `one`, + afterSettlement: [{ type: `request`, demandId: `one` }], + }) + }) + + it(`refreshes a retained row when its final released demand is reacquired`, async () => { + // Reduced from the fixed replay corpus after removing release-time pruning. + await runReplayScenario({ + initialRows: [{ id: `one`, value: 0 }], + demandIds: [`one`], + attempts: [ + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 1 }], + outcome: `resolve`, + writeBeforeSettlement: true, + }, + ], + }, + ], + settlementOrder: [0], + settlementPhases: [0], + releaseOnLastAttempt: `one`, + afterSettlement: [{ type: `request`, demandId: `one` }], + }) + }) + + it(`replaces a retained snapshot with a later empty replay`, async () => { + await runReplayScenario({ + initialRows: [{ id: `one`, value: 1 }], + demandIds: [`one`], + attempts: [ + { + loads: [{ demandId: `one`, rows: [], outcome: `reject` }], + }, + { + loads: [{ demandId: `one`, rows: [], outcome: `resolve` }], + }, + ], + settlementOrder: [0, 1], + settlementPhases: [0, 1], + afterSettlement: [], + }) + }) + + it.each([ + ...[`current`, `pending`].flatMap((scope) => + [`throw`, `reject`].map((failureMode) => ({ scope, failureMode })), + ), + // An async rejection runs after setup; only a sync failure can be held + // by an attempt whose setup stack has not returned yet. + { scope: `setup`, failureMode: `throw` }, + ])( + `drops released failure references: $scope, $failureMode`, + async ({ scope, failureMode }) => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const pendingPeer = createDeferred() + const replacement = createDeferred() + const failure = new Error(`failed owner`) + let loads = 0 + const collection = createCollection({ + id: `released-replay-failure-${scope}-${failureMode}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loads++ + if (loads <= 2) return true + if (loads === 3) { + if (failureMode === `throw`) throw failure + return Promise.reject(failure) + } + if (scope === `setup` && loads === 4) { + expect(retainedFailures()).toEqual([failure]) + subscription.requestSnapshot({ + where: peerWhere, + optimizedOnly: false, + }) + return pendingPeer.promise + } + if (scope === `setup` && loads === 5) { + begin() + truncate() + commit() + subscription.releaseSnapshot(failedWhere) + } + return loads === 4 ? pendingPeer.promise : replacement.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const failedWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`one`), + ]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const retainedFailures = () => { + // Narrow retention witness for old and new representations. Follow + // stored replay frames, not a captured map that the source discarded. + type Frame = { failures?: Map } + const session = ( + subscription as unknown as { + truncateReplaySession: Frame & { + currentAttempt: Frame + attempts?: Set + pending?: Set<{ attempt: Frame }> + } + } + ).truncateReplaySession + const frames = new Set([ + session, + session.currentAttempt, + ...(session.attempts ?? []), + ...[...(session.pending ?? [])].map(({ attempt }) => attempt), + ]) + return [...frames].flatMap((frame) => [ + ...(frame.failures?.values() ?? []), + ]) + } + const replaySource = async () => { + begin() + truncate() + commit() + await flushPromises() + } + + try { + subscription.requestSnapshot({ + where: failedWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ where: peerWhere, optimizedOnly: false }) + await replaySource() + // This is a retained-state witness, not a row oracle or GC benchmark. + // Public rows cannot reveal a released owner held by an old error map. + // Adapt this witness if the replay representation changes again. + if (scope !== `setup`) expect(retainedFailures()).toEqual([failure]) + if (scope === `pending`) await replaySource() + subscription.releaseSnapshot(failedWhere) + expect(retainedFailures()).toEqual([]) + expect(subscription.status).toBe(`loadingSubset`) + replacement.resolve() + pendingPeer.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + } finally { + replacement.resolve() + pendingPeer.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`does not start a queued replay after a newer truncate supersedes it`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const loadSignals: Array = [] + const collection = createCollection({ + id: `superseded-before-replay-setup`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: ({ signal }) => { + loadSignals.push(signal) + return loadSignals.length === 1 ? true : replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + + begin() + truncate() + commit() + begin() + truncate() + commit() + await flushPromises() + + expect(loadSignals).toHaveLength(2) + expect(loadSignals[0]?.aborted).toBe(true) + expect(loadSignals[1]?.aborted).toBe(false) + } finally { + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it.each([ + `replay`, + `additional demand`, + `additional pending demand`, + ] as const)( + `aborts a %s acquisition before a reentrant newer truncate starts`, + async (start) => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const olderReplay = createDeferred() + const newerReplay = createDeferred() + const predecessor = createDeferred() + const replaySignals: Array = [] + let loadCount = 0 + const collection = createCollection({ + id: `reentrant-newer-replay`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: ({ signal }) => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 0 } }) + commit() + return true + } + + if (loadCount === 2 && start !== `replay`) { + if (start === `additional pending demand`) { + return predecessor.promise + } + // A failed replay retains its public baseline after setup and + // all participants finish. Start the extra demand in that gap. + throw new Error(`retain the failed replay`) + } + replaySignals.push(signal) + if (loadCount === (start === `replay` ? 2 : 3)) { + begin() + truncate() + commit() + return olderReplay.promise + } + return newerReplay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + }) + + const install = (value: number) => { + begin() + write({ + type: collection.has(`one`) ? `update` : `insert`, + value: { id: `one`, value }, + }) + commit() + } + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 0 }]) + + begin() + truncate() + commit() + await flushPromises() + + if (start !== `replay`) { + if (start === `additional demand`) { + expect(subscription.lastError).toEqual( + new Error(`retain the failed replay`), + ) + } + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 0 }]) + subscription.requestSnapshot({ + where: new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]), + optimizedOnly: false, + }) + await flushPromises() + } + + expect(replaySignals).toHaveLength(start === `replay` ? 2 : 3) + expect(replaySignals[0]?.aborted).toBe(true) + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 0 }]) + + install(2) + newerReplay.resolve() + await flushPromises() + if (start === `additional pending demand`) { + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 0 }]) + predecessor.resolve() + await flushPromises() + // The returning extra demand joined the retained old attempt. Its + // transport still holds publication after that attempt's prior work. + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 0 }]) + expect(subscription.status).toBe(`loadingSubset`) + } else { + // A startup superseded before return does not hold publication. An + // ordinary demand still owns its separate readiness participant. + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + expect(subscription.status).toBe( + start === `replay` ? `ready` : `loadingSubset`, + ) + } + if (!replaySignals[0]?.aborted) install(1) + olderReplay.resolve() + await flushPromises() + + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + expect(subscription.status).toBe(`ready`) + } finally { + predecessor.resolve() + olderReplay.resolve() + newerReplay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`does not retain replay work registered after its demand is released`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const releasedReplay = createDeferred() + let loadCount = 0 + const collection = createCollection({ + id: `released-during-replay-start`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + write({ type: `insert`, value: { id: `two`, value: 1 } }) + commit() + operations.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount <= 2) return true + if (loadCount === 3) { + subscription.releaseSnapshot(firstWhere) + return releasedReplay.promise + } + begin() + write({ type: `insert`, value: { id: `two`, value: 2 } }) + commit() + return Promise.resolve() + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + }) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + + begin() + truncate() + commit() + await flushPromises() + + expect(subscription.status).toBe(`ready`) + expect(sortedRows(visible)).toEqual([{ id: `two`, value: 2 }]) + } finally { + releasedReplay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`finishes replay state and surfaces an async subscriber failure`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const listenerFailure = new Error(`replay subscriber failed`) + const queuedMicrotasks: Array = [] + let loadCount = 0 + let rejectReplacement = false + const collection = createCollection({ + id: `async-replay-subscriber-failure`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + return true + } + return replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + if (rejectReplacement) throw listenerFailure + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + + begin() + write({ type: `insert`, value: { id: `one`, value: 2 } }) + commit() + rejectReplacement = true + const queueMicrotaskSpy = vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => queuedMicrotasks.push(callback)) + try { + replay.resolve() + await flushPromises() + + expect(subscription.status).toBe(`ready`) + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + expect(queuedMicrotasks).toHaveLength(1) + expect(() => queuedMicrotasks[0]!()).toThrow(listenerFailure) + } finally { + queueMicrotaskSpy.mockRestore() + } + } finally { + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not start another demand for a replay superseded by reentrancy`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + let loadCount = 0 + let inSupersededSetup = false + let staleSecondDemandStarted = false + const collection = createCollection({ + id: `reentrant-multi-demand-replay`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + begin() + write({ type: `insert`, value: { id: `one`, value: 0 } }) + write({ type: `insert`, value: { id: `two`, value: 0 } }) + commit() + operations.markReady() + return { + loadSubset: (options) => { + loadCount++ + if (loadCount <= 2) return true + if (loadCount === 3) { + inSupersededSetup = true + queueMicrotask(() => { + inSupersededSetup = false + }) + begin() + truncate() + commit() + return true + } + if (inSupersededSetup) { + staleSecondDemandStarted = true + begin() + write({ type: `insert`, value: { id: `two`, value: 1 } }) + commit() + return true + } + if (options.where === firstWhere) { + begin() + write({ type: `insert`, value: { id: `one`, value: 2 } }) + commit() + } + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + }) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + begin() + truncate() + commit() + await flushPromises() + + expect(staleSecondDemandStarted).toBe(false) + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not start replay work retired by the loading transition`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const loads: Array = [] + const unloads: Array = [] + const collection = createCollection({ + id: `reentrant-status-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return loads.length === 1 ? true : replay.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + let releaseOnLoading = false + subscription.on(`status:loadingSubset`, () => { + if (releaseOnLoading) subscription.releaseSnapshot(where) + }) + + try { + subscription.requestSnapshot({ where, optimizedOnly: false }) + releaseOnLoading = true + begin() + truncate() + commit() + await flushPromises() + + expect(loads).toHaveLength(1) + expect(unloads).toEqual([loads[0]]) + expect(subscription.status).toBe(`ready`) + } finally { + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`releases replay work when its final publication callback throws`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const loads: Array = [] + const unloads: Array = [] + const listenerFailure = new Error(`release publication failed`) + let rejectReplacement = false + const collection = createCollection({ + id: `replay-release-callback-cleanup`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (loads.length === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + return true + } + return replay.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + truncateReplayPublication: { + start: () => {}, + succeed: () => { + if (rejectReplacement) throw listenerFailure + }, + }, + }) + + try { + subscription.requestSnapshot({ where, optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + begin() + write({ type: `insert`, value: { id: `one`, value: 2 } }) + commit() + rejectReplacement = true + + expect(() => subscription.releaseSnapshot(where)).toThrow(listenerFailure) + expect(loads[1]?.signal?.aborted).toBe(true) + expect(unloads.map((options) => loads.indexOf(options))).toEqual([0, 1]) + expect(subscription.status).toBe(`ready`) + } finally { + rejectReplacement = false + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not retain a demand released during adapter startup`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const releasedLoad = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const unloads: Array = [] + const collection = createCollection({ + id: `reentrant-new-demand-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (loads.length === 1) return true + if (loads.length === 2) return replay.promise + subscription.releaseSnapshot(secondWhere) + return releasedLoad.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + begin() + truncate() + commit() + await flushPromises() + + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + replay.resolve() + await flushPromises() + + expect(loads).toHaveLength(3) + expect(loads[2]?.signal?.aborted).toBe(true) + expect(unloads).toContain(loads[2]) + expect(subscription.status).toBe(`ready`) + expect(subscription.pendingTruncateReplacement).toBeUndefined() + } finally { + releasedLoad.resolve() + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`publishes a replay replacement before reporting ready`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + let loadCount = 0 + const collection = createCollection({ + id: `replay-ready-after-publication`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + return true + } + return replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const readyValues: Array = [] + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + }) + subscription.on(`status:ready`, () => { + readyValues.push(visible.get(`one`)?.value) + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + begin() + write({ type: `insert`, value: { id: `one`, value: 2 } }) + commit() + + replay.resolve() + await flushPromises() + + expect(readyValues).toEqual([2]) + expect(visible.get(`one`)?.value).toBe(2) + expect(subscription.status).toBe(`ready`) + } finally { + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`ignores a synchronous replay failure after its demand releases itself`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const releasedFailure = new Error(`released replay load failed`) + const reportedErrors: Array = [] + let loadCount = 0 + const collection = createCollection({ + id: `released-sync-failure`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + write({ type: `insert`, value: { id: `two`, value: 1 } }) + commit() + operations.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount <= 2) return true + if (loadCount === 3) { + subscription.releaseSnapshot(firstWhere) + throw releasedFailure + } + begin() + write({ type: `insert`, value: { id: `two`, value: 2 } }) + commit() + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + }) + subscription.on(`loadSubset:error`, ({ error }) => { + reportedErrors.push(error) + }) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + begin() + truncate() + commit() + await flushPromises() + + expect(reportedErrors).not.toContain(releasedFailure) + expect(sortedRows(visible)).toEqual([{ id: `two`, value: 2 }]) + expect(subscription.status).toBe(`ready`) + expect(subscription.pendingTruncateReplacement).toBeUndefined() + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it.each( + [false, true].flatMap((releaseDemand) => + [false, true].map((failRelease) => ({ releaseDemand, failRelease })), + ), + )( + `releases before reacquisition with releaseDemand=$releaseDemand and failRelease=$failRelease`, + async ({ releaseDemand, failRelease }) => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const loads: Array = [] + const unloads: Array = [] + let reentered = false + const releaseFailure = new Error(`old replay lease release failed`) + const collection = createCollection({ + id: `reentrant-replay-lease-replacement`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[0] && !reentered) { + reentered = true + if (releaseDemand) subscription.releaseSnapshot(where) + if (failRelease) throw releaseFailure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + try { + subscription.requestSnapshot({ where, optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + + const reacquires = !releaseDemand && !failRelease + expect(loads).toHaveLength(reacquires ? 2 : 1) + // indexOf checks the exact options object, not a structurally equal copy. + expect(unloads.map((options) => loads.indexOf(options))).toEqual([0]) + if (reacquires) expect(loads[1]!.signal?.aborted).toBe(false) + if (failRelease) expect(subscription.lastError).toBe(releaseFailure) + subscription.unsubscribe() + // A failed release or retired logical demand never starts a replacement. + expect(unloads.map((options) => loads.indexOf(options))).toEqual( + reacquires ? [0, 1] : [0], + ) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`rejects replay completion with the exact reported adapter error`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replayLoad = createDeferred() + const failure = new Error(`exact replay failure`) + const reportedErrors: Array = [] + let loadCount = 0 + const collection = createCollection({ + id: `exact-replay-completion-error`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => (++loadCount === 1 ? true : replayLoad.promise), + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { + start: () => {}, + succeed: () => {}, + }, + }) + subscription.on(`loadSubset:error`, ({ error }) => { + reportedErrors.push(error) + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + const replacement = subscription.pendingTruncateReplacement + expect(replacement).toBeInstanceOf(Promise) + + replayLoad.reject(failure) + + await expect(replacement).rejects.toBe(failure) + expect(subscription.lastError).toBe(failure) + expect(reportedErrors).toEqual([failure]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`normalizes one primitive rejection for every observer of a shared replay`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replayLoad = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const reportedErrors: Array<{ + options: LoadSubsetOptions + error: unknown + }> = [] + let loadCount = 0 + const collection = createCollection({ + id: `shared-primitive-replay-error`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + loadCount++ + return loadCount <= 2 ? true : replayLoad.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { + start: () => {}, + succeed: () => {}, + }, + }) + subscription.on(`loadSubset:error`, ({ options, error }) => { + reportedErrors.push({ options, error }) + }) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + begin() + truncate() + commit() + await flushPromises() + const replacement = subscription.pendingTruncateReplacement + expect(replacement).toBeInstanceOf(Promise) + expect(loads.map(({ where }) => where)).toEqual([ + firstWhere, + secondWhere, + firstWhere, + secondWhere, + ]) + + replayLoad.reject(undefined) + + let replacementError: unknown + try { + await replacement + } catch (error) { + replacementError = error + } + expect(reportedErrors).toHaveLength(2) + expect(reportedErrors.map(({ options }) => options)).toEqual([ + loads[2], + loads[3], + ]) + expect(reportedErrors[0]?.error).toBeInstanceOf(Error) + expect(reportedErrors[1]?.error).toBe(reportedErrors[0]?.error) + expect(subscription.lastError).toBe(reportedErrors[0]?.error) + expect(replacementError).toBe(reportedErrors[0]?.error) + } finally { + replayLoad.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it.each([`none`, `first`, `second`, `both`] as const)( + `normalizes one primitive rejection for ordinary shared loads after releasing %s demand`, + async (released) => { + const sharedLoad = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`two`), + ]) + const loads: Array = [] + const unloads: Array = [] + const reportedErrors: Array<{ + options: LoadSubsetOptions + error: unknown + }> = [] + const collection = createCollection({ + id: `shared-primitive-ordinary-error-${released}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return sharedLoad.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ options, error }) => { + reportedErrors.push({ options, error }) + }) + + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + expect(loads.map(({ where }) => where)).toEqual([ + firstWhere, + secondWhere, + ]) + + if (released === `first` || released === `both`) { + subscription.releaseSnapshot(firstWhere) + } + if (released === `second` || released === `both`) { + subscription.releaseSnapshot(secondWhere) + } + expect(loads[0]?.signal?.aborted).toBe( + released === `first` || released === `both`, + ) + expect(loads[1]?.signal?.aborted).toBe( + released === `second` || released === `both`, + ) + + sharedLoad.reject(undefined) + await flushPromises() + + const activeLoads = loads.filter((load) => !load.signal?.aborted) + expect(reportedErrors.map(({ options }) => options)).toEqual( + activeLoads, + ) + if (activeLoads.length > 0) { + expect(reportedErrors[0]?.error).toBeInstanceOf(Error) + for (const { error } of reportedErrors) { + expect(error).toBe(reportedErrors[0]?.error) + } + expect(subscription.lastError).toBe(reportedErrors[0]?.error) + } else { + expect(subscription.lastError).toBeUndefined() + } + expect(subscription.status).toBe(`ready`) + } finally { + sharedLoad.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(loads.length) + for (const load of loads) { + expect(unloads.filter((options) => options === load)).toHaveLength(1) + } + }, + ) + + it.each([`first`, `second`, `both`] as const)( + `settles a shared replay rejection after releasing %s demand`, + async (released) => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replayLoad = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`two`), + ]) + const loads: Array = [] + const unloads: Array = [] + const reportedErrors: Array<{ + options: LoadSubsetOptions + error: unknown + }> = [] + let loadCount = 0 + let replayStarts = 0 + let replaySuccesses = 0 + const collection = createCollection({ + id: `released-shared-primitive-replay-error-${released}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + loadCount++ + return loadCount <= 2 ? true : replayLoad.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { + start: () => replayStarts++, + succeed: () => replaySuccesses++, + }, + }) + subscription.on(`loadSubset:error`, ({ options, error }) => { + reportedErrors.push({ options, error }) + }) + + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + begin() + truncate() + commit() + await flushPromises() + const replacement = subscription.pendingTruncateReplacement + expect(replacement).toBeInstanceOf(Promise) + expect(loads.map(({ where }) => where)).toEqual([ + firstWhere, + secondWhere, + firstWhere, + secondWhere, + ]) + const settlement = replacement!.then( + () => ({ status: `resolved` as const }), + (error: unknown) => ({ status: `rejected` as const, error }), + ) + expect({ replayStarts, replaySuccesses }).toEqual({ + replayStarts: 1, + replaySuccesses: 0, + }) + + if (released === `first` || released === `both`) { + subscription.releaseSnapshot(firstWhere) + } + if (released === `second` || released === `both`) { + subscription.releaseSnapshot(secondWhere) + } + expect(loads[2]?.signal?.aborted).toBe( + released === `first` || released === `both`, + ) + expect(loads[3]?.signal?.aborted).toBe( + released === `second` || released === `both`, + ) + + if (released === `both`) { + const result = await settlement + expect(result).toMatchObject({ + status: `rejected`, + error: { name: `AbortError` }, + }) + expect(reportedErrors).toEqual([]) + expect(subscription.lastError).toBeUndefined() + expect(subscription.status).toBe(`ready`) + expect(replaySuccesses).toBe(1) + } else { + expect(subscription.pendingTruncateReplacement).toBe(replacement) + expect(subscription.status).toBe(`loadingSubset`) + replayLoad.reject(undefined) + const result = await settlement + expect(result.status).toBe(`rejected`) + + const activeIndex = released === `first` ? 3 : 2 + expect(reportedErrors).toHaveLength(1) + expect(reportedErrors[0]?.options).toBe(loads[activeIndex]) + expect(reportedErrors[0]?.error).toBeInstanceOf(Error) + expect(subscription.lastError).toBe(reportedErrors[0]?.error) + expect(result).toMatchObject({ + status: `rejected`, + error: reportedErrors[0]?.error, + }) + } + } finally { + replayLoad.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(loads.length) + for (const load of loads) { + expect(unloads.filter((options) => options === load)).toHaveLength(1) + } + expect(replayStarts).toBe(1) + expect(replaySuccesses).toBe(released === `both` ? 1 : 0) + }, + ) + + it(`ignores a retired demand's replay failure once surviving demand succeeds`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const firstReplay = createDeferred() + const secondReplay = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const failure = new Error(`retired demand failed`) + let replaySuccesses = 0 + const collection = createCollection({ + id: `retired-replay-failure`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (loads.length <= 2) return true + return loads.length === 3 + ? firstReplay.promise + : secondReplay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { + start: () => {}, + succeed: () => replaySuccesses++, + }, + }) + + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + begin() + truncate() + commit() + await flushPromises() + const replacement = subscription.pendingTruncateReplacement + expect(replacement).toBeInstanceOf(Promise) + + firstReplay.reject(failure) + await flushPromises() + subscription.releaseSnapshot(firstWhere) + secondReplay.resolve() + + await expect(replacement).resolves.toBeUndefined() + expect(replaySuccesses).toBe(1) + expect(subscription.status).toBe(`ready`) + } finally { + firstReplay.resolve() + secondReplay.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`retains successful peer rows after failed replay demand retires`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const failed = createDeferred() + const successful = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const unloads: Array = [] + const visible = new Map() + const batches: Array> = [] + let survivingRows: Array = [] + const collection = createCollection({ + id: `settled-replay-peer`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + const id = options.where === firstWhere ? `one` : `two` + begin() + write({ + type: `insert`, + value: { id, value: loads.length <= 2 ? 1 : 2 }, + }) + commit() + if (loads.length <= 2) return true + return id === `one` ? failed.promise : successful.promise + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges((changes) => { + batches.push(recordPublishedChanges(visible, changes)) + }) + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + expect(sortedRows(visible)).toEqual([ + { id: `one`, value: 1 }, + { id: `two`, value: 1 }, + ]) + begin() + truncate() + commit() + await flushPromises() + expect(loads).toHaveLength(4) + failed.reject(new Error(`first demand replay failed`)) + successful.resolve() + await flushPromises() + expect(sortedRows(visible)).toEqual([ + { id: `one`, value: 1 }, + { id: `two`, value: 1 }, + ]) + subscription.releaseSnapshot(firstWhere) + await flushPromises() + // Retirement leaves only the successful demand. Its replacement rows + // must survive failure handling for the now-retired peer. + survivingRows = sortedRows(visible) + subscription.releaseSnapshot(secondWhere) + // Outside replay, release ends acquisition ownership; this adapter does + // not evict its cached rows. The still-live subscriber observes deletion + // when the source actually removes the row. + expect(sortedRows(visible)).toEqual([ + { id: `one`, value: 2 }, + { id: `two`, value: 2 }, + ]) + begin() + write({ type: `delete`, key: `two` }) + commit() + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + } finally { + failed.resolve() + successful.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(4) + for (const load of loads) { + expect(unloads.filter((options) => options === load)).toHaveLength(1) + } + expect(survivingRows).toEqual([ + { id: `one`, value: 2 }, + { id: `two`, value: 2 }, + ]) + }) + + it(`keeps replay completion failure separate from a peer release failure`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const failed = createDeferred() + const peer = createDeferred() + const replayFailure = new Error(`replay failed`) + const releaseFailure = new Error(`peer release failed`) + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const unloadAttempts: Array = [] + const unloaded: Array = [] + let failRelease = false + const succeeded = vi.fn() + const collection = createCollection({ + id: `replay-error-ownership`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (loads.length <= 2) return true + return options.where === firstWhere + ? failed.promise + : peer.promise + }, + unloadSubset: (options) => { + unloadAttempts.push(options) + if (failRelease && options === loads[3]) { + failRelease = false + throw releaseFailure + } + unloaded.push(options) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { start: () => {}, succeed: succeeded }, + }) + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: peerWhere }) + begin() + truncate() + commit() + await flushPromises() + expect(loads).toHaveLength(4) + const completion = subscription.pendingTruncateReplacement + expect(completion).toBeInstanceOf(Promise) + const observed = completion!.then( + () => ({ status: `resolved` as const }), + (error: unknown) => ({ status: `rejected` as const, error }), + ) + failed.reject(replayFailure) + await flushPromises() + failRelease = true + expect(() => subscription.releaseSnapshot(peerWhere)).toThrow( + releaseFailure, + ) + expect(succeeded).not.toHaveBeenCalled() + await expect(observed).resolves.toEqual({ + status: `rejected`, + error: replayFailure, + }) + const result = await observed + expect(`error` in result ? result.error : undefined).toBe(replayFailure) + peer.resolve() + await flushPromises() + await expect(observed).resolves.toEqual({ + status: `rejected`, + error: replayFailure, + }) + } finally { + failed.resolve() + peer.resolve() + failRelease = false + subscription.unsubscribe() + await collection.cleanup() + } + expect( + unloadAttempts.filter((options) => options === loads[3]), + ).toHaveLength(1) + expect(unloaded).toHaveLength(3) + for (const load of loads) { + expect(unloaded.filter((options) => options === load)).toHaveLength( + load === loads[3] ? 0 : 1, + ) + } + }) + + it(`waits for a new async demand acquired while unloading a replay lease`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const nested = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const nestedWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const unloads: Array = [] + const ready = vi.fn() + const visible = new Map() + const publications: Array> = [] + const collection = createCollection({ + id: `replay-new-demand-readiness`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + begin() + write({ + type: `insert`, + value: { + id: options.where === nestedWhere ? `two` : `one`, + value: loads.length === 1 ? 1 : 2, + }, + }) + commit() + if (loads.length === 1) return true + return options.where === nestedWhere + ? nested.promise + : replay.promise + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[0]) { + subscription.requestSnapshot({ where: nestedWhere }) + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + recordPublishedChanges(visible, changes) + publications.push(sortedRows(visible)) + }, + { includeInitialState: false }, + ) + try { + subscription.requestSnapshot({ where: firstWhere }) + publications.length = 0 + subscription.on(`status:ready`, ready) + begin() + truncate() + commit() + await flushPromises() + expect(loads.map(({ where }) => where)).toEqual([ + firstWhere, + nestedWhere, + firstWhere, + ]) + expect(unloads).toEqual([loads[0]]) + const completion = subscription.pendingTruncateReplacement + expect(completion).toBeInstanceOf(Promise) + const settled = vi.fn() + void completion!.then(settled, settled) + replay.resolve() + await flushPromises() + expect(subscription.status).not.toBe(`ready`) + expect(ready).not.toHaveBeenCalled() + expect(settled).not.toHaveBeenCalled() + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 1 }]) + expect(publications).toEqual([]) + nested.resolve() + await completion + await flushPromises() + expect(subscription.status).toBe(`ready`) + expect(ready).toHaveBeenCalledTimes(1) + expect(settled).toHaveBeenCalledTimes(1) + expect(sortedRows(visible)).toEqual([ + { id: `one`, value: 2 }, + { id: `two`, value: 2 }, + ]) + expect(publications).toEqual([ + [ + { id: `one`, value: 2 }, + { id: `two`, value: 2 }, + ], + ]) + } finally { + replay.resolve() + nested.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(3) + for (const load of loads) { + expect(unloads.filter((options) => options === load)).toHaveLength(1) + } + }) + + it.each([`after-release`, `during-ready`, `during-unload`] as const)( + `reacquires a final released replay demand %s without waiting for obsolete work`, + async (reacquireTiming) => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const replayLoad = createDeferred() + const reacquiredLoad = createDeferred() + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const loads: Array = [] + const unloads: Array = [] + let reacquireOnReady = false + let reacquireInUnload = false + const collection = createCollection({ + id: `final-replay-reacquire-${reacquireTiming}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (loads.length === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + return true + } + if (loads.length === 2) { + begin() + write({ type: `insert`, value: { id: `one`, value: 2 } }) + commit() + return replayLoad.promise + } + return reacquireTiming === `during-unload` + ? reacquiredLoad.promise + : true + }, + unloadSubset: (options) => { + unloads.push(options) + if (reacquireInUnload && options === loads[1]) { + reacquireInUnload = false + subscription.requestSnapshot({ where }) + } + }, + } + }, + }, + }) + const visible = new Map() + const batches: Array> = [] + const subscription: CollectionSubscription = collection.subscribeChanges( + (changes) => { + batches.push(recordPublishedChanges(visible, changes)) + }, + ) + const readyRows: Array> = [] + subscription.on(`status:ready`, () => { + readyRows.push(sortedRows(visible)) + if (reacquireOnReady) { + reacquireOnReady = false + subscription.requestSnapshot({ where }) + } + }) + + try { + subscription.requestSnapshot({ where }) + begin() + truncate() + commit() + await flushPromises() + const replacement = subscription.pendingTruncateReplacement + expect(replacement).toBeInstanceOf(Promise) + const settlement = replacement!.then( + () => ({ status: `resolved` as const }), + (error: unknown) => ({ status: `rejected` as const, error }), + ) + + // Release has no synthetic delete callback. Reenter from its actual + // ready notification instead; the old replay is already retired then. + reacquireOnReady = reacquireTiming === `during-ready` + reacquireInUnload = reacquireTiming === `during-unload` + subscription.releaseSnapshot(where) + if (reacquireTiming === `after-release`) { + await expect(settlement).resolves.toMatchObject({ + status: `rejected`, + error: { name: `AbortError` }, + }) + subscription.requestSnapshot({ where }) + } else if (reacquireTiming === `during-unload`) { + let settled = false + void settlement.then(() => { + settled = true + }) + const batchesBeforePendingFlush = batches.length + await flushPromises() + expect(settled).toBe(false) + expect(subscription.status).not.toBe(`ready`) + expect(readyRows).toEqual([]) + expect(batches).toHaveLength(batchesBeforePendingFlush) + const batchesBeforeObsoleteSettlement = batches.length + replayLoad.resolve() + await flushPromises() + expect(settled).toBe(false) + expect(subscription.status).not.toBe(`ready`) + expect(readyRows).toEqual([]) + expect(batches).toHaveLength(batchesBeforeObsoleteSettlement) + reacquiredLoad.resolve() + await expect(settlement).resolves.toEqual({ status: `resolved` }) + } else { + expect(subscription.pendingTruncateReplacement).toBeUndefined() + await expect(settlement).resolves.toMatchObject({ + status: `rejected`, + error: { name: `AbortError` }, + }) + } + + expect(subscription.pendingTruncateReplacement).toBeUndefined() + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + expect(sortedChanges(batches[0]!)).toEqual([ + { type: `insert`, key: `one`, value: { id: `one`, value: 1 } }, + ]) + expect(sortedChanges(batches.at(-1)!)).toEqual([ + { + type: `update`, + key: `one`, + value: { id: `one`, value: 2 }, + previousValue: { id: `one`, value: 1 }, + }, + ]) + expect(batches.filter((batch) => batch.length > 0)).toHaveLength(2) + expect(loads).toHaveLength(3) + expect(loads.map(({ where: requestWhere }) => requestWhere)).toEqual([ + where, + where, + where, + ]) + if (reacquireTiming === `during-unload`) { + expect(readyRows).toEqual([[{ id: `one`, value: 2 }]]) + } + } finally { + replayLoad.resolve() + reacquiredLoad.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(loads.length) + for (const load of loads) { + expect(unloads.filter((options) => options === load)).toHaveLength(1) + } + }, + ) + + it(`does not start replacement work after release unsubscribes`, () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const loads: Array = [] + const unloads: Array = [] + const collection = createCollection({ + id: `reentrant-replacement-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + subscription.unsubscribe() + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + let release: (() => void) | undefined + subscription.requestSnapshot({ + where, + optimizedOnly: false, + onLoadSubsetResult: (_result, _options, releaseDemand) => { + release = releaseDemand + }, + }) + expect(release).toBeTypeOf(`function`) + release!() + expect( + subscription.requestSnapshot({ + where, + optimizedOnly: false, + }), + ).toBe(false) + + expect(loads).toHaveLength(1) + expect(unloads).toEqual([loads[0]]) + }) + + it.each([`generic`, `specific`] as const)( + `does not emit a stale specific status after reentrant release from a %s listener`, + async (reentryEvent) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const load = createDeferred() + const collection = createCollection({ + id: `reentrant-specific-status`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + operations.markReady() + return { + loadSubset: () => load.promise, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const observed: Array<{ event: string; current: string }> = [] + if (reentryEvent === `generic`) { + subscription.on(`status:change`, ({ status }) => { + if (status === `loadingSubset`) subscription.releaseSnapshot(where) + }) + } else { + subscription.on(`status:loadingSubset`, () => { + subscription.releaseSnapshot(where) + }) + } + subscription.on(`status:loadingSubset`, ({ status }) => { + observed.push({ event: status, current: subscription.status }) + }) + subscription.on(`status:ready`, ({ status }) => { + observed.push({ event: status, current: subscription.status }) + }) + + try { + subscription.requestSnapshot({ where, optimizedOnly: false }) + expect(observed).toEqual([{ event: `ready`, current: `ready` }]) + } finally { + load.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each([`generic`, `specific`] as const)( + `does not resume an obsolete status transition after %s-listener ABA reentry`, + async (reentryEvent) => { + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`two`), + ]) + const load = createDeferred() + const collection = createCollection({ + id: `reentrant-status-aba-${reentryEvent}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + operations.markReady() + return { + loadSubset: () => load.promise, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const trace: Array = [] + let reentered = false + const reenter = () => { + if (reentered) return + reentered = true + subscription.releaseSnapshot(firstWhere) + subscription.requestSnapshot({ where: secondWhere }) + } + if (reentryEvent === `generic`) { + subscription.on(`status:change`, ({ status }) => { + if (status === `loadingSubset`) reenter() + }) + } else { + subscription.on(`status:loadingSubset`, reenter) + } + subscription.on(`status:change`, ({ previousStatus, status }) => { + trace.push( + `generic:${previousStatus}->${status}:${subscription.status}`, + ) + }) + subscription.on(`status:loadingSubset`, () => { + trace.push(`specific:loadingSubset:${subscription.status}`) + }) + + try { + subscription.requestSnapshot({ where: firstWhere }) + + expect(trace).toEqual( + reentryEvent === `generic` + ? [ + `generic:loadingSubset->ready:ready`, + `generic:ready->loadingSubset:loadingSubset`, + `specific:loadingSubset:loadingSubset`, + ] + : [ + `generic:ready->loadingSubset:loadingSubset`, + `generic:loadingSubset->ready:ready`, + `generic:ready->loadingSubset:loadingSubset`, + `specific:loadingSubset:loadingSubset`, + ], + ) + } finally { + load.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each([`generic`, `specific`] as const)( + `stops %s status delivery when an earlier ready listener unsubscribes`, + async (reentryEvent) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const load = createDeferred() + const collection = createCollection({ + id: `reentrant-status-unsubscribe-${reentryEvent}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + operations.markReady() + return { + loadSubset: () => load.promise, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const trace: Array = [] + const unsubscribe = () => { + trace.push(`unsubscribe`) + subscription.unsubscribe() + } + if (reentryEvent === `generic`) { + subscription.on(`status:change`, ({ status }) => { + if (status === `ready`) unsubscribe() + }) + } else { + subscription.on(`status:ready`, unsubscribe) + } + subscription.on(`status:change`, ({ status }) => { + if (status === `ready`) trace.push(`late-generic`) + }) + subscription.on(`status:ready`, () => trace.push(`late-specific`)) + subscription.on(`unsubscribed`, () => trace.push(`unsubscribed`)) + + try { + subscription.requestSnapshot({ where }) + load.resolve() + await flushPromises() + + expect(trace).toEqual( + reentryEvent === `generic` + ? [`unsubscribe`, `unsubscribed`] + : [`late-generic`, `unsubscribe`, `unsubscribed`], + ) + } finally { + load.resolve() + await collection.cleanup() + } + }, + ) + + fcTest.prop([replayScenarioArbitrary], { + numRuns: generatedRuns, + seed: 1756, + })(`matches replay and ownership laws for a fixed seed`, runReplayScenario) + + fcTest.prop( + [replayScenarioArbitrary], + oracleRandomParameters( + generatedRuns, + replayConfig, + `subscription-replay.completion`, + ), + )( + `matches replay and ownership laws for a random or replayed seed`, + runReplayScenario, + ) + + fcTest.prop( + [sequentialReplayScenarioArbitrary], + oracleRandomParameters( + generatedRuns, + replayConfig, + `subscription-replay.sequential`, + ), + )( + `matches synchronous, asynchronous, and partial-failure replay laws`, + runSequentialReplayScenario, + ) + + fcTest.prop([cleanupRestartScenarioArbitrary], { + numRuns: generatedRuns, + seed: 1757, + })( + `isolates cleanup and restart sessions for a fixed seed`, + runCleanupRestartScenario, + ) + + fcTest.prop( + [cleanupRestartScenarioArbitrary], + oracleRandomParameters( + generatedRuns, + replayConfig, + `subscription-replay.restart`, + ), + )( + `isolates cleanup and restart sessions for a random or replayed seed`, + runCleanupRestartScenario, + ) + + fcTest.prop([fc.scheduler()], { numRuns: generatedRuns, seed: 1760 })( + `keeps same-tick obsolete and current replay settlements generation-safe`, + expectScheduledReplaySettlementIsGenerationSafe, + ) + + fcTest.prop( + [fc.scheduler()], + oracleRandomParameters( + generatedRuns, + replayConfig, + `subscription-replay.same-tick`, + ), + )( + `keeps same-tick replay settlements generation-safe for a random or replayed seed`, + expectScheduledReplaySettlementIsGenerationSafe, + ) + + fcTest.prop([sharedSubscriptionScenarioArbitrary], { + numRuns: generatedRuns, + seed: 1758, + })( + `keeps independent transport and logical ownership aligned for a fixed seed`, + runSharedSubscriptionScenario, + ) + + fcTest.prop( + [sharedSubscriptionScenarioArbitrary], + oracleRandomParameters( + generatedRuns, + replayConfig, + `subscription-replay.shared`, + ), + )( + `keeps independent transport and logical ownership aligned for a random or replayed seed`, + runSharedSubscriptionScenario, + ) + + fcTest.prop([optimisticReplayScenarioArbitrary], { + numRuns: generatedRuns, + seed: 1759, + })( + `preserves optimistic overlays across replay outcomes for a fixed seed`, + runOptimisticReplayScenario, + ) + + fcTest.prop( + [optimisticReplayScenarioArbitrary], + oracleRandomParameters( + generatedRuns, + replayConfig, + `subscription-replay.optimistic`, + ), + )( + `preserves optimistic overlays across replay outcomes for a random or replayed seed`, + runOptimisticReplayScenario, + ) +}) diff --git a/packages/db/tests/collection-subscription-retention.test.ts b/packages/db/tests/collection-subscription-retention.test.ts new file mode 100644 index 0000000000..b83c5fd67e --- /dev/null +++ b/packages/db/tests/collection-subscription-retention.test.ts @@ -0,0 +1,142 @@ +import { expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { flushPromises } from './utils.js' +import type { LoadSubsetOptions } from '../src/types.js' + +const cases = ([`none`, `success`, `failure`] as const).flatMap((replay) => + [`a`, `c`].flatMap((group) => + [false, true].flatMap((overlap) => + [false, true].map((evict) => ({ replay, group, overlap, evict })), + ), + ), +) + +it.each(cases)( + `source retention controls release: replay=$replay group=$group overlap=$overlap evict=$evict`, + async ({ group, replay, overlap, evict }) => { + type Row = { id: number; group: string } + const loads: Array<{ + options: LoadSubsetOptions + deferred: ReturnType> + }> = [] + const unloads: Array = [] + const a = new Func(`eq`, [new PropRef([`group`]), new Value(`a`)]) + const b = overlap + ? new Func(`in`, [new PropRef([`group`]), new Value([`a`, `b`])]) + : new Func(`eq`, [new PropRef([`group`]), new Value(`b`)]) + const rows = [ + { id: 1, group }, + { id: 2, group: `b` }, + ] + let replaceRows!: (truncate: boolean) => void + let releaseTarget = false + const collection = createCollection({ + id: `source-retention`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync(operations) { + replaceRows = (truncate) => { + operations.begin() + if (truncate) operations.truncate() + for (const row of rows) + operations.write({ type: `insert`, value: row }) + operations.commit() + } + operations.markReady() + return { + loadSubset(options) { + const deferred = createDeferred() + void deferred.promise.catch(() => undefined) + options.signal?.addEventListener( + `abort`, + () => + deferred.reject(new DOMException(`Aborted`, `AbortError`)), + { once: true }, + ) + loads.push({ options, deferred }) + return deferred.promise + }, + unloadSubset(options) { + unloads.push(options) + if (releaseTarget && options.where === a && evict) { + // The source, not predicate membership, decides retention. + // This also covers a row unrelated to the released predicate. + operations.begin() + operations.write({ type: `delete`, key: 1 }) + operations.commit() + } + }, + } + }, + }, + }) + const visible = new Map() + let publications = 0 + const subscription = collection.subscribeChanges( + (changes) => { + publications++ + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else + visible.set(change.key, { + id: change.value.id, + group: change.value.group, + }) + } + }, + { includeInitialState: false }, + ) + try { + // These rows are independent source data, not request-scoped writes. + replaceRows(false) + subscription.requestSnapshot({ where: a }) + subscription.requestSnapshot({ where: b }) + loads.forEach(({ deferred }) => deferred.resolve()) + await flushPromises() + expect([...visible.values()]).toEqual(rows) + if (replay !== `none`) { + replaceRows(true) + await flushPromises() + expect(loads).toHaveLength(4) + } + const beforeRelease = publications + releaseTarget = true + subscription.releaseSnapshot(a) + releaseTarget = false + const released = loads[replay === `none` ? 0 : 2]! + expect(released.options.signal?.aborted).toBe(true) + expect( + unloads.filter((options) => options === released.options), + ).toHaveLength(1) + expect(collection.has(1)).toBe(!evict) + if (replay !== `none`) { + expect([...visible.values()]).toEqual(rows) + expect(publications).toBe(beforeRelease) + const failure = new Error(`peer replay failed`) + if (replay === `failure`) loads[3]!.deferred.reject(failure) + else loads[3]!.deferred.resolve() + await flushPromises() + expect(subscription.lastError).toBe( + replay === `failure` ? failure : undefined, + ) + } + expect([...visible.values()]).toEqual( + evict && replay !== `failure` ? [rows[1]] : rows, + ) + expect(publications - beforeRelease).toBe( + Number(evict && replay !== `failure`), + ) + expect(subscription.status).toBe(`ready`) + } finally { + releaseTarget = false + subscription.unsubscribe() + await collection.cleanup() + } + for (const { options } of loads) { + expect(unloads.filter((unloaded) => unloaded === options)).toHaveLength(1) + } + }, +) diff --git a/packages/db/tests/collection-subscription.test.ts b/packages/db/tests/collection-subscription.test.ts index 65465a6a8b..3ea52df1ab 100644 --- a/packages/db/tests/collection-subscription.test.ts +++ b/packages/db/tests/collection-subscription.test.ts @@ -1,8 +1,161 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' +import { Temporal } from 'temporal-polyfill' import { createCollection } from '../src/collection/index.js' +import { CollectionSubscription } from '../src/collection/subscription.js' +import { createDeferred } from '../src/deferred.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { DeduplicatedLoadSubset } from '../src/query/subset-dedupe.js' import { flushPromises } from './utils' +import type { LoadSubsetOptions } from '../src/types.js' describe(`CollectionSubscription status tracking`, () => { + it.each( + ([`release`, `restart`] as const).flatMap((boundary) => + [false, true].flatMap((rejectOld) => + [false, true].map((oldFirst) => ({ boundary, rejectOld, oldFirst })), + ), + ), + )( + `isolates pending status across retired work: %j`, + async ({ boundary, rejectOld, oldFirst }) => { + const old = createDeferred() + const current = createDeferred() + const last = createDeferred() + const pending = [old, current, last] + let loadCount = 0 + const load = vi.fn(() => pending[loadCount++]!.promise) + const collection = createCollection<{ id: string }>({ + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset: load, unloadSubset: () => {} } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const statuses: Array = [] + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + const settleOld = async () => { + if (rejectOld) old.reject(new Error(`retired work failed`)) + else old.resolve() + await flushPromises() + } + + try { + subscription.requestSnapshot({ where }) + expect(subscription.status).toBe(`loadingSubset`) + if (boundary === `release`) { + subscription.releaseSnapshot(where) + subscription.releaseSnapshot(where) + expect(subscription.status).toBe(`ready`) + subscription.requestSnapshot({ where }) + } else { + await collection.cleanup() + collection.startSyncImmediate() + } + await flushPromises() + expect(load).toHaveBeenCalledTimes(2) + expect(subscription.status).toBe(`loadingSubset`) + const before = [...statuses] + if (oldFirst) { + await settleOld() + expect(subscription.status).toBe(`loadingSubset`) + expect(statuses).toEqual(before) + } + current.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + if (!oldFirst) { + const after = [...statuses] + await settleOld() + expect(statuses).toEqual(after) + } + // A double decrement can hide until the next load starts. + subscription.requestSnapshot({ where }) + expect(load).toHaveBeenCalledTimes(3) + expect(subscription.status).toBe(`loadingSubset`) + last.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + } finally { + for (const result of pending) result.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each([ + { terms: 2, values: [0, 0] }, + { terms: 2, values: [0] }, + { terms: 1, values: [0, 0] }, + ])( + `rejects a $terms-term composite cursor before delivery or acquisition`, + async ({ terms, values }) => { + const load = vi.fn(() => true as const) + const unload = vi.fn() + const delivery = vi.fn() + const observer = vi.fn() + const collection = createCollection<{ id: string; rank: number }>({ + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row`, rank: 1 } }) + commit() + markReady() + return { loadSubset: load, unloadSubset: unload } + }, + }, + }) + const index = collection.createIndex((row) => row.rank, { + indexType: BTreeIndex, + }) + const subscription = collection.subscribeChanges(delivery, { + includeInitialState: false, + }) + subscription.setOrderByIndex(index) + const orderBy = Array.from({ length: terms }, () => ({ + expression: new PropRef([`rank`]), + compareOptions: { direction: `asc` as const, nulls: `first` as const }, + })) + try { + expect(() => + subscription.requestLimitedSnapshot({ + orderBy, + limit: 1, + minValues: values, + onLoadSubsetResult: observer, + }), + ).toThrow(`Only single-column cursors are supported`) + expect(delivery).not.toHaveBeenCalled() + expect(load).not.toHaveBeenCalled() + expect(observer).not.toHaveBeenCalled() + expect(subscription.status).toBe(`ready`) + // A rejected input must not consume local sent keys or an acquisition slot. + subscription.requestLimitedSnapshot({ + orderBy: orderBy.slice(0, 1), + limit: 1, + minValues: [0], + }) + expect(delivery).toHaveBeenCalledTimes(1) + expect(load).toHaveBeenCalledTimes(1) + expect(load.mock.calls[0]).toBeDefined() + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + expect(unload).toHaveBeenCalledTimes(1) + }, + ) + it(`subscription starts with status 'ready'`, () => { const collection = createCollection<{ id: string; value: string }>({ id: `test`, @@ -43,7 +196,6 @@ describe(`CollectionSubscription status tracking`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, }) - expect(subscription.status).toBe(`ready`) // Trigger a snapshot request that will call loadSubset @@ -208,6 +360,112 @@ describe(`CollectionSubscription status tracking`, () => { subscription.unsubscribe() }) + it.each( + ([`generic`, `specific`] as const).flatMap((eventKind) => + ([`clean`, `throw`] as const).map((releaseKind) => ({ + eventKind, + releaseKind, + })), + ), + )( + `stops status delivery when a $eventKind loading listener unsubscribes with $releaseKind cleanup`, + async ({ eventKind, releaseKind }) => { + const pending = createDeferred() + const releaseFailure = new Error(`release failed during status callback`) + const deferredMicrotasks: Array = [] + const queueMicrotaskSpy = vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => deferredMicrotasks.push(callback)) + const collection = createCollection<{ id: string }>({ + id: `unsubscribe-during-${eventKind}-loading-status-${releaseKind}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => pending.promise, + unloadSubset: () => { + if (releaseKind === `throw`) throw releaseFailure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const eventsAfterTeardown: Array = [] + let teardownStarted = false + const unsubscribeOnLoading = () => { + teardownStarted = true + subscription.unsubscribe() + } + const recordAfterTeardown = (event: { status: string }) => { + if (teardownStarted) eventsAfterTeardown.push(event.status) + } + + if (eventKind === `generic`) { + subscription.on(`status:change`, ({ status }) => { + if (status === `loadingSubset`) unsubscribeOnLoading() + }) + subscription.on(`status:change`, recordAfterTeardown) + } else { + subscription.on(`status:loadingSubset`, unsubscribeOnLoading) + subscription.on(`status:loadingSubset`, recordAfterTeardown) + subscription.on(`status:change`, recordAfterTeardown) + } + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + expect(eventsAfterTeardown).toEqual([]) + expect(collection.subscriberCount).toBe(0) + expect(deferredMicrotasks).toHaveLength(releaseKind === `throw` ? 1 : 0) + if (releaseKind === `throw`) { + expect(() => deferredMicrotasks[0]!()).toThrow(releaseFailure) + } + + pending.resolve() + await flushPromises() + expect(eventsAfterTeardown).toEqual([]) + } finally { + queueMicrotaskSpy.mockRestore() + await collection.cleanup() + } + }, + ) + + it(`unsubscribes once when an unsubscribed listener reenters`, async () => { + const deferredMicrotasks: Array = [] + const queueMicrotaskSpy = vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => deferredMicrotasks.push(callback)) + const collection = createCollection<{ id: string }>({ + id: `reentrant-unsubscribed-event`, + getKey: ({ id }) => id, + sync: { sync: ({ markReady }) => markReady() }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + let events = 0 + subscription.on(`unsubscribed`, () => { + events++ + if (events === 1) subscription.unsubscribe() + }) + + try { + subscription.unsubscribe() + + expect(events).toBe(1) + expect(collection.subscriberCount).toBe(0) + expect(deferredMicrotasks).toEqual([]) + } finally { + queueMicrotaskSpy.mockRestore() + await collection.cleanup() + } + }) + it(`promise rejection still cleans up and sets status back to 'ready'`, async () => { let rejectLoadSubset: (error: Error) => void const loadSubsetPromise = new Promise((_, reject) => { @@ -246,6 +504,2167 @@ describe(`CollectionSubscription status tracking`, () => { subscription.unsubscribe() }) + it(`records the last rejected subset load without hiding ready data`, async () => { + const error = new Error(`incremental subset failed`) + const collection = createCollection<{ id: string; value: string }>({ + id: `subset-error-recording`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { id: `cached`, value: `available` }, + }) + commit() + markReady() + return { + loadSubset: () => Promise.reject(error), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const failures: Array = [] + subscription.on(`loadSubset:error`, (event) => failures.push(event.error)) + + subscription.requestSnapshot({ optimizedOnly: false }) + await flushPromises() + + expect(subscription.status).toBe(`ready`) + expect(collection.get(`cached`)).toMatchObject({ value: `available` }) + expect(subscription.lastError).toBe(error) + expect(failures).toEqual([error]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`records a synchronously thrown subset failure`, async () => { + const error = new Error(`synchronous subset failure`) + const collection = createCollection<{ id: string }>({ + id: `synchronous-subset-error`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + throw error + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const failures: Array = [] + subscription.on(`loadSubset:error`, (event) => failures.push(event.error)) + + expect(() => + subscription.requestSnapshot({ optimizedOnly: false }), + ).toThrow(error) + expect(subscription.lastError).toBe(error) + expect(failures).toEqual([error]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`does not unload a subset when loadSubset throws before acquisition`, async () => { + const failure = new Error(`subset failed before acquisition`) + const unloadedOptions: Array = [] + const collection = createCollection<{ id: string }>({ + id: `failed-subset-acquisition`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + throw failure + }, + unloadSubset: (options) => unloadedOptions.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + expect(() => + subscription.requestSnapshot({ optimizedOnly: false }), + ).toThrow(failure) + subscription.unsubscribe() + + expect(unloadedOptions).toEqual([]) + await collection.cleanup() + }) + + it(`releases a subset when its load-result observer throws`, async () => { + const failure = new Error(`load-result observer failed`) + let acquiredOptions: unknown + const unloadedOptions: Array = [] + const collection = createCollection<{ id: string }>({ + id: `subset-observer-failure`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + acquiredOptions = options + return true + }, + unloadSubset: (options) => unloadedOptions.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + expect(() => + subscription.requestSnapshot({ + optimizedOnly: false, + onLoadSubsetResult: () => { + throw failure + }, + }), + ).toThrow(failure) + subscription.unsubscribe() + + expect(unloadedOptions).toEqual([acquiredOptions]) + await collection.cleanup() + }) + + it.each([ + { position: `failed-first`, nestedCleanup: `clean` }, + { position: `failed-first`, nestedCleanup: `throw` }, + { position: `failed-last`, nestedCleanup: `clean` }, + { position: `failed-last`, nestedCleanup: `throw` }, + ] as const)( + `re-finds the $position demand after reentrant $nestedCleanup cleanup`, + async ({ position, nestedCleanup }) => { + const primaryFailure = new Error(`request failed after acquisition`) + const cleanupFailure = new Error(`nested cleanup failed`) + const loaded: Array = [] + const unloaded: Array = [] + const reported: Array = [] + let caughtCleanup: unknown + const collection = createCollection<{ id: string }>({ + id: `reentrant-primary-release-${position}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loaded.push(options) + return true + }, + unloadSubset: (options) => { + unloaded.push(options) + if ( + nestedCleanup === `throw` && + options === loaded[0] && + unloaded.filter((entry) => entry === loaded[0]).length === 1 + ) { + throw cleanupFailure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const firstWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`first`), + ]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`second`), + ]) + let releaseFirst: + | ((primaryFailure?: { error: unknown }) => void) + | undefined + let releaseSecond: + | ((primaryFailure?: { error: unknown }) => void) + | undefined + + subscription.requestSnapshot({ + where: firstWhere, + onLoadSubsetResult: (_result, _options, release) => { + releaseFirst = release + }, + }) + subscription.requestSnapshot({ + where: secondWhere, + onLoadSubsetResult: (_result, _options, release) => { + releaseSecond = release + }, + }) + subscription.on(`loadSubset:error`, ({ error }) => { + reported.push(error) + try { + subscription.releaseSnapshot(firstWhere) + } catch (cleanupError) { + caughtCleanup = cleanupError + } + }) + + if (position === `failed-first`) { + releaseFirst!({ error: primaryFailure }) + expect(unloaded).toEqual([loaded[0]]) + } else { + releaseSecond!({ error: primaryFailure }) + expect(unloaded).toEqual([loaded[0], loaded[1]]) + } + expect(subscription.lastError).toBe(primaryFailure) + expect(reported).toEqual([primaryFailure]) + expect(caughtCleanup).toBe( + nestedCleanup === `throw` ? cleanupFailure : undefined, + ) + + subscription.unsubscribe() + expect(unloaded).toEqual([loaded[0], loaded[1]]) + expect(subscription.lastError).toBe(primaryFailure) + expect(reported).toEqual([primaryFailure]) + await collection.cleanup() + }, + ) + + it.each([`releaseSnapshot`, `unsubscribe`] as const)( + `attempts a failed exact release only once through %s`, + async (releaseMode) => { + const loads: Array = [] + const unloads: Array = [] + const failure = new Error(`release failed`) + const collection = createCollection<{ id: string }>({ + id: `failed-exact-release-${releaseMode}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: false, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return Promise.resolve() + }, + unloadSubset: (options) => { + unloads.push(options) + if (unloads.length === 1) throw failure + }, + } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const where = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`requested`), + ]) + + try { + subscription.requestSnapshot({ + where, + limit: 1, + optimizedOnly: false, + }) + collection._resumeSyncStart() + await flushPromises() + + expect(loads).toHaveLength(1) + const firstRelease = () => + releaseMode === `releaseSnapshot` + ? subscription.releaseSnapshot(where) + : subscription.unsubscribe() + expect(firstRelease).toThrow(failure) + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads).toEqual([loads[0]]) + + subscription.unsubscribe() + expect(unloads).toHaveLength(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each( + ([`before`, `after`] as const).flatMap((throwAt) => + [false, true].map((reenter) => ({ throwAt, reenter })), + ), + )( + `bounds throwing adapter cleanup at $throwAt release, reentry=$reenter`, + async ({ throwAt, reenter }) => { + const failure = new Error(`adapter cleanup failed`) + const loads: Array = [] + const unloads: Array = [] + const externalLeases = new Set() + let dispose = () => {} + const collection = createCollection<{ id: string }>({ + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + externalLeases.add(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[0]) { + if (reenter) dispose() + if (throwAt === `before`) throw failure + externalLeases.delete(options) + throw failure + } + externalLeases.delete(options) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + dispose = () => subscription.unsubscribe() + try { + subscription.requestSnapshot({ where: new Value(true) }) + subscription.requestSnapshot({ where: new Value(false) }) + expect(dispose).toThrow(failure) + expect(dispose).not.toThrow() + expect(unloads).toEqual(loads) + expect(loads).toHaveLength(2) + expect(loads.every(({ signal }) => signal?.aborted)).toBe(true) + expect(collection.subscriberCount).toBe(0) + expect(subscription.lastError).toBe(failure) + // Intentional support boundary: core cannot repair an adapter that throws + // before freeing its resource, nor safely repeat a possibly completed release. + expect([...externalLeases]).toEqual( + throwAt === `before` ? [loads[0]] : [], + ) + } finally { + dispose() + await collection.cleanup() + } + }, + ) + + it(`preserves a primary error across reentrant teardown failure`, async () => { + const primaryFailure = new Error(`request failed after acquisition`) + const cleanupFailure = new Error(`teardown failed`) + const loads: Array = [] + const unloads: Array = [] + const reported: Array = [] + let releaseFailedDemand: + | ((primaryFailure?: { error: unknown }) => void) + | undefined + let cleanupAttempts = 0 + let caughtCleanup: unknown + const collection = createCollection<{ id: string }>({ + id: `primary-error-reentrant-teardown`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[0] && cleanupAttempts++ === 0) { + throw cleanupFailure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where: new Value(true) }) + subscription.requestSnapshot({ + where: new Value(false), + onLoadSubsetResult: (_result, _options, release) => { + releaseFailedDemand = release + }, + }) + subscription.on(`loadSubset:error`, ({ error }) => { + reported.push(error) + if (error !== primaryFailure) return + try { + subscription.unsubscribe() + } catch (cleanupError) { + caughtCleanup = cleanupError + } + }) + + releaseFailedDemand!({ error: primaryFailure }) + + expect(caughtCleanup).toBe(cleanupFailure) + expect(reported).toEqual([primaryFailure]) + expect(subscription.lastError).toBe(primaryFailure) + expect(unloads).toEqual([loads[0], loads[1]]) + + subscription.unsubscribe() + expect(unloads).toEqual([loads[0], loads[1]]) + expect(subscription.lastError).toBe(primaryFailure) + await collection.cleanup() + }) + + it.each([`sync`, `async`, `replay`] as const)( + `preserves a %s adapter error across reentrant teardown failure`, + async (failureMode) => { + const primaryFailure = new Error(`${failureMode} load failed`) + const cleanupFailure = new Error(`teardown failed`) + const victimWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`victim`), + ]) + const failedWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`failed`), + ]) + const loads: Array = [] + const reported: Array = [] + let truncateSource = () => {} + let cleanupAttempts = 0 + let caughtCleanup: unknown + let deliveringPrimary = false + const collection = createCollection<{ id: string }>({ + id: `primary-${failureMode}-reentrant-teardown`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, commit, markReady, truncate }) => { + truncateSource = () => { + begin() + truncate() + commit() + } + markReady() + return { + loadSubset: (options) => { + loads.push(options) + const shouldFail = + failureMode === `replay` + ? loads.length === 4 + : loads.length === 2 + if (!shouldFail) return true + if (failureMode === `sync`) throw primaryFailure + return Promise.reject(primaryFailure) + }, + unloadSubset: () => { + if (deliveringPrimary && cleanupAttempts++ === 0) { + throw cleanupFailure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => { + reported.push(error) + if (error !== primaryFailure) return + deliveringPrimary = true + try { + subscription.releaseSnapshot(victimWhere) + } catch (cleanupError) { + caughtCleanup = cleanupError + } finally { + deliveringPrimary = false + } + }) + + try { + subscription.requestSnapshot({ where: victimWhere }) + if (failureMode === `replay`) { + subscription.requestSnapshot({ where: failedWhere }) + truncateSource() + } else { + const request = () => + subscription.requestSnapshot({ where: failedWhere }) + if (failureMode === `sync`) { + expect(request).toThrow(primaryFailure) + } else { + request() + } + } + await flushPromises() + + expect(caughtCleanup).toBe(cleanupFailure) + expect(reported).toEqual([primaryFailure]) + expect(subscription.lastError).toBe(primaryFailure) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`does not unload a synchronous acquisition that never started`, async () => { + const failure = new Error(`load failed before acquisition`) + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`failed`)]) + const unloads: Array = [] + const collection = createCollection<{ id: string }>({ + id: `reentrant-failed-start-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + throw failure + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => { + if (error === failure) subscription.releaseSnapshot(where) + }) + + try { + expect(() => subscription.requestSnapshot({ where })).toThrow(failure) + expect(unloads).toEqual([]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not replay a logically retired demand after its unload fails`, async () => { + const loads: Array = [] + const unloads: Array = [] + const releaseError = new Error(`release failed`) + let allowUnload = false + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const collection = createCollection<{ id: string }>({ + id: `failed-release-is-not-replayed`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return Promise.resolve() + }, + unloadSubset: (options) => { + unloads.push(options) + if (!allowUnload && options === loads[0]) throw releaseError + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`first`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`second`), + ]) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + await flushPromises() + + expect(() => subscription.releaseSnapshot(firstWhere)).toThrow( + releaseError, + ) + + begin() + truncate() + commit() + await flushPromises() + + // The release attempt retired the demand, even though the adapter threw. + // It must not join later replays or be released a second time. + expect(loads).toHaveLength(3) + expect(loads[2]?.where).toBe(secondWhere) + } finally { + allowUnload = true + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`retires pending status per demand even when physical cleanup fails`, async () => { + const firstLoad = createDeferred() + const secondLoad = createDeferred() + const loads: Array = [] + const unloads: Array = [] + const releaseError = new Error(`release failed`) + let firstReleaseAttempts = 0 + const collection = createCollection<{ id: string }>({ + id: `retired-pending-status-and-cleanup-debt`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return loads.length === 1 ? firstLoad.promise : secondLoad.promise + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[0] && ++firstReleaseAttempts < 3) { + throw releaseError + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`first`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`second`), + ]) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + expect(subscription.status).toBe(`loadingSubset`) + + expect(() => subscription.releaseSnapshot(firstWhere)).toThrow( + releaseError, + ) + expect(subscription.status).toBe(`loadingSubset`) + + secondLoad.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + + expect(() => subscription.unsubscribe()).not.toThrow() + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads).toEqual([loads[0], loads[1]]) + } finally { + firstLoad.resolve() + secondLoad.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`attempts both leases once when throwing cleanup reenters teardown`, async () => { + const releaseFailure = new Error(`release failed`) + const duplicateFailure = new Error(`duplicate release`) + const loads: Array = [] + const unloads: Array = [] + const attempts = new Map() + const collection = createCollection<{ id: string }>({ + id: `reentrant-cleanup-debt-retirement`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + const attempt = (attempts.get(options) ?? 0) + 1 + attempts.set(options, attempt) + if (attempt > 1) throw duplicateFailure + if (options === loads[0]) { + subscription.unsubscribe() + } + throw releaseFailure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`first`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`second`), + ]) + + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + expect(() => subscription.releaseSnapshot(firstWhere)).toThrow( + releaseFailure, + ) + expect(() => subscription.releaseSnapshot(secondWhere)).not.toThrow() + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads).toEqual([loads[0], loads[1]]) + expect(collection.subscriberCount).toBe(0) + + subscription.unsubscribe() + expect(unloads).toHaveLength(2) + } finally { + try { + subscription.unsubscribe() + } catch { + // Keep cleanup available after a red assertion. + } + await collection.cleanup() + } + }) + + it.each([`sync`, `async`] as const)( + `reopens a failed %s replay only after its last logical demand retires`, + async (failureMode) => { + const failure = new Error(`replay failed`) + let begin!: () => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + let replayStarts = 0 + let replaySuccesses = 0 + const collection = createCollection<{ id: string }>({ + id: `failed-replay-logical-demand-cardinality-${failureMode}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount += 1 + if (loadCount <= 2) return true + if (failureMode === `sync`) throw failure + return Promise.reject(failure) + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { + start: () => { + replayStarts += 1 + }, + succeed: () => { + replaySuccesses += 1 + }, + }, + }) + const firstWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`first`), + ]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`second`), + ]) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + + begin() + truncate() + commit() + await flushPromises() + expect(loadCount).toBe(4) + expect(replayStarts).toBe(1) + expect(replaySuccesses).toBe(0) + + subscription.releaseSnapshot(firstWhere) + expect(replaySuccesses).toBe(0) + + subscription.releaseSnapshot(secondWhere) + expect(replaySuccesses).toBe(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`attempts the exact in-flight replay release once`, async () => { + const replay = createDeferred() + const loads: Array = [] + const unloads: Array = [] + const failure = new Error(`replay release failed`) + let failed = false + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const collection = createCollection<{ id: string }>({ + id: `failed-replay-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return loads.length === 1 ? Promise.resolve() : replay.promise + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[1] && !failed) { + failed = true + throw failure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + await flushPromises() + begin() + truncate() + commit() + await flushPromises() + + expect(loads).toHaveLength(2) + expect(() => subscription.unsubscribe()).toThrow(failure) + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads.filter((options) => options === loads[0])).toEqual([ + loads[0], + ]) + expect(unloads.filter((options) => options === loads[1])).toEqual([ + loads[1], + ]) + } finally { + replay.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it.each( + ([`direct`, `deferred`] as const).flatMap((start) => + ([`return`, `resolve`] as const).map((result) => ({ + name: `${start} ${result}`, + start, + result, + })), + ), + )( + `publishes ownership before a reentrant unsubscribe: $name`, + async ({ start, result }) => { + const loads: Array = [] + const unloads: Array = [] + let unsubscribeDuringLoad = () => {} + const collection = createCollection<{ id: string }>({ + id: `reentrant-ownership-${start}-${result}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: start === `direct`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + unsubscribeDuringLoad() + return result === `return` ? true : Promise.resolve() + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + if (start === `deferred`) expect(collection._deferSyncStart()).toBe(true) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + unsubscribeDuringLoad = () => subscription.unsubscribe() + + try { + subscription.requestSnapshot({ limit: 1, optimizedOnly: false }) + if (start === `deferred`) collection._resumeSyncStart() + await flushPromises() + + expect(loads).toHaveLength(1) + expect(unloads).toEqual([loads[0]]) + subscription.unsubscribe() + expect(unloads).toHaveLength(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`does not register a subscription closed during its automatic snapshot`, async () => { + const loads: Array = [] + const unloads: Array = [] + const onChange = vi.fn() + let writeAfterUnsubscribe = () => {} + const collection = createCollection<{ id: string }>({ + id: `closed-during-automatic-snapshot`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + writeAfterUnsubscribe = () => { + begin() + write({ type: `insert`, value: { id: `later` } }) + commit() + } + markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (!(options.subscription instanceof CollectionSubscription)) { + throw new Error(`automatic snapshot requires its subscription`) + } + options.subscription.unsubscribe() + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + + const subscription = collection.subscribeChanges(onChange, { + includeInitialState: true, + }) + + expect(loads).toHaveLength(1) + expect(unloads).toEqual(loads) + expect(collection.subscriberCount).toBe(0) + writeAfterUnsubscribe() + expect(onChange).not.toHaveBeenCalled() + + subscription.unsubscribe() + expect(unloads).toHaveLength(1) + await collection.cleanup() + }) + + it(`does not deliver a direct snapshot after adapter work unsubscribes`, async () => { + type Row = { id: string; rank: number } + const loads: Array = [] + const unloads: Array = [] + const callbacks: Array> = [] + let unsubscribeDuringLoad = () => {} + const collection = createCollection({ + id: `direct-snapshot-reentrant-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row`, rank: 1 } }) + commit() + markReady() + return { + loadSubset: (options) => { + loads.push(options) + unsubscribeDuringLoad() + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges((changes) => { + callbacks.push(changes.map(({ value }) => value.id)) + }) + unsubscribeDuringLoad = () => subscription.unsubscribe() + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + + expect(callbacks).toEqual([]) + expect(loads).toHaveLength(1) + expect(unloads).toEqual([loads[0]]) + + subscription.requestSnapshot({ optimizedOnly: false }) + expect(callbacks).toEqual([]) + expect(loads).toHaveLength(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not continue a direct snapshot after its result hook unsubscribes`, async () => { + type Row = { id: string } + const callbacks: Array> = [] + const collection = createCollection({ + id: `direct-result-hook-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row` } }) + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }) + const subscription = collection.subscribeChanges((changes) => { + callbacks.push(changes.map(({ value }) => value.id)) + }) + + try { + subscription.requestSnapshot({ + optimizedOnly: false, + onLoadSubsetResult: () => subscription.unsubscribe(), + }) + + expect(callbacks).toEqual([]) + expect(subscription.status).toBe(`ready`) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not continue an unoptimized snapshot after its hook unsubscribes`, async () => { + type Row = { id: string } + const callbacks: Array> = [] + const collection = createCollection({ + id: `direct-unoptimized-hook-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row` } }) + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }) + const subscription = collection.subscribeChanges((changes) => { + callbacks.push(changes.map(({ value }) => value.id)) + }) + + try { + subscription.requestSnapshot({ + where: new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]), + onUnoptimized: () => subscription.unsubscribe(), + }) + + expect(callbacks).toEqual([]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not start limited adapter work after local delivery unsubscribes`, async () => { + type Row = { id: string; rank: number } + const loads: Array = [] + const unloads: Array = [] + const collection = createCollection({ + id: `limited-snapshot-reentrant-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row`, rank: 1 } }) + commit() + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const index = collection.createIndex((row) => row.rank, { + indexType: BTreeIndex, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => subscription.unsubscribe(), + ) + subscription.setOrderByIndex(index) + + try { + subscription.requestLimitedSnapshot({ + orderBy: [ + { + expression: new PropRef([`rank`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + }) + + expect(loads).toEqual([]) + expect(unloads).toEqual([]) + + subscription.requestLimitedSnapshot({ + orderBy: [ + { + expression: new PropRef([`rank`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + }) + expect(loads).toEqual([]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not observe limited adapter work after it unsubscribes`, async () => { + type Row = { id: string; rank: number } + const pending = createDeferred() + let resultCallbacks = 0 + const collection = createCollection({ + id: `limited-adapter-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + subscription.unsubscribe() + return pending.promise + }, + } + }, + }, + }) + const index = collection.createIndex((row) => row.rank, { + indexType: BTreeIndex, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + ) + subscription.setOrderByIndex(index) + + try { + subscription.requestLimitedSnapshot({ + orderBy: [ + { + expression: new PropRef([`rank`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + onLoadSubsetResult: () => resultCallbacks++, + }) + + expect(resultCallbacks).toBe(0) + expect(subscription.status).toBe(`ready`) + } finally { + pending.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not release one acquisition twice during nested unsubscribe`, async () => { + const unloads: Array = [] + let reentered = false + const collection = createCollection<{ id: string }>({ + id: `nested-unsubscribe-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: (options) => { + unloads.push(options) + if (!reentered) { + reentered = true + subscription.unsubscribe() + } + }, + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + subscription.unsubscribe() + + expect(unloads).toHaveLength(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it.each( + ([false, true] as const).flatMap((adapterCatches) => + ([`return`, `resolve`] as const).map((result) => ({ + name: `${adapterCatches ? `caught` : `escaped`} ${result}`, + adapterCatches, + result, + })), + ), + )( + `attempts a deferred failed release once after adapter startup: $name`, + async ({ adapterCatches, result }) => { + const failure = new Error(`reentrant release failed`) + const loads: Array = [] + const unloads: Array = [] + let observedReleaseError: unknown + let unsubscribeDuringLoad = () => {} + const collection = createCollection<{ id: string }>({ + id: `reentrant-release-${adapterCatches}-${result}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (adapterCatches) { + try { + unsubscribeDuringLoad() + } catch (error) { + observedReleaseError = error + } + } else { + unsubscribeDuringLoad() + } + return result === `return` ? true : Promise.resolve() + }, + unloadSubset: (options) => { + unloads.push(options) + if (unloads.length === 1) throw failure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + unsubscribeDuringLoad = () => subscription.unsubscribe() + + try { + const request = () => + subscription.requestSnapshot({ limit: 1, optimizedOnly: false }) + expect(request).toThrow(failure) + expect(observedReleaseError).toBeUndefined() + await flushPromises() + + expect(unloads).toEqual([loads[0]]) + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads).toEqual([loads[0]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`releases each acquisition once when synchronous replay drops its demand`, async () => { + const loads: Array = [] + const unloads: Array = [] + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`requested`)]) + let replay = () => {} + let releaseDuringReplay = () => {} + const collection = createCollection<{ id: string }>({ + id: `synchronous-replay-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, commit, markReady, truncate }) => { + replay = () => { + begin() + truncate() + commit() + } + markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (loads.length === 2) releaseDuringReplay() + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + releaseDuringReplay = () => subscription.releaseSnapshot(where) + + try { + subscription.requestSnapshot({ where, optimizedOnly: false }) + replay() + await flushPromises() + + expect(loads).toHaveLength(2) + expect(unloads.map((options) => loads.indexOf(options)).sort()).toEqual([ + 0, 1, + ]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`reports a rejected subset replay after truncate`, async () => { + const error = new Error(`truncate replay failed`) + let truncateSource: () => void = () => { + throw new Error(`source has not started`) + } + let loadCount = 0 + const collection = createCollection<{ id: string }>({ + id: `truncate-subset-error`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, commit, markReady, truncate }) => { + markReady() + truncateSource = () => { + begin() + truncate() + commit() + } + return { + loadSubset: () => { + loadCount++ + return loadCount === 1 ? Promise.resolve() : Promise.reject(error) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const failures: Array = [] + subscription.on(`loadSubset:error`, (event) => failures.push(event.error)) + + subscription.requestSnapshot({ optimizedOnly: false }) + await flushPromises() + truncateSource() + await flushPromises() + + expect(subscription.status).toBe(`ready`) + expect(subscription.lastError).toBe(error) + expect(failures).toEqual([error]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`waits for every logical demand that shares one replay promise`, async () => { + type Row = { id: string; value: number } + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + let transportCalls = 0 + const dedupe = new DeduplicatedLoadSubset({ + loadSubset: () => { + transportCalls++ + if (transportCalls === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + return Promise.resolve() + } + return replay.promise + }, + }) + const collection = createCollection({ + id: `shared-replay-promise`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: dedupe.loadSubset, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else visible.set(change.key, change.value) + } + }, + { includeInitialState: false }, + ) + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + + try { + subscription.requestSnapshot({ where }) + await flushPromises() + subscription.requestSnapshot({ where }) + expect(transportCalls).toBe(1) + expect( + [...visible.values()].map(({ id, value }) => ({ id, value })), + ).toEqual([{ id: `one`, value: 1 }]) + + dedupe.reset() + begin() + truncate() + commit() + await flushPromises() + // Replay creates a fresh abortable acquisition for each logical demand, + // even when the adapter happens to return the same promise for both. + expect(transportCalls).toBe(3) + + subscription.releaseSnapshot(where) + const failure = new Error(`shared replay failed`) + replay.reject(failure) + await flushPromises() + + expect(subscription.lastError).toBe(failure) + expect( + [...visible.values()].map(({ id, value }) => ({ id, value })), + ).toEqual([{ id: `one`, value: 1 }]) + } finally { + replay.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`retries detached demand after retiring the old lease fails`, async () => { + const replay = createDeferred() + const loads: Array = [] + const unloads: Array = [] + let truncate!: () => void + let begin!: () => void + let commit!: () => void + const collection = createCollection<{ id: string }>({ + id: `failed-replay-lease-replacement`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return loads.length === 1 ? true : replay.promise + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[0] && unloads.length === 1) { + throw new Error(`old lease release failed`) + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + let unsubscribed = false + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + + expect(loads).toHaveLength(1) + expect(unloads).toEqual([loads[0]]) + expect(subscription.status).toBe(`ready`) + expect(subscription.lastError).toEqual( + new Error(`old lease release failed`), + ) + + begin() + truncate() + commit() + await flushPromises() + expect(loads).toHaveLength(2) + expect(subscription.status).toBe(`loadingSubset`) + expect(loads[1]?.signal?.aborted).toBe(false) + expect(unloads).toEqual([loads[0]]) + replay.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + unsubscribed = true + expect(unloads).toEqual([loads[0], loads[1]]) + } finally { + replay.resolve() + if (!unsubscribed) subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not become ready while replay setup still has a surviving demand`, async () => { + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const firstReplay = createDeferred() + const statusEvents: Array<{ status: string; loadCount: number }> = [] + let begin!: () => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const collection = createCollection<{ id: string }>({ + id: `replay-setup-readiness`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loadCount++ + return loadCount > 2 && options.where === firstWhere + ? firstReplay.promise + : true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + let releaseFirstReplay = false + subscription.on(`status:change`, ({ status }) => { + statusEvents.push({ status, loadCount }) + if (releaseFirstReplay && status === `loadingSubset`) { + releaseFirstReplay = false + subscription.releaseSnapshot(firstWhere) + } + }) + + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + releaseFirstReplay = true + begin() + truncate() + commit() + await flushPromises() + + expect(loadCount).toBe(3) + expect(subscription.status).toBe(`ready`) + expect(statusEvents).toEqual([ + { status: `loadingSubset`, loadCount: 2 }, + { status: `ready`, loadCount: 3 }, + ]) + } finally { + firstReplay.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not publish a pending replay after collection cleanup`, async () => { + type Row = { id: string; version: number } + const replay = createDeferred() + const visible = new Map() + const statusEvents: Array = [] + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const collection = createCollection({ + id: `cleanup-pending-replay`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + const version = ++loadCount + begin() + write({ type: `insert`, value: { id: `row`, version } }) + commit() + return version === 1 ? true : replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else visible.set(change.key, change.value) + } + }, + { includeInitialState: false }, + ) + subscription.on(`status:change`, ({ status }) => { + statusEvents.push(status) + }) + + try { + subscription.requestSnapshot() + expect(visible.get(`row`)?.version).toBe(1) + begin() + truncate() + commit() + await flushPromises() + expect(subscription.status).toBe(`loadingSubset`) + + await collection.cleanup() + const eventsAfterCleanup = [...statusEvents] + replay.resolve() + await flushPromises() + + expect(visible.get(`row`)?.version).toBe(1) + expect(statusEvents).toEqual(eventsAfterCleanup) + } finally { + replay.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`retains a subset after a synchronous truncate replay failure`, async () => { + const error = new Error(`synchronous truncate replay failed`) + let truncateSource: () => void = () => { + throw new Error(`source has not started`) + } + let loadCount = 0 + let unloadCount = 0 + const collection = createCollection<{ id: string }>({ + id: `synchronous-truncate-subset-error`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, commit, markReady, truncate }) => { + markReady() + truncateSource = () => { + begin() + truncate() + commit() + } + return { + loadSubset: () => { + loadCount++ + if (loadCount === 2) throw error + return true + }, + unloadSubset: () => { + unloadCount++ + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + subscription.requestSnapshot({ optimizedOnly: false }) + truncateSource() + await flushPromises() + truncateSource() + await flushPromises() + + expect(loadCount).toBe(3) + expect(subscription.lastError).toBe(error) + + subscription.unsubscribe() + // The initial load and the later successful replay each acquired a lease. + expect(unloadCount).toBe(2) + await collection.cleanup() + }) + + it.each([`throw`, `reject`] as const)( + `keeps the last published snapshot when truncate replay fails ($0)`, + async (delivery) => { + type Row = { id: string } + const error = new Error(`truncate replay failed before replacement`) + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + let failReplay = true + const collection = createCollection({ + id: `truncate-replay-preserves-snapshot`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount > 1 && failReplay) { + if (delivery === `throw`) throw error + return Promise.reject(error) + } + begin() + write({ type: `insert`, value: { id: `one` } }) + commit() + return true + }, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else visible.set(change.key, change.value) + } + }, + { includeInitialState: false }, + ) + + subscription.requestSnapshot({ optimizedOnly: false }) + expect([...visible.keys()]).toEqual([`one`]) + + begin() + truncate() + commit() + await flushPromises() + + expect(subscription.lastError).toBe(error) + expect([...visible.keys()]).toEqual([`one`]) + + begin() + write({ type: `insert`, value: { id: `two` } }) + commit() + await flushPromises() + + // Ordinary source changes do not establish a complete replacement. + // Keep the last coherent generation until a later replay succeeds. + expect([...visible.keys()]).toEqual([`one`]) + + failReplay = false + begin() + truncate() + commit() + await flushPromises() + + expect([...visible.keys()]).toEqual([`one`]) + + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + it(`publishes one coherent snapshot after overlapping truncate replays`, async () => { + type Row = { id: string } + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const resolveReplays: Array<() => void> = [] + const collection = createCollection({ + id: `overlapping-truncate-replays`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: `old` } }) + commit() + return true + } + if (loadCount === 3) { + begin() + write({ type: `insert`, value: { id: `new` } }) + commit() + } + return new Promise((resolve) => + resolveReplays.push(resolve), + ) + }, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else visible.set(change.key, change.value) + } + }, + { includeInitialState: false }, + ) + + subscription.requestSnapshot({ optimizedOnly: false }) + expect([...visible.keys()]).toEqual([`old`]) + + begin() + truncate() + commit() + await flushPromises() + + begin() + truncate() + commit() + await flushPromises() + + resolveReplays[1]!() + await flushPromises() + expect([...visible.keys()]).toEqual([`old`]) + + resolveReplays[0]!() + await flushPromises() + expect([...visible.keys()]).toEqual([`new`]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`does not become ready between reentrant truncate replacements`, async () => { + type Row = { id: string; version: number } + const replays = [createDeferred(), createDeferred()] + const statusEvents: Array = [] + const visible = new Map() + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + let startedNestedReplay = false + const collection = createCollection({ + id: `reentrant-truncate-ready-barrier`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: () => { + const version = ++loadCount + begin() + write({ type: `insert`, value: { id: `row`, version } }) + commit() + return version === 1 ? true : replays[version - 2]!.promise + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else visible.set(change.key, change.value) + } + if (visible.get(`row`)?.version === 2 && !startedNestedReplay) { + startedNestedReplay = true + begin() + truncate() + commit() + } + }, + { includeInitialState: false }, + ) + subscription.on(`status:change`, ({ status }) => { + statusEvents.push(status) + }) + + try { + subscription.requestSnapshot() + expect(visible.get(`row`)?.version).toBe(1) + + begin() + truncate() + commit() + await flushPromises() + expect(subscription.status).toBe(`loadingSubset`) + + replays[0]!.resolve() + await flushPromises() + expect(loadCount).toBe(3) + expect(visible.get(`row`)?.version).toBe(2) + expect(subscription.status).toBe(`loadingSubset`) + expect(statusEvents).toEqual([`loadingSubset`]) + + replays[1]!.resolve() + await flushPromises() + expect(visible.get(`row`)?.version).toBe(3) + expect(subscription.status).toBe(`ready`) + expect(statusEvents).toEqual([`loadingSubset`, `ready`]) + } finally { + for (const replay of replays) replay.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`scopes a subset failure to the subscription that requested it`, async () => { + const error = new Error(`first subscription failed`) + let loadCount = 0 + const collection = createCollection<{ id: string }>({ + id: `scoped-subset-error`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loadCount++ + return loadCount === 1 ? Promise.reject(error) : Promise.resolve() + }, + } + }, + }, + }) + const failing = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const healthy = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + failing.requestSnapshot({ optimizedOnly: false }) + healthy.requestSnapshot({ optimizedOnly: false }) + await flushPromises() + + expect(collection.status).toBe(`ready`) + expect(failing.lastError).toBe(error) + expect(healthy.lastError).toBeUndefined() + + failing.unsubscribe() + healthy.unsubscribe() + await collection.cleanup() + }) + + it(`does not report an aborted subset request as a failure`, async () => { + const cancellation = new Error(`obsolete subset request`) + cancellation.name = `AbortError` + const collection = createCollection<{ id: string }>({ + id: `aborted-subset-request`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: ({ signal }) => + new Promise((_resolve, reject) => { + signal?.addEventListener(`abort`, () => reject(cancellation), { + once: true, + }) + }), + } + }, + }, + }) + const controller = new AbortController() + const failures: Array = [] + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, (event) => failures.push(event.error)) + + subscription.requestSnapshot({ + optimizedOnly: false, + signal: controller.signal, + }) + controller.abort() + await flushPromises() + + expect(subscription.status).toBe(`ready`) + expect(subscription.lastError).toBeUndefined() + expect(failures).toEqual([]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it.each([ + [`Temporal`, Temporal.PlainDate.from(`2026-08-24`)], + [ + `opaque class`, + new (class Sortable { + valueOf() { + return 24 + } + })(), + ], + ])( + `passes a %s range operand through to the adapter`, + async (_name, operand) => { + let received: LoadSubsetOptions | undefined + const collection = createCollection<{ id: string }>({ + id: `range-operand-subset`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + received = options + return true + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const where = new Func(`gt`, [new PropRef([`value`]), new Value(operand)]) + + expect(() => + subscription.requestSnapshot({ where, optimizedOnly: false }), + ).not.toThrow() + expect(((received?.where as Func).args[1] as Value).value).toBe(operand) + + subscription.unsubscribe() + await collection.cleanup() + }, + ) + it(`unsubscribe clears event listeners`, () => { const collection = createCollection<{ id: string; value: string }>({ id: `test`, diff --git a/packages/db/tests/collection-sync-reentrancy.test.ts b/packages/db/tests/collection-sync-reentrancy.test.ts new file mode 100644 index 0000000000..9ed3b80cf0 --- /dev/null +++ b/packages/db/tests/collection-sync-reentrancy.test.ts @@ -0,0 +1,1429 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { oracleRandomParameters, readOracleRunConfig } from './oracle-config.js' +import { flushPromises } from './utils.js' +import type { SyncConfig } from '../src/types.js' + +type Row = { + id: number + value: string +} + +type SyncOps = Parameters[`sync`]>[0] + +type OrderedRow = Row & { rank: number } +type OrderedSync = Parameters[`sync`]>[0] + +type LayoutCallback = { + changes: Array + keys: Array + values: Array + markedReceiptSettled: boolean + revision: number +} + +type ListenerAction = `commit` | `abort` + +type ListenerScenario = { + beforeOpen: ReadonlyArray + leaveOpen: boolean + afterOpen: ReadonlyArray +} + +const listenerActionArbitrary = fc.constantFrom( + `commit`, + `abort`, +) + +const listenerScenarioArbitrary: fc.Arbitrary = fc.record({ + beforeOpen: fc.array(listenerActionArbitrary, { maxLength: 2 }), + leaveOpen: fc.boolean(), + afterOpen: fc.array(listenerActionArbitrary, { maxLength: 2 }), +}) + +function enumerateActions(maxLength: number): Array> { + const histories: Array> = [[]] + for (let length = 1; length <= maxLength; length++) { + const previous = histories.filter( + (history) => history.length === length - 1, + ) + histories.push( + ...previous.flatMap((history) => + ([`commit`, `abort`] as const).map((action) => [...history, action]), + ), + ) + } + return histories +} + +const exhaustiveListenerScenarios: Array = enumerateActions( + 2, +).flatMap((beforeOpen) => + enumerateActions(2).flatMap((afterOpen) => + [false, true].map((leaveOpen) => ({ + beforeOpen, + leaveOpen, + afterOpen, + })), + ), +) + +let generatedHarnessId = 0 + +function createSyncHarness(id: string) { + let sync!: SyncOps + const collection = createCollection({ + id, + getKey: (row) => row.id, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + ops.markReady() + }, + }, + }) + + return { + collection, + get sync() { + return sync + }, + } +} + +function stageInsert( + sync: SyncOps, + row: Row, + options?: { immediate?: boolean }, +): void { + sync.begin(options) + sync.write({ type: `insert`, value: row }) +} + +function installInitialOrderedRows(sync: OrderedSync): void { + sync.begin({ immediate: true }) + sync.write({ + type: `insert`, + value: { id: 1, value: `one`, rank: 0 }, + }) + sync.write({ + type: `insert`, + value: { id: 2, value: `two`, rank: 1 }, + }) + sync.commit() + sync.markReady() +} + +async function runListenerScenario(scenario: ListenerScenario): Promise { + const harness = createSyncHarness( + `generated-listener-sync-${generatedHarnessId++}`, + ) + const { collection } = harness + const appliedKeys: Array = [] + const originalSet = collection._state.syncedData.set.bind( + collection._state.syncedData, + ) + vi.spyOn(collection._state.syncedData, `set`).mockImplementation( + (key, value) => { + appliedKeys.push(key) + return originalSet(key, value) + }, + ) + const batches: Array> = [] + const committedKeys: Array = [] + const committedReceipts: Array> = [] + const abortedReceipts: Array> = [] + let openKey: number | undefined + let nextKey = 2 + let listenerDepth = 0 + let maxListenerDepth = 0 + let ranActions = false + + const runAction = (action: ListenerAction) => { + const key = nextKey++ + stageInsert(harness.sync, { id: key, value: action }) + if (action === `commit`) { + committedKeys.push(key) + const receipt = harness.sync.commit() + if (receipt !== true) committedReceipts.push(receipt) + return + } + + const controller = new AbortController() + controller.abort() + const receipt = harness.sync.commit(controller.signal) + if (receipt !== true) { + void receipt.then( + () => abortedReceipts.push({ status: `fulfilled`, value: undefined }), + (reason) => abortedReceipts.push({ status: `rejected`, reason }), + ) + } + } + + const subscription = collection.subscribeChanges((changes) => { + listenerDepth++ + maxListenerDepth = Math.max(maxListenerDepth, listenerDepth) + batches.push(changes.map((change) => change.key as number)) + + if (!ranActions && changes.some(({ key }) => key === 1)) { + ranActions = true + scenario.beforeOpen.forEach(runAction) + if (scenario.leaveOpen) { + openKey = nextKey++ + stageInsert(harness.sync, { id: openKey, value: `open` }) + } + scenario.afterOpen.forEach(runAction) + } + + listenerDepth-- + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect(appliedKeys).toEqual([1, ...committedKeys]) + expect(batches).toEqual([ + [1], + ...(committedKeys.length > 0 ? [committedKeys] : []), + ]) + expect(maxListenerDepth).toBe(1) + await Promise.all(committedReceipts) + await flushPromises() + expect(committedReceipts).toHaveLength(committedKeys.length) + expect(abortedReceipts).toHaveLength( + scenario.beforeOpen.filter((action) => action === `abort`).length + + scenario.afterOpen.filter((action) => action === `abort`).length, + ) + expect(abortedReceipts.every(({ status }) => status === `rejected`)).toBe( + true, + ) + + if (openKey !== undefined) { + harness.sync.commit() + expect(appliedKeys).toEqual([1, ...committedKeys, openKey]) + expect(batches.at(-1)).toEqual([openKey]) + } + + expect(collection._state.pendingSyncedTransactions).toHaveLength(0) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } +} + +const { multiplier, ...replay } = readOracleRunConfig() +const generatedRuns = 30 * multiplier + +describe(`sync publication reentrancy`, () => { + it(`publishes nested deferrals as one coherent batch`, async () => { + const harness = createSyncHarness(`nested-publication-cycle`) + const { collection } = harness + const callbacks: Array<{ changes: Array; visibleValue: string }> = + [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map((change) => change.value.value), + visibleValue: collection.get(1)!.value, + }) + }, + { includeInitialState: false }, + ) + + try { + const outer = collection._deferPublication() + stageInsert(harness.sync, { id: 1, value: `first` }, { immediate: true }) + harness.sync.commit() + const inner = collection._deferPublication() + harness.sync.begin({ immediate: true }) + harness.sync.write({ + type: `update`, + value: { id: 1, value: `second` }, + }) + harness.sync.commit() + + inner.publish() + expect(callbacks).toEqual([]) + outer.publish() + expect(callbacks).toEqual([ + { changes: [`first`, `second`], visibleValue: `second` }, + ]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`starts a fresh publication after the previous one closes`, async () => { + const harness = createSyncHarness(`successive-publication-cycles`) + const { collection } = harness + const callbacks: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => callbacks.push(changes.map((change) => change.value.value)), + { includeInitialState: false }, + ) + + try { + const first = collection._deferPublication() + stageInsert(harness.sync, { id: 1, value: `first` }, { immediate: true }) + harness.sync.commit() + first.publish() + + const second = collection._deferPublication() + harness.sync.begin({ immediate: true }) + harness.sync.write({ + type: `update`, + value: { id: 1, value: `second` }, + }) + harness.sync.commit() + second.publish() + + expect(callbacks).toEqual([[`first`], [`second`]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not let a discarded deferral poison the next publication`, async () => { + const harness = createSyncHarness(`discarded-publication-cycle`) + const { collection } = harness + const callbacks: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => callbacks.push(changes.map((change) => change.value.value)), + { includeInitialState: false }, + ) + + try { + const discarded = collection._deferPublication() + stageInsert( + harness.sync, + { id: 1, value: `discarded` }, + { immediate: true }, + ) + harness.sync.commit() + discarded.discard() + expect(callbacks).toEqual([]) + + const published = collection._deferPublication() + harness.sync.begin({ immediate: true }) + harness.sync.write({ + type: `update`, + value: { id: 1, value: `published` }, + }) + harness.sync.commit() + published.publish() + expect(callbacks).toEqual([[`published`]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`lets a publication callback start the next publication cycle`, async () => { + const harness = createSyncHarness(`publication-cycle-from-callback`) + const { collection } = harness + const callbacks: Array<{ + changes: Array + visibleValue: string + revision: number + }> = [] + const initialRevision = collection._stateRevision + const write = (type: `insert` | `update`, value: string) => { + harness.sync.begin({ immediate: true }) + harness.sync.write({ type, value: { id: 1, value } }) + harness.sync.commit() + } + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map((change) => change.value.value), + visibleValue: collection.get(1)!.value, + revision: collection._stateRevision, + }) + + if (changes[0]?.value.value === `first`) { + const secondPublication = collection._deferPublication() + write(`update`, `second`) + secondPublication.publish() + } + }, + { includeInitialState: false }, + ) + + try { + const firstPublication = collection._deferPublication() + write(`insert`, `first`) + firstPublication.publish() + + expect(callbacks).toEqual([ + { + changes: [`first`], + visibleValue: `first`, + revision: initialRevision + 1, + }, + { + changes: [`second`], + visibleValue: `second`, + revision: initialRevision + 2, + }, + ]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`publishes an internal layout swap with unchanged endpoints`, async () => { + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-middle-swap`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + ops.begin({ immediate: true }) + for (let id = 1; id <= 5; id++) { + ops.write({ + type: `insert`, + value: { id, value: `value-${id}`, rank: id }, + }) + } + ops.commit() + ops.markReady() + }, + }, + }) + const callbacks: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled: false, + revision: collection._layoutRevision, + }) + }, + { includeInitialState: false }, + ) + + try { + const revisionBeforeSwap = collection._layoutRevision + sync.begin({ immediate: true }) + sync.write({ + type: `update`, + value: { id: 3, value: `value-3`, rank: 4 }, + }) + sync.write({ + type: `update`, + value: { id: 4, value: `value-4`, rank: 3 }, + }) + sync.collection._markLayoutChange() + expect(sync.commit()).toBe(true) + + expect([...collection.keys()]).toEqual([1, 2, 4, 3, 5]) + expect(collection._layoutRevision).toBe(revisionBeforeSwap + 1) + expect(callbacks).toEqual([ + { + changes: [3, 4], + keys: [1, 2, 4, 3, 5], + values: [`value-1`, `value-2`, `value-4`, `value-3`, `value-5`], + markedReceiptSettled: false, + revision: revisionBeforeSwap + 1, + }, + ]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`compares layout with the public state before an immediate prefix drain`, async () => { + const updatePersistence = createDeferred() + const insertPersistence = createDeferred() + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-prefix-drain`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + ops.begin({ immediate: true }) + ops.write({ + type: `insert`, + value: { id: 1, value: `one`, rank: 0 }, + }) + ops.write({ + type: `insert`, + value: { id: 2, value: `two`, rank: 1 }, + }) + ops.commit() + ops.markReady() + }, + }, + onUpdate: () => updatePersistence.promise, + onInsert: () => insertPersistence.promise, + }) + const callbacks: Array<{ + changes: Array + keys: Array + values: Array + }> = [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + }) + }, + { includeInitialState: false }, + ) + const update = collection.update(1, (draft) => { + draft.value = `optimistic-one` + }) + let insert: ReturnType | undefined + + try { + sync.begin() + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.collection._markLayoutChange() + const firstReceipt = sync.commit() + expect(firstReceipt).not.toBe(true) + + insert = collection.insert({ + id: 3, + value: `optimistic-three`, + rank: 3, + }) + callbacks.length = 0 + const revisionBeforeDrain = collection._layoutRevision + expect([...collection.keys()]).toEqual([1, 2, 3]) + + sync.begin({ immediate: true }) + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 0 }, + }) + sync.collection._markLayoutChange() + const secondReceipt = sync.commit() + + expect([...collection.keys()]).toEqual([1, 2, 3]) + expect(collection.toArray.map(({ value }) => value)).toEqual([ + `optimistic-one`, + `two`, + `optimistic-three`, + ]) + expect(callbacks).toEqual([]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain) + await Promise.all( + [firstReceipt, secondReceipt] + .filter((receipt) => receipt !== true) + .map((receipt) => receipt), + ) + + updatePersistence.resolve() + insertPersistence.resolve() + await Promise.all([ + update.isPersisted.promise, + insert.isPersisted.promise, + ]) + } finally { + updatePersistence.resolve() + insertPersistence.resolve() + await update.isPersisted.promise.catch(() => undefined) + await insert?.isPersisted.promise.catch(() => undefined) + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`uses the post-removal public layout before an unmarked prefix drain`, async () => { + const updatePersistence = createDeferred() + const deletePersistence = createDeferred() + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-prefix-removal-drain`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + installInitialOrderedRows(ops) + }, + }, + onUpdate: () => updatePersistence.promise, + onDelete: () => deletePersistence.promise, + }) + const callbacks: Array = [] + let parkedReceiptSettled = false + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled: parkedReceiptSettled, + revision: collection._layoutRevision, + }) + }, + { includeInitialState: false }, + ) + const update = collection.update(2, (draft) => { + draft.value = `optimistic-two` + }) + let deletion: ReturnType | undefined + + try { + sync.begin() + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.collection._markLayoutChange() + const parkedReceipt = sync.commit() + expect(parkedReceipt).not.toBe(true) + if (parkedReceipt !== true) { + void parkedReceipt.then(() => { + parkedReceiptSettled = true + }) + } + + deletion = collection.delete(1) + callbacks.length = 0 + const revisionBeforeDrain = collection._layoutRevision + await Promise.resolve() + expect(parkedReceiptSettled).toBe(false) + expect([...collection.keys()]).toEqual([2]) + + sync.begin({ immediate: true }) + sync.write({ + type: `update`, + value: { id: 2, value: `server-two`, rank: 1 }, + }) + const drainReceipt = sync.commit() + + expect(drainReceipt).toBe(true) + expect([...collection.keys()]).toEqual([2]) + expect(collection.toArray.map(({ value }) => value)).toEqual([ + `optimistic-two`, + ]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain) + expect(callbacks).toEqual([]) + if (parkedReceipt !== true) await parkedReceipt + expect(parkedReceiptSettled).toBe(true) + } finally { + subscription.unsubscribe() + updatePersistence.resolve() + deletePersistence.resolve() + await update.isPersisted.promise.catch(() => undefined) + await deletion?.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } + }) + + it(`publishes a parked layout mark when optimistic persistence drains it`, async () => { + const updatePersistence = createDeferred() + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-prefix-normal-drain`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + installInitialOrderedRows(ops) + }, + }, + onUpdate: () => updatePersistence.promise, + }) + let markedReceiptSettled = false + const callbacks: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled, + revision: collection._layoutRevision, + }) + }, + { includeInitialState: false }, + ) + const update = collection.update(2, (draft) => { + draft.value = `optimistic-two` + }) + + try { + sync.begin() + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.collection._markLayoutChange() + const receipt = sync.commit() + expect(receipt).not.toBe(true) + if (receipt !== true) { + void receipt.then(() => { + markedReceiptSettled = true + }) + } + + callbacks.length = 0 + const revisionBeforeDrain = collection._layoutRevision + await Promise.resolve() + expect(markedReceiptSettled).toBe(false) + expect([...collection.keys()]).toEqual([1, 2]) + + updatePersistence.resolve() + await update.isPersisted.promise + if (receipt !== true) await receipt + + expect([...collection.keys()]).toEqual([2, 1]) + expect(collection.toArray.map(({ value }) => value)).toEqual([ + `two`, + `one`, + ]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain + 1) + // Deltas form one batch; their order is not the collection's row order. + // Assert exact membership while retaining ordered public-read assertions. + expect( + callbacks.map((callback) => ({ + ...callback, + changes: [...callback.changes].sort((a, b) => a - b), + })), + ).toEqual([ + { + changes: [1, 2], + keys: [2, 1], + values: [`two`, `one`], + markedReceiptSettled: false, + revision: revisionBeforeDrain + 1, + }, + ]) + expect(markedReceiptSettled).toBe(true) + } finally { + updatePersistence.resolve() + await update.isPersisted.promise.catch(() => undefined) + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it.each([`first`, `middle`, `last`, `first-and-middle`] as const)( + `honors %s layout marks in an immediate causal prefix`, + async (markPosition) => { + const updatePersistence = createDeferred() + const insertPersistence = createDeferred() + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-prefix-immediate-${markPosition}`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + installInitialOrderedRows(ops) + }, + }, + onUpdate: () => updatePersistence.promise, + onInsert: () => insertPersistence.promise, + }) + let firstReceiptSettled = false + const callbacks: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled: firstReceiptSettled, + revision: collection._layoutRevision, + }) + }, + { includeInitialState: false }, + ) + const update = collection.update(2, (draft) => { + draft.value = `optimistic-two` + }) + let insert: ReturnType | undefined + + try { + sync.begin() + sync.write({ + type: `update`, + value: + markPosition === `first` || markPosition === `first-and-middle` + ? { id: 1, value: `one`, rank: 2 } + : { id: 2, value: `server-two-a`, rank: 1 }, + }) + if (markPosition === `first` || markPosition === `first-and-middle`) { + sync.collection._markLayoutChange() + } + const firstReceipt = sync.commit() + expect(firstReceipt).not.toBe(true) + if (firstReceipt !== true) { + void firstReceipt.then(() => { + firstReceiptSettled = true + }) + } + await Promise.resolve() + expect(firstReceiptSettled).toBe(false) + + insert = collection.insert({ + id: 3, + value: `optimistic-three`, + rank: 3, + }) + + sync.begin() + sync.write({ + type: `update`, + value: + markPosition === `middle` + ? { id: 1, value: `one`, rank: 2 } + : { id: 2, value: `server-two-b`, rank: 1 }, + }) + if (markPosition === `middle` || markPosition === `first-and-middle`) { + sync.collection._markLayoutChange() + } + const middleReceipt = sync.commit() + expect(middleReceipt).not.toBe(true) + + callbacks.length = 0 + const revisionBeforeDrain = collection._layoutRevision + expect([...collection.keys()]).toEqual([1, 2, 3]) + + sync.begin({ immediate: true }) + sync.write({ + type: `update`, + value: + markPosition === `last` + ? { id: 1, value: `one`, rank: 2 } + : { id: 2, value: `server-two-c`, rank: 1 }, + }) + if (markPosition === `last`) sync.collection._markLayoutChange() + const lastReceipt = sync.commit() + + expect(lastReceipt).toBe(true) + expect([...collection.keys()]).toEqual([2, 1, 3]) + expect(collection.toArray.map(({ value }) => value)).toEqual([ + `optimistic-two`, + `one`, + `optimistic-three`, + ]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain + 1) + expect(callbacks).toEqual([ + { + changes: [1], + keys: [2, 1, 3], + values: [`optimistic-two`, `one`, `optimistic-three`], + markedReceiptSettled: false, + revision: revisionBeforeDrain + 1, + }, + ]) + + await Promise.all( + [firstReceipt, middleReceipt] + .filter((receipt) => receipt !== true) + .map((receipt) => receipt), + ) + expect(firstReceiptSettled).toBe(true) + } finally { + subscription.unsubscribe() + updatePersistence.resolve() + insertPersistence.resolve() + await update.isPersisted.promise.catch(() => undefined) + await insert?.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } + }, + ) + + it(`honors a parked layout mark when truncate drains its causal prefix`, async () => { + const updatePersistence = createDeferred() + const insertPersistence = createDeferred() + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-prefix-truncate-drain`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + installInitialOrderedRows(ops) + }, + }, + onUpdate: () => updatePersistence.promise, + onInsert: () => insertPersistence.promise, + }) + let markedReceiptSettled = false + const callbacks: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled, + revision: collection._layoutRevision, + }) + }, + { includeInitialState: false }, + ) + const update = collection.update(1, (draft) => { + draft.value = `optimistic-one` + }) + let insert: ReturnType | undefined + + try { + sync.begin() + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.collection._markLayoutChange() + const firstReceipt = sync.commit() + expect(firstReceipt).not.toBe(true) + if (firstReceipt !== true) { + void firstReceipt.then(() => { + markedReceiptSettled = true + }) + } + + insert = collection.insert({ + id: 3, + value: `optimistic-three`, + rank: 3, + }) + callbacks.length = 0 + const revisionBeforeDrain = collection._layoutRevision + expect([...collection.keys()]).toEqual([1, 2, 3]) + + sync.begin() + sync.truncate() + sync.write({ + type: `insert`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.write({ + type: `insert`, + value: { id: 2, value: `two`, rank: 1 }, + }) + const truncateReceipt = sync.commit() + + expect(truncateReceipt).toBe(true) + expect([...collection.keys()]).toEqual([2, 1, 3]) + expect(collection.toArray.map(({ value }) => value)).toEqual([ + `two`, + `optimistic-one`, + `optimistic-three`, + ]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain + 1) + expect(callbacks).toEqual([ + { + // Reapply whole snapshots in transaction order. Public + // layout, batch membership/count and receipt timing stay unchanged. + changes: [2, 1, 3, 1, 3, 1, 2], + keys: [2, 1, 3], + values: [`two`, `optimistic-one`, `optimistic-three`], + markedReceiptSettled: false, + revision: revisionBeforeDrain + 1, + }, + ]) + if (firstReceipt !== true) await firstReceipt + expect(markedReceiptSettled).toBe(true) + } finally { + subscription.unsubscribe() + updatePersistence.resolve() + insertPersistence.resolve() + await update.isPersisted.promise.catch(() => undefined) + await insert?.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } + }) + + it(`captures a fresh layout boundary for each reentrant causal prefix`, async () => { + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-reentrant-prefixes`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + installInitialOrderedRows(ops) + }, + }, + }) + let listenerDepth = 0 + let maxListenerDepth = 0 + let queuedRestore = false + let innerReceipt: Promise | undefined + let innerReceiptSettled = false + const callbacks: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + listenerDepth++ + maxListenerDepth = Math.max(maxListenerDepth, listenerDepth) + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled: innerReceiptSettled, + revision: collection._layoutRevision, + }) + + if (!queuedRestore) { + queuedRestore = true + sync.begin() + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 0 }, + }) + sync.collection._markLayoutChange() + const receipt = sync.commit() + if (receipt === true) { + throw new Error(`Expected listener-created work to queue`) + } + innerReceipt = receipt + void receipt.then(() => { + innerReceiptSettled = true + }) + } + + listenerDepth-- + }, + { includeInitialState: false }, + ) + + try { + const revisionBeforeDrain = collection._layoutRevision + sync.begin({ immediate: true }) + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.collection._markLayoutChange() + expect(sync.commit()).toBe(true) + + expect([...collection.keys()]).toEqual([1, 2]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain + 2) + expect(callbacks).toEqual([ + { + changes: [1], + keys: [2, 1], + values: [`two`, `one`], + markedReceiptSettled: false, + revision: revisionBeforeDrain + 1, + }, + { + changes: [1], + keys: [1, 2], + values: [`one`, `two`], + markedReceiptSettled: false, + revision: revisionBeforeDrain + 2, + }, + ]) + expect(maxListenerDepth).toBe(1) + expect(innerReceipt).toBeDefined() + await innerReceipt + expect(innerReceiptSettled).toBe(true) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`preserves sync work opened by a listener until it is committed`, async () => { + const harness = createSyncHarness(`listener-opened-sync-work`) + const { collection } = harness + let openedInnerTransaction = false + const batches: Array> = [] + + const subscription = collection.subscribeChanges((changes) => { + batches.push(changes.map((change) => change.key as number)) + if (!openedInnerTransaction && changes.some(({ key }) => key === 1)) { + openedInnerTransaction = true + stageInsert(harness.sync, { id: 2, value: `inner` }) + } + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect(openedInnerTransaction).toBe(true) + expect(collection.get(2)).toBeUndefined() + + expect(() => harness.sync.commit()).not.toThrow() + expect(collection.get(2)).toMatchObject({ id: 2, value: `inner` }) + expect(batches).toEqual([[1], [2]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`publishes listener-committed sync work after the outer batch exactly once`, async () => { + const harness = createSyncHarness(`listener-committed-sync-work`) + const { collection } = harness + const appliedKeys: Array = [] + const originalSet = collection._state.syncedData.set.bind( + collection._state.syncedData, + ) + vi.spyOn(collection._state.syncedData, `set`).mockImplementation( + (key, value) => { + appliedKeys.push(key) + return originalSet(key, value) + }, + ) + const batches: Array> = [] + let listenerDepth = 0 + let maxListenerDepth = 0 + let committedInnerTransaction = false + + const subscription = collection.subscribeChanges((changes) => { + listenerDepth++ + maxListenerDepth = Math.max(maxListenerDepth, listenerDepth) + batches.push(changes.map((change) => change.key as number)) + + if (!committedInnerTransaction && changes.some(({ key }) => key === 1)) { + committedInnerTransaction = true + stageInsert(harness.sync, { id: 2, value: `inner` }) + harness.sync.commit() + } + + listenerDepth-- + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect(appliedKeys).toEqual([1, 2]) + expect(batches).toEqual([[1], [2]]) + expect(maxListenerDepth).toBe(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`keeps callback work FIFO across committed, aborted, and open transactions`, async () => { + const harness = createSyncHarness(`listener-sync-action-order`) + const { collection } = harness + const batches: Array> = [] + let ranListenerActions = false + + const subscription = collection.subscribeChanges((changes) => { + batches.push(changes.map((change) => change.key as number)) + if (ranListenerActions || !changes.some(({ key }) => key === 1)) return + ranListenerActions = true + + stageInsert(harness.sync, { id: 2, value: `left-open` }) + + stageInsert(harness.sync, { id: 3, value: `committed` }) + harness.sync.metadata!.row.set(3, { source: `listener` }) + harness.sync.metadata!.collection.set(`listener:commit`, 3) + harness.sync.commit() + + stageInsert(harness.sync, { id: 4, value: `aborted` }) + const controller = new AbortController() + controller.abort() + const abortedReceipt = harness.sync.commit(controller.signal) + if (abortedReceipt !== true) { + void abortedReceipt.catch(() => undefined) + } + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect(collection.get(3)).toMatchObject({ id: 3, value: `committed` }) + expect(collection.get(4)).toBeUndefined() + expect(collection._state.syncedMetadata.get(3)).toEqual({ + source: `listener`, + }) + expect( + collection._state.syncedCollectionMetadata.get(`listener:commit`), + ).toBe(3) + + harness.sync.commit() + expect(collection.get(2)).toMatchObject({ id: 2, value: `left-open` }) + expect(batches).toEqual([[1], [3], [2]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`drains listener-committed transactions in staging order`, async () => { + const harness = createSyncHarness(`listener-sync-fifo`) + const { collection } = harness + const batches: Array> = [] + let stagedInnerTransactions = false + + const subscription = collection.subscribeChanges((changes) => { + batches.push(changes.map((change) => change.key as number)) + if (stagedInnerTransactions || !changes.some(({ key }) => key === 1)) { + return + } + stagedInnerTransactions = true + + stageInsert(harness.sync, { id: 2, value: `first` }) + harness.sync.commit() + stageInsert(harness.sync, { id: 3, value: `second` }) + harness.sync.commit() + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect([...collection._state.syncedData.keys()]).toEqual([1, 2, 3]) + expect(batches).toEqual([[1], [2, 3]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`drains callback work before surfacing a listener error`, async () => { + const harness = createSyncHarness(`throwing-sync-listener`) + const { collection } = harness + const failure = new Error(`listener failed`) + const appliedKeys: Array = [] + const originalSet = collection._state.syncedData.set.bind( + collection._state.syncedData, + ) + vi.spyOn(collection._state.syncedData, `set`).mockImplementation( + (key, value) => { + appliedKeys.push(key) + return originalSet(key, value) + }, + ) + let queuedReceipt: Promise | undefined + const subscription = collection.subscribeChanges((changes) => { + if (!changes.some(({ key }) => key === 1)) return + stageInsert(harness.sync, { id: 2, value: `queued` }) + const receipt = harness.sync.commit() + if (receipt === true) { + throw new Error(`Expected callback-created work to queue`) + } + queuedReceipt = receipt + throw failure + }) + + try { + stageInsert(harness.sync, { id: 1, value: `first` }) + expect(() => harness.sync.commit()).toThrow(failure) + expect(collection.get(1)).toMatchObject({ id: 1, value: `first` }) + expect(collection.get(2)).toMatchObject({ id: 2, value: `queued` }) + expect(queuedReceipt).toBeDefined() + await expect(queuedReceipt).resolves.toBeUndefined() + + stageInsert(harness.sync, { id: 3, value: `second` }) + expect(() => harness.sync.commit()).not.toThrow() + + expect(appliedKeys).toEqual([1, 2, 3]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`queues a listener truncate until the outer publication finishes`, async () => { + const harness = createSyncHarness(`listener-sync-truncate`) + const { collection } = harness + const appliedKeys: Array = [] + const originalSet = collection._state.syncedData.set.bind( + collection._state.syncedData, + ) + vi.spyOn(collection._state.syncedData, `set`).mockImplementation( + (key, value) => { + appliedKeys.push(key) + return originalSet(key, value) + }, + ) + let stagedTruncate = false + const subscription = collection.subscribeChanges((changes) => { + if (stagedTruncate || !changes.some(({ key }) => key === 1)) return + stagedTruncate = true + harness.sync.begin() + harness.sync.truncate() + harness.sync.write({ + type: `insert`, + value: { id: 2, value: `replacement` }, + }) + harness.sync.commit() + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect(appliedKeys).toEqual([1, 2]) + expect(collection.get(1)).toBeUndefined() + expect(collection.get(2)).toMatchObject({ + id: 2, + value: `replacement`, + }) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`releases subset demand from a publication callback without nested delivery`, async () => { + let sync!: SyncOps + const unloadSubset = vi.fn() + const collection = createCollection({ + id: `listener-subset-release-row-gc`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + ops.markReady() + return { + loadSubset: async () => { + stageInsert(ops, { id: 2, value: `owned` }) + const receipt = ops.commit() + if (receipt !== true) await receipt + return + }, + unloadSubset, + } + }, + }, + }) + let ownerUnsubscribed = false + const owner = collection.subscribeChanges((changes) => { + if (ownerUnsubscribed || !changes.some(({ key }) => key === 1)) return + ownerUnsubscribed = true + owner.unsubscribe() + }) + owner.requestSnapshot({ optimizedOnly: false }) + await flushPromises() + + const batches: Array> = [] + let listenerDepth = 0 + let maxListenerDepth = 0 + const observer = collection.subscribeChanges( + (changes) => { + listenerDepth++ + maxListenerDepth = Math.max(maxListenerDepth, listenerDepth) + batches.push(changes.map((change) => change.key as number)) + listenerDepth-- + }, + { includeInitialState: true }, + ) + batches.length = 0 + + try { + stageInsert(sync, { id: 1, value: `outer` }) + expect(() => sync.commit()).not.toThrow() + + expect(ownerUnsubscribed).toBe(true) + expect(unloadSubset).toHaveBeenCalledOnce() + expect(collection.get(2)).toMatchObject({ id: 2, value: `owned` }) + expect(batches).toEqual([[1]]) + expect(maxListenerDepth).toBe(1) + } finally { + owner.unsubscribe() + observer.unsubscribe() + await collection.cleanup() + } + }) + + it(`keeps normal listener sync work queued behind optimistic persistence`, async () => { + let sync!: SyncOps + const mutation = createDeferred() + const collection = createCollection({ + id: `listener-sync-with-optimistic-work`, + getKey: (row) => row.id, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + ops.markReady() + }, + }, + onInsert: () => mutation.promise, + }) + const optimisticTransaction = collection.insert({ + id: 2, + value: `optimistic`, + }) + let stagedInnerTransaction = false + const subscription = collection.subscribeChanges((changes) => { + if (stagedInnerTransaction || !changes.some(({ key }) => key === 1)) { + return + } + stagedInnerTransaction = true + stageInsert(sync, { id: 3, value: `queued` }) + sync.commit() + }) + + try { + stageInsert(sync, { id: 1, value: `outer` }, { immediate: true }) + sync.commit() + + expect(stagedInnerTransaction).toBe(true) + expect(collection.get(3)).toBeUndefined() + + mutation.resolve() + await optimisticTransaction.isPersisted.promise + + expect(collection.get(3)).toMatchObject({ id: 3, value: `queued` }) + } finally { + mutation.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`matches every bounded reentrant listener history`, async () => { + for (const scenario of exhaustiveListenerScenarios) { + await runListenerScenario(scenario) + } + }) + + fcTest.prop([listenerScenarioArbitrary], { + numRuns: generatedRuns, + seed: 1774, + })(`matches the reentrant drain laws for a fixed seed`, runListenerScenario) + + fcTest.prop( + [listenerScenarioArbitrary], + oracleRandomParameters( + generatedRuns, + replay, + `collection-sync.reentrant-drain`, + ), + )( + `matches the reentrant drain laws for a random or replayed seed`, + runListenerScenario, + ) +}) diff --git a/packages/db/tests/collection-truncate.test.ts b/packages/db/tests/collection-truncate.test.ts index b4f244258a..af2977c805 100644 --- a/packages/db/tests/collection-truncate.test.ts +++ b/packages/db/tests/collection-truncate.test.ts @@ -744,6 +744,7 @@ describe(`Collection truncate operations`, () => { | undefined let loadSubsetResolver: (() => void) | undefined let loadSubsetCallCount = 0 + let totalLoadSubsetCallCount = 0 const collection = createCollection<{ id: number; value: string }, number>({ id: `truncate-buffering-test`, @@ -758,6 +759,8 @@ describe(`Collection truncate operations`, () => { return { loadSubset: (_options: LoadSubsetOptions) => { loadSubsetCallCount++ + totalLoadSubsetCallCount++ + const requestNumber = totalLoadSubsetCallCount // Return a promise that we control return new Promise((resolve) => { @@ -766,11 +769,17 @@ describe(`Collection truncate operations`, () => { cfg.begin() cfg.write({ type: `insert`, - value: { id: 1, value: `refetched-1` }, + value: { + id: 1, + value: requestNumber === 1 ? `initial-1` : `refetched-1`, + }, }) cfg.write({ type: `insert`, - value: { id: 2, value: `refetched-2` }, + value: { + id: 2, + value: requestNumber === 1 ? `initial-2` : `refetched-2`, + }, }) cfg.commit() resolve() @@ -804,8 +813,8 @@ describe(`Collection truncate operations`, () => { // Verify initial data arrived expect(stripChanges(changeEvents)).toEqual([ - { type: `insert`, key: 1, value: { id: 1, value: `refetched-1` } }, - { type: `insert`, key: 2, value: { id: 2, value: `refetched-2` } }, + { type: `insert`, key: 1, value: { id: 1, value: `initial-1` } }, + { type: `insert`, key: 2, value: { id: 2, value: `initial-2` } }, ]) // Clear events for next phase @@ -831,15 +840,14 @@ describe(`Collection truncate operations`, () => { // Wait for buffered events to be flushed await vi.waitFor(() => expect(changeEvents.length).toBeGreaterThan(0)) - // Verify we got all events in one batch (deletes + inserts) - // The subscription should have received: - // - Delete events for the old data (from truncate) - // - Insert events for the new data (from refetch) + // The raw truncate/refetch stream is reduced to one semantic replacement. const deletes = changeEvents.filter((e) => e.type === `delete`) const inserts = changeEvents.filter((e) => e.type === `insert`) + const updates = changeEvents.filter((e) => e.type === `update`) - expect(deletes.length).toBe(2) // Deleted the old items - expect(inserts.length).toBe(2) // Inserted the refetched items + expect(deletes).toHaveLength(0) + expect(inserts).toHaveLength(0) + expect(updates).toHaveLength(2) // Verify final state is correct expect(collection.state.size).toBe(2) @@ -988,6 +996,7 @@ describe(`Collection truncate operations`, () => { | undefined let loadSubsetResolver: (() => void) | undefined let loadSubsetCallCount = 0 + let totalLoadSubsetCallCount = 0 const collection = createCollection<{ id: number; value: string }, number>({ id: `truncate-loadedInitialState-test`, @@ -1002,17 +1011,25 @@ describe(`Collection truncate operations`, () => { return { loadSubset: (_options: LoadSubsetOptions) => { loadSubsetCallCount++ + totalLoadSubsetCallCount++ + const requestNumber = totalLoadSubsetCallCount return new Promise((resolve) => { loadSubsetResolver = () => { cfg.begin() cfg.write({ type: `insert`, - value: { id: 1, value: `item-1` }, + value: { + id: 1, + value: requestNumber === 1 ? `item-1` : `refetched-1`, + }, }) cfg.write({ type: `insert`, - value: { id: 2, value: `item-2` }, + value: { + id: 2, + value: requestNumber === 1 ? `item-2` : `refetched-2`, + }, }) cfg.commit() resolve() @@ -1065,10 +1082,11 @@ describe(`Collection truncate operations`, () => { // Wait for events to be emitted await vi.waitFor(() => expect(changeEvents.length).toBeGreaterThan(0)) - // The key assertion: we should have received delete events - // Without the fix, sentKeys would be empty and deletes would be filtered out - const deletes = changeEvents.filter((e) => e.type === `delete`) - expect(deletes.length).toBe(2) // Must have delete events! + // Even with loadedInitialState, the replacement must publish the exact + // semantic changes rather than filtering the truncate stream away. + expect( + changeEvents.filter((event) => event.type === `update`), + ).toHaveLength(2) subscription.unsubscribe() }) @@ -1083,6 +1101,7 @@ describe(`Collection truncate operations`, () => { let syncOps: | Parameters[`sync`]>[0] | undefined + let loadSubsetCallCount = 0 const collection = createCollection<{ id: number; value: string }, number>({ id: `truncate-sync-loadSubset-test`, @@ -1097,15 +1116,17 @@ describe(`Collection truncate operations`, () => { return { // loadSubset returns true (synchronous) - data already available loadSubset: (_options: LoadSubsetOptions) => { + loadSubsetCallCount++ + const prefix = loadSubsetCallCount === 1 ? `sync` : `refetched` // Synchronously write data cfg.begin() cfg.write({ type: `insert`, - value: { id: 1, value: `sync-item-1` }, + value: { id: 1, value: `${prefix}-item-1` }, }) cfg.write({ type: `insert`, - value: { id: 2, value: `sync-item-2` }, + value: { id: 2, value: `${prefix}-item-2` }, }) cfg.commit() return true // Synchronous return @@ -1139,28 +1160,24 @@ describe(`Collection truncate operations`, () => { // Wait for events to settle await vi.advanceTimersByTimeAsync(10) - // We should have received delete events even though loadSubset was sync + // The synchronous replay publishes one semantic replacement. const deletes = changeEvents.filter((e) => e.type === `delete`) const inserts = changeEvents.filter((e) => e.type === `insert`) + const updates = changeEvents.filter((e) => e.type === `update`) - expect(deletes.length).toBe(2) // Should have 2 deletes - expect(inserts.length).toBe(2) // Should have 2 inserts - - // Verify correct ordering: deletes should come before inserts - // (truncate clears old data, then refetch adds new data) - const firstDeleteIdx = changeEvents.findIndex((e) => e.type === `delete`) - const firstInsertIdx = changeEvents.findIndex((e) => e.type === `insert`) - expect(firstDeleteIdx).toBeLessThan(firstInsertIdx) + expect(deletes).toHaveLength(0) + expect(inserts).toHaveLength(0) + expect(updates).toHaveLength(2) // Verify collection state is correct expect(collection.state.size).toBe(2) expect(getStateValue(collection, 1)).toEqual({ id: 1, - value: `sync-item-1`, + value: `refetched-item-1`, }) expect(getStateValue(collection, 2)).toEqual({ id: 2, - value: `sync-item-2`, + value: `refetched-item-2`, }) subscription.unsubscribe() diff --git a/packages/db/tests/collection.test-d.ts b/packages/db/tests/collection.test-d.ts index 32edff2f8d..8a29fa3ff8 100644 --- a/packages/db/tests/collection.test-d.ts +++ b/packages/db/tests/collection.test-d.ts @@ -1,6 +1,7 @@ import { assertType, describe, expectTypeOf, it } from 'vitest' import { z } from 'zod' import { createCollection } from '../src/collection/index.js' +import { DbClient, collectionOptions } from '../src/index.js' import type { OutputWithVirtual } from './utils' import type { OperationConfig } from '../src/types' import type { StandardSchemaV1 } from '@standard-schema/spec' @@ -162,6 +163,42 @@ describe(`Collection type resolution tests`, () => { }) }) +describe(`DbClient type tests`, () => { + type Todo = { id: string; text: string } + + it(`materializes typed collection options`, () => { + const todos = collectionOptions({ + id: `todos`, + getKey: (todo) => todo.id, + sync: { sync: () => {} }, + }) + + const client = new DbClient() + const collection = client.collection(todos) + + expectTypeOf(collection.get(`1`)).toEqualTypeOf< + OutputWithVirtual | undefined + >() + }) + + it(`accepts materialization initialData`, () => { + const todos = collectionOptions({ + id: `todos`, + getKey: (todo) => todo.id, + sync: { sync: () => {} }, + }) + + const client = new DbClient() + const collection = client.collection(todos, { + initialData: [{ id: `1`, text: `Write tests` }], + }) + + expectTypeOf(collection.toArray).toEqualTypeOf< + Array> + >() + }) +}) + describe(`Schema Input/Output Type Distinction`, () => { // Define schema with different input/output types const userSchemaWithDefaults = z.object({ diff --git a/packages/db/tests/collection.test.ts b/packages/db/tests/collection.test.ts index 7fc5d67b46..9388be9a73 100644 --- a/packages/db/tests/collection.test.ts +++ b/packages/db/tests/collection.test.ts @@ -1,11 +1,16 @@ import mitt from 'mitt' import { describe, expect, it, vi } from 'vitest' -import { createCollection } from '../src/collection/index.js' +import { + createCollection, + withCollectionSyncConfigFactory, +} from '../src/collection/index.js' import { CollectionRequiresConfigError, DuplicateKeyError, + DuplicateKeySyncError, InvalidKeyError, KeyUpdateNotAllowedError, + LoadSubsetOperationAbortedError, MissingDeleteHandlerError, MissingInsertHandlerError, MissingUpdateHandlerError, @@ -18,7 +23,12 @@ import { stripVirtualProps, withExpectedRejection, } from './utils' -import type { ChangeMessage, MutationFn, PendingMutation } from '../src/types' +import type { + ChangeMessage, + MutationFn, + PendingMutation, + SyncConfig, +} from '../src/types' const getStateValue = ( collection: { state: Map }, @@ -37,12 +47,80 @@ const getStateEntries = < ]) describe(`Collection`, () => { + it.each([false, true])( + `owns binding utilities only when a sync factory exists: %s`, + async (bind) => { + let value = 1 + const read = vi.fn(() => value) + const utilities = Object.create( + { inherited: () => `prototype` }, + { + current: { get: read, enumerable: true }, + }, + ) as { readonly current: number; inherited: () => string } + const source: SyncConfig<{ id: number }> = { sync: () => {} } + const factory = vi.fn((sync: typeof source) => sync) + const sync = bind + ? withCollectionSyncConfigFactory(source, factory) + : source + const options = { + getKey: (row: { id: number }) => row.id, + sync, + utils: utilities, + } + const a = createCollection(options) + const b = createCollection(options) + try { + expect(read).not.toHaveBeenCalled() + expect(a.utils === utilities).toBe(!bind) + expect(a.utils === b.utils).toBe(!bind) + expect(a.utils.inherited()).toBe(`prototype`) + value = 2 + expect(a.utils.current).toBe(2) + expect(b.utils.current).toBe(2) + expect(factory).toHaveBeenCalledTimes(bind ? 2 : 0) + } finally { + await a.cleanup() + await b.cleanup() + } + }, + ) + it(`should throw if there's no sync config`, () => { // @ts-expect-error we're testing for throwing when there's no config passed in expect(() => createCollection()).toThrow(CollectionRequiresConfigError) }) - it(`removes optimistic insert when sync confirms with a different server-generated key`, async () => { + it(`throws DuplicateKeySyncError instead of TypeError when config has no utils`, async () => { + let begin!: () => void + let write!: Parameters< + SyncConfig<{ id: number; text: string }, number>[`sync`] + >[0][`write`] + + const collection = createCollection<{ id: number; text: string }, number>({ + id: `duplicate-key-no-utils-test`, + getKey: (item) => item.id, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + params.begin() + params.write({ type: `insert`, value: { id: 1, text: `one` } }) + params.commit() + params.markReady() + }, + }, + }) + + await collection.stateWhenReady() + + begin() + expect(() => + write({ type: `insert`, value: { id: 1, text: `changed` } }), + ).toThrow(DuplicateKeySyncError) + }) + + it(`keeps ambiguous server-key sync queued while a temp-key optimistic insert is pending`, async () => { const options = mockSyncCollectionOptionsNoInitialState<{ id: number text: string @@ -67,6 +145,11 @@ describe(`Collection`, () => { options.utils.commit() // The sync commit is held while the local insert transaction is persisting. + // Without an explicit temp-key -> server-key mapping, core cannot know + // whether key 24 is this optimistic insert's server echo or an unrelated + // row, so it must not expose both rows at the same time. + expect(tx.isPersisted.isPending()).toBe(true) + expect(collection.has(24)).toBe(false) expect(getStateEntries(collection)).toEqual([ [4733, { id: 4733, text: `two` }], ]) @@ -92,14 +175,14 @@ describe(`Collection`, () => { const liveCollection = createLiveQueryCollection((q) => q .from({ collection }) - .where(({ collection }) => eq(collection.project_id, 1)) - .select(({ collection }) => ({ - id: collection.id, - text: collection.text, - project_id: collection.project_id, - $synced: collection.$synced, - $origin: collection.$origin, - $key: collection.$key, + .where(({ collection: item }) => eq(item.project_id, 1)) + .select(({ collection: item }) => ({ + id: item.id, + text: item.text, + project_id: item.project_id, + $synced: item.$synced, + $origin: item.$origin, + $key: item.$key, })), ) @@ -2142,6 +2225,45 @@ describe(`Collection isLoadingSubset property`, () => { expect(collection.isLoadingSubset).toBe(false) }) + it(`cleanup isolates subset loading state from a later sync session`, async () => { + const resolveLoads: Array<() => void> = [] + const collection = createCollection<{ id: string; value: string }>({ + id: `cleanup-isolates-subset-loading`, + getKey: (item) => item.id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => + new Promise((resolve) => resolveLoads.push(resolve)), + } + }, + }, + }) + + collection._sync.loadSubset({}) + expect(collection.isLoadingSubset).toBe(true) + + await collection.cleanup() + expect(collection.isLoadingSubset).toBe(false) + + collection.startSyncImmediate() + collection._sync.loadSubset({}) + expect(collection.isLoadingSubset).toBe(true) + + resolveLoads[0]!() + await flushPromises() + expect(collection.isLoadingSubset).toBe(true) + + resolveLoads[1]!() + await flushPromises() + expect(collection.isLoadingSubset).toBe(false) + + await collection.cleanup() + }) + it(`emits loadingSubset:change event`, async () => { let resolveLoadSubset: () => void const loadSubsetPromise = new Promise((resolve) => { @@ -2250,4 +2372,84 @@ describe(`Collection isLoadingSubset property`, () => { expect(result).toBe(true) expect(collection.isLoadingSubset).toBe(false) }) + + it(`rejects an already-aborted subset request before the adapter branch`, async () => { + const loadSubset = vi.fn(() => true as const) + const collection = createCollection<{ id: string; value: string }>({ + id: `already-aborted-subset-request`, + getKey: (item) => item.id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset } + }, + }, + }) + const request = new AbortController() + request.abort() + + await expect( + collection._sync.loadSubset({ signal: request.signal }), + ).rejects.toBeInstanceOf(LoadSubsetOperationAbortedError) + + expect(loadSubset).not.toHaveBeenCalled() + expect(collection.isLoadingSubset).toBe(false) + await collection.cleanup() + }) + + it(`rejects an already-aborted subset request before the eager return`, async () => { + const loadSubset = vi.fn(() => true as const) + const collection = createCollection<{ id: string; value: string }>({ + id: `already-aborted-eager-subset-request`, + getKey: (item) => item.id, + syncMode: `eager`, + startSync: true, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset } + }, + }, + }) + const request = new AbortController() + request.abort() + + await expect( + collection._sync.loadSubset({ signal: request.signal }), + ).rejects.toMatchObject({ name: `AbortError` }) + + expect(loadSubset).not.toHaveBeenCalled() + expect(collection.isLoadingSubset).toBe(false) + await collection.cleanup() + }) + + it(`rejects an already-aborted subset request before deferred start`, async () => { + const loadSubset = vi.fn(() => true as const) + const collection = createCollection<{ id: string; value: string }>({ + id: `already-aborted-deferred-subset-request`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const request = new AbortController() + request.abort() + + await expect( + collection._sync.loadSubset({ signal: request.signal }), + ).rejects.toMatchObject({ name: `AbortError` }) + + expect(loadSubset).not.toHaveBeenCalled() + expect(collection.isLoadingSubset).toBe(false) + collection._resumeSyncStart() + expect(loadSubset).not.toHaveBeenCalled() + await collection.cleanup() + }) }) diff --git a/packages/db/tests/comparison.property.test.ts b/packages/db/tests/comparison.property.test.ts index dd62790011..8b3fea8332 100644 --- a/packages/db/tests/comparison.property.test.ts +++ b/packages/db/tests/comparison.property.test.ts @@ -375,38 +375,53 @@ describe(`normalizeValue property-based tests`, () => { }) fcTest.prop([fc.uint8Array({ minLength: 0, maxLength: 128 })])( - `small Uint8Arrays normalize to string representation`, + `small Uint8Arrays normalize to a stable key`, (arr) => { const normalized = normalizeValue(arr) expect(typeof normalized).toBe(`string`) - expect(normalized).toMatch(/^__u8__/) + expect(normalized).toBe(normalizeValue(new Uint8Array(arr))) }, ) fcTest.prop([fc.uint8Array({ minLength: 129, maxLength: 200 })])( - `large Uint8Arrays are not normalized`, + `large Uint8Arrays normalize to a stable linear-size key`, (arr) => { const normalized = normalizeValue(arr) - expect(normalized).toBe(arr) + expect(typeof normalized).toBe(`string`) + expect(normalized).toBe(normalizeValue(new Uint8Array(arr))) + expect((normalized as string).length - arr.length).toBeLessThan(32) }, ) - fcTest.prop([fc.string()])(`strings pass through unchanged`, (str) => { - expect(normalizeValue(str)).toBe(str) - }) + fcTest.prop([fc.string()])( + `strings preserve equality after normalization`, + (str) => { + expect(normalizeValue(str)).toBe(normalizeValue(`${str}`)) + }, + ) fcTest.prop([fc.integer()])(`integers pass through unchanged`, (n) => { expect(normalizeValue(n)).toBe(n) }) fcTest.prop([fc.uint8Array({ minLength: 0, maxLength: 128 })])( - `normalization is idempotent for Uint8Arrays`, + `binary keys cannot collide with user strings`, (arr) => { - const normalized1 = normalizeValue(arr) - // For strings (which small arrays become), normalizing again should be identity - expect(normalizeValue(normalized1)).toBe(normalized1) + const normalized = normalizeValue(arr) + expect(normalizeValue(normalized)).not.toBe(normalized) }, ) + + fcTest(`reads binary keys from indexed bytes, not custom iteration`, () => { + const bytes = new Uint8Array([2]) + Object.defineProperty(bytes, Symbol.iterator, { + value: function* () { + yield 1 + }, + }) + + expect(normalizeValue(bytes)).toBe(normalizeValue(new Uint8Array([2]))) + }) }) describe(`areValuesEqual property-based tests`, () => { diff --git a/packages/db/tests/comparison.test.ts b/packages/db/tests/comparison.test.ts new file mode 100644 index 0000000000..870b23978b --- /dev/null +++ b/packages/db/tests/comparison.test.ts @@ -0,0 +1,96 @@ +import { describe, expect, it } from 'vitest' +import { Temporal } from 'temporal-polyfill' +import { + ascComparator, + compareValues, + defaultComparator, +} from '../src/utils/comparison' +import { DEFAULT_COMPARE_OPTIONS } from '../src/utils' + +describe(`ascComparator - PostgreSQL float semantics for NaN`, () => { + const opts = DEFAULT_COMPARE_OPTIONS // nulls: `first` + + it(`orders NaN greater than every number`, () => { + expect(ascComparator(NaN, 5, opts)).toBeGreaterThan(0) + expect(ascComparator(5, NaN, opts)).toBeLessThan(0) + }) + + it(`treats NaN as equal to NaN`, () => { + expect(ascComparator(NaN, NaN, opts)).toBe(0) + }) + + it(`produces a stable total order with NaN sorting last`, () => { + const sorted = [3, NaN, 1, 5, NaN].sort((a, b) => defaultComparator(a, b)) + + expect(sorted.slice(0, 3)).toEqual([1, 3, 5]) + expect(sorted.slice(3).every((v) => Number.isNaN(v))).toBe(true) + }) + + it(`keeps null before non-null values regardless of NaN`, () => { + // nulls still sort first by default; NaN sorts last (greatest non-null) + const sorted = [5, NaN, null, 1].sort((a, b) => defaultComparator(a, b)) + + expect(sorted[0]).toBe(null) + expect(sorted[1]).toBe(1) + expect(sorted[2]).toBe(5) + expect(Number.isNaN(sorted[3])).toBe(true) + }) + + it(`orders an invalid Date greater than valid Dates`, () => { + const invalid = new Date(`not a date`) + const valid = new Date(`2023-01-01`) + + expect(ascComparator(invalid, valid, opts)).toBeGreaterThan(0) + expect(ascComparator(valid, invalid, opts)).toBeLessThan(0) + }) +}) + +describe(`ascComparator - Temporal values`, () => { + const opts = DEFAULT_COMPARE_OPTIONS + + it(`orders PlainDate values by calendar date`, () => { + const earlier = new Temporal.PlainDate(2024, 1, 1) + const later = new Temporal.PlainDate(2024, 6, 1) + expect(ascComparator(earlier, later, opts)).toBeLessThan(0) + expect(ascComparator(later, earlier, opts)).toBeGreaterThan(0) + expect( + ascComparator(earlier, new Temporal.PlainDate(2024, 1, 1), opts), + ).toBe(0) + }) + + it(`treats ZonedDateTime values at the same instant as equal regardless of zone`, () => { + // 2024-01-15T06:30Z and 2024-01-15T12:00+05:30 are the same instant; + // lexicographic toString comparison would order them differently. + const utc = Temporal.ZonedDateTime.from(`2024-01-15T06:30:00+00:00[+00:00]`) + const ist = Temporal.ZonedDateTime.from(`2024-01-15T12:00:00+05:30[+05:30]`) + expect(ascComparator(utc, ist, opts)).toBe(0) + }) +}) + +describe(`ascComparator - symbols`, () => { + const opts = DEFAULT_COMPARE_OPTIONS + + it(`gives symbols a stable total order`, () => { + const first = Symbol(`group`) + const second = Symbol(`group`) + + expect(ascComparator(first, first, opts)).toBe(0) + expect(ascComparator(first, second, opts)).toBeLessThan(0) + expect(ascComparator(second, first, opts)).toBeGreaterThan(0) + expect(ascComparator(first, 1, opts)).toBeGreaterThan(0) + expect(ascComparator(1, first, opts)).toBeLessThan(0) + }) +}) + +describe(`compareValues - NaN behavior`, () => { + // NaN satisfies neither < nor >, so the fallback returns 0. In practice + // gt/gte/lt/lte catch NaN via isUnorderable before reaching compareValues. + it(`treats NaN as equal to NaN`, () => { + expect(compareValues(NaN, NaN)).toBe(0) + }) + + it(`returns 0 for NaN vs a finite number — neither < nor > holds for NaN`, () => { + expect(compareValues(NaN, 5)).toBe(0) + expect(compareValues(5, NaN)).toBe(0) + }) +}) diff --git a/packages/db/tests/conformance/contract.ts b/packages/db/tests/conformance/contract.ts new file mode 100644 index 0000000000..11534231b0 --- /dev/null +++ b/packages/db/tests/conformance/contract.ts @@ -0,0 +1,162 @@ +/** + * Cross-adapter live-query conformance harness — shared contract. + * + * ONE behavioral spec for `useLiveQuery`, run against every framework adapter. + * Each adapter provides a thin `LiveQueryDriver` and the shared suite in + * `suite.ts` does the rest. + * + * Realm safety: the driver — not the scenarios — creates source collections and + * supplies query operators, both imported from the *adapter's* copy of + * `@tanstack/db`. This keeps collection instances and expression nodes in the + * same module realm as the adapter's hook, avoiding the dual-package + * `instanceof CollectionImpl` mismatch. Scenarios never import `@tanstack/db`. + * + * Expected-fail policy: `knownGaps` lists scenario KEYS this adapter does not + * yet satisfy. Populate it EMPIRICALLY — port a behavior, run it, and only add + * the key if it actually fails. The behavior matrix tells you where to look; + * the test run tells you what's broken. When a gap closes, `it.fails` errors + * ("expected to fail but passed") prompting you to delete the key. + */ +import type { Collection } from '@tanstack/db' + +/** Default row shape used by the base scenarios (a "person"). */ +export interface Row { + id: string + name: string + age: number + team: string +} + +/** + * A realm-correct source collection plus sync-driven mutators. Generic over the + * row type so relational scenarios (join, includes) can build differently-shaped + * related collections; keyed by `id` in every case. + */ +export interface SourceHandle { + collection: Collection + insert: (row: T) => void + update: (row: T) => void + remove: (row: T) => void +} + +/** + * A source whose readiness the scenario controls, for loading/eager/ready + * transitions. Starts in `loading` (synced, not ready) so scenarios can `emit` + * rows while still loading and then `markReady`. + */ +export interface DeferredSourceHandle< + T extends { id: string } = Row, +> extends SourceHandle { + /** Write rows without marking ready — exercises the eager (visible-while-loading) path. */ + emit: (rows: ReadonlyArray) => void + /** Transition the source from `loading` to `ready`. */ + markReady: () => void +} + +/** + * The subset of `@tanstack/db` query operators scenarios need, supplied by the + * driver from the adapter's realm. Grows as engine scenarios are ported. + */ +export interface DbOps { + eq: (a: any, b: any) => any + gt: (a: any, b: any) => any + count: (a: any) => any + sum: (a: any) => any + coalesce: (...args: Array) => any + /** Build an optimistic action: onMutate applies optimistic state, mutationFn confirms. */ + createOptimisticAction: (config: { + onMutate: (variables: any) => void + mutationFn: (variables: any) => Promise + }) => (variables?: any) => { isPersisted: { promise: Promise } } +} + +/** Normalized, adapter-agnostic view of a live query's current result. */ +export interface ConformanceResult { + /** Array for list queries; a single row (or undefined) for `findOne`. */ + data: any + /** + * The keyed result map (`undefined` when disabled). Exposed so scenarios can + * assert the granular map stays in sync with `data` — e.g. that stale keys + * from a previous collection don't linger after a recompile. + */ + state: ReadonlyMap | undefined + status: string + isReady: boolean + isError: boolean + isEnabled: boolean +} + +/** A query-builder callback, e.g. `(q) => q.from({ items: source.collection })`. */ +export type QueryBuild = (q: any) => any + +/** A mounted live query under test. */ +export interface LiveQueryHandle { + current: () => ConformanceResult + /** Let the framework scheduler + core sync settle, then resolve. */ + flush: () => Promise + /** + * Run a state-mutating callback inside the framework's update scope, then + * settle (React `act`, Vue `nextTick`, Svelte `flushSync`, Solid `batch`). + * Needed when a mutation notifies synchronously, e.g. optimistic actions. + */ + apply: (fn: () => void) => Promise + unmount: () => void +} + +/** + * A mounted query whose input parameter can change after mount, for + * recompilation and disabled/enabled transitions. `setParam` re-renders with the + * new value and settles. + */ +export interface ControllableHandle

                      extends LiveQueryHandle { + setParam: (param: P) => Promise +} + +/** What each adapter package implements and hands to `runSuite`. */ +export interface LiveQueryDriver { + name: string + /** Operators from the adapter's `@tanstack/db` realm. */ + ops: DbOps + /** Create a realm-correct source collection + mutators, keyed by `id`. */ + makeSource: ( + initialData: ReadonlyArray, + ) => SourceHandle + /** Create a source that starts `loading` and readies on demand (keyed by `id`). */ + makeDeferredSource: () => DeferredSourceHandle + /** + * Create a pre-built live-query collection to pass straight to the hook. + * `startSync: false` yields a not-yet-syncing collection (isReady false). + */ + makePrecreated: ( + build: QueryBuild, + opts?: { startSync?: boolean }, + ) => { collection: Collection } + /** Create a source whose sync fails, driving it into `error` status. */ + makeErrorSource: () => { collection: Collection } + /** Mount a live query from a query-builder callback. */ + mount: (build: QueryBuild) => LiveQueryHandle + /** + * Mount a live query whose input depends on a parameter that can change after + * mount. `build` returns a query, or `null`/`undefined` to represent disabled. + */ + mountControllable:

                      ( + build: (q: any, param: P) => any, + initial: P, + ) => ControllableHandle

                      + /** Mount a pre-created collection passed directly to the hook. */ + mountCollection: (collection: Collection) => LiveQueryHandle + /** Mount via the config-object input form (`{ query: build }`). */ + mountConfig: (build: QueryBuild) => LiveQueryHandle + /** Mount an explicitly-disabled query (adapter's own null/undefined form). */ + mountDisabled: () => LiveQueryHandle + /** Scenario keys this adapter is empirically known NOT to satisfy yet. */ + knownGaps?: ReadonlyArray + /** + * How the adapter surfaces a query error (see the `error-status` scenario): + * - `flag` (default): a readable `isError`/`status === 'error'` on the result. + * - `throw`: reading the errored result throws, for a framework error boundary + * to catch (e.g. Solid's `createResource`/`` model). + */ + errorSurface?: `flag` | `throw` + features?: { serverSnapshot?: boolean; suspense?: boolean } +} diff --git a/packages/db/tests/conformance/infinite-contract.ts b/packages/db/tests/conformance/infinite-contract.ts new file mode 100644 index 0000000000..09fd67b37e --- /dev/null +++ b/packages/db/tests/conformance/infinite-contract.ts @@ -0,0 +1,101 @@ +/** + * Cross-adapter contract for `useLiveInfiniteQuery`. + * + * Drivers keep framework scheduling and package-realm details out of the shared + * scenarios. Unlike the ordinary live-query contract, controllable handles can + * mutate inputs without settling so the suite can exercise imperative calls in + * the invalidation-to-subscription interval. + */ +import type { Collection } from '@tanstack/db' +import type { QueryBuild, SourceHandle } from './contract' + +export interface InfiniteQueryConfig { + pageSize?: number + initialPageParam?: number +} + +export interface InfiniteQueryResult { + data: Array + pages: Array> + pageParams: Array + hasNextPage: boolean + isFetchingNextPage: boolean + error: unknown + status: string + collection: Collection +} + +export interface InfiniteQueryHandle { + current: () => InfiniteQueryResult + /** Invoke a page fetch and wait until its observable request settles. */ + fetchNextPage: () => Promise + flush: () => Promise + apply: (fn: () => void) => Promise + unmount: () => void +} + +export interface InfiniteQueryControllableHandle< + P, +> extends InfiniteQueryHandle { + /** Change a query dependency without waiting for the framework to settle. */ + setParamSync: (param: P) => void +} + +export interface InfiniteQueryCollectionHandle extends InfiniteQueryHandle { + /** Replace the input collection without waiting for the framework to settle. */ + replaceCollectionSync: (collection: Collection) => void +} + +export interface InfiniteQueryConfigHandle extends InfiniteQueryHandle { + /** Replace reactive page-shape options without waiting for the framework. */ + setConfigSync: (config: InfiniteQueryConfig) => void +} + +export interface InfiniteQueryInputHandle extends InfiniteQueryHandle { + setInputKindSync: (kind: `collection` | `query`) => void +} + +export interface InfiniteQueryDriver { + name: string + gt: (a: any, b: any) => any + makeSource: ( + initialData: ReadonlyArray, + ) => SourceHandle + makeOnDemandSource: ( + data: ReadonlyArray, + asyncDelay?: number, + ) => { + collection: Collection + calls: Array<{ limit?: number }> + } + makePrecreated: (build: QueryBuild) => { + collection: Collection + } + mount: ( + build: QueryBuild, + config?: InfiniteQueryConfig, + ) => InfiniteQueryHandle + mountControllable:

                      ( + build: (q: any, param: P) => any, + initial: P, + config?: InfiniteQueryConfig, + ) => InfiniteQueryControllableHandle

                      + mountCollection: ( + collection: Collection, + config?: InfiniteQueryConfig, + ) => InfiniteQueryHandle + mountCollectionControllable: ( + collection: Collection, + config?: InfiniteQueryConfig, + ) => InfiniteQueryCollectionHandle + mountConfigControllable: ( + build: QueryBuild, + initial: InfiniteQueryConfig, + ) => InfiniteQueryConfigHandle + mountInputControllable: ( + collection: Collection, + build: QueryBuild, + config?: InfiniteQueryConfig, + ) => InfiniteQueryInputHandle + knownGaps?: ReadonlyArray +} diff --git a/packages/db/tests/conformance/infinite-on-demand.ts b/packages/db/tests/conformance/infinite-on-demand.ts new file mode 100644 index 0000000000..0b265b1260 --- /dev/null +++ b/packages/db/tests/conformance/infinite-on-demand.ts @@ -0,0 +1,62 @@ +import { createFilterFunctionFromExpression } from '../../src/collection/change-events.js' +import type { Collection, LoadSubsetOptions } from '../../src/index.js' + +interface Runtime { + BTreeIndex: unknown + createCollection: ( + options: any, + ) => Collection +} + +let sequence = 0 + +export function makeInfiniteOnDemandSource< + T extends { id: string; rank: number }, +>(runtime: Runtime, data: ReadonlyArray, asyncDelay?: number) { + const calls: Array = [] + const collection = runtime.createCollection({ + id: `infinite-conformance-on-demand-${sequence++}`, + getKey: (row: T) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: runtime.BTreeIndex, + sync: { + sync: ({ markReady, begin, write, commit }: any) => { + markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + calls.push({ ...options }) + let requested = [...data].sort((a, b) => b.rank - a.rank) + if (options.cursor) { + const filter = createFilterFunctionFromExpression( + options.cursor.whereFrom, + ) + requested = requested.filter(filter) + } + if (options.limit !== undefined) { + requested = requested.slice(0, options.limit) + } + + const load = () => { + begin() + for (const row of requested) write({ type: `insert`, value: row }) + commit() + } + if (asyncDelay === undefined) { + load() + return true + } + return new Promise((resolve) => { + setTimeout(() => { + load() + resolve() + }, asyncDelay) + }) + }, + } + }, + }, + }) + return { collection, calls } +} diff --git a/packages/db/tests/conformance/infinite-suite.ts b/packages/db/tests/conformance/infinite-suite.ts new file mode 100644 index 0000000000..829fa6bf44 --- /dev/null +++ b/packages/db/tests/conformance/infinite-suite.ts @@ -0,0 +1,1050 @@ +/** Shared behavioral suite for every `useLiveInfiniteQuery` adapter. */ +import { describe, expect, it, vi } from 'vitest' +import type { + InfiniteQueryDriver, + InfiniteQueryHandle, +} from './infinite-contract' + +interface InfiniteRow { + id: string + label: string + rank: number +} + +function rows(count: number, prefix = ``): Array { + return Array.from({ length: count }, (_, index) => ({ + id: `${prefix}${index + 1}`, + label: `${prefix || `row`}-${index + 1}`, + rank: count - index, + })) +} + +async function captureError(fn: () => InfiniteQueryHandle): Promise { + try { + const handle = fn() + await handle.flush() + handle.unmount() + return undefined + } catch (error) { + return error + } +} + +async function waitFor(check: () => boolean): Promise { + for (let attempt = 0; attempt < 20; attempt++) { + if (check()) return + await Promise.resolve() + } + throw new Error(`Condition did not become true`) +} + +async function waitForAsync(check: () => boolean): Promise { + for (let attempt = 0; attempt < 50; attempt++) { + if (check()) return + await new Promise((resolve) => setTimeout(resolve, 5)) + } + throw new Error(`Condition did not become true`) +} + +export function runInfiniteQuerySuite(rawDriver: InfiniteQueryDriver): void { + const gaps = new Set(rawDriver.knownGaps ?? []) + const registeredKeys = new Set() + let mounted: Array | null = null + + const track = (handle: H): H => { + mounted?.push(handle) + return handle + } + const driver: InfiniteQueryDriver = { + ...rawDriver, + mount: (build, config) => track(rawDriver.mount(build, config)), + mountControllable: (build, initial, config) => + track(rawDriver.mountControllable(build, initial, config)), + mountCollection: (collection, config) => + track(rawDriver.mountCollection(collection, config)), + mountCollectionControllable: (collection, config) => + track(rawDriver.mountCollectionControllable(collection, config)), + mountConfigControllable: (build, config) => + track(rawDriver.mountConfigControllable(build, config)), + mountInputControllable: (collection, build, config) => + track(rawDriver.mountInputControllable(collection, build, config)), + } + + const scenario = ( + key: string, + name: string, + fn: () => Promise | void, + ) => { + registeredKeys.add(key) + const expectFail = gaps.has(key) + const label = `[${key}] ${name}${expectFail ? ` (expected-fail)` : ``}` + const run = async () => { + const handles: Array = [] + mounted = handles + try { + await fn() + } finally { + mounted = null + for (const handle of handles) { + try { + handle.unmount() + } catch { + // Teardown is best-effort and idempotent. + } + } + } + } + if (expectFail) it.fails(label, run) + else it(label, run) + } + + describe(`infinite-query conformance :: ${driver.name}`, () => { + scenario( + `page-expansion`, + `loads the initial page and expands through the final partial page`, + async () => { + const source = driver.makeSource(rows(8)) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3, initialPageParam: 4 }, + ) + await handle.flush() + + expect(handle.current().data.map((row) => row.id)).toEqual([ + `1`, + `2`, + `3`, + ]) + expect(handle.current().pages.map((page) => page.length)).toEqual([3]) + expect(handle.current().pageParams).toEqual([4]) + expect(handle.current().hasNextPage).toBe(true) + + await handle.fetchNextPage() + await handle.flush() + expect(handle.current().data.map((row) => row.id)).toEqual([ + `1`, + `2`, + `3`, + `4`, + `5`, + `6`, + ]) + expect(handle.current().pageParams).toEqual([4, 5]) + + await handle.fetchNextPage() + await handle.flush() + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 3, 2, + ]) + expect(handle.current().pageParams).toEqual([4, 5, 6]) + expect(handle.current().hasNextPage).toBe(false) + }, + ) + + scenario( + `boundary-noop`, + `does not add a page after the end of the result`, + async () => { + const source = driver.makeSource(rows(2)) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await handle.flush() + + await handle.fetchNextPage() + await handle.flush() + expect(handle.current().pages.map((page) => page.length)).toEqual([2]) + expect(handle.current().hasNextPage).toBe(false) + }, + ) + + scenario( + `empty-result`, + `represents an empty result as one empty page`, + async () => { + const source = driver.makeSource(rows(0)) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await handle.flush() + + expect(handle.current().data).toEqual([]) + expect(handle.current().pages).toEqual([[]]) + expect(handle.current().hasNextPage).toBe(false) + }, + ) + + scenario( + `exact-boundary`, + `detects the end when the result fills the final page exactly`, + async () => { + const source = driver.makeSource(rows(6)) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await handle.flush() + await handle.fetchNextPage() + await handle.flush() + + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 3, + ]) + expect(handle.current().hasNextPage).toBe(false) + }, + ) + + scenario( + `live-window`, + `keeps all committed pages live when a row enters the window`, + async () => { + const source = driver.makeSource(rows(8)) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await handle.flush() + await handle.fetchNextPage() + await handle.flush() + + await handle.apply(() => { + source.insert({ id: `new`, label: `new`, rank: 100 }) + }) + expect(handle.current().data.map((row) => row.id)).toEqual([ + `new`, + `1`, + `2`, + `3`, + `4`, + `5`, + ]) + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 3, + ]) + }, + ) + + scenario( + `live-deletion`, + `backfills committed pages when rows are deleted`, + async () => { + const data = rows(8) + const source = driver.makeSource(data) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await handle.flush() + await handle.fetchNextPage() + await handle.flush() + + await handle.apply(() => source.remove(data[1]!)) + expect(handle.current().data.map((row) => row.id)).toEqual([ + `1`, + `3`, + `4`, + `5`, + `6`, + `7`, + ]) + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 3, + ]) + }, + ) + + scenario( + `partial-page-deletion`, + `removes rows from a partial page in either order direction`, + async () => { + for (const direction of [`desc`, `asc`] as const) { + const data = rows(5, direction) + const source = driver.makeSource(data) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, direction), + { pageSize: 20 }, + ) + await handle.flush() + + const removed = direction === `desc` ? data[0]! : data[4]! + await handle.apply(() => source.remove(removed)) + expect(handle.current().data.map((row) => row.id)).not.toContain( + removed.id, + ) + expect(handle.current().pages.map((page) => page.length)).toEqual([4]) + expect(handle.current().hasNextPage).toBe(false) + handle.unmount() + } + }, + ) + + scenario( + `live-has-next-page`, + `updates hasNextPage when a row is inserted beyond the visible page`, + async () => { + const source = driver.makeSource(rows(3)) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await handle.flush() + expect(handle.current().hasNextPage).toBe(false) + + await handle.apply(() => { + source.insert({ id: `last`, label: `last`, rank: 0 }) + }) + expect(handle.current().data.map((row) => row.id)).toEqual([ + `1`, + `2`, + `3`, + ]) + expect(handle.current().hasNextPage).toBe(true) + }, + ) + + scenario( + `concurrent-fetch`, + `coalesces concurrent next-page requests`, + async () => { + const source = driver.makeSource(rows(8)) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await handle.flush() + + const utils = handle.current().collection.utils as { + setWindow: (window: { + offset: number + limit: number + }) => true | Promise + } + const originalSetWindow = utils.setWindow.bind(utils) + let calls = 0 + let resolveWindow: (() => void) | undefined + utils.setWindow = (window) => { + calls++ + originalSetWindow(window) + return new Promise((resolve) => { + resolveWindow = resolve + }) + } + + try { + const first = handle.fetchNextPage() + const second = handle.fetchNextPage() + let secondSettled = false + void second.then( + () => { + secondSettled = true + }, + () => { + secondSettled = true + }, + ) + await waitFor(() => resolveWindow !== undefined) + + expect(calls).toBe(1) + expect(handle.current().isFetchingNextPage).toBe(true) + expect(secondSettled).toBe(false) + resolveWindow?.() + await Promise.all([first, second]) + await handle.flush() + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 3, + ]) + } finally { + utils.setWindow = originalSetWindow + } + }, + ) + + scenario( + `fetch-settlement`, + `settles the driver operation with the window request`, + async () => { + const source = driver.makeSource(rows(8)) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await handle.flush() + + const utils = handle.current().collection.utils as { + setWindow: (window: { + offset: number + limit: number + }) => true | Promise + } + const originalSetWindow = utils.setWindow.bind(utils) + let resolveWindow: (() => void) | undefined + utils.setWindow = (window) => { + originalSetWindow(window) + return new Promise((resolve) => { + resolveWindow = resolve + }) + } + + try { + let settled = false + const fetch = handle.fetchNextPage().then(() => { + settled = true + }) + await waitFor(() => resolveWindow !== undefined) + await Promise.resolve() + expect(settled).toBe(false) + resolveWindow?.() + await fetch + expect(settled).toBe(true) + } finally { + utils.setWindow = originalSetWindow + } + }, + ) + + scenario( + `on-demand-paging`, + `uses peek-ahead windows while paging an on-demand source`, + async () => { + const source = driver.makeOnDemandSource(rows(8)) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await handle.flush() + + expect(source.calls.some((call) => call.limit === 4)).toBe(true) + expect(handle.current().data.map((row) => row.id)).toEqual([ + `1`, + `2`, + `3`, + ]) + expect(handle.current().hasNextPage).toBe(true) + + await handle.fetchNextPage() + await handle.fetchNextPage() + await handle.flush() + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 3, 2, + ]) + expect(handle.current().hasNextPage).toBe(false) + }, + ) + + scenario( + `on-demand-async`, + `tracks an asynchronous on-demand page load`, + async () => { + const source = driver.makeOnDemandSource(rows(8), 5) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await waitForAsync(() => handle.current().data.length === 3) + + const fetch = handle.fetchNextPage() + await waitForAsync(() => handle.current().isFetchingNextPage) + await fetch + await handle.flush() + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 3, + ]) + expect(handle.current().isFetchingNextPage).toBe(false) + }, + ) + + scenario( + `window-failure`, + `surfaces a failed window request in state without rejecting`, + async () => { + const source = driver.makeSource(rows(8)) + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await handle.flush() + + const failure = new Error(`window failed`) + const utils = handle.current().collection.utils as { + setWindow: (window: { + offset: number + limit: number + }) => true | Promise + } + const originalSetWindow = utils.setWindow.bind(utils) + utils.setWindow = (window) => { + originalSetWindow(window) + return Promise.reject(failure) + } + + try { + await expect(handle.fetchNextPage()).resolves.toBeUndefined() + await handle.flush() + expect(handle.current().pages.map((page) => page.length)).toEqual([3]) + expect(handle.current().status).toBe(`error`) + expect(handle.current().error).toBe(failure) + + utils.setWindow = originalSetWindow + await handle.fetchNextPage() + await handle.flush() + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 3, + ]) + expect(handle.current().error).toBeUndefined() + } finally { + utils.setWindow = originalSetWindow + } + }, + ) + + scenario( + `dependency-immediate-fetch`, + `fetches from the replacement query before the framework settles`, + async () => { + const source = driver.makeSource(rows(10)) + const handle = driver.mountControllable( + (q, minimum: number) => + q + .from({ items: source.collection }) + .where(({ items }: any) => driver.gt(items.rank, minimum)) + .orderBy(({ items }: any) => items.rank, `desc`), + 0, + { pageSize: 3 }, + ) + await handle.flush() + await handle.fetchNextPage() + await handle.flush() + + handle.setParamSync(5) + await handle.fetchNextPage() + await handle.flush() + + expect(handle.current().data.map((row) => row.rank)).toEqual([ + 10, 9, 8, 7, 6, + ]) + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 2, + ]) + }, + ) + + scenario( + `equal-dependency-depth`, + `preserves loaded pages for a structurally equal dependency`, + async () => { + const source = driver.makeSource(rows(10)) + const handle = driver.mountControllable( + (q, filter: { minimum: number }) => + q + .from({ items: source.collection }) + .where(({ items }: any) => driver.gt(items.rank, filter.minimum)) + .orderBy(({ items }: any) => items.rank, `desc`), + { minimum: 0 }, + { pageSize: 3 }, + ) + await handle.flush() + await handle.fetchNextPage() + await handle.flush() + + handle.setParamSync({ minimum: 0 }) + await handle.flush() + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 3, + ]) + }, + ) + + scenario( + `circular-dependency`, + `preserves page depth for a structurally equal circular dependency`, + async () => { + const source = driver.makeSource(rows(8)) + const dependency: { self?: unknown } = {} + dependency.self = dependency + const handle = driver.mountControllable( + (q, _dependency: unknown) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + dependency, + { pageSize: 3 }, + ) + await handle.flush() + await handle.fetchNextPage() + await handle.flush() + + const replacement: { self?: unknown } = {} + replacement.self = replacement + handle.setParamSync(replacement) + await handle.flush() + + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 3, + ]) + }, + ) + + scenario( + `page-shape-change`, + `preserves committed page depth when reactive page options change`, + async () => { + const source = driver.makeSource(rows(20)) + const handle = driver.mountConfigControllable( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3, initialPageParam: 4 }, + ) + await handle.flush() + await handle.fetchNextPage() + await handle.fetchNextPage() + await handle.flush() + + handle.setConfigSync({ pageSize: 4, initialPageParam: 8 }) + await handle.flush() + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 4, 4, 4, + ]) + expect(handle.current().pageParams).toEqual([8, 9, 10]) + }, + ) + + scenario( + `invalid-page-size`, + `normalizes invalid and unsafe page sizes to the default`, + async () => { + const source = driver.makeSource(rows(21)) + for (const pageSize of [ + 0, + -1, + 2.5, + Number.NaN, + Number.POSITIVE_INFINITY, + Number.MAX_SAFE_INTEGER, + ]) { + const handle = driver.mount( + (q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize }, + ) + await handle.flush() + expect(handle.current().pages[0]).toHaveLength(20) + expect(handle.current().hasNextPage).toBe(true) + handle.unmount() + } + }, + ) + + scenario( + `collection-immediate-fetch`, + `fetches from a replacement collection before the framework settles`, + async () => { + const first = driver.makeSource(rows(8, `a`)) + const second = driver.makeSource(rows(8, `b`)) + const firstQuery = driver.makePrecreated((q) => + q + .from({ items: first.collection }) + .orderBy(({ items }: any) => items.rank, `desc`) + .limit(4), + ).collection + const secondQuery = driver.makePrecreated((q) => + q + .from({ items: second.collection }) + .orderBy(({ items }: any) => items.rank, `desc`) + .limit(4), + ).collection + const handle = driver.mountCollectionControllable(firstQuery, { + pageSize: 3, + }) + await handle.flush() + await handle.fetchNextPage() + await handle.flush() + + handle.replaceCollectionSync(secondQuery) + await handle.fetchNextPage() + await handle.flush() + + expect(handle.current().collection).toBe(secondQuery) + expect(handle.current().data.map((row) => row.id)).toEqual([ + `b1`, + `b2`, + `b3`, + `b4`, + `b5`, + `b6`, + ]) + expect(handle.current().pages.map((page) => page.length)).toEqual([ + 3, 3, + ]) + }, + ) + + scenario( + `input-kind-switch`, + `switches between a supplied collection and a query callback`, + async () => { + const collectionSource = driver.makeSource(rows(6, `a`)) + const querySource = driver.makeSource(rows(6, `b`)) + const collection = driver.makePrecreated((q) => + q + .from({ items: collectionSource.collection }) + .orderBy(({ items }: any) => items.rank, `desc`) + .limit(4), + ).collection + const handle = driver.mountInputControllable( + collection, + (q) => + q + .from({ items: querySource.collection }) + .orderBy(({ items }: any) => items.rank, `desc`), + { pageSize: 3 }, + ) + await handle.flush() + expect(handle.current().data.map((row) => row.id)).toEqual([ + `a1`, + `a2`, + `a3`, + ]) + + handle.setInputKindSync(`query`) + await handle.flush() + expect(handle.current().data.map((row) => row.id)).toEqual([ + `b1`, + `b2`, + `b3`, + ]) + + handle.setInputKindSync(`collection`) + await handle.flush() + expect(handle.current().data.map((row) => row.id)).toEqual([ + `a1`, + `a2`, + `a3`, + ]) + }, + ) + + scenario( + `stale-window`, + `ignores a window promise from a replaced query`, + async () => { + const source = driver.makeSource(rows(10)) + const handle = driver.mountControllable( + (q, minimum: number) => + q + .from({ items: source.collection }) + .where(({ items }: any) => driver.gt(items.rank, minimum)) + .orderBy(({ items }: any) => items.rank, `desc`), + 0, + { pageSize: 3 }, + ) + await handle.flush() + + const oldUtils = handle.current().collection.utils as { + setWindow: (window: { + offset: number + limit: number + }) => true | Promise + } + const originalSetWindow = oldUtils.setWindow.bind(oldUtils) + let resolveWindow: (() => void) | undefined + oldUtils.setWindow = (window) => { + const result = originalSetWindow(window) + if (resolveWindow !== undefined) return result + return new Promise((resolve) => { + resolveWindow = resolve + }) + } + + try { + const staleFetch = handle.fetchNextPage() + await waitFor(() => resolveWindow !== undefined) + handle.setParamSync(8) + await handle.flush() + resolveWindow?.() + await staleFetch.catch(() => {}) + await handle.flush() + + expect(handle.current().data.map((row) => row.rank)).toEqual([10, 9]) + expect(handle.current().pages.map((page) => page.length)).toEqual([2]) + } finally { + oldUtils.setWindow = originalSetWindow + } + }, + ) + + scenario( + `callback-once`, + `invokes a zero-arity-compatible query callback once`, + async () => { + const source = driver.makeSource(rows(4)) + let calls = 0 + const callback = (...args: Array) => { + calls++ + return args[0] + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`) + } + const handle = driver.mount(callback, { pageSize: 3 }) + await handle.flush() + + expect(calls).toBe(1) + expect(handle.current().data).toHaveLength(3) + }, + ) + + scenario( + `callback-error`, + `surfaces an error thrown while constructing the query`, + async () => { + const failure = new Error(`query construction failed`) + const error = await captureError(() => + driver.mount( + ((..._args: Array) => { + throw failure + }) as any, + { pageSize: 3 }, + ), + ) + expect(error).toBe(failure) + }, + ) + + scenario( + `disabled-callback`, + `rejects a nullable query callback through the shared input policy`, + async () => { + const error = await captureError(() => + driver.mount((() => null) as any, { pageSize: 3 }), + ) + expect(error).toBeInstanceOf(Error) + expect((error as Error).message).toContain( + `Disabled null or undefined queries are not supported`, + ) + }, + ) + + scenario( + `findone-runtime`, + `rejects a single-result query at runtime`, + async () => { + const source = driver.makeSource(rows(4)) + const error = await captureError(() => + driver.mount( + ((q: any) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`) + .findOne()) as any, + { pageSize: 3 }, + ), + ) + expect(error).toBeInstanceOf(Error) + expect((error as Error).message).toContain(`Remove .findOne()`) + }, + ) + + scenario( + `unordered-collection`, + `rejects a pre-created collection without orderBy`, + async () => { + const source = driver.makeSource(rows(4)) + const unordered = driver.makePrecreated((q) => + q.from({ items: source.collection }), + ).collection + const error = await captureError(() => + driver.mountCollection(unordered, { pageSize: 3 }), + ) + expect(error).toBeInstanceOf(Error) + expect((error as Error).message).toMatch(/orderBy|ORDER BY/) + }, + ) + + scenario( + `unordered-query`, + `rejects a query callback without orderBy`, + async () => { + const source = driver.makeSource(rows(4)) + const error = await captureError(() => + driver.mount((q) => q.from({ items: source.collection }), { + pageSize: 3, + }), + ) + expect(error).toBeInstanceOf(Error) + expect((error as Error).message).toMatch(/orderBy|ORDER BY/) + }, + ) + + scenario( + `findone-collection`, + `rejects a pre-created single-result collection`, + async () => { + const source = driver.makeSource(rows(4)) + const single = driver.makePrecreated(((q: any) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`) + .findOne()) as any).collection + const error = await captureError(() => + driver.mountCollection(single, { pageSize: 3 }), + ) + expect(error).toBeInstanceOf(Error) + expect((error as Error).message).toContain(`Remove .findOne()`) + }, + ) + + scenario( + `collection-window-normalization`, + `normalizes a pre-created collection to the first peek-ahead window`, + async () => { + const source = driver.makeSource(rows(8)) + const collection = driver.makePrecreated((q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`) + .offset(1) + .limit(2), + ).collection + const warn = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const handle = driver.mountCollection(collection, { pageSize: 3 }) + await handle.flush() + + expect(warn).toHaveBeenCalledWith( + expect.stringContaining(`Pre-created collection has window`), + ) + expect( + (collection.utils as { getWindow: () => unknown }).getWindow(), + ).toEqual({ offset: 0, limit: 4 }) + expect(handle.current().data.map((row) => row.id)).toEqual([ + `1`, + `2`, + `3`, + ]) + warn.mockRestore() + }, + ) + + scenario( + `collection-live-update`, + `keeps a supplied pre-created collection live`, + async () => { + const source = driver.makeSource(rows(6)) + const collection = driver.makePrecreated((q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`) + .limit(4), + ).collection + const handle = driver.mountCollection(collection, { pageSize: 3 }) + await handle.flush() + + await handle.apply(() => { + source.insert({ id: `new`, label: `new`, rank: 100 }) + }) + expect(handle.current().data.map((row) => row.id)).toEqual([ + `new`, + `1`, + `2`, + ]) + }, + ) + + scenario( + `invalid-input`, + `rejects a first argument that is neither a query nor a collection`, + async () => { + const error = await captureError(() => + driver.mount(null as unknown as Parameters[0], { + pageSize: 3, + }), + ) + expect(error).toBeInstanceOf(Error) + expect((error as Error).message).toContain(`First argument`) + }, + ) + + scenario( + `shared-window-release`, + `releases shared window leases and restores the initial window`, + async () => { + const source = driver.makeSource(rows(12)) + const collection = driver.makePrecreated((q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.rank, `desc`) + .limit(4), + ).collection + const warn = vi.spyOn(console, `warn`).mockImplementation(() => {}) + try { + const larger = driver.mountCollection(collection, { pageSize: 3 }) + const smaller = driver.mountCollection(collection, { pageSize: 1 }) + await larger.flush() + await smaller.flush() + await larger.fetchNextPage() + await smaller.fetchNextPage() + await larger.flush() + + const getWindow = () => + (collection.utils as { getWindow: () => unknown }).getWindow() + expect(getWindow()).toEqual({ offset: 0, limit: 7 }) + larger.unmount() + await smaller.flush() + expect(getWindow()).toEqual({ offset: 0, limit: 3 }) + smaller.unmount() + expect(getWindow()).toEqual({ offset: 0, limit: 4 }) + expect(warn).not.toHaveBeenCalled() + } finally { + warn.mockRestore() + } + }, + ) + + it(`has no stale known-gap keys`, () => { + expect([...gaps].filter((key) => !registeredKeys.has(key))).toEqual([]) + }) + }) +} diff --git a/packages/db/tests/conformance/suite.ts b/packages/db/tests/conformance/suite.ts new file mode 100644 index 0000000000..39759017bc --- /dev/null +++ b/packages/db/tests/conformance/suite.ts @@ -0,0 +1,729 @@ +/** + * Shared live-query conformance suite. + * + * Sourced bottom-up from the union of the five adapters' existing test suites + * (the "spine" + framework-agnostic "gap-closers"), plus a small tail of + * behaviors no adapter tests yet but all should (encoded as expected-fail). + * + * Each scenario has a stable KEY. An adapter marks a key in `driver.knownGaps` + * (populated empirically by running, not from the coverage matrix) to assert it + * as expected-fail. `UNIVERSAL_EXPECTED_FAIL` keys fail on every adapter until + * the underlying core gap is fixed. + * + * Coverage: query/where/select, live insert/update/delete, orderBy, join, + * groupBy/aggregate, nested aggregates, `.includes` subqueries, findOne + * cardinality, disabled + transitions, deferred readiness / eager / ready-with- + * no-data, param recompilation, optimistic mutation, pre-created & config-object + * inputs, error status, and the order-only-move tail (expected-fail). + */ +import { describe, expect, it } from 'vitest' +import type { LiveQueryDriver, LiveQueryHandle, Row } from './contract' + +const SEED: Array = [ + { id: `1`, name: `John Doe`, age: 30, team: `a` }, + { id: `2`, name: `Jane Doe`, age: 25, team: `b` }, + { id: `3`, name: `John Smith`, age: 35, team: `a` }, +] + +interface Issue { + id: string + title: string + userId: string +} + +// Issues reference SEED people: John(1) has 2, Jane(2) has 1, John Smith(3) has 0. +const ISSUES: Array = [ + { id: `i1`, title: `Issue 1`, userId: `1` }, + { id: `i2`, title: `Issue 2`, userId: `2` }, + { id: `i3`, title: `Issue 3`, userId: `1` }, +] + +/** Keys that are expected to fail on ALL adapters (core gaps, not adapter drift). */ +const UNIVERSAL_EXPECTED_FAIL = new Set([]) + +export function runSuite(rawDriver: LiveQueryDriver) { + const { ops } = rawDriver + const gaps = new Set(rawDriver.knownGaps ?? []) + + // Every scenario key registered below, used to validate `knownGaps` / + // `UNIVERSAL_EXPECTED_FAIL` don't reference a stale or misspelled key. + const registeredKeys = new Set() + + // Track every handle mounted during the current scenario so it is always torn + // down, even when an (expected-fail) scenario throws before its own + // `h.unmount()`. Wrapping the driver's `mount*` methods records handles + // automatically, so scenario bodies need no `try/finally` of their own. + let mounted: Array | null = null + const track = (handle: H): H => { + mounted?.push(handle) + return handle + } + const driver: LiveQueryDriver = { + ...rawDriver, + mount: (build) => track(rawDriver.mount(build)), + mountControllable: (build, initial) => + track(rawDriver.mountControllable(build, initial)), + mountCollection: (collection) => + track(rawDriver.mountCollection(collection)), + mountConfig: (build) => track(rawDriver.mountConfig(build)), + mountDisabled: () => track(rawDriver.mountDisabled()), + } + + /** Register a scenario as `it` or `it.fails` based on known gaps. */ + const scenario = ( + key: string, + name: string, + fn: () => Promise | void, + ) => { + registeredKeys.add(key) + const expectFail = gaps.has(key) || UNIVERSAL_EXPECTED_FAIL.has(key) + const label = `[${key}] ${name}${expectFail ? ` (expected-fail)` : ``}` + const run = async () => { + const handles: Array = [] + mounted = handles + try { + await fn() + } finally { + mounted = null + for (const handle of handles) { + try { + handle.unmount() + } catch { + // teardown is best-effort / idempotent + } + } + } + } + if (expectFail) it.fails(label, run) + else it(label, run) + } + + describe(`live-query conformance :: ${driver.name}`, () => { + // ---- spine: query + liveness ---------------------------------------- + + scenario( + `basic-select`, + `from + where + select returns matching rows`, + async () => { + const source = driver.makeSource(SEED) + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .where(({ items }: any) => ops.gt(items.age, 30)) + .select(({ items }: any) => ({ id: items.id, name: items.name })), + ) + await h.flush() + + expect(h.current().data).toHaveLength(1) + expect(h.current().data[0]).toMatchObject({ + id: `3`, + name: `John Smith`, + }) + h.unmount() + }, + ) + + scenario(`live-insert`, `a sync insert appears in the result`, async () => { + const source = driver.makeSource(SEED) + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .select(({ items }: any) => ({ id: items.id })), + ) + await h.flush() + expect(h.current().data).toHaveLength(SEED.length) + + source.insert({ id: `4`, name: `Dave`, age: 40, team: `b` }) + await h.flush() + + expect(h.current().data).toHaveLength(SEED.length + 1) + h.unmount() + }) + + scenario( + `live-delete`, + `a sync delete removes from the result`, + async () => { + const source = driver.makeSource(SEED) + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .select(({ items }: any) => ({ id: items.id })), + ) + await h.flush() + + source.remove(SEED[0]!) + await h.flush() + + expect(h.current().data).toHaveLength(SEED.length - 1) + h.unmount() + }, + ) + + scenario(`orderby`, `orderBy yields rows in sorted order`, async () => { + const source = driver.makeSource(SEED) + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.age) + .select(({ items }: any) => ({ id: items.id })), + ) + await h.flush() + + expect(h.current().data.map((r: any) => r.id)).toEqual([`2`, `1`, `3`]) + h.unmount() + }) + + // ---- gap-closer: cardinality (matrix: Vue tests this 0 times) -------- + + scenario( + `findone-cardinality`, + `findOne returns a single row, not an array`, + async () => { + const source = driver.makeSource(SEED) + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .where(({ items }: any) => ops.eq(items.id, `3`)) + .findOne(), + ) + await h.flush() + + expect(Array.isArray(h.current().data)).toBe(false) + expect(h.current().data).toMatchObject({ id: `3`, name: `John Smith` }) + h.unmount() + }, + ) + + // ---- gap-closer: disabled (matrix: Svelte tests this 0 times) -------- + + scenario( + `disabled-explicit`, + `a disabled query reports isEnabled=false with no data`, + async () => { + const h = driver.mountDisabled() + await h.flush() + + expect(h.current().isEnabled).toBe(false) + expect(h.current().data ?? []).toHaveLength(0) + h.unmount() + }, + ) + + // ---- spine: lifecycle invariant -------------------------------------- + + scenario( + `no-updates-after-unmount`, + `no result mutation after unmount`, + async () => { + const source = driver.makeSource(SEED) + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .select(({ items }: any) => ({ id: items.id })), + ) + await h.flush() + const before = h.current().data.length + + h.unmount() + source.insert({ id: `99`, name: `Zed`, age: 1, team: `b` }) + await h.flush() + + expect(h.current().data.length).toBe(before) + }, + ) + + // ---- spine: relational + aggregate queries --------------------------- + + scenario(`join`, `join across two collections`, async () => { + const people = driver.makeSource(SEED) + const issues = driver.makeSource(ISSUES) + const h = driver.mount((q) => + q + .from({ issues: issues.collection }) + .join({ persons: people.collection }, ({ issues: i, persons }: any) => + ops.eq(i.userId, persons.id), + ) + .select(({ issues: i, persons }: any) => ({ + id: i.id, + title: i.title, + name: persons.name, + })), + ) + await h.flush() + + expect(h.current().data).toHaveLength(ISSUES.length) + expect(h.current().data.find((r: any) => r.id === `i1`)).toMatchObject({ + title: `Issue 1`, + name: `John Doe`, + }) + h.unmount() + }) + + scenario( + `groupby-aggregate`, + `groupBy + count aggregates per group`, + async () => { + const people = driver.makeSource(SEED) + const h = driver.mount((q) => + q + .from({ items: people.collection }) + .groupBy(({ items }: any) => items.team) + .select(({ items }: any) => ({ + team: items.team, + count: ops.count(items.id), + })), + ) + await h.flush() + + const byTeam = new Map( + h.current().data.map((r: any) => [r.team, r.count]), + ) + expect(byTeam.get(`a`)).toBe(2) + expect(byTeam.get(`b`)).toBe(1) + h.unmount() + }, + ) + + // ---- gap-closers: engine features tested only by React today --------- + + scenario( + `nested-aggregates`, + `coalesce(count(...), 0) in a joined subquery`, + async () => { + const people = driver.makeSource(SEED) + const issues = driver.makeSource(ISSUES) + const h = driver.mount((q) => { + const issueCounts = q + .from({ issues: issues.collection }) + .groupBy(({ issues: i }: any) => i.userId) + .select(({ issues: i }: any) => ({ + userId: i.userId, + issueCount: ops.coalesce(ops.count(i.id), 0), + })) + return q + .from({ persons: people.collection }) + .leftJoin({ ic: issueCounts }, ({ persons, ic }: any) => + ops.eq(persons.id, ic.userId), + ) + .select(({ persons, ic }: any) => ({ + name: persons.name, + issueCount: ic.issueCount, + })) + }) + await h.flush() + + const byName = new Map( + h.current().data.map((r: any) => [r.name, r.issueCount]), + ) + expect(byName.get(`John Doe`)).toBe(2) + expect(byName.get(`Jane Doe`)).toBe(1) + h.unmount() + }, + ) + + scenario( + `includes-subquery`, + `select with a nested subquery produces child collections`, + async () => { + const people = driver.makeSource(SEED) + const issues = driver.makeSource(ISSUES) + const h = driver.mount((q) => + q.from({ persons: people.collection }).select(({ persons }: any) => ({ + id: persons.id, + name: persons.name, + issues: q + .from({ issues: issues.collection }) + .where(({ issues: i }: any) => ops.eq(i.userId, persons.id)) + .select(({ issues: i }: any) => ({ id: i.id, title: i.title })), + })), + ) + await h.flush() + + expect(h.current().data).toHaveLength(SEED.length) + const john = h.current().data.find((r: any) => r.id === `1`) + // `john.issues` is a child collection; read its contents through the + // collection API. John (id 1) has issues i1 and i3. + expect(john.issues).toBeDefined() + const johnIssueIds = Array.from(john.issues.values()) + .map((i: any) => i.id) + .sort() + expect(johnIssueIds).toEqual([`i1`, `i3`]) + h.unmount() + }, + ) + + // ---- free ports (no new capability needed) --------------------------- + + scenario(`live-update`, `a sync update is reflected in place`, async () => { + const source = driver.makeSource(SEED) + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .select(({ items }: any) => ({ id: items.id, name: items.name })), + ) + await h.flush() + + source.update({ id: `1`, name: `Johnny Doe`, age: 30, team: `a` }) + await h.flush() + + expect(h.current().data.find((r: any) => r.id === `1`).name).toBe( + `Johnny Doe`, + ) + h.unmount() + }) + + scenario( + `findone-reactive`, + `findOne updates in place and becomes undefined on delete`, + async () => { + const source = driver.makeSource(SEED) + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .where(({ items }: any) => ops.eq(items.id, `3`)) + .findOne(), + ) + await h.flush() + expect(h.current().data).toMatchObject({ name: `John Smith` }) + + source.update({ id: `3`, name: `Johnny Smith`, age: 35, team: `a` }) + await h.flush() + expect(h.current().data).toMatchObject({ name: `Johnny Smith` }) + + source.remove({ id: `3`, name: `Johnny Smith`, age: 35, team: `a` }) + await h.flush() + expect(h.current().data ?? undefined).toBeUndefined() + h.unmount() + }, + ) + + // ---- Tier 2: deferred readiness -------------------------------------- + + scenario( + `isready-transition`, + `isReady flips from false to true when the source readies`, + async () => { + const source = driver.makeDeferredSource() + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .select(({ items }: any) => ({ id: items.id })), + ) + await h.flush() + expect(h.current().isReady).toBe(false) + + source.markReady() + await h.flush() + expect(h.current().isReady).toBe(true) + h.unmount() + }, + ) + + scenario( + `eager-visible-while-loading`, + `rows emitted before ready are visible while still loading`, + async () => { + const source = driver.makeDeferredSource() + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .select(({ items }: any) => ({ id: items.id })), + ) + await h.flush() + + source.emit(SEED) + await h.flush() + + expect(h.current().isReady).toBe(false) + expect(h.current().data).toHaveLength(SEED.length) + + source.markReady() + await h.flush() + expect(h.current().isReady).toBe(true) + h.unmount() + }, + ) + + scenario( + `isready-no-data`, + `isReady becomes true even when the source readies with no rows`, + async () => { + const source = driver.makeDeferredSource() + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .select(({ items }: any) => ({ id: items.id })), + ) + await h.flush() + + source.markReady() + await h.flush() + + expect(h.current().isReady).toBe(true) + expect(h.current().data ?? []).toHaveLength(0) + h.unmount() + }, + ) + + // ---- Tier 2: controllable input -------------------------------------- + + scenario( + `param-recompile`, + `changing a query parameter recompiles the result`, + async () => { + const source = driver.makeSource(SEED) + const h = driver.mountControllable( + (q, minAge) => + q + .from({ items: source.collection }) + .where(({ items }: any) => ops.gt(items.age, minAge)) + .select(({ items }: any) => ({ id: items.id })), + 30, + ) + await h.flush() + expect(h.current().data).toHaveLength(1) // age > 30 → John Smith + + await h.setParam(20) + expect(h.current().data).toHaveLength(3) // all + + await h.setParam(50) + expect(h.current().data).toHaveLength(0) // none + h.unmount() + }, + ) + + scenario( + `recompile-drops-stale-keys`, + `recompiling to a narrower result drops keys from the previous collection`, + async () => { + const source = driver.makeSource(SEED) + const h = driver.mountControllable( + (q, minAge) => + q + .from({ items: source.collection }) + .where(({ items }: any) => ops.gt(items.age, minAge)) + .select(({ items }: any) => ({ id: items.id })), + 10, + ) + await h.flush() + expect(h.current().data).toHaveLength(3) // all ages > 10 + // The keyed `state` map must mirror `data` exactly. + expect(h.current().state?.size).toBe(3) + + // Narrowing the filter recompiles into a *new* underlying collection + // holding fewer keys. `includeInitialState` only inserts the new rows; + // if the adapter reuses a persistent keyed map without clearing it, the + // dropped keys leak into `state` even though `data` looks correct. + await h.setParam(32) // only John Smith (age 35) survives + expect(h.current().data).toHaveLength(1) + expect(h.current().state?.size).toBe(1) + h.unmount() + }, + ) + + scenario( + `disabled-transition`, + `disabled -> enabled -> disabled toggles correctly`, + async () => { + const source = driver.makeSource(SEED) + const h = driver.mountControllable( + (q, enabled) => + enabled + ? q + .from({ items: source.collection }) + .select(({ items }: any) => ({ id: items.id })) + : null, + false, + ) + await h.flush() + expect(h.current().isEnabled).toBe(false) + + await h.setParam(true) + expect(h.current().isEnabled).toBe(true) + expect(h.current().data).toHaveLength(SEED.length) + + await h.setParam(false) + expect(h.current().isEnabled).toBe(false) + h.unmount() + }, + ) + + // ---- Tier 2: optimistic mutation ------------------------------------- + + scenario( + `optimistic-insert`, + `optimistic insert is visible immediately, then reconciles to the server key`, + async () => { + const source = driver.makeSource(SEED) + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .select(({ items }: any) => ({ id: items.id })), + ) + await h.flush() + + const temp: Row = { id: `temp`, name: `New`, age: 20, team: `c` } + const perm: Row = { id: `p9`, name: `New`, age: 20, team: `c` } + // The "server" confirms only when we release it, so the optimistic + // window is deterministic rather than racing the settle. + let confirmServer!: () => void + const serverConfirmed = new Promise((resolve) => { + confirmServer = resolve + }) + const add = ops.createOptimisticAction({ + onMutate: () => source.collection.insert(temp), + mutationFn: async () => { + await serverConfirmed + source.remove(temp) + source.insert(perm) + }, + }) + + let tx!: { isPersisted: { promise: Promise } } + await h.apply(() => { + tx = add() + }) + // Optimistic row is visible before the server confirms. + expect(h.current().data.find((r: any) => r.id === `temp`)).toBeDefined() + + confirmServer() + await tx.isPersisted.promise + await h.flush() + // Reconciled: temp replaced by the permanent key. + expect( + h.current().data.find((r: any) => r.id === `temp`), + ).toBeUndefined() + expect(h.current().data.find((r: any) => r.id === `p9`)).toBeDefined() + h.unmount() + }, + ) + + // ---- Tier 3: input variants + error status --------------------------- + + scenario( + `precreated-collection-ready`, + `accepts a pre-created (syncing) live-query collection`, + async () => { + const source = driver.makeSource(SEED) + const pre = driver.makePrecreated( + (q) => + q + .from({ items: source.collection }) + .select(({ items }: any) => ({ id: items.id })), + { startSync: true }, + ) + const h = driver.mountCollection(pre.collection) + await h.flush() + + expect(h.current().isReady).toBe(true) + expect(h.current().data).toHaveLength(SEED.length) + h.unmount() + }, + ) + + scenario( + `precreated-not-syncing-isready-false`, + `a pre-created collection over a not-ready source reports isReady=false`, + async () => { + // Both the live query (startSync: false) and its source are not ready. + // Even if the adapter eagerly starts the collection on mount, it cannot + // become ready because the source never readies — so isReady stays false. + const source = driver.makeDeferredSource() + const pre = driver.makePrecreated( + (q) => + q + .from({ items: source.collection }) + .select(({ items }: any) => ({ id: items.id })), + { startSync: false }, + ) + const h = driver.mountCollection(pre.collection) + await h.flush() + expect(h.current().isReady).toBe(false) + h.unmount() + }, + ) + + scenario( + `config-object-input`, + `accepts the { query } config-object input form`, + async () => { + const source = driver.makeSource(SEED) + const h = driver.mountConfig((q) => + q + .from({ items: source.collection }) + .where(({ items }: any) => ops.eq(items.id, `3`)) + .select(({ items }: any) => ({ id: items.id, name: items.name })), + ) + await h.flush() + + expect(h.current().data).toHaveLength(1) + expect(h.current().data[0]).toMatchObject({ id: `3` }) + h.unmount() + }, + ) + + scenario( + `error-status`, + `a failing source surfaces an error (flag or boundary)`, + async () => { + const source = driver.makeErrorSource() + const h = driver.mountCollection(source.collection) + await h.flush() + + if (driver.errorSurface === `throw`) { + // Boundary model: reading the errored result throws (for an error + // boundary to catch), rather than exposing a readable flag. + expect(() => h.current()).toThrow() + } else { + expect(h.current().status).toBe(`error`) + expect(h.current().isError).toBe(true) + } + h.unmount() + }, + ) + + // ---- tail: universal expected-fail --------------------------- + + scenario( + `order-only-move`, + `an order-only move republishes the ordered result`, + async () => { + const source = driver.makeSource(SEED) + // Project only id+name; sort by age. Changing age reorders the result + // WITHOUT changing any projected row value. + const h = driver.mount((q) => + q + .from({ items: source.collection }) + .orderBy(({ items }: any) => items.age) + .select(({ items }: any) => ({ id: items.id, name: items.name })), + ) + await h.flush() + const first = h.current().data.map((r: any) => r.id) // ['2','1','3'] + + source.update({ id: `2`, name: `Jane Doe`, age: 99, team: `b` }) + await h.flush() + + expect(h.current().data.map((r: any) => r.id)).not.toEqual(first) + h.unmount() + }, + ) + + // ---- meta: guard against stale/misspelled expected-fail keys --------- + + it(`every knownGap / universal expected-fail references a real scenario`, () => { + for (const key of rawDriver.knownGaps ?? []) { + expect( + registeredKeys.has(key), + `${driver.name} knownGaps has "${key}", which is not a scenario key`, + ).toBe(true) + } + for (const key of UNIVERSAL_EXPECTED_FAIL) { + expect( + registeredKeys.has(key), + `UNIVERSAL_EXPECTED_FAIL has "${key}", which is not a scenario key`, + ).toBe(true) + } + }) + }) +} diff --git a/packages/db/tests/cursor.property.test.ts b/packages/db/tests/cursor.property.test.ts index 04c2fa5368..45c45f8761 100644 --- a/packages/db/tests/cursor.property.test.ts +++ b/packages/db/tests/cursor.property.test.ts @@ -1,370 +1,190 @@ -import { describe, expect, it } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' -import { buildCursor } from '../src/utils/cursor' -import { Func, PropRef, Value } from '../src/query/ir' -import type { OrderBy, OrderByClause } from '../src/query/ir' -import type { CompareOptions } from '../src/query/builder/types' - -/** - * Property-based tests for cursor building - * - * Key properties: - * 1. Empty inputs return undefined - * 2. Single column produces simple gt/lt based on direction - * 3. Direction affects operator choice (asc = gt, desc = lt) - * 4. Determinism - same inputs always produce same output - * 5. Result structure is always valid - */ - -// Arbitraries for generating test data -const arbitraryDirection = fc.constantFrom(`asc`, `desc`) - -const arbitraryNulls = fc.constantFrom(`first`, `last`) - -const arbitraryStringSort = fc.constantFrom(`locale`, `lexical`) - -const arbitraryCompareOptions = fc.record({ - direction: arbitraryDirection, - nulls: arbitraryNulls, - stringSort: arbitraryStringSort, -}) as fc.Arbitrary - -const arbitraryPropRef = fc - .array(fc.string({ minLength: 1, maxLength: 10 }), { - minLength: 1, - maxLength: 3, - }) - .map((path) => new PropRef(path)) - -const arbitraryOrderByClause = fc - .tuple(arbitraryPropRef, arbitraryCompareOptions) - .map( - ([expr, compareOptions]): OrderByClause => ({ - expression: expr, - compareOptions, - }), - ) - -const arbitraryOrderBy = ( - minLength: number, - maxLength: number, -): fc.Arbitrary => - fc.array(arbitraryOrderByClause, { minLength, maxLength }) +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { PropRef } from '../src/query/ir.js' +import { buildCursor } from '../src/utils/cursor.js' +import { evaluateReferenceExpression } from './reference-expression.js' +import type { OrderBy } from '../src/query/ir.js' + +type Term = { + direction: `asc` | `desc` + nulls: `first` | `last` +} -const arbitraryValue = fc.oneof( - fc.string(), - fc.integer(), - fc.double({ noNaN: true }), - fc.boolean(), +const termArbitrary = fc.record({ + direction: fc.constantFrom(`asc`, `desc`), + nulls: fc.constantFrom(`first`, `last`), +}) +const valueArbitrary = fc.oneof( + fc.integer({ min: -2, max: 2 }), fc.constant(null), + fc.constant(undefined), ) -const arbitraryValues = ( - minLength: number, - maxLength: number, -): fc.Arbitrary> => - fc.array(arbitraryValue, { minLength, maxLength }) +function compareValue(left: unknown, right: unknown, term: Term): number { + if (left == null && right == null) return 0 + if (left == null) return term.nulls === `first` ? -1 : 1 + if (right == null) return term.nulls === `first` ? 1 : -1 + const compared = left === right ? 0 : left < right ? -1 : 1 + return term.direction === `asc` ? compared : -compared +} -// Helper to check if result is a Func -function isFunc(expr: unknown): expr is Func { - return expr instanceof Func +function compareTuple( + left: ReadonlyArray, + right: ReadonlyArray, + terms: ReadonlyArray, +): number { + for (let index = 0; index < terms.length; index++) { + const compared = compareValue(left[index], right[index], terms[index]!) + if (compared !== 0) return compared + } + return 0 } -// Helper to get operator name from Func -function getFuncName(expr: Func): string { - return expr.name +function orderBy(terms: ReadonlyArray): OrderBy { + return terms.map((compareOptions, index) => ({ + expression: new PropRef([`column${index}`]), + compareOptions, + })) } -// Helper to recursively count operators in an expression -function countOperators(expr: unknown, name: string): number { - if (!isFunc(expr)) return 0 - const selfCount = expr.name === name ? 1 : 0 - return ( - selfCount + - expr.args.reduce((sum, arg) => sum + countOperators(arg, name), 0) +function row(values: ReadonlyArray): Record { + return Object.fromEntries( + values.map((value, index) => [`column${index}`, value]), ) } -describe(`buildCursor property-based tests`, () => { - describe(`empty input handling`, () => { - fcTest.prop([arbitraryOrderBy(0, 5)])( - `returns undefined for empty values array`, - (orderBy) => { - const result = buildCursor(orderBy, []) - expect(result).toBeUndefined() - }, - ) - - fcTest.prop([arbitraryValues(0, 5)])( - `returns undefined for empty orderBy array`, - (values) => { - const result = buildCursor([], values) - expect(result).toBeUndefined() - }, - ) - - it(`returns undefined for both empty`, () => { - expect(buildCursor([], [])).toBeUndefined() - }) - }) - - describe(`single column cursor`, () => { - fcTest.prop([arbitraryOrderByClause, arbitraryValue])( - `produces a simple comparison for single column`, - (clause, value) => { - const result = buildCursor([clause], [value]) - - expect(result).toBeDefined() - expect(isFunc(result!)).toBe(true) - - // Should be either 'gt' or 'lt' based on direction - const func = result as Func - expect([`gt`, `lt`]).toContain(getFuncName(func)) - }, - ) - - fcTest.prop([ - arbitraryPropRef, - arbitraryNulls, - arbitraryStringSort, - arbitraryValue, - ])( - `ascending direction produces gt operator`, - (expr, nulls, stringSort, value) => { - const clause: OrderByClause = { - expression: expr, - compareOptions: { - direction: `asc`, - nulls: nulls as `first` | `last`, - stringSort: stringSort as `locale` | `lexical`, - }, - } - const result = buildCursor([clause], [value]) - - expect(result).toBeDefined() - expect(getFuncName(result as Func)).toBe(`gt`) - }, - ) - - fcTest.prop([ - arbitraryPropRef, - arbitraryNulls, - arbitraryStringSort, - arbitraryValue, - ])( - `descending direction produces lt operator`, - (expr, nulls, stringSort, value) => { - const clause: OrderByClause = { - expression: expr, - compareOptions: { - direction: `desc`, - nulls: nulls as `first` | `last`, - stringSort: stringSort as `locale` | `lexical`, - }, - } - const result = buildCursor([clause], [value]) - - expect(result).toBeDefined() - expect(getFuncName(result as Func)).toBe(`lt`) - }, - ) - }) - - describe(`multi-column cursor structure`, () => { - fcTest.prop([arbitraryOrderBy(2, 4), arbitraryValues(2, 4)])( - `multi-column produces or at top level when matching lengths`, - (orderBy, values) => { - // Ensure we have matching lengths for a valid multi-column cursor - const minLen = Math.min(orderBy.length, values.length) - if (minLen < 2) return // Skip if not enough for multi-column - - const trimmedOrderBy = orderBy.slice(0, minLen) - const trimmedValues = values.slice(0, minLen) - - const result = buildCursor(trimmedOrderBy, trimmedValues) - - expect(result).toBeDefined() - expect(isFunc(result!)).toBe(true) - - // For 2+ columns, top level should be 'or' - const func = result as Func - expect(getFuncName(func)).toBe(`or`) - }, +function expectCursorDenotation( + terms: ReadonlyArray, + boundary: ReadonlyArray, + candidate: ReadonlyArray, +): void { + if (terms.length !== 1 || boundary.length !== 1) { + expect(() => buildCursor(orderBy(terms), [...boundary])).toThrow( + `Only single-column cursors are supported`, ) + return + } + const length = Math.min(terms.length, boundary.length) + const usedTerms = terms.slice(0, length) + const usedBoundary = boundary.slice(0, length) + const cursor = buildCursor(orderBy(terms), [...boundary]) + expect(cursor).toBeDefined() + expect(Boolean(evaluateReferenceExpression(cursor!, row(candidate)))).toBe( + compareTuple(candidate, usedBoundary, usedTerms) > 0, + ) +} - fcTest.prop([ - fc.tuple(arbitraryOrderByClause, arbitraryOrderByClause), - fc.tuple(arbitraryValue, arbitraryValue), - ])( - `two columns produces correct structure`, - ([clause1, clause2], [val1, val2]) => { - const result = buildCursor([clause1, clause2], [val1, val2]) - - expect(result).toBeDefined() - const func = result as Func - - // Top level should be 'or' - expect(getFuncName(func)).toBe(`or`) - - // Should have structure: or(comparison1, and(eq, comparison2)) - expect(func.args).toHaveLength(2) - - // First arg should be direct gt/lt - expect(isFunc(func.args[0])).toBe(true) - expect([`gt`, `lt`]).toContain(getFuncName(func.args[0] as Func)) - - // Second arg should be 'and' combining eq and comparison - expect(isFunc(func.args[1])).toBe(true) - expect(getFuncName(func.args[1] as Func)).toBe(`and`) - }, - ) - }) - - describe(`determinism`, () => { - fcTest.prop([arbitraryOrderBy(1, 3), arbitraryValues(1, 3)])( - `buildCursor is deterministic`, - (orderBy, values) => { - const result1 = buildCursor(orderBy, values) - const result2 = buildCursor(orderBy, values) - - // Both should be defined or both undefined - expect(result1 === undefined).toBe(result2 === undefined) - - if (result1 !== undefined && result2 !== undefined) { - // Compare structure by JSON representation - expect(JSON.stringify(result1)).toBe(JSON.stringify(result2)) - } +// Keep the nullable mixed-direction ordering law at the retained production +// snapshot boundary even though direct composite cursor construction is removed. +async function expectLocalTupleOrder( + terms: ReadonlyArray, + boundary: ReadonlyArray, + candidate: ReadonlyArray, +): Promise { + const collection = createCollection<{ id: string; [key: string]: unknown }>({ + getKey: (value) => value.id, + autoIndex: `off`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { ...row(candidate), id: `candidate` } }) + write({ type: `insert`, value: { ...row(boundary), id: `boundary` } }) + commit() + markReady() }, - ) + }, }) + try { + await collection.preload() + const expected = + compareTuple(candidate, boundary, terms) >= 0 + ? [`boundary`, `candidate`] + : [`candidate`, `boundary`] + for (const limit of [1, 2]) { + expect( + collection + .currentStateAsChanges({ + orderBy: [ + ...orderBy(terms), + { + expression: new PropRef([`id`]), + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `lexical`, + }, + }, + ], + limit, + }) + ?.map(({ key }) => key), + ).toEqual(expected.slice(0, limit)) + } + } finally { + await collection.cleanup() + } +} - describe(`value preservation`, () => { - fcTest.prop([arbitraryOrderByClause, arbitraryValue])( - `cursor contains the provided value`, - (clause, value) => { - const result = buildCursor([clause], [value]) - - expect(result).toBeDefined() - const func = result as Func - - // Second argument should be a Value containing our value - expect(func.args[1]).toBeInstanceOf(Value) - expect((func.args[1] as Value).value).toBe(value) - }, - ) - - fcTest.prop([arbitraryOrderByClause, arbitraryValue])( - `cursor references the correct property`, - (clause, value) => { - const result = buildCursor([clause], [value]) +const exactCursorArbitrary = fc + .integer({ min: 1, max: 4 }) + .chain((length) => + fc.tuple( + fc.array(termArbitrary, { minLength: length, maxLength: length }), + fc.array(valueArbitrary, { minLength: length, maxLength: length }), + fc.array(valueArbitrary, { minLength: length, maxLength: length }), + ), + ) - expect(result).toBeDefined() - const func = result as Func +const partialCursorArbitrary = fc + .tuple( + fc.array(termArbitrary, { minLength: 1, maxLength: 4 }), + fc.array(valueArbitrary, { minLength: 1, maxLength: 4 }), + fc.array(valueArbitrary, { minLength: 4, maxLength: 4 }), + ) + .filter(([terms, boundary]) => terms.length !== boundary.length) - // First argument should be the same PropRef - expect(func.args[0]).toBeInstanceOf(PropRef) - expect((func.args[0] as PropRef).path).toEqual( - (clause.expression as PropRef).path, - ) - }, +describe(`buildCursor properties`, () => { + it(`returns no cursor without boundary values and rejects a boundary without an order`, () => { + expect(() => buildCursor([], [1])).toThrow( + `Only single-column cursors are supported`, ) + expect(buildCursor([], [])).toBeUndefined() + expect( + buildCursor(orderBy([{ direction: `asc`, nulls: `first` }]), []), + ).toBeUndefined() }) - describe(`length mismatch handling`, () => { - fcTest.prop([arbitraryOrderBy(3, 5), arbitraryValues(1, 2)])( - `handles more orderBy columns than values gracefully`, - (orderBy, values) => { - // Should use the minimum of the two lengths - const result = buildCursor(orderBy, values) - - if (values.length === 0) { - expect(result).toBeUndefined() - } else { - expect(result).toBeDefined() - expect(isFunc(result!)).toBe(true) - } - }, - ) + fcTest.prop([exactCursorArbitrary], { numRuns: 300 })( + `preserves nullable mixed-direction ordering while restricting cursor width`, + async ([terms, boundary, candidate]) => { + expectCursorDenotation(terms, boundary, candidate) + await expectLocalTupleOrder(terms, boundary, candidate) + }, + ) - fcTest.prop([arbitraryOrderBy(1, 2), arbitraryValues(3, 5)])( - `handles more values than orderBy columns gracefully`, - (orderBy, values) => { - // Should use the minimum of the two lengths - const result = buildCursor(orderBy, values) + fcTest.prop([partialCursorArbitrary], { numRuns: 200 })( + `rejects mismatched cursor widths without restricting local tuple ordering`, + async ([terms, boundary, candidate]) => { + expectCursorDenotation(terms, boundary, candidate) + await expectLocalTupleOrder(terms, boundary, candidate) + }, + ) - if (orderBy.length === 0) { - expect(result).toBeUndefined() - } else { - expect(result).toBeDefined() - expect(isFunc(result!)).toBe(true) + fcTest.prop([exactCursorArbitrary], { numRuns: 100 })( + `repeats the same cursor or unsupported-width error`, + ([terms, boundary]) => { + if (terms.length !== 1) { + for (let attempt = 0; attempt < 2; attempt++) { + expect(() => buildCursor(orderBy(terms), [...boundary])).toThrow( + `Only single-column cursors are supported`, + ) } - }, - ) - }) - - describe(`operator consistency`, () => { - fcTest.prop([ - fc.array( - fc.tuple(arbitraryPropRef, arbitraryNulls, arbitraryStringSort), - { minLength: 2, maxLength: 4 }, - ), - arbitraryValues(2, 4), - ])(`all ascending columns use gt operators`, (clauseParts, values) => { - const orderBy: OrderBy = clauseParts.map(([expr, nulls, stringSort]) => ({ - expression: expr, - compareOptions: { - direction: `asc` as const, - nulls: nulls as `first` | `last`, - stringSort: stringSort as `locale` | `lexical`, - }, - })) - - const minLen = Math.min(orderBy.length, values.length) - const result = buildCursor( - orderBy.slice(0, minLen), - values.slice(0, minLen), - ) - - if (result) { - // Count gt operators - should equal number of columns - const gtCount = countOperators(result, `gt`) - expect(gtCount).toBe(minLen) - // Should have no lt operators - const ltCount = countOperators(result, `lt`) - expect(ltCount).toBe(0) + return } - }) - - fcTest.prop([ - fc.array( - fc.tuple(arbitraryPropRef, arbitraryNulls, arbitraryStringSort), - { minLength: 2, maxLength: 4 }, - ), - arbitraryValues(2, 4), - ])(`all descending columns use lt operators`, (clauseParts, values) => { - const orderBy: OrderBy = clauseParts.map(([expr, nulls, stringSort]) => ({ - expression: expr, - compareOptions: { - direction: `desc` as const, - nulls: nulls as `first` | `last`, - stringSort: stringSort as `locale` | `lexical`, - }, - })) - - const minLen = Math.min(orderBy.length, values.length) - const result = buildCursor( - orderBy.slice(0, minLen), - values.slice(0, minLen), + expect(buildCursor(orderBy(terms), [...boundary])).toEqual( + buildCursor(orderBy(terms), [...boundary]), ) - - if (result) { - // Count lt operators - should equal number of columns - const ltCount = countOperators(result, `lt`) - expect(ltCount).toBe(minLen) - // Should have no gt operators - const gtCount = countOperators(result, `gt`) - expect(gtCount).toBe(0) - } - }) - }) + }, + ) }) diff --git a/packages/db/tests/cursor.test.ts b/packages/db/tests/cursor.test.ts index 5d8f4a2f47..1b269bbf54 100644 --- a/packages/db/tests/cursor.test.ts +++ b/packages/db/tests/cursor.test.ts @@ -1,226 +1,99 @@ import { describe, expect, it } from 'vitest' -import { buildCursor } from '../src/utils/cursor.js' -import { Func, PropRef, Value } from '../src/query/ir.js' -import type { OrderBy, OrderByClause } from '../src/query/ir.js' -import type { CompareOptions } from '../src/query/builder/types.js' - -// Helper to create an OrderByClause for testing -function createOrderByClause( - path: string, - direction: `asc` | `desc`, -): OrderByClause { - const compareOptions: CompareOptions = { - direction, - nulls: direction === `asc` ? `first` : `last`, - } - return { - expression: new PropRef([`t`, path]), - compareOptions, - } +import { PropRef } from '../src/query/ir.js' +import { buildCursor, canExpressCursorOrder } from '../src/utils/cursor.js' +import { evaluateReferenceExpression } from './reference-expression.js' +import type { OrderBy } from '../src/query/ir.js' + +function orderBy( + ...terms: ReadonlyArray +): OrderBy { + return terms.map(([path, direction, nulls]) => ({ + expression: new PropRef([path]), + compareOptions: { direction, nulls }, + })) } -// Helper to check if a Func has the expected structure -function isFuncWithName(expr: unknown, name: string): expr is Func { - return expr instanceof Func && expr.name === name +function matches( + order: OrderBy, + boundary: Array, + row: object, +): boolean { + const cursor = buildCursor(order, boundary) + if (!cursor) throw new Error(`expected a cursor`) + return Boolean(evaluateReferenceExpression(cursor, row)) } describe(`buildCursor`, () => { - describe(`edge cases`, () => { - it(`returns undefined for empty values array`, () => { - const orderBy: OrderBy = [createOrderByClause(`col1`, `asc`)] - expect(buildCursor(orderBy, [])).toBeUndefined() - }) - - it(`returns undefined for empty orderBy array`, () => { - expect(buildCursor([], [1, 2, 3])).toBeUndefined() - }) - - it(`returns undefined for both empty`, () => { - expect(buildCursor([], [])).toBeUndefined() - }) + it(`uses direction for one non-null term`, () => { + expect(matches(orderBy([`rank`, `asc`, `first`]), [10], { rank: 11 })).toBe( + true, + ) + expect(matches(orderBy([`rank`, `asc`, `first`]), [10], { rank: 9 })).toBe( + false, + ) + expect(matches(orderBy([`rank`, `desc`, `first`]), [10], { rank: 9 })).toBe( + true, + ) + expect( + matches(orderBy([`rank`, `desc`, `first`]), [10], { rank: 11 }), + ).toBe(false) }) - describe(`single column`, () => { - it(`produces gt() for ASC direction`, () => { - const orderBy: OrderBy = [createOrderByClause(`col1`, `asc`)] - const result = buildCursor(orderBy, [10]) - - expect(result).toBeInstanceOf(Func) - expect(isFuncWithName(result, `gt`)).toBe(true) - - const func = result as Func - expect(func.args).toHaveLength(2) - expect(func.args[0]).toBeInstanceOf(PropRef) - expect((func.args[0] as PropRef).path).toEqual([`t`, `col1`]) - expect(func.args[1]).toBeInstanceOf(Value) - expect((func.args[1] as Value).value).toBe(10) - }) - - it(`produces lt() for DESC direction`, () => { - const orderBy: OrderBy = [createOrderByClause(`col1`, `desc`)] - const result = buildCursor(orderBy, [10]) - - expect(result).toBeInstanceOf(Func) - expect(isFuncWithName(result, `lt`)).toBe(true) - - const func = result as Func - expect(func.args).toHaveLength(2) - expect(func.args[0]).toBeInstanceOf(PropRef) - expect((func.args[0] as PropRef).path).toEqual([`t`, `col1`]) - expect(func.args[1]).toBeInstanceOf(Value) - expect((func.args[1] as Value).value).toBe(10) - }) - - it(`handles string cursor values`, () => { - const orderBy: OrderBy = [createOrderByClause(`name`, `asc`)] - const result = buildCursor(orderBy, [`alice`]) + it(`places nullish values according to the term`, () => { + const nullsFirst = orderBy([`rank`, `asc`, `first`]) + expect(matches(nullsFirst, [null], { rank: 0 })).toBe(true) + expect(matches(nullsFirst, [null], { rank: undefined })).toBe(false) - expect(result).toBeInstanceOf(Func) - const func = result as Func - expect(func.args[1]).toBeInstanceOf(Value) - expect((func.args[1] as Value).value).toBe(`alice`) - }) - - it(`handles null cursor values`, () => { - const orderBy: OrderBy = [createOrderByClause(`col1`, `asc`)] - const result = buildCursor(orderBy, [null]) - - expect(result).toBeInstanceOf(Func) - const func = result as Func - expect((func.args[1] as Value).value).toBeNull() - }) + const nullsLast = orderBy([`rank`, `asc`, `last`]) + expect(matches(nullsLast, [0], { rank: null })).toBe(true) + expect(matches(nullsLast, [null], { rank: 0 })).toBe(false) }) - describe(`multi-column composite cursor`, () => { - it(`produces or(gt(col1), and(eq(col1), gt(col2))) for two ASC columns`, () => { - const orderBy: OrderBy = [ - createOrderByClause(`col1`, `asc`), - createOrderByClause(`col2`, `asc`), - ] - const result = buildCursor(orderBy, [10, 20]) - - // Should be: or(gt(col1, 10), and(eq(col1, 10), gt(col2, 20))) - expect(result).toBeInstanceOf(Func) - expect(isFuncWithName(result, `or`)).toBe(true) - - const orFunc = result as Func - expect(orFunc.args).toHaveLength(2) - - // First arg: gt(col1, 10) - const gtCol1 = orFunc.args[0] - expect(isFuncWithName(gtCol1, `gt`)).toBe(true) - expect((gtCol1 as Func).args[0]).toBeInstanceOf(PropRef) - expect(((gtCol1 as Func).args[0] as PropRef).path).toEqual([`t`, `col1`]) - expect(((gtCol1 as Func).args[1] as Value).value).toBe(10) - - // Second arg: and(eq(col1, 10), gt(col2, 20)) - const andClause = orFunc.args[1] - expect(isFuncWithName(andClause, `and`)).toBe(true) - const andFunc = andClause as Func - expect(andFunc.args).toHaveLength(2) - - // eq(col1, 10) - expect(isFuncWithName(andFunc.args[0], `eq`)).toBe(true) - const eqCol1 = andFunc.args[0] as Func - expect((eqCol1.args[0] as PropRef).path).toEqual([`t`, `col1`]) - expect((eqCol1.args[1] as Value).value).toBe(10) - - // gt(col2, 20) - expect(isFuncWithName(andFunc.args[1], `gt`)).toBe(true) - const gtCol2 = andFunc.args[1] as Func - expect((gtCol2.args[0] as PropRef).path).toEqual([`t`, `col2`]) - expect((gtCol2.args[1] as Value).value).toBe(20) - }) + it(`rejects composite cursors with mixed-direction terms`, () => { + const order = orderBy([`group`, `asc`, `first`], [`rank`, `desc`, `last`]) - it(`handles mixed ASC/DESC directions`, () => { - const orderBy: OrderBy = [ - createOrderByClause(`col1`, `asc`), - createOrderByClause(`col2`, `desc`), - ] - const result = buildCursor(orderBy, [10, 20]) - - // Should be: or(gt(col1, 10), and(eq(col1, 10), lt(col2, 20))) - expect(isFuncWithName(result, `or`)).toBe(true) - - const orFunc = result as Func - const andClause = orFunc.args[1] as Func - - // Second column should use lt() for DESC - expect(isFuncWithName(andClause.args[1], `lt`)).toBe(true) - }) - - it(`handles three columns correctly`, () => { - const orderBy: OrderBy = [ - createOrderByClause(`col1`, `asc`), - createOrderByClause(`col2`, `asc`), - createOrderByClause(`col3`, `desc`), - ] - const result = buildCursor(orderBy, [1, 2, 3]) - - // Should be: or( - // gt(col1, 1), - // and(eq(col1, 1), gt(col2, 2)), - // and(eq(col1, 1), eq(col2, 2), lt(col3, 3)) - // ) - expect(isFuncWithName(result, `or`)).toBe(true) - - const outerOr = result as Func - // The structure is: or(or(gt, and), and) due to reduce - expect(outerOr.args).toHaveLength(2) - - // First arg is or(gt(col1, 1), and(eq(col1, 1), gt(col2, 2))) - const innerOr = outerOr.args[0] - expect(isFuncWithName(innerOr, `or`)).toBe(true) - - // Second arg is and(and(eq(col1, 1), eq(col2, 2)), lt(col3, 3)) - const thirdClause = outerOr.args[1] - expect(isFuncWithName(thirdClause, `and`)).toBe(true) - - // The innermost and should have eq conditions and lt for col3 - const innerAnd = thirdClause as Func - // Due to reduce, the structure is nested: and(and(eq, eq), lt) - expect(isFuncWithName(innerAnd.args[1], `lt`)).toBe(true) - const ltCol3 = innerAnd.args[1] as Func - expect((ltCol3.args[0] as PropRef).path).toEqual([`t`, `col3`]) - expect((ltCol3.args[1] as Value).value).toBe(3) - }) + expect(() => buildCursor(order, [1, 10])).toThrow( + `Only single-column cursors are supported`, + ) + expect(canExpressCursorOrder(order, [1, 10])).toBe(false) }) - describe(`partial values`, () => { - it(`handles fewer values than orderBy columns`, () => { - const orderBy: OrderBy = [ - createOrderByClause(`col1`, `asc`), - createOrderByClause(`col2`, `asc`), - createOrderByClause(`col3`, `asc`), - ] - const result = buildCursor(orderBy, [10, 20]) - - // Should only use first two columns - expect(isFuncWithName(result, `or`)).toBe(true) - - const orFunc = result as Func - expect(orFunc.args).toHaveLength(2) - - // First clause: gt(col1, 10) - expect(isFuncWithName(orFunc.args[0], `gt`)).toBe(true) - - // Second clause: and(eq(col1, 10), gt(col2, 20)) - expect(isFuncWithName(orFunc.args[1], `and`)).toBe(true) - }) - - it(`handles single value for multi-column orderBy`, () => { - const orderBy: OrderBy = [ - createOrderByClause(`col1`, `asc`), - createOrderByClause(`col2`, `asc`), - ] - const result = buildCursor(orderBy, [10]) - - // Should just be gt(col1, 10) since only one value provided - expect(isFuncWithName(result, `gt`)).toBe(true) + it(`rejects partial composite cursors instead of silently dropping terms`, () => { + const order = orderBy([`first`, `asc`, `first`], [`second`, `asc`, `first`]) + expect(() => buildCursor(order, [1])).toThrow( + `Only single-column cursors are supported`, + ) + expect(canExpressCursorOrder(order, [1])).toBe(false) + }) - const gtFunc = result as Func - expect((gtFunc.args[0] as PropRef).path).toEqual([`t`, `col1`]) - expect((gtFunc.args[1] as Value).value).toBe(10) - }) + it(`rejects cursor pushdown when predicates cannot express the order`, () => { + const localeOrder: OrderBy = [ + { + expression: new PropRef([`label`]), + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `locale`, + localeOptions: { numeric: true }, + }, + }, + ] + + expect(canExpressCursorOrder(localeOrder, [`item2`])).toBe(false) + expect( + canExpressCursorOrder( + [ + { + ...localeOrder[0]!, + compareOptions: { + ...localeOrder[0]!.compareOptions, + stringSort: `lexical`, + }, + }, + ], + [`item2`], + ), + ).toBe(true) + expect(canExpressCursorOrder(localeOrder, [{ rank: 1 }])).toBe(false) }) }) diff --git a/packages/db/tests/d2-source-reconciliation-oracle.property.test.ts b/packages/db/tests/d2-source-reconciliation-oracle.property.test.ts new file mode 100644 index 0000000000..3d1b29cc8d --- /dev/null +++ b/packages/db/tests/d2-source-reconciliation-oracle.property.test.ts @@ -0,0 +1,830 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { expect, it } from 'vitest' +import { + createCollection, + createLiveQueryCollection, + eq, +} from '../src/index.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { createEffect } from '../src/query/effect.js' +import { reconcileChangesForD2 } from '../src/query/live/utils.js' +import { oraclePropertyOptions, oracleRuns } from './oracle-config.js' +import { flushPromises } from './utils.js' +import type { ChangeMessage, SyncConfig } from '../src/types.js' + +type SourceRow = { + id: number + revision: number + value: number +} + +type SourceSyncActions = Parameters[`sync`]>[0] + +type SourceKey = string | number + +type SourceOperation = + | { + type: `upsert` + key: SourceKey + row: SourceRow + reportedPreviousValue: SourceRow + } + | { + type: `rawUpdate` + key: SourceKey + row: SourceRow + reportedPreviousValue: SourceRow + } + | { type: `replay`; key: SourceKey } + | { type: `delete`; key: SourceKey; reportedValue: SourceRow } + +type ReconciliationStep = + | { type: `batch`; operations: ReadonlyArray } + | { type: `truncate` } + | { type: `teardown` } + | { type: `restart` } + +type ReconciliationModel = { + sourceRows: Map + sentRows: Map + relation: Map + graphActive: boolean +} + +const sourceRowArbitrary = fc.record({ + id: fc.integer({ min: 0, max: 3 }), + revision: fc.integer({ min: 0, max: 4 }), + value: fc.integer({ min: -2, max: 2 }), +}) + +const sourceKeyArbitrary: fc.Arbitrary = fc.oneof( + fc.integer({ min: 0, max: 2 }), + fc.constantFrom(`0`, `1`, `source`), +) + +const sourceOperationArbitrary: fc.Arbitrary = fc.oneof( + fc + .record({ + key: sourceKeyArbitrary, + row: sourceRowArbitrary, + reportedPreviousValue: sourceRowArbitrary, + }) + .map((operation) => ({ type: `upsert` as const, ...operation })), + fc + .record({ + key: sourceKeyArbitrary, + row: sourceRowArbitrary, + reportedPreviousValue: sourceRowArbitrary, + }) + .map((operation) => ({ type: `rawUpdate` as const, ...operation })), + sourceKeyArbitrary.map((key) => ({ type: `replay` as const, key })), + fc + .record({ + key: sourceKeyArbitrary, + reportedValue: sourceRowArbitrary, + }) + .map((operation) => ({ type: `delete` as const, ...operation })), +) + +const reconciliationStepArbitrary: fc.Arbitrary = fc.oneof( + { + weight: 8, + arbitrary: fc + .array(sourceOperationArbitrary, { minLength: 1, maxLength: 5 }) + .map((operations) => ({ type: `batch` as const, operations })), + }, + { weight: 1, arbitrary: fc.constant({ type: `truncate` as const }) }, + { weight: 1, arbitrary: fc.constant({ type: `teardown` as const }) }, + { weight: 1, arbitrary: fc.constant({ type: `restart` as const }) }, +) + +const reconciliationHistoryArbitrary = fc.array(reconciliationStepArbitrary, { + minLength: 1, + maxLength: 30, +}) + +function sourceOperationForKeyArbitrary( + key: SourceKey, +): fc.Arbitrary { + return fc.oneof( + fc + .tuple(sourceRowArbitrary, sourceRowArbitrary) + .map(([row, reportedPreviousValue]) => ({ + type: `upsert` as const, + key, + row, + reportedPreviousValue, + })), + fc + .tuple(sourceRowArbitrary, sourceRowArbitrary) + .map(([row, reportedPreviousValue]) => ({ + type: `rawUpdate` as const, + key, + row, + reportedPreviousValue, + })), + fc.constant({ type: `replay` as const, key }), + sourceRowArbitrary.map((reportedValue) => ({ + type: `delete` as const, + key, + reportedValue, + })), + ) +} + +const disjointHistoriesArbitrary = fc.tuple( + fc.array(sourceOperationForKeyArbitrary(0), { + minLength: 1, + maxLength: 8, + }), + fc.array(sourceOperationForKeyArbitrary(`other`), { + minLength: 1, + maxLength: 8, + }), +) + +function rowIdentity(row: SourceRow): string { + return `${row.id}:${row.revision}:${row.value}` +} + +function expectedWeightedRowIdentity(key: SourceKey, row: SourceRow): string { + const sourceIdentity = [typeof key, String(key)].join(`:`) + const payloadIdentity = [row.id, row.revision, row.value] + .map(String) + .join(`:`) + return `${sourceIdentity}|${payloadIdentity}` +} + +function addWeight( + relation: Map, + key: SourceKey, + row: SourceRow, + weight: 1 | -1, +): void { + const identity = `${typeof key}:${String(key)}|${rowIdentity(row)}` + const nextWeight = (relation.get(identity) ?? 0) + weight + if (nextWeight === 0) relation.delete(identity) + else relation.set(identity, nextWeight) +} + +function applyToRelation( + relation: Map, + changes: ReadonlyArray>, +): void { + for (const change of changes) { + if (change.type === `insert`) { + addWeight(relation, change.key, change.value, 1) + } else if (change.type === `update`) { + addWeight(relation, change.key, change.previousValue!, -1) + addWeight(relation, change.key, change.value, 1) + } else { + addWeight(relation, change.key, change.value, -1) + } + } +} + +function sourceChangesFor( + operations: ReadonlyArray, + sourceRows: Map, +): Array> { + const changes: Array> = [] + for (const operation of operations) { + if (operation.type === `upsert`) { + const previousValue = sourceRows.get(operation.key) + changes.push( + previousValue === undefined + ? { type: `insert`, key: operation.key, value: operation.row } + : { + type: `update`, + key: operation.key, + value: operation.row, + previousValue: operation.reportedPreviousValue, + }, + ) + sourceRows.set(operation.key, operation.row) + } else if (operation.type === `rawUpdate`) { + changes.push({ + type: `update`, + key: operation.key, + value: operation.row, + previousValue: operation.reportedPreviousValue, + }) + sourceRows.set(operation.key, operation.row) + } else if (operation.type === `replay`) { + const row = sourceRows.get(operation.key) + if (row !== undefined) { + changes.push({ type: `insert`, key: operation.key, value: row }) + } + } else { + changes.push({ + type: `delete`, + key: operation.key, + value: operation.reportedValue, + }) + sourceRows.delete(operation.key) + } + } + return changes +} + +function expectTrackerMatchesSource( + sourceRows: ReadonlyMap, + sentRows: ReadonlyMap, +): void { + const compareEntries = ( + [a]: readonly [SourceKey, SourceRow], + [b]: readonly [SourceKey, SourceRow], + ) => `${typeof a}:${String(a)}`.localeCompare(`${typeof b}:${String(b)}`) + expect([...sentRows.entries()].sort(compareEntries)).toEqual( + [...sourceRows.entries()].sort(compareEntries), + ) +} + +function expectWeightedRelationMatchesSource( + sourceRows: ReadonlyMap, + relation: ReadonlyMap, +): void { + expect( + [...relation.entries()].sort(([a], [b]) => a.localeCompare(b)), + ).toEqual( + [...sourceRows.entries()] + .map(([key, row]) => [expectedWeightedRowIdentity(key, row), 1] as const) + .sort(([a], [b]) => a.localeCompare(b)), + ) +} + +function createReconciliationModel(): ReconciliationModel { + return { + sourceRows: new Map(), + sentRows: new Map(), + relation: new Map(), + graphActive: true, + } +} + +function snapshotModel(model: ReconciliationModel): { + source: Array + sent: Array + relation: Array +} { + const rows = (entries: ReadonlyMap) => + [...entries] + .map( + ([key, row]) => + `${typeof key}:${String(key)}|${row.id}:${row.revision}:${row.value}`, + ) + .sort() + return { + source: rows(model.sourceRows), + sent: rows(model.sentRows), + relation: [...model.relation].sort(([left], [right]) => + left.localeCompare(right), + ), + } +} + +function applyReconciliationStep( + model: ReconciliationModel, + step: ReconciliationStep, +): void { + if (step.type === `truncate`) { + // Truncate is only an early lifecycle signal. Its later source batch + // still needs the retained exact rows to retract the active graph. + } else if (step.type === `teardown`) { + model.sentRows.clear() + model.relation.clear() + model.graphActive = false + } else if (step.type === `restart`) { + if (!model.graphActive) { + const replay = [...model.sourceRows].map(([key, value]) => ({ + type: `insert` as const, + key, + value, + })) + applyToRelation( + model.relation, + reconcileChangesForD2(replay, model.sentRows), + ) + model.graphActive = true + } + } else { + const changes = sourceChangesFor(step.operations, model.sourceRows) + if (model.graphActive) { + const reconciled = reconcileChangesForD2(changes, model.sentRows) + applyToRelation(model.relation, reconciled) + } + } + + if (model.graphActive) { + expectTrackerMatchesSource(model.sourceRows, model.sentRows) + expectWeightedRelationMatchesSource(model.sourceRows, model.relation) + } else { + expect(model.sentRows.size).toBe(0) + expect(model.relation.size).toBe(0) + } +} + +function upsert( + key: SourceKey, + row: SourceRow, + reportedPreviousValue: SourceRow = row, +): ReconciliationStep { + return { + type: `batch`, + operations: [{ type: `upsert`, key, row, reportedPreviousValue }], + } +} + +function createOrderedSourceHarness(id: string) { + let sync!: SourceSyncActions + let loadSubsetCalls = 0 + const replayResolvers: Array<() => void> = [] + const contributed = { id: 1, revision: 1, value: 1 } + const staleDelete = { id: 1, revision: 2, value: 1 } + const replacement = { id: 1, revision: 3, value: 2 } + const source = createCollection({ + id, + getKey: (row) => row.id, + startSync: true, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (actions) => { + sync = actions + actions.markReady() + return { + loadSubset: () => { + loadSubsetCalls++ + // Initial page and its exact tie-boundary refinement are immediate; + // later calls are truncate replays controlled by the test. + if (loadSubsetCalls > 2) { + return new Promise((resolve) => replayResolvers.push(resolve)) + } + return true + }, + } + }, + }, + }) + sync.begin() + sync.write({ type: `insert`, value: contributed }) + expect(sync.commit()).toBe(true) + + let sourceCallback: Parameters[0] | undefined + let suppressSourceChanges = false + const subscribeChanges = source.subscribeChanges.bind(source) + source.subscribeChanges = ((callback, options) => { + sourceCallback = callback + return subscribeChanges((changes) => { + if (!suppressSourceChanges) callback(changes) + }, options) + }) as typeof source.subscribeChanges + + return { + contributed, + replacement, + source, + staleDelete, + suppressSourceChanges: () => { + suppressSourceChanges = true + }, + publish: (changes: Array>) => { + if (sourceCallback === undefined) { + throw new Error(`Query did not subscribe to its source`) + } + const publish = sourceCallback as unknown as ( + messages: Array>, + ) => void + publish(changes) + }, + truncate: () => { + sync.begin() + sync.truncate() + expect(sync.commit()).toBe(true) + }, + resolveReplay: async () => { + if (replayResolvers.length === 0) { + throw new Error(`No truncate replay is pending`) + } + for (let pass = 0; replayResolvers.length > 0; pass++) { + if (pass === 20) { + throw new Error( + `Truncate replay did not reach a fixed point after 20 passes`, + ) + } + for (const resolve of replayResolvers.splice(0)) resolve() + await flushPromises() + } + }, + } +} + +it(`ignores unknown deletes and inserts unknown updates at the D2 boundary`, () => { + const sentRows = new Map() + const stale = { id: 1, revision: 1, value: 1 } + const current = { id: 2, revision: 2, value: 2 } + + expect( + reconcileChangesForD2( + [{ type: `delete`, key: `row`, value: stale }], + sentRows, + ), + ).toEqual([]) + expect( + reconcileChangesForD2( + [ + { + type: `update`, + key: `row`, + previousValue: stale, + value: current, + }, + ], + sentRows, + ), + ).toEqual([{ type: `insert`, key: `row`, value: current }]) + expect(sentRows).toEqual(new Map([[`row`, current]])) +}) + +it(`retracts the exact Effect source row after an ordered truncate`, async () => { + const harness = createOrderedSourceHarness( + `d2-effect-truncate-reconciliation`, + ) + const { contributed, source, staleDelete } = harness + const events: Array<{ + type: string + value: { id: number; revision: number; value: number } + }> = [] + const effect = createEffect({ + query: (query) => + query + .from({ row: source }) + .where(({ row }) => eq(row.revision, contributed.revision)) + .orderBy(({ row }) => row.value) + .limit(1), + onBatch: (batch) => { + events.push(...batch) + }, + }) + try { + await flushPromises() + expect(events).toHaveLength(1) + expect(events[0]).toMatchObject({ + type: `enter`, + key: 1, + value: contributed, + }) + const publishedValue = events[0]!.value + + harness.suppressSourceChanges() + harness.truncate() + await flushPromises() + expect(events).toEqual([{ type: `enter`, key: 1, value: publishedValue }]) + + harness.publish([{ type: `delete`, key: 1, value: staleDelete }]) + await flushPromises() + expect(events).toEqual([ + { type: `enter`, key: 1, value: publishedValue }, + { type: `exit`, key: 1, value: publishedValue }, + ]) + } finally { + await effect.dispose() + await source.cleanup() + } +}) + +it(`retracts the exact live-query source row after ordered replay settles`, async () => { + const harness = createOrderedSourceHarness( + `d2-live-query-truncate-reconciliation`, + ) + const { contributed, source, staleDelete } = harness + const live = createLiveQueryCollection({ + id: `d2-live-query-truncate-result`, + query: (query) => + query + .from({ row: source }) + .where(({ row }) => eq(row.revision, contributed.revision)) + .orderBy(({ row }) => row.value) + .limit(1), + startSync: true, + }) + + try { + await live.preload() + expect(live.get(contributed.id)).toMatchObject(contributed) + + harness.suppressSourceChanges() + harness.truncate() + await flushPromises() + expect(live.get(contributed.id)).toMatchObject(contributed) + + harness.publish([{ type: `delete`, key: 1, value: staleDelete }]) + await flushPromises() + expect(live.get(contributed.id)).toMatchObject(contributed) + + await harness.resolveReplay() + expect(live.get(contributed.id)).toBeUndefined() + } finally { + await live.cleanup() + await source.cleanup() + } +}) + +it(`replaces the retained Effect source row after an ordered truncate`, async () => { + const harness = createOrderedSourceHarness(`d2-effect-truncate-replacement`) + const { contributed, replacement, source, staleDelete } = harness + const batches: Array< + Array<{ + type: string + value: SourceRow + previousValue?: SourceRow + }> + > = [] + const effect = createEffect({ + query: (query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.value) + .limit(1), + onBatch: (batch) => { + batches.push(batch) + }, + }) + + try { + await flushPromises() + expect(batches).toHaveLength(1) + expect(batches[0]).toHaveLength(1) + expect(batches[0]![0]).toMatchObject({ + type: `enter`, + key: 1, + value: contributed, + }) + const publishedValue = batches[0]![0]!.value + + harness.suppressSourceChanges() + harness.truncate() + await flushPromises() + expect(batches).toHaveLength(1) + + harness.publish([ + { + type: `update`, + key: 1, + previousValue: staleDelete, + value: replacement, + }, + ]) + await flushPromises() + expect(batches).toHaveLength(2) + expect(batches[1]).toHaveLength(1) + expect(batches[1]![0]).toMatchObject({ + type: `update`, + key: 1, + value: replacement, + }) + expect(batches[1]![0]!.previousValue).toBe(publishedValue) + } finally { + await effect.dispose() + await source.cleanup() + } +}) + +it(`replaces the retained live-query source row after ordered replay settles`, async () => { + const harness = createOrderedSourceHarness( + `d2-live-query-truncate-replacement`, + ) + const { contributed, replacement, source, staleDelete } = harness + const live = createLiveQueryCollection({ + id: `d2-live-query-truncate-replacement-result`, + query: (query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.value) + .limit(1), + startSync: true, + }) + const batches: Array>> = [] + + try { + await live.preload() + expect(live.get(contributed.id)).toMatchObject(contributed) + const publishedValue = live.get(contributed.id) + const subscription = live.subscribeChanges( + (changes) => batches.push(changes), + { includeInitialState: false }, + ) + + harness.suppressSourceChanges() + harness.truncate() + await flushPromises() + expect(batches).toEqual([]) + expect(live.get(contributed.id)).toBe(publishedValue) + + harness.publish([ + { + type: `update`, + key: 1, + previousValue: staleDelete, + value: replacement, + }, + ]) + await flushPromises() + expect(batches).toEqual([]) + expect(live.get(contributed.id)).toBe(publishedValue) + + await harness.resolveReplay() + expect(batches).toHaveLength(1) + expect(batches[0]).toHaveLength(1) + expect(batches[0]![0]).toMatchObject({ + type: `update`, + key: 1, + value: replacement, + }) + expect(batches[0]![0]!.previousValue).toEqual(publishedValue) + expect(live.get(replacement.id)).toMatchObject(replacement) + subscription.unsubscribe() + } finally { + await live.cleanup() + await source.cleanup() + } +}) + +it(`keeps revision and value in weighted row identity`, () => { + const key = `row` + const base = { id: 1, revision: 1, value: 1 } + const differentRevision = { id: 1, revision: 2, value: 1 } + const differentValue = { id: 1, revision: 1, value: 2 } + const relation = new Map() + + addWeight(relation, key, base, 1) + addWeight(relation, key, differentRevision, 1) + addWeight(relation, key, differentValue, 1) + + expect(relation).toEqual( + new Map([ + [expectedWeightedRowIdentity(key, base), 1], + [expectedWeightedRowIdentity(key, differentRevision), 1], + [expectedWeightedRowIdentity(key, differentValue), 1], + ]), + ) +}) + +it(`keeps numeric and string source keys distinct across restart`, () => { + const model = createReconciliationModel() + const row = { id: 1, revision: 1, value: 1 } + const keys = [0, `0`] as const + + applyReconciliationStep(model, { + type: `batch`, + operations: keys.map((key) => ({ + type: `upsert` as const, + key, + row, + reportedPreviousValue: row, + })), + }) + expect(model.sourceRows).toEqual( + new Map([ + [0, row], + [`0`, row], + ]), + ) + expect(model.sentRows).toEqual(model.sourceRows) + expect(model.relation).toEqual( + new Map([ + [expectedWeightedRowIdentity(0, row), 1], + [expectedWeightedRowIdentity(`0`, row), 1], + ]), + ) + + applyReconciliationStep(model, { type: `teardown` }) + applyReconciliationStep(model, { type: `restart` }) + expect(model.sourceRows).toEqual( + new Map([ + [0, row], + [`0`, row], + ]), + ) + expect(model.sentRows).toEqual(model.sourceRows) + expect(model.relation).toEqual( + new Map([ + [expectedWeightedRowIdentity(0, row), 1], + [expectedWeightedRowIdentity(`0`, row), 1], + ]), + ) +}) + +it(`preserves external source rows across graph teardown and restart`, () => { + const model = createReconciliationModel() + const row = { id: 1, revision: 1, value: 1 } + + applyReconciliationStep(model, upsert(`row`, row)) + applyReconciliationStep(model, { type: `teardown` }) + expect(model.graphActive).toBe(false) + expect(model.sourceRows).toEqual(new Map([[`row`, row]])) + expect(model.sentRows).toEqual(new Map()) + expect(model.relation).toEqual(new Map()) + + applyReconciliationStep(model, { type: `restart` }) + expect(model.graphActive).toBe(true) + expect(model.sourceRows).toEqual(new Map([[`row`, row]])) + expect(model.sentRows).toEqual(new Map([[`row`, row]])) + expect(model.relation).toEqual( + new Map([[expectedWeightedRowIdentity(`row`, row), 1]]), + ) +}) + +it(`replays external source changes made while the graph is down`, () => { + const model = createReconciliationModel() + const first = { id: 1, revision: 1, value: 1 } + const replacement = { id: 1, revision: 2, value: 2 } + + applyReconciliationStep(model, upsert(`row`, first)) + applyReconciliationStep(model, { type: `teardown` }) + applyReconciliationStep(model, upsert(`row`, replacement, first)) + expect(model.sourceRows).toEqual(new Map([[`row`, replacement]])) + expect(model.sentRows).toEqual(new Map()) + expect(model.relation).toEqual(new Map()) + + applyReconciliationStep(model, { type: `restart` }) + expect(model.sourceRows).toEqual(new Map([[`row`, replacement]])) + expect(model.sentRows).toEqual(new Map([[`row`, replacement]])) + expect(model.relation).toEqual( + new Map([[expectedWeightedRowIdentity(`row`, replacement), 1]]), + ) +}) + +it(`generates teardown, down-state source changes, and restart`, () => { + const histories = fc.sample(reconciliationHistoryArbitrary, { + seed: 1780, + numRuns: 500, + }) + + expect( + histories.some((steps) => { + let graphActive = true + let sawTeardown = false + let sawDownStateSourceChange = false + for (const step of steps) { + if (step.type === `teardown`) { + graphActive = false + sawTeardown = true + } else if (step.type === `restart`) { + if (!graphActive && sawTeardown && sawDownStateSourceChange) { + return true + } + graphActive = true + } else if (step.type === `batch` && !graphActive) { + sawDownStateSourceChange = true + } + } + return false + }), + ).toBe(true) +}) + +fcTest.prop( + [reconciliationHistoryArbitrary], + oraclePropertyOptions(200, `d2-source.exact-retractions`), +)( + `keeps one exact D2 contribution per source key across batched histories`, + (steps) => { + const model = createReconciliationModel() + for (const step of steps) { + applyReconciliationStep(model, step) + } + }, +) + +const assertDisjointHistoriesCommute = ([left, right]: [ + Array, + Array, +]) => { + const leftThenRight = createReconciliationModel() + applyReconciliationStep(leftThenRight, { type: `batch`, operations: left }) + applyReconciliationStep(leftThenRight, { type: `batch`, operations: right }) + + const rightThenLeft = createReconciliationModel() + applyReconciliationStep(rightThenLeft, { type: `batch`, operations: right }) + applyReconciliationStep(rightThenLeft, { type: `batch`, operations: left }) + + expect(snapshotModel(rightThenLeft)).toEqual(snapshotModel(leftThenRight)) +} + +fcTest.prop([disjointHistoriesArbitrary], { + numRuns: oracleRuns(100), + seed: 1781, +})( + `commutes independent source histories for a fixed seed`, + assertDisjointHistoriesCommute, +) + +fcTest.prop( + [disjointHistoriesArbitrary], + oraclePropertyOptions(100, `d2-source.disjoint-commutation`), +)( + `commutes independent source histories for a random or replayed seed`, + assertDisjointHistoriesCommute, +) diff --git a/packages/db/tests/db-client.test-d.ts b/packages/db/tests/db-client.test-d.ts new file mode 100644 index 0000000000..74cb1943f0 --- /dev/null +++ b/packages/db/tests/db-client.test-d.ts @@ -0,0 +1,115 @@ +import { describe, expectTypeOf, it } from 'vitest' +import { z } from 'zod' +import { DbClient, collectionOptions, eq } from '../src' +import type { + DehydratedCollectionChunk, + DehydratedDbState, + DehydratedLiveQueryResult, +} from '../src' + +type Todo = { + id: string + title: string +} + +describe(`DbClient type assertions`, () => { + it(`types explicit dependencies`, () => { + const queryClient = { + invalidateQueries: () => Promise.resolve(), + } + const client = new DbClient({ queryClient }) + + expectTypeOf( + client.getDependency(`queryClient`), + ).toEqualTypeOf() + expectTypeOf( + client.requireDependency(`queryClient`), + ).toEqualTypeOf() + }) + + it(`infers collections from client-aware descriptor factories`, () => { + const descriptor = collectionOptions(`todos`, (client) => { + expectTypeOf(client).toEqualTypeOf() + + return { + id: `todos`, + getKey: (todo: Todo) => todo.id, + sync: { + sync: () => {}, + }, + } + }) + const client = new DbClient() + const collection = client.collection(descriptor, { + initialData: [{ id: `1`, title: `Ship SSR` }], + }) + + expectTypeOf(collection.get(`1`)).toMatchTypeOf() + collection.insert({ id: `2`, title: `Keep inference` }) + }) + + it(`types holistic and incremental hydration payloads`, () => { + const client = new DbClient() + const state: DehydratedDbState = { + collections: [ + { + collectionId: `todos`, + rows: [ + { + key: `1`, + value: { id: `1`, title: `Ship SSR` }, + }, + ], + }, + ], + } + const chunk: DehydratedCollectionChunk = state + .collections[0] as DehydratedCollectionChunk + + client.hydrate(state) + client.applyCollectionChunk(chunk) + + expectTypeOf(client.dehydrate()).toEqualTypeOf() + }) + + it(`types live-query preload and result snapshots`, () => { + const descriptor = collectionOptions(`live-todos`, () => ({ + id: `live-todos`, + getKey: (todo: Todo) => todo.id, + sync: { sync: () => {} }, + })) + const client = new DbClient() + const preload = client.preloadLiveQuery({ + query: (q) => + q.from({ todo: descriptor }).where(({ todo }) => eq(todo.id, `1`)), + }) + const snapshot: DehydratedLiveQueryResult = { + rows: [{ key: `1`, value: { id: `1`, title: `Ship SSR` } }], + } + + expectTypeOf(preload).toEqualTypeOf>() + expectTypeOf(snapshot.rows[0]!.value).toEqualTypeOf() + }) + + it(`preserves schema input and output through descriptor factories`, () => { + const schema = z.object({ + id: z.string(), + createdAt: z.string().transform((value) => new Date(value)), + }) + const descriptor = collectionOptions(`schema-items`, () => ({ + id: `schema-items`, + schema, + getKey: (item: z.output) => item.id, + sync: { + sync: () => {}, + }, + })) + const collection = new DbClient().collection(descriptor, { + initialData: [{ id: `1`, createdAt: `2026-01-01T00:00:00.000Z` }], + }) + + expectTypeOf(collection.get(`1`)?.createdAt).toEqualTypeOf< + Date | undefined + >() + }) +}) diff --git a/packages/db/tests/db-client.test.ts b/packages/db/tests/db-client.test.ts new file mode 100644 index 0000000000..a6a61bab81 --- /dev/null +++ b/packages/db/tests/db-client.test.ts @@ -0,0 +1,1174 @@ +import { describe, expect, it, vi } from 'vitest' +import { z } from 'zod' +import { + DbClient, + collectionOptions, + createLiveQueryCollection, + createTransaction, + eq, + liveQueryCollectionOptions, + localOnlyCollectionOptions, +} from '../src' +import { mockSyncCollectionOptions } from './utils' +import type { DehydratedLiveQueryResult, InitialQueryBuilder } from '../src' + +type Person = { + id: string + name: string + status?: string +} + +const people: Array = [ + { id: `1`, name: `Tanner`, status: `active` }, + { id: `2`, name: `Kyle`, status: `inactive` }, +] + +describe(`DbClient`, () => { + it(`memoizes materialized collections per client and isolates clients`, () => { + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: people, + }), + ) + + const clientA = new DbClient() + const clientB = new DbClient() + + const peopleA1 = clientA.collection(descriptor) + const peopleA2 = clientA.collection(descriptor) + const peopleB = clientB.collection(descriptor) + + expect(peopleA1).toBe(peopleA2) + expect(peopleA1).not.toBe(peopleB) + expect(peopleA1.toArray).toHaveLength(2) + expect(peopleB.toArray).toHaveLength(2) + }) + + it(`materializes independent adapter state for each client`, async () => { + const descriptor = collectionOptions( + localOnlyCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + }), + ) + const clientA = new DbClient() + const clientB = new DbClient() + const peopleA = clientA.collection(descriptor) + const peopleB = clientB.collection(descriptor) + + const transaction = peopleA.insert(people[0]!) + await transaction.isPersisted.promise + + expect(peopleA.get(`1`)).toMatchObject(people[0]!) + expect(peopleB.get(`1`)).toBeUndefined() + }) + + it(`does not reuse concrete configs across clients`, () => { + const descriptor = collectionOptions({ + id: `people`, + getKey: (person: Person) => person.id, + sync: { sync: () => {} }, + }) + + new DbClient().collection(descriptor) + + expect(() => new DbClient().collection(descriptor)).toThrow( + /cannot be safely reused across DbClient instances/, + ) + }) + + it(`isolates ambient transactions between clients`, async () => { + const descriptor = collectionOptions( + localOnlyCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + }), + ) + const clientA = new DbClient() + const clientB = new DbClient() + const peopleA = clientA.collection(descriptor) + const peopleB = clientB.collection(descriptor) + const transactionA = clientA.createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + const rolledBack = transactionA.isPersisted.promise.catch(() => undefined) + let transactionB: ReturnType | undefined + + transactionA.mutate(() => { + expect(peopleA.insert(people[0]!)).toBe(transactionA) + transactionB = peopleB.insert(people[1]!) + expect(transactionB).not.toBe(transactionA) + expect(clientA.activeTransaction).toBe(transactionA) + expect(clientB.activeTransaction).toBeUndefined() + }) + + await transactionB!.isPersisted.promise + transactionA.rollback() + await rolledBack + + expect(peopleA.get(`1`)).toBeUndefined() + expect(peopleB.get(`2`)).toMatchObject(people[1]!) + }) + + it(`binds the backwards-compatible createTransaction API to one client`, async () => { + const descriptor = collectionOptions( + localOnlyCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + }), + ) + const clientA = new DbClient() + const clientB = new DbClient() + const peopleA = clientA.collection(descriptor) + const peopleB = clientB.collection(descriptor) + const transaction = createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + const rolledBack = transaction.isPersisted.promise.catch(() => undefined) + + transaction.mutate(() => { + expect(peopleA.insert(people[0]!)).toBe(transaction) + expect(() => peopleB.insert(people[1]!)).toThrow( + /cannot mutate collections from multiple DbClient instances/, + ) + }) + + transaction.rollback() + await rolledBack + + expect(peopleA.get(`1`)).toBeUndefined() + expect(peopleB.get(`2`)).toBeUndefined() + }) + + it(`cleans up materialized collections and allows rematerialization`, async () => { + const cleanup = vi.fn() + const descriptor = collectionOptions(`people`, () => ({ + id: `people`, + getKey: (person: Person) => person.id, + startSync: true, + sync: { + sync: () => ({ cleanup }), + }, + })) + const client = new DbClient() + const first = client.collection(descriptor) + + await client.cleanup() + + expect(cleanup).toHaveBeenCalledOnce() + expect(client.dehydrate()).toEqual({ collections: [] }) + expect(client.collection(descriptor)).not.toBe(first) + }) + + it(`serializes collection rows and sync metadata from explicit ids`, () => { + let syncMeta = { version: 1, cursor: `a` } + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: people, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: people[0]!, + metadata: { source: `server` }, + }) + commit() + markReady() + }, + exportSyncMeta: () => syncMeta, + importSyncMeta: (meta) => { + syncMeta = meta as typeof syncMeta + }, + mergeSyncMeta: (_current, incoming) => incoming, + }, + }), + ) + + const client = new DbClient() + client.collection(descriptor) + + const dehydrated = client.dehydrate() + + expect(dehydrated).toEqual({ + collections: [ + { + collectionId: `people`, + rows: [ + { + key: `1`, + value: people[0], + metadata: { source: `server` }, + }, + ], + syncMeta: { version: 1, cursor: `a` }, + }, + ], + }) + }) + + it(`serializes only collections materialized through the client`, () => { + const peopleDescriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: people, + }), + ) + collectionOptions( + mockSyncCollectionOptions({ + id: `unused-people`, + getKey: (person) => person.id, + initialData: [{ id: `3`, name: `Unused` }], + }), + ) + + const client = new DbClient() + + expect(client.dehydrate()).toEqual({ collections: [] }) + + client.collection(peopleDescriptor) + + expect( + client.dehydrate().collections.map((chunk) => chunk.collectionId), + ).toEqual([`people`]) + }) + + it(`uses the collection id to reuse separately-created descriptors`, () => { + const firstFactory = vi.fn(() => + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: [people[0]!], + }), + ) + const secondFactory = vi.fn(() => + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: [people[1]!], + }), + ) + const firstDescriptor = collectionOptions(`people`, firstFactory) + const secondDescriptor = collectionOptions(`people`, secondFactory) + + const client = new DbClient() + const first = client.collection(firstDescriptor) + const second = client.collection(secondDescriptor) + + expect(second).toBe(first) + expect(firstFactory).toHaveBeenCalledOnce() + expect(secondFactory).not.toHaveBeenCalled() + expect(second.toArray).toHaveLength(1) + expect(second.toArray[0]).toMatchObject(people[0]!) + }) + + it(`reuses a same-id collection materialized by a descriptor factory`, () => { + const client = new DbClient() + const nestedDescriptor = collectionOptions(`people`, () => + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: [people[0]!], + }), + ) + const nestedCollections: Array = [] + const outerDescriptor = collectionOptions(`people`, () => { + nestedCollections.push(client.collection(nestedDescriptor)) + return mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: [people[1]!], + }) + }) + + const collection = client.collection(outerDescriptor) + + expect(collection).toBe(nestedCollections[0]) + expect(collection.toArray).toHaveLength(1) + expect(collection.toArray[0]).toMatchObject(people[0]!) + }) + + it(`requires a stable explicit collection id when creating a descriptor`, () => { + expect(() => + collectionOptions( + mockSyncCollectionOptions({ + id: undefined as unknown as string, + getKey: (person) => person.id, + initialData: people, + }), + ), + ).toThrow(/collectionOptions requires a non-empty explicit id/) + }) + + it(`rejects an empty collection descriptor id`, () => { + expect(() => + collectionOptions( + mockSyncCollectionOptions({ + id: ``, + getKey: (person) => person.id, + initialData: people, + }), + ), + ).toThrow(/collectionOptions requires a non-empty explicit id/) + }) + + it(`requires a factory when using the explicit id overload`, () => { + expect(() => + Reflect.apply(collectionOptions, undefined, [`people`]), + ).toThrow(/collectionOptions\("people"\) requires a factory/) + }) + + it(`hydrates pending collection rows when the collection materializes`, () => { + const importedMeta = vi.fn() + const lifecycleOrder: Array = [] + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: [], + sync: { + sync: ({ markReady }) => { + lifecycleOrder.push(`sync`) + markReady() + }, + importSyncMeta: (meta) => { + lifecycleOrder.push(`import`) + importedMeta(meta) + }, + }, + }), + ) + + const client = new DbClient() + client.hydrate({ + collections: [ + { + collectionId: `people`, + rows: [ + { + key: `1`, + value: people[0]!, + metadata: { source: `ssr` }, + }, + ], + syncMeta: { version: 1, cursor: `ssr` }, + }, + ], + }) + + const collection = client.collection(descriptor) + + expect(collection.get(`1`)).toMatchObject(people[0]!) + expect(collection._state.syncedMetadata.get(`1`)).toEqual({ + source: `ssr`, + }) + expect(importedMeta).toHaveBeenCalledWith({ version: 1, cursor: `ssr` }) + expect(lifecycleOrder).toEqual([`import`, `sync`]) + expect(collection.status).toBe(`ready`) + }) + + it(`defers adapter sync and replays subset loads after hydrated rows render`, () => { + const lifecycleOrder: Array = [] + const descriptor = collectionOptions(`people`, () => ({ + id: `people`, + getKey: (person: Person) => person.id, + syncMode: `on-demand` as const, + sync: { + sync: ({ begin, write, commit, markReady }) => { + lifecycleOrder.push(`sync`) + markReady() + return { + loadSubset: () => { + lifecycleOrder.push(`load`) + begin({ immediate: true }) + write({ + type: `insert`, + value: { id: `1`, name: `fresh` }, + }) + commit() + return true + }, + } + }, + }, + })) + const client = new DbClient() + + client.hydrate({ + collections: [ + { + collectionId: `people`, + rows: [{ key: `1`, value: { id: `1`, name: `stale` } }], + }, + ], + }) + + const collection = client._materializeCollectionForRender(descriptor) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: true, + }) + + expect(collection.get(`1`)).toMatchObject({ id: `1`, name: `stale` }) + expect(lifecycleOrder).toEqual([]) + + collection._resumeSyncStart() + + expect(lifecycleOrder).toEqual([`sync`, `load`]) + expect(collection.get(`1`)).toMatchObject({ id: `1`, name: `fresh` }) + subscription.unsubscribe() + }) + + it(`keeps deferred subset loads pending until the replayed adapter load finishes`, async () => { + let resolveLoad!: () => void + const adapterLoad = new Promise((resolve) => { + resolveLoad = resolve + }) + const loadSubset = vi.fn(() => adapterLoad) + const descriptor = collectionOptions(`people`, () => ({ + id: `people`, + getKey: (person: Person) => person.id, + syncMode: `on-demand` as const, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset } + }, + }, + })) + const client = new DbClient() + const collection = client._materializeCollectionForRender(descriptor) + + const deferredLoad = collection._sync.loadSubset({}) + expect(deferredLoad).toBeInstanceOf(Promise) + expect(collection.isLoadingSubset).toBe(true) + expect(loadSubset).not.toHaveBeenCalled() + + collection._resumeSyncStart() + expect(loadSubset).toHaveBeenCalledOnce() + expect(collection.isLoadingSubset).toBe(true) + + resolveLoad() + await deferredLoad + expect(collection.isLoadingSubset).toBe(false) + }) + + it(`lets the first sync snapshot replace stale hydrated rows`, () => { + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: [{ id: `1`, name: `fresh` }], + }), + ) + const client = new DbClient() + + client.hydrate({ + collections: [ + { + collectionId: `people`, + rows: [{ key: `1`, value: { id: `1`, name: `stale` } }], + }, + ], + }) + + const collection = client.collection(descriptor) + + expect(collection.get(`1`)).toMatchObject({ id: `1`, name: `fresh` }) + }) + + it(`merges sync metadata before importing hydration metadata`, () => { + let syncMeta: unknown = { version: 1, cursor: `client` } + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: [], + sync: { + sync: ({ markReady }) => { + markReady() + }, + exportSyncMeta: () => syncMeta, + importSyncMeta: (meta) => { + syncMeta = meta + }, + mergeSyncMeta: (current, incoming) => ({ current, incoming }), + }, + }), + ) + + const client = new DbClient() + client.collection(descriptor) + + client.hydrate({ + collections: [ + { + collectionId: `people`, + rows: [], + syncMeta: { version: 1, cursor: `server` }, + }, + ], + }) + + expect(syncMeta).toEqual({ + current: { version: 1, cursor: `client` }, + incoming: { version: 1, cursor: `server` }, + }) + }) + + it(`applies streaming collection chunks and live queries react from collection state`, async () => { + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: [], + sync: { + sync: ({ markReady }) => { + markReady() + }, + }, + }), + ) + + const client = new DbClient() + const collection = client.collection(descriptor) + const activePeople = createLiveQueryCollection((q) => + q + .from({ person: collection }) + .where(({ person }) => eq(person.status, `active`)), + ) + await activePeople.preload() + + client.applyCollectionChunk({ + collectionId: `people`, + rows: [{ key: `1`, value: people[0]! }], + }) + + expect(activePeople.toArray.map((person) => person.id)).toEqual([`1`]) + }) + + it(`streams pending live queries as result snapshots`, async () => { + const descriptor = collectionOptions(`people`, () => ({ + id: `people`, + getKey: (person: Person) => person.id, + sync: { + sync: ({ markReady }) => markReady(), + }, + })) + const serverClient = new DbClient() + serverClient.collection(descriptor) + const listener = vi.fn() + serverClient.subscribe(listener) + + let resolveLoad!: (snapshot: { + rows: Array<{ key: string; value: Person }> + }) => void + const loadPromise = new Promise<{ + rows: Array<{ key: string; value: Person }> + }>((resolve) => { + resolveLoad = resolve + }) + serverClient._registerLiveQuery(`active-people`, loadPromise) + + expect(listener).toHaveBeenCalledWith( + expect.objectContaining({ + type: `liveQueryAdded`, + query: expect.objectContaining({ + queryHash: `active-people`, + status: `pending`, + }), + }), + ) + expect(serverClient.dehydrate().liveQueries).toBeUndefined() + + const dehydrated = serverClient.dehydrate({ + shouldDehydrateCollection: () => false, + shouldDehydrateLiveQuery: () => true, + }) + expect(dehydrated.collections).toEqual([]) + expect(dehydrated.liveQueries).toHaveLength(1) + + const browserClient = new DbClient() + browserClient.hydrate(dehydrated) + const browserQuery = browserClient._getLiveQuery(`active-people`) + expect(browserQuery?.status).toBe(`pending`) + + resolveLoad({ + rows: [{ key: `1`, value: people[0]! }], + }) + await browserQuery?.promise + + expect(browserClient._getLiveQuery(`active-people`)?.status).toBe(`success`) + expect(browserQuery?.snapshot).toEqual({ + rows: [{ key: `1`, value: people[0]! }], + }) + expect(browserClient.collection(descriptor).get(`1`)).toBeUndefined() + }) + + it(`propagates streamed live query failures to the hydrated client`, async () => { + const serverClient = new DbClient() + let rejectLoad!: (error: Error) => void + const loadPromise = new Promise<{ + rows: Array<{ key: string; value: Person }> + }>((_resolve, reject) => { + rejectLoad = reject + }) + serverClient._registerLiveQuery(`active-people`, loadPromise) + const dehydrated = serverClient.dehydrate({ + shouldDehydrateLiveQuery: () => true, + }) + const browserClient = new DbClient() + + browserClient.hydrate(dehydrated) + const browserQuery = browserClient._getLiveQuery(`active-people`) + const error = new Error(`Server load failed`) + rejectLoad(error) + + await expect(browserQuery?.promise).rejects.toBe(error) + expect(browserQuery?.status).toBe(`error`) + expect(browserQuery?.error).toBe(error) + }) + + it(`settles waiters when newer hydration supersedes a pending live query`, async () => { + const client = new DbClient() + let resolveOriginal!: (snapshot: { + rows: Array<{ key: string; value: Person }> + }) => void + const originalResult = new Promise<{ + rows: Array<{ key: string; value: Person }> + }>((resolve) => { + resolveOriginal = resolve + }) + const originalPromise = client._registerLiveQuery( + `active-people`, + originalResult, + ) + const originalRecord = client._getLiveQuery(`active-people`)! + + client.hydrate({ + collections: [], + liveQueries: [ + { + queryHash: `active-people`, + dehydratedAt: originalRecord.dehydratedAt + 1, + snapshot: { + rows: [{ key: `1`, value: people[0]! }], + }, + }, + ], + }) + + await expect(originalPromise).resolves.toBeUndefined() + expect(client._getLiveQuery(`active-people`)).toMatchObject({ + status: `success`, + snapshot: { rows: [{ key: `1`, value: people[0]! }] }, + }) + + resolveOriginal({ rows: [] }) + }) + + it(`handles a rejected duplicate live-query registration`, async () => { + const client = new DbClient() + await client._registerLiveQuery( + `active-people`, + Promise.resolve({ rows: [] }), + ) + + let rejectDuplicate!: (error: Error) => void + const duplicate = new Promise( + (_resolve, reject) => { + rejectDuplicate = reject + }, + ) + + await expect( + client._registerLiveQuery(`active-people`, duplicate), + ).resolves.toBeUndefined() + rejectDuplicate(new Error(`duplicate failed`)) + await new Promise((resolve) => setTimeout(resolve, 0)) + }) + + it(`explicit collection preload dehydrates source collection rows`, async () => { + const descriptor = collectionOptions({ + id: `people`, + getKey: (person: Person) => person.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + + return { + loadSubset: () => { + begin({ immediate: true }) + for (const person of people) { + write({ + type: `insert`, + value: person, + }) + } + commit() + return true + }, + } + }, + }, + }) + + const client = new DbClient() + const collection = client.collection(descriptor) + const activePeople = createLiveQueryCollection((q) => + q + .from({ person: collection }) + .where(({ person }) => eq(person.status, `active`)), + ) + + await activePeople.preload() + + expect(activePeople.toArray.map((person) => person.id)).toEqual([`1`]) + expect(client.dehydrate()).toEqual({ + collections: [ + { + collectionId: `people`, + rows: people.map((person) => ({ + key: person.id, + value: person, + })), + syncMeta: undefined, + }, + ], + }) + }) + + it(`does not dehydrate collections materialized only as query sources`, () => { + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: people, + }), + ) + const client = new DbClient() + + client._materializeCollectionForRender(descriptor) + expect(client.dehydrate()).toEqual({ collections: [] }) + + client.collection(descriptor) + expect(client.dehydrate().collections).toHaveLength(1) + }) + + it(`preloads and dehydrates a live query result without its source rows`, async () => { + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: people, + }), + ) + const client = new DbClient() + + await client.preloadLiveQuery({ + query: (q) => + q + .from({ person: descriptor }) + .where(({ person }) => eq(person.status, `active`)), + }) + + const dehydrated = client.dehydrate() + expect(dehydrated.collections).toEqual([]) + expect(dehydrated.liveQueries).toHaveLength(1) + expect(dehydrated.liveQueries![0]!.promise).toBeUndefined() + expect(dehydrated.liveQueries![0]!.snapshot?.rows).toEqual([ + { + key: `1`, + value: expect.objectContaining(people[0]!), + }, + ]) + expect( + client.dehydrate({ shouldDehydrateLiveQuery: () => false }).liveQueries, + ).toBeUndefined() + }) + + it(`cleans up a failed live-query preload before retrying it`, async () => { + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `retry-people`, + getKey: (person) => person.id, + initialData: people, + }), + ) + const client = new DbClient() + const options = { + query: (q: InitialQueryBuilder) => + q + .from({ person: descriptor }) + .where(({ person }) => eq(person.status, `active`)), + } + + await client.preloadLiveQuery(options) + const internals = client as unknown as { + liveQueries: Map + preloadedLiveQueries: Map< + string, + { collection: { cleanup: () => Promise } } + > + } + const failedQuery = Array.from(internals.liveQueries.values())[0]! + failedQuery.status = `error` + failedQuery.error = new Error(`failed`) + const failedCollection = Array.from( + internals.preloadedLiveQueries.values(), + )[0]!.collection + const originalCleanup = failedCollection.cleanup.bind(failedCollection) + const cleanup = vi + .spyOn(failedCollection, `cleanup`) + .mockImplementationOnce(async () => { + await originalCleanup() + throw new Error(`cleanup failed`) + }) + + await expect(client.preloadLiveQuery(options)).resolves.toBeUndefined() + + expect(cleanup).toHaveBeenCalledOnce() + expect(client.dehydrate().liveQueries?.[0]?.snapshot?.rows).toHaveLength(1) + }) + + it(`releases source deferrals when preload returns an existing result`, async () => { + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `preload-existing-source`, + getKey: (person) => person.id, + initialData: people, + }), + ) + const client = new DbClient() + const options = { + query: (q: InitialQueryBuilder) => q.from({ person: descriptor }), + } + + await client.preloadLiveQuery(options) + const source = client.collection(descriptor) + await source.cleanup() + await client.preloadLiveQuery(options) + + await expect( + Promise.race([ + source.preload().then(() => `ready`), + new Promise<`timeout`>((resolve) => + setTimeout(() => resolve(`timeout`), 20), + ), + ]), + ).resolves.toBe(`ready`) + }) + + it(`cleans up live queries before their source collections`, async () => { + const errorSpy = vi.spyOn(console, `error`).mockImplementation(() => {}) + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `cleanup-order-source`, + getKey: (person) => person.id, + initialData: people, + }), + ) + const client = new DbClient() + + try { + await client.preloadLiveQuery({ + query: (q: InitialQueryBuilder) => q.from({ person: descriptor }), + }) + await client.cleanup() + + expect(errorSpy).not.toHaveBeenCalledWith( + expect.stringContaining(`was manually cleaned up while live query`), + ) + } finally { + errorSpy.mockRestore() + } + }) + + it(`does not dehydrate explicitly client-bound live query result collections`, async () => { + const peopleDescriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: people, + }), + ) + const activePeopleDescriptor = collectionOptions( + `active-people`, + (client) => + liveQueryCollectionOptions({ + id: `active-people`, + query: (q) => + q + .from({ person: client.collection(peopleDescriptor) }) + .where(({ person }) => eq(person.status, `active`)), + }), + ) + const client = new DbClient() + const activePeople = client.collection(activePeopleDescriptor) + + await activePeople.preload() + + expect(activePeople.toArray.map((person) => person.id)).toEqual([`1`]) + expect( + client.dehydrate().collections.map((chunk) => chunk.collectionId), + ).toEqual([`people`]) + }) + + it(`hydrates rows without running mutation handlers or creating optimistic state`, () => { + const onInsert = vi.fn() + const descriptor = collectionOptions({ + id: `people`, + getKey: (person: Person) => person.id, + sync: { + sync: ({ markReady }) => { + markReady() + }, + }, + onInsert, + }) + + const client = new DbClient() + const collection = client.collection(descriptor) + + client.hydrate({ + collections: [ + { + collectionId: `people`, + rows: [{ key: `1`, value: people[0]! }], + }, + ], + }) + + expect(onInsert).not.toHaveBeenCalled() + expect(collection._state.optimisticUpserts.size).toBe(0) + expect(collection._state.optimisticDeletes.size).toBe(0) + expect(collection.get(`1`)).toMatchObject(people[0]!) + }) + + it(`validates and transforms hydrated rows through the collection schema`, () => { + const descriptor = collectionOptions({ + id: `schema-hydration`, + schema: z.object({ + id: z.string().transform((id) => `person:${id}`), + createdAt: z.coerce.date(), + }), + getKey: (row) => row.id, + sync: { sync: ({ markReady }) => markReady() }, + }) + const client = new DbClient() + const collection = client.collection(descriptor) + + client.hydrate({ + collections: [ + { + collectionId: `schema-hydration`, + rows: [ + { + key: `1`, + value: { id: `1`, createdAt: `2026-08-14T00:00:00.000Z` }, + }, + ], + }, + ], + }) + + expect(collection.get(`person:1`)?.createdAt).toBeInstanceOf(Date) + expect(collection.get(`1`)).toBeUndefined() + }) + + it(`lets adapter inserts replace hydration applied to a ready collection`, async () => { + let adapterWrite!: (person: Person) => void + const descriptor = collectionOptions({ + id: `ready-hydration-seed`, + getKey: (person: Person) => person.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + adapterWrite = (person) => { + begin() + write({ type: `insert`, value: person }) + commit() + } + markReady() + }, + }, + }) + const client = new DbClient() + const collection = client.collection(descriptor) + await collection.preload() + + client.hydrate({ + collections: [ + { + collectionId: `ready-hydration-seed`, + rows: [{ key: `1`, value: { id: `1`, name: `hydrated` } }], + }, + ], + }) + + expect(() => adapterWrite({ id: `1`, name: `adapter` })).not.toThrow() + expect(collection.get(`1`)?.name).toBe(`adapter`) + + client.hydrate({ + collections: [ + { + collectionId: `ready-hydration-seed`, + rows: [{ key: `1`, value: { id: `1`, name: `late hydration` } }], + }, + ], + }) + + expect(collection.get(`1`)?.name).toBe(`adapter`) + }) + + it(`does not let a late stream chunk overwrite adapter rows or metadata`, async () => { + const descriptor = collectionOptions({ + id: `adapter-authority`, + getKey: (person: Person) => person.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { id: `1`, name: `adapter` }, + metadata: { source: `adapter` }, + }) + commit() + markReady() + }, + }, + }) + const client = new DbClient() + const collection = client.collection(descriptor) + await collection.preload() + + expect(collection._state.syncedData.has(`1`)).toBe(true) + expect(collection._state.hydrationSeedKeys.has(`1`)).toBe(false) + + client.applyCollectionChunk({ + collectionId: `adapter-authority`, + rows: [{ key: `1`, value: { id: `1`, name: `stale stream` } }], + }) + + expect(collection.get(`1`)?.name).toBe(`adapter`) + expect(collection._state.syncedMetadata.get(`1`)).toEqual({ + source: `adapter`, + }) + }) + + it(`does not serialize optimistic pending mutations`, async () => { + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: [people[0]!], + }), + ) + + const client = new DbClient() + const collection = client.collection(descriptor) + const tx = collection.insert({ id: `3`, name: `Pending` }) + + expect(collection._state.optimisticUpserts.has(`3`)).toBe(true) + expect(client.dehydrate()).toEqual({ + collections: [ + { + collectionId: `people`, + rows: [ + { + key: `1`, + value: people[0], + }, + ], + syncMeta: undefined, + }, + ], + }) + + collection.utils.resolveSync() + await tx.isPersisted.promise + }) + + it(`applies initialData precedence before hydrated rows`, () => { + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `people`, + getKey: (person) => person.id, + initialData: [], + }), + ) + + const client = new DbClient() + const collection = client.collection(descriptor, { + initialData: [{ id: `1`, name: `materialized` }], + }) + + expect(collection.get(`1`)).toMatchObject({ + id: `1`, + name: `materialized`, + }) + + client.hydrate({ + collections: [ + { + collectionId: `people`, + rows: [{ key: `1`, value: { id: `1`, name: `hydrated` } }], + }, + ], + }) + + expect(collection.get(`1`)).toMatchObject({ + id: `1`, + name: `hydrated`, + }) + }) + + it(`seeds initialData without marking adapter sync as ready`, () => { + const descriptor = collectionOptions({ + id: `people`, + getKey: (person) => person.id, + sync: { + sync: () => {}, + }, + }) + + const client = new DbClient() + const collection = client.collection(descriptor, { + initialData: [people[0]!], + }) + + expect(collection.get(`1`)).toMatchObject(people[0]!) + expect(collection.status).not.toBe(`ready`) + }) + + it(`validates and transforms materialization initialData before keying`, () => { + const personSchema = z.object({ + id: z.string().transform((id) => `person:${id}`), + name: z.string(), + }) + const descriptor = collectionOptions({ + id: `people`, + schema: personSchema, + getKey: (person) => person.id, + sync: { + sync: () => {}, + }, + }) + + const collection = new DbClient().collection(descriptor, { + initialData: [{ id: `1`, name: `Tanner` }], + }) + + expect(collection.get(`person:1`)).toMatchObject({ + id: `person:1`, + name: `Tanner`, + }) + expect(collection.get(`1`)).toBeUndefined() + }) +}) diff --git a/packages/db/tests/deep-equals-work.test.ts b/packages/db/tests/deep-equals-work.test.ts new file mode 100644 index 0000000000..7ced0d481e --- /dev/null +++ b/packages/db/tests/deep-equals-work.test.ts @@ -0,0 +1,29 @@ +import { describe, expect, it, vi } from 'vitest' +import { deepEquals } from '../src/utils.js' + +describe(`deep equality enumeration work`, () => { + it.each([false, true])( + `avoids intermediate filtered key arrays with symbols=%s`, + (symbols) => { + const key = Symbol(`field`) + const createRow = () => ({ + id: 1, + nested: { value: 2 }, + ...(symbols ? { [key]: 3 } : {}), + }) + const left = createRow() + const right = createRow() + const spy = vi.spyOn(Array.prototype, `filter`) + let calls: number + let equal: boolean + try { + equal = deepEquals(left, right) + calls = spy.mock.calls.length + } finally { + spy.mockRestore() + } + expect(equal).toBe(true) + expect(calls).toBe(0) + }, + ) +}) diff --git a/packages/db/tests/deterministic-ordering.test.ts b/packages/db/tests/deterministic-ordering.test.ts index 9ce9a326d6..160d03284b 100644 --- a/packages/db/tests/deterministic-ordering.test.ts +++ b/packages/db/tests/deterministic-ordering.test.ts @@ -489,5 +489,38 @@ describe(`Deterministic Ordering`, () => { const keys = changes?.map((c) => c.key) expect(keys).toEqual([`a`, `b`, `c`]) }) + + it(`should place NaN values consistently when ordering`, () => { + type Item = { id: string; score: number } + + const options = mockSyncCollectionOptions({ + id: `test-collection-changes-nan`, + getKey: (item) => item.id, + initialData: [], + }) + + const collection = createCollection(options) + + options.utils.begin() + options.utils.write({ type: `insert`, value: { id: `a`, score: 5 } }) + options.utils.write({ type: `insert`, value: { id: `nan`, score: NaN } }) + options.utils.write({ type: `insert`, value: { id: `b`, score: 1 } }) + options.utils.write({ type: `insert`, value: { id: `c`, score: 3 } }) + options.utils.commit() + + const changes = collection.currentStateAsChanges({ + orderBy: [ + { + expression: new PropRef([`score`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + }) + + // Under PostgreSQL float semantics NaN is the greatest value, so the + // numbers sort ascending first and NaN sorts last. + const keys = changes?.map((c) => c.key) + expect(keys).toEqual([`b`, `c`, `a`, `nan`]) + }) }) }) diff --git a/packages/db/tests/effect-disposal-oracle.test.ts b/packages/db/tests/effect-disposal-oracle.test.ts new file mode 100644 index 0000000000..7428721eb7 --- /dev/null +++ b/packages/db/tests/effect-disposal-oracle.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, it } from 'vitest' +import { createCollection, createEffect } from '../src/index.js' +import { createDeferred } from '../src/deferred.js' +import { flushPromises } from './utils.js' + +// One disposal attempt has one outcome, even when abort/release callbacks +// reenter it. Counting physical releases alone misses divergent caller results. +const scenarios = ([`abort`, `release`] as const).flatMap((reentry) => + [false, true].flatMap((pendingHandler) => + ([`success`, `error`, `undefined`] as const).map((outcome) => ({ + reentry, + pendingHandler, + outcome, + })), + ), +) + +describe(`Effect disposal outcome oracle`, () => { + it.each(scenarios)( + `joins all callers to one attempt: %j`, + async ({ reentry, pendingHandler, outcome }) => { + const failure = new Error(`release failed`) + const handler = createDeferred() + let nested: Promise | undefined + let releases = 0 + const source = createCollection<{ id: number }>({ + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: () => { + begin() + write({ type: `insert`, value: { id: 1 } }) + commit() + return true + }, + unloadSubset: () => { + releases++ + if (reentry === `release`) nested = effect.dispose() + if (outcome === `error`) throw failure + if (outcome === `undefined`) throw undefined + }, + } + }, + }, + }) + const effect: ReturnType = createEffect({ + query: (q) => q.from({ row: source }), + onBatch: (_events, { signal }) => { + if (reentry === `abort`) + signal.addEventListener( + `abort`, + () => { + nested = effect.dispose() + }, + { once: true }, + ) + return pendingHandler ? handler.promise : undefined + }, + }) + try { + await flushPromises() + const outer = effect.dispose() + // Observe every promise before any assertion can throw. + const results = Promise.allSettled([outer, nested!, effect.dispose()]) + let settled = false + void results.then(() => { + settled = true + }) + expect(nested).toBeDefined() + expect(effect.disposed).toBe(true) + expect(source.subscriberCount).toBe(0) + expect(releases).toBe(1) + if (pendingHandler) { + await flushPromises() + expect(settled).toBe(false) + } + handler.resolve() + const observed = await results + for (const result of observed) { + expect(result.status).toBe( + outcome === `success` ? `fulfilled` : `rejected`, + ) + if (result.status === `rejected`) { + if (outcome === `error`) expect(result.reason).toBe(failure) + else expect(result.reason).toMatchObject({ message: `undefined` }) + } + expect(result).toEqual(observed[0]) + } + // A settled failed attempt does not make the source lease retryable. + await effect.dispose() + expect(releases).toBe(1) + } finally { + handler.resolve() + await Promise.allSettled([nested, effect.dispose()]) + await source.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/effect.test.ts b/packages/db/tests/effect.test.ts index 5c557087c8..302aa5fd25 100644 --- a/packages/db/tests/effect.test.ts +++ b/packages/db/tests/effect.test.ts @@ -1,11 +1,15 @@ import { describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' import { Query, createEffect, createTransaction, eq } from '../src/index.js' import { mockSyncCollectionOptions, mockSyncCollectionOptionsNoInitialState, } from './utils.js' -import type { DeltaEvent } from '../src/index.js' +import type { + DeltaEvent, + SubscriptionLoadSubsetErrorEvent, +} from '../src/index.js' // --------------------------------------------------------------------------- // Test types and helpers @@ -644,6 +648,177 @@ describe(`createEffect`, () => { await effect.dispose() // Should not throw expect(effect.disposed).toBe(true) }) + + it(`joins cleanup that is already in progress`, async () => { + const users = createUsersCollection([sampleUsers[0]!]) + let resolveHandler!: () => void + const handlerPending = new Promise((resolve) => { + resolveHandler = resolve + }) + const effect = createEffect({ + query: (q) => q.from({ user: users }), + onEnter: () => handlerPending, + }) + + await flushPromises() + const firstDispose = effect.dispose() + let secondDisposeSettled = false + const secondDispose = effect.dispose().finally(() => { + secondDisposeSettled = true + }) + + await Promise.resolve() + expect(secondDisposeSettled).toBe(false) + + resolveHandler() + await Promise.all([firstDispose, secondDispose]) + expect(secondDisposeSettled).toBe(true) + }) + + it(`reports one in-progress cleanup failure to every disposer`, async () => { + const failure = new Error(`source release failed`) + let resolveHandler!: () => void + const handlerPending = new Promise((resolve) => { + resolveHandler = resolve + }) + const source = createCollection<{ id: number }>({ + id: `joined-effect-cleanup-error`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: () => { + begin() + write({ type: `insert`, value: { id: 1 } }) + commit() + return true + }, + unloadSubset: () => { + throw failure + }, + } + }, + }, + }) + const effect = createEffect({ + query: (q) => q.from({ source }), + onEnter: () => handlerPending, + }) + + await flushPromises() + const firstDispose = effect.dispose() + const secondDispose = effect.dispose() + resolveHandler() + + await expect(firstDispose).rejects.toBe(failure) + await expect(secondDispose).rejects.toBe(failure) + await source.cleanup() + }) + + it.each([ + { name: `undefined`, failure: undefined }, + { name: `null`, failure: null }, + { name: `false`, failure: false }, + { name: `zero`, failure: 0 }, + { name: `empty string`, failure: `` }, + { name: `NaN`, failure: Number.NaN }, + ])( + `reports a falsy cleanup failure once: $name`, + async ({ name, failure }) => { + let unloadCount = 0 + const source = createCollection<{ id: number }>({ + id: `effect-falsy-cleanup-${name}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloadCount++ + if (unloadCount === 1) throw failure + }, + } + }, + }, + }) + const effect = createEffect({ + query: (q) => q.from({ source }), + onBatch: () => {}, + }) + + try { + await flushPromises() + let didReject = false + let rejection: unknown + try { + await effect.dispose() + } catch (error) { + didReject = true + rejection = error + } + expect(didReject).toBe(true) + expect(rejection).toBeInstanceOf(Error) + expect((rejection as Error).message).toBe(String(failure)) + expect(unloadCount).toBe(1) + + await effect.dispose() + expect(unloadCount).toBe(1) + } finally { + await effect.dispose() + await source.cleanup() + } + }, + ) + + it(`does not repeat a failed source release across reentrant disposal`, async () => { + const failure = new Error(`outer source release failed`) + let unloadCount = 0 + const source = createCollection<{ id: number }>({ + id: `effect-reentrant-cleanup-error`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloadCount++ + if (unloadCount === 1) { + void effect.dispose() + throw failure + } + }, + } + }, + }, + }) + const effect = createEffect({ + query: (q) => q.from({ source }), + onBatch: () => {}, + }) + + try { + await flushPromises() + await expect(effect.dispose()).rejects.toBe(failure) + // Reentrant disposal cannot repeat an unload still on the stack. + expect(unloadCount).toBe(1) + expect(source.subscriberCount).toBe(0) + + await effect.dispose() + // Finishing the failed attempt does not make the lease retryable. + expect(unloadCount).toBe(1) + await effect.dispose() + expect(unloadCount).toBe(1) + } finally { + await effect.dispose() + await source.cleanup() + } + }) }) describe(`auto-generated IDs`, () => { @@ -1434,9 +1609,695 @@ describe(`createEffect`, () => { await effect.dispose() }) + + it(`loads each ordered sibling subquery through its own source`, async () => { + const left = createCollection( + mockSyncCollectionOptions({ + id: `ordered-sibling-left`, + getKey: (user) => user.id, + initialData: [ + { id: 1, name: `Alice`, active: false }, + { id: 2, name: `Bob`, active: true }, + ], + autoIndex: `eager`, + }), + ) + const right = createCollection( + mockSyncCollectionOptions({ + id: `ordered-sibling-right`, + getKey: (user) => user.id, + initialData: [ + { id: 1, name: `Amy`, active: false }, + { id: 2, name: `Bea`, active: true }, + ], + autoIndex: `eager`, + }), + ) + type JoinedActiveUser = { + id: number + leftName: string + rightName: string + } + const events: Array> = [] + const effect = createEffect({ + query: (q) => { + const leftTop = q + .from({ item: left }) + .where(({ item }) => eq(item.active, true)) + .orderBy(({ item }) => item.name, `asc`) + .limit(1) + const rightTop = q + .from({ item: right }) + .where(({ item }) => eq(item.active, true)) + .orderBy(({ item }) => item.name, `asc`) + .limit(1) + + return q + .from({ left: leftTop }) + .join({ right: rightTop }, ({ left: leftRow, right: rightRow }) => + eq(leftRow.id, rightRow.id), + ) + .select(({ left: leftRow, right: rightRow }) => ({ + id: leftRow.id, + leftName: leftRow.name, + rightName: rightRow.name, + })) + }, + onEnter: (event) => { + events.push(event) + }, + }) + + try { + await flushPromises() + expect(events.map(({ value }) => value)).toEqual([ + { id: 2, leftName: `Bob`, rightName: `Bea` }, + ]) + } finally { + await effect.dispose() + await Promise.all([left.cleanup(), right.cleanup()]) + } + }) }) describe(`source error handling`, () => { + it(`does not subscribe later sources after startup disposes the effect`, async () => { + const failure = new Error(`synchronous source failure`) + const users = createUsersCollection([sampleUsers[0]!]) + const issues = createIssuesCollection([sampleIssues[0]!]) + const subscribeChanges = users.subscribeChanges.bind(users) + + vi.spyOn(users, `subscribeChanges`).mockImplementation((( + callback, + options, + ) => { + const subscription = subscribeChanges(callback, { + ...options, + includeInitialState: false, + }) + const errorEvent: SubscriptionLoadSubsetErrorEvent = { + type: `loadSubset:error`, + subscription, + options: { subscription }, + error: failure, + } + options?.onLoadSubsetError?.(errorEvent) + return subscription + }) as typeof users.subscribeChanges) + + const sourceErrors: Array = [] + const effect = createEffect({ + query: (q) => + q + .from({ user: users }) + .leftJoin({ issue: issues }, ({ user, issue }) => + eq(user.id, issue.userId), + ) + .select(({ user, issue }) => ({ + id: user.id, + issueId: issue.id, + })), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }) + + expect(sourceErrors).toEqual([failure]) + expect(effect.disposed).toBe(true) + expect(users.subscriberCount).toBe(0) + expect(issues.subscriberCount).toBe(0) + await Promise.all([users.cleanup(), issues.cleanup()]) + }) + + it(`releases every source when one unsubscriber throws`, async () => { + const failure = new Error(`first source unload failed`) + const createSource = (id: string, unloadSubset: () => void) => + createCollection<{ id: number }>({ + id, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset, + } + }, + }, + }) + const left = createSource(`effect-cleanup-left`, () => { + throw failure + }) + const right = createSource(`effect-cleanup-right`, () => {}) + const effect = createEffect({ + query: (q) => + q + .from({ left }) + .leftJoin({ right }, ({ left: leftRow, right: rightRow }) => + eq(leftRow.id, rightRow.id), + ), + onBatch: () => {}, + }) + + expect(left.subscriberCount).toBe(1) + expect(right.subscriberCount).toBe(1) + + await expect(effect.dispose()).rejects.toBe(failure) + expect(left.subscriberCount).toBe(0) + expect(right.subscriberCount).toBe(0) + + await Promise.all([left.cleanup(), right.cleanup()]) + }) + + it(`preserves a startup error when cleanup also fails`, async () => { + const startupFailure = new Error(`second source failed to subscribe`) + const cleanupFailure = new Error(`first source failed to unload`) + const left = createCollection<{ id: number }>({ + id: `effect-startup-error-left`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + throw cleanupFailure + }, + } + }, + }, + }) + const right = createCollection<{ id: number }>({ + id: `effect-startup-error-right`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset: () => true } + }, + }, + }) + vi.spyOn(right, `subscribeChanges`).mockImplementation(() => { + throw startupFailure + }) + const consoleErrorSpy = vi + .spyOn(console, `error`) + .mockImplementation(() => {}) + + try { + expect(() => + createEffect({ + query: (q) => + q + .from({ left }) + .leftJoin({ right }, ({ left: leftRow, right: rightRow }) => + eq(leftRow.id, rightRow.id), + ), + onBatch: () => {}, + }), + ).toThrow(startupFailure) + expect(consoleErrorSpy).toHaveBeenCalledWith( + expect.stringContaining(`failed to dispose after a startup error`), + cleanupFailure, + ) + expect(left.subscriberCount).toBe(0) + expect(right.subscriberCount).toBe(0) + } finally { + consoleErrorSpy.mockRestore() + await Promise.all([left.cleanup(), right.cleanup()]) + } + }) + + it(`releases source ownership when the automatic subset load throws`, async () => { + const failure = new Error(`automatic subset failed`) + const users = createCollection({ + id: `effect-synchronous-subset-error`, + getKey: (user) => user.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + throw failure + }, + } + }, + }, + }) + const sourceErrors: Array = [] + + expect(() => + createEffect({ + query: (q) => q.from({ user: users }), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }), + ).toThrow(failure) + + expect(sourceErrors).toEqual([failure]) + expect(users.subscriberCount).toBe(0) + await users.cleanup() + }) + + it(`releases an ordered source when its initial subset load throws`, async () => { + const failure = new Error(`initial ordered subset failed`) + const users = createCollection({ + id: `effect-initial-ordered-subset-error`, + getKey: (user) => user.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + throw failure + }, + } + }, + }, + }) + const sourceErrors: Array = [] + + expect(() => + createEffect({ + query: (q) => + q + .from({ user: users }) + .orderBy(({ user }) => user.name, `asc`) + .limit(1), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }), + ).toThrow(failure) + + expect(sourceErrors).toEqual([failure]) + expect(users.subscriberCount).toBe(0) + await users.cleanup() + }) + + it(`releases every source when initial lazy demand throws`, async () => { + const failure = new Error(`initial lazy demand failed`) + const users = createUsersCollection([sampleUsers[0]!]) + const issues = createCollection({ + id: `effect-initial-lazy-subset-error`, + getKey: (issue) => issue.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + throw failure + }, + } + }, + }, + }) + const sourceErrors: Array = [] + + expect(() => + createEffect({ + query: (q) => + q + .from({ user: users }) + .leftJoin({ issue: issues }, ({ user, issue }) => + eq(user.id, issue.userId), + ) + .select(({ user, issue }) => ({ + id: user.id, + issueId: issue.id, + })), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }), + ).toThrow(failure) + + expect(sourceErrors).toEqual([failure]) + expect(users.subscriberCount).toBe(0) + expect(issues.subscriberCount).toBe(0) + await Promise.all([users.cleanup(), issues.cleanup()]) + }) + + it(`isolates synchronous lazy-demand failure from an established source commit`, async () => { + const failure = new Error(`incremental effect lazy demand failed`) + const users = createCollection( + mockSyncCollectionOptions({ + id: `incremental-effect-users`, + getKey: (user) => user.id, + initialData: [], + }), + ) + const issues = createCollection({ + id: `incremental-effect-issues`, + getKey: (issue) => issue.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + throw failure + }, + } + }, + }, + }) + const sourceErrors: Array = [] + const effect = createEffect({ + query: (q) => + q + .from({ user: users }) + .leftJoin({ issue: issues }, ({ user, issue }) => + eq(user.id, issue.userId), + ) + .select(({ user, issue }) => ({ + id: user.id, + issueId: issue.id, + })), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }) + + try { + let commitError: unknown + try { + users.utils.begin() + users.utils.write({ type: `insert`, value: sampleUsers[0]! }) + users.utils.commit() + } catch (error) { + commitError = error + } + + expect(commitError).toBeUndefined() + expect(sourceErrors).toEqual([failure]) + expect(effect.disposed).toBe(true) + } finally { + await effect.dispose() + await Promise.all([users.cleanup(), issues.cleanup()]) + } + }) + + it(`reports failed obsolete-demand release without failing the source commit`, async () => { + const failure = new Error(`obsolete effect demand release failed`) + const users = createCollection( + mockSyncCollectionOptions({ + id: `effect-obsolete-release-users`, + getKey: (user) => user.id, + initialData: [sampleUsers[0]!], + }), + ) + let loadCount = 0 + let unloadCount = 0 + const consoleError = vi + .spyOn(console, `error`) + .mockImplementation(() => {}) + const issues = createCollection({ + id: `effect-obsolete-release-issues`, + getKey: (issue) => issue.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loadCount++ + return true + }, + unloadSubset: () => { + unloadCount++ + if (unloadCount <= 2) throw failure + }, + } + }, + }, + }) + const sourceErrors: Array = [] + const effect = createEffect({ + query: (q) => + q + .from({ user: users }) + .leftJoin({ issue: issues }, ({ user, issue }) => + eq(user.id, issue.userId), + ) + .select(({ user, issue }) => ({ + id: user.id, + issueId: issue.id, + })), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }) + + try { + await flushPromises() + expect(loadCount).toBe(1) + + expect(() => { + users.utils.begin() + users.utils.write({ type: `delete`, value: sampleUsers[0]! }) + users.utils.commit() + }).not.toThrow() + await flushPromises() + + expect(sourceErrors).toEqual([failure]) + expect(effect.disposed).toBe(true) + expect(unloadCount).toBe(1) + + await effect.dispose() + expect(unloadCount).toBe(1) + } finally { + await effect.dispose() + await Promise.all([users.cleanup(), issues.cleanup()]) + consoleError.mockRestore() + } + }) + + it(`reports a rejected ordered subset load and disposes the effect`, async () => { + const failure = new Error(`ordered subset failed`) + let loadCount = 0 + const users = createCollection({ + id: `effect-rejected-ordered-users`, + getKey: (user) => user.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount > 1) return Promise.reject(failure) + begin() + write({ type: `insert`, value: sampleUsers[0]! }) + commit() + return Promise.resolve() + }, + } + }, + }, + }) + const sourceErrors: Array = [] + const effect = createEffect({ + query: (q) => + q + .from({ user: users }) + .orderBy(({ user }) => user.name, `asc`) + .limit(1), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }) + + try { + await flushPromises() + expect(sourceErrors).toEqual([failure]) + expect(effect.disposed).toBe(true) + } finally { + await effect.dispose() + await users.cleanup() + } + }) + + it(`reports a cleanup failure from automatic disposal`, async () => { + const loadFailure = new Error(`ordered subset failed`) + const cleanupFailure = new Error(`ordered subset cleanup failed`) + let loadCount = 0 + let removeVisibleRow: () => void = () => { + throw new Error(`source has not started`) + } + const users = createCollection({ + id: `effect-rejected-ordered-cleanup-users`, + getKey: (user) => user.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + removeVisibleRow = () => { + begin() + write({ type: `delete`, value: sampleUsers[0]! }) + commit() + } + return { + loadSubset: () => { + loadCount++ + if (loadCount > 1) return Promise.reject(loadFailure) + begin() + write({ type: `insert`, value: sampleUsers[0]! }) + commit() + return Promise.resolve() + }, + unloadSubset: () => { + throw cleanupFailure + }, + } + }, + }, + }) + const sourceErrors: Array = [] + const consoleErrorSpy = vi + .spyOn(console, `error`) + .mockImplementation(() => {}) + const effect = createEffect({ + query: (q) => + q + .from({ user: users }) + .orderBy(({ user }) => user.name, `asc`) + .limit(1), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }) + + try { + await flushPromises() + removeVisibleRow() + await flushPromises() + + expect(sourceErrors).toEqual([loadFailure]) + expect(effect.disposed).toBe(true) + expect(consoleErrorSpy).toHaveBeenCalledWith( + expect.stringContaining(`failed to dispose after a source error`), + cleanupFailure, + ) + } finally { + await expect(effect.dispose()).resolves.toBeUndefined() + consoleErrorSpy.mockRestore() + await users.cleanup() + } + }) + + it(`reports a rejected lazy subset load and disposes the effect`, async () => { + const users = createUsersCollection([sampleUsers[0]!]) + const issues = createCollection({ + id: `effect-rejected-lazy-issues`, + getKey: (issue) => issue.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: () => ({ + loadSubset: () => Promise.reject(new Error(`lazy load failed`)), + }), + }, + }) + const sourceErrors: Array = [] + const effect = createEffect({ + query: (q) => + q + .from({ user: users }) + .leftJoin({ issue: issues }, ({ user, issue }) => + eq(user.id, issue.userId), + ) + .select(({ user, issue }) => ({ + id: user.id, + issueId: issue.id, + })), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }) + + try { + await flushPromises() + expect(sourceErrors).toEqual([ + expect.objectContaining({ message: `lazy load failed` }), + ]) + expect(effect.disposed).toBe(true) + } finally { + await effect.dispose() + await Promise.all([users.cleanup(), issues.cleanup()]) + } + }) + + it(`keeps the effect alive when obsolete lazy demand is aborted`, async () => { + const users = createUsersCollection([sampleUsers[0]!]) + const cancellation = new Error(`obsolete lazy demand`) + cancellation.name = `AbortError` + let capturedSignal: AbortSignal | undefined + const issues = createCollection({ + id: `effect-aborted-lazy-issues`, + getKey: (issue) => issue.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: ({ signal }) => { + capturedSignal = signal + return new Promise((_resolve, reject) => { + signal?.addEventListener( + `abort`, + () => reject(cancellation), + { once: true }, + ) + }) + }, + } + }, + }, + }) + const sourceErrors: Array = [] + const effect = createEffect({ + query: (q) => + q + .from({ user: users }) + .leftJoin({ issue: issues }, ({ user, issue }) => + eq(user.id, issue.userId), + ) + .select(({ user, issue }) => ({ + id: user.id, + issueId: issue.id, + })), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }) + + try { + await flushPromises() + expect(capturedSignal?.aborted).toBe(false) + + users.utils.begin() + users.utils.write({ type: `delete`, value: sampleUsers[0]! }) + users.utils.commit() + await flushPromises() + + expect(capturedSignal?.aborted).toBe(true) + expect(sourceErrors).toEqual([]) + expect(effect.disposed).toBe(false) + } finally { + await effect.dispose() + await Promise.all([users.cleanup(), issues.cleanup()]) + } + }) + it(`should auto-dispose when source collection is cleaned up`, async () => { const users = createUsersCollection() const events: Array> = [] @@ -1571,6 +2432,101 @@ describe(`createEffect`, () => { await effect.dispose() }) + it.each( + ([`projection`, `delivery`] as const).flatMap((phase) => + [false, true].map((throwRelease) => ({ phase, throwRelease })), + ), + )( + `isolates in-turn disposal and nested publication: %j`, + async ({ phase, throwRelease }) => { + const users = createUsersCollection([]) + const issues = createIssuesCollection([]) + const peerEvents: Array> = [] + const peer = createEffect({ + query: (q) => q.from({ user: users }), + onEnter: (event) => { + peerEvents.push(event) + }, + }) + const failure = new Error(`unsubscribe failed after releasing`) + let shouldThrow = throwRelease + const subscribe = users.subscribeChanges.bind(users) + vi.spyOn(users, `subscribeChanges`).mockImplementation((...args) => { + const subscription = subscribe(...args) + const unsubscribe = subscription.unsubscribe.bind(subscription) + vi.spyOn(subscription, `unsubscribe`).mockImplementation(() => { + unsubscribe() + if (shouldThrow) { + shouldThrow = false + throw failure + } + }) + return subscription + }) + const events: Array> = [] + let disposeInTurn: (() => void) | undefined + let outcome: Promise | undefined + const effect = createEffect({ + query: (q) => + q + .from({ user: users }) + .leftJoin({ issue: issues }, ({ user, issue }) => + eq(user.id, issue.userId), + ) + .fn.select(({ user }) => { + if (phase === `projection`) disposeInTurn?.() + return user + }), + onEnter: (event) => { + events.push(event) + if (phase === `delivery`) disposeInTurn?.() + }, + }) + try { + await flushPromises() + expect(users.subscriberCount).toBe(2) + expect(issues.subscriberCount).toBe(1) + disposeInTurn = () => { + disposeInTurn = undefined + outcome = effect.dispose().then( + () => ({ status: `fulfilled` }), + (error: unknown) => ({ status: `rejected`, reason: error }), + ) + users.utils.begin() + users.utils.write({ + type: `insert`, + value: { id: 3, name: `Nested`, active: true }, + }) + users.utils.commit() + } + users.utils.begin() + for (const id of [1, 2]) { + users.utils.write({ + type: `insert`, + value: { id, name: `User ${id}`, active: true }, + }) + } + users.utils.commit() + await flushPromises() + expect(outcome).toBeDefined() + expect(await outcome).toEqual( + throwRelease + ? { status: `rejected`, reason: failure } + : { status: `fulfilled` }, + ) + expect(effect.disposed).toBe(true) + expect(events).toHaveLength(phase === `projection` ? 0 : 1) + expect(peerEvents.map(({ key }) => key).sort()).toEqual([1, 2, 3]) + expect(users.subscriberCount).toBe(1) + expect(issues.subscriberCount).toBe(0) + } finally { + await effect.dispose() + await peer.dispose() + await Promise.all([users.cleanup(), issues.cleanup()]) + } + }, + ) + it(`disposing inside handler should not throw and should stop further events`, async () => { const users = createUsersCollection() const events: Array> = [] diff --git a/packages/db/tests/expected-failure.test.ts b/packages/db/tests/expected-failure.test.ts new file mode 100644 index 0000000000..deb9bd092c --- /dev/null +++ b/packages/db/tests/expected-failure.test.ts @@ -0,0 +1,86 @@ +import { describe, expect, it } from 'vitest' +import { expectAssertionFailure } from './expected-failure.js' +import { TraceAssertionError } from './trace-runner.js' + +function assertionMismatch(checkpoint: number): Promise { + try { + expect(`observed`).toBe(`expected`) + return Promise.resolve() + } catch (error) { + return Promise.reject(new TraceAssertionError(checkpoint, error)) + } +} + +describe(`expected failure guard`, () => { + it(`accepts an assertion mismatch at the expected checkpoint`, async () => { + const guarded = expectAssertionFailure(() => assertionMismatch(2), { + checkpoint: 2, + }) + + await guarded() + }) + + it(`rejects an assertion mismatch from the wrong checkpoint`, async () => { + const guarded = expectAssertionFailure( + () => + Promise.reject( + new TraceAssertionError(0, new Error(`startup mismatch`)), + ), + { checkpoint: 2 }, + ) + + await expect(guarded()).rejects.toBeInstanceOf(Error) + }) + + it(`rejects a runtime error from the expected checkpoint`, async () => { + const guarded = expectAssertionFailure( + () => + Promise.reject( + new TraceAssertionError(2, new TypeError(`projection failed`)), + ), + { checkpoint: 2 }, + ) + + await expect(guarded()).rejects.toBeInstanceOf(Error) + }) + + it(`accepts an assertion mismatch with the expected difference`, async () => { + const guarded = expectAssertionFailure(() => assertionMismatch(2), { + checkpoint: 2, + classify: ({ actual, expected }) => + actual === `observed` && expected === `expected`, + }) + + await guarded() + }) + + it(`rejects an assertion mismatch with a different shape`, async () => { + const guarded = expectAssertionFailure(() => assertionMismatch(2), { + checkpoint: 2, + classify: ({ actual }) => actual === `different`, + }) + + await expect(guarded()).rejects.toBeInstanceOf(Error) + }) + + it(`accepts an assertion mismatch with the expected message`, async () => { + const guarded = expectAssertionFailure( + () => + Promise.resolve().then(() => { + expect([`actual`]).toEqual([`expected`]) + }), + { message: /expected/ }, + ) + + await guarded() + }) + + it(`rejects runtime errors that happen to have the expected message`, async () => { + const runtimeError = new TypeError(`expected value is missing`) + const guarded = expectAssertionFailure(() => Promise.reject(runtimeError), { + message: /expected/, + }) + + await expect(guarded()).rejects.toBeInstanceOf(Error) + }) +}) diff --git a/packages/db/tests/expected-failure.ts b/packages/db/tests/expected-failure.ts new file mode 100644 index 0000000000..f5a437f0c3 --- /dev/null +++ b/packages/db/tests/expected-failure.ts @@ -0,0 +1,66 @@ +import { expect } from 'vitest' + +export type AssertionDifference = { + actual: unknown + expected: unknown +} + +type ExpectedAssertionFailure = + | { + checkpoint: number + classify?: (difference: AssertionDifference) => boolean + } + | { message: string | RegExp } + +function assertionDifference(error: unknown): AssertionDifference { + if ( + typeof error !== `object` || + error === null || + !(`cause` in error) || + typeof error.cause !== `object` || + error.cause === null || + !(`actual` in error.cause) || + !(`expected` in error.cause) + ) { + throw new Error(`Expected an assertion difference`) + } + + return { + actual: error.cause.actual, + expected: error.cause.expected, + } +} + +export function expectAssertionFailure>( + assertion: (...args: TArgs) => Promise, + expected: ExpectedAssertionFailure, +): (...args: TArgs) => Promise { + return async (...args) => { + if (`checkpoint` in expected) { + let error: unknown + try { + await assertion(...args) + } catch (caught) { + error = caught + } + + expect(error).toMatchObject({ + name: `TraceAssertionError`, + checkpoint: expected.checkpoint, + cause: { name: `AssertionError` }, + }) + if (expected.classify) { + expect(expected.classify(assertionDifference(error))).toBe(true) + } + return + } + + await expect(assertion(...args)).rejects.toMatchObject({ + name: `AssertionError`, + message: + typeof expected.message === `string` + ? expected.message + : expect.stringMatching(expected.message), + }) + } +} diff --git a/packages/db/tests/facade-retention.probe.ts b/packages/db/tests/facade-retention.probe.ts new file mode 100644 index 0000000000..fa45d249b0 --- /dev/null +++ b/packages/db/tests/facade-retention.probe.ts @@ -0,0 +1,117 @@ +// Run manually: node --expose-gc --import tsx tests/facade-retention.probe.ts +// This probes reachability, not total application heap size or GC latency. +import assert from 'node:assert/strict' +import { setImmediate } from 'node:timers/promises' +import { D2, MultiSet } from '@tanstack/db-ivm' +import { BucketFacadeAdapter } from '../src/query/live/bucket-facade-adapter.js' +import { BUCKET_FACADE_REF } from '../src/query/live/materialized-pipeline.js' +import type { Collection } from '../src/collection/index.js' +import type { + BucketFacadeRef, + BucketRow, +} from '../src/query/live/materialized-pipeline.js' + +const gc = globalThis.gc +if (!gc) throw new Error(`Run this probe with --expose-gc`) + +function capture( + released: boolean, + holder: `view` | `method`, + pendingUpdate: boolean, +) { + const graph = new D2() + const rows = graph.newInput<[string, BucketRow]>() + const activeBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `retention-probe`, + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: false }], + () => {}, + ) + graph.finalize() + const value = { id: 1, payload: new ArrayBuffer(1024 * 1024) } + const bucketKey = `group` + activeBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + rows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { + publicKey: 1, + value, + order: undefined, + }, + ], + 1, + ], + ]), + ) + graph.run() + const ref: BucketFacadeRef = { + [BUCKET_FACADE_REF]: { edgeId: `children`, bucketKey }, + } + adapter.flush().publish() + const view = adapter.resolve(ref) as unknown as Collection< + typeof value, + number + > + assert.equal(view.get(1)?.id, 1) + const retained = holder === `view` ? view : view.get.bind(view) + if (pendingUpdate) { + rows.sendData( + new MultiSet([ + [[bucketKey, { publicKey: 2, value: { id: 2 }, order: undefined }], 1], + ]), + ) + graph.run() + } + if (released) adapter.cleanup() + return { retained, value: new WeakRef(value), adapter: new WeakRef(adapter) } +} + +const cells = [false, true].flatMap((released) => + ([`view`, `method`] as const).flatMap((holder) => + [false, true].map((pendingUpdate) => ({ released, holder, pendingUpdate })), + ), +) +const results = cells.map((cell) => ({ + ...cell, + samples: Array.from({ length: 10 }, () => + capture(cell.released, cell.holder, cell.pendingUpdate), + ), +})) + +// WeakRef targets stay alive through the creating job. Cross job boundaries +// before forcing collection and avoid dereferencing targets inside this loop. +for (let turn = 0; turn < 5; turn++) { + await setImmediate() + gc() +} +await setImmediate() + +const report = results.map(({ released, holder, pendingUpdate, samples }) => { + const retainedValues = samples.filter( + (sample) => sample.value.deref() !== undefined, + ).length + const retainedAdapters = samples.filter( + (sample) => sample.adapter.deref() !== undefined, + ).length + // Live public facades are the positive control: this probe must detect them. + assert.equal(retainedValues, released ? 0 : samples.length) + assert.equal(retainedAdapters, 0) + assert.equal(samples.length, 10) + // Keep each public handle or captured method observably reachable to the end. + for (const { retained } of samples) { + const row = typeof retained === `function` ? retained(1) : retained.get(1) + assert.equal(row?.id, released ? undefined : 1) + } + return { + released, + holder, + pendingUpdate, + samples: samples.length, + retainedValues, + retainedAdapters, + } +}) +console.log(JSON.stringify({ node: process.version, report }, null, 2)) diff --git a/packages/db/tests/gc-process-exit.test.ts b/packages/db/tests/gc-process-exit.test.ts new file mode 100644 index 0000000000..43bdacda5d --- /dev/null +++ b/packages/db/tests/gc-process-exit.test.ts @@ -0,0 +1,34 @@ +// @vitest-environment node +import { execFile } from 'node:child_process' +import { fileURLToPath } from 'node:url' +import { promisify } from 'node:util' +import { expect, it } from 'vitest' + +it(`allows Node to exit with unused eagerly synced collections`, async () => { + const packageRoot = fileURLToPath(new URL(`..`, import.meta.url)) + const { stdout } = await promisify(execFile)( + process.execPath, + [ + `--import`, + `tsx`, + `--input-type=module`, + `-e`, + `import { createCollection } from './src/collection/index.ts'; + createCollection({ + getKey: row => row.id, + startSync: true, + sync: { sync: ({ markReady }) => markReady() }, + }); + console.log('finished');`, + ], + { + cwd: packageRoot, + env: { + ...process.env, + TSX_TSCONFIG_PATH: `${packageRoot}/tsconfig.json`, + }, + timeout: 15000, + }, + ) + expect(stdout.trim()).toBe(`finished`) +}, 20000) diff --git a/packages/db/tests/get-key-query-planning.test.ts b/packages/db/tests/get-key-query-planning.test.ts new file mode 100644 index 0000000000..728e427725 --- /dev/null +++ b/packages/db/tests/get-key-query-planning.test.ts @@ -0,0 +1,50 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { currentStateAsChanges } from '../src/collection/change-events.js' +import { Func, PropRef, Value } from '../src/query/ir.js' + +describe(`getKey query planning`, () => { + it(`does not treat arbitrary getKey code as query field metadata`, async () => { + type Row = { id: string; fallback: string } + let getKeyCalls = 0 + const collection = createCollection({ + id: `conditional-get-key-query-planning`, + getKey: (row) => { + getKeyCalls++ + return row.id === `special` ? row.fallback : row.id + }, + autoIndex: `off`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { id: `special`, fallback: `actual-key` }, + }) + commit() + markReady() + }, + }, + }) + + try { + await collection.stateWhenReady() + const callsAfterSync = getKeyCalls + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`special`)]) + + expect( + currentStateAsChanges(collection, { where, optimizedOnly: true }), + ).toBeUndefined() + expect(getKeyCalls).toBe(callsAfterSync) + expect( + currentStateAsChanges(collection, { where })?.map( + (change) => change.key, + ), + ).toEqual([`actual-key`]) + expect(getKeyCalls).toBe(callsAfterSync) + } finally { + await collection.cleanup() + } + }) +}) diff --git a/packages/db/tests/index-domain-recovery-work.test.ts b/packages/db/tests/index-domain-recovery-work.test.ts new file mode 100644 index 0000000000..74d2436364 --- /dev/null +++ b/packages/db/tests/index-domain-recovery-work.test.ts @@ -0,0 +1,91 @@ +import { expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { BasicIndex } from '../src/indexes/basic-index.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import type { SyncConfig } from '../src/types.js' + +type Row = { id: number; value: number | Array } + +it.each( + [BasicIndex, BTreeIndex].flatMap((IndexType) => + [100, 10000].flatMap((size) => + ([`delete`, `update`] as const).map((retirement) => ({ + name: IndexType.name, + IndexType, + size, + retirement, + })), + ), + ), +)( + `$name restores range lookup after $retirement in $size rows`, + async ({ IndexType, size, retirement }) => { + let sync!: Parameters[`sync`]>[0] + const collection = createCollection({ + getKey: (row) => row.id, + autoIndex: `off`, + sync: { + sync: (context) => { + sync = context + context.begin() + for (let id = 0; id < size; id++) { + context.write({ type: `insert`, value: { id, value: id } }) + } + context.commit() + context.markReady() + }, + }, + }) + await collection.preload() + const index = collection.createIndex((row) => row.value, { + indexType: IndexType, + }) + const entries = collection.entries.bind(collection) + let scanned = 0 + const spy = vi + .spyOn(collection, `entries`) + .mockImplementation(function* () { + for (const entry of entries()) { + scanned++ + yield entry + } + }) + const where = new Func(`gt`, [ + new PropRef([`value`]), + new Value(size - 2), + ]) + const visits: Array = [] + const read = (expected: Array, repeats = 1) => { + scanned = 0 + for (let i = 0; i < repeats; i++) { + expect( + collection + .currentStateAsChanges({ where }) + ?.map(({ key }) => key) + .sort((a, b) => Number(a) - Number(b)), + ).toEqual(expected) + } + visits.push(scanned) + } + try { + read([size - 1]) + sync.begin() + sync.write({ type: `insert`, value: { id: size, value: [size + 5] } }) + sync.commit() + read([size - 1, size]) + sync.begin() + if (retirement === `delete`) sync.write({ type: `delete`, key: size }) + else sync.write({ type: `update`, value: { id: size, value: 0 } }) + sync.commit() + read([size - 1], 3) + index.build(entries()) + read([size - 1]) + // The transient foreign domain must not leave every future snapshot scanning. + expect(visits).toEqual([0, size + 1, 0, 0]) + } finally { + spy.mockRestore() + await collection.cleanup() + } + }, +) diff --git a/packages/db/tests/index-reader.test-d.ts b/packages/db/tests/index-reader.test-d.ts new file mode 100644 index 0000000000..88ea904bc4 --- /dev/null +++ b/packages/db/tests/index-reader.test-d.ts @@ -0,0 +1,13 @@ +import { expectTypeOf, test } from 'vitest' +import type { + IndexReader, + ReverseIndex, + findIndexForField, +} from '../src/index.js' + +test(`resolved indexes expose a named read interface`, () => { + expectTypeOf>().toEqualTypeOf< + IndexReader | undefined + >() + expectTypeOf>().toMatchTypeOf>() +}) diff --git a/packages/db/tests/index-update-short-circuit.test.ts b/packages/db/tests/index-update-short-circuit.test.ts new file mode 100644 index 0000000000..c23d9ae2c7 --- /dev/null +++ b/packages/db/tests/index-update-short-circuit.test.ts @@ -0,0 +1,147 @@ +import { describe, expect, it, vi } from 'vitest' +import { BasicIndex } from '../src/indexes/basic-index.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { PropRef } from '../src/query/ir.js' +import { normalizeValue } from '../src/utils/comparison.js' +import { valueMapData } from './utils' +import type { BaseIndex } from '../src/indexes/base-index.js' + +type IndexConstructor = new ( + id: number, + expression: PropRef, + name?: string, + options?: unknown, +) => BaseIndex + +const indexTypes: Array<[string, IndexConstructor]> = [ + [`BasicIndex`, BasicIndex as IndexConstructor], + [`BTreeIndex`, BTreeIndex as IndexConstructor], +] + +describe.each(indexTypes)(`%s update`, (_indexName, IndexType) => { + function createIndex(options?: unknown) { + return new IndexType(1, new PropRef([`value`]), `test_index`, options) + } + + it(`looks up rows without collecting timing diagnostics`, () => { + const index = createIndex() + index.add(`a`, { value: 1 }) + const now = vi.spyOn(performance, `now`) + try { + expect(index.lookup(`eq`, 1)).toEqual(new Set([`a`])) + expect(index.lookup(`in`, [1, 2])).toEqual(new Set([`a`])) + expect(now).not.toHaveBeenCalled() + } finally { + now.mockRestore() + } + }) + + it(`keeps the existing bucket when the indexed value does not change`, () => { + const index = createIndex() + index.add(`a`, { value: 1, version: 1 }) + const bucket = valueMapData(index).get(1) + const add = vi.spyOn(index, `add`) + const remove = vi.spyOn(index, `remove`) + + index.update(`a`, { value: 1, version: 1 }, { value: 1, version: 2 }) + + expect(valueMapData(index).get(1)).toBe(bucket) + expect(add).not.toHaveBeenCalled() + expect(remove).not.toHaveBeenCalled() + expect(index.lookup(`eq`, 1)).toEqual(new Set([`a`])) + }) + + it.each([ + [`undefined`, undefined, undefined], + [`NaN`, Number.NaN, Number.NaN], + [`signed zero`, 0, -0], + [`equal dates`, new Date(1000), new Date(1000)], + [`equal byte arrays`, new Uint8Array([1, 2]), new Uint8Array([1, 2])], + ])(`keeps the existing bucket for %s`, (_caseName, oldValue, newValue) => { + const index = createIndex() + index.add(`a`, { value: oldValue }) + const bucket = valueMapData(index).get(normalizeValue(oldValue)) + expect(bucket).toBeDefined() + + index.update(`a`, { value: oldValue }, { value: newValue }) + + expect(valueMapData(index).get(normalizeValue(newValue))).toBe(bucket) + }) + + it(`moves the key when the indexed value changes`, () => { + const index = createIndex() + index.add(`a`, { value: 1 }) + + index.update(`a`, { value: 1 }, { value: 2 }) + + expect(index.lookup(`eq`, 1)).toEqual(new Set()) + expect(index.lookup(`eq`, 2)).toEqual(new Set([`a`])) + }) + + it(`does not conflate undefined and null`, () => { + const index = createIndex() + index.add(`a`, { value: undefined }) + + index.update(`a`, { value: undefined }, { value: null }) + + expect(index.lookup(`eq`, undefined)).toEqual(new Set()) + expect(index.lookup(`eq`, null)).toEqual(new Set([`a`])) + }) + + it(`does not use comparator equality to skip an update`, () => { + const index = createIndex({ + compareFn: (a: string, b: string) => + a.toLowerCase().localeCompare(b.toLowerCase()), + }) + index.add(`a`, { value: `A` }) + + index.update(`a`, { value: `A` }, { value: `a` }) + + expect(index.lookup(`eq`, `A`)).toEqual(new Set()) + expect(index.lookup(`eq`, `a`)).toEqual(new Set([`a`])) + }) + + it(`preserves the previous error behavior when evaluation fails`, () => { + const index = createIndex() + index.add(`a`, { value: 1 }) + const newItem = Object.defineProperty({}, `value`, { + get() { + throw new Error(`evaluation failed`) + }, + }) + + expect(() => index.update(`a`, { value: 1 }, newItem)).toThrow( + `evaluation failed`, + ) + + expect(index.lookup(`eq`, 1)).toEqual(new Set()) + expect(index.keyCount).toBe(0) + }) +}) + +describe(`BasicIndex update bookkeeping`, () => { + it(`repairs indexed key membership after a failed removal`, () => { + const index = new BasicIndex(1, new PropRef([`value`])) + index.add(`a`, { value: 1 }) + const itemWithThrowingValue = Object.defineProperty({}, `value`, { + get() { + throw new Error(`evaluation failed`) + }, + }) + const warn = vi.spyOn(console, `warn`).mockImplementation(() => {}) + + try { + index.remove(`a`, itemWithThrowingValue) + } finally { + warn.mockRestore() + } + + expect(index.lookup(`eq`, 1)).toEqual(new Set([`a`])) + expect(index.keyCount).toBe(0) + + index.update(`a`, { value: 1 }, { value: 1 }) + + expect(index.lookup(`eq`, 1)).toEqual(new Set([`a`])) + expect(index.keyCount).toBe(1) + }) +}) diff --git a/packages/db/tests/index-update.property.test.ts b/packages/db/tests/index-update.property.test.ts new file mode 100644 index 0000000000..501c940920 --- /dev/null +++ b/packages/db/tests/index-update.property.test.ts @@ -0,0 +1,369 @@ +import { describe, expect, expectTypeOf, test } from 'vitest' +import { fc, test as fcTest } from '@fast-check/vitest' +import { compareKeys } from '@tanstack/db-ivm' +import { BasicIndex } from '../src/indexes/basic-index.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { PropRef } from '../src/query/ir.js' +import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' +import { makeComparator } from '../src/utils/comparison.js' +import { indexedKeysSet, orderedEntriesArray, valueMapData } from './utils' +import type { BaseIndex, IndexInterface } from '../src/indexes/base-index.js' + +type IndexValue = number + +type IndexConstructor = new ( + id: number, + expression: PropRef, + name?: string, + options?: { + compareFn?: (left: unknown, right: unknown) => number + compareOptions?: typeof DEFAULT_COMPARE_OPTIONS + }, +) => BaseIndex + +type IndexAction = + | { type: `put`; key: string; value: IndexValue } + | { type: `delete`; key: string } + +const indexTypes: Array<[string, IndexConstructor]> = [ + [`BasicIndex`, BasicIndex as IndexConstructor], + [`BTreeIndex`, BTreeIndex as IndexConstructor], +] + +const arbitraryValue: fc.Arbitrary = fc.integer({ + min: -3, + max: 3, +}) + +const arbitraryAction: fc.Arbitrary = fc.oneof( + fc.record({ + type: fc.constant(`put` as const), + key: fc.integer({ min: 0, max: 7 }).map(String), + value: arbitraryValue, + }), + fc.record({ + type: fc.constant(`delete` as const), + key: fc.integer({ min: 0, max: 7 }).map(String), + }), +) + +const probeValues: Array = [-3, -2, -1, -0, 0, 1, 2, 3, 99] +const rangeBoundaries: Array = [-2, 0, 2] + +function groupKeysByValue( + rows: Map, +): Map> { + const groups = new Map>() + for (const [key, value] of rows) { + const keys = groups.get(value) + if (keys) { + keys.add(key) + } else { + groups.set(value, new Set([key])) + } + } + return groups +} + +function expectIndexMatchesModel( + index: BaseIndex, + rows: Map, +): void { + const groups = groupKeysByValue(rows) + + expect(index.keyCount).toBe(rows.size) + expect(indexedKeysSet(index)).toEqual(new Set(rows.keys())) + expect(valueMapData(index)).toEqual(groups) + expect(orderedEntriesArray(index)).toEqual( + [...groups].sort(([left], [right]) => left - right), + ) + + for (const value of probeValues) { + expect(index.lookup(`eq`, value)).toEqual(groups.get(value) ?? new Set()) + } + + for (const boundary of rangeBoundaries) { + const keysAtOrAbove = new Set( + [...rows].filter(([, value]) => value >= boundary).map(([key]) => key), + ) + const keysAtOrBelow = new Set( + [...rows].filter(([, value]) => value <= boundary).map(([key]) => key), + ) + + expect(index.rangeQuery({ from: boundary })).toEqual(keysAtOrAbove) + expect(index.rangeQuery({ to: boundary })).toEqual(keysAtOrBelow) + expect(index.rangeQueryReversed({ from: boundary })).toEqual(keysAtOrBelow) + expect(index.rangeQueryReversed({ to: boundary })).toEqual(keysAtOrAbove) + } + expect(index.rangeQueryReversed({})).toEqual(new Set(rows.keys())) +} + +describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { + fcTest.prop([ + fc.array(arbitraryAction, { + minLength: 1, + maxLength: 100, + }), + ])( + `matches a reference model across valid operation sequences`, + (actions) => { + const index = new IndexType(1, new PropRef([`value`])) + const rows = new Map() + + for (const action of actions) { + if (action.type === `put`) { + if (rows.has(action.key)) { + index.update( + action.key, + { value: rows.get(action.key) }, + { value: action.value }, + ) + } else { + index.add(action.key, { value: action.value }) + } + rows.set(action.key, action.value) + } else if (rows.has(action.key)) { + index.remove(action.key, { value: rows.get(action.key) }) + rows.delete(action.key) + } + + expectIndexMatchesModel(index, rows) + } + + const rebuilt = new IndexType(2, new PropRef([`value`])) + rebuilt.build([...rows].map(([key, value]) => [key, { value }] as const)) + expectIndexMatchesModel(rebuilt, rows) + }, + ) + + test(`tracks range-domain safety through updates, rebuilds, and clear`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const other = [20] + + index.add(`number`, { value: 50 }) + expect(index.canOptimizeRangeFor(100)).toBe(true) + + index.add(`other`, { value: other }) + expect(index.canOptimizeRangeFor(100)).toBe(false) + + index.update(`other`, { value: other }, { value: 20 }) + expect(index.canOptimizeRangeFor(100)).toBe(true) + + index.update(`number`, { value: 50 }, { value: new Date(50) }) + expect(index.canOptimizeRangeFor(100)).toBe(false) + index.remove(`other`, { value: 20 }) + expect(index.canOptimizeRangeFor(new Date(100))).toBe(true) + + index.clear() + expect(index.canOptimizeRangeFor(100)).toBe(true) + + index.build([ + [`number`, { value: 50 }], + [`other`, { value: [20] }], + ]) + expect(index.canOptimizeRangeFor(100)).toBe(false) + }) + + test(`accepts indexed values rather than row keys through the index interface`, () => { + const index: IndexInterface = new IndexType( + 1, + new PropRef([`value`]), + ) + expectTypeOf< + Parameters[`take`]>[1] + >().toEqualTypeOf() + expectTypeOf< + Parameters[`takeReversed`]>[1] + >().toEqualTypeOf() + expectTypeOf< + Parameters[`take`]>[1] + >().toEqualTypeOf() + expectTypeOf< + Parameters[`takeReversed`]>[1] + >().toEqualTypeOf() + index.add(`undefined`, { value: undefined }) + index.add(`zero`, { value: 0 }) + index.add(`one`, { value: 1 }) + + expect(index.take(3, 0)).toEqual([`one`]) + expect(index.takeReversed(3, 1)).toEqual([`zero`, `undefined`]) + expect(index.take(3, undefined)).toEqual([`zero`, `one`]) + expect(index.takeReversed(3, undefined)).toEqual([]) + }) + + test(`distinguishes explicit undefined range and cursor bounds`, () => { + const index = new IndexType(1, new PropRef([`value`])) + index.add(`undefined`, { value: undefined }) + index.add(`null`, { value: null }) + index.add(`one`, { value: 1 }) + + expect(index.rangeQuery({ to: undefined })).toEqual( + new Set([`undefined`, `null`]), + ) + expect(index.rangeQueryReversed({ from: undefined })).toEqual( + new Set([`undefined`, `null`]), + ) + expect(index.take(3, undefined)).toEqual([`one`]) + expect(index.takeReversed(3, undefined)).toEqual([]) + }) + + test(`executes the ordering advertised by compare options`, () => { + const compareOptions = { + ...DEFAULT_COMPARE_OPTIONS, + nulls: `last` as const, + stringSort: `lexical` as const, + } + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareOptions, + }) + index.add(`undefined`, { value: undefined }) + index.add(`null`, { value: null }) + index.add(`one`, { value: 1 }) + + expect(index.matchesCompareOptions(compareOptions)).toBe(true) + expect(index.takeFromStart(3)).toEqual([`one`, `null`, `undefined`]) + expect(index.rangeQuery({ to: 1 })).toEqual(new Set([`one`])) + }) +}) + +describe.each(indexTypes)(`%s comparator groups`, (_indexName, IndexType) => { + fcTest.prop([ + fc.array(fc.integer({ min: 0, max: 4 }), { + minLength: 2, + maxLength: 20, + }), + ])( + `preserves exact equality while ordered traversal retains every row`, + (groupIds) => { + const symbols = new Map() + const rows = groupIds.map((groupId, position) => { + const symbol = symbols.get(groupId) ?? Symbol(String(groupId)) + symbols.set(groupId, symbol) + return { + key: String(position), + value: [symbol], + groupId, + } + }) + const index = new IndexType(1, new PropRef([`value`])) + + const expectMatchesModel = ( + subject: BaseIndex, + currentRows: typeof rows, + ) => { + const groups = new Map() + for (const row of currentRows) { + const group = groups.get(row.groupId) ?? [] + group.push(row) + groups.set(row.groupId, group) + } + const compare = makeComparator(DEFAULT_COMPARE_OPTIONS) + const orderedGroups = [...groups.values()].sort((left, right) => + compare(left[0]!.value, right[0]!.value), + ) + const forward = orderedGroups.flatMap((group) => + group.map((row) => row.key).sort(compareKeys), + ) + const reversed = [...orderedGroups].reverse().flatMap((group) => + group + .map((row) => row.key) + .sort(compareKeys) + .reverse(), + ) + + expect(subject.takeFromStart(currentRows.length)).toEqual(forward) + expect(subject.takeReversedFromEnd(currentRows.length)).toEqual( + reversed, + ) + for (const [representative, keys] of orderedEntriesArray(subject)) { + expect( + currentRows.some( + (row) => row.value === representative && keys.has(row.key), + ), + ).toBe(true) + } + for (const row of currentRows) { + expect(subject.equalityLookup(row.value)).toEqual(new Set([row.key])) + expect( + subject.rangeQuery({ from: row.value, to: row.value }), + ).toEqual( + new Set( + currentRows + .filter((candidate) => candidate.groupId === row.groupId) + .map((candidate) => candidate.key), + ), + ) + } + } + + for (const row of rows) index.add(row.key, row) + expectMatchesModel(index, rows) + + const removed = rows.shift()! + index.remove(removed.key, removed) + expectMatchesModel(index, rows) + + const changed = rows[0]! + const previous = { ...changed } + changed.groupId = 99 + changed.value = [Symbol(`updated`)] + index.update(changed.key, previous, changed) + expectMatchesModel(index, rows) + + const rebuilt = new IndexType(2, new PropRef([`value`])) + rebuilt.build(rows.map((row) => [row.key, row])) + expectMatchesModel(rebuilt, rows) + }, + ) + + fcTest.prop([ + fc.array(fc.integer({ min: 0, max: 4 }), { + minLength: 2, + maxLength: 20, + }), + ])(`matches an independent custom-comparator model`, (generatedGroups) => { + const groupIds = [...generatedGroups, generatedGroups[0]!] + const rows = groupIds.map((groupId, position) => ({ + key: String(position).padStart(2, `0`), + value: { groupId, position }, + })) + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareFn: (left, right) => + (left as { groupId: number }).groupId - + (right as { groupId: number }).groupId, + }) + + const expectMatchesModel = (currentRows: typeof rows) => { + const ordered = [...currentRows].sort( + (left, right) => + left.value.groupId - right.value.groupId || + (left.key < right.key ? -1 : left.key > right.key ? 1 : 0), + ) + const forward = ordered.map(({ key }) => key) + expect(index.takeFromStart(currentRows.length)).toEqual(forward) + expect(index.takeReversedFromEnd(currentRows.length)).toEqual( + [...forward].reverse(), + ) + + for (const row of currentRows) { + expect(index.equalityLookup(row.value)).toEqual(new Set([row.key])) + expect(index.rangeQuery({ from: row.value, to: row.value })).toEqual( + new Set( + currentRows + .filter( + (candidate) => candidate.value.groupId === row.value.groupId, + ) + .map(({ key }) => key), + ), + ) + } + } + + for (const row of rows) index.add(row.key, row) + expectMatchesModel(rows) + + const removed = rows[0]! + index.remove(removed.key, removed) + expectMatchesModel(rows.slice(1)) + }) +}) diff --git a/packages/db/tests/integration/uint8array-id-comparison.test.ts b/packages/db/tests/integration/uint8array-id-comparison.test.ts index 7b13c04f63..c0329a5937 100644 --- a/packages/db/tests/integration/uint8array-id-comparison.test.ts +++ b/packages/db/tests/integration/uint8array-id-comparison.test.ts @@ -79,8 +79,7 @@ describe(`Uint8Array ID comparison (user reproduction)`, () => { expect(resultByName?.name).toBe(makeItemName(selectedItemIndex)) }) - it(`should use reference equality for large Uint8Arrays (> 128 bytes)`, async () => { - // Create a large Uint8Array (> 128 bytes) that should use reference equality + it(`should use content equality for large Uint8Arrays`, async () => { const largeId = new Uint8Array(200).fill(42) interface LargeItem { @@ -102,7 +101,6 @@ describe(`Uint8Array ID comparison (user reproduction)`, () => { }), ) - // Query with the exact same reference - this should work const queryWithSameRef = createLiveQueryCollection((q) => q .from({ item: collection }) @@ -113,12 +111,9 @@ describe(`Uint8Array ID comparison (user reproduction)`, () => { await queryWithSameRef.preload() const resultWithSameRef = Array.from(queryWithSameRef.entries())[0]?.[1] - // Should find the item because we're using the same reference expect(resultWithSameRef).toBeDefined() expect(resultWithSameRef?.name).toBe(`Large Item`) - // Query with a different instance but same content - this will NOT work - // because large arrays use reference equality const differentInstance = new Uint8Array(200).fill(42) const queryWithDifferentRef = createLiveQueryCollection((q) => q @@ -132,8 +127,7 @@ describe(`Uint8Array ID comparison (user reproduction)`, () => { queryWithDifferentRef.entries(), )[0]?.[1] - // Should NOT find the item because large arrays use reference equality - // This is expected behavior to avoid memory overhead - expect(resultWithDifferentRef).toBeUndefined() + expect(resultWithDifferentRef).toBeDefined() + expect(resultWithDifferentRef?.name).toBe(`Large Item`) }) }) diff --git a/packages/db/tests/live-query-observer.test.ts b/packages/db/tests/live-query-observer.test.ts new file mode 100644 index 0000000000..fe74985f77 --- /dev/null +++ b/packages/db/tests/live-query-observer.test.ts @@ -0,0 +1,1219 @@ +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { DbClient, collectionOptions } from '../src/client.js' +import { createLiveQueryCollection } from '../src/query/index.js' +import { createLiveQueryObserver } from '../src/live-query-observer.js' +import { + mockSyncCollectionOptions, + mockSyncCollectionOptionsNoInitialState, +} from './utils.js' +import type { ChangeMessage } from '../src/types.js' + +interface Row { + id: string + name: string +} + +const SEED: Array = [ + { id: `1`, name: `A` }, + { id: `2`, name: `B` }, +] + +let seq = 0 +function makeSource(data: Array = SEED) { + return createCollection( + mockSyncCollectionOptions({ + id: `observer-test-${seq++}`, + getKey: (r) => r.id, + initialData: data, + }), + ) +} + +/** A collection that is syncing but not yet ready, with a manual `markReady`. */ +function makeLoadingSource() { + const collection = createCollection( + mockSyncCollectionOptionsNoInitialState({ + id: `observer-loading-${seq++}`, + getKey: (r) => r.id, + }), + ) + collection.startSyncImmediate() + return collection +} + +/** An on-demand collection whose sync exposes a loadSubset spy. */ +function makeLoadSubsetSource() { + const loadSubsetCalls: Array = [] + let writeRow: (type: `insert` | `delete`, row: Row) => void + const collection = createCollection({ + id: `observer-loadsubset-${seq++}`, + getKey: (r) => r.id, + startSync: false, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const row of SEED) write({ type: `insert`, value: row }) + commit() + markReady() + writeRow = (type, row) => { + begin() + write({ type, value: row }) + commit() + } + return { + loadSubset: (options: unknown) => { + loadSubsetCalls.push(options) + return true as const + }, + } + }, + }, + }) + return { + collection, + loadSubsetCalls, + writeRow: (type: `insert` | `delete`, row: Row) => writeRow(type, row), + } +} + +function makeControlledTruncateSource() { + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let resolveLoad!: () => void + let loadCalls = 0 + + const collection = createCollection({ + id: `observer-truncate-${seq++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (ops) => { + begin = ops.begin + write = ops.write + commit = ops.commit + truncate = ops.truncate + ops.markReady() + + return { + loadSubset: () => { + loadCalls++ + return new Promise((resolve) => { + resolveLoad = () => { + begin() + write({ type: `insert`, value: SEED[0]! }) + commit() + resolve() + } + }) + }, + } + }, + }, + }) + + return { + collection, + syncOps: { + begin: () => begin(), + truncate: () => truncate(), + commit: () => commit(), + }, + resolveLoad: () => resolveLoad(), + loadCount: () => loadCalls, + } +} + +describe(`createLiveQueryObserver`, () => { + it.each( + ([`granular`, `wholesale`] as const).flatMap((mode) => + ([`ordinary`, `reentrant`, `dispose`] as const).flatMap((scenario) => + [false, true].map((throwUndefined) => ({ + mode, + scenario, + throwUndefined, + })), + ), + ), + )( + `delivers peer publications before reporting a listener failure: %j`, + async ({ mode, scenario, throwUndefined }) => { + const source = makeSource() + const observer = createLiveQueryObserver(source, { mode }) + const firstError = throwUndefined + ? undefined + : new Error(`First listener failed`) + const secondError = new Error(`Peer listener failed`) + const peerRows = new Map() + const publications: Array> = [] + let armed = false + observer.subscribe(() => { + if (!armed) return + armed = false + if (scenario === `reentrant`) { + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `4`, name: `D` } }) + source.utils.commit() + } + if (scenario === `dispose`) observer.dispose() + throw firstError + }) + observer.subscribe((changes) => { + if (mode === `wholesale`) { + peerRows.clear() + for (const [key, row] of observer.getSnapshot().state ?? []) + peerRows.set(key, row) + } else { + for (const change of changes ?? []) { + if (change.type === `delete`) peerRows.delete(change.key) + else peerRows.set(change.key, change.value) + } + } + publications.push([...peerRows.keys()].sort()) + if (peerRows.has(`3`)) throw secondError + }) + publications.length = 0 + armed = true + try { + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `3`, name: `C` } }) + let caught: { error: unknown } | undefined + try { + source.utils.commit() + } catch (error) { + caught = { error } + } + expect(caught).toEqual({ error: firstError }) + expect(publications).toEqual( + scenario === `dispose` + ? [] + : scenario === `reentrant` + ? [ + [`1`, `2`, `3`], + [`1`, `2`, `3`, `4`], + ] + : [[`1`, `2`, `3`]], + ) + } finally { + observer.dispose() + await source.cleanup() + } + }, + ) + + it(`registers SSR live-query resources for client-owned cleanup`, async () => { + const errorSpy = vi.spyOn(console, `error`).mockImplementation(() => {}) + const client = new DbClient() + const source = client.collection( + collectionOptions({ + id: `observer-client-cleanup-source`, + getKey: (row: Row) => row.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: SEED[0]! }) + commit() + markReady() + }, + }, + }), + ) + const liveQuery = createLiveQueryCollection((q) => q.from({ source })) + client._setSsrServerCleanupEnabled(true) + const sourceObserver = createLiveQueryObserver(source, { + client, + queryHash: `observer-source-cleanup`, + }) + const liveQueryObserver = createLiveQueryObserver(liveQuery, { + client, + queryHash: `observer-client-cleanup`, + }) + sourceObserver.getServerSnapshot() + liveQueryObserver.getServerSnapshot() + liveQuery.startSyncImmediate() + + try { + await client.cleanup() + + expect(errorSpy).not.toHaveBeenCalledWith( + expect.stringContaining(`was manually cleaned up while live query`), + ) + } finally { + errorSpy.mockRestore() + } + }) + + it(`publishes a live error instead of pinning a hydration seed as ready`, () => { + const collection = makeLoadingSource() + const client = new DbClient() + client.hydrate({ + collections: [], + liveQueries: [ + { + queryHash: `seeded-error`, + dehydratedAt: 1, + snapshot: { + rows: [{ key: `1`, value: { id: `1`, name: `server` } }], + }, + }, + ], + }) + const observer = createLiveQueryObserver(collection, { + client, + queryHash: `seeded-error`, + mode: `wholesale`, + }) + const listener = vi.fn() + observer.subscribe(listener) + + collection._lifecycle.setStatus(`error`) + + expect(listener).toHaveBeenCalled() + expect(observer.getSnapshot().status).toBe(`error`) + observer.dispose() + }) + + it(`exposes a streamed query error while a hydration seed is active`, async () => { + const collection = makeLoadingSource() + const client = new DbClient() + client.hydrate({ + collections: [], + liveQueries: [ + { + queryHash: `seeded-stream-error`, + dehydratedAt: 1, + snapshot: { + rows: [{ key: `1`, value: { id: `1`, name: `server` } }], + }, + }, + ], + }) + const observer = createLiveQueryObserver(collection, { + client, + queryHash: `seeded-stream-error`, + mode: `wholesale`, + }) + observer.subscribe(() => {}) + const failure = new Error(`stream failed`) + + client.hydrate({ + collections: [], + liveQueries: [ + { + queryHash: `seeded-stream-error`, + dehydratedAt: 2, + promise: Promise.reject(failure), + }, + ], + }) + await Promise.resolve() + + expect(observer.getError()).toBe(failure) + observer.dispose() + }) + + it(`retries preload after settlement and replaces cached error records`, async () => { + const collection = makeSource() + const preload = vi.spyOn(collection, `preload`).mockResolvedValue(undefined) + const client = new DbClient() + const failure = new Error(`first preload failed`) + await expect( + client._registerLiveQuery(`retry-preload`, Promise.reject(failure)), + ).rejects.toBe(failure) + const observer = createLiveQueryObserver(collection, { + client, + queryHash: `retry-preload`, + }) + + await expect(observer.preload()).resolves.toBeUndefined() + await expect(observer.preload()).resolves.toBeUndefined() + + expect(preload).toHaveBeenCalledTimes(2) + observer.dispose() + }) + + it(`shows a hydrated result until the live collection is authoritative`, async () => { + const collection = makeLoadingSource() + const client = new DbClient() + client.hydrate({ + collections: [], + liveQueries: [ + { + queryHash: `people`, + dehydratedAt: 1, + snapshot: { + rows: [{ key: `1`, value: { id: `1`, name: `From server` } }], + }, + }, + ], + }) + const observer = createLiveQueryObserver(collection as any, { + client, + queryHash: `people`, + mode: `wholesale`, + }) + + expect(observer.getSnapshot()).toMatchObject({ + status: `ready`, + data: [{ id: `1`, name: `From server` }], + }) + + const visibleSnapshots: Array> = [] + observer.subscribe(() => { + visibleSnapshots.push(observer.getSnapshot().data as ReadonlyArray) + }) + collection.utils.begin() + collection.utils.write({ + type: `insert`, + value: { id: `2`, name: `From live sync` }, + }) + collection.utils.commit() + + expect(observer.getSnapshot().data).toEqual([ + { id: `1`, name: `From server` }, + ]) + expect(visibleSnapshots).toEqual([]) + + collection.utils.markReady() + await Promise.resolve() + + expect(observer.getSnapshot().data).toEqual([ + expect.objectContaining({ id: `2`, name: `From live sync` }), + ]) + expect(visibleSnapshots).toHaveLength(1) + expect(visibleSnapshots[0]).toEqual([ + expect.objectContaining({ id: `2`, name: `From live sync` }), + ]) + observer.dispose() + }) + + it(`delivers an atomic hydrated-to-live diff to granular consumers`, async () => { + const collection = makeLoadingSource() + const client = new DbClient() + client.hydrate({ + collections: [], + liveQueries: [ + { + queryHash: `people`, + dehydratedAt: 1, + snapshot: { + rows: [{ key: `1`, value: { id: `1`, name: `From server` } }], + }, + }, + ], + }) + const observer = createLiveQueryObserver(collection as any, { + client, + queryHash: `people`, + }) + const changes: Array> = [] + observer.subscribe((batch) => changes.push(...(batch ?? []))) + + expect(changes).toEqual([ + { + type: `insert`, + key: `1`, + value: { id: `1`, name: `From server` }, + }, + ]) + changes.length = 0 + + collection.utils.begin() + collection.utils.write({ + type: `insert`, + value: { id: `2`, name: `From live sync` }, + }) + collection.utils.commit() + expect(changes).toEqual([]) + + collection.utils.markReady() + await Promise.resolve() + expect(changes).toEqual([ + { + type: `delete`, + key: `1`, + value: { id: `1`, name: `From server` }, + }, + { + type: `insert`, + key: `2`, + value: expect.objectContaining({ id: `2`, name: `From live sync` }), + }, + ]) + observer.dispose() + }) + + it(`does not replay a consumed server snapshot to a later observer`, async () => { + const collection = makeLoadingSource() + const client = new DbClient() + client.hydrate({ + collections: [], + liveQueries: [ + { + queryHash: `people`, + dehydratedAt: 1, + snapshot: { + rows: [{ key: `1`, value: { id: `1`, name: `From server` } }], + }, + }, + ], + }) + const observer = createLiveQueryObserver(collection as any, { + client, + queryHash: `people`, + mode: `wholesale`, + }) + observer.subscribe(() => {}) + + collection.utils.begin() + collection.utils.write({ + type: `insert`, + value: { id: `2`, name: `From live sync` }, + }) + collection.utils.commit() + collection.utils.markReady() + await Promise.resolve() + + const laterObserver = createLiveQueryObserver( + collection as any, + { client, queryHash: `people`, mode: `wholesale` }, + ) + expect(laterObserver.getSnapshot().data).toEqual([ + expect.objectContaining({ id: `2`, name: `From live sync` }), + ]) + expect(client._getLiveQuery(`people`)).toBeUndefined() + observer.dispose() + laterObserver.dispose() + }) + + it(`does not consume a shared hydration result during an abandoned render read`, () => { + const collection = makeLoadingSource() + const client = new DbClient() + client.hydrate({ + collections: [], + liveQueries: [ + { + queryHash: `shared-render-result`, + dehydratedAt: 1, + snapshot: { + rows: [{ key: `1`, value: { id: `1`, name: `From server` } }], + }, + }, + ], + }) + const abandoned = createLiveQueryObserver(collection, { + client, + queryHash: `shared-render-result`, + mode: `wholesale`, + }) + + expect(abandoned.getSnapshot().data).toEqual([ + { id: `1`, name: `From server` }, + ]) + abandoned.dispose() + + const sibling = createLiveQueryObserver(collection, { + client, + queryHash: `shared-render-result`, + mode: `wholesale`, + }) + expect(sibling.getSnapshot().data).toEqual([ + { id: `1`, name: `From server` }, + ]) + expect(client._getLiveQuery(`shared-render-result`)).toBeDefined() + sibling.dispose() + }) + + it(`ignores a server snapshot that arrives after browser sync is ready`, () => { + const collection = makeSource() + const client = new DbClient() + const observer = createLiveQueryObserver(collection as any, { + client, + queryHash: `people`, + mode: `wholesale`, + }) + observer.subscribe(() => {}) + + client.hydrate({ + collections: [], + liveQueries: [ + { + queryHash: `people`, + dehydratedAt: 1, + snapshot: { + rows: [{ key: `server`, value: { id: `server`, name: `Stale` } }], + }, + }, + ], + }) + + expect(observer.getSnapshot().data).toEqual([ + expect.objectContaining({ id: `1`, name: `A` }), + expect.objectContaining({ id: `2`, name: `B` }), + ]) + expect(client._getLiveQuery(`people`)).toBeUndefined() + observer.dispose() + }) + + it(`ignores a server failure that arrives after browser sync is ready`, async () => { + const collection = makeSource() + const client = new DbClient() + const observer = createLiveQueryObserver(collection as any, { + client, + queryHash: `people`, + mode: `wholesale`, + }) + observer.subscribe(() => {}) + let rejectServerResult!: (error: Error) => void + const serverResult = new Promise<{ rows: [] }>((_resolve, reject) => { + rejectServerResult = reject + }) + + client.hydrate({ + collections: [], + liveQueries: [ + { + queryHash: `people`, + dehydratedAt: 1, + promise: serverResult, + }, + ], + }) + rejectServerResult(new Error(`Stale server failure`)) + await Promise.resolve() + + expect(observer.getError()).toBeUndefined() + expect(observer.getSnapshot().data).toEqual([ + expect.objectContaining({ id: `1`, name: `A` }), + expect.objectContaining({ id: `2`, name: `B` }), + ]) + expect(client._getLiveQuery(`people`)).toBeUndefined() + observer.dispose() + }) + + it(`exposes a stable snapshot of a ready collection (wholesale path)`, () => { + const observer = createLiveQueryObserver(makeSource() as any) + + const snap = observer.getSnapshot() + expect(snap.isEnabled).toBe(true) + expect(snap.isReady).toBe(true) + expect(snap.status).toBe(`ready`) + expect(snap.data).toHaveLength(2) + expect(snap.state?.get(`1`)).toMatchObject({ name: `A` }) + // Same identity when nothing changed. + expect(observer.getSnapshot()).toBe(snap) + observer.dispose() + }) + + it(`delivers initial state then change deltas to subscribers (granular path)`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + const deltas: Array> = [] + const unsub = observer.subscribe((changes) => { + if (changes) deltas.push(...changes) + }) + // Initial rows arrive synchronously as inserts (includeInitialState). + expect( + deltas + .filter((c) => c.type === `insert`) + .map((c) => c.key) + .sort(), + ).toEqual([`1`, `2`]) + + const before = observer.getSnapshot() + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `3`, name: `C` } }) + source.utils.commit() + + // Subsequent deltas keep flowing synchronously... + expect(deltas.some((c) => c.type === `insert` && c.key === `3`)).toBe(true) + // ...and wholesale consumers see a fresh, updated snapshot. + const after = observer.getSnapshot() + expect(after).not.toBe(before) + expect(after.data).toHaveLength(3) + + unsub() + observer.dispose() + }) + + it(`stops notifying after unsubscribe / dispose`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + let count = 0 + const unsub = observer.subscribe(() => { + count++ + }) + unsub() + const countAfterUnsub = count // initial-state notify may have fired + + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `9`, name: `Z` } }) + source.utils.commit() + + // No further notifications after unsubscribe. + expect(count).toBe(countAfterUnsub) + observer.dispose() + }) + + it(`represents a disabled query (null collection)`, () => { + const observer = createLiveQueryObserver(null) + const snap = observer.getSnapshot() + expect(snap.isEnabled).toBe(false) + expect(snap.status).toBe(`disabled`) + expect(snap.data).toBeUndefined() + expect(snap.state).toBeUndefined() + observer.dispose() + }) + + it(`delivers nothing synchronously during a wholesale subscribe`, () => { + const observer = createLiveQueryObserver(makeSource() as any, { + mode: `wholesale`, + }) + let notified = false + observer.subscribe(() => { + notified = true + }) + // No bootstrap replay in wholesale mode: useSyncExternalStore-style + // consumers are never notified inside their own subscribe call. + expect(notified).toBe(false) + expect(observer.getSnapshot().data).toHaveLength(2) + observer.dispose() + }) + + it(`delivers events in commit order — no notify can overtake an older one`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + const order: Array = [] + observer.subscribe((changes) => { + for (const c of changes ?? []) order.push(`${c.type}:${c.key}`) + }) + order.length = 0 // drop the bootstrap + + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `v1`, name: `V1` } }) + source.utils.commit() + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `v2`, name: `V2` } }) + source.utils.commit() + + expect(order).toEqual([`insert:v1`, `insert:v2`]) + observer.dispose() + }) + + it(`fires the ready notify once after unsubscribe-before-ready then resubscribe`, () => { + const collection = makeLoadingSource() + const observer = createLiveQueryObserver(collection as any) + + // Subscribe then unsubscribe while still loading — this registers an + // onFirstReady callback that detach() can't remove. + observer.subscribe(() => {})() + + let readyNotifications = 0 + observer.subscribe((changes) => { + if (changes === undefined) readyNotifications++ + }) + + collection.utils.markReady() + + // Only the current subscription's ready callback should fire, not the + // stale one left behind by the first (already unsubscribed) attach. + expect(readyNotifications).toBe(1) + observer.dispose() + }) + + it(`a resubscribe before a microtask cannot leak a stale bootstrap`, async () => { + const observer = createLiveQueryObserver(makeSource() as any) + + // Subscribe then unsubscribe immediately, then resubscribe. All delivery + // is synchronous now, so nothing deferred can flush later. + observer.subscribe(() => {})() + + let notifications = 0 + observer.subscribe(() => { + notifications++ + }) + expect(notifications).toBe(1) // the synchronous bootstrap replay + await Promise.resolve() + expect(notifications).toBe(1) // and nothing else afterwards + observer.dispose() + }) + + it(`dispatches nested publications FIFO, never reentrantly`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + // Listener A reacts to the insert of row 3 by synchronously deleting it — + // a nested publication while the insert is still being delivered. + observer.subscribe((changes) => { + if (changes?.some((c) => c.type === `insert` && c.key === `3`)) { + source.utils.begin() + source.utils.write({ type: `delete`, value: { id: `3`, name: `C` } }) + source.utils.commit() + } + }) + + const listenerBEvents: Array = [] + observer.subscribe((changes) => { + for (const c of changes ?? []) { + if (c.key === `3`) listenerBEvents.push(c.type) + } + }) + + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `3`, name: `C` } }) + source.utils.commit() + + // B must observe the insert before the (nested) delete. + expect(listenerBEvents).toEqual([`insert`, `delete`]) + observer.dispose() + }) + + it(`does not deliver an in-flight publication to a listener added during dispatch`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + let lateListenerRow4Deliveries = 0 + observer.subscribe((changes) => { + // Add the late listener only while the row-4 delta is being dispatched. + if (changes?.some((c) => c.key === `4`)) { + observer.subscribe((lateChanges) => { + if (lateChanges?.some((c) => c.key === `4`)) { + lateListenerRow4Deliveries++ + } + }) + } + }) + + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `4`, name: `D` } }) + source.utils.commit() + + // The late subscriber receives row 4 exactly once — via its seed of the + // already-committed state, NOT additionally via the in-flight publication. + expect(lateListenerRow4Deliveries).toBe(1) + observer.dispose() + }) + + it(`still delivers the in-flight publication to a listener removed during dispatch`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + let row5Deliveries = 0 + let unsubB: (() => void) | null = null + observer.subscribe(() => { + unsubB?.() + unsubB = null + }) + unsubB = observer.subscribe((changes) => { + if (changes?.some((c) => c.key === `5`)) row5Deliveries++ + }) + + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `5`, name: `E` } }) + source.utils.commit() + + // A removed B while the publication was in flight; B still receives it. + expect(row5Deliveries).toBe(1) + observer.dispose() + }) + + it(`treats two subscriptions with the same callback as independent`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + let calls = 0 + const shared = () => { + calls++ + } + const unsubFirst = observer.subscribe(shared) + const unsubSecond = observer.subscribe(shared) + + unsubFirst() + calls = 0 + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `6`, name: `F` } }) + source.utils.commit() + + // The second subscription survives the first one's teardown. + expect(calls).toBe(1) + unsubSecond() + + calls = 0 + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `7`, name: `G` } }) + source.utils.commit() + expect(calls).toBe(0) + observer.dispose() + }) + + it(`releases the collection subscription when a listener disposes during initial replay`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + // The initial-state replay is delivered synchronously inside subscribe(); + // disposing from the listener must not leak the collection subscription. + observer.subscribe(() => observer.dispose()) + + expect(source.subscriberCount).toBe(0) + }) + + it(`seeds a second concurrent subscriber with the current rows`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + observer.subscribe(() => {}) + + // The attach (and its initial-state replay) already happened; a late + // subscriber must still receive the current rows as inserts. + const secondSubscriberKeys: Array = [] + observer.subscribe((changes) => { + for (const c of changes ?? []) { + if (c.type === `insert`) secondSubscriberKeys.push(c.key) + } + }) + + expect(secondSubscriberKeys.sort()).toEqual([`1`, `2`]) + observer.dispose() + }) + + it(`throws when subscribing after dispose`, () => { + const observer = createLiveQueryObserver(makeSource() as any) + observer.dispose() + expect(() => observer.subscribe(() => {})).toThrow( + /disposed LiveQueryObserver/, + ) + }) + + it(`preserves snapshot identity across subscribe/unsubscribe cycles`, () => { + const observer = createLiveQueryObserver(makeSource() as any) + + const before = observer.getSnapshot() + observer.subscribe(() => {})() + observer.subscribe(() => {})() + + // Bootstrap replay is per-subscriber delivery, not a semantic revision: + // nothing observable changed, so the snapshot identity must not change. + expect(observer.getSnapshot()).toBe(before) + observer.dispose() + }) + + it(`emits exactly one post-bootstrap notification for a readiness transition`, () => { + const collection = makeLoadingSource() + const observer = createLiveQueryObserver(collection as any) + + const events: Array = [] + observer.subscribe((changes) => events.push(changes)) + + collection.utils.markReady() + + // Not the old [[], undefined, []]: empty batches carry no semantic change, + // so one readiness transition publishes exactly once. + expect(events).toEqual([undefined]) + observer.dispose() + }) + + it(`serves a fresh snapshot for rows changed while detached`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + const unsubscribe = observer.subscribe(() => {}) + const before = observer.getSnapshot() + unsubscribe() + + // Mutate while nothing is attached; the status does not change. + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `8`, name: `H` } }) + source.utils.commit() + + const after = observer.getSnapshot() + expect(after).not.toBe(before) + expect(after.state?.has(`8`)).toBe(true) + expect(after.data).toHaveLength(3) + observer.dispose() + }) + + it(`wholesale mode does not request an initial snapshot (no unfiltered loadSubset)`, () => { + const { collection, loadSubsetCalls, writeRow } = makeLoadSubsetSource() + const observer = createLiveQueryObserver(collection as any, { + mode: `wholesale`, + }) + + const notifies: Array = [] + observer.subscribe((changes) => notifies.push(changes)) + + // No initial-state request, so no loadSubset({ where: undefined }) — the + // pre-observer React/Angular loading policy. + expect(loadSubsetCalls).toHaveLength(0) + // No bootstrap replay either (only status wake-ups, which carry no + // changes); wholesale consumers read getSnapshot(). + expect(notifies.filter((n) => n !== undefined)).toHaveLength(0) + expect(observer.getSnapshot().data).toHaveLength(2) + + // Deltas — including deletes — still wake the consumer. + const deltasBefore = notifies.filter( + (changes) => changes !== undefined, + ).length + writeRow(`delete`, { id: `1`, name: `A` }) + + expect(notifies.filter((changes) => changes !== undefined)).toHaveLength( + deltasBefore + 1, + ) + expect(observer.getSnapshot().data).toHaveLength(1) + observer.dispose() + }) + + it(`granular mode still seeds from an initial snapshot`, () => { + const { collection, loadSubsetCalls } = makeLoadSubsetSource() + const observer = createLiveQueryObserver(collection as any) + const before = observer.getSnapshot() + + const inserted: Array = [] + observer.subscribe((changes) => { + for (const c of changes ?? []) { + if (c.type === `insert`) inserted.push(c.key) + } + }) + + expect(inserted.sort()).toEqual([`1`, `2`]) + expect(loadSubsetCalls).toHaveLength(1) + expect(observer.getSnapshot()).not.toBe(before) + expect(observer.getSnapshot().data).toHaveLength(2) + observer.dispose() + }) + + it(`reuses captured entries for a status-only snapshot change`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + const entriesSpy = vi.spyOn(source, `entries`) + expect(observer.getSnapshot().data).toHaveLength(2) + expect(entriesSpy).toHaveBeenCalledTimes(1) + + source._lifecycle.setStatus(`error`) + + expect(observer.getSnapshot().status).toBe(`error`) + expect(observer.getSnapshot().state?.size).toBe(2) + // Status-only changes reuse the immutable row capture. + expect(entriesSpy).toHaveBeenCalledTimes(1) + observer.dispose() + }) + + it(`does not activate sync at construction — only on first subscribe`, () => { + const collection = createCollection( + mockSyncCollectionOptionsNoInitialState({ + id: `observer-idle-${seq++}`, + getKey: (r) => r.id, + }), + ) + const observer = createLiveQueryObserver(collection as any) + + // Construction (e.g. in an abandoned React render) is inert. + expect(collection.status).toBe(`idle`) + expect(observer.getSnapshot().status).toBe(`idle`) + + const unsubscribe = observer.subscribe(() => {}) + expect(collection.status).not.toBe(`idle`) + unsubscribe() + observer.dispose() + }) + + it(`wakes consumers on status-only transitions (error, cleaned-up)`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + + const statuses: Array = [] + observer.subscribe(() => { + statuses.push(observer.getSnapshot().status) + }) + + // Status transitions carry no row changes; the observer must publish + // them through the same canonical path as data changes. + source._lifecycle.setStatus(`error`) + source._lifecycle.setStatus(`cleaned-up`) + + expect(statuses).toContain(`error`) + expect(statuses).toContain(`cleaned-up`) + observer.dispose() + }) + + it(`refreshes the snapshot when status changes without a version bump`, () => { + // A status-only loading→ready transition with no active subscription: the + // cached snapshot must not stay stale (covers the preload() case too). + const collection = makeLoadingSource() + const observer = createLiveQueryObserver(collection as any) + + expect(observer.getSnapshot().isReady).toBe(false) + expect(observer.getSnapshot().status).toBe(`loading`) + + collection.utils.markReady() + + expect(observer.getSnapshot().isReady).toBe(true) + expect(observer.getSnapshot().status).toBe(`ready`) + observer.dispose() + }) + + it(`bumps layoutRevision on a membership change that a joined key signature would collide on`, () => { + // Two keys "a","b" vs a single key "a\u0000b" join to the same string under + // any separator that can appear in a key. The revision compares the key + // sequence directly, so it must still register the membership change. + const source = makeSource([ + { id: `a`, name: `A` }, + { id: `b`, name: `B` }, + ]) + const observer = createLiveQueryObserver(source as any) + observer.subscribe(() => {}) + + const revBefore = observer.getSnapshot().layoutRevision + + source.utils.begin() + source.utils.write({ type: `delete`, value: { id: `a`, name: `A` } }) + source.utils.write({ type: `delete`, value: { id: `b`, name: `B` } }) + source.utils.write({ + type: `insert`, + value: { id: `a\u0000b`, name: `AB` }, + }) + source.utils.commit() + + expect(observer.getSnapshot().layoutRevision).not.toBe(revBefore) + observer.dispose() + }) + + it(`keeps an unread snapshot pinned to the revision when it was created`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any, { + mode: `wholesale`, + }) + const before = observer.getSnapshot() + + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `3`, name: `C` } }) + source.utils.commit() + + expect(before.data).toHaveLength(2) + expect(before.state?.has(`3`)).toBe(false) + observer.dispose() + }) + + it(`does not notify synchronously when wholesale subscribe activates an idle source`, () => { + const { collection } = makeLoadSubsetSource() + const observer = createLiveQueryObserver(collection as any, { + mode: `wholesale`, + }) + const before = observer.getSnapshot() + let insideSubscribe = true + let calledSynchronously = false + + const unsubscribe = observer.subscribe(() => { + if (insideSubscribe) calledSynchronously = true + }) + insideSubscribe = false + + expect(calledSynchronously).toBe(false) + expect(observer.getSnapshot()).not.toBe(before) + expect(observer.getSnapshot().status).toBe(`ready`) + expect(observer.getSnapshot().data).toHaveLength(2) + unsubscribe() + observer.dispose() + }) + + it(`does not reenter a late subscriber while delivering its seed`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + observer.subscribe(() => {}) + let callbackDepth = 0 + let wasReentrant = false + let wrote = false + + observer.subscribe(() => { + callbackDepth++ + if (callbackDepth > 1) wasReentrant = true + if (!wrote) { + wrote = true + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `3`, name: `C` } }) + source.utils.commit() + } + callbackDepth-- + }) + + expect(wasReentrant).toBe(false) + observer.dispose() + }) + + it(`keeps waking consumers after collection cleanup clears event listeners`, () => { + const source = makeSource() + const observer = createLiveQueryObserver(source as any) + const statuses: Array = [] + observer.subscribe(() => statuses.push(observer.getSnapshot().status)) + + void source.cleanup() + source._lifecycle.setStatus(`error`) + + expect(statuses).toContain(`cleaned-up`) + expect(statuses).toContain(`error`) + observer.dispose() + }) + + it(`invalidates snapshots for compatible collections without a state revision`, () => { + const rows = new Map([[`1`, { id: `1`, name: `A` }]]) + const listeners = new Set< + (changes: Array>) => void + >() + const collection = { + status: `ready`, + entries: () => rows.entries(), + on: () => () => {}, + subscribeChanges: ( + listener: (changes: Array>) => void, + ) => { + listeners.add(listener) + return { unsubscribe: () => listeners.delete(listener) } + }, + preload: async () => {}, + } + const observer = createLiveQueryObserver(collection as any, { + mode: `wholesale`, + }) + observer.subscribe(() => {}) + const before = observer.getSnapshot() + + const row = { id: `2`, name: `B` } + rows.set(row.id, row) + for (const listener of listeners) { + listener([{ type: `insert`, key: row.id, value: row }]) + } + + const after = observer.getSnapshot() + expect(after).not.toBe(before) + expect(after.data).toHaveLength(2) + observer.dispose() + }) + + it(`does not expose a truncate while its subscription is buffering a refetch`, async () => { + const { collection, syncOps, resolveLoad, loadCount } = + makeControlledTruncateSource() + await collection.stateWhenReady() + + const observer = createLiveQueryObserver(collection as any) + observer.subscribe(() => {}) + + resolveLoad() + await vi.waitFor(() => expect(observer.getSnapshot().data).toHaveLength(1)) + const before = observer.getSnapshot() + + syncOps.begin() + syncOps.truncate() + syncOps.commit() + await vi.waitFor(() => expect(loadCount()).toBe(2)) + + expect(observer.getSnapshot()).toBe(before) + expect(observer.getSnapshot().data).toHaveLength(1) + observer.dispose() + }) +}) diff --git a/packages/db/tests/live-query-options.test.ts b/packages/db/tests/live-query-options.test.ts new file mode 100644 index 0000000000..6c2500d428 --- /dev/null +++ b/packages/db/tests/live-query-options.test.ts @@ -0,0 +1,81 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { + getLiveQueryHash, + getPreparedLiveQueryIdentity, + prepareLiveQueryValue, +} from '../src/live-query-options.js' +import { BaseQueryBuilder } from '../src/query/builder/index.js' + +describe(`live query preparation`, () => { + it.each([undefined, null])( + `promotes a nullish config query result to a disabled query`, + (disabled) => { + const prepared = prepareLiveQueryValue( + { query: () => disabled }, + undefined, + new Set(), + ) + + expect(prepared).toBe(disabled) + }, + ) +}) + +describe(`live query identity`, () => { + it(`hashes Map values in an explicit queryKey deterministically`, () => { + const first = getLiveQueryHash(undefined, [ + new Map([ + [`b`, 2], + [`a`, 1], + ]), + ]) + const second = getLiveQueryHash(undefined, [ + new Map([ + [`a`, 1], + [`b`, 2], + ]), + ]) + + expect(first).toBe(second) + }) + + it(`hashes Set values in an explicit queryKey deterministically`, () => { + const first = getLiveQueryHash(undefined, [new Set([`b`, `a`])]) + const second = getLiveQueryHash(undefined, [new Set([`a`, `b`])]) + + expect(first).toBe(second) + }) + + it(`treats an empty queryKey as absent`, () => { + const first = createCollection<{ id: string }>({ + id: `empty-query-key-first`, + getKey: (row) => row.id, + sync: { sync: ({ markReady }) => markReady() }, + }) + const second = createCollection<{ id: string }>({ + id: `empty-query-key-second`, + getKey: (row) => row.id, + sync: { sync: ({ markReady }) => markReady() }, + }) + + expect(getLiveQueryHash(first, [])).not.toBe(getLiveQueryHash(second, [])) + }) + + it(`does not collapse configs with opaque row identity behavior`, () => { + const source = createCollection<{ id: string }>({ + id: `live-query-config-identity-source`, + getKey: (row) => row.id, + sync: { sync: ({ markReady }) => markReady() }, + }) + const query = new BaseQueryBuilder().from({ source }) + const first = { query, getKey: (row: { id: string }) => row.id } + const second = { query, getKey: (row: { id: string }) => `x-${row.id}` } + + expect(getPreparedLiveQueryIdentity(first)).not.toEqual( + getPreparedLiveQueryIdentity(second), + ) + expect(() => getLiveQueryHash(first)).toThrow(/function value/) + expect(() => getLiveQueryHash(second)).toThrow(/function value/) + }) +}) diff --git a/packages/db/tests/live-query-order-only-move.test.ts b/packages/db/tests/live-query-order-only-move.test.ts new file mode 100644 index 0000000000..5d6bf5181d --- /dev/null +++ b/packages/db/tests/live-query-order-only-move.test.ts @@ -0,0 +1,504 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { createLiveQueryCollection } from '../src/query/live-query-collection.js' +import { createLiveQueryObserver } from '../src/live-query-observer.js' +import { eq } from '../src/query/builder/functions.js' +import { mockSyncCollectionOptions } from './utils.js' + +interface Person { + id: string + name: string + age: number +} + +const SEED: Array = [ + { id: `1`, name: `Alice`, age: 30 }, + { id: `2`, name: `Bob`, age: 20 }, + { id: `3`, name: `Carol`, age: 40 }, +] + +let seq = 0 +function makeSource(data: Array = SEED) { + return createCollection( + mockSyncCollectionOptions({ + id: `order-only-move-${seq++}`, + getKey: (p) => p.id, + initialData: data, + }), + ) +} + +/** Live query ordered by `age` (NOT projected), selecting only `{ id, name }`. */ +async function makeOrderedByAge(source: ReturnType) { + const lq = createLiveQueryCollection((q) => + q + .from({ p: source }) + .orderBy(({ p }) => p.age, `asc`) + .select(({ p }) => ({ id: p.id, name: p.name })), + ) + await lq.preload() + return lq +} + +const flush = () => new Promise((r) => setTimeout(r, 0)) + +describe(`order-only move (RFC #1623 phase 4)`, () => { + it(`republishes the ordered result when a row moves but its value is unchanged`, async () => { + const source = makeSource() + const lq = await makeOrderedByAge(source) + const observer = createLiveQueryObserver< + { id: string; name: string }, + string + >(lq as any) + + let notifications = 0 + observer.subscribe(() => { + notifications++ + }) + + const before = observer.getSnapshot() + expect((before.data as Array).map((r) => r.id)).toEqual([ + `2`, + `1`, + `3`, + ]) + const revBefore = before.layoutRevision + + // Move Bob (age 20 -> 99) to the end. The projected `{ id, name }` is + // identical, so the collection's value-diff emits no row change — only the + // layout notification should republish the new order. + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `2`, name: `Bob`, age: 99 }, + }) + source.utils.commit() + await flush() + + const after = observer.getSnapshot() + expect((after.data as Array).map((r) => r.id)).toEqual([`1`, `3`, `2`]) + expect(after.layoutRevision).toBeGreaterThan(revBefore) + expect(notifications).toBeGreaterThan(0) + observer.dispose() + }) + + it(`refreshes a detached observer after an order-only move`, async () => { + const source = makeSource() + const lq = await makeOrderedByAge(source) + const observer = createLiveQueryObserver< + { id: string; name: string }, + string + >(lq as any) + + const before = observer.getSnapshot() + expect((before.data as Array).map((row) => row.id)).toEqual([ + `2`, + `1`, + `3`, + ]) + + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `2`, name: `Bob`, age: 99 }, + }) + source.utils.commit() + await flush() + + const after = observer.getSnapshot() + expect(after).not.toBe(before) + expect((after.data as Array).map((row) => row.id)).toEqual([ + `1`, + `3`, + `2`, + ]) + expect(after.layoutRevision).toBeGreaterThan(before.layoutRevision) + observer.dispose() + }) + + it(`refreshes a detached observer when an order-only sync is parked`, async () => { + const source = makeSource() + const persist = createDeferred() + const lq = createLiveQueryCollection({ + getKey: (row) => row.id, + query: (q) => + q + .from({ p: source }) + .orderBy(({ p }) => p.age, `asc`) + .select(({ p }) => ({ id: p.id, name: p.name })), + onUpdate: () => persist.promise, + }) + await lq.preload() + const observer = createLiveQueryObserver< + { id: string; name: string }, + string + >(lq as any) + + const before = observer.getSnapshot() + const collectionLayoutRevisionBefore = lq._layoutRevision + expect((before.data as Array).map((row) => row.id)).toEqual([ + `2`, + `1`, + `3`, + ]) + + const mutation = lq.update( + `1`, + { optimistic: false }, + (draft) => void (draft.name = `Pending`), + ) + expect(mutation.state).toBe(`persisting`) + + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `2`, name: `Bob`, age: 99 }, + }) + source.utils.commit() + await flush() + + const parked = observer.getSnapshot() + expect((parked.data as Array).map((row) => row.id)).toEqual([ + `2`, + `1`, + `3`, + ]) + expect(lq._layoutRevision).toBe(collectionLayoutRevisionBefore) + + persist.resolve() + await mutation.isPersisted.promise + await flush() + + const after = observer.getSnapshot() + expect((after.data as Array).map((row) => row.id)).toEqual([ + `1`, + `3`, + `2`, + ]) + expect(lq._layoutRevision).toBeGreaterThan(collectionLayoutRevisionBefore) + observer.dispose() + }) + + it(`does not bump the layout revision when nothing about the layout changes`, async () => { + const source = makeSource() + const lq = await makeOrderedByAge(source) + const observer = createLiveQueryObserver< + { id: string; name: string }, + string + >(lq as any) + let notifications = 0 + observer.subscribe(() => notifications++) + notifications = 0 + + const revBefore = observer.getSnapshot().layoutRevision + + // Update a row's `age` in a way that keeps its sort position (20 -> 21, + // still the youngest) and does not change the projected value. + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `2`, name: `Bob`, age: 21 }, + }) + source.utils.commit() + await flush() + + // Order is unchanged (`2` still first), so the layout revision is stable. + const after = observer.getSnapshot() + expect((after.data as Array).map((r) => r.id)).toEqual([`2`, `1`, `3`]) + expect(after.layoutRevision).toBe(revBefore) + expect(notifications).toBe(0) + observer.dispose() + }) + + it(`does not publish when multiple moves cancel within one transaction`, async () => { + const source = makeSource() + const lq = await makeOrderedByAge(source) + const observer = createLiveQueryObserver< + { id: string; name: string }, + string + >(lq as any) + let notifications = 0 + observer.subscribe(() => notifications++) + notifications = 0 + + const before = observer.getSnapshot() + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `2`, name: `Bob`, age: 99 }, + }) + source.utils.write({ + type: `update`, + value: { id: `2`, name: `Bob`, age: 20 }, + }) + source.utils.commit() + await flush() + + expect(observer.getSnapshot()).toBe(before) + expect(notifications).toBe(0) + observer.dispose() + }) + + it(`exposes layout-only signals as empty public change batches`, async () => { + const source = makeSource() + const lq = await makeOrderedByAge(source) + const publications: Array> = [] + const subscription = lq.subscribeChanges( + (changes) => publications.push(changes), + { includeInitialState: false }, + ) + + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `2`, name: `Bob`, age: 99 }, + }) + source.utils.commit() + await flush() + + expect(publications).toEqual([[]]) + subscription.unsubscribe() + }) + + it(`bumps the layout revision on membership changes too`, async () => { + const source = makeSource() + const lq = await makeOrderedByAge(source) + const observer = createLiveQueryObserver< + { id: string; name: string }, + string + >(lq as any) + observer.subscribe(() => {}) + + const revBefore = observer.getSnapshot().layoutRevision + + source.utils.begin() + source.utils.write({ + type: `insert`, + value: { id: `4`, name: `Dan`, age: 10 }, + }) + source.utils.commit() + await flush() + + const after = observer.getSnapshot() + expect((after.data as Array).map((r) => r.id)).toEqual([ + `4`, + `2`, + `1`, + `3`, + ]) + expect(after.layoutRevision).toBeGreaterThan(revBefore) + observer.dispose() + }) + + // Kyle's review issue 1: a commit containing both an ordinary value update + // and an order-only move must publish exactly once — the ordinary publication + // already carries the final values and ordering, so the separate layout event + // is redundant. + it(`publishes a mixed value update and order-only move exactly once`, async () => { + const source = makeSource() + const lq = await makeOrderedByAge(source) + const observer = createLiveQueryObserver< + { id: string; name: string }, + string + >(lq as any) + + let notifications = 0 + observer.subscribe(() => notifications++) + notifications = 0 // exclude subscribeChanges' initial-state publication + + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `1`, name: `Alicia`, age: 30 }, + }) + source.utils.write({ + type: `update`, + value: { id: `2`, name: `Bob`, age: 99 }, + }) + source.utils.commit() + + const after = observer.getSnapshot() + expect( + (after.data as Array).map(({ id, name }) => [id, name]), + ).toEqual([ + [`1`, `Alicia`], + [`3`, `Carol`], + [`2`, `Bob`], + ]) + expect(notifications).toBe(1) + + // The layout clock from this mixed publication must be consumed even + // though it arrived with row changes. A later legacy empty-ready event is + // not a second layout publication. + ;(lq as any)._changes.emitEmptyReadyEvent() + expect(notifications).toBe(1) + observer.dispose() + }) + + it(`publishes a mixed batch to a subscriber that filters out the row update`, async () => { + const source = makeSource() + const lq = await makeOrderedByAge(source) + const publications: Array> = [] + const subscription = lq.subscribeChanges( + (changes) => publications.push(changes), + { + includeInitialState: false, + where: (row) => eq(row.name, `Bob`), + }, + ) + + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `1`, name: `Alicia`, age: 30 }, + }) + source.utils.write({ + type: `update`, + value: { id: `2`, name: `Bob`, age: 99 }, + }) + source.utils.commit() + + expect(publications).toEqual([]) + subscription.unsubscribe() + }) + + // Kyle's review issue 2: an ordered child collection produced by `includes` + // must consume the insertion-side order metadata and publish its move when the + // projected child value is unchanged. + it(`publishes an ordered included child move exactly once`, async () => { + const parents = createCollection( + mockSyncCollectionOptions<{ id: string }>({ + id: `order-only-parents-${seq++}`, + getKey: ({ id }) => id, + initialData: [{ id: `p1` }], + }), + ) + const children = createCollection( + mockSyncCollectionOptions<{ + id: string + parentId: string + name: string + position: number + }>({ + id: `order-only-children-${seq++}`, + getKey: ({ id }) => id, + initialData: [ + { id: `c1`, parentId: `p1`, name: `One`, position: 1 }, + { id: `c2`, parentId: `p1`, name: `Two`, position: 2 }, + ], + }), + ) + const lq = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.parentId, parent.id)) + .orderBy(({ child }) => child.position) + .select(({ child }) => ({ id: child.id, name: child.name })), + })), + ) + await lq.preload() + + const childCollection = (lq.get(`p1`) as any).children + const observer = createLiveQueryObserver(childCollection) + let notifications = 0 + observer.subscribe(() => notifications++) + notifications = 0 + + children.utils.begin() + children.utils.write({ + type: `update`, + value: { id: `c1`, parentId: `p1`, name: `One`, position: 3 }, + }) + children.utils.commit() + + expect([...childCollection.values()].map(({ id }: any) => id)).toEqual([ + `c2`, + `c1`, + ]) + expect(notifications).toBe(1) + observer.dispose() + }) + + // The includes flush is recursive, so the order-only-move handling must hold + // at depth, not just one level. Two levels of ordered includes + // (org -> teams -> members); move a grandchild whose projected value is + // unchanged and assert its collection re-sorts and publishes exactly once. + it(`publishes an ordered move in a deeply-nested included child exactly once`, async () => { + const orgs = createCollection( + mockSyncCollectionOptions<{ id: string }>({ + id: `order-only-orgs-${seq++}`, + getKey: ({ id }) => id, + initialData: [{ id: `o1` }], + }), + ) + const teams = createCollection( + mockSyncCollectionOptions<{ + id: string + orgId: string + position: number + }>({ + id: `order-only-teams-${seq++}`, + getKey: ({ id }) => id, + initialData: [{ id: `t1`, orgId: `o1`, position: 1 }], + }), + ) + const members = createCollection( + mockSyncCollectionOptions<{ + id: string + teamId: string + name: string + position: number + }>({ + id: `order-only-members-${seq++}`, + getKey: ({ id }) => id, + initialData: [ + { id: `m1`, teamId: `t1`, name: `One`, position: 1 }, + { id: `m2`, teamId: `t1`, name: `Two`, position: 2 }, + ], + }), + ) + const lq = createLiveQueryCollection((q) => + q.from({ org: orgs }).select(({ org }) => ({ + id: org.id, + teams: q + .from({ team: teams }) + .where(({ team }) => eq(team.orgId, org.id)) + .orderBy(({ team }) => team.position) + .select(({ team }) => ({ + id: team.id, + members: q + .from({ member: members }) + .where(({ member }) => eq(member.teamId, team.id)) + .orderBy(({ member }) => member.position) + .select(({ member }) => ({ id: member.id, name: member.name })), + })), + })), + ) + await lq.preload() + + const teamCollection = (lq.get(`o1`) as any).teams + const memberCollection = teamCollection.get(`t1`).members + const observer = createLiveQueryObserver(memberCollection) + let notifications = 0 + observer.subscribe(() => notifications++) + notifications = 0 + + // Move m1 behind m2 (position 1 -> 3); projected { id, name } unchanged. + members.utils.begin() + members.utils.write({ + type: `update`, + value: { id: `m1`, teamId: `t1`, name: `One`, position: 3 }, + }) + members.utils.commit() + + expect([...memberCollection.values()].map(({ id }: any) => id)).toEqual([ + `m2`, + `m1`, + ]) + expect(notifications).toBe(1) + observer.dispose() + }) +}) diff --git a/packages/db/tests/live-query-window-controller.test.ts b/packages/db/tests/live-query-window-controller.test.ts new file mode 100644 index 0000000000..6526a2cafd --- /dev/null +++ b/packages/db/tests/live-query-window-controller.test.ts @@ -0,0 +1,1355 @@ +import { describe, expect, it, vi } from 'vitest' +import { BTreeIndex } from '../src/index.js' +import { createCollection } from '../src/collection/index.js' +import { createLiveQueryCollection } from '../src/query/live-query-collection.js' +import { LIVE_QUERY_INTERNAL } from '../src/query/live/internal.js' +import { LiveQueryWindowControllerDisposedError } from '../src/errors.js' +import { + createLiveQueryWindowController, + normalizeLiveQueryWindowPageSize, +} from '../src/live-query-window-controller.js' +import { mockSyncCollectionOptions } from './utils.js' +import { evaluateReferenceExpression } from './reference-expression.js' +import type { Collection } from '../src/collection/index.js' + +interface Row { + id: string + n: number +} + +const ROWS: Array = [1, 2, 3, 4, 5].map((n) => ({ id: String(n), n })) + +let seq = 0 +function makeSource(initialData: Array = ROWS) { + return createCollection( + mockSyncCollectionOptions({ + id: `window-ctrl-${seq++}`, + getKey: (r) => r.id, + initialData, + }), + ) +} + +/** Ordered live query with page 1's peek-ahead window baked in, as the React adapter builds it. */ +function makeOrderedLiveQuery(source: Collection, pageSize: number) { + return createLiveQueryCollection({ + query: (q) => + q + .from({ r: source }) + .orderBy(({ r }) => r.n, `asc`) + .limit(pageSize + 1) + .offset(0) + .select(({ r }) => ({ id: r.id, n: r.n })), + startSync: true, + gcTime: 1, + }) +} + +const flush = () => new Promise((r) => setTimeout(r, 0)) + +const ids = (snap: { data: ReadonlyArray }) => snap.data.map((r) => r.id) + +describe(`createLiveQueryWindowController`, () => { + it.each( + [0, 2, 5].flatMap((rowCount) => + [`fetch`, `reset`, `dispose`].map((action) => ({ rowCount, action })), + ), + )( + `handles $action during initial loading with $rowCount rows`, + async ({ rowCount, action }) => { + const source = makeSource(ROWS.slice(0, rowCount)) + const lq = makeOrderedLiveQuery(source, 2) + const controller = createLiveQueryWindowController(lq, { + pageSize: 2, + }) + const unsubscribe = controller.subscribe(() => {}) + try { + expect(controller.getSnapshot().isLoading).toBe(true) + const fetch = controller.fetchNextPage() + expect(controller.fetchNextPage()).toBe(fetch) + if (action === `reset`) await controller.reset() + if (action === `dispose`) controller.dispose() + await fetch + const visibleCount = action === `fetch` ? 4 : 2 + expect(ids(controller.getSnapshot())).toEqual( + ROWS.slice(0, Math.min(rowCount, visibleCount)).map((row) => row.id), + ) + expect(controller.getSnapshot().pages).toHaveLength( + action === `fetch` && rowCount > 2 ? 2 : 1, + ) + } finally { + unsubscribe() + controller.dispose() + await lq.cleanup() + await source.cleanup() + } + }, + ) + + it.each([ + { pageSize: undefined, normalized: 20 }, + { pageSize: 0, normalized: 20 }, + { pageSize: -1, normalized: 20 }, + { pageSize: 1.5, normalized: 20 }, + { pageSize: Number.POSITIVE_INFINITY, normalized: 20 }, + { pageSize: Number.MAX_SAFE_INTEGER, normalized: 20 }, + { pageSize: 1, normalized: 1 }, + { + pageSize: Number.MAX_SAFE_INTEGER - 1, + normalized: Number.MAX_SAFE_INTEGER - 1, + }, + ])( + `normalizes pageSize $pageSize to $normalized`, + ({ pageSize, normalized }) => { + expect(normalizeLiveQueryWindowPageSize(pageSize)).toBe(normalized) + }, + ) + + it(`exposes the first page with a peek-ahead hasNextPage`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await lq.preload() + await flush() + + const snap = controller.getSnapshot() + expect(ids(snap)).toEqual([`1`, `2`]) + expect(snap.pages.map((p) => p.map((r) => r.id))).toEqual([[`1`, `2`]]) + expect(snap.pageParams).toEqual([0]) + expect(snap.hasNextPage).toBe(true) + expect(snap.isFetchingNextPage).toBe(false) + controller.dispose() + }) + + it(`loads further pages via fetchNextPage until the source is exhausted`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await lq.preload() + await flush() + + controller.fetchNextPage() + await flush() + let snap = controller.getSnapshot() + expect(ids(snap)).toEqual([`1`, `2`, `3`, `4`]) + expect(snap.pages.map((p) => p.map((r) => r.id))).toEqual([ + [`1`, `2`], + [`3`, `4`], + ]) + expect(snap.pageParams).toEqual([0, 1]) + expect(snap.hasNextPage).toBe(true) + + controller.fetchNextPage() + await flush() + snap = controller.getSnapshot() + // 5 rows total; the 3rd page is a partial page and there is no peek row. + expect(ids(snap)).toEqual([`1`, `2`, `3`, `4`, `5`]) + expect(snap.pages.map((p) => p.map((r) => r.id))).toEqual([ + [`1`, `2`], + [`3`, `4`], + [`5`], + ]) + expect(snap.hasNextPage).toBe(false) + controller.dispose() + }) + + it(`fetchNextPage is a no-op when there is no next page`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 10) // pageSize > row count + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 10, + }) + controller.subscribe(() => {}) + await lq.preload() + await flush() + expect(controller.getSnapshot().hasNextPage).toBe(false) + + controller.fetchNextPage() + await flush() + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`, `3`, `4`, `5`]) + expect(controller.getSnapshot().pages).toHaveLength(1) + controller.dispose() + }) + + it(`returns the active fetch promise to concurrent callers`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await lq.preload() + + const failure = new Error(`window failed`) + const originalSetWindow = lq.utils.setWindow.bind(lq.utils) + let rejectWindow!: (error: Error) => void + vi.spyOn(lq.utils, `setWindow`).mockImplementationOnce((options) => { + originalSetWindow(options) + return new Promise((_resolve, reject) => { + rejectWindow = reject + }) + }) + + const first = controller.fetchNextPage() + const second = controller.fetchNextPage() + expect(second).toBe(first) + let secondSettled = false + const firstOutcome = first.then( + () => undefined, + (error: unknown) => error, + ) + const secondOutcome = second.then( + () => { + secondSettled = true + return undefined + }, + (error: unknown) => { + secondSettled = true + return error + }, + ) + + await Promise.resolve() + const secondWasPending = !secondSettled + rejectWindow(failure) + + expect(secondWasPending).toBe(true) + expect(await firstOutcome).toBe(failure) + expect(await secondOutcome).toBe(failure) + controller.dispose() + }) + + it(`represents an empty enabled query as one empty page`, async () => { + const lq = makeOrderedLiveQuery(makeSource([]), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await lq.preload() + + const snapshot = controller.getSnapshot() + expect(snapshot.isEnabled).toBe(true) + expect(snapshot.data).toEqual([]) + expect(snapshot.pages).toEqual([[]]) + expect(snapshot.hasNextPage).toBe(false) + controller.dispose() + }) + + it(`uses the default page size when pageSize is zero`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 0, + }) + controller.subscribe(() => {}) + await lq.preload() + + const snapshot = controller.getSnapshot() + expect(ids(snapshot)).toEqual([`1`, `2`, `3`, `4`, `5`]) + expect(snapshot.pages).toHaveLength(1) + expect(snapshot.hasNextPage).toBe(false) + controller.dispose() + }) + + it(`reset returns to the first page`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await lq.preload() + await flush() + + controller.fetchNextPage() + await flush() + expect(controller.getSnapshot().pages).toHaveLength(2) + + controller.reset() + await flush() + const snap = controller.getSnapshot() + expect(ids(snap)).toEqual([`1`, `2`]) + expect(snap.pages).toHaveLength(1) + controller.dispose() + }) + + it(`notifies subscribers on data changes and page changes`, async () => { + const source = makeSource() + const lq = makeOrderedLiveQuery(source, 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + let notifications = 0 + controller.subscribe(() => notifications++) + await lq.preload() + await flush() + + notifications = 0 + controller.fetchNextPage() + await flush() + expect(notifications).toBeGreaterThan(0) + controller.dispose() + }) + + it(`retries a window that throws synchronously`, () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const setWindow = vi + .spyOn(lq.utils, `setWindow`) + .mockImplementationOnce(() => { + throw new Error(`window failed`) + }) + .mockReturnValue(true) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + + expect(() => controller.subscribe(() => {})).toThrow(`window failed`) + const unsubscribe = controller.subscribe(() => {}) + + expect(setWindow).toHaveBeenCalledTimes(2) + unsubscribe() + controller.dispose() + }) + + it(`retains the settled public window after a graph throw until retry`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + await lq.preload() + const settledRows = lq.toArray + const builder = lq.utils[LIVE_QUERY_INTERNAL].getBuilder() + const originalWindowFn = Reflect.get(builder, `windowFn`) as (options: { + offset?: number + limit?: number + }) => void + const windowFn = vi.fn(originalWindowFn) + const originalRunGraph = Reflect.get( + builder, + `maybeRunGraphFn`, + ) as () => void + const requestedError = new Error(`requested window failed`) + const maybeRunGraph = vi.fn(originalRunGraph).mockImplementationOnce(() => { + throw requestedError + }) + Reflect.set(builder, `windowFn`, windowFn) + Reflect.set(builder, `maybeRunGraphFn`, maybeRunGraph) + + let caught: unknown + try { + lq.utils.setWindow({ offset: 0, limit: 5 }) + } catch (error) { + caught = error + } + expect(caught).toBe(requestedError) + expect(windowFn).toHaveBeenNthCalledWith(1, { offset: 0, limit: 5 }) + // Retain public state, not a rollback of the already advanced private graph. + expect(windowFn).toHaveBeenCalledTimes(1) + expect(maybeRunGraph).toHaveBeenCalledTimes(1) + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + expect(lq.toArray).toEqual(settledRows) + + await lq.utils.setWindow({ offset: 0, limit: 5 }) + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) + expect(lq.toArray.map(({ id, n }) => ({ id, n }))).toEqual(ROWS) + await lq.cleanup() + }) + + it(`keeps the committed page retryable when a window load rejects`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await lq.preload() + await flush() + expect(controller.getSnapshot().hasNextPage).toBe(true) + + const failure = new Error(`load failed`) + vi.spyOn(lq.utils, `setWindow`).mockRejectedValueOnce(failure) + + await expect(Promise.resolve(controller.fetchNextPage())).rejects.toThrow( + `load failed`, + ) + + expect(controller.getSnapshot().pages).toHaveLength(1) + expect(controller.getSnapshot().hasNextPage).toBe(true) + expect((controller.getSnapshot() as { error?: unknown }).error).toBe( + failure, + ) + + await controller.fetchNextPage() + expect(controller.getSnapshot().pages).toHaveLength(2) + controller.dispose() + }) + + it(`clears a preload error after a successful retry`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const failure = new Error(`preload failed`) + vi.spyOn(lq.utils, `setWindow`).mockRejectedValueOnce(failure) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + + await expect(controller.preload()).rejects.toBe(failure) + expect(controller.getSnapshot().error).toBe(failure) + + await controller.preload() + expect(controller.getSnapshot().isError).toBe(false) + expect(controller.getSnapshot().error).toBeUndefined() + controller.dispose() + }) + + it(`publishes one coherent loading snapshot and one settled snapshot`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + const snapshots: Array<{ pages: number; fetching: boolean }> = [] + controller.subscribe(() => { + const snapshot = controller.getSnapshot() + snapshots.push({ + pages: snapshot.pages.length, + fetching: snapshot.isFetchingNextPage, + }) + }) + await lq.preload() + await flush() + snapshots.length = 0 + + let resolveWindow!: () => void + vi.spyOn(lq.utils, `setWindow`).mockReturnValueOnce( + new Promise((resolve) => { + resolveWindow = resolve + }), + ) + + const fetch = Promise.resolve(controller.fetchNextPage()) + expect(snapshots).toEqual([{ pages: 1, fetching: true }]) + + resolveWindow() + await fetch + expect(snapshots).toEqual([ + { pages: 1, fetching: true }, + { pages: 2, fetching: false }, + ]) + controller.dispose() + }) + + it.each([`resolve`, `reject`] as const)( + `preload joins a pending page fetch that will %s`, + async (outcome) => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController( + lq as any, + { + pageSize: 2, + }, + ) + controller.subscribe(() => {}) + await lq.preload() + + const originalSetWindow = lq.utils.setWindow.bind(lq.utils) + let resolveExpansion!: () => void + let rejectExpansion!: (error: unknown) => void + const failure = new Error(`expansion failed`) + const setWindow = vi + .spyOn(lq.utils, `setWindow`) + .mockImplementationOnce((options) => { + const pending = new Promise((resolve, reject) => { + resolveExpansion = resolve + rejectExpansion = reject + }) + return Promise.resolve(originalSetWindow(options)).then(() => pending) + }) + + const expansion = controller.fetchNextPage() + const expansionOutcome = expansion.catch((error: unknown) => error) + const preload = controller.preload() + let preloadSettled = false + const preloadOutcome = preload.then( + () => { + preloadSettled = true + }, + (error: unknown) => { + preloadSettled = true + return error + }, + ) + await flush() + expect(setWindow).toHaveBeenCalledTimes(1) + expect(preloadSettled).toBe(false) + if (outcome === `reject`) { + rejectExpansion(failure) + expect(await expansionOutcome).toBe(failure) + expect(await preloadOutcome).toBe(failure) + expect(controller.getSnapshot().pages).toHaveLength(1) + expect(controller.getSnapshot().error).toBe(failure) + await controller.fetchNextPage() + } else { + resolveExpansion() + expect(await expansionOutcome).toBeUndefined() + expect(await preloadOutcome).toBeUndefined() + } + + expect(controller.getSnapshot().pages).toHaveLength(2) + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`, `3`, `4`]) + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) + controller.dispose() + }, + ) + + it(`publishes source changes while a page fetch is pending`, async () => { + const source = makeSource() + const lq = makeOrderedLiveQuery(source, 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + let notifications = 0 + controller.subscribe(() => notifications++) + await lq.preload() + notifications = 0 + + vi.spyOn(lq.utils, `setWindow`).mockReturnValueOnce( + new Promise(() => {}), + ) + void controller.fetchNextPage() + notifications = 0 + + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: `1`, n: 0 }, + }) + source.utils.commit() + await flush() + + expect(notifications).toBeGreaterThan(0) + expect(controller.getSnapshot().data[0]).toMatchObject({ id: `1`, n: 0 }) + controller.dispose() + }) + + it(`surfaces a real async subset-load failure from setWindow`, async () => { + const remoteRows = [...ROWS] + let rejectLoads = false + const failure = new Error(`remote page failed`) + const source = createCollection({ + id: `window-ctrl-rejecting-source-${seq++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + // Boundary refinement asks only for the last loaded tie class. + // The source has already supplied that row. + if (options.where) return Promise.resolve() + return new Promise((resolve, reject) => { + queueMicrotask(() => { + if (rejectLoads) { + reject(failure) + return + } + begin() + remoteRows.slice(0, options.limit).forEach((row) => { + write({ type: `insert`, value: row }) + }) + commit() + resolve() + }) + }) + }, + } + }, + }, + }) + const lq = createLiveQueryCollection({ + query: (q) => + q + .from({ r: source }) + .orderBy(({ r }) => r.n, `asc`) + .limit(3) + .offset(0) + .select(({ r }) => ({ id: r.id, n: r.n })), + startSync: true, + gcTime: 1, + utils: { + customUtility: () => `custom`, + }, + }) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await controller.preload() + expect(controller.getSnapshot().hasNextPage).toBe(true) + + rejectLoads = true + await expect(controller.fetchNextPage()).rejects.toBe(failure) + expect(controller.getSnapshot().pages).toHaveLength(1) + expect(controller.getSnapshot().error).toBe(failure) + expect(lq.utils.lastSubsetError).toBe(failure) + expect(lq.utils.customUtility()).toBe(`custom`) + controller.dispose() + }) + + it(`reset supersedes an in-flight page expansion`, async () => { + // Isolate controller generations: the mock accepts reset without source work. + // The real source publication barrier is tested separately below. + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await lq.preload() + + let resolveExpansion!: () => void + const setWindow = vi + .spyOn(lq.utils, `setWindow`) + .mockReturnValueOnce( + new Promise((resolve) => { + resolveExpansion = resolve + }), + ) + .mockReturnValueOnce(true) + + const expansion = controller.fetchNextPage() + expect(controller.getSnapshot().isFetchingNextPage).toBe(true) + + await controller.reset() + expect(controller.getSnapshot().pages).toHaveLength(1) + expect(controller.getSnapshot().isFetchingNextPage).toBe(false) + expect(setWindow).toHaveBeenNthCalledWith(1, { offset: 0, limit: 5 }) + expect(setWindow).toHaveBeenNthCalledWith(2, { offset: 0, limit: 3 }) + + resolveExpansion() + await expansion + expect(controller.getSnapshot().pages).toHaveLength(1) + controller.dispose() + }) + + it.each([`resolve`, `reject`] as const)( + `reset waits for publication-blocking source work that will %s`, + async (outcome) => { + const failure = new Error(`superseded expansion failed`) + let holdNextRequest = false + let settleExpansion: (() => void) | undefined + const loaded = new Set() + const source = createCollection({ + id: `window-reset-real-source-${seq++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + const matching = ROWS.filter( + (row) => + !options.where || + evaluateReferenceExpression(options.where, row) === true, + ) + const cursor = options.cursor + const from = cursor + ? matching.filter( + (row) => + evaluateReferenceExpression(cursor.whereFrom, row) === + true, + ) + : matching.slice(options.offset ?? 0) + const limited = from.slice(0, options.limit) + const requested = cursor + ? matching.filter( + (row) => + limited.includes(row) || + evaluateReferenceExpression( + cursor.whereCurrent, + row, + ) === true, + ) + : limited + const apply = () => { + begin() + requested.forEach((row) => { + if (loaded.has(row.id)) return + loaded.add(row.id) + write({ type: `insert`, value: row }) + }) + return commit() + } + if (!holdNextRequest) + return Promise.resolve(apply()).then(() => {}) + holdNextRequest = false + return new Promise((resolve, reject) => { + settleExpansion = () => { + if (outcome === `reject`) reject(failure) + else + void Promise.resolve(apply()).then( + () => resolve(), + reject, + ) + } + }) + }, + } + }, + }, + }) + const lq = makeOrderedLiveQuery(source, 2) + const controller = createLiveQueryWindowController( + lq as any, + { + pageSize: 2, + }, + ) + controller.subscribe(() => {}) + + try { + await controller.preload() + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`]) + holdNextRequest = true + const expansion = Promise.resolve(controller.fetchNextPage()) + const expansionOutcome = expansion.catch((error: unknown) => error) + expect(holdNextRequest).toBe(false) + expect(settleExpansion).toBeTypeOf(`function`) + const reset = Promise.resolve(controller.reset()) + let resetSettled = false + const resetOutcome = reset.then( + () => { + resetSettled = true + }, + (error: unknown) => { + resetSettled = true + return error + }, + ) + await flush() + expect(resetSettled).toBe(false) + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`]) + settleExpansion!() + if (outcome === `reject`) { + expect(await resetOutcome).toBe(failure) + expect(await expansionOutcome).toBe(failure) + expect(controller.getSnapshot().error).toBe(failure) + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`]) + await controller.reset() + } else { + expect(await resetOutcome).toBeUndefined() + expect(await expansionOutcome).toBeUndefined() + } + expect(controller.getSnapshot().pages).toHaveLength(1) + expect(controller.getSnapshot().isError).toBe(false) + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + await controller.fetchNextPage() + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`, `3`, `4`]) + } finally { + controller.dispose() + await Promise.all([lq.cleanup(), source.cleanup()]) + } + }, + ) + + it.each( + [false, true].flatMap((waitBeforeCleanup) => + [false, true].map((pendingLoad) => ({ waitBeforeCleanup, pendingLoad })), + ), + )( + `cleanup cancels unfinished operations with waitFirst=$waitBeforeCleanup, pending=$pendingLoad`, + async ({ waitBeforeCleanup, pendingLoad }) => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + await lq.preload() + + let resolveLoad!: () => void + const load = new Promise((resolve) => { + resolveLoad = resolve + }) + const operation = lq._sync.beginLoadSubsetOperation() + if (pendingLoad) lq._sync.trackLoadPromise(load) + const beforeCleanup = waitBeforeCleanup ? operation.wait() : undefined + const observe = (result: true | Promise) => + Promise.resolve(result).then( + () => undefined, + (error: unknown) => error, + ) + const observedBefore = + beforeCleanup === undefined ? undefined : observe(beforeCleanup) + lq._sync.cleanup() + const observed = observedBefore ?? observe(operation.wait()) + const outcome = await observed + if (waitBeforeCleanup && !pendingLoad) { + expect(beforeCleanup).toBe(true) + expect(outcome).toBeUndefined() + } else { + expect(outcome).toMatchObject({ name: `AbortError` }) + } + + resolveLoad() + expect(await observed).toBe(outcome) + await lq.cleanup() + }, + ) + + it(`cleanup settles every superseded load operation`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + await lq.preload() + + const firstLoad = new Promise(() => {}) + const firstOperation = lq._sync.beginLoadSubsetOperation() + lq._sync.trackLoadPromise(firstLoad) + const firstWaiting = Promise.resolve(firstOperation.wait()) + + const secondLoad = new Promise(() => {}) + const secondOperation = lq._sync.beginLoadSubsetOperation() + lq._sync.trackLoadPromise(secondLoad) + const secondWaiting = Promise.resolve(secondOperation.wait()) + const settled = [false, false] + void firstWaiting.catch(() => { + settled[0] = true + }) + void secondWaiting.catch(() => { + settled[1] = true + }) + + lq._sync.cleanup() + await Promise.resolve() + + expect(settled).toEqual([true, true]) + await expect(firstWaiting).rejects.toMatchObject({ name: `AbortError` }) + await expect(secondWaiting).rejects.toMatchObject({ name: `AbortError` }) + await lq.cleanup() + }) + + it(`retains an unsubscribed lease until overlapping requests settle`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + await lq.preload() + + const originalSetWindow = lq.utils.setWindow.bind(lq.utils) + const resolvers: Array<() => void> = [] + const setWindow = vi + .spyOn(lq.utils, `setWindow`) + .mockImplementation((options) => { + originalSetWindow(options) + if (resolvers.length >= 2) return true + return new Promise((resolve) => resolvers.push(resolve)) + }) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + + const expansion = controller.fetchNextPage() + const reset = controller.reset() + expect(setWindow).toHaveBeenCalledTimes(2) + + resolvers[0]!() + await expansion + + const competingController = createLiveQueryWindowController( + lq as any, + { pageSize: 1 }, + ) + competingController.subscribe(() => {}) + expect(setWindow).toHaveBeenCalledTimes(2) + + resolvers[1]!() + await reset + expect(controller.getSnapshot().pages).toHaveLength(1) + competingController.dispose() + controller.dispose() + }) + + it(`replays the desired window after collection cleanup`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + const unsubscribe = controller.subscribe(() => {}) + await lq.preload() + + await controller.fetchNextPage() + await flush() + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`, `3`, `4`]) + + unsubscribe() + await lq.cleanup() + + controller.subscribe(() => {}) + await lq.preload() + await flush() + + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`, `3`, `4`]) + expect(controller.getSnapshot().hasNextPage).toBe(true) + controller.dispose() + }) + + it(`establishes the desired window before preload`, async () => { + const source = makeSource() + const lq = createLiveQueryCollection({ + query: (q) => + q + .from({ r: source }) + .orderBy(({ r }) => r.n, `asc`) + .limit(2) + .select(({ r }) => ({ id: r.id, n: r.n })), + gcTime: 1, + }) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + + await controller.preload() + + expect(controller.getSnapshot().hasNextPage).toBe(true) + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + controller.dispose() + }) + + it(`coordinates the physical window across multiple controllers`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const larger = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + const smaller = createLiveQueryWindowController(lq as any, { + pageSize: 1, + }) + + larger.subscribe(() => {}) + smaller.subscribe(() => {}) + await lq.preload() + await flush() + + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + expect(ids(larger.getSnapshot())).toEqual([`1`, `2`]) + expect(larger.getSnapshot().hasNextPage).toBe(true) + + await larger.fetchNextPage() + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) + + await smaller.fetchNextPage() + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) + + const setWindow = vi.spyOn(lq.utils, `setWindow`) + larger.dispose() + expect(setWindow).toHaveBeenLastCalledWith({ offset: 0, limit: 3 }) + expect(setWindow.mock.results.at(-1)!.type).toBe(`return`) + await setWindow.mock.results.at(-1)!.value + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + expect(lq.toArray.map(({ id, n }) => ({ id, n }))).toEqual(ROWS.slice(0, 3)) + smaller.dispose() + }) + + it(`rolls a failed lease request back to its committed window`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await lq.preload() + + const failure = new Error(`window failed`) + vi.spyOn(lq.utils, `setWindow`).mockRejectedValueOnce(failure) + await expect(controller.fetchNextPage()).rejects.toBe(failure) + + const smaller = createLiveQueryWindowController(lq as any, { + pageSize: 1, + }) + smaller.subscribe(() => {}) + + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + smaller.dispose() + controller.dispose() + }) + + it(`restores the remaining lease after a pending larger lease is released`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const keeper = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + keeper.subscribe(() => {}) + await lq.preload() + await keeper.fetchNextPage() + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) + + const transient = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + transient.subscribe(() => {}) + await transient.fetchNextPage() + + const originalSetWindow = lq.utils.setWindow.bind(lq.utils) + let resolveExpansion!: () => void + vi.spyOn(lq.utils, `setWindow`).mockImplementation((options) => { + const result = originalSetWindow(options) + if (options.limit !== 7) return result + return new Promise((resolve) => { + resolveExpansion = resolve + }) + }) + + const expansion = transient.fetchNextPage() + transient.dispose() + + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) + resolveExpansion() + await expansion + keeper.dispose() + }) + + it(`repairs an externally moved physical window`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await lq.preload() + + await lq.utils.setWindow({ offset: 1, limit: 3 }) + expect(lq.utils.getWindow()).toEqual({ offset: 1, limit: 3 }) + + await controller.preload() + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + controller.dispose() + }) + + it(`restores the query's initial window after the last lease is released`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + const unsubscribe = controller.subscribe(() => {}) + await lq.preload() + await controller.fetchNextPage() + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) + + const setWindow = vi.spyOn(lq.utils, `setWindow`) + unsubscribe() + + expect(setWindow).toHaveBeenLastCalledWith({ offset: 0, limit: 3 }) + expect(setWindow.mock.results.at(-1)!.type).toBe(`return`) + await setWindow.mock.results.at(-1)!.value + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + expect(lq.toArray.map(({ id, n }) => ({ id, n }))).toEqual(ROWS.slice(0, 3)) + controller.dispose() + await lq.cleanup() + }) + + it.each([`throws`, `rejects`] as const)( + `retains the original baseline when its first restoration %s`, + async (failureMode) => { + const lq = makeOrderedLiveQuery(makeSource(), 3) + const first = createLiveQueryWindowController(lq as any, { + pageSize: 3, + }) + const unsubscribeFirst = first.subscribe(() => {}) + await lq.preload() + await first.fetchNextPage() + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 7 }) + + const setWindow = vi.spyOn(lq.utils, `setWindow`) + if (failureMode === `throws`) { + setWindow.mockImplementationOnce(() => { + throw new Error(`restore failed`) + }) + } else { + setWindow.mockRejectedValueOnce(new Error(`restore failed`)) + } + unsubscribeFirst() + await Promise.resolve() + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 7 }) + + const second = createLiveQueryWindowController(lq as any, { + pageSize: 3, + }) + const unsubscribeSecond = second.subscribe(() => {}) + unsubscribeSecond() + + expect(setWindow).toHaveBeenLastCalledWith({ offset: 0, limit: 4 }) + expect(setWindow.mock.results.at(-1)!.type).toBe(`return`) + await setWindow.mock.results.at(-1)!.value + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) + expect(lq.toArray.map(({ id, n }) => ({ id, n }))).toEqual( + ROWS.slice(0, 4), + ) + first.dispose() + second.dispose() + await lq.cleanup() + }, + ) + + it(`recaptures an externally changed window before a new lease cycle`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const first = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + const unsubscribeFirst = first.subscribe(() => {}) + await lq.preload() + unsubscribeFirst() + + await lq.utils.setWindow({ offset: 0, limit: 4 }) + const second = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + const unsubscribeSecond = second.subscribe(() => {}) + unsubscribeSecond() + + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) + first.dispose() + second.dispose() + await lq.cleanup() + }) + + it(`recaptures an external window change after standalone preload`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 3, + }) + + await controller.preload() + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) + + await lq.utils.setWindow({ offset: 0, limit: 6 }) + const unsubscribe = controller.subscribe(() => {}) + unsubscribe() + + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 6 }) + controller.dispose() + await lq.cleanup() + }) + + it(`ignores a failed attachment superseded by a new lease`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + await lq.preload() + + let rejectFirst!: (error: Error) => void + vi.spyOn(lq.utils, `setWindow`) + .mockReturnValueOnce( + new Promise((_, reject) => { + rejectFirst = reject + }), + ) + .mockReturnValue(true) + + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + const unsubscribe = controller.subscribe(() => {}) + unsubscribe() + controller.subscribe(() => {}) + + rejectFirst(new Error(`stale attachment failed`)) + await flush() + + expect(controller.getSnapshot().isError).toBe(false) + expect(controller.getSnapshot().error).toBeUndefined() + controller.dispose() + }) + + it(`does not notify synchronously while subscribing by default`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + await lq.preload() + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + let subscribing = true + let notifiedWhileSubscribing = false + + const unsubscribe = controller.subscribe(() => { + if (subscribing) notifiedWhileSubscribing = true + }) + subscribing = false + + expect(notifiedWhileSubscribing).toBe(false) + unsubscribe() + controller.dispose() + }) + + it(`keeps duplicate callback subscriptions independent`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + let notifications = 0 + const listener = () => notifications++ + const unsubscribeFirst = controller.subscribe(listener) + const unsubscribeSecond = controller.subscribe(listener) + await lq.preload() + await flush() + + notifications = 0 + unsubscribeFirst() + controller.fetchNextPage() + await flush() + + expect(notifications).toBeGreaterThan(0) + unsubscribeSecond() + controller.dispose() + }) + + it(`does not deliver an in-flight notification to a late subscriber`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + let publishing = false + let lateNotifications = 0 + let unsubscribeLate: (() => void) | undefined + const unsubscribeFirst = controller.subscribe(() => { + if (publishing && !unsubscribeLate) { + unsubscribeLate = controller.subscribe(() => lateNotifications++) + } + }) + await lq.preload() + await flush() + vi.spyOn(lq.utils, `setWindow`).mockReturnValue(true) + + publishing = true + controller.fetchNextPage() + publishing = false + + expect(lateNotifications).toBe(0) + unsubscribeLate?.() + unsubscribeFirst() + controller.dispose() + }) + + it(`rejects every subscription after disposal`, () => { + const controller = createLiveQueryWindowController( + makeOrderedLiveQuery(makeSource(), 2) as any, + { pageSize: 2 }, + ) + controller.dispose() + + expect(() => controller.subscribe(() => {})).toThrow( + LiveQueryWindowControllerDisposedError, + ) + expect(() => controller.subscribe(() => {})).toThrow( + LiveQueryWindowControllerDisposedError, + ) + }) + + it(`releases subscriptions when disposed by a listener`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + + controller.subscribe(() => controller.dispose()) + await lq.preload() + await controller.fetchNextPage() + + expect(lq.subscriberCount).toBe(0) + }) + + it(`stops an in-flight publication when a listener disposes`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + let publishing = false + let secondListenerNotifications = 0 + controller.subscribe(() => { + if (publishing) controller.dispose() + }) + controller.subscribe(() => { + if (publishing) secondListenerNotifications++ + }) + await lq.preload() + await flush() + vi.spyOn(lq.utils, `setWindow`).mockReturnValue(true) + + publishing = true + controller.fetchNextPage() + publishing = false + + expect(secondListenerNotifications).toBe(0) + }) + + it(`skips a listener unsubscribed during an in-flight publication`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + let publishing = false + let secondListenerNotifications = 0 + let unsubscribeSecond = () => {} + controller.subscribe(() => { + if (publishing) unsubscribeSecond() + }) + unsubscribeSecond = controller.subscribe(() => { + if (publishing) secondListenerNotifications++ + }) + await lq.preload() + await flush() + vi.spyOn(lq.utils, `setWindow`).mockReturnValue(true) + + publishing = true + await controller.fetchNextPage() + publishing = false + + expect(secondListenerNotifications).toBe(0) + controller.dispose() + }) + + it(`returns a stable snapshot identity when nothing changed`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await lq.preload() + await flush() + expect(controller.getSnapshot()).toBe(controller.getSnapshot()) + controller.dispose() + }) + + it(`derives status flags from a pagination error status`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + }) + controller.subscribe(() => {}) + await lq.preload() + + vi.spyOn(lq.utils, `setWindow`).mockRejectedValueOnce( + new Error(`window failed`), + ) + await expect(controller.fetchNextPage()).rejects.toThrow(`window failed`) + + expect(controller.getSnapshot()).toMatchObject({ + status: `error`, + isLoading: false, + isReady: false, + isIdle: false, + isError: true, + isCleanedUp: false, + }) + controller.dispose() + }) + + it(`normalizes a NaN initial page count to one page`, async () => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController(lq as any, { + pageSize: 2, + initialPageCount: Number.NaN, + }) + controller.subscribe(() => {}) + await lq.preload() + + expect(controller.getSnapshot().pages).toHaveLength(1) + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`]) + controller.dispose() + }) + + it(`represents a disabled controller (null collection)`, () => { + const controller = createLiveQueryWindowController(null) + const snap = controller.getSnapshot() + expect(snap.isEnabled).toBe(false) + expect(snap.data).toEqual([]) + expect(snap.hasNextPage).toBe(false) + expect(snap.pages).toEqual([]) + controller.dispose() + }) +}) diff --git a/packages/db/tests/local-storage-persistence-failure.test.ts b/packages/db/tests/local-storage-persistence-failure.test.ts new file mode 100644 index 0000000000..6aff40db6a --- /dev/null +++ b/packages/db/tests/local-storage-persistence-failure.test.ts @@ -0,0 +1,106 @@ +import { expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index' +import { localStorageCollectionOptions } from '../src/local-storage' +import { createTransaction } from '../src/transactions' + +type Row = { id: string; value: number } + +const cases = ([`insert`, `update`, `delete`] as const).flatMap((operation) => + ([`storage`, `serialization`] as const).flatMap((failure) => + [false, true].map((manual) => ({ operation, failure, manual })), + ), +) + +it.each(cases)( + `does not persist a rejected mutation on the next successful write: %j`, + async ({ operation, failure, manual }) => { + const data = new Map() + const error = new Error(`Persistence failed`) + let fail = false + const storage = { + getItem: (key: string) => data.get(key) ?? null, + removeItem: (key: string) => { + data.delete(key) + }, + setItem: (key: string, value: string) => { + if (fail && failure === `storage`) throw error + data.set(key, value) + }, + } + const makeCollection = () => + createCollection( + localStorageCollectionOptions({ + storageKey: `rows`, + storage, + storageEventApi: { addEventListener() {}, removeEventListener() {} }, + getKey: (row) => row.id, + parser: { + parse: JSON.parse, + stringify: (value: unknown) => { + // Per-row validation succeeds; serializing the full stored map fails. + if ( + fail && + failure === `serialization` && + typeof value === `object` && + value !== null && + !(`id` in value) + ) { + throw error + } + return JSON.stringify(value) + }, + }, + }), + ) + const collection = makeCollection() + const log = vi.spyOn(console, `error`).mockImplementation(() => {}) + try { + await collection.preload() + await collection.insert({ id: `seed`, value: 1 }).isPersisted.promise + const mutate = () => { + if (operation === `insert`) + return collection.insert({ id: `bad`, value: 2 }) + if (operation === `delete`) return collection.delete(`seed`) + return collection.update(`seed`, (draft) => { + draft.value = 2 + }) + } + fail = true + const failed = manual + ? createTransaction({ + autoCommit: false, + mutationFn: ({ transaction }) => { + collection.utils.acceptMutations(transaction) + return Promise.resolve() + }, + }) + : mutate() + const rejection = expect(failed.isPersisted.promise).rejects.toBe(error) + if (manual) { + failed.mutate(mutate) + await failed.commit().catch(() => {}) + } + await rejection + fail = false + await collection.insert({ id: `good`, value: 3 }).isPersisted.promise + const restored = makeCollection() + try { + await restored.preload() + expect( + [...restored.values()] + .map(({ id, value }) => ({ id, value })) + .sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual([ + { id: `good`, value: 3 }, + { id: `seed`, value: 1 }, + ]) + } finally { + await restored.cleanup() + } + } finally { + fail = false + await collection.cleanup() + log.mockRestore() + } + }, +) diff --git a/packages/db/tests/observer-cleanup-restart.test.ts b/packages/db/tests/observer-cleanup-restart.test.ts new file mode 100644 index 0000000000..07c63114f6 --- /dev/null +++ b/packages/db/tests/observer-cleanup-restart.test.ts @@ -0,0 +1,65 @@ +import { expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index' +import { createLiveQueryCollection } from '../src/query/live-query-collection' +import { createLiveQueryObserver } from '../src/live-query-observer' +import { mockSyncCollectionOptions } from './utils' + +it.each( + ([`wholesale`, `granular`] as const).flatMap((mode) => + ([`read`, `subscribe`] as const).map((resume) => ({ mode, resume })), + ), +)( + `refreshes a detached $mode snapshot after unobserved GC and empty reload on $resume`, + async ({ mode, resume }) => { + vi.useFakeTimers() + const row = { id: 1, name: `Alice` } + const source = createCollection( + mockSyncCollectionOptions({ + id: `observer-restart-source`, + getKey: (value: typeof row) => value.id, + initialData: [row], + }), + ) + const live = createLiveQueryCollection({ + startSync: true, + gcTime: 1, + query: (q) => q.from({ row: source }), + }) + const observer = createLiveQueryObserver(live, { mode }) + try { + const originalSnapshot = observer.getSnapshot() + expect(originalSnapshot.status).toBe(`ready`) + expect(originalSnapshot.data).toMatchObject([row]) + expect(live.subscriberCount).toBe(0) + + await vi.advanceTimersByTimeAsync(51) + expect(live.status).toBe(`cleaned-up`) + expect(source.subscriberCount).toBe(0) + source.utils.begin() + source.utils.write({ type: `delete`, value: row }) + source.utils.commit() + + // A separate caller reloads the query. The original observer misses + // every intermediate lifecycle state and receives no row events. + await live.preload() + expect(live.status).toBe(`ready`) + expect(live.size).toBe(0) + if (resume === `subscribe`) observer.subscribe(() => {}) + + const snapshot = observer.getSnapshot() + expect(snapshot.data).toEqual([]) + expect(snapshot.state).toEqual(new Map()) + expect(snapshot.status).toBe(`ready`) + expect(snapshot.layoutRevision).toBeGreaterThan( + originalSnapshot.layoutRevision, + ) + expect(observer.getSnapshot()).toBe(snapshot) + expect(originalSnapshot.data).toMatchObject([row]) + } finally { + observer.dispose() + await live.cleanup() + await source.cleanup() + vi.useRealTimers() + } + }, +) diff --git a/packages/db/tests/optimistic-composition.test.ts b/packages/db/tests/optimistic-composition.test.ts new file mode 100644 index 0000000000..1ad03a2ebc --- /dev/null +++ b/packages/db/tests/optimistic-composition.test.ts @@ -0,0 +1,506 @@ +import { describe, expect, it } from 'vitest' +import { z } from 'zod' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { createLiveQueryCollection } from '../src/query/index.js' +import { createTransaction } from '../src/transactions.js' +import { stripVirtualProps } from './utils.js' +import type { SyncConfig } from '../src/types.js' + +type Row = { id: number; a: string; b: string; c: string } +const initial: Row = { id: 1, a: `a0`, b: `b0`, c: `c0` } +const orders = [ + [0, 1, 2], + [0, 2, 1], + [1, 0, 2], + [1, 2, 0], + [2, 0, 1], + [2, 1, 0], +] + +function source(onUpdate = () => Promise.resolve()) { + let sync!: Parameters[`sync`]>[0] + const collection = createCollection({ + getKey: (row) => row.id, + onUpdate, + sync: { + sync: (params) => { + sync = params + params.begin() + params.write({ type: `insert`, value: initial }) + params.commit() + params.markReady() + }, + }, + }) + return { + collection, + get sync() { + return sync + }, + } +} + +function pendingTransaction() { + const done = createDeferred() + const tx = createTransaction({ + autoCommit: false, + mutationFn: () => done.promise, + }) + const settled = tx.isPersisted.promise.catch((error: unknown) => error) + return { tx, done, settled } +} + +const cases = orders.flatMap((order) => + ([`pending`, `persisting`] as const).flatMap((phase) => + [0, 1, 2].flatMap((removed) => + ([`disjoint`, `overlapping`] as const).map((fields) => ({ + order, + phase, + removed, + fields, + })), + ), + ), +) + +describe(`optimistic snapshot ownership`, () => { + it.each([false, true])( + `does not attribute a later remote write to a failed-only update, optimistic=%s`, + async (optimistic) => { + const done = createDeferred() + const fixture = source(() => done.promise) + await fixture.collection.preload() + const tx = fixture.collection.update(1, { optimistic }, (row) => { + row.a = `failed` + }) + const settled = tx.isPersisted.promise.catch(() => {}) + try { + done.reject(new Error(`failed update`)) + await settled + fixture.sync.begin() + fixture.sync.write({ + type: `update`, + value: { ...initial, a: `remote` }, + }) + await fixture.sync.commit() + expect(fixture.collection.get(1)?.$origin).toBe(`remote`) + } finally { + done.resolve() + await fixture.collection.cleanup() + } + }, + ) + + it.each([false, true])( + `attributes synchronous handler acknowledgement to its local update, optimistic=%s`, + async (optimistic) => { + const fixture = source(() => { + fixture.sync.begin() + fixture.sync.write({ + type: `update`, + value: { ...initial, a: `accepted` }, + }) + void fixture.sync.commit() + return Promise.resolve() + }) + await fixture.collection.preload() + try { + const tx = fixture.collection.update(1, { optimistic }, (row) => { + row.a = `accepted` + }) + await tx.isPersisted.promise + expect(fixture.collection.get(1)?.$origin).toBe(`local`) + expect(stripVirtualProps(fixture.collection.get(1))).toEqual({ + ...initial, + a: `accepted`, + }) + } finally { + await fixture.collection.cleanup() + } + }, + ) + + it.each([false, true])( + `keeps completed nonoptimistic origin across sibling rollback, completed first=%s`, + async (completedFirst) => { + const done = [createDeferred(), createDeferred()] + let calls = 0 + const fixture = source(() => done[calls++]!.promise) + await fixture.collection.preload() + const first = fixture.collection.update( + 1, + { optimistic: false }, + (row) => { + row.a = `accepted` + }, + ) + const second = fixture.collection.update(1, (row) => { + row.b = `rejected` + }) + const secondSettled = second.isPersisted.promise.catch(() => {}) + try { + if (completedFirst) { + done[0]!.resolve() + await first.isPersisted.promise + } + done[1]!.reject(new Error(`sibling failure`)) + await secondSettled + if (!completedFirst) { + done[0]!.resolve() + await first.isPersisted.promise + } + fixture.sync.begin() + fixture.sync.write({ + type: `update`, + value: { ...initial, a: `accepted` }, + }) + await fixture.sync.commit() + expect(fixture.collection.get(1)?.$origin).toBe(`local`) + expect(stripVirtualProps(fixture.collection.get(1))).toEqual({ + ...initial, + a: `accepted`, + }) + } finally { + done.forEach((entry) => entry.resolve()) + await Promise.allSettled([ + first.isPersisted.promise, + second.isPersisted.promise, + ]) + await fixture.collection.cleanup() + } + }, + ) + + it.each([false, true])( + `keeps its captured snapshot until queued confirmation=%s`, + async (confirm) => { + const done = createDeferred() + const fixture = source(() => done.promise) + const downstream = createLiveQueryCollection({ + query: (q) => q.from({ row: fixture.collection }), + }) + await downstream.preload() + const replica = new Map([[1, initial]]) + const subscription = fixture.collection.subscribeChanges((changes) => { + for (const change of changes) { + if (change.type === `delete`) replica.delete(change.key) + else replica.set(change.key, stripVirtualProps(change.value)) + } + }) + const check = (expected: Row) => { + expect(stripVirtualProps(fixture.collection.get(1))).toEqual(expected) + expect(replica.get(1)).toEqual(expected) + expect(stripVirtualProps(downstream.get(1))).toEqual(expected) + } + try { + const tx = fixture.collection.update(1, (row) => { + row.a = `a1` + }) + check({ ...initial, a: `a1` }) + fixture.sync.begin({ immediate: true }) + fixture.sync.write({ type: `update`, value: { ...initial, b: `b1` } }) + expect(fixture.sync.commit()).toBe(true) + check({ ...initial, a: `a1` }) + let applied: true | Promise = true + if (confirm) { + fixture.sync.begin() + fixture.sync.write({ type: `update`, value: { ...initial, a: `a1` } }) + applied = fixture.sync.commit() + expect(applied).not.toBe(true) + } + done.resolve() + await tx.isPersisted.promise + await applied + check({ ...initial, a: `a1` }) + } finally { + done.resolve() + subscription.unsubscribe() + await downstream.cleanup() + await fixture.collection.cleanup() + } + }, + ) + + it.each(orders)( + `selects whole direct snapshots across settlement order %j`, + async (...order) => { + const done = [ + createDeferred(), + createDeferred(), + createDeferred(), + ] + let calls = 0 + const fixture = source(() => done[calls++]!.promise) + const downstream = createLiveQueryCollection({ + query: (q) => q.from({ row: fixture.collection }), + }) + await downstream.preload() + const replica = new Map([[1, initial]]) + const subscription = fixture.collection.subscribeChanges((changes) => { + for (const change of changes) { + if (change.type === `delete`) replica.delete(change.key) + else replica.set(change.key, stripVirtualProps(change.value)) + } + }) + try { + const patches = [{ a: `a1` }, { b: `b2` }, { c: `c3` }] + const transactions = patches.map((patch) => + fixture.collection.update(1, (row) => { + Object.assign(row, patch) + }), + ) + const snapshots = patches.map((_, index) => + Object.assign({}, initial, ...patches.slice(0, index + 1)), + ) + const active = new Set([0, 1, 2]) + for (const index of order) { + done[index]!.resolve() + await transactions[index]!.isPersisted.promise + active.delete(index) + const expected = snapshots[active.size ? Math.max(...active) : index] + expect(stripVirtualProps(fixture.collection.get(1))).toEqual(expected) + expect(replica.get(1)).toEqual(expected) + expect(stripVirtualProps(downstream.get(1))).toEqual(expected) + } + } finally { + for (const pending of done) pending.resolve() + subscription.unsubscribe() + await downstream.cleanup() + await fixture.collection.cleanup() + } + }, + ) + + it.each([`before`, `after`] as const)( + `retains a direct survivor's captured snapshot when it completes %s sibling rollback`, + async (completion) => { + const done = [createDeferred(), createDeferred()] + let calls = 0 + const fixture = source(() => done[calls++]!.promise) + const downstream = createLiveQueryCollection({ + query: (q) => q.from({ row: fixture.collection }), + }) + await downstream.preload() + const replica = new Map([[1, initial]]) + const subscription = fixture.collection.subscribeChanges((changes) => { + for (const change of changes) { + if (change.type === `delete`) replica.delete(change.key) + else replica.set(change.key, stripVirtualProps(change.value)) + } + }) + const check = (expected: Row) => { + expect(stripVirtualProps(fixture.collection.get(1))).toEqual(expected) + expect(replica.get(1)).toEqual(expected) + expect(stripVirtualProps(downstream.get(1))).toEqual(expected) + } + try { + const first = fixture.collection.update(1, (row) => { + row.a = `a1` + }) + const firstSettled = first.isPersisted.promise.catch(() => {}) + const second = fixture.collection.update(1, (row) => { + row.b = `b2` + }) + check({ ...initial, a: `a1`, b: `b2` }) + if (completion === `before`) { + done[1]!.resolve() + await second.isPersisted.promise + check({ ...initial, a: `a1` }) + } + done[0]!.reject(new Error(`first update failed`)) + await firstSettled + check({ ...initial, a: `a1`, b: `b2` }) + if (completion === `after`) { + done[1]!.resolve() + await second.isPersisted.promise + check({ ...initial, a: `a1`, b: `b2` }) + } + fixture.sync.begin() + fixture.sync.write({ type: `update`, value: { ...initial, b: `b2` } }) + await fixture.sync.commit() + check({ ...initial, b: `b2` }) + } finally { + for (const pending of done) pending.resolve() + subscription.unsubscribe() + await downstream.cleanup() + await fixture.collection.cleanup() + } + }, + ) + + it.each(cases)( + `$order / $phase / rollback $removed / $fields`, + async ({ order, phase, removed, fields }) => { + const fixture = source() + const pending = [ + pendingTransaction(), + pendingTransaction(), + pendingTransaction(), + ] + const patches: Array> = + fields === `disjoint` + ? [{ a: `a1` }, { b: `b2` }, { c: `c3` }] + : [{ a: `a1` }, { a: `a2`, b: `b2` }, { c: `c3` }] + const snapshots: Array = [ + undefined, + undefined, + undefined, + ] + const expected = (excluded = -1) => + snapshots.reduce( + (last, row, index) => + row !== undefined && index !== excluded ? row : last, + initial, + ) + try { + await fixture.collection.preload() + for (const index of order) { + snapshots[index] = { ...expected(), ...patches[index] } + pending[index]!.tx.mutate(() => + fixture.collection.update(1, (row) => { + Object.assign(row, patches[index]) + }), + ) + } + if (phase === `persisting`) { + for (const entry of pending) void entry.tx.commit().catch(() => {}) + } + expect(stripVirtualProps(fixture.collection.get(1))).toEqual(expected()) + // Isolate one rollback's projection; conflict-cascade policy is unchanged. + pending[removed]!.tx.rollback({ isSecondaryRollback: true }) + expect(stripVirtualProps(fixture.collection.get(1))).toEqual( + expected(removed), + ) + } finally { + for (const entry of pending) { + entry.tx.rollback({ isSecondaryRollback: true }) + entry.done.resolve() + } + await Promise.all(pending.map((entry) => entry.settled)) + await fixture.collection.cleanup() + } + }, + ) + + it.each([`pending`, `persisting`] as const)( + `does not merge new synced fields into a $phase mutation snapshot`, + async (phase) => { + const fixture = source() + const entry = pendingTransaction() + try { + await fixture.collection.preload() + entry.tx.mutate(() => + fixture.collection.update(1, (row) => { + row.a = `local` + }), + ) + if (phase === `persisting`) void entry.tx.commit().catch(() => {}) + // Exercise the existing explicit immediate path, not a new queue policy. + fixture.sync.begin({ immediate: phase === `persisting` }) + fixture.sync.write({ + type: `update`, + value: { ...initial, b: `remote` }, + }) + expect(fixture.sync.commit()).toBe(true) + expect(stripVirtualProps(fixture.collection.get(1))).toEqual({ + ...initial, + a: `local`, + }) + } finally { + entry.tx.rollback() + entry.done.resolve() + await entry.settled + await fixture.collection.cleanup() + } + }, + ) + + it(`keeps a settled direct update beneath an older pending update`, async () => { + const fixture = source() + const entry = pendingTransaction() + try { + await fixture.collection.preload() + entry.tx.mutate(() => + fixture.collection.update(1, (row) => { + row.a = `local` + }), + ) + void entry.tx.commit().catch(() => {}) + const settled = fixture.collection.update(1, (row) => { + row.b = `settled` + }) + await settled.isPersisted.promise + fixture.sync.begin() + fixture.sync.write({ type: `insert`, value: { ...initial, id: 2 } }) + expect(fixture.sync.commit()).not.toBe(true) + expect(stripVirtualProps(fixture.collection.get(1))).toEqual({ + ...initial, + a: `local`, + }) + } finally { + entry.tx.rollback() + entry.done.resolve() + await entry.settled + await fixture.collection.cleanup() + } + }) + + it(`preserves insert defaults and suppresses unchanged overlay publications`, async () => { + const collection = createCollection({ + schema: z.object({ + id: z.number(), + title: z.string(), + priority: z.number().default(3), + }), + getKey: (row) => row.id, + sync: { + sync: ({ begin, commit, markReady }) => { + begin() + commit() + markReady() + }, + }, + }) + const entries = [ + pendingTransaction(), + pendingTransaction(), + pendingTransaction(), + ] + try { + await collection.preload() + entries[0]!.tx.mutate(() => collection.insert({ id: 1, title: `first` })) + entries[1]!.tx.mutate(() => + collection.update(1, (row) => { + row.title = `second` + }), + ) + expect(stripVirtualProps(collection.get(1))).toEqual({ + id: 1, + title: `second`, + priority: 3, + }) + const before = collection.get(1) + const seen: Array = [] + const subscription = collection.subscribeChanges((changes) => { + seen.push(...changes.filter((change) => change.key === 1)) + }) + try { + entries[2]!.tx.mutate(() => + collection.insert({ id: 2, title: `unrelated` }), + ) + expect(collection.get(1)).toBe(before) + expect(seen).toEqual([]) + } finally { + subscription.unsubscribe() + } + } finally { + for (const entry of entries) { + entry.tx.rollback({ isSecondaryRollback: true }) + entry.done.resolve() + } + await Promise.all(entries.map((entry) => entry.settled)) + await collection.cleanup() + } + }) +}) diff --git a/packages/db/tests/optimistic-history-oracle.ts b/packages/db/tests/optimistic-history-oracle.ts new file mode 100644 index 0000000000..44bcc6a20b --- /dev/null +++ b/packages/db/tests/optimistic-history-oracle.ts @@ -0,0 +1,428 @@ +import { expect } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { createLiveQueryCollection } from '../src/query/index.js' +import type { SyncConfig } from '../src/types.js' + +export type HistoryRow = { id: number; a: number; b: number; c: number } +type Fields = Partial> +export type OptimisticStep = + | { type: `edit`; key: number; fields: Fields; optimistic: boolean } + | { type: `settle`; slot: number; success: boolean; cascade: boolean } + | { + type: `sync` + rows: Array + truncate: boolean + immediate: boolean + copies: number + } + +type Intent = { + key: number + kind: `insert` | `update` + fields: Fields + snapshot: HistoryRow + optimistic: boolean + dependency?: number + state: `active` | `accepted` | `failed` + settled: number + retired: boolean + acknowledged: boolean + originPending: boolean +} +type ObservedRow = HistoryRow & { + $origin: `local` | `remote` + $synced: boolean +} + +/** Specification state is an event history, never a copy of production caches. + * Mutations own whole-row snapshots, never patches over changing synced rows. + * Accepted snapshots precede active snapshots. An insert supplies row existence; + * accepted updates dependent on it survive its success, but not its failed birth. + */ +class HistoryModel { + base = new Map() + origins = new Map() + intents: Array = [] + queue: Array> = [] + clock = 0 + + constructor(rows: Array) { + for (const row of rows) { + this.base.set(row.id, row) + this.origins.set(row.id, `remote`) + } + } + + private retained(intent: Intent) { + return ( + intent.state === `accepted` && + !intent.retired && + intent.optimistic && + (intent.dependency === undefined || + this.intents[intent.dependency]!.state !== `failed` || + this.intents[intent.dependency]!.acknowledged) + ) + } + + visible(): Map { + const result = new Map( + [...this.base].map(([key, row]) => [ + key, + { ...row, $origin: this.origins.get(key)!, $synced: true }, + ]), + ) + const accepted = this.intents + .filter((intent) => this.retained(intent)) + .sort((a, b) => a.settled - b.settled) + const active = this.intents.filter( + (intent) => intent.state === `active` && intent.optimistic, + ) + const apply = (intent: Intent) => { + result.set(intent.key, { + ...intent.snapshot, + $origin: `local`, + $synced: false, + }) + if (intent.kind === `insert` && !intent.acknowledged) { + // An accepted dependent snapshot belongs after its creating insert even + // when transport completion occurs in the opposite order. + for (const child of accepted) { + if (child.dependency === this.intents.indexOf(intent)) { + result.set(intent.key, { + ...child.snapshot, + $origin: `local`, + $synced: false, + }) + } + } + } + } + for (const intent of [...accepted, ...active]) apply(intent) + return result + } + + edit(step: Extract): number | undefined { + const row = this.visible().get(step.key) + if ( + row && + Object.entries(step.fields).every( + ([key, value]) => row[key as keyof Fields] === value, + ) + ) + return + const kind = row ? `update` : `insert` + const snapshot = { + ...(row ?? { id: step.key, a: 0, b: 0, c: 0 }), + ...step.fields, + } + const dependency = this.intents.reduce( + (previous, intent, index) => + kind === `update` && + intent.key === step.key && + intent.kind === `insert` && + intent.optimistic && + intent.state === `active` && + !intent.acknowledged + ? index + : previous, + -1, + ) + this.intents.push({ + key: step.key, + kind, + fields: step.fields, + snapshot, + optimistic: step.optimistic, + dependency: dependency < 0 ? undefined : dependency, + state: `active`, + settled: 0, + retired: false, + acknowledged: false, + originPending: false, + }) + return this.intents.length - 1 + } + + settle(index: number, success: boolean) { + const intent = this.intents[index]! + // A separately submitted update may succeed after the insert has already + // failed. That later server acceptance is not undone by an earlier failure. + if ( + success && + intent.dependency !== undefined && + this.intents[intent.dependency]!.state === `failed` + ) + intent.dependency = undefined + intent.state = success ? `accepted` : `failed` + if (intent.kind === `insert` && intent.acknowledged) intent.retired = true + intent.settled = ++this.clock + // A synced insert has already spent its acknowledgement. Completing its + // transport cannot turn the next unrelated remote write into a local one. + if (success && !(intent.kind === `insert` && intent.acknowledged)) + intent.originPending = true + // This grammar submits direct operations immediately. Rollback cascades + // affect pending (not already persisting) peer transactions, so none of + // these independently submitted requests is canceled by a sibling failure. + if (!this.intents.some((entry) => entry.state === `active`)) this.drain() + } + + sync(step: Extract) { + this.queue.push(step) + if ( + step.immediate || + step.truncate || + !this.intents.some((entry) => entry.state === `active`) + ) + this.drain() + } + + private drain() { + if (!this.queue.length) return + const localKeys = new Set( + this.intents + .filter((intent) => intent.state === `active`) + .map((intent) => intent.key), + ) + const replaced = this.queue.some((batch) => batch.truncate) + const written = new Set( + this.queue.flatMap((batch) => batch.rows.map((row) => row.id)), + ) + const retainedKeys = new Set( + this.intents + .filter((intent) => this.retained(intent)) + .map((intent) => intent.key), + ) + for (const batch of this.queue) { + if (batch.truncate) { + this.base.clear() + this.origins.clear() + // Origin is row-level attribution, not per-mutation acknowledgement. + // A truncate replacement of a retained optimistic row is remote unless + // a still-active request also owns that key. Do not invent finer + // acknowledgement matching between completed same-key requests. + for (const intent of this.intents) + if (intent.state === `accepted` && retainedKeys.has(intent.key)) + intent.originPending = false + } + for (const row of batch.rows) { + const local = + this.intents.some( + (intent) => intent.key === row.id && intent.originPending, + ) || localKeys.has(row.id) + this.base.set(row.id, row) + this.origins.set(row.id, local ? `local` : `remote`) + localKeys.delete(row.id) + for (const intent of this.intents) { + if (intent.key === row.id) intent.originPending = false + if ( + intent.key === row.id && + intent.kind === `insert` && + intent.state === `active` + ) + intent.acknowledged = true + } + } + // Truncate retains attribution only for rows in its own replacement, + // not for an unrelated future write after the old source was cleared. + if (batch.truncate) + for (const intent of this.intents) intent.originPending = false + } + // Ordinary source publication retires completed direct snapshots, including + // temporary keys. Truncate preserves snapshots omitted from its replacement. + for (const intent of this.intents) + if (intent.state === `accepted`) { + intent.retired ||= !replaced || written.has(intent.key) + if (retainedKeys.has(intent.key)) intent.originPending = false + } + this.queue = [] + } +} + +const plain = ({ id, a, b, c }: HistoryRow): HistoryRow => ({ id, a, b, c }) +const observed = ( + row: HistoryRow & { $origin: `local` | `remote`; $synced: boolean }, +): ObservedRow => ({ + ...plain(row), + $origin: row.$origin, + $synced: row.$synced, +}) +const sorted = (rows: Iterable) => + [...rows].sort((a, b) => a.id - b.id) + +export async function runOptimisticHistory( + initial: Array, + steps: ReadonlyArray, +) { + const model = new HistoryModel(initial) + let sync!: Parameters[`sync`]>[0] + let starting: ReturnType> | undefined + const handler = () => starting!.promise + const collection = createCollection({ + getKey: (row) => row.id, + onInsert: handler, + onUpdate: handler, + sync: { + rowUpdateMode: `full`, + sync: (actions) => { + sync = actions + actions.begin() + for (const row of initial) + actions.write({ type: `insert`, value: { ...row } }) + actions.commit() + actions.markReady() + }, + }, + }) + const downstream = createLiveQueryCollection({ + query: (q) => q.from({ row: collection }), + }) + await downstream.preload() + const replica = new Map( + [...collection.values()].map((row) => [row.id, observed(row)]), + ) + let deliveries = 0 + const sub = collection.subscribeChanges( + (batch) => { + for (const change of batch) { + deliveries++ + if (change.type === `delete`) replica.delete(Number(change.key)) + else replica.set(Number(change.key), observed(change.value)) + } + }, + { includeInitialState: true }, + ) + const operations: Array<{ + tx: + | ReturnType + | ReturnType + done: ReturnType> + outcome: Promise + }> = [] + const receipts: Array> = [] + const counts = { + edits: 0, + settlements: 0, + replacements: 0, + queued: 0, + dependencies: 0, + failures: 0, + snapshotOverrides: 0, + } + const check = (label: string) => { + for (const [index, operation] of operations.entries()) { + expect( + operation.tx.mutations[0]!.modified, + `${label}: immutable request ${index}`, + ).toMatchObject(plain(model.intents[index]!.snapshot)) + } + const expected = sorted(model.visible().values()) + const actual = sorted([...collection.values()].map(observed)) + expect( + actual, + `${label}: reads ${JSON.stringify(actual)} expected ${JSON.stringify(expected)}`, + ).toEqual(expected) + expect(sorted(replica.values()), `${label}: event replica`).toEqual( + expected, + ) + expect( + sorted([...downstream.values()].map(plain)), + `${label}: downstream`, + ).toEqual(expected.map(plain)) + } + try { + check(`initial`) + for (const [position, step] of steps.entries()) { + const before = sorted(model.visible().values()) + const deliveredBefore = deliveries + if (step.type === `edit`) { + const index = model.edit(step) + if (index === undefined) continue + const intent = model.intents[index]! + const done = createDeferred() + starting = done + const tx = + intent.kind === `insert` + ? collection.insert(plain(intent.snapshot), { + optimistic: step.optimistic, + }) + : collection.update( + step.key, + { optimistic: step.optimistic }, + (draft) => Object.assign(draft, step.fields), + ) + operations.push({ + tx, + done, + outcome: tx.isPersisted.promise.catch((error: unknown) => error), + }) + expect( + tx.mutations[0]!.modified, + `captured request snapshot`, + ).toMatchObject(plain(intent.snapshot)) + counts.edits++ + if (intent.dependency !== undefined) counts.dependencies++ + } else if (step.type === `settle`) { + const active = model.intents.flatMap((intent, index) => + intent.state === `active` ? [index] : [], + ) + if (!active.length) continue + const index = active[step.slot % active.length]! + const op = operations[index]! + model.settle(index, step.success) + if (!step.success) + op.tx.rollback({ isSecondaryRollback: !step.cascade }) + op.done.resolve() + await op.outcome + await Promise.resolve() + counts.settlements++ + if (!step.success) counts.failures++ + } else { + model.sync(step) + sync.begin({ immediate: step.immediate }) + if (step.truncate) { + sync.truncate() + counts.replacements++ + } + for (let copy = 0; copy < step.copies; copy++) { + for (const row of step.rows) + sync.write({ type: `update`, value: { ...row } }) + } + const receipt = sync.commit() + if (receipt !== true) { + receipts.push(receipt.catch((error: unknown) => error)) + counts.queued++ + } + if ( + (step.immediate || step.truncate) && + model.intents.some((intent) => intent.state === `active`) + ) + counts.snapshotOverrides++ + } + check(`${position}: ${JSON.stringify(step)}`) + // Count events as well as final values; value-only oracles miss redundant + // publications when a mutation moves into completed retention. + if ( + step.type === `settle` && + JSON.stringify(before) === + JSON.stringify(sorted(model.visible().values())) + ) { + expect(deliveries, `unchanged settlement ${position}`).toBe( + deliveredBefore, + ) + } + } + return counts + } finally { + for (const op of operations) { + if (op.tx.state === `pending` || op.tx.state === `persisting`) + op.tx.rollback({ isSecondaryRollback: true }) + op.done.resolve() + } + await Promise.all(operations.map((op) => op.outcome)) + sub.unsubscribe() + await downstream.cleanup() + await collection.cleanup() + await Promise.all(receipts) + } +} diff --git a/packages/db/tests/optimistic-settlement-boundaries.test.ts b/packages/db/tests/optimistic-settlement-boundaries.test.ts new file mode 100644 index 0000000000..409c58930b --- /dev/null +++ b/packages/db/tests/optimistic-settlement-boundaries.test.ts @@ -0,0 +1,150 @@ +import { it } from 'vitest' +import { runOptimisticHistory } from './optimistic-history-oracle.js' +import type { HistoryRow, OptimisticStep } from './optimistic-history-oracle.js' + +const row: HistoryRow = { id: 1, a: 0, b: 0, c: 0 } +const edit = ( + fields: Partial>, + optimistic = true, +): OptimisticStep => ({ type: `edit`, key: 1, fields, optimistic }) +const settle = (slot: number, success = true): OptimisticStep => ({ + type: `settle`, + slot, + success, + cascade: false, +}) +const sync = ( + rows: Array, + truncate = false, + immediate = false, + copies = 1, +): OptimisticStep => ({ type: `sync`, rows, truncate, immediate, copies }) + +// Every regression is a program for the same model, driver and checkpoint +// assertions used by generated histories. Membership work has its own generated +// law in query/derived-delete-reconciliation.test.ts. +const cases: Array<{ + name: string + initial: Array + steps: Array +}> = [ + { + name: `one confirmation for repeated queued writes`, + initial: [row], + steps: [ + edit({ a: 1 }), + sync([{ ...row, a: 1 }], false, false, 2), + settle(0), + ], + }, + { + name: `an acknowledged insert cannot remove an accepted update`, + initial: [], + steps: [ + edit({ a: 1 }), + sync([{ ...row, a: 1 }], false, true), + edit({ b: 2 }), + settle(1), + settle(0, false), + ], + }, + { + name: `failed insertion cannot remove later same-key snapshots`, + initial: [], + steps: [ + edit({ a: 1 }), + edit({ b: 1 }), + settle(1), + settle(0, false), + edit({ a: 2 }), + edit({ b: 2 }), + settle(1), + settle(0), + ], + }, + ...[true, false].map((success) => ({ + name: `dependent snapshot follows insert settlement: ${success}`, + initial: [], + steps: [edit({ a: 1 }), edit({ b: 2 }), settle(1), settle(0, success)], + })), + ...[true, false].map((truncate) => ({ + name: `failed birth does not erase accepted sibling attribution: ${truncate}`, + initial: [], + steps: [ + edit({ a: 1 }), + edit({ b: 2 }), + settle(1), + settle(0, false), + sync([row], truncate), + ], + })), + { + name: `a later accepted update survives an earlier failed insertion`, + initial: [], + steps: [edit({ a: 1 }), edit({ b: 2 }), settle(0, false), settle(0)], + }, + ...[true, false].map((truncate) => ({ + name: `sync retirement precedes rebuilding active snapshots: ${truncate}`, + initial: [], + steps: [ + edit({ a: 1 }), + edit({ b: 2 }), + settle(1), + sync([], truncate, true), + settle(0, false), + ], + })), + { + name: `unchanged persistence completion does not publish`, + initial: [row], + steps: [edit({ a: 1 }), settle(0)], + }, + { + name: `truncate preserves the captured whole row`, + initial: [row], + steps: [edit({ a: 1 }), sync([{ ...row, b: 2 }], true), settle(0)], + }, + { + name: `rollback does not rewrite a later captured request`, + initial: [row], + steps: [edit({ a: 1 }), edit({ b: 2 }), settle(0, false), settle(0)], + }, + { + name: `duplicate writes retain a local acknowledgement within a batch`, + initial: [], + steps: [edit({ a: 1 }, false), sync([{ ...row, a: 1 }], false, true, 2)], + }, + { + name: `a persisting peer does not hide rollback publication`, + initial: [], + steps: [edit({ a: 1 }, false), edit({ a: 2 }), sync([]), settle(1, false)], + }, + { + name: `truncate retains pending nonoptimistic attribution`, + initial: [], + steps: [ + edit({ a: 1 }, false), + sync([], true), + sync([{ ...row, a: 1 }], false, true), + ], + }, + { + name: `completed sibling does not erase active truncate attribution`, + initial: [], + steps: [ + edit({ a: 1 }, false), + edit({ a: 2 }), + settle(1), + sync([{ ...row, a: 1 }], true), + ], + }, + { + name: `removal of a truncate-retained snapshot reaches subscribers`, + initial: [], + steps: [edit({ a: 1 }), settle(0), sync([], true), sync([])], + }, +] + +it.each(cases)(`oracle replay: $name`, async ({ initial, steps }) => { + await runOptimisticHistory(initial, steps) +}) diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts new file mode 100644 index 0000000000..e04134b99d --- /dev/null +++ b/packages/db/tests/oracle-config.ts @@ -0,0 +1,263 @@ +type OracleEnvironment = Record + +const staticOracleProperties = [ + `trailbase.lifecycle`, + `electric.bound-descriptor-history`, + `electric.persisted-tag-history`, + `electric.match-reentry`, + `collection-sync.reentrant-drain`, + `collection-state.retention`, + `collection-state.optimistic-history`, + `derived-publication.membership-work`, + `collection-publication.metadata-cancellation`, + `collection-publication.metadata-only`, + `collection-publication.metadata-rollback`, + `coverage-registry.claim-churn`, + `coverage-registry.state-machine`, + `d2-source.exact-retractions`, + `d2-source.disjoint-commutation`, + `includes-collection.layout-swap`, + `includes-collection.optimistic-child-history`, + `includes-collection.public-key-order`, + `includes-collection.relationship-history`, + `includes-cross-formulation.equivalence`, + `includes-cross-formulation.ordered-window`, + `includes-cross-formulation.reference-context`, + `includes-cross-formulation.reference-key`, + `includes-cross-formulation.symbol-group-route`, + `includes-optimistic.ancestor-rollback`, + `includes-optimistic.confirm-different-route`, + `includes-optimistic.confirm-same-route`, + `includes-optimistic.descendant-rollback`, + `includes-optimistic.rekey-detach`, + `includes-optimistic.rekey-rollback`, + `includes-optimistic.repeated-history`, + `includes-optimistic.sibling-route-rollback`, + `includes-publication.atomic-parent-replacement`, + `includes-publication.child-scalar`, + `includes-publication.optimistic-rollback`, + `includes-publication.parent-route`, + `includes-temporal.release-reentry`, + `includes-temporal.demand-scheduling`, + `includes.alpha-renaming`, + `includes.incremental-history`, + `includes.nested-scalar-materialization`, + `includes.optimistic-convergence`, + `includes.scenario-statistics`, + `load-subset-full-flow.atomic-replacement`, + `load-subset-full-flow.automatic-progress`, + `load-subset-full-flow.boundary-provenance`, + `load-subset-full-flow.consumer-parity`, + `load-subset-full-flow.continuation-evidence`, + `load-subset-full-flow.continuation-statistics`, + `load-subset-full-flow.multi-source-ordered`, + `load-subset-full-flow.multi-source-statistics`, + `load-subset-full-flow.truncate-evidence`, + `load-subset-lifecycle.state-machine`, + `load-subset.async-settlement`, + `load-subset.changing-predicate`, + `load-subset.concurrent-dedupe`, + `load-subset.coverage`, + `load-subset.distinct-window-predicate`, + `load-subset.exact-completion`, + `load-subset.exact-inflight`, + `load-subset.ordered-window`, + `load-subset.rejected-waiter`, + `ordered-work.forward-exhaustion`, + `ordered-work.forward-prefix`, + `ordered-work.custom-comparator-fallback`, + `ordered-work.public-key-suffix`, + `ordered-work.reverse-exhaustion`, + `ordered-work.reverse-prefix`, + `ordered-work.snapshot-reuse`, + `ordered-work.consumer-parity`, + `ordered-work.lifecycle`, + `pagination.async-cursor`, + `pagination.multi-order`, + `pagination.nullable-cursor`, + `pagination.ordered-window`, + `pagination.pending-history`, + `pagination.pending-mutation`, + `pagination.window-transition`, + `predicate-subtraction.duplicate-terms`, + `predicate-subtraction.finite-world`, + `predicate-subtraction.unbounded`, + `subscription-replay.completion`, + `subscription-replay.optimistic`, + `subscription-replay.ownership`, + `subscription-replay.restart`, + `subscription-replay.sequential`, + `subscription-replay.shared`, + `subscription-replay.same-tick`, + `subscription-lifecycle.history-statistics`, + `subscription-lifecycle.publication-history`, + `subscription-lifecycle.sync-history`, + `subscription-lifecycle.async-history`, + `subscription-lifecycle.async-restart`, + `subscription-lifecycle.async-statistics`, +] as const + +const publicationProperties = [ + `parent-scalar`, + `parent-then-child`, + `optimistic-before-confirm`, + `optimistic-after-confirm`, +].flatMap((law) => + [`direct`, `joined`].flatMap((q1Shape) => + [`passThrough`, `where`, `orderBy`, `select`].map( + (q2Shape) => `includes-publication.${law}.${q1Shape}.${q2Shape}`, + ), + ), +) + +const refinementProperties = Array.from( + { length: 11 }, + (_, index) => `load-subset-refinement.${1_779_001 + index}`, +) + +export function validateOraclePropertyRegistry( + properties: ReadonlyArray, +): ReadonlySet { + const registry = new Set() + for (const property of properties) { + if (registry.has(property)) { + throw new Error(`duplicate oracle property: ${property}`) + } + registry.add(property) + } + return registry +} + +const registeredOracleProperties = validateOraclePropertyRegistry([ + ...staticOracleProperties, + ...publicationProperties, + ...refinementProperties, +]) + +function assertRegisteredOracleProperty(property: string): void { + if (!registeredOracleProperties.has(property)) { + throw new Error(`unknown oracle property: ${property}`) + } +} + +export type OracleReplayConfig = { + replaySeed: number | undefined + replayPath: string | undefined + replayProperty: string | undefined +} + +export function readOracleRunConfig( + environment: OracleEnvironment = process.env, +): OracleReplayConfig & { multiplier: number } { + const multiplierValue = environment.TANSTACK_DB_ORACLE_RUNS_MULTIPLIER ?? `1` + const multiplier = Number(multiplierValue) + if ( + multiplierValue.trim() === `` || + !Number.isSafeInteger(multiplier) || + multiplier < 1 + ) { + throw new Error( + `TANSTACK_DB_ORACLE_RUNS_MULTIPLIER must be a positive integer`, + ) + } + + const seedValue = environment.TANSTACK_DB_ORACLE_SEED + const replayPath = environment.TANSTACK_DB_ORACLE_PATH + const replayProperty = environment.TANSTACK_DB_ORACLE_PROPERTY + if (seedValue === undefined) { + if (replayPath !== undefined) { + throw new Error( + `TANSTACK_DB_ORACLE_PATH requires TANSTACK_DB_ORACLE_SEED`, + ) + } + if (replayProperty !== undefined) { + throw new Error( + `TANSTACK_DB_ORACLE_PROPERTY requires TANSTACK_DB_ORACLE_PATH`, + ) + } + return { + multiplier, + replaySeed: undefined, + replayPath: undefined, + replayProperty: undefined, + } + } + + const replaySeed = Number(seedValue) + if (seedValue.trim() === `` || !Number.isSafeInteger(replaySeed)) { + throw new Error(`TANSTACK_DB_ORACLE_SEED must be an integer`) + } + if (replayPath === undefined) { + if (replayProperty !== undefined) { + throw new Error( + `TANSTACK_DB_ORACLE_PROPERTY requires TANSTACK_DB_ORACLE_PATH`, + ) + } + return { + multiplier, + replaySeed, + replayPath: undefined, + replayProperty: undefined, + } + } + if (replayPath.trim() === ``) { + throw new Error(`TANSTACK_DB_ORACLE_PATH must be non-empty`) + } + if (!/^\d+(?::\d+)*$/.test(replayPath)) { + throw new Error( + `TANSTACK_DB_ORACLE_PATH must contain colon-separated nonnegative integers`, + ) + } + if (replayProperty === undefined || replayProperty.trim() === ``) { + throw new Error( + `TANSTACK_DB_ORACLE_PATH requires TANSTACK_DB_ORACLE_PROPERTY`, + ) + } + assertRegisteredOracleProperty(replayProperty) + return { multiplier, replaySeed, replayPath, replayProperty } +} + +export function oracleRandomParameters( + numRuns: number, + replay: OracleReplayConfig | number | undefined, + property?: string, +): { numRuns: number; seed?: number; path?: string } { + if (property !== undefined) assertRegisteredOracleProperty(property) + const { replaySeed, replayPath, replayProperty } = + typeof replay === `object` + ? replay + : { + replaySeed: replay, + replayPath: undefined, + replayProperty: undefined, + } + if (replaySeed === undefined) return { numRuns } + return { + numRuns, + seed: replaySeed, + ...(property !== undefined && + replayPath !== undefined && + replayProperty === property + ? { path: replayPath } + : {}), + } +} + +const { multiplier, ...replay } = readOracleRunConfig() + +/** Keeps ordinary CI bounded while allowing long randomized oracle campaigns. */ +export function oracleRuns(baseRuns: number): number { + return baseRuns * multiplier +} + +/** Replays broad randomized properties when a campaign seed is supplied. */ +export function oraclePropertyOptions( + baseRuns: number, + property?: string, +): { + numRuns: number + seed?: number + path?: string +} { + return oracleRandomParameters(oracleRuns(baseRuns), replay, property) +} diff --git a/packages/db/tests/orphaned-live-query-gc.test.ts b/packages/db/tests/orphaned-live-query-gc.test.ts new file mode 100644 index 0000000000..449d448a22 --- /dev/null +++ b/packages/db/tests/orphaned-live-query-gc.test.ts @@ -0,0 +1,82 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createLiveQueryCollection } from '../src/query/live-query-collection.js' +import { mockSyncCollectionOptions, resetCleanupQueue } from './utils.js' + +type Person = { id: string; name: string } + +const collections: Array<{ cleanup: () => Promise }> = [] + +const makeSource = () => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `orphan-gc-source`, + getKey: (person) => person.id, + initialData: [{ id: `1`, name: `Alice` }], + }), + ) + collections.push(source) + return source +} + +describe(`live query collections that never gain a subscriber`, () => { + beforeEach(() => { + vi.useFakeTimers() + resetCleanupQueue() + }) + + afterEach(async () => { + for (const collection of collections.reverse()) await collection.cleanup() + collections.length = 0 + await Promise.resolve() + resetCleanupQueue() + vi.useRealTimers() + }) + + it(`releases every orphan's source subscription after the grace period`, async () => { + const source = makeSource() + const orphans = Array.from({ length: 3 }, () => { + const orphan = createLiveQueryCollection({ + startSync: true, + gcTime: 1, + query: (q) => q.from({ person: source }), + }) + collections.push(orphan) + return orphan + }) + + expect(source.subscriberCount).toBe(orphans.length) + await vi.advanceTimersByTimeAsync(51) + + expect(orphans.map((orphan) => orphan.status)).toEqual([ + `cleaned-up`, + `cleaned-up`, + `cleaned-up`, + ]) + expect(source.subscriberCount).toBe(0) + }) + + it(`stops evaluating a reclaimed query when its source changes`, async () => { + const source = makeSource() + const project = vi.fn(({ person }: { person: Person }) => ({ ...person })) + const orphan = createLiveQueryCollection({ + startSync: true, + gcTime: 1, + query: (q) => q.from({ person: source }).fn.select(project), + }) + collections.push(orphan) + expect(project).toHaveBeenCalled() + + await vi.advanceTimersByTimeAsync(51) + expect(orphan.status).toBe(`cleaned-up`) + project.mockClear() + + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `2`, name: `Bob` } }) + source.utils.commit() + + expect(source.size).toBe(2) + expect(project).not.toHaveBeenCalled() + expect(source.subscriberCount).toBe(0) + }) +}) diff --git a/packages/db/tests/preload-gc.test.ts b/packages/db/tests/preload-gc.test.ts new file mode 100644 index 0000000000..453bda5fa9 --- /dev/null +++ b/packages/db/tests/preload-gc.test.ts @@ -0,0 +1,167 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index' + +describe(`preload retention`, () => { + const collections: Array<{ cleanup: () => Promise }> = [] + + beforeEach(() => vi.useFakeTimers()) + afterEach(async () => { + for (const collection of collections.splice(0)) await collection.cleanup() + vi.useRealTimers() + }) + + function makeCollection(startSync = false, gcTime = 10) { + let ready!: () => void + let fail!: (error: Error) => void + const cleanup = vi.fn() + const collection = createCollection<{ id: number }>({ + getKey: (row) => row.id, + gcTime, + startSync, + sync: { + sync: ({ begin, write, commit, markReady, markError }) => { + ready = () => { + begin() + write({ type: `insert`, value: { id: 1 } }) + commit() + markReady() + } + fail = markError + return cleanup + }, + }, + }) + collections.push(collection) + return { + collection, + ready: () => ready(), + fail: (error: Error) => fail(error), + cleanup, + } + } + + it.each([false, true])( + `retains pending data and grants a fresh grace period after readiness (already syncing: %s)`, + async (startSync) => { + const { collection, ready, cleanup } = makeCollection(startSync) + const pending = collection.preload() + const outcome = pending.then( + () => `ready`, + () => `aborted`, + ) + expect(collection.preload()).toBe(pending) + + await vi.advanceTimersByTimeAsync(1000) + expect(collection.status).toBe(`loading`) + expect(cleanup).not.toHaveBeenCalled() + + ready() + await expect(outcome).resolves.toBe(`ready`) + expect(collection.size).toBe(1) + await vi.advanceTimersByTimeAsync(49) + expect(collection.status).toBe(`ready`) + await vi.advanceTimersByTimeAsync(2) + expect(collection.status).toBe(`cleaned-up`) + expect(cleanup).toHaveBeenCalledOnce() + }, + ) + + it.each([ + { cached: false, gcTime: 10 }, + { cached: true, gcTime: 10 }, + { cached: false, gcTime: 100 }, + { cached: true, gcTime: 100 }, + ])( + `renews the full warm-preload grace interval (cached: $cached, gcTime: $gcTime)`, + async ({ cached, gcTime }) => { + const { collection, ready, cleanup } = makeCollection(true, gcTime) + ready() + const previousPreload = cached ? collection.preload() : undefined + await previousPreload + const graceTime = Math.max(50, gcTime) + await vi.advanceTimersByTimeAsync(graceTime - 1) + + const preload = collection.preload() + if (cached) expect(preload).toBe(previousPreload) + expect(collection.preload()).toBe(preload) + await preload + + await vi.advanceTimersByTimeAsync(graceTime - 1) + expect(collection.status).toBe(`ready`) + expect(collection.size).toBe(1) + expect(cleanup).not.toHaveBeenCalled() + await vi.advanceTimersByTimeAsync(2) + expect(collection.status).toBe(`cleaned-up`) + expect(cleanup).toHaveBeenCalledOnce() + }, + ) + + it(`cancels queued idle cleanup when an already-ready collection is preloaded`, async () => { + const { collection, ready, cleanup } = makeCollection(true) + ready() + await Promise.resolve() + vi.advanceTimersByTime(50) + + await collection.preload() + + await vi.advanceTimersByTimeAsync(49) + expect(collection.status).toBe(`ready`) + expect(cleanup).not.toHaveBeenCalled() + await vi.advanceTimersByTimeAsync(2) + expect(collection.status).toBe(`cleaned-up`) + expect(cleanup).toHaveBeenCalledOnce() + }) + + it(`retains a pending preload when the last subscriber leaves`, async () => { + const { collection, ready } = makeCollection() + const subscription = collection.subscribeChanges(() => {}) + const pending = collection.preload() + const outcome = pending.then( + () => `ready`, + () => `aborted`, + ) + subscription.unsubscribe() + await vi.advanceTimersByTimeAsync(1000) + expect(collection.status).toBe(`loading`) + ready() + await expect(outcome).resolves.toBe(`ready`) + }) + + it(`cancels idle cleanup when preload starts after the GC deadline`, async () => { + const { collection, ready } = makeCollection(true) + await Promise.resolve() + // Run the GC timer, leaving its destructive idle callback queued. + vi.advanceTimersByTime(50) + const outcome = collection.preload().then( + () => `ready`, + () => `aborted`, + ) + await vi.advanceTimersByTimeAsync(1000) + expect(collection.status).toBe(`loading`) + ready() + await expect(outcome).resolves.toBe(`ready`) + }) + + it(`reclaims an unused collection after preload fails`, async () => { + const { collection, fail, cleanup } = makeCollection() + const failure = new Error(`source failed`) + const outcome = collection.preload().catch((error: unknown) => error) + await vi.advanceTimersByTimeAsync(1000) + expect(collection.status).toBe(`loading`) + fail(failure) + await expect(outcome).resolves.toBe(failure) + await vi.advanceTimersByTimeAsync(51) + expect(collection.status).toBe(`cleaned-up`) + expect(cleanup).toHaveBeenCalledOnce() + }) + + it(`allows explicit cleanup to abort preload without scheduling another GC`, async () => { + const { collection, cleanup } = makeCollection() + const outcome = collection.preload().catch((error: unknown) => error) + await collection.cleanup() + await expect(outcome).resolves.toMatchObject({ name: `AbortError` }) + await vi.advanceTimersByTimeAsync(1000) + expect(cleanup).toHaveBeenCalledOnce() + expect(vi.getTimerCount()).toBe(0) + }) +}) diff --git a/packages/db/tests/proxy-detachment-contract.test.ts b/packages/db/tests/proxy-detachment-contract.test.ts new file mode 100644 index 0000000000..0b88283d50 --- /dev/null +++ b/packages/db/tests/proxy-detachment-contract.test.ts @@ -0,0 +1,241 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { withChangeTracking } from '../src/proxy.js' + +class Label { + #text: string + constructor(text: string) { + this.#text = text + } + read() { + return this.#text + } + rename(text: string) { + this.#text = text + } +} + +function storedRow(row: T & { id: number }) { + return createCollection({ + getKey: (value) => value.id, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: row }) + commit() + markReady() + }, + }, + onUpdate: () => Promise.resolve(), + }) +} + +describe(`Mutation result detachment`, () => { + it(`keeps arbitrary class instances by reference as an explicit isolation exception`, async () => { + const label = new Label(`before`) + const collection = storedRow({ id: 1, value: undefined as unknown }) + try { + const tx = collection.update(1, (draft) => { + draft.value = label + }) + expect(collection.get(1)!.value).toBe(label) + label.rename(`after`) + expect((collection.get(1)!.value as Label).read()).toBe(`after`) + await tx.isPersisted.promise + } finally { + await collection.cleanup() + } + }) + + it(`preserves and detaches a regular expression's matching position`, () => { + const expression = /x/g + expression.lastIndex = 2 + const changes = withChangeTracking( + { value: undefined as unknown }, + (draft) => { + draft.value = expression + }, + ) + expect((changes.value as RegExp).lastIndex).toBe(2) + expression.lastIndex = 0 + expect((changes.value as RegExp).lastIndex).toBe(2) + }) + + it.each([`URL`, `class`, `Date`, `RegExp`, `typed-array`] as const)( + `preserves a newly assigned %s in the stored row`, + async (kind) => { + const value = + kind === `URL` + ? new URL(`https://example.com/path`) + : kind === `class` + ? new Label(`saved`) + : kind === `Date` + ? new Date(`2026-01-01T00:00:00Z`) + : kind === `RegExp` + ? /saved/gi + : new Uint8Array([1, 2]) + const collection = storedRow({ id: 1, value: undefined as unknown }) + try { + const tx = collection.update(1, (draft) => { + draft.value = value + }) + const saved = collection.get(1)!.value + expect(Object.getPrototypeOf(saved)).toBe(Object.getPrototypeOf(value)) + if (value instanceof URL) expect((saved as URL).href).toBe(value.href) + else if (value instanceof Label) + expect((saved as Label).read()).toBe(`saved`) + else expect(saved).toEqual(value) + await tx.isPersisted.promise + } finally { + await collection.cleanup() + } + }, + ) + + it.each([`Date`, `typed-array`] as const)( + `detaches a known mutable %s after callback return`, + (kind) => { + const value = kind === `Date` ? new Date(0) : new Uint8Array([1]) + const changes = withChangeTracking( + { value: undefined as unknown }, + (draft) => { + draft.value = value + }, + ) + if (value instanceof Date) { + value.setTime(1000) + expect((changes.value as Date).getTime()).toBe(0) + } else { + value[0] = 2 + expect((changes.value as Uint8Array)[0]).toBe(1) + } + }, + ) + + it(`keeps a stored URL unchanged when the caller later changes its URL`, async () => { + const value = new URL(`https://example.com/before`) + const collection = storedRow({ id: 1, value: undefined as unknown }) + try { + const tx = collection.update(1, (draft) => { + draft.value = value + }) + value.pathname = `/after` + expect((collection.get(1)!.value as URL).pathname).toBe(`/before`) + await tx.isPersisted.promise + } finally { + await collection.cleanup() + } + }) + + it.each([`Set`, `array`] as const)( + `can commit a new %s member holding a draft handle`, + async (kind) => { + type Row = { + id: number + count: number + s: Set<{ back: Row }> + arr: Array<{ owner: Row }> + } + const collection = storedRow({ + id: 1, + count: 0, + s: new Set(), + arr: [], + }) + try { + const tx = collection.update(1, (draft) => { + draft.count = 1 + if (kind === `Set`) draft.s.add({ back: draft }) + else draft.arr.push({ owner: draft }) + }) + const saved = collection.get(1)! + const back = + kind === `Set` + ? saved.s.values().next().value!.back + : saved.arr[0]!.owner + expect(back.count).toBe(1) + // The draft becomes a detached snapshot, not the published row wrapper. + // Its containers must still lead back to that same snapshot. + expect(kind === `Set` ? back.s : back.arr).toBe( + kind === `Set` ? saved.s : saved.arr, + ) + const cycle = + kind === `Set` + ? back.s.values().next().value!.back + : back.arr[0]!.owner + expect(cycle).toBe(back) + await tx.isPersisted.promise + } finally { + await collection.cleanup() + } + }, + ) + + it.each( + ([`scalar`, `object`] as const).flatMap((kind) => + ([`object`, `array`, `Map`, `Set`] as const).map((path) => ({ + kind, + path, + })), + ), + )( + `omits an untouched $path back-reference on a $kind-only edit`, + ({ kind, path }) => { + type Row = { + count: number + value: { x: number } + child?: { + name: string + back: Row | Array | Map | Set + } + } + const row: Row = { count: 0, value: { x: 0 } } + row.child = { + name: `before`, + back: + path === `array` + ? [row] + : path === `Map` + ? new Map([[`row`, row]]) + : path === `Set` + ? new Set([row]) + : row, + } + const changes = withChangeTracking(row, (draft) => { + if (kind === `scalar`) draft.count = 1 + else draft.value = { x: 1 } + }) + expect(Object.keys(changes)).toEqual([ + kind === `scalar` ? `count` : `value`, + ]) + }, + ) + + it(`keeps a real nested edit even when that child also reaches the row`, () => { + type Row = { count: number; child?: { name: string; back: Row } } + const row: Row = { count: 0 } + row.child = { name: `before`, back: row } + const changes = withChangeTracking(row, (draft) => { + draft.count = 1 + draft.child!.name = `after` + }) + expect((changes.child as NonNullable).name).toBe(`after`) + expect(row.child.name).toBe(`before`) + }) + + it(`publishes a changed sibling alias even when it has a nested row back-reference`, () => { + type Child = { name: string; back?: Row } + type Row = { child: Child; alias: Child } + const child: Child = { name: `before` } + const row: Row = { child, alias: child } + child.back = row + const changes = withChangeTracking(row, (draft) => { + draft.alias.name = `after` + }) + const saved = { ...row, ...changes } + expect(saved.child.name).toBe(`after`) + expect(saved.child).toBe(saved.alias) + expect(child.name).toBe(`before`) + }) +}) diff --git a/packages/db/tests/proxy-iteration-contract.test.ts b/packages/db/tests/proxy-iteration-contract.test.ts new file mode 100644 index 0000000000..027ac53104 --- /dev/null +++ b/packages/db/tests/proxy-iteration-contract.test.ts @@ -0,0 +1,570 @@ +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { + createChangeProxy, + withArrayChangeTracking, + withChangeTracking, +} from '../src/proxy.js' + +// Drafts preserve native live membership, even if a snapshot iterator would +// make mutation tracking simpler. Nested field edits have separate laws. +describe.each([`Map`, `Set`] as const)(`%s draft iteration`, (kind) => { + it(`calls a read-only forEach callback once per entry without reporting changes`, () => { + const values = + kind === `Map` ? new Map([[1, { x: 1 }]]) : new Set([{ x: 1 }]) + const context = {} + const callback = + vi.fn<(value: unknown, key: unknown, collection: unknown) => void>() + const { proxy: draft, getChanges } = createChangeProxy({ values }) + const draftValues = draft.values + draftValues.forEach(callback, context) + expect(callback).toHaveBeenCalledTimes(1) + expect(getChanges()).toEqual({}) + expect(callback.mock.contexts).toEqual([context]) + const [value, key, collection] = callback.mock.calls[0]! + expect(collection).toBe(draftValues) + if (kind === `Set`) expect(key).toBe(value) + }) + + it(`rejects an invalid forEach callback even when empty`, () => { + const values = kind === `Map` ? new Map() : new Set() + withChangeTracking({ values }, (draft) => { + expect(() => draft.values.forEach(null!)).toThrow(TypeError) + }) + }) + it(`visits entries added before consuming an existing iterator`, () => { + const values = kind === `Map` ? new Map([[1, 1]]) : new Set([1]) + withChangeTracking({ values }, (draft) => { + const iterator = draft.values.values() + if (draft.values instanceof Map) draft.values.set(2, 2) + else draft.values.add(2) + expect([...iterator]).toEqual([1, 2]) + }) + }) + + it(`skips entries deleted before consuming an existing iterator`, () => { + const values = + kind === `Map` + ? new Map([ + [1, 1], + [2, 2], + ]) + : new Set([1, 2]) + withChangeTracking({ values }, (draft) => { + const iterator = draft.values.values() + draft.values.delete(2) + expect([...iterator]).toEqual([1]) + }) + }) +}) + +type Item = { x: number } + +describe.each([`Map`, `Set`] as const)( + `%s caller-owned insertion values`, + (kind) => { + it.each([`mutate`, `delete-readd`, `second-entry`] as const)( + `matches native raw-object mutation after insertion: %s`, + (operation) => { + const run = (values: Map | Set) => { + const item = { x: 1 } + if (values instanceof Map) values.set(`a`, item) + else values.add(item) + if (operation === `delete-readd`) { + if (values instanceof Map) values.delete(`a`) + else values.delete(item) + } + item.x = 2 + if (operation !== `mutate`) { + if (values instanceof Map) + values.set(operation === `second-entry` ? `b` : `a`, item) + else values.add(item) + } + } + const make = () => + kind === `Map` ? new Map() : new Set() + const expected = make() + run(expected) + const changes = withChangeTracking({ values: make() }, (draft) => + run(draft.values), + ) + expect([ + ...(changes.values as ReturnType).values(), + ]).toEqual([...expected.values()]) + }, + ) + }, +) +const protocols = [`values`, `entries`, `iterator`, `forEach`] as const + +describe(`Whole-draft identity boundary`, () => { + it.each([`object`, `array`, `Map`, `Set`] as const)( + `%s preserves existing data when a callback throws after editing a new object`, + (kind) => { + const item = { x: 1 } + const original = { x: 10 } + const input = { + original, + object: undefined as Item | undefined, + array: [] as Array, + map: new Map(), + set: new Set(), + } + const failure = new Error(`callback failed`) + expect(() => + withChangeTracking(input, (draft) => { + draft.original.x = 20 + let added: Item + if (kind === `object`) { + draft.object = item + added = draft.object + } else if (kind === `array`) { + draft.array.push(item) + added = draft.array[0]! + } else if (kind === `Map`) { + draft.map.set(`item`, item) + added = draft.map.get(`item`)! + } else { + draft.set.add(item) + added = draft.set.values().next().value! + } + added.x = 2 + expect(item.x).toBe(2) + throw failure + }), + ).toThrow(failure) + expect(item.x).toBe(2) + expect(original.x).toBe(10) + expect(input.object).toBeUndefined() + expect(input.array).toEqual([]) + expect(input.map.size).toBe(0) + expect(input.set.size).toBe(0) + }, + ) + + it.each([`Map`, `Set`] as const)( + `shares a new object across two row drafts through %s and detaches both results`, + (kind) => { + const item = { x: 1 } + const rows = [1, 2].map((id) => ({ + id, + values: kind === `Map` ? new Map() : new Set(), + })) + const changes = withArrayChangeTracking(rows, (drafts) => { + for (const draft of drafts) { + if (draft.values instanceof Map) draft.values.set(`item`, item) + else draft.values.add(item) + } + drafts[0]!.values.values().next().value!.x = 2 + expect(drafts[1]!.values.values().next().value!.x).toBe(2) + expect(item.x).toBe(2) + }) + item.x = 3 + for (const change of changes) { + expect([ + ...(change.values as (typeof rows)[number][`values`]).values(), + ]).toEqual([{ x: 2 }]) + } + expect(rows.map((row) => row.values.size)).toEqual([0, 0]) + }, + ) + + it(`reports replacing a self link while omitting an untouched self link`, () => { + type Linked = { name: string; self?: Linked } + const input: Linked = { name: `before` } + input.self = input + const untouched = withChangeTracking(input, (draft) => { + draft.name = `after` + }) + expect(untouched).toEqual({ name: `after` }) + const replaced = withChangeTracking(input, (draft) => { + draft.name = `after` + draft.self = { name: `replacement` } + }) + expect(replaced).toEqual({ name: `after`, self: { name: `replacement` } }) + expect(input.self).toBe(input) + expect(input.name).toBe(`before`) + }) + + it.each( + ([`object`, `array`, `Map`, `Set`] as const).flatMap((kind) => + [false, true].map((throughAlias) => ({ kind, throughAlias })), + ), + )( + `publishes both aliases for $kind, throughAlias=$throughAlias`, + ({ kind, throughAlias }) => { + const item = { x: 1 } + const alias = + kind === `object` + ? { item } + : kind === `array` + ? [item] + : kind === `Map` + ? new Map([[`item`, item]]) + : new Set([item]) + const input = { item, alias } + const readAlias = (container: typeof alias) => + container instanceof Map + ? container.get(`item`)! + : container instanceof Set + ? container.values().next().value! + : Array.isArray(container) + ? container[0]! + : container.item + const changes = withChangeTracking(input, (draft) => { + const value = throughAlias ? readAlias(draft.alias) : draft.item + value.x = 2 + }) + const result = { ...input, ...changes } + expect(result.item.x).toBe(2) + expect(readAlias(result.alias).x).toBe(2) + expect(readAlias(result.alias)).toBe(result.item) + expect(item.x).toBe(1) + }, + ) +}) +type Protocol = (typeof protocols)[number] + +function visit( + values: Map | Set, + protocol: Protocol, + callback: (value: Item) => void, +) { + switch (protocol) { + case `forEach`: + values.forEach(callback) + break + case `entries`: + for (const [, value] of values.entries()) callback(value) + break + case `values`: + for (const value of values.values()) callback(value) + break + case `iterator`: + if (values instanceof Map) for (const [, value] of values) callback(value) + else for (const value of values) callback(value) + } +} + +describe.each([`Map`, `Set`] as const)(`%s nested iteration laws`, (kind) => { + it(`reuses an original member handle without duplicating or splitting its draft identity`, () => { + const item = { x: 1 } + const values = kind === `Map` ? new Map([[`old`, item]]) : new Set([item]) + const changes = withChangeTracking({ values }, (draft) => { + if (draft.values instanceof Map) { + draft.values.set(`new`, item) + draft.values.get(`new`)!.x = 2 + expect(draft.values.get(`old`)!.x).toBe(2) + } else { + draft.values.add(item) + expect(draft.values.size).toBe(1) + draft.values.values().next().value!.x = 2 + } + }) + expect(item.x).toBe(1) + expect( + [...(changes.values as typeof values).values()].every( + (value) => value.x === 2, + ), + ).toBe(true) + }) + + it.each(protocols)( + `%s tracks each nested edit once and leaves the input untouched`, + (protocol) => { + const input = [{ x: 1 }, { x: 2 }] + const values = + kind === `Map` + ? new Map(input.map((value) => [value.x, value])) + : new Set(input) + let visits = 0 + const changes = withChangeTracking({ values }, (draft) => { + visit(draft.values, protocol, (value) => { + if (++visits > 2) throw new Error(`An edit reinserted an entry`) + value.x += 10 + }) + }) + expect(visits).toBe(2) + expect([...(changes.values as typeof values).values()]).toEqual([ + { x: 11 }, + { x: 12 }, + ]) + expect([...values.values()]).toEqual([{ x: 1 }, { x: 2 }]) + }, + ) + + it.each(protocols)( + `%s can write and revert without revisiting entries or reporting changes`, + (protocol) => { + const values = + kind === `Map` ? new Map([[1, { x: 1 }]]) : new Set([{ x: 1 }]) + let visits = 0 + const changes = withChangeTracking({ values }, (draft) => { + visit(draft.values, protocol, (value) => { + if (++visits > 1) throw new Error(`An edit reinserted an entry`) + value.x = 2 + value.x = 1 + }) + }) + expect(visits).toBe(1) + expect(changes).toEqual({}) + }, + ) + + it.each(protocols)( + `%s preserves sibling changes when another entry reverts`, + (protocol) => { + const values = + kind === `Map` + ? new Map([ + [1, { x: 1 }], + [2, { x: 2 }], + ]) + : new Set([{ x: 1 }, { x: 2 }]) + let visits = 0 + const changes = withChangeTracking({ values }, (draft) => { + visit(draft.values, protocol, (value) => { + if (++visits > 2) throw new Error(`An edit reinserted an entry`) + const original = value.x + value.x += 10 + if (original === 2) value.x = original + }) + }) + expect([...(changes.values as typeof values).values()]).toEqual([ + { x: 11 }, + { x: 2 }, + ]) + }, + ) + + it.each(protocols)( + `%s matches native membership changes during iteration`, + (protocol) => { + const run = (values: Map | Set) => { + const seen: Array = [] + visit(values, protocol, (value) => { + if (seen.length > 5) throw new Error(`Iteration did not terminate`) + seen.push(value.x) + value.x += 10 + if (seen.length === 1) { + if (values instanceof Map) { + values.delete(2) + values.set(3, { x: 3 }) + } else { + values.delete([...values][1]!) + values.add({ x: 3 }) + } + } + }) + return { seen, values: [...values.values()] } + } + const make = () => + kind === `Map` + ? new Map([ + [1, { x: 1 }], + [2, { x: 2 }], + ]) + : new Set([{ x: 1 }, { x: 2 }]) + const expected = run(make()) + const changes = withChangeTracking({ values: make() }, (draft) => { + expect(run(draft.values)).toEqual(expected) + }) + expect([ + ...(changes.values as Map | Set).values(), + ]).toEqual(expected.values) + }, + ) + + it(`shares newly added values during chained mutators and detaches the result`, () => { + const item = { x: 1 } + const values = kind === `Map` ? new Map() : new Set() + const changes = withChangeTracking({ values }, (draft) => { + if (draft.values instanceof Map) { + expect(draft.values.set(`a`, item).set(`b`, item)).toBe(draft.values) + draft.values.get(`a`)!.x = 2 + expect(draft.values.get(`b`)!.x).toBe(2) + } else { + expect(draft.values.add(item).add(item)).toBe(draft.values) + expect(draft.values.has(item)).toBe(true) + draft.values.values().next().value!.x = 2 + expect(draft.values.size).toBe(1) + } + }) + expect(item.x).toBe(2) + item.x = 3 + expect( + [...(changes.values as typeof values).values()].every( + (value) => value.x === 2, + ), + ).toBe(true) + }) +}) + +it.each(protocols)( + `Set %s observes clear and re-add after an edit just like a native iterator`, + (protocol) => { + const run = (values: Set) => { + const seen: Array = [] + visit(values, protocol, (value) => { + if (seen.length > 2) throw new Error(`Iteration did not terminate`) + seen.push(value.x) + if (seen.length === 1) { + value.x = 3 + values.clear() + values.add(value) + } + }) + return seen + } + const expected = run(new Set([{ x: 1 }, { x: 2 }])) + const changes = withChangeTracking( + { values: new Set([{ x: 1 }, { x: 2 }]) }, + (draft) => { + expect(run(draft.values)).toEqual(expected) + }, + ) + expect(changes.values).toEqual(new Set([{ x: 3 }])) + }, +) + +it.each(protocols)( + `Set %s keys and values expose the same draft handle`, + (protocol) => { + const changes = withChangeTracking( + { values: new Set([{ x: 1 }]) }, + (draft) => { + const key = draft.values.keys().next().value! + visit(draft.values, protocol, (value) => expect(value).toBe(key)) + key.x = 2 + }, + ) + expect(changes.values).toEqual(new Set([{ x: 2 }])) + }, +) + +it.each([...protocols, `keys`] as const)( + `Set %s handles retain membership after editing`, + (protocol) => { + const changes = withChangeTracking( + { values: new Set([{ x: 1 }]) }, + (draft) => { + let visits = 0 + const edit = (value: Item) => { + if (++visits > 1) throw new Error(`An edit reinserted an entry`) + value.x = 2 + expect(draft.values.has(value)).toBe(true) + expect(draft.values.add(value)).toBe(draft.values) + draft.values.add(value) + expect(draft.values.size).toBe(1) + expect(draft.values.delete(value)).toBe(true) + expect(draft.values.has(value)).toBe(false) + } + if (protocol === `keys`) + for (const value of draft.values.keys()) edit(value) + else visit(draft.values, protocol, edit) + }, + ) + expect(changes.values).toEqual(new Set()) + }, +) + +it(`Map for-of nested writes reach collection.update`, async () => { + const collection = createCollection<{ + id: number + values: Map + }>({ + getKey: (row) => row.id, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { id: 1, values: new Map([[`a`, { x: 1 }]]) }, + }) + commit() + markReady() + }, + }, + onUpdate: async () => {}, + }) + try { + const tx = collection.update(1, (draft) => { + for (const [, value] of draft.values) value.x = 2 + }) + expect(tx.mutations[0]?.changes.values).toEqual(new Map([[`a`, { x: 2 }]])) + expect(collection.get(1)?.values.get(`a`)?.x).toBe(2) + } finally { + await collection.cleanup() + } +}) + +it.each([`Map`, `Set`] as const)( + `%s accepts raw edits inside the callback but detaches committed values afterward`, + async (kind) => { + type Row = { id: number; values: Map | Set } + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { id: 1, values: kind === `Map` ? new Map() : new Set() }, + }) + commit() + markReady() + }, + }, + onUpdate: () => Promise.resolve(), + }) + try { + const item = { x: 1 } + const tx = collection.update(1, (draft) => { + if (draft.values instanceof Map) draft.values.set(`a`, item) + else draft.values.add(item) + item.x = 2 + }) + expect([...collection.get(1)!.values.values()]).toEqual([{ x: 2 }]) + item.x = 3 + expect([...collection.get(1)!.values.values()]).toEqual([{ x: 2 }]) + await tx.isPersisted.promise + } finally { + await collection.cleanup() + } + }, +) + +it.each([`entries`, `values`] as const)( + `taking one Map %s value does not scan every entry`, + (protocol) => { + const { proxy } = createChangeProxy({ + values: new Map(Array.from({ length: 1000 }, (_, i) => [i, i])), + }) + const values = proxy.values + let visits = 0 + const original = Map.prototype.entries + const spy = vi.spyOn(Map.prototype, `entries`).mockImplementation(function ( + this: Map, + ) { + const iterator = original.call(this) + const next = iterator.next.bind(iterator) + iterator.next = () => { + const result = next() + if (!result.done) visits++ + return result + } + return iterator + }) + try { + expect(values[protocol]().next()).toEqual({ + done: false, + value: protocol === `entries` ? [0, 0] : 0, + }) + expect(visits).toBe(1) + } finally { + spy.mockRestore() + } + }, +) diff --git a/packages/db/tests/proxy.test.ts b/packages/db/tests/proxy.test.ts index bbac0151af..d074426b76 100644 --- a/packages/db/tests/proxy.test.ts +++ b/packages/db/tests/proxy.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' import { Temporal } from 'temporal-polyfill' import { createArrayChangeProxy, @@ -8,6 +8,32 @@ import { } from '../src/proxy' describe(`Proxy Library`, () => { + it.each([null, `true`])( + `tracks reads, writes and reverts without consulting DEBUG=%s`, + (debug) => { + const getItem = vi.fn(() => debug) + const log = vi.spyOn(console, `log`).mockImplementation(() => {}) + vi.stubGlobal(`localStorage`, { getItem }) + try { + const original = { value: 1, nested: { value: 2 } } + const { proxy, getChanges } = createChangeProxy(original) + expect(proxy.value).toBe(1) + proxy.value = 3 + proxy.nested.value = 4 + expect(getChanges()).toEqual({ value: 3, nested: { value: 4 } }) + proxy.value = 1 + proxy.nested.value = 2 + expect(getChanges()).toEqual({}) + expect(original).toEqual({ value: 1, nested: { value: 2 } }) + expect(getItem).not.toHaveBeenCalled() + expect(log).not.toHaveBeenCalled() + } finally { + vi.unstubAllGlobals() + log.mockRestore() + } + }, + ) + describe(`createChangeProxy`, () => { it(`should track changes to an object`, () => { const obj = { name: `John`, age: 30 } diff --git a/packages/db/tests/query/bucket-facade-adapter.test.ts b/packages/db/tests/query/bucket-facade-adapter.test.ts new file mode 100644 index 0000000000..d3999946e8 --- /dev/null +++ b/packages/db/tests/query/bucket-facade-adapter.test.ts @@ -0,0 +1,760 @@ +import { D2, MultiSet } from '@tanstack/db-ivm' +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' +import { eq } from '../../src/query/builder/functions.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { BucketFacadeAdapter } from '../../src/query/live/bucket-facade-adapter.js' +import { CollectionConfigBuilder } from '../../src/query/live/collection-config-builder.js' +import { BUCKET_FACADE_REF } from '../../src/query/live/materialized-pipeline.js' +import { mockSyncCollectionOptions, stripVirtualProps } from '../utils.js' +import type { Collection } from '../../src/collection/index.js' +import type { SyncConfig } from '../../src/types.js' +import type { + BucketFacadeRef, + BucketRow, +} from '../../src/query/live/materialized-pipeline.js' +import type { Context } from '../../src/query/builder/types.js' + +type FacadeSync = Parameters>[`sync`]>[0] + +class ThrowingBuildIndex extends BasicIndex { + throwBeforeBuild = false + throwOnBuild = false + + override build(entries: Iterable<[number, unknown]>): void { + if (this.throwBeforeBuild) { + throw new Error(`facade index rebuild failed`) + } + super.build(entries) + if (this.throwOnBuild) { + throw new Error(`facade index rebuild failed`) + } + } +} + +describe(`BucketFacadeAdapter`, () => { + it.each( + [false, true].flatMap((present) => + [`insert`, `replace`, `cancel`].map((change) => ({ present, change })), + ), + )( + `publishes consolidated membership: $present / $change`, + async ({ present, change }) => { + const graph = new D2() + const rows = graph.newInput<[string, BucketRow]>() + const activeBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `public-membership`, + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: false }], + () => {}, + ) + graph.finalize() + const ref: BucketFacadeRef = { + [BUCKET_FACADE_REF]: { edgeId: `children`, bucketKey: `group` }, + } + const oldRow: BucketRow = { + publicKey: 1, + value: { id: 1, name: `old` }, + order: undefined, + } + const newRow: BucketRow = { + publicKey: 1, + value: { id: 1, name: `new` }, + order: undefined, + } + const send = (row: BucketRow, weight: number) => + rows.sendData(new MultiSet([[[`group`, row], weight]])) + try { + activeBuckets.sendData(new MultiSet([[[`group`, true], 1]])) + if (present) send(oldRow, 1) + graph.run() + adapter.flush().publish() + if (change === `cancel`) { + send(present ? oldRow : newRow, 1) + send(present ? oldRow : newRow, -1) + } else { + if (change === `replace`) send(oldRow, -1) + send(newRow, 1) + } + graph.run() + const published = adapter.resolve(ref) as unknown as Collection< + { id: number; name: string }, + number + > + const expected = + change !== `insert` && !present + ? [] + : [{ id: 1, name: change === `cancel` ? `old` : `new` }] + expect(published.toArray.map(stripVirtualProps)).toEqual( + present ? [{ id: 1, name: `old` }] : [], + ) + adapter.flush().publish() + expect(adapter.resolve(ref)).toBe(published) + expect(published.toArray.map(stripVirtualProps)).toEqual(expected) + } finally { + await adapter.cleanup() + } + }, + ) + + it.each( + [10, 100].flatMap((size) => + [false, true].map((ordered) => ({ size, ordered })), + ), + )( + `reads a $size-row facade without repeated scans (ordered=$ordered)`, + ({ size, ordered }) => { + const graph = new D2() + const rows = graph.newInput<[string, BucketRow]>() + const activeBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `facade-read-work`, + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: ordered }], + () => {}, + ) + graph.finalize() + const bucketKey = `group` + const values = Array.from({ length: size }, (_, id) => ({ id })) + activeBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + rows.sendData( + new MultiSet( + values.map((value) => [ + [ + bucketKey, + { + publicKey: value.id, + value, + order: ordered + ? String(size - value.id).padStart(3, `0`) + : undefined, + }, + ], + 1, + ]), + ), + ) + graph.run() + adapter.flush().publish() + const ref: BucketFacadeRef = { + [BUCKET_FACADE_REF]: { edgeId: `children`, bucketKey }, + } + const publicView = adapter.resolve(ref) as unknown as Collection< + { id: number }, + number + > + const entries = publicView.entries.bind(publicView) + let visited = 0 + const scan = vi + .spyOn(publicView, `entries`) + .mockImplementation(function* () { + for (const entry of entries()) { + visited++ + yield entry + } + }) + try { + const published = publicView + for (const { id } of values) { + expect(published.get(id)?.id).toBe(id) + expect(published.has(id)).toBe(true) + expect(published.size).toBe(size) + } + expect([...published.keys()]).toEqual( + ordered + ? values.map(({ id }) => id).reverse() + : values.map(({ id }) => id), + ) + // These counters see the real facade scan, not a modeled operation. + expect + .soft(scan.mock.calls.length, `full bucket scans`) + .toBeLessThanOrEqual(1) + expect.soft(visited, `source rows visited`).toBeLessThanOrEqual(size) + const inserted = { id: size } + rows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { publicKey: inserted.id, value: inserted, order: `999` }, + ], + 1, + ], + ]), + ) + graph.run() + adapter.flush().publish() + expect(published.get(size)?.id).toBe(size) + expect(published.size).toBe(size + 1) + } finally { + scan.mockRestore() + adapter.cleanup() + } + }, + ) + + it(`moves a row when the graph reuses its object for a new order`, async () => { + const graph = new D2() + const rows = graph.newInput<[string, BucketRow]>() + const activeBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `facade-order-parent`, + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: true }], + () => {}, + ) + graph.finalize() + + const bucketKey = `group-1` + const moving = { id: 1, value: `moving` } + const fixed = { id: 2, value: `fixed` } + activeBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + rows.sendData( + new MultiSet([ + [[bucketKey, { publicKey: moving.id, value: moving, order: `0` }], 1], + [[bucketKey, { publicKey: fixed.id, value: fixed, order: `1` }], 1], + ]), + ) + graph.run() + adapter.flush().publish() + + const facadeRef: BucketFacadeRef = { + [BUCKET_FACADE_REF]: { edgeId: `children`, bucketKey }, + } + const facade = adapter.resolve(facadeRef) as unknown as Collection< + typeof moving, + number + > + expect(facade.toArray.map(({ id }) => id)).toEqual([1, 2]) + + rows.sendData( + new MultiSet([ + [[bucketKey, { publicKey: moving.id, value: moving, order: `0` }], -1], + [[bucketKey, { publicKey: moving.id, value: moving, order: `2` }], 1], + ]), + ) + graph.run() + adapter.flush().publish() + + expect(facade.toArray.map(({ id }) => id)).toEqual([2, 1]) + await adapter.cleanup() + }) + + it(`restores facade state without public effects when a flush fails`, async () => { + const graph = new D2() + const rows = graph.newInput<[string, BucketRow]>() + const activeBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `facade-rollback-parent`, + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: true }], + () => {}, + ) + graph.finalize() + + const bucketKey = `group-1` + const original = { id: 1, value: `original` } + const fixed = { id: 3, value: `fixed` } + activeBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + rows.sendData( + new MultiSet([ + [ + [bucketKey, { publicKey: original.id, value: original, order: `0` }], + 1, + ], + [[bucketKey, { publicKey: fixed.id, value: fixed, order: `1` }], 1], + ]), + ) + graph.run() + adapter.flush().publish() + + const facadeRef: BucketFacadeRef = { + [BUCKET_FACADE_REF]: { edgeId: `children`, bucketKey }, + } + const facade = adapter.resolve(facadeRef) as unknown as Collection< + typeof original, + number + > + expect(facade.toArray.map(stripVirtualProps)).toEqual([original, fixed]) + const publications: Array = [] + const subscription = facade.subscribeChanges((changes) => { + publications.push(changes) + }) + let layoutPublications = 0 + const unsubscribeLayout = facade._subscribeLayoutChanges(() => { + layoutPublications++ + }) + let statusChanges = 0 + const unsubscribeStatus = facade.on(`status:change`, () => { + statusChanges++ + }) + let truncates = 0 + const unsubscribeTruncate = facade.on(`truncate`, () => { + truncates++ + }) + const stateRevision = facade._stateRevision + const layoutRevision = facade._layoutRevision + + const entries = ( + adapter as unknown as { + entries: Map> + } + ).entries + const sync = entries.get(`children`)?.get(bucketKey)?.sync + if (!sync) throw new Error(`Missing facade sync`) + const commit = sync.commit + let shouldThrow = true + sync.commit = () => { + const applied = commit() + if (shouldThrow) { + shouldThrow = false + throw new Error(`facade flush failed`) + } + return applied + } + + const replacement = { id: 1, value: `replacement` } + const added = { id: 2, value: `added` } + rows.sendData( + new MultiSet([ + [ + [bucketKey, { publicKey: original.id, value: original, order: `0` }], + -1, + ], + [ + [ + bucketKey, + { + publicKey: replacement.id, + value: replacement, + order: `2`, + }, + ], + 1, + ], + [[bucketKey, { publicKey: added.id, value: added, order: `3` }], 1], + ]), + ) + graph.run() + + expect(() => adapter.flush()).toThrow(`facade flush failed`) + expect(facade.toArray.map(stripVirtualProps)).toEqual([original, fixed]) + expect([ + ...( + entries.get(`children`)?.get(bucketKey) as unknown as { + currentOrder: Map + } + ).currentOrder, + ]).toEqual([ + [original.id, `0`], + [fixed.id, `1`], + ]) + expect(publications).toEqual([]) + expect(layoutPublications).toBe(0) + expect(statusChanges).toBe(0) + expect(truncates).toBe(0) + expect(facade._stateRevision).toBe(stateRevision) + expect(facade._layoutRevision).toBe(layoutRevision) + expect(facade.status).toBe(`ready`) + + const restoredOriginal = facade.get(original.id) + const restoredFixed = facade.get(fixed.id) + if (!restoredOriginal || !restoredFixed) { + throw new Error(`Missing restored facade rows`) + } + expect(facade.getKeyFromItem(restoredOriginal)).toBe(original.id) + expect(facade.getKeyFromItem(restoredFixed)).toBe(fixed.id) + + adapter.flush().publish() + + expect(facade.toArray.map(stripVirtualProps)).toEqual([ + fixed, + replacement, + added, + ]) + expect(layoutPublications).toBe(0) + expect(publications).toHaveLength(1) + expect(publications[0]).toHaveLength(2) + expect(statusChanges).toBe(0) + expect(truncates).toBe(0) + expect(facade._stateRevision).toBe(stateRevision + 1) + expect(facade._layoutRevision).toBe(layoutRevision + 1) + expect(facade.toArray.map((row) => facade.getKeyFromItem(row))).toEqual([ + fixed.id, + replacement.id, + added.id, + ]) + expect([ + ...( + entries.get(`children`)?.get(bucketKey) as unknown as { + currentOrder: Map + } + ).currentOrder, + ]).toEqual([ + [original.id, `2`], + [fixed.id, `1`], + [added.id, `3`], + ]) + + rows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { + publicKey: replacement.id, + value: replacement, + order: `2`, + }, + ], + -1, + ], + [ + [ + bucketKey, + { + publicKey: replacement.id, + value: replacement, + order: `0`, + }, + ], + 1, + ], + ]), + ) + graph.run() + adapter.flush().publish() + + expect(facade.toArray.map(stripVirtualProps)).toEqual([ + replacement, + fixed, + added, + ]) + expect(layoutPublications).toBe(1) + expect(publications).toHaveLength(2) + expect(publications[1]).toEqual([]) + expect(facade._stateRevision).toBe(stateRevision + 1) + expect(facade._layoutRevision).toBe(layoutRevision + 2) + + unsubscribeTruncate() + unsubscribeStatus() + unsubscribeLayout() + subscription.unsubscribe() + await adapter.cleanup() + }) + + it(`publishes fresh facade readiness only after every install succeeds`, async () => { + const graph = new D2() + const firstRows = graph.newInput<[string, BucketRow]>() + const firstActiveBuckets = graph.newInput<[string, true]>() + const secondRows = graph.newInput<[string, BucketRow]>() + const secondActiveBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `facade-ready-parent`, + [ + { + edgeId: `first`, + rows: firstRows, + activeBuckets: firstActiveBuckets, + hasOrderBy: false, + }, + { + edgeId: `second`, + rows: secondRows, + activeBuckets: secondActiveBuckets, + hasOrderBy: false, + }, + ], + () => {}, + ) + graph.finalize() + + const bucketKey = `group-1` + const firstFacade = adapter.resolve({ + [BUCKET_FACADE_REF]: { edgeId: `first`, bucketKey }, + } satisfies BucketFacadeRef) as unknown as Collection< + { id: number; value: string }, + number + > + const secondFacade = adapter.resolve({ + [BUCKET_FACADE_REF]: { edgeId: `second`, bucketKey }, + } satisfies BucketFacadeRef) as unknown as Collection< + { id: number; value: string }, + number + > + const firstStatuses: Array = [] + const secondStatuses: Array = [] + const unsubscribeFirst = firstFacade.on(`status:change`, ({ status }) => { + firstStatuses.push(status) + }) + const unsubscribeSecond = secondFacade.on(`status:change`, ({ status }) => { + secondStatuses.push(status) + }) + + const first = { id: 1, value: `first` } + const second = { id: 2, value: `second` } + firstActiveBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + secondActiveBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + firstRows.sendData( + new MultiSet([ + [ + [bucketKey, { publicKey: first.id, value: first, order: undefined }], + 1, + ], + ]), + ) + secondRows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { publicKey: second.id, value: second, order: undefined }, + ], + 1, + ], + ]), + ) + graph.run() + + const entries = ( + adapter as unknown as { + entries: Map> + } + ).entries + const secondSync = entries.get(`second`)?.get(bucketKey)?.sync + if (!secondSync) throw new Error(`Missing second facade sync`) + const commit = secondSync.commit + let shouldThrow = true + secondSync.commit = () => { + const applied = commit() + if (shouldThrow) { + shouldThrow = false + throw new Error(`second facade failed`) + } + return applied + } + + expect(() => adapter.flush()).toThrow(`second facade failed`) + expect(firstFacade.status).toBe(`loading`) + expect(secondFacade.status).toBe(`loading`) + expect(firstStatuses).toEqual([]) + expect(secondStatuses).toEqual([]) + expect(firstFacade.toArray).toEqual([]) + expect(secondFacade.toArray).toEqual([]) + + const retry = adapter.flush() + expect(firstFacade.status).toBe(`loading`) + expect(secondFacade.status).toBe(`loading`) + retry.prepare() + expect(firstFacade.status).toBe(`ready`) + expect(secondFacade.status).toBe(`ready`) + retry.publish() + expect(firstFacade.toArray.map(stripVirtualProps)).toEqual([first]) + expect(secondFacade.toArray.map(stripVirtualProps)).toEqual([second]) + expect(firstStatuses).toEqual([`ready`]) + expect(secondStatuses).toEqual([`ready`]) + + unsubscribeFirst() + unsubscribeSecond() + await adapter.cleanup() + }) + + it(`restores indexed facade state without rebuilding the index`, async () => { + const graph = new D2() + const rows = graph.newInput<[string, BucketRow]>() + const activeBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `facade-index-rollback-parent`, + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: false }], + () => {}, + ) + graph.finalize() + + const bucketKey = `group-1` + const original = { id: 1, value: `original` } + activeBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + rows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { publicKey: original.id, value: original, order: undefined }, + ], + 1, + ], + ]), + ) + graph.run() + adapter.flush().publish() + + const facade = adapter.resolve({ + [BUCKET_FACADE_REF]: { edgeId: `children`, bucketKey }, + } satisfies BucketFacadeRef) as unknown as Collection< + typeof original, + number + > + const index = facade.createIndex((row) => row.value, { + indexType: ThrowingBuildIndex, + }) as ThrowingBuildIndex + const publications: Array = [] + const subscription = facade.subscribeChanges( + (changes) => { + publications.push(changes) + }, + { includeInitialState: false }, + ) + const revision = facade._stateRevision + + const entry = ( + adapter as unknown as { + entries: Map> + } + ).entries + .get(`children`) + ?.get(bucketKey) + const sync = entry?.sync + if (!sync) throw new Error(`Missing facade sync`) + const commit = sync.commit + let shouldThrow = true + sync.commit = () => { + const applied = commit() + if (shouldThrow) { + shouldThrow = false + throw new Error(`facade flush failed`) + } + return applied + } + + const replacement = { id: 1, value: `replacement` } + index.throwBeforeBuild = true + rows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { publicKey: original.id, value: original, order: undefined }, + ], + -1, + ], + [ + [ + bucketKey, + { + publicKey: replacement.id, + value: replacement, + order: undefined, + }, + ], + 1, + ], + ]), + ) + graph.run() + + expect(() => adapter.flush()).toThrow(`facade flush failed`) + expect(facade.status).toBe(`ready`) + expect(facade._state.syncedData.get(original.id)).toMatchObject(original) + expect(publications).toEqual([]) + expect(facade._stateRevision).toBe(revision) + + const final = { id: 1, value: `final` } + rows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { + publicKey: replacement.id, + value: replacement, + order: undefined, + }, + ], + -1, + ], + [ + [bucketKey, { publicKey: final.id, value: final, order: undefined }], + 1, + ], + ]), + ) + graph.run() + adapter.flush().publish() + expect(facade.status).toBe(`ready`) + expect(facade.toArray.map(stripVirtualProps)).toEqual([final]) + expect(publications).toHaveLength(1) + expect(publications[0]).toHaveLength(1) + expect(facade._stateRevision).toBe(revision + 1) + expect(index.lookup(`eq`, `original`)).toEqual(new Set()) + expect(index.lookup(`eq`, `replacement`)).toEqual(new Set()) + expect(index.lookup(`eq`, `final`)).toEqual(new Set([original.id])) + + subscription.unsubscribe() + await adapter.cleanup() + }) + + it(`retries pending parent changes when facade flushing fails`, async () => { + type Parent = { id: number; groupId: number } + type Child = { id: number; groupId: number } + const parents = createCollection( + mockSyncCollectionOptions({ + id: `facade-failure-parents`, + getKey: (row) => row.id, + initialData: [{ id: 1, groupId: 1 }], + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `facade-failure-children`, + getKey: (row) => row.id, + initialData: [{ id: 10, groupId: 1 }], + autoIndex: `eager`, + }), + ) + const originalGetConfig = CollectionConfigBuilder.prototype.getConfig + let builder: + | CollectionConfigBuilder> + | undefined + CollectionConfigBuilder.prototype.getConfig = function () { + builder = this as unknown as CollectionConfigBuilder< + Context, + Record + > + return originalGetConfig.call(this) + } + const live = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)), + })), + ) + + try { + await live.preload() + const flush = vi + .spyOn(BucketFacadeAdapter.prototype, `flush`) + .mockImplementationOnce(() => { + throw new Error(`facade flush failed`) + }) + + parents.utils.begin() + parents.utils.write({ + type: `insert`, + value: { id: 2, groupId: 2 }, + }) + expect(() => parents.utils.commit()).toThrow(`facade flush failed`) + flush.mockRestore() + + const syncState = builder?.currentSyncState + if (!syncState?.flushPendingChanges) { + throw new Error(`Missing live query sync state`) + } + syncState.flushPendingChanges() + expect(live.has(2)).toBe(true) + } finally { + CollectionConfigBuilder.prototype.getConfig = originalGetConfig + vi.restoreAllMocks() + await live.cleanup() + await Promise.all([parents.cleanup(), children.cleanup()]) + } + }) +}) diff --git a/packages/db/tests/query/builder/functions.test.ts b/packages/db/tests/query/builder/functions.test.ts index cd40ce2e4c..e4d284960b 100644 --- a/packages/db/tests/query/builder/functions.test.ts +++ b/packages/db/tests/query/builder/functions.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, expectTypeOf, it } from 'vitest' import { CollectionImpl } from '../../../src/collection/index.js' import { Query, getQueryIR } from '../../../src/query/builder/index.js' import { @@ -12,6 +12,7 @@ import { coalesce, concat, count, + divide, eq, gt, gte, @@ -23,12 +24,16 @@ import { lte, max, min, + multiply, not, or, + subtract, sum, toArray, upper, } from '../../../src/query/builder/functions.js' +import { compileSingleRowExpression } from '../../../src/query/compiler/evaluators.js' +import type { BasicExpression } from '../../../src/query/ir.js' // Test schema interface Employee { @@ -324,5 +329,92 @@ describe(`QueryBuilder Functions`, () => { const select = builtQuery.select! expect((select.salary_plus_bonus as any).name).toBe(`add`) }) + + it(`subtract function works`, () => { + const query = new Query() + .from({ employees: employeesCollection }) + .select(({ employees }) => ({ + id: employees.id, + salary_minus_tax: subtract(employees.salary, 5000), + })) + + const builtQuery = getQueryIR(query) + const select = builtQuery.select! + expect((select.salary_minus_tax as any).name).toBe(`subtract`) + }) + + it(`multiply function works`, () => { + const query = new Query() + .from({ employees: employeesCollection }) + .select(({ employees }) => ({ + id: employees.id, + double_salary: multiply(employees.salary, 2), + })) + + const builtQuery = getQueryIR(query) + const select = builtQuery.select! + expect((select.double_salary as any).name).toBe(`multiply`) + }) + + it(`divide function works`, () => { + const query = new Query() + .from({ employees: employeesCollection }) + .select(({ employees }) => ({ + id: employees.id, + monthly_salary: divide(employees.salary, 12), + })) + + const builtQuery = getQueryIR(query) + const select = builtQuery.select! + expect((select.monthly_salary as any).name).toBe(`divide`) + }) + + it(`math functions can be combined for complex calculations`, () => { + const query = new Query() + .from({ employees: employeesCollection }) + .select(({ employees }) => ({ + id: employees.id, + // (salary * 1.1) - 500 = 10% raise minus deductions + adjusted_salary: subtract(multiply(employees.salary, 1.1), 500), + })) + + const builtQuery = getQueryIR(query) + const select = builtQuery.select! + expect((select.adjusted_salary as any).name).toBe(`subtract`) + }) + + it(`RED review: nullish operands are coalesced to 0 at runtime but widen types`, () => { + const subtractExpression = subtract(10, null) + expectTypeOf(subtractExpression).toEqualTypeOf>() + + const subtractResult = compileSingleRowExpression(subtractExpression)({}) + expect(subtractResult).toBe(10) + }) + + it(`RED review: divide can return null for non-null operand types`, () => { + const divideExpression = divide(10, 0) + expectTypeOf(divideExpression).toEqualTypeOf< + BasicExpression + >() + + const divideResult = compileSingleRowExpression(divideExpression)({}) + expect(divideResult).toBeNull() + }) + + it(`math functions can be used in orderBy`, () => { + const query = new Query() + .from({ employees: employeesCollection }) + .orderBy(({ employees }) => multiply(employees.salary, 2), `desc`) + .select(({ employees }) => ({ + id: employees.id, + salary: employees.salary, + })) + + const builtQuery = getQueryIR(query) + expect(builtQuery.orderBy).toBeDefined() + expect(builtQuery.orderBy).toHaveLength(1) + expect((builtQuery.orderBy![0]!.expression as any).name).toBe(`multiply`) + expect(builtQuery.orderBy![0]!.compareOptions.direction).toBe(`desc`) + }) }) }) diff --git a/packages/db/tests/query/builder/union-all.test.ts b/packages/db/tests/query/builder/union-all.test.ts index 7a2fe7b7d9..ea3d243513 100644 --- a/packages/db/tests/query/builder/union-all.test.ts +++ b/packages/db/tests/query/builder/union-all.test.ts @@ -1,11 +1,17 @@ import { describe, expect, it } from 'vitest' import { CollectionImpl } from '../../../src/collection/index.js' +import { DbClient, collectionOptions } from '../../../src/client.js' import { InvalidSourceError, InvalidSourceTypeError, QueryMustHaveFromClauseError, } from '../../../src/errors.js' -import { Query, getQueryIR } from '../../../src/query/builder/index.js' +import { + BaseQueryBuilder, + Query, + getQueryIR, +} from '../../../src/query/builder/index.js' +import { eq } from '../../../src/query/builder/functions.js' interface Employee { id: number @@ -93,6 +99,70 @@ describe(`QueryBuilder.unionAll`, () => { expect(builtQuery.from.queries).toHaveLength(2) }) + it(`preserves descriptor resolution after unioning sources`, () => { + const employeeDescriptor = collectionOptions(`union-employees`, () => ({ + id: `union-employees`, + getKey: (item: Employee) => item.id, + sync: { sync: () => {} }, + })) + const departmentDescriptor = collectionOptions(`union-departments`, () => ({ + id: `union-departments`, + getKey: (item: Department) => item.id, + sync: { sync: () => {} }, + })) + const client = new DbClient() + const builder = new BaseQueryBuilder({}, (options) => + client.collection(options), + ) + + const query = builder + .unionAll({ employees: employeeDescriptor }) + .join( + { departments: departmentDescriptor }, + ({ employees, departments }) => + eq(employees.department_id, departments.id), + `inner`, + ) + + expect(getQueryIR(query).join).toHaveLength(1) + }) + + it(`preserves descriptor resolution after unioning query branches`, () => { + const employeeDescriptor = collectionOptions(`branch-employees`, () => ({ + id: `branch-employees`, + getKey: (item: Employee) => item.id, + sync: { sync: () => {} }, + })) + const departmentDescriptor = collectionOptions( + `branch-departments`, + () => ({ + id: `branch-departments`, + getKey: (item: Department) => item.id, + sync: { sync: () => {} }, + }), + ) + const client = new DbClient() + const builder = new BaseQueryBuilder({}, (options) => + client.collection(options), + ) + const employeeRows = builder + .from({ employees: employeeDescriptor }) + .select(({ employees: employee }) => ({ id: employee.id })) + const departmentRows = builder + .from({ departments: departmentDescriptor }) + .select(({ departments: department }) => ({ id: department.id })) + + const query = builder + .unionAll(employeeRows, departmentRows) + .join( + { departments: departmentDescriptor }, + ({ id, departments }) => eq(id, departments.id), + `inner`, + ) + + expect(getQueryIR(query).join).toHaveLength(1) + }) + it(`throws helpful errors for invalid source inputs`, () => { const builder = new Query() diff --git a/packages/db/tests/query/compiler/basic.test.ts b/packages/db/tests/query/compiler/basic.test.ts index 66b7a15149..52f84b4df9 100644 --- a/packages/db/tests/query/compiler/basic.test.ts +++ b/packages/db/tests/query/compiler/basic.test.ts @@ -1,6 +1,7 @@ import { describe, expect, test } from 'vitest' import { D2, MultiSet, output } from '@tanstack/db-ivm' import { compileQuery } from '../../../src/query/compiler/index.js' +import { materializeCompilation } from '../../../src/query/live/materialized-pipeline.js' import { CollectionRef, Func, PropRef, Value } from '../../../src/query/ir.js' import type { QueryIR } from '../../../src/query/ir.js' import type { CollectionImpl } from '../../../src/collection/index.js' @@ -30,6 +31,33 @@ const sampleUsers: Array = [ describe(`Query2 Compiler`, () => { describe(`Basic Compilation`, () => { + test(`queries without includes keep their compiled pipeline`, () => { + const usersCollection = { id: `users` } as CollectionImpl + const query: QueryIR = { + from: new CollectionRef(usersCollection, `users`), + } + const graph = new D2() + const input = graph.newInput<[number, User]>() + const compilation = compileQuery( + query, + { users: input }, + { users: usersCollection }, + {}, + {}, + new Set(), + {}, + () => {}, + ) + + const materialized = materializeCompilation( + compilation, + (user: User) => user.id, + ) + + expect(materialized.pipeline).toBe(compilation.pipeline) + expect(materialized.facades).toEqual([]) + }) + test(`compiles a simple FROM query`, () => { // Create a mock collection const usersCollection = { @@ -160,6 +188,63 @@ describe(`Query2 Compiler`, () => { }) }) + test(`implicit joined results expose their lexical aliases`, () => { + type Post = { id: number; userId: number; title: string } + const usersCollection = { + id: `users`, + config: { autoIndex: `off` }, + } as CollectionImpl + const postsCollection = { + id: `posts`, + config: { autoIndex: `off` }, + } as CollectionImpl + + const resultKeys = (userAlias: string, postAlias: string) => { + const graph = new D2() + const usersInput = graph.newInput<[number, User]>() + const postsInput = graph.newInput<[number, Post]>() + const query: QueryIR = { + from: new CollectionRef(usersCollection, userAlias), + join: [ + { + type: `inner`, + from: new CollectionRef(postsCollection, postAlias), + left: new PropRef([userAlias, `id`]), + right: new PropRef([postAlias, `userId`]), + }, + ], + } + const { pipeline } = compileQuery( + query, + { [userAlias]: usersInput, [postAlias]: postsInput }, + { users: usersCollection, posts: postsCollection }, + {}, + {}, + new Set(), + {}, + () => {}, + ) + const messages: Array> = [] + pipeline.pipe(output((message) => messages.push(message))) + graph.finalize() + + usersInput.sendData(new MultiSet([[[1, sampleUsers[0]!], 1]])) + postsInput.sendData( + new MultiSet([[[10, { id: 10, userId: 1, title: `Hello` }], 1]]), + ) + graph.run() + + const result = messages + .flatMap((message) => message.getInner()) + .map(([data]) => data[1][0]) + .find((row) => row !== undefined) + return Object.keys(result).sort() + } + + expect(resultKeys(`user`, `post`)).toEqual([`post`, `user`]) + expect(resultKeys(`account`, `article`)).toEqual([`account`, `article`]) + }) + test(`compiles a query with WHERE clause`, () => { const usersCollection = { id: `users`, diff --git a/packages/db/tests/query/compiler/binary-equality-work.test.ts b/packages/db/tests/query/compiler/binary-equality-work.test.ts new file mode 100644 index 0000000000..ddbbeed670 --- /dev/null +++ b/packages/db/tests/query/compiler/binary-equality-work.test.ts @@ -0,0 +1,106 @@ +import { describe, expect, it } from 'vitest' +import { compileSingleRowExpression } from '../../../src/query/compiler/evaluators.js' +import { Func, PropRef, Value } from '../../../src/query/ir.js' + +const cases = [0, 129, 65536].flatMap((size) => + [`eq`, `in`].flatMap((operator) => + [`array`, `buffer`, `mixed`].flatMap((form) => + [`equal`, `offset`, `different`, `length`].map((shape) => ({ + size, + operator, + form, + shape, + })), + ), + ), +) + +// Count bytes encoded, without retaining one mock-call record per byte. +function measureEncoding(run: () => unknown) { + const original = String.fromCharCode + let bytes = 0 + String.fromCharCode = (...codes) => { + bytes += codes.length + return original(...codes) + } + try { + return { result: run(), bytes } + } finally { + String.fromCharCode = original + } +} + +describe(`binary equality work`, () => { + it.each(cases)( + `compares $size bytes with $operator/$form/$shape without encoding strings`, + ({ size, operator, form, shape }) => { + const left = + form === `buffer` + ? Buffer.alloc(size, 65) + : new Uint8Array(size).fill(65) + const backing = new Uint8Array(size + 2).fill(65) + backing[0] = 99 + backing[backing.length - 1] = 99 + let right: Uint8Array = + shape === `offset` + ? backing.subarray(1, size + 1) + : Uint8Array.from(left) + if (shape === `different`) { + if (size === 0) right = new Uint8Array([66]) + else right[size - 1] = 66 + } + if (shape === `length`) right = new Uint8Array(size + 1).fill(65) + if (form !== `array`) + right = Buffer.from(right.buffer, right.byteOffset, right.byteLength) + const expected = shape === `equal` || shape === `offset` + const evaluate = compileSingleRowExpression( + new Func(operator, [ + new PropRef([`blob`]), + new Value(operator === `in` ? [null, right] : right), + ]), + ) + const observed = measureEncoding(() => evaluate({ blob: left })) + expect(observed.result).toBe(expected) + expect(observed.bytes).toBe(0) + }, + ) + + it.each([`eq`, `in`])( + `compares a MiB using %s without caching mutable bytes`, + (operator) => { + const left = new Uint8Array(1024 * 1024).fill(65) + const right = left.slice() + const evaluate = compileSingleRowExpression( + new Func(operator, [ + new PropRef([`blob`]), + new Value(operator === `in` ? [right] : right), + ]), + ) + const equal = measureEncoding(() => evaluate({ blob: left })) + expect(equal).toEqual({ result: true, bytes: 0 }) + right[right.length - 1] = 66 + const different = measureEncoding(() => evaluate({ blob: left })) + expect(different).toEqual({ result: false, bytes: 0 }) + }, + ) + + it.each([`eq`, `in`])( + `keeps %s binary values separate from normalization-like strings`, + (operator) => { + const bytes = new Uint8Array([65]) + const text = '\u0000tanstack-db:binary:A' + for (const [left, right] of [ + [bytes, text], + [text, bytes], + ]) { + const evaluate = compileSingleRowExpression( + new Func(operator, [ + new PropRef([`value`]), + new Value(operator === `in` ? [right] : right), + ]), + ) + expect(evaluate({ value: left })).toBe(false) + } + }, + ) +}) diff --git a/packages/db/tests/query/compiler/evaluators.test.ts b/packages/db/tests/query/compiler/evaluators.test.ts index 69969de18a..dac457867a 100644 --- a/packages/db/tests/query/compiler/evaluators.test.ts +++ b/packages/db/tests/query/compiler/evaluators.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from 'vitest' +import { Temporal } from 'temporal-polyfill' import { compileExpression } from '../../../src/query/compiler/evaluators.js' import { Func, PropRef, Value } from '../../../src/query/ir.js' import type { NamespacedRow } from '../../../src/types.js' @@ -730,6 +731,339 @@ describe(`evaluators`, () => { expect(compiled({})).toBe(null) }) }) + + describe(`NaN (PostgreSQL float semantics)`, () => { + // Following PostgreSQL, NaN is equal to itself and greater than every + // other (non-null) value, so it has a well-defined order. + it(`treats NaN as equal to NaN`, () => { + const func = new Func(`eq`, [new Value(NaN), new Value(NaN)]) + expect(compileExpression(func)({})).toBe(true) + }) + + it(`treats NaN as not equal to a number`, () => { + const func = new Func(`eq`, [new Value(NaN), new Value(5)]) + expect(compileExpression(func)({})).toBe(false) + }) + + it(`still returns UNKNOWN when comparing NaN with null`, () => { + const func = new Func(`eq`, [new Value(NaN), new Value(null)]) + expect(compileExpression(func)({})).toBe(null) + }) + + it(`treats NaN as greater than every number`, () => { + expect( + compileExpression(new Func(`gt`, [new Value(NaN), new Value(5)]))( + {}, + ), + ).toBe(true) + expect( + compileExpression(new Func(`gt`, [new Value(5), new Value(NaN)]))( + {}, + ), + ).toBe(false) + expect( + compileExpression( + new Func(`gt`, [new Value(NaN), new Value(NaN)]), + )({}), + ).toBe(false) + }) + + it(`orders NaN with gte/lt/lte consistently`, () => { + // NaN >= anything (including NaN); nothing finite >= NaN + expect( + compileExpression( + new Func(`gte`, [new Value(NaN), new Value(5)]), + )({}), + ).toBe(true) + expect( + compileExpression( + new Func(`gte`, [new Value(NaN), new Value(NaN)]), + )({}), + ).toBe(true) + // NaN < nothing; a finite value < NaN + expect( + compileExpression(new Func(`lt`, [new Value(NaN), new Value(5)]))( + {}, + ), + ).toBe(false) + expect( + compileExpression(new Func(`lt`, [new Value(5), new Value(NaN)]))( + {}, + ), + ).toBe(true) + // NaN <= NaN; a finite value <= NaN + expect( + compileExpression( + new Func(`lte`, [new Value(NaN), new Value(NaN)]), + )({}), + ).toBe(true) + expect( + compileExpression( + new Func(`lte`, [new Value(5), new Value(NaN)]), + )({}), + ).toBe(true) + }) + + it(`matches NaN inside an IN list`, () => { + const func = new Func(`in`, [ + new Value(NaN), + new Value([NaN, 1, 2]), + ]) + expect(compileExpression(func)({})).toBe(true) + }) + }) + + describe(`Temporal objects`, () => { + const evalOp = (op: string, left: any, right: any) => + compileExpression( + new Func(op, [new Value(left), new Value(right)]), + )({}) + + describe(`Instant`, () => { + const earlier = Temporal.Instant.from(`2024-01-15T10:00:00Z`) + const later = Temporal.Instant.from(`2024-01-15T11:00:00Z`) + + it(`orders chronologically`, () => { + expect(evalOp(`gt`, later, earlier)).toBe(true) + expect(evalOp(`gt`, earlier, later)).toBe(false) + expect(evalOp(`gt`, earlier, earlier)).toBe(false) + expect(evalOp(`gte`, earlier, earlier)).toBe(true) + expect(evalOp(`lt`, earlier, later)).toBe(true) + expect(evalOp(`lte`, earlier, earlier)).toBe(true) + }) + + it(`eq returns true for distinct instances with the same instant`, () => { + const earlierCopy = Temporal.Instant.from(earlier) + expect(earlier).not.toBe(earlierCopy) + expect(evalOp(`eq`, earlier, earlierCopy)).toBe(true) + }) + }) + + describe(`PlainDate`, () => { + const jan15 = Temporal.PlainDate.from(`2024-01-15`) + const jan16 = Temporal.PlainDate.from(`2024-01-16`) + + it(`orders chronologically`, () => { + expect(evalOp(`gt`, jan16, jan15)).toBe(true) + expect(evalOp(`gt`, jan15, jan16)).toBe(false) + expect(evalOp(`gt`, jan15, jan15)).toBe(false) + expect(evalOp(`gte`, jan15, jan15)).toBe(true) + expect(evalOp(`lt`, jan15, jan16)).toBe(true) + expect(evalOp(`lte`, jan15, jan15)).toBe(true) + }) + + it(`eq returns true for distinct instances with the same date`, () => { + const jan15Copy = Temporal.PlainDate.from(jan15) + expect(jan15).not.toBe(jan15Copy) + expect(evalOp(`eq`, jan15, jan15Copy)).toBe(true) + }) + + it(`eq returns false for different dates`, () => { + expect(evalOp(`eq`, jan15, jan16)).toBe(false) + }) + }) + + describe(`PlainDateTime`, () => { + const a = Temporal.PlainDateTime.from(`2024-01-15T10:30:00`) + const b = Temporal.PlainDateTime.from(`2024-01-15T10:30:01`) + + it(`orders chronologically`, () => { + expect(evalOp(`gt`, b, a)).toBe(true) + expect(evalOp(`gt`, a, b)).toBe(false) + expect(evalOp(`gt`, a, a)).toBe(false) + expect(evalOp(`gte`, a, a)).toBe(true) + expect(evalOp(`lt`, a, b)).toBe(true) + expect(evalOp(`lte`, a, a)).toBe(true) + }) + + it(`eq returns true for distinct instances with the same value`, () => { + const aCopy = Temporal.PlainDateTime.from(a) + expect(a).not.toBe(aCopy) + expect(evalOp(`eq`, a, aCopy)).toBe(true) + }) + }) + + describe(`PlainTime`, () => { + const morning = Temporal.PlainTime.from(`08:00:00`) + const evening = Temporal.PlainTime.from(`20:00:00`) + + it(`orders by time of day`, () => { + expect(evalOp(`gt`, evening, morning)).toBe(true) + expect(evalOp(`gt`, morning, evening)).toBe(false) + expect(evalOp(`gt`, morning, morning)).toBe(false) + expect(evalOp(`gte`, morning, morning)).toBe(true) + expect(evalOp(`lt`, morning, evening)).toBe(true) + expect(evalOp(`lte`, morning, morning)).toBe(true) + }) + + it(`eq returns true for distinct instances with the same time`, () => { + const morningCopy = Temporal.PlainTime.from(morning) + expect(morning).not.toBe(morningCopy) + expect(evalOp(`eq`, morning, morningCopy)).toBe(true) + }) + + it(`eq returns false for different times`, () => { + expect(evalOp(`eq`, morning, evening)).toBe(false) + }) + }) + + describe(`PlainYearMonth`, () => { + const feb = Temporal.PlainYearMonth.from(`2024-02`) + const mar = Temporal.PlainYearMonth.from(`2024-03`) + + it(`orders chronologically`, () => { + expect(evalOp(`gt`, mar, feb)).toBe(true) + expect(evalOp(`gt`, feb, mar)).toBe(false) + expect(evalOp(`gt`, feb, feb)).toBe(false) + expect(evalOp(`gte`, feb, feb)).toBe(true) + expect(evalOp(`lt`, feb, mar)).toBe(true) + expect(evalOp(`lte`, feb, feb)).toBe(true) + }) + + it(`eq returns true for distinct instances with the same year/month`, () => { + const febCopy = Temporal.PlainYearMonth.from(feb) + expect(feb).not.toBe(febCopy) + expect(evalOp(`eq`, feb, febCopy)).toBe(true) + }) + + it(`eq returns false for different year/month pairs`, () => { + expect(evalOp(`eq`, feb, mar)).toBe(false) + }) + }) + + describe(`PlainMonthDay`, () => { + // PlainMonthDay has no static .compare() — there is no canonical ordering + // without a year (e.g. Feb 29 only sometimes follows Feb 28). Equality is + // still well-defined. + const md1 = Temporal.PlainMonthDay.from(`--03-15`) + const md2 = Temporal.PlainMonthDay.from(`--04-15`) + + it(`eq returns true for distinct instances with the same month/day`, () => { + const md1Copy = Temporal.PlainMonthDay.from(md1) + expect(md1).not.toBe(md1Copy) + expect(evalOp(`eq`, md1, md1Copy)).toBe(true) + }) + + it(`eq returns false for different month/day pairs`, () => { + expect(evalOp(`eq`, md1, md2)).toBe(false) + }) + + it(`gt throws since PlainMonthDay has no defined ordering`, () => { + expect(() => evalOp(`gt`, md1, md2)).toThrow( + /no defined ordering/, + ) + }) + }) + + describe(`ZonedDateTime`, () => { + // Same wall-clock noon in two zones is a different instant. + // Tokyo is ahead of New York, so noon Tokyo precedes noon NY by ~14h. + const tokyoNoon = Temporal.ZonedDateTime.from( + `2024-01-15T12:00:00[Asia/Tokyo]`, + ) + const nyNoon = Temporal.ZonedDateTime.from( + `2024-01-15T12:00:00[America/New_York]`, + ) + + it(`orders by underlying instant across time zones`, () => { + expect(evalOp(`gt`, nyNoon, tokyoNoon)).toBe(true) + expect(evalOp(`gt`, tokyoNoon, nyNoon)).toBe(false) + expect(evalOp(`gt`, tokyoNoon, tokyoNoon)).toBe(false) + expect(evalOp(`gte`, tokyoNoon, tokyoNoon)).toBe(true) + expect(evalOp(`lt`, tokyoNoon, nyNoon)).toBe(true) + expect(evalOp(`lte`, tokyoNoon, tokyoNoon)).toBe(true) + }) + + it(`eq returns true for distinct instances with same instant AND zone`, () => { + const tokyoNoonCopy = Temporal.ZonedDateTime.from(tokyoNoon) + expect(tokyoNoon).not.toBe(tokyoNoonCopy) + expect(evalOp(`eq`, tokyoNoon, tokyoNoonCopy)).toBe(true) + }) + + it(`eq returns false for same instant in different zones`, () => { + // ZonedDateTime.equals treats the time zone as part of identity, so two + // ZonedDateTimes for the same instant in different zones are not equal — + // even though .compare() returns 0 for them. + const inTokyo = nyNoon.withTimeZone(`Asia/Tokyo`) + expect(evalOp(`eq`, nyNoon, inTokyo)).toBe(false) + }) + + it(`ranks same-instant-different-zone as positionally equal`, () => { + // .compare() ranks by underlying instant, so two ZonedDateTimes for + // the same instant in different zones are positionally equal — even + // though .equals() returns false (see the eq test above). This + // verifies we dispatch to .compare() rather than lex-comparing + // toString output, which would order by zone name. + const inTokyo = nyNoon.withTimeZone(`Asia/Tokyo`) + expect(evalOp(`gt`, nyNoon, inTokyo)).toBe(false) + expect(evalOp(`lt`, nyNoon, inTokyo)).toBe(false) + expect(evalOp(`gte`, nyNoon, inTokyo)).toBe(true) + expect(evalOp(`lte`, nyNoon, inTokyo)).toBe(true) + }) + }) + + describe(`Duration`, () => { + // Without a relativeTo, Duration ordering is only well-defined when both + // values are time/day-only. Calendar-affected fields (years/months/weeks) + // are not exercised here. + const oneHour = Temporal.Duration.from(`PT1H`) + const twoHours = Temporal.Duration.from(`PT2H`) + + it(`orders time-only durations by length`, () => { + expect(evalOp(`gt`, twoHours, oneHour)).toBe(true) + expect(evalOp(`gt`, oneHour, twoHours)).toBe(false) + expect(evalOp(`gt`, oneHour, oneHour)).toBe(false) + expect(evalOp(`gte`, oneHour, oneHour)).toBe(true) + expect(evalOp(`lt`, oneHour, twoHours)).toBe(true) + expect(evalOp(`lte`, oneHour, oneHour)).toBe(true) + }) + + it(`eq returns true for distinct instances with the same value`, () => { + const oneHourCopy = Temporal.Duration.from(oneHour) + expect(oneHour).not.toBe(oneHourCopy) + expect(evalOp(`eq`, oneHour, oneHourCopy)).toBe(true) + }) + + it(`eq returns false for different durations`, () => { + expect(evalOp(`eq`, oneHour, twoHours)).toBe(false) + }) + + it(`equivalent forms compare equal but eq returns false`, () => { + // PT60M and PT1H represent the same duration. .compare() ranks them + // as equal, but .equals() (structural, field-by-field) does not, so + // eq returns false. This pins the asymmetry — and verifies we + // dispatch to .compare() rather than lex-comparing toString output, + // which would say "PT1H" < "PT60M". + const sixtyMinutes = Temporal.Duration.from(`PT60M`) + const oneHourLit = Temporal.Duration.from(`PT1H`) + expect(evalOp(`gt`, sixtyMinutes, oneHourLit)).toBe(false) + expect(evalOp(`lt`, sixtyMinutes, oneHourLit)).toBe(false) + expect(evalOp(`gte`, sixtyMinutes, oneHourLit)).toBe(true) + expect(evalOp(`lte`, sixtyMinutes, oneHourLit)).toBe(true) + expect(evalOp(`eq`, sixtyMinutes, oneHourLit)).toBe(false) + }) + }) + + it(`eq returns false for different Temporal types with overlapping string forms`, () => { + // PlainDate and PlainDateTime can both stringify to "2024-01-15..." but + // should compare unequal because the types differ. + const date = Temporal.PlainDate.from(`2024-01-15`) + const dateTime = Temporal.PlainDateTime.from(`2024-01-15`) + // sanity-check the strings share a prefix + expect(dateTime.toString().startsWith(date.toString())).toBe(true) + expect(evalOp(`eq`, date, dateTime)).toBe(false) + }) + + it(`gt throws when comparing different Temporal types`, () => { + // Mixed-type ordering is undefined; surfacing a TypeError beats the + // string-lex pseudo-comparison Temporal explicitly designs against. + const date = Temporal.PlainDate.from(`2024-01-15`) + const dateTime = Temporal.PlainDateTime.from(`2024-01-15T00:00:00`) + expect(() => evalOp(`gt`, date, dateTime)).toThrow( + /different types/, + ) + }) + }) }) describe(`boolean operators`, () => { diff --git a/packages/db/tests/query/compiler/group-by-pipeline.test.ts b/packages/db/tests/query/compiler/group-by-pipeline.test.ts new file mode 100644 index 0000000000..8d09872c43 --- /dev/null +++ b/packages/db/tests/query/compiler/group-by-pipeline.test.ts @@ -0,0 +1,197 @@ +import { D2, MultiSet, output } from '@tanstack/db-ivm' +import { describe, expect, test } from 'vitest' +import { NonAggregateExpressionNotInGroupByError } from '../../../src/errors.js' +import { coalesce } from '../../../src/query/builder/functions.js' +import { processGroupBy } from '../../../src/query/compiler/group-by.js' +import { createValueIdentity } from '../../../src/query/equality-value-identity.js' +import { Aggregate, Func, PropRef, Value } from '../../../src/query/ir.js' +import type { Select } from '../../../src/query/ir.js' +import type { KeyedNamespacedRow } from '../../../src/types.js' + +type Row = { id: number; group: number; amount: number; local: boolean } + +const initial: Array = [ + { id: 1, group: 1, amount: 2, local: false }, + { id: 2, group: 1, amount: 5, local: true }, + { id: 3, group: 2, amount: 9, local: false }, +] +const snapshots = [ + [], + initial, + [initial[0]!, { ...initial[1]!, group: 2, amount: 3, local: false }], + [], + initial, +] + +const cases = [false, true].flatMap((grouped) => + ([`none`, `plain`, `wrapped`] as const).flatMap((selection) => + ([`none`, `expression`, `function`, `false`, `null`] as const).map( + (having) => ({ + grouped, + selection, + having, + }), + ), + ), +) + +describe(`group-by production pipeline`, () => { + test.each([false, true])( + `validates ungrouped SELECT references only with grouping keys: %s`, + (grouped) => { + const graph = new D2() + const compile = () => + processGroupBy( + graph.newInput(), + grouped ? [new PropRef([`row`, `group`])] : [], + createValueIdentity(), + undefined, + { amount: new PropRef([`row`, `amount`]) }, + ) + if (grouped) { + expect(compile).toThrow(NonAggregateExpressionNotInGroupByError) + } else { + expect(compile).not.toThrow() + } + }, + ) + + test.each(cases)( + `recomputes rows and metadata: grouped=$grouped, select=$selection, having=$having`, + ({ grouped, selection, having }) => { + const graph = new D2() + const input = graph.newInput() + const groupRef = new PropRef([`row`, `group`]) + const total = new Aggregate(`sum`, [new PropRef([`row`, `amount`])]) + // Exercise generated-field collision avoidance as well as wrapped refs. + const totalAlias = `__tanstack_group_synced` + const select: Select | undefined = + selection === `none` + ? undefined + : { + ...(grouped ? { group: groupRef } : {}), + [totalAlias]: + selection === `plain` + ? total + : new Func(`add`, [ + coalesce(total, 0), + grouped ? groupRef : new Value(0), + ]), + } + type Result = { + key: unknown + selected: unknown + synced: unknown + origin: unknown + } + let actual = new MultiSet() + processGroupBy( + input, + grouped ? [groupRef] : [], + createValueIdentity(), + having === `expression` + ? [ + selection === `none` + ? new Value(true) + : new Func(`gt`, [ + new PropRef([`$selected`, totalAlias]), + new Value(5), + ]), + ] + : having === `false` || having === `null` + ? [ + having === `false` + ? new Value(false) + : new Func(`gt`, [new Value(null), new Value(5)]), + ] + : undefined, + select, + having === `function` + ? [ + (row: { $selected: Record }) => + selection === `none` || row.$selected[totalAlias]! > 5, + ] + : undefined, + `aggregate-result`, + ).pipe( + output((delta) => { + // Observe the public projection, not transient reducer bookkeeping. + actual = actual + .concat( + delta.map(([key, row]) => { + expect(row.$key).toBe(key) + expect(row.$collectionId).toBe(`aggregate-result`) + return { + key, + selected: row.$selected, + synced: row.$synced, + origin: row.$origin, + } + }), + ) + .consolidate() + }), + ) + graph.finalize() + + let previous = new MultiSet() + for (const rows of snapshots) { + const next = new MultiSet( + rows.map((row) => [ + [ + String(row.id), + { + row: { + ...row, + $synced: !row.local, + $origin: row.local ? `local` : `remote`, + }, + }, + ], + 1, + ]), + ) + input.sendData(previous.negate().concat(next)) + graph.run() + previous = next + + // Independent batch model: partition source rows, then sum directly. + const groups = new Map>() + for (const row of rows) { + const key = grouped ? row.group : `single_group` + groups.set(key, [...(groups.get(key) ?? []), row]) + } + const expected = [...groups].flatMap(([key, members]) => { + if (having === `false` || having === `null`) return [] + const amount = + members.reduce((sum, row) => sum + row.amount, 0) + + (selection === `wrapped` && grouped ? Number(key) : 0) + if (having !== `none` && selection !== `none` && amount <= 5) + return [] + return [ + { + key, + selected: + selection === `none` + ? grouped + ? { __key_0: key } + : {} + : { + ...(grouped ? { group: key } : {}), + [totalAlias]: amount, + }, + synced: members.every((row) => !row.local), + origin: members.some((row) => row.local) ? `local` : `remote`, + }, + ] + }) + const observed = actual.getInner().map(([row, weight]) => { + expect(weight).toBe(1) + return row + }) + expect(observed).toHaveLength(expected.length) + expect(observed).toEqual(expect.arrayContaining(expected)) + } + }, + ) +}) diff --git a/packages/db/tests/query/compiler/lazy-demand.test.ts b/packages/db/tests/query/compiler/lazy-demand.test.ts new file mode 100644 index 0000000000..b1c745f5a7 --- /dev/null +++ b/packages/db/tests/query/compiler/lazy-demand.test.ts @@ -0,0 +1,174 @@ +import { D2, output } from '@tanstack/db-ivm' +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../../src/collection/index.js' +import { compileQuery } from '../../../src/query/compiler/index.js' +import { CollectionRef, PropRef } from '../../../src/query/ir.js' +import type { LazyCollectionCallbacks } from '../../../src/query/compiler/joins.js' + +type Row = { id: number; key: unknown } +type Change = [[number, Row], number] + +function createDemandHarness(joinType: `left` | `right` | `full` = `left`) { + const source = (id: string) => + createCollection>({ + id, + getKey: ({ id: key }) => Number(key), + sync: { sync: () => {} }, + }) + const left = source(`demand-left`) + const right = source(`demand-right`) + const graph = new D2() + const leftInput = graph.newInput<[number, Row]>() + const rightInput = graph.newInput<[number, Row]>() + const callbacks: Record = {} + const lazySources = new Set() + const { pipeline } = compileQuery( + { + from: new CollectionRef(left, `left`), + join: [ + { + type: joinType, + from: new CollectionRef(right, `right`), + left: new PropRef([`left`, `key`]), + right: new PropRef([`right`, `key`]), + }, + ], + }, + { left: leftInput, right: rightInput }, + { [left.id]: left, [right.id]: right }, + {}, + callbacks, + lazySources, + {}, + () => {}, + ) + const transitions: Array> = [] + for (const state of Object.values(callbacks)) { + let previous: Array = [] + state.setDemand = (_plan, keys) => { + const next = [...keys] + // Ignore redundant notifications, but not an intervening empty demand. + if ( + next.length === previous.length && + next.every((key) => previous.includes(key)) + ) + return + previous = next + transitions.push(next) + } + } + let resultWeight = 0 + pipeline.pipe( + output((data) => { + for (const [, weight] of data.getInner()) resultWeight += weight + }), + ) + graph.finalize() + const input = joinType === `right` ? rightInput : leftInput + return { + graph, + input, + transitions, + lazySources, + resultWeight: () => resultWeight, + cleanup: async () => { + await left.cleanup() + await right.cleanup() + }, + } +} + +describe(`compiled lazy demand presence`, () => { + // Characterize the current message boundary before changing batching policy. + it.each( + ([`left`, `right`] as const).flatMap((joinType) => + ([`one-message`, `queued-messages`, `separate-turns`] as const).map( + (delivery) => ({ joinType, delivery }), + ), + ), + )( + `preserves demand transitions for $joinType with $delivery`, + async ({ joinType, delivery }) => { + const h = createDemandHarness(joinType) + const row = { id: 1, key: `shared` } + const insert: Change = [[row.id, row], 1] + const retract: Change = [[row.id, row], -1] + try { + h.input.sendData([insert]) + h.graph.run() + expect(h.lazySources.size).toBe(1) + expect(h.transitions).toEqual([[`shared`]]) + h.transitions.length = 0 + if (delivery === `one-message`) h.input.sendData([retract, insert]) + else { + h.input.sendData([retract]) + if (delivery === `separate-turns`) h.graph.run() + h.input.sendData([insert]) + } + h.graph.run() + expect(h.transitions).toEqual( + delivery === `one-message` ? [] : [[], [`shared`]], + ) + expect(h.resultWeight()).toBe(1) + } finally { + await h.cleanup() + } + }, + ) + + it.each([ + { name: `numbers`, first: 3, second: 3 }, + { name: `signed zero`, first: -0, second: 0 }, + { name: `Date values`, first: new Date(3), second: new Date(3) }, + { + name: `binary values`, + first: Buffer.from([3]), + second: new Uint8Array([3]), + }, + ])( + `retains one demand until the last $name contributor leaves`, + async ({ first, second }) => { + const h = createDemandHarness() + const a: Row = { id: 1, key: first } + const b: Row = { id: 2, key: second } + try { + h.input.sendData([ + [[a.id, a], 1], + [[b.id, b], 1], + ]) + h.graph.run() + expect(h.transitions).toHaveLength(1) + expect(h.transitions[0]).toHaveLength(1) + expect(h.resultWeight()).toBe(2) + h.input.sendData([[[a.id, a], -1]]) + h.graph.run() + expect(h.transitions).toHaveLength(1) + expect(h.resultWeight()).toBe(1) + h.input.sendData([[[b.id, b], -1]]) + h.graph.run() + expect(h.transitions).toHaveLength(2) + expect(h.transitions[1]).toEqual([]) + expect(h.resultWeight()).toBe(0) + } finally { + await h.cleanup() + } + }, + ) + + it(`does not demand nullish keys or add lazy demand for a full join`, async () => { + for (const joinType of [`left`, `full`] as const) { + const h = createDemandHarness(joinType) + try { + h.input.sendData([ + [[1, { id: 1, key: null }], 1], + [[2, { id: 2, key: undefined }], 1], + ]) + h.graph.run() + expect(h.transitions).toEqual([]) + expect(h.lazySources.size).toBe(joinType === `full` ? 0 : 1) + } finally { + await h.cleanup() + } + } + }) +}) diff --git a/packages/db/tests/query/compiler/lazy-targets.test.ts b/packages/db/tests/query/compiler/lazy-targets.test.ts new file mode 100644 index 0000000000..280b0572e1 --- /dev/null +++ b/packages/db/tests/query/compiler/lazy-targets.test.ts @@ -0,0 +1,48 @@ +import { describe, expect, it } from 'vitest' +import { getLazyLoadTargets } from '../../../src/query/compiler/lazy-targets.js' +import { CollectionRef, PropRef, QueryRef } from '../../../src/query/ir.js' +import type { QueryIR } from '../../../src/query/ir.js' +import type { CollectionImpl } from '../../../src/collection/index.js' + +describe(`lazy load target identity`, () => { + const collection = { id: `items` } as CollectionImpl + + function targetsForAlias(alias: string) { + const inner: QueryIR = { + from: new CollectionRef(collection, `inner`), + } + const query: QueryIR = { + from: new QueryRef(inner, `selected`), + } + const optimizedSource = new CollectionRef(collection, `optimized`) + + return { + optimizedSource, + targets: getLazyLoadTargets( + query, + optimizedSource, + `selected`, + new PropRef([`selected`, `id`]), + collection, + { selected: alias }, + ), + } + } + + it(`uses a fallback source when its lexical alias matches`, () => { + const { optimizedSource, targets } = targetsForAlias(`optimized`) + + expect(targets).toEqual([ + { + sourceId: optimizedSource.sourceId, + alias: `optimized`, + collection, + path: [`id`], + }, + ]) + }) + + it(`does not route demand through a fallback with another alias`, () => { + expect(targetsForAlias(`other`).targets).toEqual([]) + }) +}) diff --git a/packages/db/tests/query/compiler/select.test.ts b/packages/db/tests/query/compiler/select.test.ts index 820209b092..c5459944ae 100644 --- a/packages/db/tests/query/compiler/select.test.ts +++ b/packages/db/tests/query/compiler/select.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest' -import { processArgument } from '../../../src/query/compiler/select.js' +import { compileExpression } from '../../../src/query/compiler/evaluators.js' import { Aggregate, Func, PropRef, Value } from '../../../src/query/ir.js' describe(`select compiler`, () => { @@ -7,12 +7,12 @@ describe(`select compiler`, () => { // tests in basic.test.ts and other compiler tests. Here we focus on the standalone // functions that can be tested in isolation. - describe(`processArgument`, () => { + describe(`compileExpression`, () => { it(`processes non-aggregate expressions correctly`, () => { const arg = new PropRef([`users`, `name`]) const namespacedRow = { users: { name: `John` } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(`John`) }) @@ -20,7 +20,7 @@ describe(`select compiler`, () => { const arg = new Value(42) const namespacedRow = {} - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(42) }) @@ -28,7 +28,7 @@ describe(`select compiler`, () => { const arg = new Func(`upper`, [new Value(`hello`)]) const namespacedRow = {} - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(`HELLO`) }) @@ -37,10 +37,9 @@ describe(`select compiler`, () => { const namespacedRow = { users: { id: 1 } } expect(() => { - processArgument(arg, namespacedRow) - }).toThrow( - `Aggregate expressions are not supported in this context. Use GROUP BY clause for aggregates.`, - ) + // @ts-expect-error Aggregate IR is not a single-row expression. + compileExpression(arg)(namespacedRow) + }).toThrow(`Unknown expression type: agg`) }) it(`processes reference expressions from different tables`, () => { @@ -50,7 +49,7 @@ describe(`select compiler`, () => { orders: { amount: 100.5 }, } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(100.5) }) @@ -64,7 +63,7 @@ describe(`select compiler`, () => { }, } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(`New York`) }) @@ -72,7 +71,7 @@ describe(`select compiler`, () => { const arg = new Func(`length`, [new PropRef([`users`, `name`])]) const namespacedRow = { users: { name: `Alice` } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(5) }) @@ -89,7 +88,7 @@ describe(`select compiler`, () => { }, } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(`John Doe`) }) @@ -97,7 +96,7 @@ describe(`select compiler`, () => { const arg = new PropRef([`users`, `middleName`]) const namespacedRow = { users: { name: `John`, middleName: null } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(null) }) @@ -105,7 +104,7 @@ describe(`select compiler`, () => { const arg = new PropRef([`nonexistent`, `field`]) const namespacedRow = { users: { name: `John` } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(undefined) }) @@ -113,7 +112,7 @@ describe(`select compiler`, () => { const arg = new PropRef([`users`, `nonexistent`]) const namespacedRow = { users: { name: `John` } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(undefined) }) @@ -121,7 +120,7 @@ describe(`select compiler`, () => { const arg = new Value({ nested: { value: 42 } }) const namespacedRow = {} - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toEqual({ nested: { value: 42 } }) }) @@ -129,7 +128,7 @@ describe(`select compiler`, () => { const arg = new Func(`and`, [new Value(true), new Value(false)]) const namespacedRow = {} - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(false) }) @@ -137,7 +136,7 @@ describe(`select compiler`, () => { const arg = new Func(`gt`, [new PropRef([`users`, `age`]), new Value(18)]) const namespacedRow = { users: { age: 25 } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(true) }) @@ -153,18 +152,14 @@ describe(`select compiler`, () => { }, } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(108.5) }) }) describe(`helper functions`, () => { // Test the helper function that can be imported and tested directly - it(`correctly identifies aggregate expressions`, () => { - // This test would require accessing the isAggregateExpression function - // which is private. Since we can't test it directly, we test it indirectly - // through the processArgument function's error handling. - + it(`rejects aggregate IR at the single-row compiler boundary`, () => { const aggregateExpressions = [ new Aggregate(`count`, [new PropRef([`users`, `id`])]), new Aggregate(`sum`, [new PropRef([`orders`, `amount`])]), @@ -183,12 +178,13 @@ describe(`select compiler`, () => { // All of these should throw errors since they're aggregates aggregateExpressions.forEach((expr) => { expect(() => { - processArgument(expr, namespacedRow) - }).toThrow(`Aggregate expressions are not supported in this context`) + // @ts-expect-error Aggregate IR is not a single-row expression. + compileExpression(expr)(namespacedRow) + }).toThrow(`Unknown expression type: agg`) }) }) - it(`correctly identifies non-aggregate expressions`, () => { + it(`accepts supported single-row expression forms`, () => { const nonAggregateExpressions = [ new PropRef([`users`, `name`]), new Value(42), @@ -201,7 +197,7 @@ describe(`select compiler`, () => { // None of these should throw errors since they're not aggregates nonAggregateExpressions.forEach((expr) => { expect(() => { - processArgument(expr, namespacedRow) + compileExpression(expr)(namespacedRow) }).not.toThrow() }) }) diff --git a/packages/db/tests/query/compiler/subqueries.test.ts b/packages/db/tests/query/compiler/subqueries.test.ts index 07ba9921a2..3bf3a6f99f 100644 --- a/packages/db/tests/query/compiler/subqueries.test.ts +++ b/packages/db/tests/query/compiler/subqueries.test.ts @@ -212,10 +212,7 @@ describe(`Query2 Subqueries`, () => { }) describe(`Subqueries in JOIN clause`, () => { - const dummyCallbacks = { - loadKeys: (_: any) => {}, - loadInitialState: () => {}, - } + const dummyCallbacks = {} it(`supports subquery in join clause`, () => { // Create a subquery for active users @@ -301,11 +298,18 @@ describe(`Query2 Subqueries`, () => { ) const { pipeline } = compilation - // Since we're doing a left join, the right-side source should be handled lazily. - // For subquery-backed joins, lazy loading is marked on the concrete source - // alias that has a subscription (`user`), not the outer QueryRef alias - // (`activeUser`). - expect(lazySources).contains(`user`) + // Since we're doing a left join, the concrete lexical source inside the + // right-side subquery should be handled lazily. Aliases are query-language + // names; the compiler tracks runtime demand by the source's opaque ID. + const activeUserJoin = builtQuery.join![0]!.from + expect(activeUserJoin.type).toBe(`queryRef`) + if (activeUserJoin.type === `queryRef`) { + const activeUserSource = activeUserJoin.query.from + expect(activeUserSource.type).toBe(`collectionRef`) + if (activeUserSource.type === `collectionRef`) { + expect(lazySources).contains(activeUserSource.sourceId) + } + } const messages: Array> = [] pipeline.pipe( @@ -378,10 +382,7 @@ describe(`Query2 Subqueries`, () => { user: usersSubscription, } - const dummyCallbacks = { - loadKeys: (_: any) => {}, - loadInitialState: () => {}, - } + const dummyCallbacks = {} // Compile the query const graph = new D2() diff --git a/packages/db/tests/query/compiler/subquery-caching.test.ts b/packages/db/tests/query/compiler/subquery-caching.test.ts index 769cd7e7a0..991e78c0d4 100644 --- a/packages/db/tests/query/compiler/subquery-caching.test.ts +++ b/packages/db/tests/query/compiler/subquery-caching.test.ts @@ -263,6 +263,40 @@ describe(`Subquery Caching`, () => { expect(sharedCache.has(subquery)).toBe(true) }) + it(`does not reuse a correlated query across parent streams`, () => { + const usersCollection = createMockCollection(`users`) + const query: QueryIR = { + from: new CollectionRef(usersCollection, `u`), + select: { id: new PropRef([`u`, `id`]) }, + } + const graph = new D2() + const userInput = graph.newInput<[number, any]>() + const firstParents = graph.newInput<[number, any]>() + const secondParents = graph.newInput<[number, any]>() + const cache = new WeakMap() + const compileWithParents = (parents: typeof firstParents) => + compileQuery( + query, + { u: userInput }, + { users: usersCollection }, + {}, + {}, + new Set(), + {}, + () => {}, + cache, + new WeakMap(), + parents, + new PropRef([`u`, `id`]), + ) + + const first = compileWithParents(firstParents) + const second = compileWithParents(secondParents) + + expect(second).not.toBe(first) + expect(cache.has(query)).toBe(false) + }) + it(`should use cache to avoid recompilation in nested subqueries`, () => { const usersCollection = createMockCollection(`users`) diff --git a/packages/db/tests/query/derived-delete-reconciliation.test.ts b/packages/db/tests/query/derived-delete-reconciliation.test.ts new file mode 100644 index 0000000000..8a52552c30 --- /dev/null +++ b/packages/db/tests/query/derived-delete-reconciliation.test.ts @@ -0,0 +1,332 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { createLiveQueryCollection } from '../../src/query/index.js' +import { createTransaction } from '../../src/transactions.js' +import { stripVirtualProps } from '../utils.js' +import { oraclePropertyOptions, oracleRuns } from '../oracle-config.js' +import type { SyncConfig } from '../../src/types.js' + +type Row = { id: number; value: number } +const cases = ([`pass-through`, `order`, `select`] as const).flatMap((shape) => + [false, true].flatMap((layered) => + ([`acknowledge`, `rollback`] as const).flatMap((outcome) => + [1, 2].map((batches) => ({ shape, layered, outcome, batches })), + ), + ), +) + +// The model owns source rows; production owns graph deltas and queued output. +// Work is checked at each flush, not just for one final batch size. +const flushHistory = fc.array( + fc.record({ + inserts: fc.integer({ min: 1, max: 5 }), + update: fc.boolean(), + }), + { minLength: 1, maxLength: 12 }, +) +async function runFlushHistory( + steps: Array<{ inserts: number; update: boolean }>, +) { + let sync!: Parameters[`sync`]>[0] + const expected = new Map([[1, { id: 1, value: 0 }]]) + const source = createCollection({ + getKey: (row) => row.id, + sync: { + sync: (actions) => { + sync = actions + actions.begin() + actions.write({ type: `insert`, value: expected.get(1)! }) + actions.commit() + actions.markReady() + }, + }, + }) + const derived = createLiveQueryCollection({ + query: (q) => q.from({ row: source }), + }) + const done = createDeferred() + const tx = createTransaction({ mutationFn: () => done.promise }) + const settled = tx.isPersisted.promise.catch((error: unknown) => error) + const lookup = vi.spyOn(derived._state, `createSyncedKeyLookup`) + try { + await derived.preload() + tx.mutate(() => derived.delete(1)) + for (const [index, step] of steps.entries()) { + lookup.mockClear() + sync.begin() + if (step.update) { + const row = { id: 1, value: index + 1 } + expected.set(1, row) + sync.write({ type: `update`, value: row }) + } + for (let offset = 0; offset < step.inserts; offset++) { + const row = { id: expected.size + 1, value: index } + expected.set(row.id, row) + sync.write({ type: `insert`, value: row }) + } + expect(sync.commit()).toBe(true) + expect(lookup, `flush ${index}`).toHaveBeenCalledTimes( + step.update ? 1 : 0, + ) + expect(derived._state.pendingSyncedTransactions).toHaveLength(index + 1) + expect([...derived.values()]).toEqual([]) + } + done.resolve() + await settled + expect([...derived.values()].map((row) => stripVirtualProps(row))).toEqual([ + ...expected.values(), + ]) + } finally { + done.resolve() + await settled + lookup.mockRestore() + await derived.cleanup() + await source.cleanup() + } +} + +fcTest.prop([flushHistory], { numRuns: oracleRuns(40), seed: 41703 })( + `shares membership work only for balanced deltas across fixed queue histories`, + runFlushHistory, +) +fcTest.prop( + [flushHistory], + oraclePropertyOptions(60, `derived-publication.membership-work`), +)( + `shares membership work only for balanced deltas across random queue histories`, + runFlushHistory, +) + +describe(`derived updates beneath optimistic deletes`, () => { + it.each([false, true])( + `uses committed last writes and truncate=%s for membership`, + async (truncate) => { + const collection = createCollection({ + getKey: (row) => row.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, value: 0 } }) + write({ type: `insert`, value: { id: 2, value: 0 } }) + commit() + markReady() + }, + }, + }) + try { + await collection.preload() + const batch = ( + committed: boolean, + ): (typeof collection._state.pendingSyncedTransactions)[number] => ({ + committed, + applicationStarted: false, + layoutChanged: false, + operations: [], + deletedKeys: new Set(), + rowMetadataWrites: new Map(), + collectionMetadataWrites: new Map(), + applied: createDeferred(), + }) + const earlier = batch(true) + earlier.operations.push({ + type: `insert`, + key: 3, + value: { id: 3, value: 0 }, + }) + const later = batch(true) + later.truncate = truncate + later.operations.push( + { type: `delete`, key: 1, value: { id: 1, value: 0 } }, + { type: `insert`, key: 4, value: { id: 4, value: 0 } }, + { type: `delete`, key: 4, value: { id: 4, value: 0 } }, + { type: `insert`, key: 1, value: { id: 1, value: 1 } }, + ) + const uncommitted = batch(false) + uncommitted.truncate = true + uncommitted.operations.push({ + type: `insert`, + key: 5, + value: { id: 5, value: 0 }, + }) + // Directly exercise the membership snapshot; a real truncate normally + // drains immediately and must not be delayed just to construct this case. + collection._state.pendingSyncedTransactions.push( + earlier, + later, + uncommitted, + ) + const hasKey = collection._state.createSyncedKeyLookup() + expect([1, 2, 3, 4, 5].map(hasKey)).toEqual([ + true, + !truncate, + !truncate, + false, + false, + ]) + } finally { + for (const batch of collection._state.pendingSyncedTransactions) + batch.applied.resolve() + collection._state.pendingSyncedTransactions.length = 0 + await collection.cleanup() + } + }, + ) + + it.each([64, 256])( + `classifies %i queued updates with linear membership work`, + async (count) => { + let sync!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: (row) => row.id, + sync: { + sync: (params) => { + sync = params + params.begin() + for (let id = 0; id < count; id++) + params.write({ type: `insert`, value: { id, value: 0 } }) + params.commit() + params.markReady() + }, + }, + }) + const derived = createLiveQueryCollection({ + query: (q) => q.from({ row: source }), + }) + const done = createDeferred() + const tx = createTransaction({ mutationFn: () => done.promise }) + const settled = tx.isPersisted.promise.catch(() => {}) + try { + await derived.preload() + tx.mutate(() => derived.delete(0)) + const publish = (value: number) => { + sync.begin() + for (let id = 0; id < count; id++) + sync.write({ type: `update`, value: { id, value } }) + expect(sync.commit()).toBe(true) + } + publish(1) + let keyReads = 0 + const queued = derived._state.pendingSyncedTransactions.flatMap( + (batch) => batch.operations, + ) + expect(queued).toHaveLength(count) + for (const operation of queued) { + const key = operation.key + Object.defineProperty(operation, `key`, { + configurable: true, + get: () => { + keyReads++ + return key + }, + }) + } + publish(2) + expect(keyReads).toBeLessThanOrEqual(count * 4) + done.resolve() + await settled + expect( + [...derived.values()].map((row) => stripVirtualProps(row)), + ).toEqual(Array.from({ length: count }, (_, id) => ({ id, value: 2 }))) + } finally { + done.resolve() + await settled + await derived.cleanup() + await source.cleanup() + } + }, + ) + + it.each(cases)( + `$shape / layered=$layered / $outcome / $batches batches`, + async ({ shape, layered, outcome, batches }) => { + let sync!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: (row) => row.id, + sync: { + sync: (params) => { + sync = params + params.begin() + params.write({ type: `insert`, value: { id: 1, value: 10 } }) + params.commit() + params.markReady() + }, + }, + }) + const middle = createLiveQueryCollection({ + query: (q) => q.from({ row: source }), + }) + const derived = createLiveQueryCollection({ + query: (q) => { + const query = q.from({ row: layered ? middle : source }) + if (shape === `order`) return query.orderBy(({ row }) => row.value) + if (shape === `select`) + return query.fn.select(({ row }) => ({ ...row })) + return query + }, + }) + const downstream = createLiveQueryCollection({ + query: (q) => q.from({ row: derived }), + }) + const done = createDeferred() + const tx = createTransaction({ mutationFn: () => done.promise }) + const settled = tx.isPersisted.promise.catch((error: unknown) => error) + const read = (collection: { values: () => Iterable }) => + [...collection.values()] + .map((row) => ({ id: row.id, value: row.value })) + .sort((a, b) => a.id - b.id) + try { + await downstream.preload() + tx.mutate(() => derived.delete(1)) + const reconstructed = new Map() + const subscription = derived.subscribeChanges((changes) => { + for (const change of changes) { + if (change.type === `delete`) reconstructed.delete(change.key) + else reconstructed.set(change.key, stripVirtualProps(change.value)) + } + expect( + [...reconstructed.values()].sort((a, b) => a.id - b.id), + ).toEqual(read(derived)) + }) + try { + for (let index = 0; index < batches; index++) { + sync.begin() + sync.write({ type: `update`, value: { id: 1, value: 20 + index } }) + if (index > 0) + sync.write({ type: `update`, value: { id: 2, value: 40 } }) + sync.write({ + type: `insert`, + value: { id: 2 + index, value: 30 + index }, + }) + expect(sync.commit()).toBe(true) + // The whole graph-output batch remains queued, including the insert. + expect(read(derived)).toEqual([]) + expect(read(downstream)).toEqual([]) + } + if (outcome === `rollback`) done.reject(new Error(`Rejected delete`)) + else done.resolve() + await settled + const expected = [ + { id: 1, value: 19 + batches }, + ...Array.from({ length: batches }, (_, index) => ({ + id: 2 + index, + value: batches > 1 && index === 0 ? 40 : 30 + index, + })), + ] + expect(read(derived)).toEqual(expected) + expect(read(downstream)).toEqual(expected) + } finally { + subscription.unsubscribe() + } + } finally { + done.resolve() + await settled + await downstream.cleanup() + await derived.cleanup() + await middle.cleanup() + await source.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/query/functional-variants.test-d.ts b/packages/db/tests/query/functional-variants.test-d.ts index 15130260ee..a388579f48 100644 --- a/packages/db/tests/query/functional-variants.test-d.ts +++ b/packages/db/tests/query/functional-variants.test-d.ts @@ -1,9 +1,13 @@ import { describe, expectTypeOf, test } from 'vitest' import { + Query, + caseWhen, count, createLiveQueryCollection, eq, gt, + materialize, + toArray, } from '../../src/query/index.js' import { createCollection } from '../../src/collection/index.js' import { mockSyncCollectionOptions } from '../utils.js' @@ -144,6 +148,60 @@ describe(`Functional Variants Types`, () => { >() }) + test(`fn.select rejects child queries and materialization helpers`, () => { + // @ts-expect-error query helpers are only supported in select() + createLiveQueryCollection((q) => + q.from({ user: usersCollection }).fn.select((row) => ({ + id: row.user.id, + nested: { + departments: toArray(q.from({ department: departmentsCollection })), + }, + })), + ) + + // @ts-expect-error materialize() is only supported in select() + createLiveQueryCollection((q) => + q.from({ user: usersCollection }).fn.select((row) => ({ + id: row.user.id, + departments: materialize(q.from({ department: departmentsCollection })), + })), + ) + + // @ts-expect-error child query builders are only supported in select() + createLiveQueryCollection((q) => + q.from({ user: usersCollection }).fn.select((row) => ({ + id: row.user.id, + departments: q.from({ department: departmentsCollection }), + })), + ) + + // @ts-expect-error query expressions are only supported in select() + createLiveQueryCollection((q) => + q.from({ user: usersCollection }).fn.select((row) => ({ + id: row.user.id, + active: eq(row.user.active, true), + })), + ) + + // @ts-expect-error caseWhen() is only supported in select() + createLiveQueryCollection((q) => + q.from({ user: usersCollection }).fn.select((row) => ({ + id: row.user.id, + label: caseWhen(eq(row.user.active, true), `active`, `inactive`), + })), + ) + }) + + test(`fn.select accepts unresolved generic result types`, () => { + const query = new Query().from({ user: usersCollection }) + + function selectValue(value: T) { + return query.fn.select(() => value) + } + + selectValue({ label: `active` }) + }) + test(`fn.where with filtered original type`, () => { const liveCollection = createLiveQueryCollection({ query: (q) => diff --git a/packages/db/tests/query/functional-variants.test.ts b/packages/db/tests/query/functional-variants.test.ts index 8456526b91..f803e1f04e 100644 --- a/packages/db/tests/query/functional-variants.test.ts +++ b/packages/db/tests/query/functional-variants.test.ts @@ -1,9 +1,12 @@ import { beforeEach, describe, expect, test } from 'vitest' import { + caseWhen, count, createLiveQueryCollection, eq, gt, + materialize, + toArray, } from '../../src/query/index.js' import { createCollection } from '../../src/collection/index.js' import { mockSyncCollectionOptions, stripVirtualProps } from '../utils.js' @@ -219,6 +222,135 @@ describe(`Functional Variants Query`, () => { yearsToRetirement: 37, }) }) + + test(`rejects query-construction values returned from fn.select`, () => { + const departmentsCollection = createDepartmentsCollection() + + expect(() => + createLiveQueryCollection({ + startSync: true, + query: (q) => { + const users = q.from({ user: usersCollection }) + const departments = q.from({ department: departmentsCollection }) + + return users.fn.select( + (row) => + ({ + id: row.user.id, + nested: { + departments: toArray( + q.from({ + department: departments.fn.where( + ({ department }) => + row.user.department_id === department.id, + ), + }), + ), + }, + }) as any, + ) + }, + }), + ).toThrow( + `fn.select() cannot return toArray(). Child query builders, query expressions, and helpers`, + ) + + expect(() => + createLiveQueryCollection({ + startSync: true, + query: (q) => + q.from({ user: usersCollection }).fn.select( + (row) => + ({ + id: row.user.id, + departments: materialize( + q.from({ department: departmentsCollection }), + ), + }) as any, + ), + }), + ).toThrow(`fn.select() cannot return materialize()`) + + expect(() => + createLiveQueryCollection({ + startSync: true, + query: (q) => + q.from({ user: usersCollection }).fn.select( + (row) => + ({ + id: row.user.id, + departments: q.from({ department: departmentsCollection }), + }) as any, + ), + }), + ).toThrow(`fn.select() cannot return a child query builder`) + + class Wrapper { + constructor(readonly departments: unknown) {} + } + + expect(() => + createLiveQueryCollection({ + startSync: true, + query: (q) => + q + .from({ user: usersCollection }) + .fn.select( + () => + new Wrapper( + q.from({ department: departmentsCollection }), + ) as any, + ), + }), + ).toThrow(`fn.select() cannot return a child query builder`) + + const departments = Symbol(`departments`) + expect(() => + createLiveQueryCollection({ + startSync: true, + query: (q) => + q.from({ user: usersCollection }).fn.select( + (row) => + ({ + id: row.user.id, + [departments]: q.from({ department: departmentsCollection }), + }) as any, + ), + }), + ).toThrow(`fn.select() cannot return a child query builder`) + + expect(() => + createLiveQueryCollection({ + startSync: true, + query: (q) => + q.from({ user: usersCollection }).fn.select( + (row) => + ({ + id: row.user.id, + active: eq(row.user.active, true), + }) as any, + ), + }), + ).toThrow(`fn.select() cannot return eq()`) + + expect(() => + createLiveQueryCollection({ + startSync: true, + query: (q) => + q.from({ user: usersCollection }).fn.select( + (row) => + ({ + id: row.user.id, + label: caseWhen( + eq(row.user.active, true), + `active`, + `inactive`, + ), + }) as any, + ), + }), + ).toThrow(`fn.select() cannot return caseWhen()`) + }) }) describe(`fn.where`, () => { diff --git a/packages/db/tests/query/generic-collection.test-d.ts b/packages/db/tests/query/generic-collection.test-d.ts new file mode 100644 index 0000000000..553b5f975b --- /dev/null +++ b/packages/db/tests/query/generic-collection.test-d.ts @@ -0,0 +1,53 @@ +import { describe, test } from 'vitest' +import { createLiveQueryCollection, eq } from '../../src/query/index.js' +import type { Collection } from '../../src/collection/index.js' + +// Regression tests for https://github.com/TanStack/db/issues/1677 +// Queries over a Collection where T is an unresolved generic type parameter +// must still expose the properties guaranteed by T's constraint inside +// where/join/select callbacks. This worked in 0.6.5 and broke in 0.6.6. + +describe(`queries over generic collection row types`, () => { + test(`where callback can access properties guaranteed by the type constraint`, () => { + function findById( + items: Collection, + id: string, + ) { + return createLiveQueryCollection((q) => + q.from({ items }).where(({ items: itemsRef }) => eq(itemsRef.id, id)), + ) + } + + void findById + }) + + test(`select callback can access properties guaranteed by the type constraint`, () => { + function selectIds(items: Collection) { + return createLiveQueryCollection((q) => + q.from({ items }).select(({ items: itemsRef }) => ({ + id: itemsRef.id, + })), + ) + } + + void selectIds + }) + + test(`subquery over a generic collection can be used as a join source`, () => { + function withDiff( + a: Collection, + b: Collection<{ id: string }, string>, + ) { + return createLiveQueryCollection((q) => { + const sub = q.from({ a }) + return q + .from({ b }) + .leftJoin({ sub }, ({ b: bRef, sub: subRef }) => + eq(bRef.id, subRef.id), + ) + }) + } + + void withDiff + }) +}) diff --git a/packages/db/tests/query/group-by-work.test.ts b/packages/db/tests/query/group-by-work.test.ts new file mode 100644 index 0000000000..00153d9c88 --- /dev/null +++ b/packages/db/tests/query/group-by-work.test.ts @@ -0,0 +1,53 @@ +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createLiveQueryCollection } from '../../src/query/index.js' +import { count } from '../../src/query/builder/functions.js' +import { mockSyncCollectionOptions } from '../utils.js' + +describe(`group representative work`, () => { + it.each([16, 1024, 5000])( + `encodes changed contributions, not all %s retained members`, + async (size) => { + const source = createCollection( + mockSyncCollectionOptions<{ id: number; value: number }>({ + id: `group-work-${size}`, + getKey: (row) => row.id, + initialData: Array.from({ length: size }, (_, id) => ({ + id, + value: 1, + })), + }), + ) + const grouped = createLiveQueryCollection((q) => + q + .from({ row: source }) + .groupBy(({ row }) => row.value) + .select(({ row }) => ({ value: row.value, count: count(row.id) })), + ) + try { + await grouped.preload() + for (const type of [`insert`, `delete`] as const) { + const spy = vi.spyOn(JSON, `stringify`) + let calls: number + try { + source.utils.begin() + source.utils.write({ type, value: { id: size, value: 1 } }) + source.utils.commit() + calls = spy.mock.calls.length + } finally { + spy.mockRestore() + } + // Group reduction still scans members. Encoding its stable input + // keys must scale with the delta, not the retained group size. + expect(calls).toBeLessThanOrEqual(4) + expect(grouped.toArray).toMatchObject([ + { value: 1, count: size + (type === `insert` ? 1 : 0) }, + ]) + } + } finally { + await grouped.cleanup() + await source.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/query/group-by.test.ts b/packages/db/tests/query/group-by.test.ts index 34c225cc57..d736d8244b 100644 --- a/packages/db/tests/query/group-by.test.ts +++ b/packages/db/tests/query/group-by.test.ts @@ -1,4 +1,5 @@ import { beforeEach, describe, expect, test } from 'vitest' +import { Temporal } from 'temporal-polyfill' import { createLiveQueryCollection } from '../../src/query/index.js' import { createCollection } from '../../src/collection/index.js' import { mockSyncCollectionOptions, stripVirtualProps } from '../utils.js' @@ -222,8 +223,221 @@ function createOrdersCollection(autoIndex: `off` | `eager` = `eager`) { ) } +const equalityEquivalentGroupValues: Array< + [string, () => readonly [unknown, unknown]] +> = [ + [`a Date and its timestamp`, () => [new Date(0), 0]], + [`an invalid Date and NaN`, () => [new Date(Number.NaN), Number.NaN]], + [`signed zero`, () => [-0, 0]], + [ + `binary values with the same bytes`, + () => [Buffer.from([1, 2, 3]), new Uint8Array([1, 2, 3])], + ], + [ + `equivalent Temporal values`, + () => [ + Temporal.PlainDate.from(`2024-04-05`), + Temporal.PlainDate.from(`2024-04-05`), + ], + ], + [ + `the same symbol reference`, + () => { + const value = Symbol(`group`) + return [value, value] + }, + ], + [ + `the same cyclic object`, + () => { + const value: { self?: unknown } = {} + value.self = value + return [value, value] + }, + ], +] + +function representativeSignature(value: unknown): string { + if (value instanceof Date) return `date` + if (Buffer.isBuffer(value)) return `buffer` + if (value instanceof Uint8Array) return `uint8array` + if (typeof value === `number` && Object.is(value, -0)) return `negative-zero` + if (typeof value === `number` && Number.isNaN(value)) return `nan` + if (typeof value === `number`) return `number` + if (typeof value === `symbol`) return `symbol` + if ( + typeof value === `object` && + value !== null && + (value as { self?: unknown }).self === value + ) { + return `cyclic-object` + } + return `${typeof value}:${String(value)}` +} + function createGroupByTests(autoIndex: `off` | `eager`): void { describe(`with autoIndex ${autoIndex}`, () => { + test(`keeps opaque public group keys stable across graph scopes`, async () => { + const symbol = Symbol(`group`) + const otherSymbol = Symbol(`group`) + const valuesCollection = createCollection( + mockSyncCollectionOptions<{ id: number; value: symbol }>({ + id: `scoped-group-symbol-${autoIndex}`, + getKey: (row) => row.id, + initialData: [ + { id: 1, value: symbol }, + { id: 2, value: otherSymbol }, + ], + autoIndex, + }), + ) + const createSummary = () => + createLiveQueryCollection({ + startSync: true, + query: (q) => + q + .from({ value: valuesCollection }) + .groupBy(({ value }) => value.value) + .select(({ value }) => ({ + value: value.value, + count: count(value.id), + })), + }) + + const first = createSummary() + const second = createSummary() + + try { + const firstKeys = [...first.keys()] + const secondKeys = [...second.keys()] + expect(firstKeys).toHaveLength(2) + expect(firstKeys.every((key) => typeof key === `string`)).toBe(true) + expect(new Set(firstKeys).size).toBe(2) + expect(secondKeys).toEqual(firstKeys) + + const symbolKey = first.toArray.find( + (row) => row.value === symbol, + )!.$key + valuesCollection.utils.begin() + valuesCollection.utils.write({ + type: `delete`, + value: { id: 1, value: symbol }, + }) + valuesCollection.utils.commit() + expect(first.get(symbolKey)).toBeUndefined() + + valuesCollection.utils.begin() + valuesCollection.utils.write({ + type: `insert`, + value: { id: 1, value: symbol }, + }) + valuesCollection.utils.commit() + expect(first.get(symbolKey)?.value).toBe(symbol) + } finally { + await Promise.all([ + first.cleanup(), + second.cleanup(), + valuesCollection.cleanup(), + ]) + } + }) + + test.each(equalityEquivalentGroupValues)( + `groups %s by query equality`, + (_name, createValues) => { + const [left, right] = createValues() + const valuesCollection = createCollection( + mockSyncCollectionOptions<{ id: number; value: unknown }>({ + id: `equality-group-values-${autoIndex}`, + getKey: (row) => row.id, + initialData: [ + { id: 1, value: left }, + { id: 2, value: right }, + ], + autoIndex, + }), + ) + + const summary = createLiveQueryCollection({ + startSync: true, + query: (q) => + q + .from({ value: valuesCollection }) + .groupBy(({ value }) => value.value) + .select(({ value }) => ({ + value: value.value, + count: count(value.id), + })), + }) + + const expectSingleGroup = ( + expectedCount: number, + representative: unknown, + ) => { + expect(summary.toArray).toHaveLength(1) + expect(summary.toArray[0]?.count).toBe(expectedCount) + expect(representativeSignature(summary.toArray[0]?.value)).toBe( + representativeSignature(representative), + ) + } + + expectSingleGroup(2, left) + + valuesCollection.utils.begin() + valuesCollection.utils.write({ + type: `delete`, + value: { id: 1, value: left }, + }) + valuesCollection.utils.commit() + expectSingleGroup(1, right) + + valuesCollection.utils.begin() + valuesCollection.utils.write({ + type: `insert`, + value: { id: 1, value: left }, + }) + valuesCollection.utils.commit() + expectSingleGroup(2, left) + }, + ) + + test.each([ + `__group_value_0`, + `__key_0`, + `__tanstack_group_value_0`, + `__tanstack_group_key_0`, + ])( + `keeps the grouped value when an aggregate uses internal-looking alias %s`, + (alias) => { + const valuesCollection = createCollection( + mockSyncCollectionOptions<{ id: number; value: string }>({ + id: `group-alias-collision-${autoIndex}-${alias}`, + getKey: (row) => row.id, + initialData: [ + { id: 1, value: `x` }, + { id: 2, value: `x` }, + ], + autoIndex, + }), + ) + const summary = createLiveQueryCollection({ + startSync: true, + query: (q) => + q + .from({ value: valuesCollection }) + .groupBy(({ value }) => value.value) + .select(({ value }) => ({ + value: value.value, + [alias]: count(value.id), + })), + }) + + expect(summary.toArray.map(stripVirtualProps)).toEqual([ + { value: `x`, [alias]: 2 }, + ]) + }, + ) + describe(`Single Column Grouping`, () => { let ordersCollection: ReturnType @@ -2224,11 +2438,14 @@ function createGroupByTests(autoIndex: `off` | `eager`): void { q .from({ orders: ordersCollection }) .groupBy(({ orders }) => orders.customer_id) - .fn.select((row) => ({ - customerId: row.orders.customer_id, - totalAmount: sum(row.orders.amount), - orderCount: count(row.orders.id), - })), + .fn.select( + (row) => + ({ + customerId: row.orders.customer_id, + totalAmount: sum(row.orders.amount), + orderCount: count(row.orders.id), + }) as any, + ), }), ).toThrow(`fn.select() cannot be used with groupBy()`) }) diff --git a/packages/db/tests/query/immutable-demand-boundary.test.ts b/packages/db/tests/query/immutable-demand-boundary.test.ts new file mode 100644 index 0000000000..dccd88f5d2 --- /dev/null +++ b/packages/db/tests/query/immutable-demand-boundary.test.ts @@ -0,0 +1,138 @@ +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection' +import { eq } from '../../src/query/builder/functions' +import { Func, PropRef, Value } from '../../src/query/ir' +import { compileSingleRowExpression } from '../../src/query/compiler/evaluators' +import { DeduplicatedLoadSubset } from '../../src/query/subset-dedupe' +import type { LoadSubsetOptions } from '../../src/types' + +describe.each([`direct`, `deferred`] as const)( + `immutable demand through %s sync startup`, + (start) => { + it.each([`return`, `resolve`, `abort`] as const)( + `preserves request data and live cancellation on %s`, + async (outcome) => { + const date = Object.freeze(new Date(7)) + const candidates = Object.freeze([date]) + const reference = new PropRef([`date`]) + Object.freeze(reference.path) + Object.freeze(reference) + const where = new Func(`in`, [ + reference, + Object.freeze(new Value(candidates)), + ]) + Object.freeze(where.args) + Object.freeze(where) + const owner = new AbortController() + const options: LoadSubsetOptions = Object.freeze({ + where, + limit: 2, + signal: owner.signal, + }) + let finish = () => {} + const loads: Array = [] + const unloadSubset = vi.fn() + const deduplicated = new DeduplicatedLoadSubset({ + loadSubset: (request) => { + loads.push(request) + return outcome === `return` + ? true + : new Promise((resolve) => (finish = resolve)) + }, + }) + const collection = createCollection<{ id: number }>({ + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: start === `direct`, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset: deduplicated.loadSubset, unloadSubset } + }, + }, + }) + try { + if (start === `deferred`) { + expect(collection._deferSyncStart()).toBe(true) + } + const result = collection._sync.loadSubset(options) + if (start === `deferred`) { + expect(loads).toEqual([]) + collection._resumeSyncStart() + } + expect(loads).toHaveLength(1) + expect(loads[0]).toBe(options) + const matches = compileSingleRowExpression(loads[0]!.where!) + expect( + [new Date(7), new Date(8)].map((value) => matches({ date: value })), + ).toEqual([true, false]) + + if (outcome === `abort`) owner.abort() + expect(loads[0]!.signal!.aborted).toBe(outcome === `abort`) + if (outcome !== `return`) finish() + await result + collection._sync.unloadSubset(options) + expect(unloadSubset).toHaveBeenCalledExactlyOnceWith(options) + expect(unloadSubset.mock.calls[0]![0]).toBe(loads[0]) + expect(date.getTime()).toBe(7) + expect(candidates).toEqual([new Date(7)]) + + // New immutable data describes a new request. Completed equal data + // shares; an aborted transport establishes no reusable result. + const repeat = deduplicated.loadSubset({ + where: new Func(`in`, [ + new PropRef([`date`]), + new Value([new Date(7)]), + ]), + limit: 2, + }) + expect(loads).toHaveLength(outcome === `abort` ? 2 : 1) + if (outcome === `abort`) finish() + await repeat + } finally { + finish() + await collection.cleanup() + } + }, + ) + }, +) + +it.each([`release`, `cleanup`] as const)( + `retires a frozen queued request before adapter startup by %s`, + async (action) => { + const loadSubset = vi.fn(() => true as const) + const unloadSubset = vi.fn() + const collection = createCollection<{ id: number }>({ + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset, unloadSubset } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const options = Object.freeze({ + where: eq(new PropRef([`id`]), new Value(1)), + }) + try { + const result = collection._sync.loadSubset(options) + const settled = Promise.allSettled([result]) + if (action === `release`) collection._sync.unloadSubset(options) + else await collection.cleanup() + expect(await settled).toEqual([ + { + status: `rejected`, + reason: expect.objectContaining({ name: `AbortError` }), + }, + ]) + collection._resumeSyncStart() + expect(loadSubset).not.toHaveBeenCalled() + expect(unloadSubset).not.toHaveBeenCalled() + } finally { + await collection.cleanup() + } + }, +) diff --git a/packages/db/tests/query/includes-collection-oracle.property.test.ts b/packages/db/tests/query/includes-collection-oracle.property.test.ts new file mode 100644 index 0000000000..82b99103cf --- /dev/null +++ b/packages/db/tests/query/includes-collection-oracle.property.test.ts @@ -0,0 +1,2017 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect } from 'vitest' +import { + add, + caseWhen, + concat, + count, + createLiveQueryCollection, + eq, + gt, + lt, + materialize, + multiply, + sum, + toArray, +} from '../../src/query/index.js' +import { runTrace } from '../trace-runner.js' +import { oraclePropertyOptions } from '../oracle-config.js' +import { flushPromises, withExpectedRejection } from '../utils.js' +import { createControlledCollection as createOracleControlledCollection } from './includes-oracle-helpers.js' +import type { Collection } from '../../src/collection/index.js' +import type { ChangeMessage } from '../../src/types.js' +import type { TraceDriver, TraceProjection } from '../trace-runner.js' +import type { ControlledCollection } from './includes-oracle-helpers.js' + +type ParentRow = { + id: number + group: number +} + +type ChildRow = { + id: number + parentGroup: number + value: number +} + +type CollectionAction = + | { type: `putParent`; row: ParentRow } + | { type: `deleteParent`; id: number } + | { type: `putChild`; row: ChildRow } + | { type: `deleteChild`; id: number } + +type ProjectedParent = { + id: number + group: number + childrenReady: boolean + children: Array + arrayChildren: Array + materializedChildren: Array +} + +type CollectionObservation = { + rows: Array + publications: Array> +} + +type CollectionContext = { + parents: ControlledCollection + children: ControlledCollection + live: ReturnType + model: { + parents: Map + children: Map + } + publications: Array> + subscription?: { unsubscribe: () => void } +} + +function createControlledCollection( + name: string, + initialData: ReadonlyArray, +): ControlledCollection { + return createOracleControlledCollection(name, initialData, { + autoIndex: `eager`, + rowUpdateMode: `full`, + }) +} + +function expectedMaterializations(rows: ReadonlyArray) { + return { + facade: [...rows], + array: [...rows], + materialized: [...rows], + } +} + +function createCollectionQuery( + parents: Collection, + children: Collection, + reuseChildRelation = true, +) { + return createLiveQueryCollection((q) => + q + .from({ parent: parents }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + const createChildRows = () => + q + .from({ child: children }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + value: child.value, + })) + const childRows = createChildRows() + + return { + id: parent.id, + group: parent.group, + children: childRows, + arrayChildren: toArray( + reuseChildRelation ? childRows : createChildRows(), + ), + materializedChildren: materialize( + reuseChildRelation ? childRows : createChildRows(), + ), + } + }), + ) +} + +type IncludedChildCollection = ReturnType< + typeof createCollectionQuery +>[`toArray`][number][`children`] + +function projectLive( + live: ReturnType, +): Array { + return [...live.values()].map((parent) => { + const projectChildren = (rows: Iterable) => + [...rows].map(({ id, parentGroup, value }) => ({ + id, + parentGroup, + value, + })) + + return { + id: parent.id, + group: parent.group, + childrenReady: parent.children.isReady(), + children: projectChildren(parent.children.values()), + arrayChildren: projectChildren(parent.arrayChildren), + materializedChildren: projectChildren(parent.materializedChildren), + } + }) +} + +function recompute(context: CollectionContext): Array { + return [...context.model.parents.values()] + .sort((left, right) => left.id - right.id) + .map((parent) => { + const children = [...context.model.children.values()] + .filter((child) => child.parentGroup === parent.group) + .sort((left, right) => left.id - right.id) + .map((child) => ({ ...child })) + + return { + ...parent, + childrenReady: true, + children, + arrayChildren: children, + materializedChildren: children, + } + }) +} + +function createCollectionDriver( + initialParents: ReadonlyArray, + initialChildren: ReadonlyArray, + reuseChildRelation = true, +): TraceDriver { + return { + setup() { + const parents = createControlledCollection( + `collection-oracle-parents`, + initialParents, + ) + const children = createControlledCollection( + `collection-oracle-children`, + initialChildren, + ) + return { + parents, + children, + live: createCollectionQuery( + parents.collection, + children.collection, + reuseChildRelation, + ), + model: { + parents: new Map(initialParents.map((row) => [row.id, { ...row }])), + children: new Map(initialChildren.map((row) => [row.id, { ...row }])), + }, + publications: [], + } + }, + async start(context) { + await context.live.preload() + context.subscription = context.live.subscribeChanges( + () => context.publications.push(projectLive(context.live)), + { includeInitialState: false }, + ) + }, + apply(action, context) { + context.publications = [] + switch (action.type) { + case `putParent`: { + const type = context.model.parents.has(action.row.id) + ? `update` + : `insert` + context.model.parents.set(action.row.id, { ...action.row }) + context.parents.write(type, action.row) + return + } + case `deleteParent`: { + const previous = context.model.parents.get(action.id) + if (!previous) return + context.model.parents.delete(action.id) + context.parents.write(`delete`, previous) + return + } + case `putChild`: { + const type = context.model.children.has(action.row.id) + ? `update` + : `insert` + context.model.children.set(action.row.id, { ...action.row }) + context.children.write(type, action.row) + return + } + case `deleteChild`: { + const previous = context.model.children.get(action.id) + if (!previous) return + context.model.children.delete(action.id) + context.children.write(`delete`, previous) + } + } + }, + async cleanup(context) { + context.subscription?.unsubscribe() + await Promise.all([ + context.live.cleanup(), + context.parents.collection.cleanup(), + context.children.collection.cleanup(), + ]) + }, + } +} + +const collectionProjection: TraceProjection< + CollectionContext, + CollectionObservation +> = { + observe: (context) => ({ + rows: projectLive(context.live), + publications: context.publications, + }), + recompute: (context) => { + const rows = recompute(context) + return { + rows, + publications: context.publications.map(() => rows), + } + }, + assertEqual(observed, expected) { + expect(observed).toEqual(expected) + return undefined + }, +} + +const actionArbitrary: fc.Arbitrary = fc.oneof( + fc.record({ + type: fc.constant(`putParent` as const), + row: fc.record({ + id: fc.integer({ min: 0, max: 3 }), + group: fc.integer({ min: -3, max: 3 }), + }), + }), + fc.record({ + type: fc.constant(`deleteParent` as const), + id: fc.integer({ min: 0, max: 3 }), + }), + fc.record({ + type: fc.constant(`putChild` as const), + row: fc.record({ + id: fc.integer({ min: 10, max: 14 }), + parentGroup: fc.integer({ min: -3, max: 3 }), + value: fc.integer({ min: -5, max: 5 }), + }), + }), + fc.record({ + type: fc.constant(`deleteChild` as const), + id: fc.integer({ min: 10, max: 14 }), + }), +) + +const collectionScenarioArbitrary = fc.record({ + parentGroup: fc.integer({ min: -3, max: 3 }), + childValue: fc.integer({ min: -5, max: 5 }), + actions: fc.array(actionArbitrary, { minLength: 1, maxLength: 16 }), +}) + +const orderSwapArbitrary = fc.integer({ min: 2, max: 8 }).chain((length) => + fc.integer({ min: 0, max: length - 2 }).map((swapIndex) => ({ + length, + swapIndex, + })), +) + +function enumerateActionSequences( + actions: ReadonlyArray, + maxLength: number, +): Array> { + const sequences: Array> = [[]] + let frontier: Array> = [[]] + for (let length = 1; length <= maxLength; length++) { + frontier = frontier.flatMap((prefix) => + actions.map((action) => [...prefix, action]), + ) + sequences.push(...frontier) + } + return sequences +} + +const exhaustiveActions: ReadonlyArray = [ + { type: `putParent`, row: { id: 0, group: 0 } }, + { type: `putParent`, row: { id: 0, group: 1 } }, + { type: `deleteParent`, id: 0 }, + { type: `putChild`, row: { id: 10, parentGroup: 0, value: 0 } }, + { type: `putChild`, row: { id: 10, parentGroup: 1, value: 1 } }, + { type: `deleteChild`, id: 10 }, +] + +describe(`Collection-valued includes oracle`, () => { + fcTest.prop( + [collectionScenarioArbitrary], + oraclePropertyOptions(30, `includes-collection.relationship-history`), + )( + `keeps Collection, toArray, and materialize equivalent across generated relationship histories`, + ({ parentGroup, childValue, actions }) => + runTrace({ + steps: actions, + driver: createCollectionDriver( + [{ id: 0, group: parentGroup }], + [{ id: 10, parentGroup, value: childValue }], + ), + projection: collectionProjection, + }), + ) + + fcTest( + `exhaustively matches every two-step history in the smallest relationship domain`, + async () => { + const initialStates = [ + { parents: [] as Array, children: [] as Array }, + { parents: [{ id: 0, group: 0 }], children: [] as Array }, + { + parents: [] as Array, + children: [{ id: 10, parentGroup: 0, value: 0 }], + }, + { + parents: [{ id: 0, group: 0 }], + children: [{ id: 10, parentGroup: 0, value: 0 }], + }, + ] + const histories = enumerateActionSequences(exhaustiveActions, 2) + + for (const initial of initialStates) { + for (const steps of histories) { + try { + await runTrace({ + steps, + driver: createCollectionDriver(initial.parents, initial.children), + projection: collectionProjection, + }) + } catch (cause) { + throw new Error( + `Exhaustive Collection include history failed: ${JSON.stringify({ initial, steps })}`, + { cause }, + ) + } + } + } + }, + ) + + fcTest( + `publishes independently compiled equivalent child relations coherently`, + () => + runTrace({ + steps: [{ type: `deleteChild`, id: 10 }], + driver: createCollectionDriver( + [{ id: 0, group: 0 }], + [{ id: 10, parentGroup: 0, value: 0 }], + false, + ), + projection: collectionProjection, + }), + ) + + fcTest(`replays a dormant bucket when its first parent route activates`, () => + runTrace({ + steps: [{ type: `putParent`, row: { id: 1, group: 7 } }], + driver: createCollectionDriver( + [], + [{ id: 10, parentGroup: 7, value: 1 }], + ), + projection: collectionProjection, + }), + ) + + fcTest(`retiring a route leaves a held facade empty and ready`, async () => { + let retiredFacade: IncludedChildCollection | undefined + const driver = createCollectionDriver( + [{ id: 1, group: 1 }], + [{ id: 10, parentGroup: 1, value: 1 }], + ) + const lifecycleDriver: TraceDriver = { + ...driver, + apply(action, context, checkpoint) { + retiredFacade ??= context.live.get(1)?.children + return driver.apply(action, context, checkpoint) + }, + } + const projection: TraceProjection< + CollectionContext, + { rows: Array; retiredStatus: string | undefined } + > = { + observe: (context) => ({ + rows: projectLive(context.live), + retiredStatus: retiredFacade?.status, + }), + recompute: (context) => ({ + rows: recompute(context), + retiredStatus: + context.model.parents.size === 0 ? `ready` : retiredFacade?.status, + }), + assertEqual(observed, expected) { + expect(observed).toEqual(expected) + return undefined + }, + } + + await runTrace({ + steps: [{ type: `deleteParent`, id: 1 }], + driver: lifecycleDriver, + projection, + }) + + expect(retiredFacade?.toArray).toEqual([]) + await expect(retiredFacade?.preload()).resolves.toBeUndefined() + }) + + fcTest(`a delete event preserves the published facade identity`, async () => { + let publishedFacade: IncludedChildCollection | undefined + let previousFacadeMatched = true + const driver = createCollectionDriver( + [{ id: 1, group: 1 }], + [{ id: 10, parentGroup: 1, value: 1 }], + ) + const eventDriver: TraceDriver = { + ...driver, + async start(context) { + await driver.start?.(context) + publishedFacade = context.live.get(1)?.children + context.subscription = context.live.subscribeChanges( + (changes: Array>) => { + for (const change of changes) { + if (change.type === `delete`) { + previousFacadeMatched = + change.value.children === publishedFacade + } + } + }, + { includeInitialState: false }, + ) + }, + } + const projection: TraceProjection< + CollectionContext, + { previousFacadeMatched: boolean } + > = { + observe: () => ({ previousFacadeMatched }), + recompute: () => ({ previousFacadeMatched: true }), + assertEqual(observed, expected) { + expect(observed).toEqual(expected) + return undefined + }, + } + + await runTrace({ + steps: [{ type: `deleteParent`, id: 1 }], + driver: eventDriver, + projection, + }) + }) + + fcTest(`facade public keys survive row cloning`, async () => { + const driver = createCollectionDriver( + [{ id: 1, group: 1 }], + [{ id: 10, parentGroup: 1, value: 1 }], + ) + const projection: TraceProjection< + CollectionContext, + { clonedKey: unknown }, + { clonedKey: number } + > = { + observe(context) { + const facade = context.live.get(1)!.children + const row = facade.get(10)! + return { clonedKey: facade.getKeyFromItem({ ...row }) } + }, + recompute: () => ({ clonedKey: 10 }), + assertEqual(observed, expected) { + expect(observed).toEqual(expected) + return undefined + }, + } + + await runTrace({ steps: [], driver, projection }) + }) + + fcTest(`reactivating a retired route restores its current snapshot`, () => + runTrace({ + steps: [ + { type: `putParent`, row: { id: 1, group: 2 } }, + { type: `putParent`, row: { id: 1, group: 1 } }, + ], + driver: createCollectionDriver( + [{ id: 1, group: 1 }], + [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 2, value: 2 }, + ], + ), + projection: collectionProjection, + }), + ) + + fcTest( + `facade application failure leaves no partial state or publication`, + async () => { + const driver = createCollectionDriver( + [{ id: 1, group: 1 }], + [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 1, value: 2 }, + ], + ) + const context = await driver.setup() + await driver.start?.(context) + const facade = context.live.get(1)!.children + const changes: Array = [] + const subscription = facade.subscribeChanges( + (batch) => changes.push(...batch), + { includeInitialState: false }, + ) + const originalGetKey = facade.config.getKey + facade.config.getKey = (row) => { + if (row.id === 20) throw new Error(`facade key failed`) + return originalGetKey(row) + } + + try { + expect(() => + context.children.writeBatch([ + { + type: `update`, + value: { id: 10, parentGroup: 1, value: 10 }, + }, + { + type: `update`, + value: { id: 20, parentGroup: 1, value: 20 }, + }, + ]), + ).toThrow(`facade key failed`) + expect(projectLive(context.live)).toEqual([ + { + id: 1, + group: 1, + childrenReady: true, + children: [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 1, value: 2 }, + ], + arrayChildren: [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 1, value: 2 }, + ], + materializedChildren: [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 1, value: 2 }, + ], + }, + ]) + expect(changes).toEqual([]) + + facade.config.getKey = originalGetKey + context.parents.write(`insert`, { id: 2, group: 2 }) + await flushPromises() + expect(context.live.get(1)!.children).toBe(facade) + expect(projectLive(context.live)[0]!.children).toEqual([ + { id: 10, parentGroup: 1, value: 10 }, + { id: 20, parentGroup: 1, value: 20 }, + ]) + expect(changes).toHaveLength(2) + } finally { + facade.config.getKey = originalGetKey + subscription.unsubscribe() + await driver.cleanup(context) + } + }, + ) + + fcTest( + `root application failure rolls back a prepared facade publication`, + async () => { + type NodeRow = { + id: number + kind: `parent` | `child` + group: number + value: number + } + const initialParent: NodeRow = { + id: 1, + kind: `parent`, + group: 1, + value: 1, + } + const initialChild: NodeRow = { + id: 10, + kind: `child`, + group: 1, + value: 1, + } + const nodes = createControlledCollection(`rollback-nodes`, [ + initialParent, + initialChild, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: nodes.collection }) + .where(({ parent }) => eq(parent.kind, `parent`)) + .select(({ parent }) => ({ + id: parent.id, + value: parent.value, + children: q + .from({ child: nodes.collection }) + .where(({ child }) => eq(child.kind, `child`)) + .where(({ child }) => eq(child.group, parent.group)), + })), + ) + + await live.preload() + const facade = live.get(1)!.children + const rootPublications: Array = [] + const childPublications: Array = [] + const rootSubscription = live.subscribeChanges( + (batch) => rootPublications.push(...batch), + { includeInitialState: false }, + ) + const childSubscription = facade.subscribeChanges( + (batch) => childPublications.push(...batch), + { includeInitialState: false }, + ) + const originalGetKey = live.config.getKey + live.config.getKey = (row) => { + if (row.value === 2) throw new Error(`root key failed`) + return originalGetKey(row) + } + + try { + expect(() => + nodes.writeBatch([ + { + type: `update`, + value: { ...initialParent, value: 2 }, + }, + { + type: `update`, + value: { ...initialChild, value: 2 }, + }, + ]), + ).toThrow(`root key failed`) + expect(live.get(1)!.value).toBe(1) + expect(facade.get(10)!.value).toBe(1) + expect(rootPublications).toEqual([]) + expect(childPublications).toEqual([]) + + live.config.getKey = originalGetKey + nodes.writeBatch([ + { + type: `update`, + value: { ...initialParent, value: 3 }, + }, + { + type: `update`, + value: { ...initialChild, value: 3 }, + }, + ]) + expect(live.get(1)!.value).toBe(3) + expect(facade.get(10)!.value).toBe(3) + expect(rootPublications).toHaveLength(1) + expect(childPublications).toHaveLength(1) + } finally { + live.config.getKey = originalGetKey + rootSubscription.unsubscribe() + childSubscription.unsubscribe() + await Promise.all([live.cleanup(), nodes.collection.cleanup()]) + } + }, + ) + + fcTest( + `child-only changes flush the facade without republishing the parent`, + async () => { + const parents = createControlledCollection(`facade-only-parents`, [ + { id: 1, group: 1 }, + ]) + const children = createControlledCollection(`facade-only-children`, [ + { id: 10, parentGroup: 1, value: 1 }, + ]) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)), + })), + ) + + await live.preload() + const facade = live.get(1)!.children + const rootPublications: Array = [] + const childPublications: Array = [] + const rootSubscription = live.subscribeChanges( + (batch) => rootPublications.push(...batch), + { includeInitialState: false }, + ) + const childSubscription = facade.subscribeChanges( + (batch) => childPublications.push(...batch), + { includeInitialState: false }, + ) + + try { + children.write(`update`, { + id: 10, + parentGroup: 1, + value: 2, + }) + + expect(rootPublications).toEqual([]) + expect(childPublications).toHaveLength(1) + expect(live.get(1)!.children).toBe(facade) + expect( + [...facade.values()].map(({ id, parentGroup, value }) => ({ + id, + parentGroup, + value, + })), + ).toEqual([{ id: 10, parentGroup: 1, value: 2 }]) + } finally { + rootSubscription.unsubscribe() + childSubscription.unsubscribe() + await Promise.all([ + live.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest( + `outer fn.select recomputes nested values after a union branch include changes`, + async () => { + const callbackRows: Array> = [] + const messages = createControlledCollection(`fn-select-messages`, [ + { id: 1, group: 1 }, + ]) + const tools = createControlledCollection(`fn-select-tools`, [ + { id: 2, group: 2 }, + ]) + const children = createControlledCollection(`fn-select-children`, [ + { id: 10, parentGroup: 1, value: 1 }, + ]) + const live = createLiveQueryCollection((q) => { + const messageRows = q + .from({ message: messages.collection }) + .select(({ message }) => ({ + kind: `message` as const, + id: message.id, + children: toArray( + q + .from({ messageChild: children.collection }) + .where(({ messageChild }) => + eq(messageChild.parentGroup, message.group), + ) + .select(({ messageChild }) => ({ + id: messageChild.id, + value: messageChild.value, + })), + ), + })) + const toolRows = q + .from({ tool: tools.collection }) + .select(({ tool }) => ({ + kind: `tool` as const, + id: tool.id, + })) + + return q.unionAll(messageRows, toolRows).fn.select((row) => { + callbackRows.push(row) + return { + kind: row.kind, + id: row.id, + payload: { children: row.children }, + } + }) + }) + + try { + await live.preload() + const message = live.toArray.find((row) => row.kind === `message`)! + expect(message.payload.children).toEqual([{ id: 10, value: 1 }]) + + children.write(`update`, { + id: 10, + parentGroup: 1, + value: 2, + }) + expect( + live.toArray.find((row) => row.kind === `message`)!.payload.children, + ).toEqual([{ id: 10, value: 2 }]) + expect( + callbackRows.flatMap((row) => Object.getOwnPropertySymbols(row)), + ).toEqual([]) + } finally { + await Promise.all([ + live.cleanup(), + messages.collection.cleanup(), + tools.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest( + `outer fn.select rejects a bare union include before invoking the callback`, + async () => { + class Box { + constructor(readonly child: unknown) {} + } + + const callbackChildren: Array = [] + const messages = createControlledCollection(`fn-select-bare-messages`, [ + { id: 1, group: 1 }, + ]) + const tools = createControlledCollection(`fn-select-bare-tools`, [ + { id: 2, group: 2 }, + ]) + const children = createControlledCollection(`fn-select-bare-children`, [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 2, value: 2 }, + ]) + const buildQuery = () => + createLiveQueryCollection((q) => { + const messageRows = q + .from({ message: messages.collection }) + .select(({ message }) => ({ + kind: `message` as const, + id: message.id, + children: q + .from({ messageChild: children.collection }) + .where(({ messageChild }) => + eq(messageChild.parentGroup, message.group), + ), + })) + const toolRows = q + .from({ tool: tools.collection }) + .select(({ tool }) => ({ + kind: `tool` as const, + id: tool.id, + })) + + return q.unionAll(messageRows, toolRows).fn.select((row) => { + const child = `children` in row ? row.children : undefined + callbackChildren.push(child) + return { kind: row.kind, id: row.id, box: new Box(child) } + }) + }) + + try { + expect(buildQuery).toThrow( + `fn.select() cannot consume Collection-valued includes`, + ) + expect(callbackChildren).toEqual([]) + } finally { + await Promise.all([ + messages.collection.cleanup(), + tools.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest( + `fn.select rejects query values returned during include rematerialization`, + async () => { + const messages = createControlledCollection(`fn-select-reject-messages`, [ + { id: 1, group: 1 }, + ]) + const tools = createControlledCollection(`fn-select-reject-tools`, [ + { id: 2, group: 2 }, + ]) + const children = createControlledCollection(`fn-select-reject-children`, [ + { id: 10, parentGroup: 1, value: 1 }, + ]) + const live = createLiveQueryCollection((q) => { + const messageRows = q + .from({ message: messages.collection }) + .select(({ message }) => ({ + kind: `message` as const, + id: message.id, + children: toArray( + q + .from({ messageChild: children.collection }) + .where(({ messageChild }) => + eq(messageChild.parentGroup, message.group), + ), + ), + })) + const toolRows = q + .from({ tool: tools.collection }) + .select(({ tool }) => ({ + kind: `tool` as const, + id: tool.id, + })) + + return q.unionAll(messageRows, toolRows).fn.select((row) => { + const includedChildren = + row.kind === `message` + ? (row.children as typeof row.children | null) + : null + + return { + kind: row.kind, + id: row.id, + leakedQuery: + includedChildren?.[0]?.value === 2 + ? q.from({ child: children.collection }) + : null, + } as any + }) + }) + + try { + await live.preload() + + expect(() => + children.write(`update`, { + id: 10, + parentGroup: 1, + value: 2, + }), + ).toThrow(`fn.select() cannot return a child query builder`) + } finally { + await Promise.all([ + live.cleanup(), + messages.collection.cleanup(), + tools.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest( + `rejects collapsed contributors that disagree by value, order, or outgoing route`, + async () => { + type CommentRow = { + id: number + userId: number + text: string + } + const users = createControlledCollection(`congruence-users`, [ + { id: 1, group: 1 }, + ]) + + const expectIncrementalRejection = async ( + mode: `value` | `order`, + ): Promise => { + const comments = createControlledCollection( + `congruence-${mode}-comments`, + [{ id: 1, userId: 1, text: `first` }], + ) + const live = createLiveQueryCollection({ + query: (q) => { + const joined = q + .from({ comment: comments.collection }) + .join({ user: users.collection }, ({ comment, user }) => + eq(comment.userId, user.id), + ) + return mode === `value` + ? joined.select(({ comment }) => ({ + publicId: comment.userId, + visible: comment.text, + })) + : joined + .orderBy(({ comment }) => comment.id) + .select(({ comment }) => ({ + publicId: comment.userId, + visible: `same`, + })) + }, + getKey: (row) => row.publicId, + }) + + try { + await live.preload() + expect(() => + comments.write(`insert`, { + id: 2, + userId: 1, + text: mode === `value` ? `second` : `ignored`, + }), + ).toThrow(`not congruent`) + } finally { + await live.cleanup() + await comments.collection.cleanup() + } + } + + await expectIncrementalRejection(`value`) + await expectIncrementalRejection(`order`) + + const parents = createControlledCollection(`congruence-parents`, [ + { id: 1, group: 1 }, + ]) + const children = createControlledCollection( + `congruence-children`, + [], + ) + const routed = createLiveQueryCollection({ + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + publicId: 1, + visible: `same`, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)), + ), + })), + getKey: (row) => row.publicId, + }) + + try { + await routed.preload() + expect(() => parents.write(`insert`, { id: 2, group: 2 })).toThrow( + `not congruent`, + ) + } finally { + await routed.cleanup() + await Promise.all([ + parents.collection.cleanup(), + children.collection.cleanup(), + users.collection.cleanup(), + ]) + } + }, + ) + + fcTest(`facade events observe the matching root publication`, async () => { + const driver = createCollectionDriver( + [{ id: 1, group: 1 }], + [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 2, value: 2 }, + ], + ) + const context = await driver.setup() + await driver.start?.(context) + const oldFacade = context.live.get(1)!.children + const callbackSnapshots: Array<{ + group: number | undefined + usesOldFacade: boolean + rows: Array + }> = [] + const subscription = oldFacade.subscribeChanges( + () => { + const current = context.live.get(1) + callbackSnapshots.push({ + group: current?.group, + usesOldFacade: current?.children === oldFacade, + rows: current + ? [...current.children.keys()].filter( + (key): key is number => typeof key === `number`, + ) + : [], + }) + }, + { includeInitialState: false }, + ) + + try { + context.parents.write(`update`, { id: 1, group: 2 }) + expect(callbackSnapshots).toEqual([ + { group: 2, usesOldFacade: false, rows: [20] }, + ]) + } finally { + subscription.unsubscribe() + await driver.cleanup(context) + } + }) + + fcTest( + `cleanup during root publication suppresses the prepared facade callback`, + async () => { + type NodeRow = { + id: number + kind: `parent` | `child` + group: number + value: number + } + const nodes = createControlledCollection(`publication-cleanup`, [ + { id: 1, kind: `parent`, group: 1, value: 1 }, + { id: 10, kind: `child`, group: 1, value: 1 }, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: nodes.collection }) + .where(({ parent }) => eq(parent.kind, `parent`)) + .select(({ parent }) => ({ + id: parent.id, + value: parent.value, + children: q + .from({ child: nodes.collection }) + .where(({ child }) => eq(child.kind, `child`)) + .where(({ child }) => eq(child.group, parent.group)), + })), + ) + + await live.preload() + const facade = live.get(1)!.children + const rootSnapshots: Array> = [] + const facadeSnapshots: Array> = [] + let cleanup: Promise | undefined + const rootSubscription = live.subscribeChanges( + () => { + rootSnapshots.push(facade.toArray.map(({ value }) => value)) + cleanup = facade.cleanup() + }, + { includeInitialState: false }, + ) + const facadeSubscription = facade.subscribeChanges( + () => facadeSnapshots.push(facade.toArray.map(({ value }) => value)), + { includeInitialState: false }, + ) + + try { + nodes.writeBatch([ + { + type: `update`, + value: { id: 1, kind: `parent`, group: 1, value: 2 }, + }, + { + type: `update`, + value: { id: 10, kind: `child`, group: 1, value: 2 }, + }, + ]) + await cleanup + + expect(rootSnapshots).toEqual([[2]]) + expect(facadeSnapshots).toEqual([]) + expect(facade.status).toBe(`cleaned-up`) + expect(facade.toArray).toEqual([]) + } finally { + rootSubscription.unsubscribe() + facadeSubscription.unsubscribe() + await Promise.all([live.cleanup(), nodes.collection.cleanup()]) + } + }, + ) + + fcTest( + `shared facades remain active until their last parent departs`, + async () => { + const driver = createCollectionDriver( + [ + { id: 1, group: 1 }, + { id: 2, group: 1 }, + ], + [{ id: 10, parentGroup: 1, value: 1 }], + ) + const context = await driver.setup() + await driver.start?.(context) + const sharedFacade = context.live.get(1)!.children + + try { + expect(context.live.get(2)!.children).toBe(sharedFacade) + context.parents.write(`delete`, { id: 1, group: 1 }) + expect(context.live.get(2)!.children).toBe(sharedFacade) + expect([...sharedFacade.keys()]).toEqual([10]) + expect(sharedFacade.status).toBe(`ready`) + } finally { + await driver.cleanup(context) + } + }, + ) + + fcTest(`a matched null singleton remains null`, async () => { + type NullableChild = { id: number; parentGroup: number; value: null } + const parents = createControlledCollection(`nullable-oracle-parents`, [ + { id: 1, group: 1 }, + ]) + const children = createControlledCollection( + `nullable-oracle-children`, + [{ id: 10, parentGroup: 1, value: null }], + ) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + value: materialize( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .select(({ child }) => child.value) + .findOne(), + ), + })), + ) + + try { + await live.preload() + expect(live.get(1)!.value).toBeNull() + } finally { + await Promise.all([ + live.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }) + + fcTest.prop( + [ + fc.record({ + smallId: fc.integer({ min: 2, max: 9 }), + wideId: fc.integer({ min: 10, max: 19 }), + }), + ], + oraclePropertyOptions(20, `includes-collection.public-key-order`), + )( + `uses one raw public-key order across Collection and inline materializations`, + async ({ smallId, wideId }) => { + const parents = createControlledCollection(`ordering-oracle-parents`, [ + { id: 1, group: 1 }, + ]) + const children = createControlledCollection(`ordering-oracle-children`, [ + { id: smallId, parentGroup: 1, value: smallId }, + { id: wideId, parentGroup: 1, value: wideId }, + ]) + const live = createLiveQueryCollection((q) => { + return q.from({ parent: parents.collection }).select(({ parent }) => { + const childRows = () => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .select(({ child }) => ({ id: child.id, value: child.value })) + + return { + id: parent.id, + facade: childRows(), + array: toArray(childRows()), + materialized: materialize(childRows()), + first: materialize(childRows().findOne()), + joined: concat( + toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .select(({ child }) => child.value), + ), + ), + } + }) + }) + + try { + await live.preload() + const result = live.get(1)! + const expectedIds = [smallId, wideId] + const facadeIds = result.facade.toArray.map((child) => child.id) + + expect(facadeIds).toEqual(expectedIds) + expect(result.array.map((child) => child.id)).toEqual(expectedIds) + expect(result.materialized.map((child) => child.id)).toEqual( + expectedIds, + ) + expect(result.first?.id).toBe(smallId) + expect(result.joined).toBe(`${smallId}${wideId}`) + } finally { + await Promise.all([ + live.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest.prop( + [orderSwapArbitrary], + oraclePropertyOptions(20, `includes-collection.layout-swap`), + )( + `propagates generated order-only child swaps through every materialization`, + async ({ length, swapIndex }) => { + type OrderedChild = ChildRow & { position: number; label: string } + const parents = createControlledCollection(`order-move-parents`, [ + { id: 1, group: 1 }, + ]) + const initialRows: Array = Array.from( + { length }, + (_, index) => ({ + id: index + 1, + parentGroup: 1, + value: index + 1, + position: index, + label: String(index + 1), + }), + ) + const children = createControlledCollection( + `order-move-children`, + initialRows, + ) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + const childRows = () => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + position: child.position, + label: child.label, + })) + + return { + id: parent.id, + facade: childRows(), + array: toArray(childRows()), + materialized: materialize(childRows()), + first: materialize(childRows().findOne()), + joined: concat( + toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => child.label), + ), + ), + } + }), + ) + + const project = () => { + const row = live.get(1)! + return { + facade: row.facade.toArray.map(({ id }) => id), + array: row.array.map(({ id }) => id), + materialized: row.materialized.map(({ id }) => id), + first: row.first?.id, + joined: row.joined, + } + } + + try { + await live.preload() + const facade = live.get(1)!.facade + const initialIds = initialRows.map(({ id }) => id) + expect(project()).toEqual({ + ...expectedMaterializations(initialIds), + first: initialIds[0], + joined: initialRows.map(({ label }) => label).join(``), + }) + + const first = initialRows[swapIndex]! + const second = initialRows[swapIndex + 1]! + children.writeBatch([ + { type: `update`, value: { ...first, position: second.position } }, + { type: `update`, value: { ...second, position: first.position } }, + ]) + const expectedIds = [...initialIds] + ;[expectedIds[swapIndex], expectedIds[swapIndex + 1]] = [ + expectedIds[swapIndex + 1]!, + expectedIds[swapIndex]!, + ] + + expect(live.get(1)!.facade).toBe(facade) + expect(project()).toEqual({ + ...expectedMaterializations(expectedIds), + first: expectedIds[0], + joined: expectedIds.join(``), + }) + } finally { + await Promise.all([ + live.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest( + `reconstructs nested conditional includes through guard transitions`, + async () => { + type GuardedParent = ParentRow & { active: boolean } + const parents = createControlledCollection( + `guarded-parents`, + [{ id: 1, group: 1, active: true }], + ) + const children = createControlledCollection(`guarded-children`, [ + { id: 10, parentGroup: 1, value: 1 }, + ]) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + const childRows = () => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + value: child.value, + })) + + return { + id: parent.id, + profile: caseWhen( + eq(parent.active, true), + { + kind: `active` as const, + facade: childRows(), + array: toArray(childRows()), + materialized: materialize(childRows()), + }, + { kind: `inactive` as const }, + ), + } + }), + ) + + const project = () => { + const profile = live.get(1)!.profile + if (profile.kind === `inactive`) return profile + const rows = (values: Iterable) => + [...values].map(({ id, value }) => ({ id, value })) + return { + kind: profile.kind, + facade: rows(profile.facade.values()), + array: rows(profile.array), + materialized: rows(profile.materialized), + } + } + + try { + await live.preload() + const initialProfile = live.get(1)!.profile + if (initialProfile.kind === `inactive`) { + throw new Error( + `Expected the initial conditional branch to be active`, + ) + } + const initialFacade = initialProfile.facade + expect(project()).toEqual({ + kind: `active`, + ...expectedMaterializations([{ id: 10, value: 1 }]), + }) + + parents.write(`update`, { id: 1, group: 1, active: false }) + expect(project()).toEqual({ kind: `inactive` }) + expect(initialFacade.toArray).toEqual([]) + expect(initialFacade.status).toBe(`ready`) + children.write(`insert`, { id: 20, parentGroup: 1, value: 2 }) + expect(project()).toEqual({ kind: `inactive` }) + + parents.write(`update`, { id: 1, group: 1, active: true }) + const reactivatedProfile = live.get(1)!.profile + if (reactivatedProfile.kind === `inactive`) { + throw new Error(`Expected the conditional branch to reactivate`) + } + expect(reactivatedProfile.facade).not.toBe(initialFacade) + expect(project()).toEqual({ + kind: `active`, + ...expectedMaterializations([ + { id: 10, value: 1 }, + { id: 20, value: 2 }, + ]), + }) + } finally { + await Promise.all([ + live.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest( + `matches recomputation for correlated aggregate child relations`, + async () => { + const parents = createControlledCollection(`aggregate-parents`, [ + { id: 1, group: 1, factor: 1 }, + { id: 2, group: 1, factor: -1 }, + { id: 3, group: 2, factor: 2 }, + ]) + const children = createControlledCollection(`aggregate-children`, [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 1, value: 2 }, + { id: 30, parentGroup: 2, value: 5 }, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + const summaries = () => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ + parentGroup: child.parentGroup, + count: count(child.id), + total: sum(multiply(child.value, parent.factor)), + })) + const total = () => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .select(({ child }) => ({ + count: count(child.id), + total: sum(multiply(child.value, parent.factor)), + })) + + return { + id: parent.id, + facade: summaries(), + array: toArray(summaries()), + materialized: materialize(summaries()), + implicit: materialize(total()), + } + }), + ) + + type Summary = { parentGroup: number; count: number; total: number } + const project = () => + live.toArray.map((row) => { + const clean = (values: Iterable

                      ) => + [...values].map(({ parentGroup, count: size, total }) => ({ + parentGroup, + count: size, + total, + })) + return { + id: row.id, + facade: clean(row.facade.values()), + array: clean(row.array), + materialized: clean(row.materialized), + implicit: row.implicit.map(({ count: size, total }) => ({ + count: size, + total, + })), + } + }) + + const expected = (groupOneTotal: number, groupOneCount: number) => [ + { + id: 1, + ...expectedMaterializations([ + { parentGroup: 1, count: groupOneCount, total: groupOneTotal }, + ]), + implicit: [{ count: groupOneCount, total: groupOneTotal }], + }, + { + id: 2, + ...expectedMaterializations([ + { + parentGroup: 1, + count: groupOneCount, + total: -groupOneTotal, + }, + ]), + implicit: [{ count: groupOneCount, total: -groupOneTotal }], + }, + { + id: 3, + ...expectedMaterializations([ + { parentGroup: 2, count: 1, total: 10 }, + ]), + implicit: [{ count: 1, total: 10 }], + }, + ] + + try { + await live.preload() + expect(project()).toEqual(expected(3, 2)) + + children.write(`update`, { id: 20, parentGroup: 1, value: 7 }) + expect(project()).toEqual(expected(8, 2)) + + children.write(`delete`, { id: 10, parentGroup: 1, value: 1 }) + expect(project()).toEqual(expected(7, 1)) + } finally { + await Promise.all([ + live.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest( + `evaluates correlated having clauses with parent context`, + async () => { + type HavingParent = ParentRow & { threshold: number } + const parents = createControlledCollection( + `having-parents`, + [ + { id: 1, group: 1, threshold: 1 }, + { id: 2, group: 1, threshold: 3 }, + ], + ) + const children = createControlledCollection(`having-children`, [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 1, value: 2 }, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + const summaries = () => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .groupBy(({ child }) => child.parentGroup) + .having(({ child }) => gt(count(child.id), parent.threshold)) + .select(({ child }) => ({ count: count(child.id) })) + + return { + id: parent.id, + facade: summaries(), + array: toArray(summaries()), + materialized: materialize(summaries()), + } + }), + ) + + const project = () => + live.toArray.map((row) => { + const counts = (values: Iterable<{ count: number }>) => + [...values].map(({ count: size }) => size) + return { + id: row.id, + facade: counts(row.facade.values()), + array: counts(row.array), + materialized: counts(row.materialized), + } + }) + + try { + await live.preload() + expect(project()).toEqual([ + { id: 1, ...expectedMaterializations([2]) }, + { id: 2, ...expectedMaterializations([]) }, + ]) + + parents.write(`update`, { id: 2, group: 1, threshold: 1 }) + expect(project()).toEqual([ + { id: 1, ...expectedMaterializations([2]) }, + { id: 2, ...expectedMaterializations([2]) }, + ]) + } finally { + await Promise.all([ + live.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest(`routes changes to non-key parent filter inputs`, async () => { + type FilterParent = ParentRow & { threshold: number } + const parents = createControlledCollection( + `parent-filter-input-parents`, + [{ id: 1, group: 1, threshold: 2 }], + ) + const children = createControlledCollection( + `parent-filter-input-children`, + [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 1, value: 3 }, + ], + ) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + const childRows = () => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .where(({ child }) => lt(child.value, parent.threshold)) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ id: child.id, value: child.value })) + + return { + id: parent.id, + facade: childRows(), + array: toArray(childRows()), + materialized: materialize(childRows()), + } + }), + ) + + const project = () => { + const row = live.get(1)! + const ids = (values: Iterable<{ id: number }>) => + [...values].map(({ id }) => id) + return { + facade: ids(row.facade.values()), + array: ids(row.array), + materialized: ids(row.materialized), + } + } + + try { + await live.preload() + expect(project()).toEqual(expectedMaterializations([10])) + + parents.write(`update`, { id: 1, group: 1, threshold: 4 }) + expect(project()).toEqual(expectedMaterializations([10, 20])) + + parents.write(`update`, { id: 1, group: 1, threshold: 0 }) + expect(project()).toEqual(expectedMaterializations([])) + } finally { + await Promise.all([ + live.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }) + + fcTest(`routes changes to parent-dependent child ordering`, async () => { + type OrderingParent = ParentRow & { direction: number } + const parents = createControlledCollection( + `parent-order-input-parents`, + [ + { id: 1, group: 1, direction: 1 }, + { id: 2, group: 1, direction: -1 }, + ], + ) + const children = createControlledCollection(`parent-order-input-children`, [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 1, value: 2 }, + ]) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + const childRows = () => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .orderBy(({ child }) => multiply(child.value, parent.direction)) + .select(({ child }) => ({ id: child.id, value: child.value })) + + return { + id: parent.id, + facade: childRows(), + array: toArray(childRows()), + materialized: materialize(childRows()), + } + }), + ) + + const project = (id: number) => { + const row = live.get(id)! + const rows = (values: Iterable<{ id: number; value: number }>) => + [...values].map(({ id: childId, value }) => ({ id: childId, value })) + return { + facade: rows(row.facade.values()), + array: rows(row.array), + materialized: rows(row.materialized), + } + } + + try { + await live.preload() + const ascendingFacade = live.get(1)!.facade + const descendingFacade = live.get(2)!.facade + expect(project(1)).toEqual( + expectedMaterializations([ + { id: 10, value: 1 }, + { id: 20, value: 2 }, + ]), + ) + expect(project(2)).toEqual( + expectedMaterializations([ + { id: 20, value: 2 }, + { id: 10, value: 1 }, + ]), + ) + expect(ascendingFacade).not.toBe(descendingFacade) + + parents.write(`update`, { id: 1, group: 1, direction: -1 }) + expect(live.get(1)!.facade).toBe(descendingFacade) + expect(ascendingFacade.toArray).toEqual([]) + expect(project(1)).toEqual( + expectedMaterializations([ + { id: 20, value: 2 }, + { id: 10, value: 1 }, + ]), + ) + } finally { + await Promise.all([ + live.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }) + + fcTest(`routes changes to parent-dependent child joins`, async () => { + type JoinParent = ParentRow & { offset: number } + type JoinedChild = ChildRow & { tagId: number } + type Tag = { id: number; label: string } + const parents = createControlledCollection( + `parent-join-parents`, + [ + { id: 1, group: 1, offset: 0 }, + { id: 2, group: 1, offset: 1 }, + ], + ) + const children = createControlledCollection( + `parent-join-children`, + [{ id: 10, parentGroup: 1, value: 1, tagId: 1 }], + ) + const tags = createControlledCollection(`parent-join-tags`, [ + { id: 1, label: `direct` }, + { id: 2, label: `offset` }, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + const childRows = () => + q + .from({ child: children.collection }) + .innerJoin({ tag: tags.collection }, ({ child, tag }) => + eq(tag.id, add(child.tagId, parent.offset)), + ) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .select(({ child, tag }) => ({ + id: child.id, + label: tag.label, + })) + + return { + id: parent.id, + facade: childRows(), + array: toArray(childRows()), + materialized: materialize(childRows()), + } + }), + ) + + const project = (id: number) => { + const row = live.get(id)! + const labels = (values: Iterable<{ label: string }>) => + [...values].map(({ label }) => label) + return { + facade: labels(row.facade.values()), + array: labels(row.array), + materialized: labels(row.materialized), + } + } + + try { + await live.preload() + const directFacade = live.get(1)!.facade + const offsetFacade = live.get(2)!.facade + expect(directFacade).not.toBe(offsetFacade) + expect(project(1)).toEqual(expectedMaterializations([`direct`])) + expect(project(2)).toEqual(expectedMaterializations([`offset`])) + + parents.write(`update`, { id: 1, group: 1, offset: 1 }) + expect(live.get(1)!.facade).toBe(offsetFacade) + expect(directFacade.toArray).toEqual([]) + expect(project(1)).toEqual(expectedMaterializations([`offset`])) + } finally { + await Promise.all([ + live.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + tags.collection.cleanup(), + ]) + } + }) + + fcTest.prop( + [ + fc.record({ + group: fc.integer({ min: -10, max: 10 }), + insertedId: fc.integer({ min: 10, max: 100 }), + confirmedId: fc.integer({ min: 101, max: 200 }), + value: fc.integer({ min: -10, max: 10 }), + }), + ], + oraclePropertyOptions(20, `includes-collection.optimistic-child-history`), + )( + `matches recomputation through optimistic child insert and delete confirmation and rollback`, + async ({ group, insertedId, confirmedId, value }) => { + type OptimisticAction = + | { type: `insert`; row: ChildRow; settlement: `confirm` | `rollback` } + | { type: `delete`; id: number; settlement: `confirm` | `rollback` } + const base = createCollectionDriver([{ id: 1, group }], []) + const driver: TraceDriver = { + ...base, + async apply(action, context, checkpoint) { + context.publications = [] + if (action.type === `insert`) { + const transaction = context.children.collection.insert({ + ...action.row, + }) + context.model.children.set(action.row.id, { ...action.row }) + checkpoint() + context.publications = [] + if (action.settlement === `confirm`) { + context.children.write(`insert`, action.row) + context.children.resolveSync() + await transaction.isPersisted.promise + } else { + context.model.children.delete(action.row.id) + const message = `rollback insert` + const persisted = transaction.isPersisted.promise.catch( + () => undefined, + ) + await withExpectedRejection(message, async () => { + context.children.rejectSync(new Error(message)) + await persisted + await flushPromises() + }) + } + return + } + + const previous = context.model.children.get(action.id) + if (!previous) throw new Error(`Missing optimistic delete row`) + const transaction = context.children.collection.delete(action.id) + context.model.children.delete(action.id) + checkpoint() + context.publications = [] + if (action.settlement === `confirm`) { + context.children.write(`delete`, previous) + context.children.resolveSync() + await transaction.isPersisted.promise + } else { + context.model.children.set(action.id, previous) + const message = `rollback delete` + const persisted = transaction.isPersisted.promise.catch( + () => undefined, + ) + await withExpectedRejection(message, async () => { + context.children.rejectSync(new Error(message)) + await persisted + await flushPromises() + }) + } + }, + } + const rolledBack = { + id: insertedId, + parentGroup: group, + value, + } + const confirmed = { + id: confirmedId, + parentGroup: group, + value: value + 1, + } + + await runTrace({ + steps: [ + { type: `insert`, row: rolledBack, settlement: `rollback` }, + { type: `insert`, row: confirmed, settlement: `confirm` }, + { type: `delete`, id: confirmedId, settlement: `rollback` }, + { type: `delete`, id: confirmedId, settlement: `confirm` }, + ], + driver, + projection: collectionProjection, + }) + }, + ) +}) diff --git a/packages/db/tests/query/includes-context-transport-oracle.test.ts b/packages/db/tests/query/includes-context-transport-oracle.test.ts new file mode 100644 index 0000000000..6fbc31fca2 --- /dev/null +++ b/packages/db/tests/query/includes-context-transport-oracle.test.ts @@ -0,0 +1,2181 @@ +import { describe, expect, test } from 'vitest' +import { + add, + and, + coalesce, + count, + createLiveQueryCollection, + eq, + gt, + lt, + lte, + materialize, + multiply, + sum, + toArray, +} from '../../src/query/index.js' +import { + attachRouteMetadata, + stripInternalRouteMetadata, +} from '../../src/query/compiler/route-metadata.js' +import { createControlledCollection } from './includes-oracle-helpers.js' +import type { Collection } from '../../src/collection/index.js' +import type { Context, QueryBuilder } from '../../src/query/builder/index.js' +import type { ControlledCollection } from './includes-oracle-helpers.js' + +type Cleanable = { cleanup: () => Promise } +type MaterializationForm = (typeof materializationForms)[number] + +type MaterializedForms = { + collection: { values: () => Iterable } + array: Iterable + materialized: Iterable +} + +const materializationForms = [`collection`, `array`, `materialized`] as const +const checkpoints = [`initial`, `parent-update`, `child-update`] as const + +const routeContextGrammar = { + parentProjection: { + shapes: [`field`, `whole-row`] as const, + }, + correlationDomain: { + states: [`unmatched`, `null`] as const, + }, + lexicalScope: { + scopes: [`immediate-parent`, `lexical-ancestor`] as const, + }, + aggregation: { + groupings: [`implicit`, `explicit`] as const, + placements: [`inside-aggregate`, `wrapped-aggregate`] as const, + }, + recursiveSource: { + boundaries: [`from-query-ref`, `joined-query-ref`, `union-branch`] as const, + phases: [ + `filter`, + `projection`, + `aggregate`, + `having`, + `order-window`, + ] as const, + }, + join: { + keySides: [`main`, `joined`] as const, + correlationAttachments: [`main`, `joined`] as const, + }, + unionIdentity: { + forms: [`union-from`, `union-all`] as const, + }, + derivedResult: { + boundaries: [`from-query-ref`, `joined-query-ref`, `union-all`] as const, + selections: [`expression`, `functional`] as const, + domains: [`non-null`, `nullable`] as const, + }, + namespaceCollision: { + locations: [`parent-alias`, `selected-field`] as const, + boundaries: [`direct`, `query-ref`, `join`, `group`] as const, + names: [ + `__parentContextIdentity`, + `__parentContext`, + `__correlationKey`, + `value`, + `identity`, + ] as const, + }, + publicSurface: { + shapes: [ + `object-query-ref-scalar`, + `nested-functional-spread`, + `opaque-wrapper`, + `functional-having-input`, + `nested-reference`, + `adversarial-key`, + `user-symbol`, + `implicit-join`, + ] as const, + }, +} as const + +const queryRefMetadataGrammar = { + routeModes: [`plain`, `routed`] as const, +} as const + +type QueryRefMetadataMode = (typeof queryRefMetadataGrammar.routeModes)[number] + +type ParentProjectionCell = { + family: `parent-projection` + shape: (typeof routeContextGrammar.parentProjection.shapes)[number] +} + +type CorrelationDomainCell = { + family: `correlation-domain` + state: (typeof routeContextGrammar.correlationDomain.states)[number] +} + +type LexicalScopeCell = { + family: `lexical-scope` + scope: (typeof routeContextGrammar.lexicalScope.scopes)[number] +} + +type AggregationCell = { + family: `aggregation` + grouping: (typeof routeContextGrammar.aggregation.groupings)[number] + placement: (typeof routeContextGrammar.aggregation.placements)[number] +} + +type RecursiveSourceCell = { + family: `recursive-source` + boundary: (typeof routeContextGrammar.recursiveSource.boundaries)[number] + phase: (typeof routeContextGrammar.recursiveSource.phases)[number] +} + +type JoinCell = { + family: `join` + keySide: (typeof routeContextGrammar.join.keySides)[number] + correlationAttachment: (typeof routeContextGrammar.join.correlationAttachments)[number] +} + +type UnionIdentityCell = { + family: `union-identity` + form: (typeof routeContextGrammar.unionIdentity.forms)[number] +} + +type DerivedResultCell = { + family: `derived-result` + boundary: (typeof routeContextGrammar.derivedResult.boundaries)[number] + selection: (typeof routeContextGrammar.derivedResult.selections)[number] + domain: (typeof routeContextGrammar.derivedResult.domains)[number] +} + +type NamespaceCollisionCell = { + family: `namespace-collision` + location: (typeof routeContextGrammar.namespaceCollision.locations)[number] + boundary: (typeof routeContextGrammar.namespaceCollision.boundaries)[number] + name: (typeof routeContextGrammar.namespaceCollision.names)[number] +} + +type PublicSurfaceCell = { + family: `public-surface` + shape: (typeof routeContextGrammar.publicSurface.shapes)[number] +} + +type GrammarCell = + | ParentProjectionCell + | CorrelationDomainCell + | LexicalScopeCell + | AggregationCell + | RecursiveSourceCell + | JoinCell + | UnionIdentityCell + | DerivedResultCell + | NamespaceCollisionCell + | PublicSurfaceCell + +const grammarCells: Array = [ + ...routeContextGrammar.parentProjection.shapes.map( + (shape): ParentProjectionCell => ({ + family: `parent-projection`, + shape, + }), + ), + ...routeContextGrammar.correlationDomain.states.map( + (state): CorrelationDomainCell => ({ + family: `correlation-domain`, + state, + }), + ), + ...routeContextGrammar.lexicalScope.scopes.map( + (scope): LexicalScopeCell => ({ family: `lexical-scope`, scope }), + ), + ...routeContextGrammar.aggregation.groupings.flatMap((grouping) => + routeContextGrammar.aggregation.placements.map( + (placement): AggregationCell => ({ + family: `aggregation`, + grouping, + placement, + }), + ), + ), + ...routeContextGrammar.recursiveSource.boundaries.flatMap((boundary) => + routeContextGrammar.recursiveSource.phases.map( + (phase): RecursiveSourceCell => ({ + family: `recursive-source`, + boundary, + phase, + }), + ), + ), + ...routeContextGrammar.join.keySides.flatMap((keySide) => + routeContextGrammar.join.correlationAttachments.map( + (correlationAttachment): JoinCell => ({ + family: `join`, + keySide, + correlationAttachment, + }), + ), + ), + ...routeContextGrammar.unionIdentity.forms.map( + (form): UnionIdentityCell => ({ family: `union-identity`, form }), + ), + ...routeContextGrammar.derivedResult.boundaries.flatMap((boundary) => + routeContextGrammar.derivedResult.selections.flatMap((selection) => + routeContextGrammar.derivedResult.domains.map( + (domain): DerivedResultCell => ({ + family: `derived-result`, + boundary, + selection, + domain, + }), + ), + ), + ), + ...routeContextGrammar.namespaceCollision.locations.flatMap((location) => + routeContextGrammar.namespaceCollision.boundaries.flatMap((boundary) => + routeContextGrammar.namespaceCollision.names.map( + (name): NamespaceCollisionCell => ({ + family: `namespace-collision`, + location, + boundary, + name, + }), + ), + ), + ), + ...routeContextGrammar.publicSurface.shapes.map( + (shape): PublicSurfaceCell => ({ family: `public-surface`, shape }), + ), +] + +async function cleanup( + live: Cleanable, + sources: Array<{ collection: Cleanable }>, +): Promise { + await live.cleanup() + await Promise.all(sources.map(({ collection }) => collection.cleanup())) +} + +function createGrammarCollection( + name: string, + rows: ReadonlyArray, +): ControlledCollection { + return createControlledCollection(name, rows, { autoIndex: `eager` }) +} + +function includeInEveryForm( + query: QueryBuilder, +) { + return { + collection: query, + array: toArray(query), + materialized: materialize(query), + } +} + +function readEveryForm( + forms: MaterializedForms, + project: (rows: Iterable) => U, +): Record { + return { + collection: project(forms.collection.values()), + array: project(forms.array), + materialized: project(forms.materialized), + } +} + +function expectEveryForm( + forms: MaterializedForms, + project: (rows: Iterable) => U, + expected: U, +): void { + expect(readEveryForm(forms, project)).toEqual( + Object.fromEntries(materializationForms.map((form) => [form, expected])), + ) +} + +function ids(rows: Iterable<{ id: number }>): Array { + return [...rows].map((row) => row.id) +} + +function grammarCellName(cell: GrammarCell): string { + switch (cell.family) { + case `parent-projection`: + return `${cell.family} / ${cell.shape}` + case `correlation-domain`: + return `${cell.family} / ${cell.state}` + case `lexical-scope`: + return `${cell.family} / ${cell.scope}` + case `aggregation`: + return `${cell.family} / ${cell.grouping} / ${cell.placement}` + case `recursive-source`: + return `${cell.family} / ${cell.boundary} / ${cell.phase}` + case `join`: + return `${cell.family} / ${cell.keySide}-side key / ${cell.correlationAttachment}-side correlation` + case `union-identity`: + return `${cell.family} / ${cell.form}` + case `derived-result`: + return `${cell.family} / ${cell.boundary} / ${cell.selection} / ${cell.domain}` + case `namespace-collision`: + return `${cell.family} / ${cell.location} / ${cell.boundary} / ${cell.name}` + case `public-surface`: + return `${cell.family} / ${cell.shape}` + } +} + +async function runParentProjectionCell({ + shape, +}: ParentProjectionCell): Promise { + type ParentRow = { id: number; group: number; token: string } + const parentRows: Array = [ + { id: 1, group: 1, token: `one` }, + { id: 2, group: 1, token: `two` }, + ] + const parents = createGrammarCollection( + `projection-${shape}-parents`, + parentRows, + ) + const children = createGrammarCollection(`projection-${shape}-children`, [ + { id: 10, parentGroup: 1 }, + { id: 20, parentGroup: 1 }, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + const correlated = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + const rows = correlated + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + parentSnapshot: + shape === `field` + ? { token: parent.token } + : coalesce(parent, null), + })) + return { id: parent.id, ...includeInEveryForm(rows) } + }), + ) + + const expected = (parentId: number) => { + const parent = parents.collection.get(parentId)! + return children.collection.toArray + .filter((child) => child.parentGroup === parent.group) + .map(({ id }) => ({ + id, + parentSnapshot: + shape === `field` ? { token: parent.token } : { ...parent }, + })) + } + const project = (rows: Iterable<{ id: number; parentSnapshot: unknown }>) => + [...rows].map(({ id, parentSnapshot }) => ({ id, parentSnapshot })) + const assertParents = () => { + for (const parent of parents.collection.toArray) { + expectEveryForm(live.get(parent.id)!, project, expected(parent.id)) + } + } + + try { + await live.preload() + assertParents() + + const updatedParent = { id: 1, group: 1, token: `updated` } + parents.write(`update`, updatedParent) + assertParents() + + children.write(`insert`, { id: 30, parentGroup: 1 }) + assertParents() + } finally { + await cleanup(live, [parents, children]) + } +} + +async function runCorrelationDomainCell({ + state, +}: CorrelationDomainCell): Promise { + type ParentRow = { id: number; group: number | null } + type ChildRow = { id: number; parentGroup: number | null } + const parents = createGrammarCollection( + `correlation-${state}-parents`, + [{ id: 1, group: state === `null` ? null : 999 }], + ) + const children = createGrammarCollection( + `correlation-${state}-children`, + [ + { id: 10, parentGroup: null }, + { id: 20, parentGroup: 1 }, + ], + ) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + const rows = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ id: child.id })) + return { id: parent.id, ...includeInEveryForm(rows) } + }), + ) + + const expected = () => { + const parentGroup = parents.collection.get(1)!.group + return children.collection.toArray + .filter( + (child) => + parentGroup != null && + child.parentGroup != null && + child.parentGroup === parentGroup, + ) + .map(({ id }) => id) + } + const assertResult = () => expectEveryForm(live.get(1)!, ids, expected()) + + try { + await live.preload() + assertResult() + + parents.write(`update`, { id: 1, group: 1 }) + assertResult() + + children.write(`insert`, { id: 30, parentGroup: 1 }) + assertResult() + } finally { + await cleanup(live, [parents, children]) + } +} + +async function runLexicalScopeCell({ scope }: LexicalScopeCell): Promise { + const parents = createGrammarCollection(`scope-${scope}-parents`, [ + { id: 1, group: 1, threshold: 2 }, + { id: 2, group: 1, threshold: 4 }, + ]) + const children = createGrammarCollection(`scope-${scope}-children`, [ + { id: 10, parentGroup: 1, group: 10, value: 1 }, + { id: 20, parentGroup: 1, group: 20, value: 3 }, + ]) + const grandchildren = createGrammarCollection( + `scope-${scope}-grandchildren`, + [ + { id: 100, parentGroup: 10, value: 1 }, + { id: 200, parentGroup: 10, value: 3 }, + ], + ) + + if (scope === `immediate-parent`) { + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + const childRows = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .where(({ child }) => lt(child.value, parent.threshold)) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ id: child.id })) + return { id: parent.id, ...includeInEveryForm(childRows) } + }), + ) + + const expected = (parentId: number) => { + const parent = parents.collection.get(parentId)! + return children.collection.toArray + .filter( + (child) => + child.parentGroup === parent.group && + child.value < parent.threshold, + ) + .map(({ id }) => id) + .sort((left, right) => left - right) + } + + try { + await live.preload() + for (const parent of parents.collection.toArray) { + expectEveryForm(live.get(parent.id)!, ids, expected(parent.id)) + } + + parents.write(`update`, { id: 1, group: 1, threshold: 4 }) + expectEveryForm(live.get(1)!, ids, expected(1)) + + children.write(`insert`, { + id: 30, + parentGroup: 1, + group: 30, + value: 2, + }) + for (const parent of parents.collection.toArray) { + expectEveryForm(live.get(parent.id)!, ids, expected(parent.id)) + } + } finally { + await cleanup(live, [parents, children, grandchildren]) + } + return + } + + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + const childRows = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .select(({ child }) => { + const grandchildRows = q + .from({ grandchild: grandchildren.collection }) + .where(({ grandchild }) => + eq(grandchild.parentGroup, child.group), + ) + .where(({ grandchild }) => lt(grandchild.value, parent.threshold)) + .orderBy(({ grandchild }) => grandchild.id) + .select(({ grandchild }) => ({ id: grandchild.id })) + + return { + id: child.id, + ...includeInEveryForm(grandchildRows), + } + }) + + return { id: parent.id, ...includeInEveryForm(childRows) } + }), + ) + + const expected = (parentId: number, childGroup: number) => { + const parent = parents.collection.get(parentId)! + return grandchildren.collection.toArray + .filter( + (grandchild) => + grandchild.parentGroup === childGroup && + grandchild.value < parent.threshold, + ) + .map(({ id }) => id) + .sort((left, right) => left - right) + } + + const projectOuterForm = ( + parentId: number, + outerForm: MaterializationForm, + ) => { + const parentRow = live.get(parentId)! + const outerRows = + outerForm === `collection` + ? parentRow.collection.values() + : parentRow[outerForm] + return [...outerRows].map((child) => ({ + id: child.id, + grandchildren: readEveryForm(child, ids), + })) + } + + const assertNestedProduct = (parentId: number) => { + const expectedRows = children.collection.toArray + .filter( + (child) => + child.parentGroup === parents.collection.get(parentId)!.group, + ) + .map((child) => ({ + id: child.id, + grandchildren: Object.fromEntries( + materializationForms.map((form) => [ + form, + expected(parentId, child.group), + ]), + ), + })) + for (const outerForm of materializationForms) { + expect(projectOuterForm(parentId, outerForm)).toEqual(expectedRows) + } + } + + try { + await live.preload() + assertNestedProduct(1) + assertNestedProduct(2) + + parents.write(`update`, { id: 1, group: 1, threshold: 4 }) + assertNestedProduct(1) + + grandchildren.write(`insert`, { id: 300, parentGroup: 10, value: 2 }) + assertNestedProduct(1) + assertNestedProduct(2) + } finally { + await cleanup(live, [parents, children, grandchildren]) + } +} + +async function runAggregationCell({ + grouping, + placement, +}: AggregationCell): Promise { + const name = `${grouping}-${placement}` + const parents = createGrammarCollection(`aggregate-${name}-parents`, [ + { id: 1, group: 1, factor: 2 }, + { id: 2, group: 1, factor: -1 }, + ]) + const children = createGrammarCollection(`aggregate-${name}-children`, [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 1, value: 2 }, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + const correlated = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + const rows = + grouping === `explicit` + ? correlated + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ + score: + placement === `inside-aggregate` + ? sum(add(child.value, parent.factor)) + : add(sum(child.value), parent.factor), + })) + : correlated.select(({ child }) => ({ + score: + placement === `inside-aggregate` + ? sum(add(child.value, parent.factor)) + : add(sum(child.value), parent.factor), + })) + return { id: parent.id, ...includeInEveryForm(rows) } + }), + ) + + const expected = (parentId: number) => { + const parent = parents.collection.get(parentId)! + const total = children.collection.toArray + .filter((child) => child.parentGroup === parent.group) + .reduce((result, child) => result + child.value, 0) + const childCount = children.collection.toArray.filter( + (child) => child.parentGroup === parent.group, + ).length + return [ + placement === `inside-aggregate` + ? total + childCount * parent.factor + : total + parent.factor, + ] + } + + const project = (rows: Iterable<{ score: number }>) => + [...rows].map(({ score }) => score) + + try { + await live.preload() + for (const parent of parents.collection.toArray) { + expectEveryForm(live.get(parent.id)!, project, expected(parent.id)) + } + + parents.write(`update`, { id: 2, group: 1, factor: 3 }) + expectEveryForm(live.get(2)!, project, expected(2)) + + children.write(`insert`, { id: 30, parentGroup: 1, value: 4 }) + for (const parent of parents.collection.toArray) { + expectEveryForm(live.get(parent.id)!, project, expected(parent.id)) + } + } finally { + await cleanup(live, [parents, children]) + } +} + +type CandidateRow = { + id: number + parentGroup: number + value: number +} + +async function runRecursiveSourceCell({ + boundary, + phase, +}: RecursiveSourceCell): Promise { + const name = `${boundary}-${phase}` + const initialParameter = + phase === `order-window` + ? [1, -1] + : phase === `projection` || phase === `aggregate` + ? [10, 20] + : phase === `having` + ? [0, 1] + : [1, 2] + const parents = createGrammarCollection(`recursive-${name}-parents`, [ + { id: 1, group: 1, parameter: initialParameter[0]! }, + { id: 2, group: 1, parameter: initialParameter[1]! }, + ]) + const initialCandidates: Array = [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 1, value: 2 }, + { id: 30, parentGroup: 1, value: 3 }, + { id: 40, parentGroup: 1, value: 4 }, + ] + const candidates = createGrammarCollection( + `recursive-${name}-candidates`, + initialCandidates, + ) + const left = createGrammarCollection( + `recursive-${name}-left`, + initialCandidates.filter(({ id }) => id % 20 === 10), + ) + const right = createGrammarCollection( + `recursive-${name}-right`, + initialCandidates.filter(({ id }) => id % 20 === 0), + ) + const anchors = createGrammarCollection(`recursive-${name}-anchors`, [ + ...initialCandidates.map(({ id, parentGroup }) => ({ id, parentGroup })), + { id: 50, parentGroup: 1 }, + ]) + + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + const buildCandidates = (source: Collection) => { + const correlated = q + .from({ candidate: source }) + .where(({ candidate }) => eq(candidate.parentGroup, parent.group)) + switch (phase) { + case `filter`: + return correlated + .where(({ candidate }) => + lte(candidate.value, parent.parameter), + ) + .select(({ candidate }) => ({ + id: candidate.id, + parentGroup: candidate.parentGroup, + value: candidate.id, + })) + case `projection`: + return correlated.select(({ candidate }) => ({ + id: candidate.id, + parentGroup: candidate.parentGroup, + value: add(candidate.value, parent.parameter), + })) + case `aggregate`: + return correlated + .groupBy(({ candidate }) => [ + candidate.id, + candidate.parentGroup, + ]) + .select(({ candidate }) => ({ + id: candidate.id, + parentGroup: candidate.parentGroup, + value: add(count(candidate.id), parent.parameter), + })) + case `having`: + return correlated + .groupBy(({ candidate }) => [ + candidate.id, + candidate.parentGroup, + ]) + .having(({ candidate }) => + gt(count(candidate.id), parent.parameter), + ) + .select(({ candidate }) => ({ + id: candidate.id, + parentGroup: candidate.parentGroup, + value: count(candidate.id), + })) + case `order-window`: + return correlated + .orderBy(({ candidate }) => + multiply(candidate.value, parent.parameter), + ) + .orderBy(({ candidate }) => candidate.id) + .limit(1) + .select(({ candidate }) => ({ + id: candidate.id, + parentGroup: candidate.parentGroup, + value: candidate.id, + })) + } + } + + // unionAll branches must use distinct lexical aliases. Keep these two + // adapters explicit so the grammar exercises the public builder rules + // without erasing their types behind a cast. + const buildLeftCandidates = () => { + const correlated = q + .from({ leftCandidate: left.collection }) + .where(({ leftCandidate }) => + eq(leftCandidate.parentGroup, parent.group), + ) + switch (phase) { + case `filter`: + return correlated + .where(({ leftCandidate }) => + lte(leftCandidate.value, parent.parameter), + ) + .select(({ leftCandidate }) => ({ + id: leftCandidate.id, + parentGroup: leftCandidate.parentGroup, + value: leftCandidate.id, + })) + case `projection`: + return correlated.select(({ leftCandidate }) => ({ + id: leftCandidate.id, + parentGroup: leftCandidate.parentGroup, + value: add(leftCandidate.value, parent.parameter), + })) + case `aggregate`: + return correlated + .groupBy(({ leftCandidate }) => [ + leftCandidate.id, + leftCandidate.parentGroup, + ]) + .select(({ leftCandidate }) => ({ + id: leftCandidate.id, + parentGroup: leftCandidate.parentGroup, + value: add(count(leftCandidate.id), parent.parameter), + })) + case `having`: + return correlated + .groupBy(({ leftCandidate }) => [ + leftCandidate.id, + leftCandidate.parentGroup, + ]) + .having(({ leftCandidate }) => + gt(count(leftCandidate.id), parent.parameter), + ) + .select(({ leftCandidate }) => ({ + id: leftCandidate.id, + parentGroup: leftCandidate.parentGroup, + value: count(leftCandidate.id), + })) + case `order-window`: + return correlated + .orderBy(({ leftCandidate }) => + multiply(leftCandidate.value, parent.parameter), + ) + .orderBy(({ leftCandidate }) => leftCandidate.id) + .limit(1) + .select(({ leftCandidate }) => ({ + id: leftCandidate.id, + parentGroup: leftCandidate.parentGroup, + value: leftCandidate.id, + })) + } + } + + const buildRightCandidates = () => { + const correlated = q + .from({ rightCandidate: right.collection }) + .where(({ rightCandidate }) => + eq(rightCandidate.parentGroup, parent.group), + ) + switch (phase) { + case `filter`: + return correlated + .where(({ rightCandidate }) => + lte(rightCandidate.value, parent.parameter), + ) + .select(({ rightCandidate }) => ({ + id: rightCandidate.id, + parentGroup: rightCandidate.parentGroup, + value: rightCandidate.id, + })) + case `projection`: + return correlated.select(({ rightCandidate }) => ({ + id: rightCandidate.id, + parentGroup: rightCandidate.parentGroup, + value: add(rightCandidate.value, parent.parameter), + })) + case `aggregate`: + return correlated + .groupBy(({ rightCandidate }) => [ + rightCandidate.id, + rightCandidate.parentGroup, + ]) + .select(({ rightCandidate }) => ({ + id: rightCandidate.id, + parentGroup: rightCandidate.parentGroup, + value: add(count(rightCandidate.id), parent.parameter), + })) + case `having`: + return correlated + .groupBy(({ rightCandidate }) => [ + rightCandidate.id, + rightCandidate.parentGroup, + ]) + .having(({ rightCandidate }) => + gt(count(rightCandidate.id), parent.parameter), + ) + .select(({ rightCandidate }) => ({ + id: rightCandidate.id, + parentGroup: rightCandidate.parentGroup, + value: count(rightCandidate.id), + })) + case `order-window`: + return correlated + .orderBy(({ rightCandidate }) => + multiply(rightCandidate.value, parent.parameter), + ) + .orderBy(({ rightCandidate }) => rightCandidate.id) + .limit(1) + .select(({ rightCandidate }) => ({ + id: rightCandidate.id, + parentGroup: rightCandidate.parentGroup, + value: rightCandidate.id, + })) + } + } + + switch (boundary) { + case `from-query-ref`: { + const routed = q + .from({ result: buildCandidates(candidates.collection) }) + .where(({ result }) => eq(result.parentGroup, parent.group)) + .select(({ result }) => ({ + id: result.id, + value: result.value, + })) + return { id: parent.id, ...includeInEveryForm(routed) } + } + case `joined-query-ref`: { + const routed = q + .from({ anchor: anchors.collection }) + .innerJoin( + { result: buildCandidates(candidates.collection) }, + ({ anchor, result }) => eq(anchor.id, result.id), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ result }) => ({ + id: result.id, + value: result.value, + })) + return { id: parent.id, ...includeInEveryForm(routed) } + } + case `union-branch`: { + const routed = q + .unionAll(buildLeftCandidates(), buildRightCandidates()) + .innerJoin({ anchor: anchors.collection }, ({ id, anchor }) => + eq(id, anchor.id), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ id, value }) => ({ id, value })) + return { id: parent.id, ...includeInEveryForm(routed) } + } + } + }), + ) + + const modelRows = new Map(initialCandidates.map((row) => [row.id, row])) + const leftModelIds = new Set( + initialCandidates.filter(({ id }) => id % 20 === 10).map(({ id }) => id), + ) + const expected = (parentId: number) => { + const parent = parents.collection.get(parentId)! + const correlated = [...modelRows.values()].filter( + (candidate) => candidate.parentGroup === parent.group, + ) + switch (phase) { + case `filter`: + return correlated + .filter((candidate) => candidate.value <= parent.parameter) + .map((candidate) => ({ id: candidate.id, value: candidate.id })) + case `projection`: + return correlated.map((candidate) => ({ + id: candidate.id, + value: candidate.value + parent.parameter, + })) + case `aggregate`: + return correlated.map((candidate) => ({ + id: candidate.id, + value: 1 + parent.parameter, + })) + case `having`: + return parent.parameter < 1 + ? correlated.map((candidate) => ({ id: candidate.id, value: 1 })) + : [] + case `order-window`: { + const partitions = + boundary === `union-branch` + ? [ + correlated.filter(({ id }) => leftModelIds.has(id)), + correlated.filter(({ id }) => !leftModelIds.has(id)), + ] + : [correlated] + return partitions.flatMap((partition) => + partition + .sort( + (leftRow, rightRow) => + leftRow.value * parent.parameter - + rightRow.value * parent.parameter || leftRow.id - rightRow.id, + ) + .slice(0, 1) + .map((candidate) => ({ id: candidate.id, value: candidate.id })), + ) + } + } + } + + const project = (rows: Iterable<{ id: number; value: number }>) => + [...rows] + .map(({ id, value }) => ({ id, value })) + .sort((leftRow, rightRow) => leftRow.id - rightRow.id) + const assertParents = () => { + for (const parent of parents.collection.toArray) { + expectEveryForm(live.get(parent.id)!, project, expected(parent.id)) + } + } + + try { + await live.preload() + assertParents() + + const updatedParameter = + phase === `order-window` + ? -1 + : phase === `projection` || phase === `aggregate` + ? 30 + : phase === `having` + ? 1 + : 4 + parents.write(`update`, { id: 1, group: 1, parameter: updatedParameter }) + assertParents() + + const inserted = { + id: 50, + parentGroup: 1, + value: phase === `filter` ? 1 : 5, + } + modelRows.set(inserted.id, inserted) + if (boundary === `union-branch`) right.write(`insert`, inserted) + else candidates.write(`insert`, inserted) + assertParents() + } finally { + await cleanup(live, [parents, candidates, left, right, anchors]) + } +} + +async function runUnionIdentityCell({ + form, +}: UnionIdentityCell): Promise { + const parents = createGrammarCollection(`identity-${form}-parents`, [ + { id: 1, group: 1, parameter: 10 }, + { id: 2, group: 1, parameter: 20 }, + ]) + const left = createGrammarCollection(`identity-${form}-left`, [ + { id: 10, parentGroup: 1, value: 1 }, + ]) + const right = createGrammarCollection( + `identity-${form}-right`, + [{ id: 20, parentGroup: 1, value: 2 }], + ) + const anchors = createGrammarCollection(`identity-${form}-anchors`, [ + { id: 10, parentGroup: 1 }, + { id: 20, parentGroup: 1 }, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + const unionAllRows = () => { + const leftRows = q + .from({ leftCandidate: left.collection }) + .select(({ leftCandidate }) => ({ + id: leftCandidate.id, + value: add(leftCandidate.value, parent.parameter), + })) + const rightRows = q + .from({ rightCandidate: right.collection }) + .select(({ rightCandidate }) => ({ + id: rightCandidate.id, + value: add(rightCandidate.value, parent.parameter), + })) + return q + .unionAll(leftRows, rightRows) + .innerJoin({ anchor: anchors.collection }, ({ id, anchor }) => + eq(id, anchor.id), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ id, value }) => ({ id, value })) + } + const unionFromRows = () => + q + .unionAll({ + leftCandidate: left.collection, + rightCandidate: right.collection, + }) + .innerJoin( + { anchor: anchors.collection }, + ({ leftCandidate, rightCandidate, anchor }) => + eq(coalesce(leftCandidate.id, rightCandidate.id), anchor.id), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ leftCandidate, rightCandidate }) => ({ + id: coalesce(leftCandidate.id, rightCandidate.id), + value: add( + coalesce(leftCandidate.value, rightCandidate.value), + parent.parameter, + ), + })) + return form === `union-all` + ? { id: parent.id, ...includeInEveryForm(unionAllRows()) } + : { id: parent.id, ...includeInEveryForm(unionFromRows()) } + }), + ) + + const project = (rows: Iterable<{ id: number; value: number }>) => + [...rows] + .map(({ id, value }) => ({ id, value })) + .sort((leftRow, rightRow) => leftRow.id - rightRow.id) + const expected = (parentId: number) => { + const parent = parents.collection.get(parentId)! + return [...left.collection.toArray, ...right.collection.toArray] + .filter((candidate) => candidate.parentGroup === parent.group) + .map((candidate) => ({ + id: candidate.id, + value: candidate.value + parent.parameter, + })) + .sort((leftRow, rightRow) => leftRow.id - rightRow.id) + } + const keys = (parentId: number) => { + const collection = live.get(parentId)!.collection + return new Map( + collection.toArray.map((row) => [row.id, collection.getKeyFromItem(row)]), + ) + } + const assertParents = () => { + for (const parent of parents.collection.toArray) { + expectEveryForm(live.get(parent.id)!, project, expected(parent.id)) + } + } + const assertRouteIndependentKeys = () => expect(keys(2)).toEqual(keys(1)) + + try { + await live.preload() + assertParents() + assertRouteIndependentKeys() + const initialKeys = keys(1) + + parents.write(`update`, { id: 1, group: 1, parameter: 30 }) + assertParents() + expect(keys(1)).toEqual(initialKeys) + assertRouteIndependentKeys() + + right.write(`update`, { id: 20, parentGroup: 1, value: 5 }) + assertParents() + expect(keys(1)).toEqual(initialKeys) + assertRouteIndependentKeys() + } finally { + await cleanup(live, [parents, left, right, anchors]) + } +} + +type DerivedCandidateRow = { + id: number + value: number | null +} + +async function runDerivedResultCell({ + boundary, + selection, + domain, +}: DerivedResultCell): Promise { + const name = `${boundary}-${selection}-${domain}` + const parents = createGrammarCollection(`derived-${name}-parents`, [ + { id: 1, group: 1 }, + { id: 2, group: 2 }, + ]) + const initialCandidates: Array = [ + { id: 10, value: 10 }, + { id: 15, value: domain === `nullable` ? null : 15 }, + { id: 20, value: 20 }, + ] + const candidates = createGrammarCollection( + `derived-${name}-candidates`, + initialCandidates, + ) + const left = createGrammarCollection( + `derived-${name}-left`, + initialCandidates.filter(({ id }) => id !== 20), + ) + const right = createGrammarCollection( + `derived-${name}-right`, + initialCandidates.filter(({ id }) => id === 20), + ) + const anchors = createGrammarCollection(`derived-${name}-anchors`, [ + { id: 100, parentGroup: 1, value: 10 }, + { id: 200, parentGroup: 2, value: 20 }, + { id: 300, parentGroup: 2, value: 30 }, + ]) + + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + if (boundary === `union-all`) { + const leftRows = q.from({ leftCandidate: left.collection }) + const rightRows = q.from({ rightCandidate: right.collection }) + const values = + selection === `expression` + ? q.unionAll( + leftRows.select(({ leftCandidate }) => + coalesce(leftCandidate.value, null), + ), + rightRows.select(({ rightCandidate }) => + coalesce(rightCandidate.value, null), + ), + ) + : q.unionAll( + leftRows.fn.select( + ({ leftCandidate }) => leftCandidate.value, + ), + rightRows.fn.select( + ({ rightCandidate }) => rightCandidate.value, + ), + ) + const rows = q + .from({ anchor: anchors.collection }) + .innerJoin({ value: values }, ({ anchor, value }) => + eq(anchor.value, value), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ anchor, value }) => ({ id: anchor.id, value })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + if (selection === `expression`) { + const values = q + .from({ candidate: candidates.collection }) + .select(({ candidate }) => coalesce(candidate.value, null)) + if (boundary === `from-query-ref`) { + const rows = q + .from({ value: values }) + .innerJoin({ anchor: anchors.collection }, ({ value, anchor }) => + eq(value, anchor.value), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ anchor, value }) => ({ id: anchor.id, value })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + const rows = q + .from({ anchor: anchors.collection }) + .innerJoin({ value: values }, ({ anchor, value }) => + eq(anchor.value, value), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ anchor, value }) => ({ id: anchor.id, value })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + const values = q + .from({ candidate: candidates.collection }) + .fn.select(({ candidate }) => candidate.value) + if (boundary === `from-query-ref`) { + const rows = q + .from({ value: values }) + .innerJoin({ anchor: anchors.collection }, ({ value, anchor }) => + eq(value, anchor.value), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ anchor, value }) => ({ id: anchor.id, value })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + const rows = q + .from({ anchor: anchors.collection }) + .innerJoin({ value: values }, ({ anchor, value }) => + eq(anchor.value, value), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ anchor, value }) => ({ id: anchor.id, value })) + return { id: parent.id, ...includeInEveryForm(rows) } + }), + ) + + const modelRows = new Map(initialCandidates.map((row) => [row.id, row])) + const expected = (parentId: number) => { + const parent = parents.collection.get(parentId)! + return [...modelRows.values()] + .flatMap((candidate) => + anchors.collection.toArray + .filter( + (anchor) => + candidate.value != null && + anchor.value === candidate.value && + anchor.parentGroup === parent.group, + ) + .map((anchor) => ({ id: anchor.id, value: candidate.value })), + ) + .sort((leftRow, rightRow) => leftRow.id - rightRow.id) + } + const project = ( + rows: Iterable<{ id: number; value: number | null | undefined }>, + ) => + [...rows] + .map(({ id, value }) => ({ id, value })) + .sort((leftRow, rightRow) => leftRow.id - rightRow.id) + const assertParents = () => { + for (const parent of parents.collection.toArray) { + expectEveryForm(live.get(parent.id)!, project, expected(parent.id)) + } + } + + try { + await live.preload() + assertParents() + + parents.write(`update`, { id: 1, group: 2 }) + assertParents() + + const updated = { id: 20, value: 30 } + modelRows.set(updated.id, updated) + if (boundary === `union-all`) right.write(`update`, updated) + else candidates.write(`update`, updated) + assertParents() + } finally { + await cleanup(live, [parents, candidates, left, right, anchors]) + } +} + +function expectNoRouteMetadata(row: object): void { + expect(Object.hasOwn(row, `__correlationKey`)).toBe(false) + expect(Object.hasOwn(row, `__parentContext`)).toBe(false) +} + +async function runQueryRefMetadataCell( + routeMode: QueryRefMetadataMode, +): Promise { + const anchors = createGrammarCollection(`metadata-${routeMode}-anchors`, [ + { id: 1, candidateId: 10, parentGroup: 1 }, + ]) + const candidates = createGrammarCollection( + `metadata-${routeMode}-candidates`, + [{ id: 10, label: `ten` }], + ) + + if (routeMode === `plain`) { + const live = createLiveQueryCollection((q) => { + const projectedCandidates = q + .from({ candidate: candidates.collection }) + .select(({ candidate }) => ({ + id: candidate.id, + label: candidate.label, + })) + return q + .from({ anchor: anchors.collection }) + .innerJoin( + { candidateResult: projectedCandidates }, + ({ anchor, candidateResult }) => + eq(anchor.candidateId, candidateResult.id), + ) + .select(({ candidateResult }) => candidateResult) + }) + + try { + await live.preload() + expectNoRouteMetadata(live.toArray[0]!) + + candidates.write(`update`, { id: 10, label: `updated` }) + expectNoRouteMetadata(live.toArray[0]!) + } finally { + await cleanup(live, [anchors, candidates]) + } + return + } + + const parents = createGrammarCollection(`metadata-routed-parents`, [ + { id: 1, group: 1 }, + ]) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + const projectedCandidates = q + .from({ candidate: candidates.collection }) + .select(({ candidate }) => ({ + id: candidate.id, + label: candidate.label, + })) + const rows = q + .from({ anchor: anchors.collection }) + .innerJoin( + { candidateResult: projectedCandidates }, + ({ anchor, candidateResult }) => + eq(anchor.candidateId, candidateResult.id), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ candidateResult }) => candidateResult) + return { id: parent.id, ...includeInEveryForm(rows) } + }), + ) + const assertClean = () => { + const forms = live.get(1)! + for (const rows of [ + forms.collection.values(), + forms.array, + forms.materialized, + ]) { + for (const row of rows) expectNoRouteMetadata(row) + } + } + + try { + await live.preload() + assertClean() + + parents.write(`update`, { id: 1, group: 2 }) + assertClean() + + anchors.write(`update`, { id: 1, candidateId: 10, parentGroup: 2 }) + assertClean() + } finally { + await cleanup(live, [parents, anchors, candidates]) + } +} + +async function runJoinCell({ + keySide, + correlationAttachment, +}: JoinCell): Promise { + const name = `${keySide}-${correlationAttachment}` + const parents = createGrammarCollection(`join-${name}-parents`, [ + { id: 1, group: 1, offset: 0 }, + { id: 2, group: 2, offset: 1 }, + ]) + const children = createGrammarCollection(`join-${name}-children`, [ + { id: 10, parentGroup: 1, tagId: 2 }, + { id: 20, parentGroup: 2, tagId: 2 }, + ]) + const tags = createGrammarCollection(`join-${name}-tags`, [ + { id: 1, parentGroup: 2, label: `one` }, + { id: 2, parentGroup: 1, label: `two` }, + { id: 3, parentGroup: 2, label: `three` }, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => { + const joined = + keySide === `main` + ? q + .from({ child: children.collection }) + .innerJoin({ tag: tags.collection }, ({ child, tag }) => + eq(tag.id, add(child.tagId, parent.offset)), + ) + : q + .from({ child: children.collection }) + .innerJoin({ tag: tags.collection }, ({ child, tag }) => + eq(add(tag.id, parent.offset), child.tagId), + ) + const routed = ( + correlationAttachment === `main` + ? joined.where(({ child }) => eq(child.parentGroup, parent.group)) + : joined.where(({ tag }) => eq(tag.parentGroup, parent.group)) + ) + .orderBy(({ child }) => child.id) + .select(({ child, tag }) => ({ id: child.id, value: tag.label })) + return { id: parent.id, ...includeInEveryForm(routed) } + }), + ) + + const expected = (parentId: number) => { + const parent = parents.collection.get(parentId)! + return children.collection.toArray + .flatMap((child) => + tags.collection.toArray + .filter((tag) => { + const keyMatches = + keySide === `main` + ? tag.id === child.tagId + parent.offset + : tag.id + parent.offset === child.tagId + const routeMatches = + correlationAttachment === `main` + ? child.parentGroup === parent.group + : tag.parentGroup === parent.group + return keyMatches && routeMatches + }) + .map((tag) => ({ id: child.id, value: tag.label })), + ) + .sort((leftRow, rightRow) => leftRow.id - rightRow.id) + } + + const project = (rows: Iterable<{ id: number; value: string }>) => + [...rows].map(({ id, value }) => ({ id, value })) + const assertParents = () => { + for (const parent of parents.collection.toArray) { + expectEveryForm(live.get(parent.id)!, project, expected(parent.id)) + } + } + + try { + await live.preload() + assertParents() + + parents.write(`update`, { id: 1, group: 1, offset: 1 }) + assertParents() + + children.write(`insert`, { id: 30, parentGroup: 1, tagId: 2 }) + assertParents() + } finally { + await cleanup(live, [parents, children, tags]) + } +} + +async function runNamespaceCollisionCell({ + location, + boundary, + name, +}: NamespaceCollisionCell): Promise { + const cellName = `collision-${location}-${boundary}-${name}` + const parents = createGrammarCollection(`${cellName}-parents`, [ + { id: 1, group: 1, token: `one` }, + ]) + const children = createGrammarCollection(`${cellName}-children`, [ + { id: 10, parentGroup: 1, token: `one`, label: `one` }, + { id: 20, parentGroup: 2, token: `two`, label: `two` }, + ]) + const tags = createGrammarCollection(`${cellName}-tags`, [ + { id: 10 }, + { id: 20 }, + ]) + const live = + location === `parent-alias` + ? createLiveQueryCollection((q) => + q.from({ [name]: parents.collection }).select((sources) => { + // This computed alias names the sole, non-optional main source. + const parent = sources[name]! + const correlated = q + .from({ child: children.collection }) + .where(({ child }) => + and( + eq(child.parentGroup, parent.group), + eq(child.token, parent.token), + ), + ) + const forms = (() => { + switch (boundary) { + case `direct`: + return includeInEveryForm( + correlated.select(({ child }) => ({ + id: child.id, + value: child.label, + })), + ) + case `query-ref`: { + const projected = correlated.select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + label: child.label, + })) + return includeInEveryForm( + q + .from({ result: projected }) + .where(({ result }) => + eq(result.parentGroup, parent.group), + ) + .select(({ result }) => ({ + id: result.id, + value: result.label, + })), + ) + } + case `join`: + return includeInEveryForm( + correlated + .innerJoin({ tag: tags.collection }, ({ child, tag }) => + eq(child.id, tag.id), + ) + .select(({ child }) => ({ + id: child.id, + value: child.label, + })), + ) + case `group`: + return includeInEveryForm( + correlated + .groupBy(({ child }) => [child.id, child.label]) + .select(({ child }) => ({ + id: child.id, + value: child.label, + })), + ) + } + })() + return { id: parent.id, ...forms } + }), + ) + : createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + const correlated = q + .from({ child: children.collection }) + .where(({ child }) => + and( + eq(child.parentGroup, parent.group), + eq(child.token, parent.token), + ), + ) + const forms = (() => { + switch (boundary) { + case `direct`: + return includeInEveryForm( + correlated.select(({ child }) => ({ + id: child.id, + [name]: child.label, + })), + ) + case `query-ref`: { + const projected = correlated.select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + label: child.label, + })) + return includeInEveryForm( + q + .from({ result: projected }) + .where(({ result }) => + eq(result.parentGroup, parent.group), + ) + .select(({ result }) => ({ + id: result.id, + [name]: result.label, + })), + ) + } + case `join`: + return includeInEveryForm( + correlated + .innerJoin({ tag: tags.collection }, ({ child, tag }) => + eq(child.id, tag.id), + ) + .select(({ child }) => ({ + id: child.id, + [name]: child.label, + })), + ) + case `group`: + return includeInEveryForm( + correlated + .groupBy(({ child }) => [child.id, child.label]) + .select(({ child }) => ({ + id: child.id, + [name]: child.label, + })), + ) + } + })() + return { id: parent.id, ...forms } + }), + ) + + const expected = () => { + const group = parents.collection.get(1)!.group + return children.collection.toArray + .filter( + (child) => + child.parentGroup === group && + child.token === parents.collection.get(1)!.token, + ) + .map((child) => ({ id: child.id, value: child.label })) + } + const project = (rows: Iterable>) => + [...rows].map((row) => ({ + id: row.id, + value: location === `parent-alias` ? row.value : row[name], + })) + const assertCurrent = () => + expectEveryForm( + live.get(1)! as MaterializedForms>, + project, + expected(), + ) + + try { + await live.preload() + assertCurrent() + + parents.write(`update`, { id: 1, group: 2, token: `two` }) + assertCurrent() + + children.write(`update`, { + id: 20, + parentGroup: 2, + token: `two`, + label: `updated`, + }) + assertCurrent() + } finally { + await cleanup(live, [parents, children, tags]) + } +} + +class PublicSurfaceBox { + readonly map: Map + readonly set: Set + + constructor(readonly row: unknown) { + this.map = new Map([[`row`, row]]) + this.set = new Set([row]) + } +} + +function expectNoPrivateSymbolsDeep( + value: unknown, + allowedSymbols: ReadonlySet, + seen = new WeakSet(), +): void { + if (value == null || typeof value !== `object` || seen.has(value)) return + seen.add(value) + + for (const key of Reflect.ownKeys(value)) { + if (typeof key === `symbol`) { + expect( + allowedSymbols.has(key), + `unexpected private symbol in public query output`, + ).toBe(true) + } + expectNoPrivateSymbolsDeep( + (value as Record)[key], + allowedSymbols, + seen, + ) + } + + if (value instanceof Map) { + for (const [key, entry] of value) { + expectNoPrivateSymbolsDeep(key, allowedSymbols, seen) + expectNoPrivateSymbolsDeep(entry, allowedSymbols, seen) + } + } else if (value instanceof Set) { + for (const entry of value) { + expectNoPrivateSymbolsDeep(entry, allowedSymbols, seen) + } + } +} + +async function runPublicSurfaceCell({ + shape, +}: PublicSurfaceCell): Promise { + const callbackRows: Array = [] + const parents = createGrammarCollection(`surface-${shape}-parents`, [ + { id: 1, group: 1 }, + ]) + const first = new Date(`2026-01-01T00:00:00.000Z`) + const second = new Date(`2026-01-02T00:00:00.000Z`) + const third = new Date(`2026-01-03T00:00:00.000Z`) + const firstPayload = { token: `first` } + const secondPayload = { token: `second` } + const userSymbol = Symbol(`user-owned`) + const createAdversarialPayload = (marker: string) => { + const value: { + safe: string + __proto__: { marker: string } + row?: unknown + } = { safe: marker, [`__proto__`]: { marker } } + return value + } + const firstAdversarial = createAdversarialPayload(`first`) + const secondAdversarial = createAdversarialPayload(`second`) + const children = createGrammarCollection(`surface-${shape}-children`, [ + { + id: 10, + parentGroup: 1, + value: first, + payload: firstPayload, + adversarial: firstAdversarial, + symbols: { [userSymbol]: `first` }, + label: `ten`, + }, + { + id: 20, + parentGroup: 2, + value: second, + payload: secondPayload, + adversarial: secondAdversarial, + symbols: { [userSymbol]: `second` }, + label: `twenty`, + }, + ]) + const candidates = createGrammarCollection(`surface-${shape}-candidates`, [ + { id: 10, value: first }, + { id: 20, value: second }, + ]) + const anchors = createGrammarCollection(`surface-${shape}-anchors`, [ + { id: 100, parentGroup: 1, value: first }, + { id: 200, parentGroup: 2, value: second }, + { id: 300, parentGroup: 2, value: third }, + ]) + const tags = createGrammarCollection(`surface-${shape}-tags`, [ + { id: 1000, childId: 10, label: `first` }, + { id: 2000, childId: 20, label: `second` }, + ]) + + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + if (shape === `object-query-ref-scalar`) { + const values = q + .from({ candidate: candidates.collection }) + .fn.select(({ candidate }) => candidate.value) + const rows = q + .from({ anchor: anchors.collection }) + .innerJoin({ value: values }, ({ anchor, value }) => + eq(anchor.value, value), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ anchor, value }) => ({ id: anchor.id, value })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + const correlated = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + if (shape === `nested-functional-spread`) { + const rows = correlated.fn.select((row) => ({ + id: row.child.id, + nested: { ...row }, + })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + if (shape === `opaque-wrapper`) { + const rows = correlated.fn + .where((row) => { + callbackRows.push(row) + return true + }) + .fn.select((row) => ({ + id: row.child.id, + box: new PublicSurfaceBox(row), + })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + if (shape === `functional-having-input`) { + const rows = correlated + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ + parentGroup: child.parentGroup, + total: count(child.id), + })) + .fn.having((row) => { + callbackRows.push(row) + return true + }) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + if (shape === `nested-reference`) { + const rows = correlated.select(({ child }) => ({ + id: child.id, + payload: child.payload, + })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + if (shape === `adversarial-key`) { + const rows = correlated.fn.select((row) => { + const payload = createAdversarialPayload(row.child.adversarial.safe) + payload.row = row + return { id: row.child.id, payload } + }) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + if (shape === `user-symbol`) { + const rows = correlated.fn.select((row) => ({ + id: row.child.id, + payload: { ...row.child.symbols, row }, + })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + const rows = correlated.innerJoin( + { tag: tags.collection }, + ({ child, tag }) => eq(child.id, tag.childId), + ) + return { id: parent.id, ...includeInEveryForm(rows) } + }), + ) + + const assertCurrent = () => { + const forms = live.get(1)! + const rowsByForm = [ + [...forms.collection.values()], + [...forms.array], + [...forms.materialized], + ] + for (const rows of rowsByForm) { + for (const row of rows) { + expectNoPrivateSymbolsDeep(row, new Set([userSymbol])) + } + } + // Retain earlier callback values: later graph work must not contaminate + // objects already handed to user code with private route metadata. + for (const row of callbackRows) { + expectNoPrivateSymbolsDeep(row, new Set([userSymbol])) + } + + if (shape === `object-query-ref-scalar`) { + const expected = + parents.collection.get(1)!.group === 1 + ? [{ id: 100, value: first }] + : candidates.collection.get(20)!.value === second + ? [{ id: 200, value: second }] + : [{ id: 300, value: third }] + for (const rows of rowsByForm) { + expect(rows).toHaveLength(1) + expect((rows[0] as any).id).toBe(expected[0]!.id) + expect((rows[0] as any).value).toBe(expected[0]!.value) + } + return + } + + if (shape === `nested-functional-spread`) { + const child = + parents.collection.get(1)!.group === 1 + ? children.collection.get(10)! + : children.collection.get(20)! + for (const rows of rowsByForm) { + expect( + rows.map((row: any) => ({ + id: row.id, + child: row.nested.child, + })), + ).toEqual([{ id: child.id, child }]) + } + return + } + + const child = + parents.collection.get(1)!.group === 1 + ? children.collection.get(10)! + : children.collection.get(20)! + + if (shape === `opaque-wrapper`) { + for (const rows of rowsByForm) { + expect(rows).toHaveLength(1) + const row = rows[0] as any + expect(row.box).toBeInstanceOf(PublicSurfaceBox) + expect(row.box.row.child.id).toBe(child.id) + expect(row.box.row.child.label).toBe(child.label) + expect(row.box.map.get(`row`)).toBe(row.box.row) + expect(row.box.set.has(row.box.row)).toBe(true) + } + return + } + + if (shape === `nested-reference`) { + for (const rows of rowsByForm) { + expect((rows[0] as any).payload).toBe(child.payload) + } + return + } + + if (shape === `functional-having-input`) { + const total = children.collection.toArray.filter( + ({ parentGroup }) => parentGroup === parents.collection.get(1)!.group, + ).length + for (const rows of rowsByForm) { + expect( + rows.map((row: any) => ({ + parentGroup: row.parentGroup, + total: row.total, + })), + ).toEqual([{ parentGroup: child.parentGroup, total }]) + } + return + } + + if (shape === `adversarial-key`) { + for (const rows of rowsByForm) { + const payload = (rows[0] as any).payload + expect(Object.prototype.hasOwnProperty.call(payload, `__proto__`)).toBe( + true, + ) + expect(Object.getPrototypeOf(payload)).toBe(Object.prototype) + expect(payload.__proto__).toEqual({ + marker: child.adversarial.__proto__.marker, + }) + expect(payload.row.child.id).toBe(child.id) + } + return + } + + if (shape === `user-symbol`) { + for (const [index, rows] of rowsByForm.entries()) { + expect( + (rows[0] as any).payload[userSymbol], + materializationForms[index], + ).toBe(child.symbols[userSymbol]) + expect((rows[0] as any).payload.row.child.id).toBe(child.id) + } + return + } + + const tag = tags.collection.toArray.find( + ({ childId }) => childId === child.id, + )! + for (const rows of rowsByForm) { + expect( + rows.map((row: any) => ({ child: row.child, tag: row.tag })), + ).toEqual([{ child, tag }]) + } + } + + try { + await live.preload() + assertCurrent() + + parents.write(`update`, { id: 1, group: 2 }) + assertCurrent() + + if (shape === `object-query-ref-scalar`) { + candidates.write(`update`, { id: 20, value: third }) + } else if (shape === `functional-having-input`) { + children.write(`insert`, { + id: 30, + parentGroup: 2, + value: third, + payload: { token: `third` }, + adversarial: createAdversarialPayload(`third`), + symbols: { [userSymbol]: `third` }, + label: `thirty`, + }) + } else if (shape === `nested-reference`) { + children.write(`update`, { + ...children.collection.get(20)!, + payload: { token: `updated` }, + }) + } else if (shape === `adversarial-key`) { + children.write(`update`, { + ...children.collection.get(20)!, + adversarial: createAdversarialPayload(`updated`), + }) + } else if (shape === `user-symbol`) { + children.write(`update`, { + ...children.collection.get(20)!, + symbols: { [userSymbol]: `updated` }, + }) + } else { + children.write(`update`, { + ...children.collection.get(20)!, + label: `updated`, + }) + } + assertCurrent() + } finally { + await cleanup(live, [parents, children, candidates, anchors, tags]) + } +} + +async function runGrammarCell(cell: GrammarCell): Promise { + switch (cell.family) { + case `parent-projection`: + return runParentProjectionCell(cell) + case `correlation-domain`: + return runCorrelationDomainCell(cell) + case `lexical-scope`: + return runLexicalScopeCell(cell) + case `aggregation`: + return runAggregationCell(cell) + case `recursive-source`: + return runRecursiveSourceCell(cell) + case `join`: + return runJoinCell(cell) + case `union-identity`: + return runUnionIdentityCell(cell) + case `derived-result`: + return runDerivedResultCell(cell) + case `namespace-collision`: + return runNamespaceCollisionCell(cell) + case `public-surface`: + return runPublicSurfaceCell(cell) + } +} + +describe(`correlated include route-context transport grammar`, () => { + test(`expands every declared product without duplicate cells`, () => { + const expectedCellCount = + routeContextGrammar.parentProjection.shapes.length + + routeContextGrammar.correlationDomain.states.length + + routeContextGrammar.lexicalScope.scopes.length + + routeContextGrammar.aggregation.groupings.length * + routeContextGrammar.aggregation.placements.length + + routeContextGrammar.recursiveSource.boundaries.length * + routeContextGrammar.recursiveSource.phases.length + + routeContextGrammar.join.keySides.length * + routeContextGrammar.join.correlationAttachments.length + + routeContextGrammar.unionIdentity.forms.length + + routeContextGrammar.derivedResult.boundaries.length * + routeContextGrammar.derivedResult.selections.length * + routeContextGrammar.derivedResult.domains.length + + routeContextGrammar.namespaceCollision.locations.length * + routeContextGrammar.namespaceCollision.boundaries.length * + routeContextGrammar.namespaceCollision.names.length + + routeContextGrammar.publicSurface.shapes.length + const names = grammarCells.map(grammarCellName) + + expect(grammarCells).toHaveLength(expectedCellCount) + expect(new Set(names)).toHaveLength(expectedCellCount) + expect( + grammarCells.length * materializationForms.length * checkpoints.length, + ).toBe(819) + }) + + test(`preserves cycles while removing nested route metadata`, () => { + const routed = attachRouteMetadata({ id: 1 }, 1, null) + const value: Record = { routed } + value.self = value + + const cleaned = stripInternalRouteMetadata(value) as typeof value + + expect(cleaned).not.toBe(value) + expect(cleaned.self).toBe(cleaned) + expectNoPrivateSymbolsDeep(cleaned, new Set()) + }) + + test(`does not evaluate unused accessors while cleaning routed callback rows`, async () => { + let reads = 0 + const payload = {} + Object.defineProperty(payload, `unused`, { + get() { + reads++ + throw new Error(`unused getter evaluated`) + }, + enumerable: true, + }) + const parents = createGrammarCollection(`getter-parents`, [ + { id: 1, group: 1 }, + ]) + const children = createGrammarCollection(`getter-children`, [ + { id: 10, parentGroup: 1, payload }, + ]) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .fn.where(() => true) + .fn.select(({ child }) => ({ id: child.id })), + ), + })), + ) + + try { + await live.preload() + expect(live.get(1)?.children).toEqual([{ id: 10 }]) + expect(reads).toBe(0) + } finally { + await cleanup(live, [parents, children]) + } + }) + + for (const cell of grammarCells) { + test(`${grammarCellName(cell)} × every materialization form × parent/child updates`, () => + runGrammarCell(cell)) + } + + for (const routeMode of queryRefMetadataGrammar.routeModes) { + test(`query-ref metadata / ${routeMode}`, () => + runQueryRefMetadataCell(routeMode)) + } +}) diff --git a/packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts b/packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts new file mode 100644 index 0000000000..8e50137897 --- /dev/null +++ b/packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts @@ -0,0 +1,1001 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, test } from 'vitest' +import { Temporal } from 'temporal-polyfill' +import { createCollection } from '../../src/collection/index.js' +import { createFilterFunctionFromExpression } from '../../src/collection/change-events.js' +import { + and, + count, + createLiveQueryCollection, + eq, + isNull, + lt, + not, + queryOnce, + toArray, +} from '../../src/query/index.js' +import { oraclePropertyOptions } from '../oracle-config.js' +import { flushPromises, stripVirtualProps } from '../utils.js' +import { createControlledCollection as createOracleControlledCollection } from './includes-oracle-helpers.js' +import type { Collection } from '../../src/collection/index.js' +import type { LoadSubsetOptions } from '../../src/types.js' +import type { ControlledCollection } from './includes-oracle-helpers.js' + +type ParentRow = { + id: number + group: number + position: number +} + +type ChildRow = { + id: number + parentGroup: number + score: number | null + position: number +} + +type CrossFormulationAction = + | { type: `putParent`; row: ParentRow } + | { type: `deleteParent`; id: number } + | { type: `putChild`; row: ChildRow } + | { type: `deleteChild`; id: number } + +type CrossFormulationScenario = { + parents: Array + children: Array + pivot: number + actions: Array +} + +type NormalizedParent = ParentRow & { + children: Array +} + +type FlatRow = { + parentId: number + parentGroup: number + parentPosition: number + child: ChildRow | undefined +} + +type ReferenceKey = { code: number } + +type ReferenceParent = { + id: number + group: ReferenceKey +} + +type ReferenceChild = { + id: number + parentGroup: ReferenceKey +} + +type ReferenceContextParent = { + id: number + group: number + expected: ReferenceKey +} + +type ReferenceContextChild = { + id: number + group: number + token: ReferenceKey +} + +function createControlledCollection( + name: string, + initialData: ReadonlyArray, +): ControlledCollection { + return createOracleControlledCollection(name, initialData, { + autoIndex: `eager`, + rowUpdateMode: `full`, + }) +} + +function compareParents(left: ParentRow, right: ParentRow): number { + return left.position - right.position || left.id - right.id +} + +function compareChildren(left: ChildRow, right: ChildRow): number { + return left.position - right.position || left.id - right.id +} + +function normalizeChild(child: ChildRow): ChildRow { + return { + id: child.id, + parentGroup: child.parentGroup, + score: child.score, + position: child.position, + } +} + +function normalizeNested( + rows: ReadonlyArray, +): Array { + return rows + .map((parent) => ({ + id: parent.id, + group: parent.group, + position: parent.position, + children: parent.children.map(normalizeChild).sort(compareChildren), + })) + .sort(compareParents) +} + +function normalizeFlat(rows: ReadonlyArray): Array { + const parents = new Map() + for (const row of rows) { + const parent = parents.get(row.parentId) ?? { + id: row.parentId, + group: row.parentGroup, + position: row.parentPosition, + children: [], + } + if (row.child) parent.children.push(normalizeChild(row.child)) + parents.set(row.parentId, parent) + } + return normalizeNested([...parents.values()]) +} + +function recompute( + parents: Map, + children: Map, +): Array { + return normalizeNested( + [...parents.values()].map((parent) => ({ + ...parent, + children: [...children.values()].filter( + (child) => child.parentGroup === parent.group, + ), + })), + ) +} + +function createNestedQuery( + parents: Collection, + children: Collection, +) { + return createLiveQueryCollection({ + getKey: (row) => row.id, + query: (q) => + q + .from({ parent: parents }) + .orderBy(({ parent }) => parent.position) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => ({ + id: parent.id, + group: parent.group, + position: parent.position, + children: toArray( + q + .from({ child: children }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + score: child.score, + position: child.position, + })), + ), + })), + }) +} + +function createWindowedNestedQuery( + parents: Collection, + children: Collection, + offset: number, + limit: number, +) { + return createLiveQueryCollection({ + getKey: (row) => row.id, + query: (q) => + q + .from({ parent: parents }) + .orderBy(({ parent }) => parent.position) + .orderBy(({ parent }) => parent.id) + .select(({ parent }) => ({ + id: parent.id, + group: parent.group, + position: parent.position, + children: toArray( + q + .from({ child: children }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .offset(offset) + .limit(limit) + .select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + score: child.score, + position: child.position, + })), + ), + })), + }) +} + +function createFlatQuery( + parents: Collection, + children: Collection, +) { + return createLiveQueryCollection({ + getKey: (row) => `${row.parentId}:${row.child?.id ?? `empty`}`, + query: (q) => + q + .from({ parent: parents }) + .leftJoin({ child: children }, ({ parent, child }) => + eq(parent.group, child.parentGroup), + ) + .select(({ parent, child }) => ({ + parentId: parent.id, + parentGroup: parent.group, + parentPosition: parent.position, + child, + })), + }) +} + +type ChildPartition = `all` | `predicate` | `complement` | `unknown` + +async function queryChildren( + children: Collection, + parentGroup: number, + pivot: number, + partition: ChildPartition, +): Promise> { + return queryOnce((q) => { + const correlated = q + .from({ child: children }) + .where(({ child }) => eq(child.parentGroup, parentGroup)) + const partitioned = (() => { + switch (partition) { + case `all`: + return correlated + case `predicate`: + return correlated.where(({ child }) => lt(child.score, pivot)) + case `complement`: + return correlated.where(({ child }) => not(lt(child.score, pivot))) + case `unknown`: + return correlated.where(({ child }) => isNull(lt(child.score, pivot))) + } + })() + + return partitioned + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + score: child.score, + position: child.position, + })) + }) +} + +async function queryPerParent( + parents: ReadonlyArray, + children: Collection, + pivot: number, + useTlp: boolean, +): Promise> { + return normalizeNested( + await Promise.all( + parents.map(async (parent) => { + const childRows = useTlp + ? ( + await Promise.all( + ([`predicate`, `complement`, `unknown`] as const).map( + (partition) => + queryChildren(children, parent.group, pivot, partition), + ), + ) + ).flat() + : await queryChildren(children, parent.group, pivot, `all`) + + return { ...parent, children: childRows } + }), + ), + ) +} + +function applyAction( + action: CrossFormulationAction, + parentSource: ControlledCollection, + childSource: ControlledCollection, + parents: Map, + children: Map, +): void { + switch (action.type) { + case `putParent`: { + const type = parents.has(action.row.id) ? `update` : `insert` + parents.set(action.row.id, { ...action.row }) + parentSource.write(type, action.row) + return + } + case `deleteParent`: { + const previous = parents.get(action.id) + if (!previous) return + parents.delete(action.id) + parentSource.write(`delete`, previous) + return + } + case `putChild`: { + const type = children.has(action.row.id) ? `update` : `insert` + children.set(action.row.id, { ...action.row }) + childSource.write(type, action.row) + return + } + case `deleteChild`: { + const previous = children.get(action.id) + if (!previous) return + children.delete(action.id) + childSource.write(`delete`, previous) + } + } +} + +async function expectFormulationsEquivalent( + scenario: CrossFormulationScenario, +): Promise { + const parentSource = createControlledCollection( + `cross-form-parents`, + scenario.parents, + ) + const childSource = createControlledCollection( + `cross-form-children`, + scenario.children, + ) + const parents = new Map(scenario.parents.map((row) => [row.id, { ...row }])) + const children = new Map(scenario.children.map((row) => [row.id, { ...row }])) + const nested = createNestedQuery( + parentSource.collection, + childSource.collection, + ) + const flat = createFlatQuery(parentSource.collection, childSource.collection) + + const assertEquivalent = async () => { + const expected = recompute(parents, children) + const nestedResult = normalizeNested(nested.toArray) + const flatResult = normalizeFlat(flat.toArray) + const parentRows = [...parents.values()] + const standaloneResult = await queryPerParent( + parentRows, + childSource.collection, + scenario.pivot, + false, + ) + const tlpResult = await queryPerParent( + parentRows, + childSource.collection, + scenario.pivot, + true, + ) + + expect({ + nested: nestedResult, + flat: flatResult, + standalone: standaloneResult, + tlp: tlpResult, + }).toEqual({ + nested: expected, + flat: expected, + standalone: expected, + tlp: expected, + }) + } + + try { + await Promise.all([nested.preload(), flat.preload()]) + await assertEquivalent() + for (const action of scenario.actions) { + applyAction(action, parentSource, childSource, parents, children) + await flushPromises() + await assertEquivalent() + } + } finally { + await Promise.allSettled([ + nested.cleanup(), + flat.cleanup(), + parentSource.collection.cleanup(), + childSource.collection.cleanup(), + ]) + } +} + +async function expectWindowedIncludeMatches( + scenario: CrossFormulationScenario, + offset: number, + limit: number, +): Promise { + const parentSource = createControlledCollection( + `windowed-cross-form-parents`, + scenario.parents, + ) + const childSource = createControlledCollection( + `windowed-cross-form-children`, + scenario.children, + ) + const parents = new Map(scenario.parents.map((row) => [row.id, { ...row }])) + const children = new Map(scenario.children.map((row) => [row.id, { ...row }])) + const nested = createWindowedNestedQuery( + parentSource.collection, + childSource.collection, + offset, + limit, + ) + + const assertEquivalent = () => { + const expected = normalizeNested( + [...parents.values()].map((parent) => ({ + ...parent, + children: [...children.values()] + .filter((child) => child.parentGroup === parent.group) + .sort(compareChildren) + .slice(offset, offset + limit), + })), + ) + expect(normalizeNested(nested.toArray)).toEqual(expected) + } + + try { + await nested.preload() + assertEquivalent() + for (const action of scenario.actions) { + applyAction(action, parentSource, childSource, parents, children) + await flushPromises() + assertEquivalent() + } + } finally { + await Promise.allSettled([ + nested.cleanup(), + parentSource.collection.cleanup(), + childSource.collection.cleanup(), + ]) + } +} + +const parentRowArbitrary = (id: number) => + fc.record({ + id: fc.constant(id), + group: fc.integer({ min: -1, max: 1 }), + position: fc.integer({ min: -2, max: 2 }), + }) + +const childRowArbitrary = (id: number) => + fc.record({ + id: fc.constant(id), + parentGroup: fc.integer({ min: -1, max: 1 }), + score: fc.option(fc.integer({ min: -2, max: 2 }), { nil: null }), + position: fc.integer({ min: -2, max: 2 }), + }) + +const actionArbitrary: fc.Arbitrary = fc.oneof( + fc.record({ + type: fc.constant(`putParent` as const), + row: fc.integer({ min: 0, max: 2 }).chain(parentRowArbitrary), + }), + fc.record({ + type: fc.constant(`deleteParent` as const), + id: fc.integer({ min: 0, max: 2 }), + }), + fc.record({ + type: fc.constant(`putChild` as const), + row: fc.integer({ min: 10, max: 14 }).chain(childRowArbitrary), + }), + fc.record({ + type: fc.constant(`deleteChild` as const), + id: fc.integer({ min: 10, max: 14 }), + }), +) + +const scenarioArbitrary: fc.Arbitrary = fc.record({ + parents: fc.tuple(parentRowArbitrary(0), parentRowArbitrary(1)), + children: fc.tuple( + childRowArbitrary(10), + childRowArbitrary(11), + childRowArbitrary(12), + ), + pivot: fc.integer({ min: -2, max: 2 }), + actions: fc.array(actionArbitrary, { minLength: 1, maxLength: 5 }), +}) + +const windowedScenarioArbitrary = fc.record({ + scenario: scenarioArbitrary, + offset: fc.integer({ min: 0, max: 2 }), + limit: fc.integer({ min: 0, max: 3 }), +}) + +describe(`includes cross-formulation oracle`, () => { + fcTest.prop( + [fc.integer()], + oraclePropertyOptions(4, `includes-cross-formulation.reference-context`), + )( + `parent-context routing preserves reference-sensitive predicate values across transitions`, + async (code) => { + const firstToken = { code } + const secondToken = { code } + const parentRows: Array = [ + { id: 1, group: 1, expected: firstToken }, + { id: 2, group: 1, expected: secondToken }, + ] + const childRows: Array = [ + { id: 10, group: 1, token: firstToken }, + { id: 20, group: 1, token: secondToken }, + ] + const parents = createControlledCollection( + `reference-context-parents`, + parentRows, + ) + const fullyLoadedChildren = createControlledCollection( + `reference-context-full-children`, + childRows, + ) + const loadedChildIds = new Set() + const lazyChildren = createCollection({ + id: `reference-context-lazy-children`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: (options: LoadSubsetOptions) => { + const matches = options.where + ? createFilterFunctionFromExpression( + options.where, + ) + : () => true + begin() + for (const row of childRows) { + if (!loadedChildIds.has(row.id) && matches(row)) { + loadedChildIds.add(row.id) + write({ type: `insert`, value: row }) + } + } + commit() + markReady() + return Promise.resolve() + }, + }), + }, + }) + const createReferenceContextQuery = ( + children: Collection, + ) => + createLiveQueryCollection({ + getKey: (row) => row.id, + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children }) + .where(({ child }) => + and( + eq(child.group, parent.group), + eq(child.token, parent.expected), + ), + ) + .select(({ child }) => child.id), + ), + })), + }) + const fullyLoaded = createReferenceContextQuery( + fullyLoadedChildren.collection, + ) + const lazy = createReferenceContextQuery(lazyChildren) + + try { + await Promise.all([fullyLoaded.preload(), lazy.preload()]) + expect(lazy.toArray.map(stripVirtualProps)).toEqual([ + { id: 1, children: [10] }, + { id: 2, children: [20] }, + ]) + expect(lazy.toArray.map(stripVirtualProps)).toEqual( + fullyLoaded.toArray.map(stripVirtualProps), + ) + + parents.write(`delete`, parentRows[0]!) + await flushPromises() + expect(lazy.toArray.map(stripVirtualProps)).toEqual([ + { id: 2, children: [20] }, + ]) + expect(lazy.toArray.map(stripVirtualProps)).toEqual( + fullyLoaded.toArray.map(stripVirtualProps), + ) + + parents.write(`insert`, parentRows[0]!) + await flushPromises() + expect(lazy.toArray.map(stripVirtualProps)).toEqual([ + { id: 1, children: [10] }, + { id: 2, children: [20] }, + ]) + } finally { + await Promise.allSettled([ + fullyLoaded.cleanup(), + lazy.cleanup(), + parents.collection.cleanup(), + fullyLoadedChildren.collection.cleanup(), + lazyChildren.cleanup(), + ]) + } + }, + ) + + test.each([ + [`Date and number`, () => [new Date(0), 0] as const], + [ + `Buffer and Uint8Array`, + () => [Buffer.from([1, 2, 3]), new Uint8Array([1, 2, 3])] as const, + ], + [ + `equivalent Temporal values`, + () => + [ + Temporal.PlainDate.from(`2024-04-05`), + Temporal.PlainDate.from(`2024-04-05`), + ] as const, + ], + ])( + `grouped includes use query equality for %s routes`, + async (_name, createValues) => { + const [parentGroup, equivalentChildGroup] = createValues() + const parents = createControlledCollection(`equality-route-parents`, [ + { id: 1, group: parentGroup as unknown }, + ]) + const children = createControlledCollection(`equality-route-children`, [ + { id: 10, parentGroup: parentGroup as unknown }, + { id: 11, parentGroup: equivalentChildGroup as unknown }, + ]) + const nested = createLiveQueryCollection({ + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + summaries: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ count: count(child.id) })), + ), + })), + }) + const standalone = createLiveQueryCollection({ + query: (q) => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parentGroup)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ count: count(child.id) })), + }) + + try { + await Promise.all([nested.preload(), standalone.preload()]) + const nestedCounts = nested + .get(1) + ?.summaries.map(({ count: childCount }) => ({ count: childCount })) + const standaloneCounts = standalone.toArray.map( + ({ count: childCount }) => ({ count: childCount }), + ) + expect(nestedCounts).toEqual(standaloneCounts) + expect(nestedCounts).toEqual([{ count: 2 }]) + } finally { + await Promise.allSettled([ + nested.cleanup(), + standalone.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + test.each([`__correlationKey`, `__tanstack_group_correlation_key`])( + `grouped includes preserve internal-looking aggregate alias %s`, + async (alias) => { + const parents = createControlledCollection(`aggregate-alias-parents`, [ + { id: 1, group: 1 }, + ]) + const children = createControlledCollection(`aggregate-alias-children`, [ + { id: 10, parentGroup: 1 }, + { id: 11, parentGroup: 1 }, + ]) + const nested = createLiveQueryCollection({ + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + summaries: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ [alias]: count(child.id) })), + ), + })), + }) + + try { + await nested.preload() + expect(nested.get(1)?.summaries.map((row) => row[alias])).toEqual([2]) + } finally { + await Promise.allSettled([ + nested.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest.prop( + [fc.integer()], + oraclePropertyOptions(4, `includes-cross-formulation.reference-key`), + )( + `lazy materialization matches fully loaded materialization for reference-sensitive correlation keys`, + async (code) => { + const firstKey = { code } + const secondKey = { code } + const parentRows: Array = [ + { id: 1, group: firstKey }, + { id: 2, group: secondKey }, + ] + const childRows: Array = [ + { id: 10, parentGroup: firstKey }, + { id: 20, parentGroup: secondKey }, + ] + const parents = createControlledCollection( + `reference-key-parents`, + parentRows, + ) + const fullyLoadedChildren = createControlledCollection( + `reference-key-full-children`, + childRows, + ) + const loadedChildIds = new Set() + const lazyChildren = createCollection({ + id: `reference-key-lazy-children`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: (options: LoadSubsetOptions) => { + const matches = options.where + ? createFilterFunctionFromExpression( + options.where, + ) + : () => true + begin() + for (const row of childRows) { + if (!loadedChildIds.has(row.id) && matches(row)) { + loadedChildIds.add(row.id) + write({ type: `insert`, value: row }) + } + } + commit() + markReady() + return Promise.resolve() + }, + }), + }, + }) + + const createReferenceQuery = (children: Collection) => + createLiveQueryCollection({ + getKey: (row) => row.id, + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .select(({ child }) => child.id), + ), + })), + }) + + const createGroupedReferenceQuery = ( + children: Collection, + ) => + createLiveQueryCollection({ + getKey: (row) => row.id, + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + summaries: toArray( + q + .from({ child: children }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ count: count(child.id) })), + ), + })), + }) + + const fullyLoaded = createReferenceQuery(fullyLoadedChildren.collection) + const lazy = createReferenceQuery(lazyChildren) + const fullyLoadedGrouped = createGroupedReferenceQuery( + fullyLoadedChildren.collection, + ) + const lazyGrouped = createGroupedReferenceQuery(lazyChildren) + const groupedRows = ( + query: typeof fullyLoadedGrouped, + ): Array<{ id: number; summaries: Array<{ count: number }> }> => + query.toArray.map((row) => ({ + id: row.id, + summaries: row.summaries.map(({ count: childCount }) => ({ + count: childCount, + })), + })) + + try { + await Promise.all([ + fullyLoaded.preload(), + lazy.preload(), + fullyLoadedGrouped.preload(), + lazyGrouped.preload(), + ]) + const fullyLoadedRows = fullyLoaded.toArray.map(stripVirtualProps) + const lazyRows = lazy.toArray.map(stripVirtualProps) + expect(lazyRows).toEqual(fullyLoadedRows) + expect(lazyRows).toEqual([ + { id: 1, children: [10] }, + { id: 2, children: [20] }, + ]) + expect(groupedRows(lazyGrouped)).toEqual( + groupedRows(fullyLoadedGrouped), + ) + expect(groupedRows(lazyGrouped)).toEqual([ + { id: 1, summaries: [{ count: 1 }] }, + { id: 2, summaries: [{ count: 1 }] }, + ]) + + parents.write(`delete`, parentRows[0]!) + await flushPromises() + expect(lazy.toArray.map(stripVirtualProps)).toEqual([ + { id: 2, children: [20] }, + ]) + expect(lazy.toArray.map(stripVirtualProps)).toEqual( + fullyLoaded.toArray.map(stripVirtualProps), + ) + expect(groupedRows(lazyGrouped)).toEqual([ + { id: 2, summaries: [{ count: 1 }] }, + ]) + expect(groupedRows(lazyGrouped)).toEqual( + groupedRows(fullyLoadedGrouped), + ) + + parents.write(`insert`, parentRows[0]!) + await flushPromises() + expect(lazy.toArray.map(stripVirtualProps)).toEqual([ + { id: 1, children: [10] }, + { id: 2, children: [20] }, + ]) + expect(groupedRows(lazyGrouped)).toEqual([ + { id: 1, summaries: [{ count: 1 }] }, + { id: 2, summaries: [{ count: 1 }] }, + ]) + } finally { + await Promise.allSettled([ + fullyLoaded.cleanup(), + lazy.cleanup(), + fullyLoadedGrouped.cleanup(), + lazyGrouped.cleanup(), + parents.collection.cleanup(), + fullyLoadedChildren.collection.cleanup(), + lazyChildren.cleanup(), + ]) + } + }, + ) + + fcTest.prop( + [fc.integer()], + oraclePropertyOptions(4, `includes-cross-formulation.symbol-group-route`), + )( + `grouped includes agree with standalone groups for symbol routes`, + async (code) => { + const firstGroup = Symbol(`first-${code}`) + const secondGroup = Symbol(`second-${code}`) + const parents = createControlledCollection(`symbol-route-parents`, [ + { id: 1, group: firstGroup }, + { id: 2, group: secondGroup }, + ]) + const children = createControlledCollection(`symbol-route-children`, [ + { id: 10, parentGroup: firstGroup }, + { id: 11, parentGroup: firstGroup }, + { id: 20, parentGroup: secondGroup }, + ]) + + const nested = createLiveQueryCollection({ + getKey: (row) => row.id, + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + summaries: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ count: count(child.id) })), + ), + })), + }) + const standalone = [firstGroup, secondGroup].map((group) => + createLiveQueryCollection({ + query: (q) => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, group)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ count: count(child.id) })), + }), + ) + + try { + await Promise.all([ + nested.preload(), + ...standalone.map((query) => query.preload()), + ]) + expect( + nested.toArray.map((row) => ({ + id: row.id, + summaries: row.summaries.map(({ count: childCount }) => ({ + count: childCount, + })), + })), + ).toEqual( + standalone.map((query, index) => ({ + id: index + 1, + summaries: query.toArray.map(({ count: childCount }) => ({ + count: childCount, + })), + })), + ) + } finally { + await Promise.allSettled([ + nested.cleanup(), + ...standalone.map((query) => query.cleanup()), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest(`shared-route child deletion agrees across formulations`, () => + expectFormulationsEquivalent({ + parents: [ + { id: 0, group: 0, position: 0 }, + { id: 1, group: 0, position: 0 }, + ], + children: [ + { id: 10, parentGroup: 0, score: null, position: 0 }, + { id: 11, parentGroup: 0, score: null, position: 0 }, + { id: 12, parentGroup: 0, score: null, position: 0 }, + ], + pivot: 0, + actions: [{ type: `deleteChild`, id: 10 }], + }), + ) + + fcTest.prop( + [scenarioArbitrary], + oraclePropertyOptions(8, `includes-cross-formulation.equivalence`), + )( + `agrees across nested includes, flat joins, per-parent queries, and TLP partitions`, + expectFormulationsEquivalent, + ) + + fcTest.prop( + [windowedScenarioArbitrary], + oraclePropertyOptions(12, `includes-cross-formulation.ordered-window`), + )( + `matches recomputation for ordered offset and limit child windows`, + ({ scenario, offset, limit }) => + expectWindowedIncludeMatches(scenario, offset, limit), + ) +}) diff --git a/packages/db/tests/query/includes-functional-input-boundary.test.ts b/packages/db/tests/query/includes-functional-input-boundary.test.ts new file mode 100644 index 0000000000..0e47a21a86 --- /dev/null +++ b/packages/db/tests/query/includes-functional-input-boundary.test.ts @@ -0,0 +1,408 @@ +import { describe, expect, it } from 'vitest' +import { + createLiveQueryCollection, + eq, + materialize, + toArray, +} from '../../src/query/index.js' +import { createControlledCollection } from './includes-oracle-helpers.js' + +describe(`functional include input boundary`, () => { + it.each( + [`array`, `materialized`].flatMap((form) => + [`none`, `first`, `second`].map((failureAt) => ({ form, failureAt })), + ), + )( + `keeps chained $form projections coherent through $failureAt failure`, + async ({ form, failureAt }) => { + const parents = createControlledCollection(`chain-parent`, [ + { id: 1, group: 1 }, + ]) + const children = createControlledCollection(`chain-child`, [ + { id: 10, group: 1 }, + { id: 20, group: 2 }, + ]) + const peers = createControlledCollection(`chain-peer`, [ + { id: 100, group: 1 }, + { id: 200, group: 2 }, + ]) + const failure = new Error(`projection failed`) + let failing = false + const query = createLiveQueryCollection((q) => { + const source = q.from({ parent: parents.collection }) + const included = + form === `array` + ? source.select(({ parent }) => ({ + id: parent.id, + group: parent.group, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.group, parent.group)), + ), + })) + : source.select(({ parent }) => ({ + id: parent.id, + group: parent.group, + children: materialize( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.group, parent.group)), + ), + })) + const first = q.from({ row: included }).fn.select(({ row }) => { + if (failing && failureAt === `first`) throw failure + return { + id: row.id, + group: row.group, + ids: row.children.map((child) => child.id), + } + }) + const projected = q.from({ row: first }) + const combined = + form === `array` + ? projected.select(({ row }) => ({ + id: row.id, + ids: row.ids, + peers: toArray( + q + .from({ peer: peers.collection }) + .where(({ peer }) => eq(peer.group, row.group)), + ), + })) + : projected.select(({ row }) => ({ + id: row.id, + ids: row.ids, + peers: materialize( + q + .from({ peer: peers.collection }) + .where(({ peer }) => eq(peer.group, row.group)), + ), + })) + return q.from({ row: combined }).fn.select(({ row }) => { + if (failing && failureAt === `second`) throw failure + return { + id: row.id, + ids: [...row.ids, ...row.peers.map((peer) => peer.id)], + } + }) + }) + try { + await query.preload() + const old = query.get(1)! + expect(old.ids).toEqual([10, 100]) + failing = failureAt !== `none` + if (failing) { + expect(() => parents.write(`update`, { id: 1, group: 2 })).toThrow( + failure, + ) + expect(query.get(1)).toBe(old) + expect(old.ids).toEqual([10, 100]) + } else { + parents.write(`update`, { id: 1, group: 2 }) + expect(query.get(1)!.ids).toEqual([20, 200]) + expect(old.ids).toEqual([10, 100]) + } + await query.cleanup() + failing = false + await query.preload() + expect(query.get(1)!.ids).toEqual([20, 200]) + peers.write(`insert`, { id: 201, group: 2 }) + expect(query.get(1)!.ids).toEqual([20, 200, 201]) + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + await peers.collection.cleanup() + } + }, + ) + + it(`keeps singleton materialization reactive through absence`, async () => { + const parents = createControlledCollection(`singleton-parent`, [{ id: 1 }]) + const children = createControlledCollection(`singleton-child`, [ + { id: 10, parentId: 2, value: 3 }, + ]) + const query = createLiveQueryCollection((q) => { + const included = q + .from({ parent: parents.collection }) + .select(({ parent }) => ({ + id: parent.id, + child: materialize( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)) + .findOne(), + ), + })) + return q + .from({ row: included }) + .fn.select(({ row }) => ({ id: row.id, value: row.child?.value ?? 0 })) + }) + try { + await query.preload() + expect(query.get(1)!.value).toBe(0) + children.write(`update`, { id: 10, parentId: 1, value: 3 }) + expect(query.get(1)!.value).toBe(3) + children.write(`update`, { id: 10, parentId: 1, value: 7 }) + expect(query.get(1)!.value).toBe(7) + children.write(`delete`, { id: 10, parentId: 1, value: 7 }) + expect(query.get(1)!.value).toBe(0) + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }) + + it.each([ + `read`, + `pass-through`, + `ignore`, + `subscribe`, + `create-index`, + ] as const)( + `rejects a Collection input before the callback can %s it`, + async (use) => { + const parents = createControlledCollection(`boundary-parent`, [{ id: 1 }]) + const children = createControlledCollection(`boundary-child`, [ + { id: 10, parentId: 1 }, + ]) + let calls = 0 + let query: ReturnType | undefined + try { + await expect( + (async () => { + query = createLiveQueryCollection((q) => + q + .from({ + row: q + .from({ parent: parents.collection }) + .select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + })), + }) + .fn.select(({ row }) => { + calls++ + if (use === `subscribe`) + row.children.subscribeChanges(() => {}) + if (use === `create-index`) + row.children.createIndex((child) => child.id) + return { + id: row.id, + value: + use === `read` + ? row.children.size + : use === `pass-through` + ? row.children + : null, + } + }), + ) + await query.preload() + })(), + ).rejects.toThrow( + `fn.select() cannot consume Collection-valued includes`, + ) + expect(calls).toBe(0) + } finally { + await query?.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it.each([`array`, `materialized`] as const)( + `keeps %s calculations reactive after child-only changes`, + async (form) => { + const parents = createControlledCollection(`inline-parent`, [{ id: 1 }]) + const children = createControlledCollection(`inline-child`, [ + { id: 10, parentId: 1, value: 3 }, + ]) + let calls = 0 + const query = createLiveQueryCollection((q) => { + const source = q.from({ parent: parents.collection }) + const included = + form === `array` + ? source.select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + ), + })) + : source.select(({ parent }) => ({ + id: parent.id, + children: materialize( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + ), + })) + return q.from({ row: included }).fn.select(({ row }) => { + calls++ + return { + id: row.id, + count: row.children.length, + sum: row.children.reduce((sum, child) => sum + child.value, 0), + found: row.children.find((child) => child.id === 10)?.value, + } + }) + }) + try { + await query.preload() + expect(query.get(1)).toMatchObject({ count: 1, sum: 3, found: 3 }) + const initialCalls = calls + children.write(`update`, { id: 10, parentId: 1, value: 7 }) + expect(query.get(1)).toMatchObject({ count: 1, sum: 7, found: 7 }) + expect(calls).toBeGreaterThan(initialCalls) + children.write(`insert`, { id: 11, parentId: 1, value: 5 }) + expect(query.get(1)).toMatchObject({ count: 2, sum: 12, found: 7 }) + children.write(`delete`, { id: 10, parentId: 1, value: 7 }) + expect(query.get(1)).toMatchObject({ + count: 1, + sum: 5, + found: undefined, + }) + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it(`keeps a bare Collection live through an expression projection`, async () => { + const parents = createControlledCollection(`expression-parent`, [{ id: 1 }]) + const children = createControlledCollection(`expression-child`, [ + { id: 10, parentId: 1 }, + ]) + const query = createLiveQueryCollection((q) => + q + .from({ + row: q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + })), + }) + .select(({ row }) => ({ id: row.id, children: row.children })), + ) + try { + await query.preload() + const held = query.get(1)!.children + expect(held.get(10)?.id).toBe(10) + children.write(`insert`, { id: 11, parentId: 1 }) + expect(query.get(1)!.children).toBe(held) + expect(held.get(11)?.id).toBe(11) + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }) + + it(`keeps parent-only functional work before adding live children`, async () => { + const parents = createControlledCollection(`parent-first`, [{ id: 1 }]) + const children = createControlledCollection(`parent-first-child`, [ + { id: 10, parentId: 1 }, + ]) + const query = createLiveQueryCollection((q) => { + const projected = q + .from({ parent: parents.collection }) + .fn.select(({ parent }) => ({ + id: parent.id, + label: `Parent ${parent.id}`, + })) + return q.from({ row: projected }).select(({ row }) => ({ + id: row.id, + label: row.label, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, row.id)), + })) + }) + let publications = 0 + const subscription = query.subscribeChanges(() => { + publications++ + }) + try { + await query.preload() + const held = query.get(1)!.children + expect(query.get(1)!.label).toBe(`Parent 1`) + expect(held.get(10)?.id).toBe(10) + publications = 0 + children.write(`insert`, { id: 11, parentId: 1 }) + expect(held.get(11)?.id).toBe(11) + expect(publications).toBe(0) + } finally { + subscription.unsubscribe() + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }) + + it.each([false, true])( + `checks nested Collection inputs (inline=%s)`, + async (inline) => { + const parents = createControlledCollection(`nested-parent`, [{ id: 1 }]) + const children = createControlledCollection(`nested-child`, [ + { id: 10, parentId: 1 }, + ]) + const leaves = createControlledCollection(`nested-leaf`, [ + { id: 100, childId: 10 }, + ]) + let cleanup = async () => {} + try { + const run = async () => { + const query = createLiveQueryCollection((q) => { + const included = q + .from({ parent: parents.collection }) + .select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)) + .select(({ child }) => { + const leafQuery = q + .from({ leaf: leaves.collection }) + .where(({ leaf }) => eq(leaf.childId, child.id)) + return { + id: child.id, + leaves: inline ? toArray(leafQuery) : leafQuery, + } + }), + ), + })) + return q + .from({ row: included }) + .fn.select(({ row }) => ({ id: row.id, children: row.children })) + }) + cleanup = () => query.cleanup() + await query.preload() + expect(query.get(1)).toMatchObject({ + children: [{ id: 10, leaves: [{ id: 100 }] }], + }) + } + if (inline) await run() + else + await expect(run()).rejects.toThrow( + `fn.select() cannot consume Collection-valued includes`, + ) + } finally { + await cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + await leaves.collection.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/query/includes-functional-projection-oracle.test.ts b/packages/db/tests/query/includes-functional-projection-oracle.test.ts new file mode 100644 index 0000000000..1c10fda82d --- /dev/null +++ b/packages/db/tests/query/includes-functional-projection-oracle.test.ts @@ -0,0 +1,1536 @@ +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { BucketFacadeAdapter } from '../../src/query/live/bucket-facade-adapter.js' +import { + createLiveQueryCollection, + eq, + materialize, + toArray, +} from '../../src/query/index.js' +import { flushPromises } from '../utils.js' +import { createControlledCollection } from './includes-oracle-helpers.js' + +const boundaries = [`query-ref`, `recursive-query-ref`, `union`] as const +const forms = [`collection`, `array`, `materialized`] as const +const outputs = [`expression`, `record`, `opaque-root`] as const +const initialStates = [`empty`, `populated`] as const +const cells = boundaries.flatMap((boundary) => + forms.flatMap((form) => + outputs.flatMap((output) => + initialStates.map((initial) => ({ boundary, form, output, initial })), + ), + ), +) +const consumers = [`expression`, `functional`] as const +const valueShapes = [`number`, `null`, `date`, `dropped-record`] as const +const valueCells = forms.flatMap((form) => + consumers.flatMap((consumer) => + valueShapes.flatMap((shape) => + [false, true].map((withInclude) => ({ + form, + consumer, + shape, + withInclude, + })), + ), + ), +) +const renamedCells = forms.flatMap((form) => + consumers.flatMap((consumer) => + consumers.flatMap((projection) => + [false, true].map((withSibling) => ({ + form, + consumer, + projection, + withSibling, + })), + ), + ), +) +const operatorCells = forms.flatMap((form) => + ([`custom-key`, `selected-order`, `distinct`] as const).flatMap((operator) => + [false, true].map((readsInclude) => ({ form, operator, readsInclude })), + ), +) + +type Child = { id: number; parentGroup: number; value: number } +type Input = { id: number; kind: string; children?: unknown } +type Phase = `initial` | `child-update` | `sibling-update` | `route-move` +type ChildView = { + valid: boolean + ready: boolean | undefined + rows: Array +} + +function readChildren(value: unknown, form: (typeof forms)[number]): ChildView { + if (form !== `collection`) { + return { + valid: Array.isArray(value), + ready: undefined, + rows: Array.isArray(value) ? value : [], + } + } + if ( + typeof value !== `object` || + value === null || + !(`toArray` in value) || + !(`isReady` in value) || + typeof value.isReady !== `function` + ) { + return { valid: false, ready: undefined, rows: [] } + } + return { + valid: Array.isArray(value.toArray), + ready: value.isReady(), + rows: Array.isArray(value.toArray) ? value.toArray : [], + } +} + +// Keep only the selected public fields in row comparisons. Callback-time shape +// and facade readiness have their own assertions rather than being normalized away. +function publicRows(rows: ReadonlyArray) { + return rows + .map(({ id, parentGroup, value }) => ({ id, parentGroup, value })) + .sort((left, right) => left.id - right.id) +} + +const collectionInputError = `fn.select() cannot consume Collection-valued includes` +function rejectsCollectionInput( + form: string, + ...functional: Array +): boolean { + return form === `collection` && functional.some(Boolean) +} + +class Projection { + constructor( + readonly id: number, + readonly kind: string, + readonly children: unknown, + readonly total: number, + ) {} +} + +describe(`functional projection output compatibility`, () => { + it.each( + ([`expression`, `functional`] as const).flatMap((projection) => + ([`resolve`, `reject`, `cleanup-resolve`, `cleanup-reject`] as const).map( + (settlement) => ({ projection, settlement }), + ), + ), + )( + `$projection projection fences $settlement of a pending child load`, + async ({ projection, settlement }) => { + const parents = createControlledCollection(`pending-parent`, [ + { id: 1, groupId: 1 }, + ]) + const requests: Array<{ + gate: ReturnType> + signal: AbortSignal | undefined + }> = [] + const children = createCollection<{ id: number; groupId: number }>({ + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: ({ signal }) => { + const gate = createDeferred() + requests.push({ gate, signal }) + return gate.promise.then(async () => { + if (signal?.aborted) return + begin() + write({ type: `insert`, value: { id: 10, groupId: 1 } }) + await commit() + markReady() + }) + }, + }), + }, + }) + const captured: Array> = [] + const buildQuery = () => + createLiveQueryCollection((q) => { + const source = q.from({ + row: q + .from({ parent: parents.collection }) + .select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)), + })), + }) + return projection === `expression` + ? source.select(({ row }) => row) + : source.fn.select(({ row }) => { + captured.push(row.children) + return { id: row.id, children: row.children } + }) + }) + if (rejectsCollectionInput(`collection`, projection === `functional`)) { + try { + expect(buildQuery).toThrow(collectionInputError) + expect(captured).toEqual([]) + } finally { + await parents.collection.cleanup() + await children.cleanup() + } + return + } + const query = buildQuery() + const failure = new Error(`pending child failed`) + // Attach both outcomes immediately; no pending-length assertion may + // leave a rejected preload promise unobserved. + const preload = () => { + const result: { settled: boolean; error?: unknown } = { settled: false } + const observed = query.preload().then( + () => { + result.settled = true + }, + (error) => { + result.settled = true + result.error = error + }, + ) + return { result, observed } + } + try { + const initial = preload() + await flushPromises() + expect(requests).toHaveLength(1) + expect(initial.result.settled).toBe(false) + expect(query.isReady()).toBe(false) + const held = query.get(1)!.children + expect(held.toArray).toEqual([]) + if (projection === `functional`) expect(captured).toHaveLength(1) + const obsoleteViews = [...captured, held] + + if (settlement.startsWith(`cleanup`)) { + await query.cleanup() + await children.cleanup() + await initial.observed + expect(initial.result.error).toMatchObject({ name: `AbortError` }) + expect(requests[0]!.signal?.aborted).toBe(true) + const restarted = preload() + await flushPromises() + expect(requests).toHaveLength(2) + const current = query.get(1)!.children + if (settlement === `cleanup-reject`) requests[0]!.gate.reject(failure) + else requests[0]!.gate.resolve() + await flushPromises() + expect( + restarted.result.settled, + `obsolete completion cannot finish preload`, + ).toBe(false) + expect(query.isReady()).toBe(false) + expect(current.toArray).toEqual([]) + requests[1]!.gate.resolve() + await restarted.observed + expect(restarted.result.error).toBeUndefined() + expect(query.isReady()).toBe(true) + expect(current.toArray.map((child) => child.id)).toEqual([10]) + for (const view of obsoleteViews) expect(view.toArray).toEqual([]) + } else { + if (settlement === `reject`) requests[0]!.gate.reject(failure) + else requests[0]!.gate.resolve() + await initial.observed + if (settlement === `reject`) { + expect(initial.result.error).toBe(failure) + expect(query.isReady()).toBe(false) + expect(held.toArray).toEqual([]) + } else { + expect(initial.result.error).toBeUndefined() + expect(query.isReady()).toBe(true) + expect(held.toArray.map((child) => child.id)).toEqual([10]) + for (const view of captured) + expect(view.toArray.map((child) => child.id)).toEqual([10]) + } + } + } finally { + await query.cleanup() + for (const { gate } of requests) gate.resolve() + await children.cleanup() + await parents.collection.cleanup() + } + }, + ) + + it.each( + ([`publication`, `after-preload`] as const).flatMap((subscribeAt) => + ([`none`, `callback`, `flush`] as const).map((failureAt) => ({ + subscribeAt, + failureAt, + })), + ), + )( + `keeps $subscribeAt subscriptions isolated through $failureAt failure and restart`, + async ({ subscribeAt, failureAt }) => { + const parents = createControlledCollection(`subscription-parent`, [ + { id: 1, groupId: 1 }, + ]) + const children = createControlledCollection(`subscription-child`, [ + { id: 10, groupId: 1 }, + { id: 20, groupId: 2 }, + ]) + const observers: Array<{ + rows: Set + batches: Array> + view: Pick + }> = [] + const releases: Array<() => void> = [] + const observe = (view: (typeof observers)[number][`view`]) => { + const rows = new Set() + const batches: Array> = [] + const subscription = view.subscribeChanges( + (changes) => { + batches.push( + changes.map((change) => `${change.type}:${change.value.id}`), + ) + for (const change of changes) { + if (change.type === `delete`) rows.delete(change.value.id) + else rows.add(change.value.id) + } + }, + { includeInitialState: true }, + ) + releases.push(() => subscription.unsubscribe()) + observers.push({ rows, batches, view }) + } + const failure = new Error(`projection subscription ${failureAt} failure`) + let failing = false + let flushReached = false + const originalFlush = BucketFacadeAdapter.prototype.flush + // Fail after actual facade writes, before any deferred public events. + // An event-listener throw is asynchronous and would not test rollback. + const flush = + failureAt === `flush` + ? vi + .spyOn(BucketFacadeAdapter.prototype, `flush`) + .mockImplementation(function (this: BucketFacadeAdapter) { + const publication = originalFlush.call(this) + return { + ...publication, + prepare: () => { + publication.prepare() + if (failing) { + flushReached = true + throw failure + } + }, + } + }) + : undefined + const query = createLiveQueryCollection((q) => { + const projected = q + .from({ parent: parents.collection }) + .fn.select(({ parent }) => { + if (failing && failureAt === `callback`) throw failure + return parent + }) + return q.from({ row: projected }).select(({ row }) => ({ + id: row.id, + groupId: row.groupId, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.groupId, row.groupId)), + })) + }) + if (subscribeAt === `publication`) { + const rootSubscription = query.subscribeChanges( + (changes) => { + for (const change of changes) + if (change.type !== `delete`) observe(change.value.children) + }, + { includeInitialState: true }, + ) + releases.push(() => rootSubscription.unsubscribe()) + } + const ids = (observer: (typeof observers)[number]) => + [...observer.rows].sort((a, b) => a - b) + try { + await query.preload() + if (subscribeAt === `after-preload`) observe(query.get(1)!.children) + const first = observers[0]! + expect(ids(first), `initial subscription snapshot`).toEqual([10]) + children.write(`insert`, { id: 11, groupId: 1 }) + expect(ids(first), `initial live insert`).toEqual([10, 11]) + const originalRow = query.get(1) + const beforeFailure = first.batches.length + failing = failureAt !== `none` + const move = () => parents.write(`update`, { id: 1, groupId: 2 }) + if (failing) { + let thrown: unknown + try { + move() + } catch (error) { + thrown = error + } + expect(thrown).toBe(failure) + expect(query.get(1), `root rollback`).toBe(originalRow) + expect(ids(first), `old subscriber rollback`).toEqual([10, 11]) + expect(first.batches.length, `no partial public events`).toBe( + beforeFailure, + ) + expect(observers).toHaveLength(1) + if (failureAt === `flush`) expect(flushReached).toBe(true) + } else { + move() + if (subscribeAt === `after-preload`) observe(query.get(1)!.children) + expect(ids(first), `retired route`).toEqual([]) + expect(ids(observers[1]!), `destination subscription`).toEqual([20]) + } + + // Keep the external subscriptions alive across cleanup. They belong to + // the old graph, not the next graph created by preload on this query. + await query.cleanup() + const oldObservers = [...observers] + const oldBatches = oldObservers.map( + (observer) => observer.batches.length, + ) + failing = false + await query.preload() + if (subscribeAt === `after-preload`) observe(query.get(1)!.children) + expect(observers).toHaveLength(oldObservers.length + 1) + expect(query.get(1)!.groupId, `restart uses current source`).toBe(2) + const restarted = observers.at(-1)! + expect(ids(restarted), `restart subscription snapshot`).toEqual([20]) + children.write(`insert`, { id: 22, groupId: 2 }) + expect(ids(restarted), `restart live insert`).toEqual([20, 22]) + expect( + oldObservers.map((observer) => observer.batches.length), + `old graph receives no fresh events`, + ).toEqual(oldBatches) + for (const observer of oldObservers) { + expect( + observer.view.toArray, + `old graph exposes no fresh rows`, + ).toEqual([]) + } + } finally { + failing = false + flush?.mockRestore() + for (const release of releases) release() + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + const readSurfaces = [ + `toArray`, + `get`, + `has`, + `size`, + `keys`, + `values`, + `entries`, + `iterator`, + `forEach`, + `map`, + `state`, + `virtual-key`, + `virtual-metadata`, + `index`, + ] as const + it.each( + readSurfaces.flatMap((surface) => + [false, true].map((ordered) => ({ surface, ordered })), + ), + )( + `keeps published $surface reads live (ordered=$ordered)`, + async ({ surface, ordered }) => { + const parents = createControlledCollection(`read-api-parent`, [ + { id: 1, groupId: 1 }, + ]) + const children = createControlledCollection(`read-api-child`, [ + { id: 10, groupId: 1 }, + { id: 11, groupId: 1 }, + { id: 20, groupId: 2 }, + { id: 21, groupId: 2 }, + ]) + const readers = new Map< + number, + (keys: Array) => Array + >() + const query = createLiveQueryCollection((q) => + q + .from({ + row: q.from({ parent: parents.collection }).select(({ parent }) => { + const childQuery = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + return { + id: parent.id, + groupId: parent.groupId, + children: ordered + ? childQuery.orderBy(({ child }) => child.id, `desc`) + : childQuery, + } + }), + }) + .select(({ row }) => row), + ) + const capture = (row: NonNullable>) => { + const view = row.children + const expectedKeys = [row.groupId * 10, row.groupId * 10 + 1] + const createIndex = view.createIndex.bind(view) + const read = (keys: Array) => { + let ids: Array + switch (surface) { + case `toArray`: + ids = view.toArray.map((child) => child.id) + break + case `get`: + ids = keys.flatMap((key) => view.get(key)?.id ?? []) + break + case `has`: + ids = keys.filter((key) => view.has(key)) + break + case `size`: + ids = [view.size] + break + case `keys`: + ids = [...view.keys()] + break + case `values`: + ids = [...view.values()].map((child) => child.id) + break + case `entries`: + ids = [...view.entries()].map(([key]) => key) + break + case `iterator`: + ids = [...view].map(([key]) => key) + break + case `forEach`: + ids = [] + view.forEach((child) => ids.push(child.id)) + break + case `map`: + ids = view.map((child) => child.id) + break + case `state`: + ids = [...view.state.keys()] + break + case `virtual-key`: + ids = view.toArray.map((child) => child.$key) + break + case `virtual-metadata`: + ids = view.toArray.map((child) => { + expect(child.$collectionId).toBe(children.collection.id) + expect(child.$synced).toBe(true) + expect(child.$origin).toBe(`remote`) + return child.id + }) + break + case `index`: { + const index = createIndex((child) => child.id, { + indexType: BasicIndex, + }) + ids = keys.flatMap((key) => [...index.lookup(`eq`, key)]) + break + } + } + return ids + } + readers.set(row.groupId, read) + const ids = read(expectedKeys) + return { id: row.id, ids, children: view } + } + const expected = (group: number) => { + if (surface === `size`) return [2] + const ids = [group * 10, group * 10 + 1] + return ordered && ![`get`, `has`, `index`].includes(surface) + ? ids.reverse() + : ids + } + const checkPublished = ( + group: number, + ids: Array, + phase: string, + ) => { + const actual = readers.get(group)!([ + group * 10, + group * 10 + 1, + group * 10 + 2, + ]) + const result = + surface === `size` + ? [ids.length] + : ordered && ![`get`, `has`, `index`].includes(surface) + ? [...ids].reverse() + : ids + expect.soft(actual, phase).toEqual(result) + } + try { + await query.preload() + const initialRead = capture(query.get(1)!) + checkPublished(1, [10, 11], `initial published read`) + expect + .soft(initialRead.ids, `initial published input`) + .toEqual(expected(1)) + parents.write(`update`, { id: 1, groupId: 2 }) + const movedRead = capture(query.get(1)!) + expect.soft(movedRead.ids, `moved published input`).toEqual(expected(2)) + checkPublished(1, [], `retired route read`) + checkPublished(2, [20, 21], `moved published read`) + const held = query.get(1)!.children + children.write(`insert`, { id: 22, groupId: 2 }) + expect(held.toArray.map((child) => child.id)).toEqual( + ordered ? [22, 21, 20] : [20, 21, 22], + ) + checkPublished(1, [], `retired route ignores later insert`) + checkPublished(2, [20, 21, 22], `published insertion read`) + children.write(`delete`, { id: 21, groupId: 2 }) + checkPublished(2, [20, 22], `published deletion read`) + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it.each([`expression`, `plain`, `opaque`, `closure`] as const)( + `keeps retained views live across a same-route parent update through a %s holder`, + async (holder) => { + const parents = createControlledCollection(`facade-identity-parent`, [ + { id: 1, groupId: 1, label: `first` }, + { id: 2, groupId: 1, label: `second` }, + ]) + const childSource = createControlledCollection(`facade-identity-child`, [ + { id: 10, groupId: 1 }, + ]) + class Holder { + constructor(readonly children: T) {} + } + const query = createLiveQueryCollection((q) => { + const source = q.from({ + row: q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + label: parent.label, + children: q + .from({ child: childSource.collection }) + .where(({ child }) => eq(child.groupId, parent.groupId)), + })), + }) + return source.select(({ row }) => ({ + id: row.id, + label: row.label, + box: { children: row.children }, + })) + }) + try { + await query.preload() + const childrenAtPublication = query.get(1)!.box.children + const retained = + holder === `opaque` + ? new Holder(childrenAtPublication) + : holder === `closure` + ? { + get children() { + return childrenAtPublication + }, + } + : { children: childrenAtPublication } + const held = retained.children + expect + .soft( + held.toArray.map((child) => child.id), + `initial rows`, + ) + .toEqual([10]) + expect + .soft(query.get(2)!.box.children, `initial shared identity`) + .toBe(held) + parents.write(`update`, { id: 1, groupId: 1, label: `changed` }) + expect + .soft(query.get(1)!.label, `parent update is visible`) + .toBe(`changed`) + expect(query.get(1)!.box.children).toBe(held) + expect + .soft( + query.get(2)!.box.children, + `unchanged parent keeps shared facade`, + ) + .toBe(held) + childSource.write(`insert`, { id: 11, groupId: 1 }) + for (const facade of [ + held, + query.get(1)!.box.children, + query.get(2)!.box.children, + ]) { + expect + .soft( + facade.toArray.map((child) => child.id).sort(), + `retained view stays live`, + ) + .toEqual([10, 11]) + } + } finally { + await query.cleanup() + await parents.collection.cleanup() + await childSource.collection.cleanup() + } + }, + ) + + it.each([`rows`, `index`, `callback-read`, `captured-method`] as const)( + `keeps held facade %s unchanged when a parent projection throws`, + async (surface) => { + const parents = createControlledCollection(`snapshot-parent`, [ + { id: 1, groupId: 1 }, + ]) + const children = createControlledCollection(`snapshot-child`, [ + { id: 10, groupId: 1 }, + { id: 20, groupId: 2 }, + ]) + const failure = new Error(`parent projection failed`) + let fail = false + let readPublished: (() => Array) | undefined + let capturedGet: ((key: number) => { id: number } | undefined) | undefined + let observed: Array | undefined + const query = createLiveQueryCollection((q) => { + const projected = q + .from({ parent: parents.collection }) + .fn.select(({ parent }) => { + if (fail) { + observed = readPublished?.() + throw failure + } + return parent + }) + return q.from({ row: projected }).select(({ row }) => ({ + id: row.id, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.groupId, row.groupId)), + })) + }) + try { + await query.preload() + const original = query.get(1)! + const held = original.children + capturedGet = held.get.bind(held) + readPublished = () => + surface === `captured-method` + ? [capturedGet?.(10)?.id].filter((id) => id !== undefined) + : held.toArray.map((child) => child.id) + const index = held.createIndex((child) => child.id, { + indexType: BasicIndex, + }) + expect(held.toArray.map((child) => child.id)).toEqual([10]) + expect(index.lookup(`eq`, 10)).toEqual(new Set([10])) + fail = true + expect(() => parents.write(`update`, { id: 1, groupId: 2 })).toThrow( + failure, + ) + expect(query.get(1)).toBe(original) + if (surface === `rows`) { + expect(held.toArray.map((child) => child.id)).toEqual([10]) + } else if (surface === `index`) { + expect(index.lookup(`eq`, 10)).toEqual(new Set([10])) + } else { + expect(observed).toEqual([10]) + } + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it.each([false, true])( + `preserves projection through public-facade changes and parent restoration (reads=%s)`, + async (readsFacade) => { + const parents = createControlledCollection(`published-parent`, [ + { id: 1 }, + ]) + const children = createControlledCollection(`published-child`, [ + { id: 10, parentId: 1 }, + ]) + const source = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + })), + ) + const projected = createLiveQueryCollection((q) => + q + .from({ row: source }) + .fn.select(({ row }) => { + const count = readsFacade + ? readChildren(row.children, `collection`).rows.length + : 1 + return { id: row.id, count } + }) + .distinct(), + ) + const rows = () => + projected.toArray.map(({ id, count }) => ({ id, count })) + try { + await source.preload() + await projected.preload() + expect(rows()).toEqual([{ id: 1, count: 1 }]) + children.write(`insert`, { id: 11, parentId: 1 }) + expect( + readChildren(source.toArray[0]?.children, `collection`).rows, + ).toHaveLength(2) + // A stable facade does not make its scalar reads child dependencies. + expect(rows()).toEqual([{ id: 1, count: 1 }]) + parents.write(`delete`, { id: 1 }) + expect(rows()).toEqual([]) + children.write(`delete`, { id: 11, parentId: 1 }) + parents.write(`insert`, { id: 1 }) + expect(rows()).toEqual([{ id: 1, count: 1 }]) + } finally { + await projected.cleanup() + await source.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it(`covers the declared output and renamed-field products`, () => { + expect(valueCells).toHaveLength(48) + expect(renamedCells).toHaveLength(24) + expect(operatorCells).toHaveLength(18) + expect(new Set(valueCells.map((cell) => JSON.stringify(cell))).size).toBe( + 48, + ) + expect(new Set(renamedCells.map((cell) => JSON.stringify(cell))).size).toBe( + 24, + ) + expect( + new Set(operatorCells.map((cell) => JSON.stringify(cell))).size, + ).toBe(18) + }) + + it.each(operatorCells)( + `$form / $operator / reads-include=$readsInclude consumes the projected value`, + async ({ form, operator, readsInclude }) => { + const parents = createControlledCollection(`operator-parent`, [ + { id: 1, group: 1, base: 3 }, + { id: 2, group: 2, base: 5 }, + ]) + const children = createControlledCollection(`operator-child`, [ + { id: 10, parentGroup: 1, value: 3 }, + { id: 20, parentGroup: 2, value: 5 }, + ]) + // This model stores the scalar computed on a parent projection. A live + // Collection handle does not make its scalar reads child dependencies. + const expectedScores = new Map([ + [1, 3], + [2, 5], + ]) + const observed: Array = [] + const buildQuery = () => + createLiveQueryCollection({ + query: (q) => { + const source = q + .from({ parent: parents.collection }) + .select(({ parent }) => { + const childRows = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + return { + id: parent.id, + base: parent.base, + children: + form === `collection` + ? childRows + : form === `array` + ? toArray(childRows) + : materialize(childRows), + } + }) + const projected = q.from({ row: source }).fn.select(({ row }) => { + const view = readsInclude + ? readChildren(row.children, form) + : undefined + if (view) observed.push({ ...view, rows: publicRows(view.rows) }) + return { + id: operator === `distinct` ? 0 : row.id, + score: view + ? view.rows.reduce((sum, child) => sum + child.value, 0) + : row.base, + } + }) + if (operator === `distinct`) return projected.distinct() + if (operator === `selected-order`) + return projected + .orderBy(({ $selected }) => $selected.score, `desc`) + .orderBy(({ $selected }) => $selected.id) + .limit(1) + return projected + }, + getKey: + operator === `custom-key` ? (row) => `result:${row.id}` : undefined, + }) + if (rejectsCollectionInput(form, true)) { + try { + expect(buildQuery).toThrow(collectionInputError) + expect(observed).toEqual([]) + } finally { + await parents.collection.cleanup() + await children.collection.cleanup() + } + return + } + const live = buildQuery() + const check = () => { + let expected = [...expectedScores].map(([id, score]) => ({ id, score })) + if (operator === `distinct`) + expected = [...new Set(expected.map((row) => row.score))].map( + (score) => ({ id: 0, score }), + ) + if (operator === `selected-order`) + expected = expected + .sort( + (left, right) => right.score - left.score || left.id - right.id, + ) + .slice(0, 1) + const sort = (rows: Array<{ id: number; score: number }>) => + rows.sort( + (left, right) => left.id - right.id || left.score - right.score, + ) + expect + .soft(sort(live.toArray.map(({ id, score }) => ({ id, score })))) + .toEqual(sort(expected)) + if (operator === `custom-key`) + expect + .soft([...live.keys()].sort()) + .toEqual(expected.map((row) => `result:${row.id}`).sort()) + for (const view of observed) { + expect.soft(view.valid, `operator callback input form`).toBe(true) + if (form === `collection`) + expect + .soft(view.ready, `operator callback input readiness`) + .toBe(true) + } + observed.length = 0 + } + try { + await live.preload() + check() + if (readsInclude && form !== `collection`) expectedScores.set(1, 7) + children.write(`update`, { id: 10, parentGroup: 1, value: 7 }) + check() + expectedScores.set(1, 5) + parents.write(`update`, { id: 1, group: 2, base: 5 }) + check() + expectedScores.delete(1) + parents.write(`delete`, { id: 1, group: 2, base: 5 }) + check() + } finally { + await live.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it.each(valueCells)( + `$form / $consumer / $shape / include=$withInclude preserves arbitrary output`, + async ({ form, consumer, shape, withInclude }) => { + const parents = createControlledCollection(`output-parent`, [ + { id: 1, value: 2 }, + ]) + const children = createControlledCollection(`output-child`, [ + { id: 10, parentId: 1 }, + ]) + let expectedValue = 2 + const assertValue = (value: unknown, expected?: number) => { + switch (shape) { + case `number`: + expect.soft(typeof value).toBe(`number`) + if (expected !== undefined) expect.soft(value).toBe(expected) + break + case `null`: + expect.soft(value).toBeNull() + break + case `date`: + expect.soft(value instanceof Date).toBe(true) + if (expected !== undefined && value instanceof Date) + expect.soft(value.getTime()).toBe(expected * 1000) + break + case `dropped-record`: + expect.soft(value !== null && typeof value === `object`).toBe(true) + if (value !== null && typeof value === `object`) { + // Virtual properties are public metadata. Check the selected field + // and forbid input paths without imposing a new metadata contract. + expect.soft(`children` in value || `row` in value).toBe(false) + expect.soft(`code` in value).toBe(true) + if (expected !== undefined && `code` in value) + expect.soft(value.code).toBe(expected) + } + } + } + const buildQuery = () => + createLiveQueryCollection((q) => { + const source = q + .from({ parent: parents.collection }) + .select(({ parent }) => { + const childRows = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)) + return { + id: parent.id, + value: parent.value, + ...(withInclude + ? { + children: + form === `collection` + ? childRows + : form === `array` + ? toArray(childRows) + : materialize(childRows), + } + : {}), + } + }) + const projected = q.from({ row: source }).fn.select(({ row }) => { + switch (shape) { + case `number`: + return row.value + case `null`: + return null + case `date`: + return new Date(row.value * 1000) + case `dropped-record`: + return { code: row.value } + } + }) + const outer = q.from({ result: projected }) + return consumer === `expression` + ? outer.select(({ result }) => ({ value: result })) + : outer.fn.select(({ result }) => { + // Observe the value on entry, including retract callbacks. Those + // may carry an earlier value, but must still have its proper type. + assertValue(result) + return { value: result } + }) + }) + if (rejectsCollectionInput(form, withInclude)) { + try { + expect(buildQuery).toThrow(collectionInputError) + } finally { + await parents.collection.cleanup() + await children.collection.cleanup() + } + return + } + const live = buildQuery() + const check = () => { + expect.soft(live.toArray).toHaveLength(1) + assertValue(live.toArray[0]?.value, expectedValue) + } + try { + await live.preload() + check() + children.write(`insert`, { id: 11, parentId: 1 }) + check() + expectedValue = 4 + parents.write(`update`, { id: 1, value: expectedValue }) + check() + parents.write(`delete`, { id: 1, value: expectedValue }) + expect.soft(live.toArray).toEqual([]) + } finally { + await live.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it.each(renamedCells)( + `$form / $projection / $consumer / sibling=$withSibling materializes inputs before renaming them`, + async ({ form, consumer, projection, withSibling }) => { + const parents = createControlledCollection(`renamed-parent`, [ + { id: 1, group: 1, siblingGroup: 2 }, + ]) + const initial: Array = [ + { id: 10, parentGroup: 1, value: 3 }, + { id: 20, parentGroup: 2, value: 5 }, + ] + const children = createControlledCollection(`renamed-child`, initial) + const truth = new Map(initial.map((row) => [row.id, row])) + let group = 1 + let phase: Phase = `initial` + const calls: Array<{ + phase: Phase + stage: `projection` | `consumer` + primary: ChildView + sibling?: ChildView + }> = [] + const inspect = ( + stage: `projection` | `consumer`, + primary: unknown, + sibling: unknown, + ) => { + const view = readChildren(primary, form) + const second = withSibling ? readChildren(sibling, form) : undefined + calls.push({ + phase, + stage, + primary: { ...view, rows: publicRows(view.rows) }, + sibling: second && { ...second, rows: publicRows(second.rows) }, + }) + return ( + view.rows.reduce((sum, row) => sum + row.value, 0) + + (second?.rows.reduce((sum, row) => sum + row.value, 0) ?? 0) + ) + } + const buildQuery = () => + createLiveQueryCollection((q) => { + const source = q + .from({ parent: parents.collection }) + .select(({ parent }) => { + const primary = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + const sibling = q + .from({ other: children.collection }) + .where(({ other }) => + eq(other.parentGroup, parent.siblingGroup), + ) + return { + id: parent.id, + children: + form === `collection` + ? primary + : form === `array` + ? toArray(primary) + : materialize(primary), + ...(withSibling + ? { + sibling: + form === `collection` + ? sibling + : form === `array` + ? toArray(sibling) + : materialize(sibling), + } + : {}), + } + }) + const input = q.from({ row: source }) + const projected = + projection === `expression` + ? input.select(({ row }) => ({ + id: row.id, + renamed: { primary: row.children, sibling: row.sibling }, + total: 0, + })) + : input.fn.select(({ row }) => ({ + id: row.id, + renamed: { primary: row.children, sibling: row.sibling }, + total: inspect(`projection`, row.children, row.sibling), + })) + const outer = q.from({ result: projected }) + return consumer === `expression` + ? outer.select(({ result }) => ({ value: result })) + : outer.fn.select(({ result }) => ({ + value: { + id: result.id, + renamed: result.renamed, + total: inspect( + `consumer`, + result.renamed.primary, + result.renamed.sibling, + ), + }, + })) + }) + if ( + rejectsCollectionInput( + form, + projection === `functional`, + consumer === `functional`, + ) + ) { + try { + expect(buildQuery).toThrow(collectionInputError) + expect(calls).toEqual([]) + } finally { + await parents.collection.cleanup() + await children.collection.cleanup() + } + return + } + const live = buildQuery() + const check = () => { + const row = live.toArray[0]?.value + expect.soft(live.toArray, `${phase}: row count`).toHaveLength(1) + expect.soft(row?.id, `${phase}: public id`).toBe(1) + const expectedPrimary = publicRows( + [...truth.values()].filter((child) => child.parentGroup === group), + ) + const expectedSibling = withSibling + ? publicRows( + [...truth.values()].filter((child) => child.parentGroup === 2), + ) + : [] + for (const [value, expected] of [ + [row?.renamed.primary, expectedPrimary], + ...(withSibling + ? [[row?.renamed.sibling, expectedSibling] as const] + : []), + ] as const) { + const view = readChildren(value, form) + expect.soft(view.valid, `${phase}: renamed public form`).toBe(true) + expect + .soft(publicRows(view.rows), `${phase}: renamed public rows`) + .toEqual(expected) + if (form === `collection`) + expect + .soft(view.ready, `${phase}: renamed public readiness`) + .toBe(true) + } + if (row) + expect + .soft( + `children` in row || `row` in row, + `${phase}: input paths do not leak`, + ) + .toBe(false) + if ( + form !== `collection` || + phase === `initial` || + phase === `route-move` + ) { + if (projection === `functional` || consumer === `functional`) + expect + .soft(row?.total, `${phase}: derived total`) + .toBe( + [...expectedPrimary, ...expectedSibling].reduce( + (sum, child) => sum + child.value, + 0, + ), + ) + for (const stage of [ + ...(projection === `functional` ? [`projection` as const] : []), + ...(consumer === `functional` ? [`consumer` as const] : []), + ]) { + expect + .soft( + calls.filter( + (call) => call.phase === phase && call.stage === stage, + ).length, + `${phase}: ${stage} callback reach`, + ) + .toBeGreaterThan(0) + } + } + for (const call of calls.filter((item) => item.phase === phase)) { + for (const view of [ + call.primary, + ...(call.sibling ? [call.sibling] : []), + ]) { + expect.soft(view.valid, `${phase}: callback input form`).toBe(true) + if (form === `collection`) + expect + .soft(view.ready, `${phase}: callback input readiness`) + .toBe(true) + } + } + } + try { + await live.preload() + check() + phase = `child-update` + const changed = { id: 10, parentGroup: 1, value: 7 } + truth.set(10, changed) + children.write(`update`, changed) + check() + if (withSibling) { + phase = `sibling-update` + const sibling = { id: 20, parentGroup: 2, value: 11 } + truth.set(20, sibling) + children.write(`update`, sibling) + check() + } + phase = `route-move` + group = 2 + parents.write(`update`, { id: 1, group, siblingGroup: 2 }) + check() + } finally { + await live.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) +}) + +describe(`functional include projection boundary grammar`, () => { + it(`preserves a scalar result when a functional projection drops its include`, async () => { + const parents = createControlledCollection(`scalar-projection-parent`, [ + { id: 1 }, + ]) + const children = createControlledCollection(`scalar-projection-child`, [ + { id: 10, parentId: 1 }, + ]) + const live = createLiveQueryCollection((q) => { + const included = q + .from({ parent: parents.collection }) + .select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + ), + })) + const scalar = q.from({ row: included }).fn.select(({ row }) => row.id) + return q + .from({ result: scalar }) + .select(({ result }) => ({ value: result })) + }) + try { + await live.preload() + expect(live.toArray.map((row) => row.value)).toEqual([1]) + } finally { + await live.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }) + + it(`preserves opaque-root fields without include materialization`, async () => { + const parents = createControlledCollection(`opaque-root-control`, [ + { id: 1 }, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .fn.select( + ({ parent }) => new Projection(parent.id, `plain`, undefined, 0), + ), + ) + try { + await live.preload() + const row = live.toArray[0] + expect(row?.id).toBe(1) + expect(row?.kind).toBe(`plain`) + expect(row?.total).toBe(0) + // Collection root records already flatten prototypes without includes. + // This matrix checks their fields, not a new prototype-preservation API. + } finally { + await live.cleanup() + await parents.collection.cleanup() + } + }) + + it(`covers every declared boundary product without duplicate cells`, () => { + expect(cells).toHaveLength(54) + expect(new Set(cells.map((cell) => JSON.stringify(cell))).size).toBe(54) + }) + + it.each(cells)( + `$boundary / $form / $output / $initial`, + async ({ boundary, form, output, initial }) => { + const parents = createControlledCollection(`projection-parents`, [ + { id: 1, group: 1 }, + ]) + const absent = createControlledCollection(`projection-absent`, [ + { id: 2 }, + ]) + const initialChildren: Array = [ + ...(initial === `populated` + ? [{ id: 10, parentGroup: 1, value: 3 }] + : []), + { id: 20, parentGroup: 2, value: 5 }, + ] + const children = createControlledCollection( + `projection-children`, + initialChildren, + ) + const truth = new Map(initialChildren.map((row) => [row.id, row])) + let group = 1 + let phase: Phase = `initial` + const calls: Array<{ + phase: Phase + kind: string + child: unknown + view: ChildView + }> = [] + const project = (row: Input) => { + const child = row.children + const view = readChildren(child, form) + // Capture readiness and contents NOW, not through a reference read after preload. + calls.push({ + phase, + kind: row.kind, + child, + view: { ...view, rows: publicRows(view.rows) }, + }) + const total = view.rows.reduce((sum, item) => sum + item.value, 0) + return output === `opaque-root` + ? new Projection(row.id, row.kind, child, total) + : { id: row.id, kind: row.kind, children: child, total } + } + const buildQuery = () => + createLiveQueryCollection((q) => { + const included = q + .from({ parent: parents.collection }) + .select(({ parent }) => { + const childRows = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + value: child.value, + })) + return { + id: parent.id, + kind: `included`, + total: 0, + children: + form === `collection` + ? childRows + : form === `array` + ? toArray(childRows) + : materialize(childRows), + } + }) + if (boundary === `union`) { + const withoutInclude = q + .from({ other: absent.collection }) + .select(({ other }) => ({ + id: other.id, + kind: `absent`, + total: 0, + })) + const union = q.unionAll(included, withoutInclude) + return output === `expression` ? union : union.fn.select(project) + } + if (boundary === `recursive-query-ref`) { + const intermediate = q + .from({ inner: included }) + .select(({ inner }) => inner) + const outer = q.from({ row: intermediate }) + return output === `expression` + ? outer.select(({ row }) => row) + : outer.fn.select(({ row }) => project(row)) + } + const outer = q.from({ row: included }) + return output === `expression` + ? outer.select(({ row }) => row) + : outer.fn.select(({ row }) => project(row)) + }) + if (rejectsCollectionInput(form, output !== `expression`)) { + try { + expect(buildQuery).toThrow(collectionInputError) + expect(calls).toEqual([]) + } finally { + await parents.collection.cleanup() + await children.collection.cleanup() + await absent.collection.cleanup() + } + return + } + const live = buildQuery() + let facade: unknown + const check = () => { + const row: (Input & { total: number }) | undefined = live.toArray.find( + (item) => item.kind === `included`, + ) + expect.soft(row, `${phase}: included public row`).toBeDefined() + if (!row) return + const expected = publicRows( + [...truth.values()].filter((item) => item.parentGroup === group), + ) + const view = readChildren(row.children, form) + expect.soft(view.valid, `${phase}: public include form`).toBe(true) + expect + .soft(publicRows(view.rows), `${phase}: public children`) + .toEqual(expected) + if (form === `collection`) { + expect.soft(view.ready, `${phase}: public facade ready`).toBe(true) + if (phase === `initial`) facade = row.children + else if (phase === `child-update`) + expect.soft(row.children, `child-only facade identity`).toBe(facade) + else + expect + .soft(row.children, `route move replaces facade`) + .not.toBe(facade) + } + // A Collection is a live handle, not a dependency-tracked scalar read. + // Assert derived scalars when the parent projection runs, not on child-only + // changes to a retained facade. Inline values do drive parent recomputation. + if ( + output !== `expression` && + (form !== `collection` || phase !== `child-update`) + ) { + expect + .soft(row.total, `${phase}: derived scalar`) + .toBe(expected.reduce((sum, item) => sum + item.value, 0)) + } + if (boundary === `union`) { + const other: Input | undefined = live.toArray.find( + (item) => item.kind === `absent`, + ) + expect.soft(other, `${phase}: absent branch survives`).toBeDefined() + expect + .soft(other?.children, `${phase}: absent branch value`) + .toBeUndefined() + } + const current = calls.filter( + (call) => call.phase === phase && call.kind === `included`, + ) + if ( + output !== `expression` && + (form !== `collection` || phase !== `child-update`) + ) + expect + .soft(current.length, `${phase}: callback reach`) + .toBeGreaterThan(0) + for (const call of current) { + expect + .soft(call.view.valid, `${phase}: callback include form`) + .toBe(true) + if (form === `collection`) + expect + .soft(call.view.ready, `${phase}: callback facade ready`) + .toBe(true) + } + for (const call of calls.filter( + (item) => item.phase === phase && item.kind === `absent`, + )) { + expect + .soft(call.child, `${phase}: valid callback absence`) + .toBeUndefined() + } + } + try { + await live.preload() + check() + phase = `child-update` + const changed = { id: 10, parentGroup: 1, value: 7 } + truth.set(10, changed) + children.write(initial === `empty` ? `insert` : `update`, changed) + check() + phase = `route-move` + group = 2 + parents.write(`update`, { id: 1, group }) + check() + } finally { + await live.cleanup() + await Promise.all([ + parents.collection.cleanup(), + children.collection.cleanup(), + absent.collection.cleanup(), + ]) + } + }, + ) +}) diff --git a/packages/db/tests/query/includes-lazy-loading.test.ts b/packages/db/tests/query/includes-lazy-loading.test.ts index 8e8eccace3..f2ed8f5bc3 100644 --- a/packages/db/tests/query/includes-lazy-loading.test.ts +++ b/packages/db/tests/query/includes-lazy-loading.test.ts @@ -8,7 +8,11 @@ import { } from '../../src/query/index.js' import { createCollection } from '../../src/collection/index.js' import { extractSimpleComparisons } from '../../src/query/expression-helpers.js' -import { flushPromises, stripVirtualProps } from '../utils.js' +import { + flushPromises, + mockSyncCollectionOptions, + stripVirtualProps, +} from '../utils.js' import type { LoadSubsetOptions } from '../../src/types.js' /** @@ -127,6 +131,85 @@ describe(`includes lazy loading`, () => { ]) }) + it(`targets the joined source in a lazy self-join`, async () => { + type SelfItem = { + id: number + peerId: number + rootId: number + label: string + } + const roots = createCollection( + mockSyncCollectionOptions<{ id: number }>({ + id: `includes-lazy-self-join-roots`, + getKey: (root) => root.id, + initialData: [{ id: 1 }], + }), + ) + const rows: Array = [ + { id: 1, peerId: 2, rootId: 999, label: `left` }, + { id: 2, peerId: 0, rootId: 1, label: `right` }, + ] + const installed = new Set() + const items = createCollection({ + id: `includes-lazy-self-join-items`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: (options) => { + const rootIds = new Set( + extractSimpleComparisons(options.where).flatMap((comparison) => + comparison.field[0] === `rootId` && + comparison.operator === `in` && + Array.isArray(comparison.value) + ? comparison.value + : [], + ), + ) + begin() + for (const row of rows) { + if ( + installed.has(row.id) || + (rootIds.size > 0 && !rootIds.has(row.rootId)) + ) { + continue + } + installed.add(row.id) + write({ type: `insert`, value: row }) + } + commit() + markReady() + return Promise.resolve() + }, + }), + }, + }) + + const live = createLiveQueryCollection((q) => + q.from({ root: roots }).select(({ root }) => ({ + id: root.id, + matches: toArray( + q + .from({ left: items }) + .join({ right: items }, ({ left, right }) => + eq(left.peerId, right.id), + ) + .where(({ right }) => eq(right.rootId, root.id)) + .select(({ left, right }) => ({ + left: left.label, + right: right.label, + })), + ), + })), + ) + + await live.preload() + expect(stripVirtualProps(live.get(1))).toMatchObject({ + id: 1, + matches: [{ left: `left`, right: `right` }], + }) + }) + it(`should produce correct query results with lazy-loaded includes`, async () => { const roots = createRootsCollection() const { collection: items } = createItemsCollectionWithTracking() @@ -434,12 +517,12 @@ describe(`includes child where clauses in loadSubset`, () => { * through to the child collection's loadSubset/queryFn. */ - type Root = { + type FilterRoot = { id: number name: string } - type Item = { + type FilterItem = { id: number rootId: number status: string @@ -447,12 +530,12 @@ describe(`includes child where clauses in loadSubset`, () => { title: string } - const sampleRoots: Array = [ + const filterRoots: Array = [ { id: 1, name: `Root A` }, { id: 2, name: `Root B` }, ] - const sampleItems: Array = [ + const filterItems: Array = [ { id: 10, rootId: 1, status: `active`, priority: 3, title: `A1 active` }, { id: 11, @@ -466,13 +549,13 @@ describe(`includes child where clauses in loadSubset`, () => { ] function createRootsCollection() { - return createCollection({ + return createCollection({ id: `child-where-roots`, getKey: (r) => r.id, sync: { sync: ({ begin, write, commit, markReady }) => { begin() - for (const root of sampleRoots) { + for (const root of filterRoots) { write({ type: `insert`, value: root }) } commit() @@ -485,14 +568,14 @@ describe(`includes child where clauses in loadSubset`, () => { function createItemsCollectionWithTracking() { const loadSubsetCalls: Array = [] - const collection = createCollection({ + const collection = createCollection({ id: `child-where-items`, getKey: (item) => item.id, syncMode: `on-demand`, sync: { sync: ({ begin, write, commit, markReady }) => { begin() - for (const item of sampleItems) { + for (const item of filterItems) { write({ type: `insert`, value: item }) } commit() diff --git a/packages/db/tests/query/includes-optimistic-oracle.property.test.ts b/packages/db/tests/query/includes-optimistic-oracle.property.test.ts new file mode 100644 index 0000000000..23b4bf762d --- /dev/null +++ b/packages/db/tests/query/includes-optimistic-oracle.property.test.ts @@ -0,0 +1,771 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect } from 'vitest' +import { + createLiveQueryCollection, + eq, + toArray, +} from '../../src/query/index.js' +import { flushPromises, withExpectedRejection } from '../utils.js' +import { runTrace } from '../trace-runner.js' +import { oraclePropertyOptions } from '../oracle-config.js' +import { createControlledCollection as createOracleControlledCollection } from './includes-oracle-helpers.js' +import type { TraceDriver, TraceProjection } from '../trace-runner.js' +import type { OracleSyncChange as SyncChange } from './includes-oracle-helpers.js' + +type RootRow = { + id: number + group: number + value: number + position: number +} + +type ChildRow = RootRow & { + parentGroup: number +} + +type ChildLevel = 1 | 2 | 3 + +type ChildPatch = Partial< + Pick +> + +type OptimisticStep = { + type: `optimistic` + handle: string + level: ChildLevel + id: number + patch: ChildPatch +} + +type OptimisticRelationshipStep = + | OptimisticStep + | { + type: `optimisticRollback` + level: ChildLevel + id: number + patch: ChildPatch + beforeRollback?: { + level: ChildLevel + changes: ReadonlyArray> + } + } + | { + type: `confirm` + handle: string + authoritative: ChildRow + } + | { type: `rollback`; handle: string } + | { + type: `sync` + level: ChildLevel + changes: ReadonlyArray> + } + +type OracleNode = RootRow & { + children?: Array +} + +type ControlledCollection = ReturnType< + typeof createControlledCollection +> + +type Sources = { + roots: ControlledCollection + levels: readonly [ + ControlledCollection, + ControlledCollection, + ControlledCollection, + ] +} + +type LevelRows = readonly [ + ReadonlyArray, + ReadonlyArray, + ReadonlyArray, +] + +type SettlingTransaction = { + isPersisted: { promise: Promise } +} + +type PendingOptimisticChange = { + transaction: SettlingTransaction + level: ChildLevel + id: number + row: ChildRow +} + +type OptimisticContext = { + sources: Sources + live: ReturnType + roots: Map + levels: Array> + pending: Map +} + +function assertCanStartOptimisticChange( + pending: ReadonlyMap>, + level: ChildLevel, + handle?: string, +): void { + if (handle !== undefined && pending.has(handle)) { + throw new Error(`Duplicate optimistic handle ${handle}`) + } + const sameLevel = [...pending.entries()].find( + ([, change]) => change.level === level, + ) + if (sameLevel) { + throw new Error( + `Level ${level} already has pending optimistic handle ${sameLevel[0]}`, + ) + } +} + +function assertMatchingConfirmation( + pending: Pick, + authoritative: Pick, +): void { + if (authoritative.id !== pending.id) { + throw new Error( + `Confirmation row ${authoritative.id} does not match pending row ${pending.id}`, + ) + } +} + +type RouteValues = { + rootA: number + rootB: number + rootC: number + original: number + optimistic: number + authoritative: number +} + +function createControlledCollection( + name: string, + initialData: ReadonlyArray, +) { + return createOracleControlledCollection(name, initialData, { + rowUpdateMode: `full`, + }) +} + +function createSources( + roots: ReadonlyArray, + levels: LevelRows, +): Sources { + return { + roots: createControlledCollection(`optimistic-oracle-roots`, roots), + levels: [ + createControlledCollection(`optimistic-oracle-level-1`, levels[0]), + createControlledCollection(`optimistic-oracle-level-2`, levels[1]), + createControlledCollection(`optimistic-oracle-level-3`, levels[2]), + ], + } +} + +function childSource(sources: Sources, level: ChildLevel) { + switch (level) { + case 1: + return sources.levels[0] + case 2: + return sources.levels[1] + case 3: + return sources.levels[2] + } +} + +function createOptimisticQuery(sources: Sources) { + const [children, grandchildren, leaves] = sources.levels + return createLiveQueryCollection((q) => + q + .from({ root: sources.roots.collection }) + .orderBy(({ root }) => root.position) + .orderBy(({ root }) => root.id) + .select(({ root }) => ({ + id: root.id, + group: root.group, + value: root.value, + position: root.position, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, root.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + group: child.group, + value: child.value, + position: child.position, + children: toArray( + q + .from({ grandchild: grandchildren.collection }) + .where(({ grandchild }) => + eq(grandchild.parentGroup, child.group), + ) + .orderBy(({ grandchild }) => grandchild.position) + .orderBy(({ grandchild }) => grandchild.id) + .select(({ grandchild }) => ({ + id: grandchild.id, + group: grandchild.group, + value: grandchild.value, + position: grandchild.position, + children: toArray( + q + .from({ leaf: leaves.collection }) + .where(({ leaf }) => + eq(leaf.parentGroup, grandchild.group), + ) + .orderBy(({ leaf }) => leaf.position) + .orderBy(({ leaf }) => leaf.id) + .select(({ leaf }) => ({ + id: leaf.id, + group: leaf.group, + value: leaf.value, + position: leaf.position, + })), + ), + })), + ), + })), + ), + })), + ) +} + +function cloneMap(rows: ReadonlyArray) { + return new Map(rows.map((row) => [row.id, { ...row }])) +} + +function applyPatch(row: ChildRow, patch: ChildPatch): ChildRow { + return { ...row, ...patch } +} + +function visibleLevels(context: OptimisticContext) { + const levels = context.levels.map( + (level) => new Map([...level].map(([id, row]) => [id, { ...row }])), + ) + + for (const { level, id, row } of context.pending.values()) { + const rows = levels[level - 1]! + rows.set(id, { ...row }) + } + return levels +} + +function compareRows(left: RootRow, right: RootRow) { + return left.position - right.position || left.id - right.id +} + +function recompute(context: OptimisticContext): Array { + const levels = visibleLevels(context) + const materialize = (level: number, parentGroup: number): Array => + [...levels[level]!.values()] + .filter((row) => row.parentGroup === parentGroup) + .sort(compareRows) + .map(({ parentGroup: _parentGroup, ...row }) => ({ + ...row, + ...(level + 1 < levels.length + ? { children: materialize(level + 1, row.group) } + : {}), + })) + + return [...context.roots.values()].sort(compareRows).map((root) => ({ + ...root, + children: materialize(0, root.group), + })) +} + +function stripVirtualProperties(value: unknown): unknown { + if (Array.isArray(value)) return value.map(stripVirtualProperties) + if (!value || typeof value !== `object`) return value + return Object.fromEntries( + Object.entries(value) + .filter(([key]) => !key.startsWith(`$`)) + .map(([key, entry]) => [key, stripVirtualProperties(entry)]), + ) +} + +function updateModel( + model: Map, + changes: ReadonlyArray>, +) { + for (const change of changes) { + if (change.type === `delete`) model.delete(change.value.id) + else model.set(change.value.id, { ...change.value }) + } +} + +async function rollback( + source: ControlledCollection, + transaction: SettlingTransaction, +) { + const message = `optimistic relationship oracle rollback` + const persisted = transaction.isPersisted.promise.catch(() => undefined) + await withExpectedRejection(message, async () => { + source.rejectSync(new Error(message)) + await persisted + await flushPromises() + }) +} + +function createDriver( + roots: ReadonlyArray, + levelRows: LevelRows, +): TraceDriver { + return { + setup: () => { + const sources = createSources(roots, levelRows) + return { + sources, + live: createOptimisticQuery(sources), + roots: cloneMap(roots), + levels: levelRows.map(cloneMap), + pending: new Map(), + } + }, + start: ({ live }) => live.preload(), + apply: async (step, context, checkpoint) => { + if (step.type === `optimisticRollback`) { + assertCanStartOptimisticChange(context.pending, step.level) + const source = childSource(context.sources, step.level) + const transaction = source.collection.update(step.id, (draft) => { + Object.assign(draft, step.patch) + }) + if (step.beforeRollback) { + const beforeRollbackSource = childSource( + context.sources, + step.beforeRollback.level, + ) + beforeRollbackSource.writeBatch(step.beforeRollback.changes) + updateModel( + context.levels[step.beforeRollback.level - 1]!, + step.beforeRollback.changes, + ) + } + // This compound action checks the settled state. The immediate state is + // checked separately so its known mismatch cannot abort the rollback. + await rollback(source, transaction) + return + } + + if (step.type === `optimistic`) { + assertCanStartOptimisticChange(context.pending, step.level, step.handle) + const source = childSource(context.sources, step.level) + const current = visibleLevels(context)[step.level - 1]!.get(step.id) + if (!current) throw new Error(`Unknown optimistic row ${step.id}`) + const transaction = source.collection.update(step.id, (draft) => { + Object.assign(draft, step.patch) + }) + context.pending.set(step.handle, { + transaction, + level: step.level, + id: step.id, + row: applyPatch(current, step.patch), + }) + return + } + + if (step.type === `sync`) { + const source = childSource(context.sources, step.level) + source.writeBatch(step.changes) + updateModel(context.levels[step.level - 1]!, step.changes) + return + } + + const pending = context.pending.get(step.handle) + if (!pending) throw new Error(`Unknown optimistic handle ${step.handle}`) + const source = childSource(context.sources, pending.level) + + if (step.type === `rollback`) { + context.pending.delete(step.handle) + await rollback(source, pending.transaction) + return + } + + assertMatchingConfirmation(pending, step.authoritative) + source.writeBatch([{ type: `update`, value: step.authoritative }]) + context.levels[pending.level - 1]!.set(step.authoritative.id, { + ...step.authoritative, + }) + // Sync delivery must not displace the pending optimistic projection. + checkpoint() + source.resolveSync() + await pending.transaction.isPersisted.promise + context.pending.delete(step.handle) + }, + cleanup: async ({ live, sources, pending }) => { + for (const [handle, change] of pending) { + pending.delete(handle) + await rollback(childSource(sources, change.level), change.transaction) + } + await live.cleanup() + await Promise.all([ + sources.roots.collection.cleanup(), + ...sources.levels.map(({ collection }) => collection.cleanup()), + ]) + }, + } +} + +const projection: TraceProjection< + OptimisticContext, + unknown, + Array +> = { + observe: ({ live }) => stripVirtualProperties(live.toArray), + recompute, + assertEqual: (actual, expected) => { + expect(actual).toEqual(expected) + }, +} + +function firstChild(routes: RouteValues, patch: ChildPatch = {}): ChildRow { + return { + id: 11, + parentGroup: routes.rootA, + group: routes.original, + value: 110, + position: 0, + ...patch, + } +} + +function fixture(routes: RouteValues) { + const roots: Array = [ + { id: 1, group: routes.rootA, value: 10, position: 0 }, + { id: 2, group: routes.rootB, value: 20, position: 1 }, + { id: 3, group: routes.rootC, value: 30, position: 2 }, + ] + const levels: LevelRows = [ + [firstChild(routes)], + [ + { + id: 21, + parentGroup: routes.original, + group: routes.original + 1000, + value: 210, + position: 0, + }, + ], + [ + { + id: 31, + parentGroup: routes.original + 1000, + group: routes.original + 2000, + value: 310, + position: 0, + }, + ], + ] + return { roots, levels } +} + +async function expectHistoryMatches( + routes: RouteValues, + steps: ReadonlyArray, +) { + const { roots, levels } = fixture(routes) + await runTrace({ steps, driver: createDriver(roots, levels), projection }) +} + +const routeValuesArbitrary: fc.Arbitrary = fc.record({ + // Routes are equality keys. Disjoint ranges preserve distinctness while + // letting FastCheck shrink each semantic role independently. + rootA: fc.integer({ min: 10, max: 90 }), + rootB: fc.integer({ min: 100, max: 180 }), + rootC: fc.integer({ min: 200, max: 280 }), + original: fc.integer({ min: 300, max: 380 }), + optimistic: fc.integer({ min: 400, max: 480 }), + authoritative: fc.integer({ min: 500, max: 580 }), +}) + +describe(`optimistic relationship-transition oracle`, () => { + fcTest( + `rejects optimistic handles the sync mock cannot settle independently`, + () => { + const pending = new Map([[`first`, { level: 1 as const }]]) + + expect(() => assertCanStartOptimisticChange(pending, 2, `first`)).toThrow( + /Duplicate optimistic handle first/, + ) + expect(() => + assertCanStartOptimisticChange(pending, 1, `second`), + ).toThrow(/Level 1 already has pending optimistic handle first/) + expect(() => assertMatchingConfirmation({ id: 11 }, { id: 12 })).toThrow( + /Confirmation row 12 does not match pending row 11/, + ) + }, + ) + + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.rekey-detach`), + )( + `an optimistic rekey detaches its old descendants immediately`, + async (routes) => { + await expectHistoryMatches(routes, [ + { + type: `optimistic`, + handle: `rekey`, + level: 1, + id: 11, + patch: { group: routes.optimistic }, + }, + ]) + }, + ) + + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.rekey-rollback`), + )( + `restores the authoritative relationship after an optimistic rekey rolls back`, + async (routes) => { + await expectHistoryMatches(routes, [ + { + type: `optimisticRollback`, + level: 1, + id: 11, + patch: { group: routes.optimistic }, + }, + ]) + }, + ) + + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.descendant-rollback`), + )( + `rolls back a descendant update made while its ancestor is reparented`, + async (routes) => { + await expectHistoryMatches(routes, [ + { + type: `optimistic`, + handle: `reparent`, + level: 1, + id: 11, + patch: { parentGroup: routes.rootB }, + }, + { + type: `optimistic`, + handle: `descendant`, + level: 2, + id: 21, + patch: { value: 211 }, + }, + { type: `rollback`, handle: `descendant` }, + { type: `rollback`, handle: `reparent` }, + ]) + }, + ) + + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.ancestor-rollback`), + )( + `rolls back a reparented ancestor while its descendant update remains pending`, + async (routes) => { + await expectHistoryMatches(routes, [ + { + type: `optimistic`, + handle: `reparent`, + level: 1, + id: 11, + patch: { parentGroup: routes.rootB }, + }, + { + type: `optimistic`, + handle: `descendant`, + level: 2, + id: 21, + patch: { value: 211 }, + }, + { type: `rollback`, handle: `reparent` }, + { type: `rollback`, handle: `descendant` }, + ]) + }, + ) + + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.confirm-same-route`), + )( + `settles a confirmed optimistic reparent on the same authoritative route`, + async (routes) => { + await expectHistoryMatches(routes, [ + { + type: `optimistic`, + handle: `reparent`, + level: 1, + id: 11, + patch: { parentGroup: routes.rootB }, + }, + { + type: `confirm`, + handle: `reparent`, + authoritative: firstChild(routes, { + parentGroup: routes.rootB, + }), + }, + { + type: `sync`, + level: 2, + changes: [ + { + type: `update`, + value: { + id: 21, + parentGroup: routes.original, + group: routes.original + 1000, + value: 211, + position: 0, + }, + }, + ], + }, + ]) + }, + ) + + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.confirm-different-route`), + )( + `settles a confirmed optimistic reparent on a different authoritative route`, + async (routes) => { + await expectHistoryMatches(routes, [ + { + type: `optimistic`, + handle: `reparent`, + level: 1, + id: 11, + patch: { parentGroup: routes.rootB }, + }, + { + type: `confirm`, + handle: `reparent`, + authoritative: firstChild(routes, { + parentGroup: routes.rootC, + group: routes.authoritative, + value: 111, + }), + }, + { + type: `sync`, + level: 2, + changes: [ + { + type: `update`, + value: { + id: 21, + parentGroup: routes.authoritative, + group: routes.original + 1000, + value: 211, + position: 0, + }, + }, + ], + }, + ]) + }, + ) + + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.sibling-route-rollback`), + )(`restores a rekey after a sibling enters its old route`, async (routes) => { + await expectHistoryMatches(routes, [ + { + type: `optimisticRollback`, + level: 1, + id: 11, + patch: { group: routes.optimistic }, + beforeRollback: { + level: 1, + changes: [ + { + type: `insert`, + value: { + id: 12, + parentGroup: routes.rootA, + group: routes.original, + value: 120, + position: 1, + }, + }, + ], + }, + }, + { + type: `sync`, + level: 2, + changes: [ + { + type: `update`, + value: { + id: 21, + parentGroup: routes.original, + group: routes.original + 1000, + value: 211, + position: 0, + }, + }, + ], + }, + ]) + }) + + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.repeated-history`), + )(`supports repeated rollback and confirmation histories`, async (routes) => { + await expectHistoryMatches(routes, [ + { + type: `optimistic`, + handle: `first`, + level: 1, + id: 11, + patch: { parentGroup: routes.rootB }, + }, + { type: `rollback`, handle: `first` }, + { + type: `optimistic`, + handle: `second`, + level: 1, + id: 11, + patch: { parentGroup: routes.rootB }, + }, + { + type: `confirm`, + handle: `second`, + authoritative: firstChild(routes, { + parentGroup: routes.rootB, + }), + }, + { + type: `optimisticRollback`, + level: 1, + id: 11, + patch: { group: routes.optimistic }, + }, + { + type: `sync`, + level: 2, + changes: [ + { + type: `update`, + value: { + id: 21, + parentGroup: routes.original, + group: routes.original + 1000, + value: 212, + position: 0, + }, + }, + ], + }, + ]) + }) +}) diff --git a/packages/db/tests/query/includes-oracle-helpers.ts b/packages/db/tests/query/includes-oracle-helpers.ts new file mode 100644 index 0000000000..77f08543b5 --- /dev/null +++ b/packages/db/tests/query/includes-oracle-helpers.ts @@ -0,0 +1,58 @@ +import { createCollection } from '../../src/collection/index.js' +import { mockSyncCollectionOptions } from '../utils.js' +import type { Collection } from '../../src/collection/index.js' + +export type OracleSyncChange = { + type: `insert` | `update` | `delete` + value: T +} + +export type ControlledCollection = { + collection: Collection + write: (type: OracleSyncChange[`type`], value: T) => void + writeBatch: (changes: ReadonlyArray>) => void + resolveSync: () => void + rejectSync: (error: Error) => void +} + +type ControlledCollectionOptions = { + autoIndex?: `off` | `eager` + rowUpdateMode?: `partial` | `full` +} + +let nextControlledCollectionId = 0 + +export function createControlledCollection( + name: string, + initialData: ReadonlyArray = [], + options: ControlledCollectionOptions = {}, +): ControlledCollection { + const collectionOptions = mockSyncCollectionOptions({ + id: `${name}-${nextControlledCollectionId++}`, + getKey: (row) => row.id, + initialData: initialData.map((row) => ({ ...row })), + ...(options.autoIndex ? { autoIndex: options.autoIndex } : {}), + }) + collectionOptions.sync.rowUpdateMode = options.rowUpdateMode ?? `partial` + const collection = createCollection(collectionOptions) + const writeBatch: ControlledCollection[`writeBatch`] = (changes) => { + collectionOptions.utils.begin() + for (const change of changes) { + collectionOptions.utils.write({ + type: change.type, + value: { ...change.value }, + }) + } + collectionOptions.utils.commit() + } + + return { + collection, + write(type, value) { + writeBatch([{ type, value }]) + }, + writeBatch, + resolveSync: collectionOptions.utils.resolveSync, + rejectSync: collectionOptions.utils.rejectSync, + } +} diff --git a/packages/db/tests/query/includes-oracle.property.test.ts b/packages/db/tests/query/includes-oracle.property.test.ts new file mode 100644 index 0000000000..a1e4ec2018 --- /dev/null +++ b/packages/db/tests/query/includes-oracle.property.test.ts @@ -0,0 +1,4744 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect } from 'vitest' +import { + concat, + createLiveQueryCollection, + eq, + materialize, + queryOnce, + toArray, +} from '../../src/query/index.js' +import { flushPromises, withExpectedRejection } from '../utils.js' +import { oraclePropertyOptions, oracleRuns } from '../oracle-config.js' +import { runTrace } from '../trace-runner.js' +import { createControlledCollection as createOracleControlledCollection } from './includes-oracle-helpers.js' +import type { + TraceCheckpoint, + TraceDriver, + TraceProjection, +} from '../trace-runner.js' +import type { OracleSyncChange as SyncChange } from './includes-oracle-helpers.js' + +type IncludeDepth = 1 | 2 | 3 | 4 + +type RootRow = { + id: number + group: number + value: number + position: number +} + +type ChildRow = RootRow & { + parentGroup: number +} + +type HistoryAction = { + type: `put` | `delete` | `optimisticConfirm` | `optimisticRollback` + level: 0 | IncludeDepth + id: number + parentGroup: number + group: number + value: number + position: number +} + +type Scenario = { + depth: IncludeDepth + history: Array +} + +type OracleNode = RootRow & { + children?: Array +} + +type RelationshipNode = { + id: number + value?: unknown + children?: unknown +} + +function isRelationshipNode(value: unknown): value is RelationshipNode { + return ( + typeof value === `object` && + value !== null && + `id` in value && + typeof value.id === `number` + ) +} + +function findRelationshipNode( + value: unknown, + id: number, +): RelationshipNode | undefined { + if (!Array.isArray(value)) return undefined + + for (const entry of value) { + if (!isRelationshipNode(entry)) continue + if (entry.id === id) return entry + const nested = findRelationshipNode(entry.children, id) + if (nested) return nested + } + return undefined +} + +function hasDirectChild(value: unknown, parentId: number, childId: number) { + const parent = findRelationshipNode(value, parentId) + return ( + Array.isArray(parent?.children) && + parent.children.some( + (child) => isRelationshipNode(child) && child.id === childId, + ) + ) +} + +type RelationshipProjectionNode = { + id: number + children?: Array +} + +function relationshipOnly( + nodes: ReadonlyArray, +): Array { + return nodes.map(({ id, children }) => ({ + id, + ...(children ? { children: relationshipOnly(children) } : {}), + })) +} + +type MaterializeRoot = { id: number; middleId: number } +type MaterializeMiddle = { id: number; sharedId: number } +type MaterializeShared = { id: number; leafId: number } +type MaterializeLeaf = { id: number; value: number } + +type MaterializeTree = { + id: number + middle: + | { + id: number + sharedId: number + shared: + | { + id: number + leafId: number + leaf: MaterializeLeaf | undefined + } + | undefined + } + | undefined +} + +type MaterializeInsert = + | `root-1` + | `root-2` + | `middle-1` + | `middle-2` + | `shared-1` + | `shared-2` + | `leaf-1` + | `leaf-2` + +type MaterializeScenario = { + sharedIntermediate: boolean + insertOrder: Array +} + +const depthArbitrary = fc.constantFrom(1, 2, 3, 4) + +function levelArbitrary( + depth: IncludeDepth, +): fc.Arbitrary { + switch (depth) { + case 1: + return fc.constantFrom(0, 1) + case 2: + return fc.constantFrom(0, 1, 2) + case 3: + return fc.constantFrom(0, 1, 2, 3) + case 4: + return fc.constantFrom(0, 1, 2, 3, 4) + } +} + +function actionArbitrary(depth: IncludeDepth): fc.Arbitrary { + return levelArbitrary(depth).chain((level) => + fc.record({ + // Root delete/reinsert has a focused history matrix below. Keep this + // unconstrained corpus small enough to shrink to one clear action. + type: + level === 0 + ? fc.constantFrom( + `put` as const, + `optimisticConfirm` as const, + `optimisticRollback` as const, + ) + : fc.constantFrom( + `put` as const, + `delete` as const, + `optimisticConfirm` as const, + `optimisticRollback` as const, + ), + level: fc.constant(level), + id: fc.integer({ min: 0, max: 5 }), + parentGroup: fc.integer({ min: 0, max: 2 }), + group: fc.integer({ min: 0, max: 2 }), + value: fc.integer({ min: -3, max: 3 }), + position: fc.integer({ min: -2, max: 2 }), + }), + ) +} + +function ensureActionsTargetRows( + history: Array, +): Array { + const keysByLevel = Array.from({ length: 5 }, () => new Set()) + const positionsByLevel = Array.from( + { length: 5 }, + () => new Map(), + ) + + return history.map((action) => { + const keys = keysByLevel[action.level]! + const positions = positionsByLevel[action.level]! + const normalizePut = (): HistoryAction => { + keys.add(action.id) + const position = positions.get(action.id) ?? action.position + positions.set(action.id, position) + return { ...action, type: `put`, position } + } + + if (action.type === `put`) { + return normalizePut() + } + if (keys.size === 0) { + return normalizePut() + } + + const existingKeys = [...keys] + const id = existingKeys[action.id % existingKeys.length]! + if (action.type === `delete`) { + keys.delete(id) + positions.delete(id) + } + return { ...action, id } + }) +} + +const scenarioArbitrary: fc.Arbitrary = depthArbitrary.chain( + (depth) => + fc + .array(actionArbitrary(depth), { minLength: 1, maxLength: 18 }) + .map((history) => ({ + depth, + history: ensureActionsTargetRows(history), + })), +) + +function classifyScenarioCoverage({ depth, history }: Scenario): Array { + const routesByLevel = Array.from( + { length: depth + 1 }, + () => new Map(), + ) + let relationshipChanges = 0 + + for (const action of history) { + const routes = routesByLevel[action.level]! + if (action.type === `delete`) { + routes.delete(action.id) + continue + } + + const previous = routes.get(action.id) + if ( + previous && + (previous.group !== action.group || + (action.level > 0 && previous.parentGroup !== action.parentGroup)) + ) { + relationshipChanges += 1 + } + routes.set(action.id, { + parentGroup: action.parentGroup, + group: action.group, + }) + } + + return [ + `depth=${depth}`, + `relationship-changes=${ + relationshipChanges === 0 + ? `none` + : relationshipChanges === 1 + ? `one` + : `many` + }`, + `optimistic=${history.some((action) => + action.type.startsWith(`optimistic`), + )}`, + `delete=${history.some((action) => action.type === `delete`)}`, + ] +} + +if (process.env.TANSTACK_DB_ORACLE_STATISTICS === `1`) { + fc.statistics( + scenarioArbitrary, + classifyScenarioCoverage, + oraclePropertyOptions(1_000, `includes.scenario-statistics`), + ) +} + +const materializeScenarioArbitrary: fc.Arbitrary = fc + .boolean() + .chain((sharedIntermediate) => { + const inserts: Array = sharedIntermediate + ? [`root-1`, `root-2`, `middle-1`, `middle-2`, `shared-1`, `leaf-1`] + : [ + `root-1`, + `root-2`, + `middle-1`, + `middle-2`, + `shared-1`, + `shared-2`, + `leaf-1`, + `leaf-2`, + ] + + return fc + .shuffledSubarray(inserts, { + minLength: inserts.length, + maxLength: inserts.length, + }) + .map((insertOrder) => ({ sharedIntermediate, insertOrder })) + }) + +const sharedMaterializeSeed: MaterializeScenario = { + sharedIntermediate: true, + insertOrder: [ + `root-2`, + `middle-2`, + `root-1`, + `middle-1`, + `shared-1`, + `leaf-1`, + ], +} + +const confirmedChildReorderSeed: Scenario = { + depth: 2, + history: [ + { + type: `put`, + level: 0, + id: 0, + parentGroup: 0, + group: 0, + value: 0, + position: 0, + }, + { + type: `put`, + level: 1, + id: 1, + parentGroup: 0, + group: 0, + value: 0, + position: -1, + }, + { + type: `put`, + level: 1, + id: 2, + parentGroup: 0, + group: 0, + value: 0, + position: -1, + }, + { + type: `put`, + level: 1, + id: 1, + parentGroup: 0, + group: 0, + value: 0, + position: 0, + }, + ], +} + +function createControlledCollection( + name: string, + initialData: Array = [], + rowUpdateMode: `partial` | `full` = `partial`, +) { + return createOracleControlledCollection(name, initialData, { + rowUpdateMode, + }) +} + +function compareRows(left: RootRow, right: RootRow): number { + return left.position - right.position || left.id - right.id +} + +// This is intentionally independent of the live-query implementation. It is +// the simple, full-recompute semantics reference for the incremental system. +function recompute( + roots: Map, + levels: Array>, + depth: IncludeDepth, +): Array { + const materializeLevel = ( + level: number, + parentGroup: number, + ): Array => + [...levels[level]!.values()] + .filter((row) => row.parentGroup === parentGroup) + .sort(compareRows) + .map((row) => { + const node: OracleNode = { + id: row.id, + group: row.group, + value: row.value, + position: row.position, + } + if (level + 1 < depth) { + node.children = materializeLevel(level + 1, row.group) + } + return node + }) + + return [...roots.values()].sort(compareRows).map((root) => ({ + id: root.id, + group: root.group, + value: root.value, + position: root.position, + children: materializeLevel(0, root.group), + })) +} + +// Collection delivery metadata is independent of include materialization. +// Compare only the user-defined row shape modeled by the recompute oracle. +function stripVirtualProperties(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map(stripVirtualProperties) + } + if (!value || typeof value !== `object`) { + return value + } + + return Object.fromEntries( + Object.entries(value) + .filter(([key]) => !key.startsWith(`$`)) + .map(([key, entry]) => [key, stripVirtualProperties(entry)]), + ) +} + +type Sources = ReturnType + +function createIncrementalQuery(depth: IncludeDepth, sources: Sources) { + const { roots } = sources + const [level1, level2, level3, level4] = sources.levels + + switch (depth) { + case 1: + return createLiveQueryCollection((q) => + q + .from({ root: roots.collection }) + .orderBy(({ root }) => root.position) + .orderBy(({ root }) => root.id) + .select(({ root }) => ({ + id: root.id, + group: root.group, + value: root.value, + position: root.position, + children: toArray( + q + .from({ child: level1.collection }) + .where(({ child }) => eq(child.parentGroup, root.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + group: child.group, + value: child.value, + position: child.position, + })), + ), + })), + ) + case 2: + return createLiveQueryCollection((q) => + q + .from({ root: roots.collection }) + .orderBy(({ root }) => root.position) + .orderBy(({ root }) => root.id) + .select(({ root }) => ({ + id: root.id, + group: root.group, + value: root.value, + position: root.position, + children: toArray( + q + .from({ child: level1.collection }) + .where(({ child }) => eq(child.parentGroup, root.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + group: child.group, + value: child.value, + position: child.position, + children: toArray( + q + .from({ grandchild: level2.collection }) + .where(({ grandchild }) => + eq(grandchild.parentGroup, child.group), + ) + .orderBy(({ grandchild }) => grandchild.position) + .orderBy(({ grandchild }) => grandchild.id) + .select(({ grandchild }) => ({ + id: grandchild.id, + group: grandchild.group, + value: grandchild.value, + position: grandchild.position, + })), + ), + })), + ), + })), + ) + case 3: + return createLiveQueryCollection((q) => + q + .from({ root: roots.collection }) + .orderBy(({ root }) => root.position) + .orderBy(({ root }) => root.id) + .select(({ root }) => ({ + id: root.id, + group: root.group, + value: root.value, + position: root.position, + children: toArray( + q + .from({ child: level1.collection }) + .where(({ child }) => eq(child.parentGroup, root.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + group: child.group, + value: child.value, + position: child.position, + children: toArray( + q + .from({ grandchild: level2.collection }) + .where(({ grandchild }) => + eq(grandchild.parentGroup, child.group), + ) + .orderBy(({ grandchild }) => grandchild.position) + .orderBy(({ grandchild }) => grandchild.id) + .select(({ grandchild }) => ({ + id: grandchild.id, + group: grandchild.group, + value: grandchild.value, + position: grandchild.position, + children: toArray( + q + .from({ greatGrandchild: level3.collection }) + .where(({ greatGrandchild }) => + eq(greatGrandchild.parentGroup, grandchild.group), + ) + .orderBy( + ({ greatGrandchild }) => greatGrandchild.position, + ) + .orderBy( + ({ greatGrandchild }) => greatGrandchild.id, + ) + .select(({ greatGrandchild }) => ({ + id: greatGrandchild.id, + group: greatGrandchild.group, + value: greatGrandchild.value, + position: greatGrandchild.position, + })), + ), + })), + ), + })), + ), + })), + ) + case 4: + return createLiveQueryCollection((q) => + q + .from({ root: roots.collection }) + .orderBy(({ root }) => root.position) + .orderBy(({ root }) => root.id) + .select(({ root }) => ({ + id: root.id, + group: root.group, + value: root.value, + position: root.position, + children: toArray( + q + .from({ child: level1.collection }) + .where(({ child }) => eq(child.parentGroup, root.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + group: child.group, + value: child.value, + position: child.position, + children: toArray( + q + .from({ grandchild: level2.collection }) + .where(({ grandchild }) => + eq(grandchild.parentGroup, child.group), + ) + .orderBy(({ grandchild }) => grandchild.position) + .orderBy(({ grandchild }) => grandchild.id) + .select(({ grandchild }) => ({ + id: grandchild.id, + group: grandchild.group, + value: grandchild.value, + position: grandchild.position, + children: toArray( + q + .from({ greatGrandchild: level3.collection }) + .where(({ greatGrandchild }) => + eq(greatGrandchild.parentGroup, grandchild.group), + ) + .orderBy( + ({ greatGrandchild }) => greatGrandchild.position, + ) + .orderBy( + ({ greatGrandchild }) => greatGrandchild.id, + ) + .select(({ greatGrandchild }) => ({ + id: greatGrandchild.id, + group: greatGrandchild.group, + value: greatGrandchild.value, + position: greatGrandchild.position, + children: toArray( + q + .from({ finalChild: level4.collection }) + .where(({ finalChild }) => + eq( + finalChild.parentGroup, + greatGrandchild.group, + ), + ) + .orderBy( + ({ finalChild }) => finalChild.position, + ) + .orderBy(({ finalChild }) => finalChild.id) + .select(({ finalChild }) => ({ + id: finalChild.id, + group: finalChild.group, + value: finalChild.value, + position: finalChild.position, + })), + ), + })), + ), + })), + ), + })), + ), + })), + ) + } +} + +function createSources(rowUpdateMode: `partial` | `full` = `partial`) { + return { + roots: createControlledCollection( + `oracle-roots`, + [], + rowUpdateMode, + ), + levels: [ + createControlledCollection(`oracle-level-1`, [], rowUpdateMode), + createControlledCollection(`oracle-level-2`, [], rowUpdateMode), + createControlledCollection(`oracle-level-3`, [], rowUpdateMode), + createControlledCollection(`oracle-level-4`, [], rowUpdateMode), + ] as const, + } +} + +function sameRoot(left: RootRow, right: RootRow): boolean { + return ( + left.group === right.group && + left.value === right.value && + left.position === right.position + ) +} + +function sameChild(left: ChildRow, right: ChildRow): boolean { + return left.parentGroup === right.parentGroup && sameRoot(left, right) +} + +async function settleOptimisticAction( + action: HistoryAction, + resolveSync: () => void, + rejectSync: (error: Error) => void, + persistedPromise: Promise, +): Promise { + if (action.type === `optimisticConfirm`) { + resolveSync() + await persistedPromise + return + } + + const message = `oracle optimistic rollback` + const persisted = persistedPromise.catch(() => undefined) + await withExpectedRejection(message, async () => { + rejectSync(new Error(message)) + await persisted + await flushPromises() + }) +} + +async function applyAction( + action: HistoryAction, + sources: Sources, + roots: Map, + levels: Array>, + assertMatches: TraceCheckpoint, +): Promise { + if (action.level === 0) { + const current = roots.get(action.id) + if (action.type === `delete`) { + if (current) { + sources.roots.write(`delete`, current) + roots.delete(action.id) + } + return + } + + const next: RootRow = { + id: action.id, + group: action.group, + value: action.value, + position: + action.type === `put` ? action.position : (current?.position ?? 0), + } + + if (action.type !== `put` && !current) return + if (action.type !== `put` && current && sameRoot(current, next)) return + + if ( + action.type === `optimisticConfirm` || + action.type === `optimisticRollback` + ) { + const transaction = sources.roots.collection.update( + action.id, + (draft) => { + draft.group = next.group + draft.value = next.value + }, + ) + roots.set(action.id, next) + assertMatches() + + if (action.type === `optimisticConfirm`) { + sources.roots.write(`update`, next) + } + await settleOptimisticAction( + action, + sources.roots.resolveSync, + sources.roots.rejectSync, + transaction.isPersisted.promise, + ) + if (action.type === `optimisticRollback`) { + roots.set(action.id, current!) + assertMatches() + } + return + } + + sources.roots.write(current ? `update` : `insert`, next) + roots.set(action.id, next) + return + } + + const level = action.level - 1 + const model = levels[level]! + const source = sources.levels[level]! + const current = model.get(action.id) + if (action.type === `delete`) { + if (current) { + source.write(`delete`, current) + model.delete(action.id) + } + return + } + + const next: ChildRow = { + id: action.id, + parentGroup: action.parentGroup, + group: action.group, + value: action.value, + position: + action.type === `put` ? action.position : (current?.position ?? 0), + } + + if (action.type !== `put` && !current) return + if (action.type !== `put` && current && sameChild(current, next)) return + + if ( + action.type === `optimisticConfirm` || + action.type === `optimisticRollback` + ) { + const transaction = source.collection.update(action.id, (draft) => { + draft.parentGroup = next.parentGroup + draft.group = next.group + draft.value = next.value + }) + model.set(action.id, next) + assertMatches() + + if (action.type === `optimisticConfirm`) { + source.write(`update`, next) + } + await settleOptimisticAction( + action, + source.resolveSync, + source.rejectSync, + transaction.isPersisted.promise, + ) + if (action.type === `optimisticRollback`) { + model.set(action.id, current!) + assertMatches() + } + return + } + + source.write(current ? `update` : `insert`, next) + model.set(action.id, next) +} + +async function cleanupSources(sources: Sources) { + await Promise.all( + [sources.roots, ...sources.levels].map(({ collection }) => + collection.cleanup(), + ), + ) +} + +type StructuralTraceContext = { + depth: IncludeDepth + sources: Sources + incremental: ReturnType + roots: Map + levels: Array> +} + +function createStructuralTraceContext( + depth: IncludeDepth, + rowUpdateMode: `partial` | `full` = `partial`, +): StructuralTraceContext { + const sources = createSources(rowUpdateMode) + return { + depth, + sources, + incremental: createIncrementalQuery(depth, sources), + roots: new Map(), + levels: Array.from({ length: 4 }, () => new Map()), + } +} + +async function cleanupStructuralTrace({ + incremental, + sources, +}: Pick & { + incremental: { cleanup: () => Promise } +}): Promise { + await incremental.cleanup() + await cleanupSources(sources) +} + +function createStructuralTraceDriver( + depth: IncludeDepth, +): TraceDriver { + return { + setup: () => createStructuralTraceContext(depth), + start: ({ incremental }) => incremental.preload(), + apply: (action, context, checkpoint) => + applyAction( + action, + context.sources, + context.roots, + context.levels, + checkpoint, + ), + cleanup: cleanupStructuralTrace, + } +} + +type FullRowBatchStep = + | { level: 0; changes: Array> } + | { level: IncludeDepth; changes: Array> } + +type FullRowBatchInput = { + level: 0 | IncludeDepth + changes: Array<{ + type: `put` | `delete` + id: number + parentGroup: number + group: number + value: number + position: number + }> +} + +type FullRowBatchScenario = { + depth: IncludeDepth + steps: Array +} + +function updateModel( + model: Map, + changes: ReadonlyArray>, +): void { + for (const change of changes) { + if (change.type === `delete`) { + model.delete(change.value.id) + } else { + model.set(change.value.id, change.value) + } + } +} + +function updateFullRowBatchModels( + step: FullRowBatchStep, + roots: Map, + levels: Array>, +): void { + if (step.level === 0) { + updateModel(roots, step.changes) + } else { + updateModel(levels[step.level - 1]!, step.changes) + } +} + +function createFullRowBatchTraceDriver( + depth: IncludeDepth, +): TraceDriver { + return { + setup: () => createStructuralTraceContext(depth, `full`), + start: ({ incremental }) => incremental.preload(), + apply: (step, { sources, roots, levels }) => { + if (step.level === 0) { + sources.roots.writeBatch(step.changes) + } else { + sources.levels[step.level - 1]!.writeBatch(step.changes) + } + updateFullRowBatchModels(step, roots, levels) + }, + cleanup: cleanupStructuralTrace, + } +} + +function batchRoot( + id: number, + group: number, + value = id * 10, + position = id - 1, +): RootRow { + return { id, group, value, position } +} + +function batchChild( + id: number, + parentGroup: number, + value = id * 10, + position = id - 1, +): ChildRow { + return { id, parentGroup, group: id * 10, value, position } +} + +const fullRowBatchTrace: Array = [ + { + level: 0, + changes: [ + { type: `insert`, value: batchRoot(1, 1) }, + { type: `insert`, value: batchRoot(2, 2) }, + ], + }, + { + level: 1, + changes: [ + { type: `insert`, value: batchChild(1, 1, 10, 0) }, + { type: `insert`, value: batchChild(2, 1, 20, 1) }, + { type: `insert`, value: batchChild(3, 2, 30, 0) }, + ], + }, + { + level: 1, + changes: [ + { type: `update`, value: batchChild(1, 2, 11, 0) }, + { type: `update`, value: batchChild(2, 1, 22, 1) }, + ], + }, + { + level: 1, + changes: [ + { type: `delete`, value: batchChild(3, 2, 30, 0) }, + { type: `insert`, value: batchChild(4, 2, 40, 1) }, + ], + }, +] + +const fullRowSharedRoutingSeed: FullRowBatchScenario = { + depth: 1, + steps: [ + { + level: 0, + changes: [ + { type: `insert`, value: batchRoot(0, 2, 0, 0) }, + { type: `insert`, value: batchRoot(1, 2, 0, 0) }, + ], + }, + { + level: 0, + changes: [{ type: `delete`, value: batchRoot(1, 2, 0, 0) }], + }, + { + level: 0, + changes: [{ type: `insert`, value: batchRoot(1, 0, 0, 0) }], + }, + { + level: 1, + changes: [{ type: `insert`, value: batchChild(0, 2, 0, 0) }], + }, + ], +} + +function fullRowBatchInputArbitrary( + depth: IncludeDepth, + levels: `all` | `children`, + maxBatchSize = 3, +): fc.Arbitrary { + const levelsArbitrary = + levels === `all` + ? levelArbitrary(depth) + : fc.integer({ min: 1, max: depth }).map((level) => level as IncludeDepth) + + return levelsArbitrary.chain((level) => + fc + .uniqueArray( + fc.record({ + // Root delete/reinsert has a known failing seed below. Keep the + // generated green corpus out of that class while still generating + // inserts, replacements, and multi-change batches at the root. + type: + level === 0 + ? fc.constant(`put` as const) + : fc.constantFrom(`put` as const, `delete` as const), + id: fc.integer({ min: 0, max: 5 }), + parentGroup: fc.integer({ min: 0, max: 4 }), + group: fc.integer({ min: 0, max: 4 }), + value: fc.integer({ min: -3, max: 3 }), + position: fc.integer({ min: -2, max: 2 }), + }), + { + selector: (change) => change.id, + minLength: 1, + maxLength: maxBatchSize, + }, + ) + .map((changes) => ({ level, changes })), + ) +} + +function createConnectedBatchPrefix( + depth: IncludeDepth, +): Array { + // Generated ids stop at 5, so this path survives every later batch and + // guarantees that the selected depth is observable at every checkpoint. + const steps: Array = [ + { + level: 0, + changes: [{ type: `insert`, value: batchRoot(100, 100, 100, 0) }], + }, + ] + + for (let level = 1; level <= depth; level++) { + steps.push({ + level: level as IncludeDepth, + changes: [ + { + type: `insert`, + value: { + ...batchChild(100 + level, 99 + level, 100 + level, 0), + group: 100 + level, + }, + }, + ], + }) + } + + return steps +} + +type ConnectedBranch = { + idBase: number + groupBase: number +} + +function createConnectedBatchBranches( + depth: IncludeDepth, + branches: ReadonlyArray = [ + { idBase: 100, groupBase: 100 }, + { idBase: 200, groupBase: 200 }, + ], +): Array { + const steps: Array = [ + { + level: 0, + changes: branches.map(({ idBase, groupBase }) => ({ + type: `insert`, + value: batchRoot(idBase, groupBase, idBase, 0), + })), + }, + ] + + for (let level = 1; level <= depth; level++) { + steps.push({ + level: level as IncludeDepth, + changes: branches.map(({ idBase, groupBase }) => ({ + type: `insert`, + value: { + ...batchChild( + idBase + level, + groupBase + level - 1, + idBase + level, + 0, + ), + group: groupBase + level, + }, + })), + }) + } + + return steps +} + +function normalizeFullRowBatchInputs( + depth: IncludeDepth, + inputs: Array, +): FullRowBatchScenario { + const roots = new Map([[100, batchRoot(100, 100, 100, 0)]]) + const levels = Array.from( + { length: 4 }, + (_, level) => + new Map( + level < depth + ? [ + [ + 101 + level, + { + ...batchChild(101 + level, 100 + level, 101 + level, 0), + group: 101 + level, + }, + ], + ] + : [], + ), + ) + const steps = createConnectedBatchPrefix(depth) + + for (const input of inputs) { + if (input.level === 0) { + const changes = input.changes.map((change): SyncChange => { + const current = roots.get(change.id) + if (change.type === `delete` && current) { + roots.delete(change.id) + return { type: `delete`, value: current } + } + + const value: RootRow = { + id: change.id, + group: current ? current.group : change.group, + value: change.value, + position: change.position, + } + roots.set(value.id, value) + return { type: current ? `update` : `insert`, value } + }) + steps.push({ level: 0, changes }) + continue + } + + const model = levels[input.level - 1]! + const changes = input.changes.map((change): SyncChange => { + const current = model.get(change.id) + if (change.type === `delete` && current) { + model.delete(change.id) + return { type: `delete`, value: current } + } + + const value: ChildRow = { + id: change.id, + parentGroup: current ? current.parentGroup : change.parentGroup, + group: current ? current.group : change.group, + value: change.value, + position: change.position, + } + model.set(value.id, value) + return { type: current ? `update` : `insert`, value } + }) + steps.push({ level: input.level, changes }) + } + + return { depth, steps } +} + +function fullRowBatchScenarioAtDepthArbitrary( + depth: IncludeDepth, +): fc.Arbitrary { + return fc + .array(fullRowBatchInputArbitrary(depth, `all`), { + minLength: 1, + maxLength: 10, + }) + .map((inputs) => { + const noise = normalizeFullRowBatchInputs(depth, inputs).steps.slice( + depth + 1, + ) + const changes: Array> = [100, 200].map((rootId) => ({ + type: `update`, + value: { + ...batchChild( + rootId + depth, + rootId + depth - 1, + rootId + depth + 1, + 0, + ), + group: rootId + depth, + }, + })) + + return { + depth, + steps: [ + ...createConnectedBatchBranches(depth), + ...noise, + { level: depth, changes }, + ], + } + }) +} + +type VisibleRelationshipTransition = `reparent` | `rekey` +type BranchDeliveryOrder = `forward` | `reverse` +const branchDeliveryOrders: ReadonlyArray = [ + `forward`, + `reverse`, +] + +function otherBranch(branch: 0 | 1): 0 | 1 { + return branch === 0 ? 1 : 0 +} + +function deliverBranches( + branches: readonly [ConnectedBranch, ConnectedBranch], + order: BranchDeliveryOrder, +): readonly [ConnectedBranch, ConnectedBranch] { + return order === `forward` ? branches : [branches[1], branches[0]] +} + +function deliveredBranchIndex( + branch: 0 | 1, + order: BranchDeliveryOrder, +): 0 | 1 { + return order === `forward` ? branch : otherBranch(branch) +} + +type VisibleRelationshipScenario = FullRowBatchScenario & { + transitionStepIndex: number +} + +type VisibleRelationshipScenarios = { + transitionOnly: VisibleRelationshipScenario + stateful: VisibleRelationshipScenario +} + +type VisibleScalarNoise = { + side: `before` | `after` + level: 0 | IncludeDepth + branch: 0 | 1 + value: number + position: number +} + +type VisibleRelationshipScenarioOptions = { + depth: IncludeDepth + targetLevel: IncludeDepth + sourceBranch: 0 | 1 + branches: readonly [ConnectedBranch, ConnectedBranch] + noise: ReadonlyArray +} & ( + | { transition: `reparent`; rekeyGroup?: never } + | { transition: `rekey`; rekeyGroup: number } +) + +function assertDisjointRelationshipKeys( + depth: IncludeDepth, + branches: readonly [ConnectedBranch, ConnectedBranch], + { + extraIds = [], + extraGroups = [], + }: { + extraIds?: ReadonlyArray + extraGroups?: ReadonlyArray + } = {}, +): void { + const ids = [ + ...branches.flatMap(({ idBase }) => + Array.from({ length: depth + 1 }, (_, level) => idBase + level), + ), + ...extraIds, + ] + const groups = [ + ...branches.flatMap(({ groupBase }) => + Array.from({ length: depth + 1 }, (_, level) => groupBase + level), + ), + ...extraGroups, + ] + if ( + new Set(ids).size !== ids.length || + new Set(groups).size !== groups.length + ) { + throw new Error(`Visible relationship keys overlap`) + } +} + +function createVisibleRelationshipScenario( + options: VisibleRelationshipScenarioOptions, +): VisibleRelationshipScenario { + const { depth, transition, targetLevel, sourceBranch, branches, noise } = + options + assertDisjointRelationshipKeys( + depth, + branches, + transition === `rekey` ? { extraGroups: [options.rekeyGroup] } : {}, + ) + const steps = createConnectedBatchBranches(depth, branches) + const roots = new Map() + const levels = Array.from({ length: 4 }, () => new Map()) + + for (const step of steps) { + updateFullRowBatchModels(step, roots, levels) + } + + const appendNoise = (entry: VisibleScalarNoise): void => { + const branch = branches[entry.branch] + if (entry.level === 0) { + const current = roots.get(branch.idBase)! + const value = { + ...current, + value: entry.value, + position: entry.position, + } + roots.set(value.id, value) + steps.push({ level: 0, changes: [{ type: `update`, value }] }) + return + } + + const model = levels[entry.level - 1]! + const current = model.get(branch.idBase + entry.level)! + const value = { + ...current, + value: entry.value, + position: entry.position, + } + model.set(value.id, value) + steps.push({ + level: entry.level, + changes: [{ type: `update`, value }], + }) + } + + for (const entry of noise.filter(({ side }) => side === `before`)) { + appendNoise(entry) + } + + const source = branches[sourceBranch] + const destination = branches[otherBranch(sourceBranch)] + const targetModel = levels[targetLevel - 1]! + const current = targetModel.get(source.idBase + targetLevel)! + const value: ChildRow = { + ...current, + parentGroup: + transition === `reparent` + ? destination.groupBase + targetLevel - 1 + : current.parentGroup, + group: transition === `rekey` ? options.rekeyGroup : current.group, + } + const transitionStepIndex = steps.length + steps.push({ + level: targetLevel, + changes: [{ type: `update`, value }], + }) + targetModel.set(value.id, value) + + for (const entry of noise.filter(({ side }) => side === `after`)) { + appendNoise(entry) + } + + return { + depth, + steps, + transitionStepIndex, + } +} + +function visibleRelationshipScenarioArbitrary( + depth: IncludeDepth, + transition: VisibleRelationshipTransition, + targetLevel: IncludeDepth, +): fc.Arbitrary { + const branchArbitrary = fc.constantFrom<0 | 1>(0, 1) + const scalarNoiseArbitrary = ( + side: VisibleScalarNoise[`side`], + branch: fc.Arbitrary<0 | 1>, + ): fc.Arbitrary => + fc.record({ + side: fc.constant(side), + level: levelArbitrary(depth), + branch, + value: fc.integer({ min: -3, max: 3 }), + position: fc.integer({ min: -2, max: 2 }), + }) + + return fc + .record({ + sourceBranch: branchArbitrary, + leftIdBase: fc.integer({ min: 100, max: 500 }), + leftGroupBase: fc.integer({ min: 600, max: 1_000 }), + rightIdBase: fc.integer({ min: 1_100, max: 1_500 }), + rightGroupBase: fc.integer({ min: 1_600, max: 2_000 }), + rekeyGroup: fc.integer({ min: 2_100, max: 2_500 }), + beforeNoise: scalarNoiseArbitrary(`before`, branchArbitrary), + extraBeforeNoise: fc.array( + scalarNoiseArbitrary(`before`, branchArbitrary), + { maxLength: 4 }, + ), + afterValues: fc.array(scalarNoiseArbitrary(`after`, branchArbitrary), { + minLength: 1, + maxLength: 5, + }), + }) + .map( + ({ + sourceBranch, + leftIdBase, + leftGroupBase, + rightIdBase, + rightGroupBase, + rekeyGroup, + beforeNoise, + extraBeforeNoise, + afterValues, + }) => { + const connectedNoise: Array = [ + beforeNoise, + ...extraBeforeNoise, + ...afterValues, + ] + const options = { + depth, + targetLevel, + sourceBranch, + branches: [ + { idBase: leftIdBase, groupBase: leftGroupBase }, + { idBase: rightIdBase, groupBase: rightGroupBase }, + ], + ...(transition === `rekey` + ? { transition, rekeyGroup } + : { transition }), + } satisfies Omit + + return { + transitionOnly: createVisibleRelationshipScenario({ + ...options, + noise: [], + }), + stateful: createVisibleRelationshipScenario({ + ...options, + noise: connectedNoise, + }), + } + }, + ) +} + +type GeneratedBranchOptions = { + leftIdBase: number + leftGroupBase: number + rightIdBase: number + rightGroupBase: number +} + +const generatedBranchArbitraries = { + leftIdBase: fc.integer({ min: 100, max: 500 }), + leftGroupBase: fc.integer({ min: 600, max: 1_000 }), + rightIdBase: fc.integer({ min: 1_100, max: 1_500 }), + rightGroupBase: fc.integer({ min: 1_600, max: 2_000 }), +} + +function createGeneratedBranches({ + leftIdBase, + leftGroupBase, + rightIdBase, + rightGroupBase, +}: GeneratedBranchOptions): readonly [ConnectedBranch, ConnectedBranch] { + return [ + { idBase: leftIdBase, groupBase: leftGroupBase }, + { idBase: rightIdBase, groupBase: rightGroupBase }, + ] +} + +type TransitionHistoryScenario = FullRowBatchScenario & { + historyStartStepIndex: number +} + +type TransitionHistoryScenarioOptions = { + depth: IncludeDepth + targetLevel: IncludeDepth + firstTransition: VisibleRelationshipTransition + secondTransition: VisibleRelationshipTransition + sourceBranch: 0 | 1 + branches: readonly [ConnectedBranch, ConnectedBranch] + rekeyGroups: readonly [number, number] + insertedLevels: readonly [IncludeDepth, IncludeDepth] + insertedValues: readonly [number, number] + insertedPositions: readonly [number, number] +} + +function createTransitionHistoryScenario({ + depth, + targetLevel, + firstTransition, + secondTransition, + sourceBranch, + branches, + rekeyGroups, + insertedLevels, + insertedValues, + insertedPositions, +}: TransitionHistoryScenarioOptions): TransitionHistoryScenario { + // Fresh keys keep this matrix green through both transitions. Separate + // state-aware families below reuse retired routes and replace existing rows, + // so those histories remain shrinkable without masking later steps. + const insertedRows = [ + { + id: 3_000, + group: 2_700, + value: insertedValues[0], + position: insertedPositions[0], + }, + { + id: 3_001, + group: 2_800, + value: insertedValues[1], + position: insertedPositions[1], + }, + ] as const + assertDisjointRelationshipKeys(depth, branches, { + extraIds: insertedRows.map(({ id }) => id), + extraGroups: [...rekeyGroups, ...insertedRows.map(({ group }) => group)], + }) + + const steps = createConnectedBatchBranches(depth, branches) + const historyStartStepIndex = steps.length + const source = branches[sourceBranch] + let current = { + ...batchChild( + source.idBase + targetLevel, + source.groupBase + targetLevel - 1, + source.idBase + targetLevel, + 0, + ), + group: source.groupBase + targetLevel, + } + const appendStep = ( + level: IncludeDepth, + changes: Array>, + ): void => { + steps.push({ level, changes }) + } + const insertRow = ( + row: (typeof insertedRows)[number], + level: IncludeDepth, + ): ChildRow => { + if (level !== targetLevel && level !== targetLevel + 1) { + throw new Error(`History inserts must touch the target or its child`) + } + return { + ...row, + parentGroup: level === targetLevel ? current.parentGroup : current.group, + } + } + const transition = ( + kind: VisibleRelationshipTransition, + rekeyGroup: number, + ): void => { + const currentParentBranch = branches.findIndex( + ({ groupBase }) => current.parentGroup === groupBase + targetLevel - 1, + ) + if (currentParentBranch !== 0 && currentParentBranch !== 1) { + throw new Error(`Transition target has no visible parent branch`) + } + const destination = branches[otherBranch(currentParentBranch)] + current = { + ...current, + parentGroup: + kind === `reparent` + ? destination.groupBase + targetLevel - 1 + : current.parentGroup, + group: kind === `rekey` ? rekeyGroup : current.group, + } + appendStep(targetLevel, [{ type: `update`, value: current }]) + } + const interleaveTransition = ( + kind: VisibleRelationshipTransition, + rekeyGroup: number, + row: (typeof insertedRows)[number], + level: IncludeDepth, + ): void => { + if (kind === `rekey`) { + if (level !== targetLevel + 1) { + throw new Error(`Rekey histories must seed the new child route`) + } + transition(kind, rekeyGroup) + appendStep(level, [{ type: `insert`, value: insertRow(row, level) }]) + return + } + + const inserted = insertRow(row, level) + appendStep(level, [{ type: `insert`, value: inserted }]) + transition(kind, rekeyGroup) + appendStep(level, [{ type: `delete`, value: inserted }]) + } + + interleaveTransition( + firstTransition, + rekeyGroups[0], + insertedRows[0], + insertedLevels[0], + ) + interleaveTransition( + secondTransition, + rekeyGroups[1], + insertedRows[1], + insertedLevels[1], + ) + + return { + depth, + steps, + historyStartStepIndex, + } +} + +function transitionHistoryPlacements( + depth: IncludeDepth, + firstTransition: VisibleRelationshipTransition, + secondTransition: VisibleRelationshipTransition, +): Array<{ + targetLevel: IncludeDepth + insertedLevels: readonly [IncludeDepth, IncludeDepth] +}> { + // A rekey needs one child level so it changes relationship membership. It + // also fails when two descendant levels remain. The single-transition matrix + // pins that failure and will turn red when this continuation can be widened. + const minimumTargetLevel = + firstTransition === `rekey` || secondTransition === `rekey` + ? Math.max(1, depth - 1) + : 1 + const maximumTargetLevel = + firstTransition === `rekey` || secondTransition === `rekey` + ? depth - 1 + : depth + const placements: Array<{ + targetLevel: IncludeDepth + insertedLevels: readonly [IncludeDepth, IncludeDepth] + }> = [] + + for (let level = minimumTargetLevel; level <= maximumTargetLevel; level++) { + const targetLevel = level as IncludeDepth + const insertedLevels = ( + transition: VisibleRelationshipTransition, + ): ReadonlyArray => + transition === `rekey` + ? [(targetLevel + 1) as IncludeDepth] + : targetLevel < depth + ? [targetLevel, (targetLevel + 1) as IncludeDepth] + : [targetLevel] + + for (const firstInsertedLevel of insertedLevels(firstTransition)) { + for (const secondInsertedLevel of insertedLevels(secondTransition)) { + placements.push({ + targetLevel, + insertedLevels: [firstInsertedLevel, secondInsertedLevel], + }) + } + } + } + + return placements +} + +function transitionHistoryScenariosArbitrary( + depth: IncludeDepth, + firstTransition: VisibleRelationshipTransition, + secondTransition: VisibleRelationshipTransition, + sourceBranch: 0 | 1, +): fc.Arbitrary> { + const placements = transitionHistoryPlacements( + depth, + firstTransition, + secondTransition, + ) + + return fc + .record({ + ...generatedBranchArbitraries, + firstRekeyGroup: fc.integer({ min: 2_100, max: 2_300 }), + secondRekeyGroup: fc.integer({ min: 2_400, max: 2_600 }), + firstInsertedValue: fc.integer({ min: -3, max: 3 }), + secondInsertedValue: fc.integer({ min: -3, max: 3 }), + firstInsertedPosition: fc.integer({ min: -2, max: 2 }), + secondInsertedPosition: fc.integer({ min: -2, max: 2 }), + }) + .map( + ({ + leftIdBase, + leftGroupBase, + rightIdBase, + rightGroupBase, + firstRekeyGroup, + secondRekeyGroup, + firstInsertedValue, + secondInsertedValue, + firstInsertedPosition, + secondInsertedPosition, + }) => + placements.map(({ targetLevel, insertedLevels }) => + createTransitionHistoryScenario({ + depth, + targetLevel, + firstTransition, + secondTransition, + sourceBranch, + branches: createGeneratedBranches({ + leftIdBase, + leftGroupBase, + rightIdBase, + rightGroupBase, + }), + rekeyGroups: [firstRekeyGroup, secondRekeyGroup], + insertedLevels, + insertedValues: [firstInsertedValue, secondInsertedValue], + insertedPositions: [firstInsertedPosition, secondInsertedPosition], + }), + ), + ) +} + +function expectEveryHistoryStepVisible( + scenario: TransitionHistoryScenario, +): void { + for ( + let stepIndex = scenario.historyStartStepIndex; + stepIndex < scenario.steps.length; + stepIndex++ + ) { + const before = relationshipOnly( + recomputeFullRowBatchScenario(scenario, stepIndex), + ) + const after = relationshipOnly( + recomputeFullRowBatchScenario(scenario, stepIndex + 1), + ) + expect(after).not.toEqual(before) + } +} + +type RouteDestination = { + strategy: `fresh` | `restore` | `merge` | `split` | `retired` + route: number +} + +type RouteTransitionDescriptor = { + row: 0 | 1 + stepsBefore?: ReadonlyArray +} & ( + | { + kind: `reparent` + level: IncludeDepth + destination: { strategy: `merge`; route: number } + } + | { + kind: `rekey` + level: 0 | IncludeDepth + destination: RouteDestination + } +) + +type RouteLifecycleScenario = FullRowBatchScenario & { + transitionStepIndexes: ReadonlyArray +} + +type RouteLifecycleScenarioOptions = { + depth: IncludeDepth + branches: readonly [ConnectedBranch, ConnectedBranch] + descriptors: ReadonlyArray + prefixSteps?: ReadonlyArray + trailingSteps?: ReadonlyArray +} + +function rowAt( + branches: readonly [ConnectedBranch, ConnectedBranch], + row: 0 | 1, + level: 0 | IncludeDepth, +): number { + return branches[row].idBase + level +} + +function applyRouteLifecycleStep( + step: FullRowBatchStep, + roots: Map, + levels: Array>, + seenRoutes: Set, + retiredRouteOwners: Map, +): void { + const rows = step.level === 0 ? roots : levels[step.level - 1]! + const previousRouteOwners = new Map>() + + for (const change of step.changes) { + const previous = rows.get(change.value.id) + if (!previous) continue + const owners = previousRouteOwners.get(previous.group) ?? new Set() + owners.add(previous.id) + previousRouteOwners.set(previous.group, owners) + } + + updateFullRowBatchModels(step, roots, levels) + + for (const row of rows.values()) { + const route = routeIdentity(step.level, row.group) + seenRoutes.add(route) + retiredRouteOwners.delete(route) + } + for (const [route, previousOwners] of previousRouteOwners) { + if ([...rows.values()].some((row) => row.group === route)) continue + const retiredRoute = routeIdentity(step.level, route) + if (previousOwners.size === 1) { + retiredRouteOwners.set(retiredRoute, [...previousOwners][0]!) + } else { + retiredRouteOwners.delete(retiredRoute) + } + } +} + +function routeIdentity(level: 0 | IncludeDepth, route: number): string { + // This oracle has one include edge per level, so level plus correlation value + // is the complete route identity. Equal values at different levels are not + // the same subscription lifecycle. + return `${level}:${route}` +} + +function createRouteLifecycleScenario({ + depth, + branches, + descriptors, + prefixSteps = createConnectedBatchBranches(depth, branches), + trailingSteps = [], +}: RouteLifecycleScenarioOptions): RouteLifecycleScenario { + const seenRoutes = new Set() + const retiredRouteOwners = new Map() + + const steps = [...prefixSteps] + const roots = new Map() + const levels = Array.from({ length: 4 }, () => new Map()) + for (const step of steps) { + applyRouteLifecycleStep(step, roots, levels, seenRoutes, retiredRouteOwners) + } + + const transitionStepIndexes: Array = [] + for (const descriptor of descriptors) { + const destination = descriptor.destination as RouteDestination + if (descriptor.kind === `reparent` && destination.strategy !== `merge`) { + throw new Error( + `reparent transitions only support live merge destinations`, + ) + } + for (const step of descriptor.stepsBefore ?? []) { + steps.push(step) + applyRouteLifecycleStep( + step, + roots, + levels, + seenRoutes, + retiredRouteOwners, + ) + } + + const id = rowAt(branches, descriptor.row, descriptor.level) + const rowsAtLevel = + descriptor.level === 0 + ? [...roots.values()] + : [...levels[descriptor.level - 1]!.values()] + const current = rowsAtLevel.find((row) => row.id === id) + if (!current) throw new Error(`Missing transition row ${id}`) + const currentRoute = + descriptor.kind === `rekey` + ? current.group + : (current as ChildRow).parentGroup + const destinationRows = + descriptor.kind === `rekey` + ? rowsAtLevel + : descriptor.level === 1 + ? [...roots.values()] + : [...levels[descriptor.level - 2]!.values()] + const destinationIsLive = destinationRows.some( + (row) => row.group === descriptor.destination.route, + ) + const currentRouteUsers = rowsAtLevel.filter((row) => + descriptor.kind === `rekey` + ? row.group === currentRoute + : (row as ChildRow).parentGroup === currentRoute, + ).length + const destinationRoute = routeIdentity( + descriptor.level, + descriptor.destination.route, + ) + const retiredOwner = retiredRouteOwners.get(destinationRoute) + + switch (descriptor.destination.strategy) { + case `fresh`: + if (seenRoutes.has(destinationRoute)) { + throw new Error(`fresh route must never have been used`) + } + break + case `restore`: + if (destinationIsLive || retiredOwner !== id) { + throw new Error(`restore route must have been retired by this row`) + } + break + case `merge`: + if ( + !destinationIsLive || + descriptor.destination.route === currentRoute + ) { + throw new Error(`merge route must be live and different`) + } + break + case `split`: + if (seenRoutes.has(destinationRoute)) { + throw new Error(`split destination must be unused`) + } + if (currentRouteUsers < 2) { + throw new Error(`split source route must be shared`) + } + break + case `retired`: + if ( + destinationIsLive || + retiredOwner === undefined || + retiredOwner === id + ) { + throw new Error(`retired route must have been retired by another row`) + } + break + } + + let step: FullRowBatchStep + if (descriptor.level === 0) { + const root = roots.get(id) + if (!root) throw new Error(`Missing transition root ${id}`) + step = { + level: 0, + changes: [ + { + type: `update`, + value: { ...root, group: descriptor.destination.route }, + }, + ], + } + } else { + const child = levels[descriptor.level - 1]!.get(id) + if (!child) throw new Error(`Missing transition child ${id}`) + step = { + level: descriptor.level, + changes: [ + { + type: `update`, + value: + descriptor.kind === `rekey` + ? { ...child, group: descriptor.destination.route } + : { + ...child, + parentGroup: descriptor.destination.route, + }, + }, + ], + } + } + transitionStepIndexes.push(steps.length) + steps.push(step) + applyRouteLifecycleStep(step, roots, levels, seenRoutes, retiredRouteOwners) + } + + steps.push(...trailingSteps) + return { depth, steps, transitionStepIndexes } +} + +function expectEveryRouteTransitionVisible( + scenario: RouteLifecycleScenario, +): void { + for (const stepIndex of scenario.transitionStepIndexes) { + const before = relationshipOnly( + recomputeFullRowBatchScenario(scenario, stepIndex), + ) + const after = relationshipOnly( + recomputeFullRowBatchScenario(scenario, stepIndex + 1), + ) + expect(after).not.toEqual(before) + } +} + +const independentTransitionShapes = [ + `ancestor-descendant`, + `descendant-ancestor`, + `sibling`, + `cross-branch`, + `root`, +] as const + +type IndependentTransitionShape = (typeof independentTransitionShapes)[number] + +function independentFreshRoutesArbitrary( + shape: IndependentTransitionShape, +): fc.Arbitrary { + const first = fc.integer({ min: 2_100, max: 2_400 }) + if (shape === `sibling`) { + return fc.tuple(first, fc.integer({ min: 2_500, max: 2_800 })) + } + if (shape === `ancestor-descendant` || shape === `root`) { + return first.map((route) => [route, 2_500] as const) + } + return fc.constant([2_100, 2_500] as const) +} + +function independentTransitionDescriptors( + shape: IndependentTransitionShape, + branches: readonly [ConnectedBranch, ConnectedBranch], + freshRoutes: readonly [number, number], +): ReadonlyArray { + switch (shape) { + case `ancestor-descendant`: + return [ + { + kind: `reparent`, + level: 1, + row: 0, + destination: { + strategy: `merge`, + route: branches[1].groupBase, + }, + }, + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `fresh`, route: freshRoutes[0] }, + }, + ] + case `descendant-ancestor`: + return [ + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `fresh`, route: freshRoutes[0] }, + }, + { + kind: `reparent`, + level: 1, + row: 0, + destination: { + strategy: `merge`, + route: branches[1].groupBase, + }, + }, + ] + case `sibling`: + return [ + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `fresh`, route: freshRoutes[0] }, + }, + { + kind: `rekey`, + level: 2, + row: 1, + destination: { strategy: `fresh`, route: freshRoutes[1] }, + }, + ] + case `cross-branch`: + return [ + { + kind: `reparent`, + level: 2, + row: 0, + destination: { + strategy: `merge`, + route: branches[1].groupBase + 1, + }, + }, + { + kind: `reparent`, + level: 2, + row: 1, + destination: { + strategy: `merge`, + route: branches[0].groupBase + 1, + }, + }, + ] + case `root`: + return [ + { + kind: `rekey`, + level: 0, + row: 0, + destination: { strategy: `fresh`, route: freshRoutes[0] }, + }, + { + kind: `reparent`, + level: 1, + row: 1, + destination: { strategy: `merge`, route: freshRoutes[0] }, + }, + ] + } +} + +function independentTransitionPrefix( + shape: IndependentTransitionShape, + branches: readonly [ConnectedBranch, ConnectedBranch], +): Array { + const prefix = createConnectedBatchBranches(3, branches) + if (shape !== `sibling`) return prefix + + const secondSiblingId = rowAt(branches, 1, 2) + return prefix.map((step) => + step.level === 2 + ? { + ...step, + changes: step.changes.map((change) => + change.value.id === secondSiblingId + ? { + ...change, + value: { + ...change.value, + parentGroup: branches[0].groupBase + 1, + }, + } + : change, + ), + } + : step, + ) +} + +function independentTransitionScenarioArbitrary( + shape: IndependentTransitionShape, +): fc.Arbitrary { + return fc + .record({ + ...generatedBranchArbitraries, + freshRoutes: independentFreshRoutesArbitrary(shape), + }) + .map(({ freshRoutes, ...branchOptions }) => { + const branches = createGeneratedBranches(branchOptions) + return createRouteLifecycleScenario({ + depth: 3, + branches, + prefixSteps: independentTransitionPrefix(shape, branches), + descriptors: independentTransitionDescriptors( + shape, + branches, + freshRoutes, + ), + }) + }) +} + +const destinationHistories = [ + `fresh`, + `restore`, + `merge-split`, + `retired`, +] as const + +type DestinationHistory = (typeof destinationHistories)[number] + +function destinationHistoryDescriptors( + history: DestinationHistory, + branches: readonly [ConnectedBranch, ConnectedBranch], + freshRoute: number, +): ReadonlyArray { + const originalRoute = branches[0].groupBase + 2 + const sharedRoute = branches[1].groupBase + 2 + const first: RouteTransitionDescriptor = { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `fresh`, route: freshRoute }, + } + + switch (history) { + case `fresh`: + return [first] + case `restore`: + return [ + first, + { + ...first, + destination: { strategy: `restore`, route: originalRoute }, + }, + ] + case `merge-split`: + return [ + { + ...first, + destination: { strategy: `merge`, route: sharedRoute }, + }, + { + ...first, + destination: { strategy: `split`, route: freshRoute }, + }, + ] + case `retired`: + return [ + first, + { + kind: `rekey`, + level: 2, + row: 1, + destination: { strategy: `retired`, route: originalRoute }, + }, + ] + } +} + +function destinationHistoryScenarioArbitrary( + history: DestinationHistory, +): fc.Arbitrary { + return fc + .record({ + ...generatedBranchArbitraries, + freshRoute: fc.integer({ min: 2_100, max: 2_400 }), + }) + .map(({ freshRoute, ...branchOptions }) => { + const branches = createGeneratedBranches(branchOptions) + return createRouteLifecycleScenario({ + depth: 3, + branches, + descriptors: destinationHistoryDescriptors( + history, + branches, + freshRoute, + ), + }) + }) +} + +function createInitiallySharedRoutePrefix( + depth: IncludeDepth, + parentLevel: 0 | 1 | 2, + branches: readonly [ConnectedBranch, ConnectedBranch], + enteringRow: 0 | 1 = 1, +): Array { + const enteringId = rowAt(branches, enteringRow, parentLevel) + const sharedRoute = branches[otherBranch(enteringRow)].groupBase + parentLevel + return createConnectedBatchBranches(depth, branches).map((step) => { + if (step.level !== parentLevel) return step + + if (step.level === 0) { + return { + level: 0, + changes: step.changes.map((change) => + change.value.id === enteringId + ? { ...change, value: { ...change.value, group: sharedRoute } } + : change, + ), + } + } + + return { + level: step.level, + changes: step.changes.map((change) => + change.value.id === enteringId + ? { ...change, value: { ...change.value, group: sharedRoute } } + : change, + ), + } + }) +} + +function createMergeIntoSharedRouteScenarios( + parentLevel: 0 | 1 | 2, + enteringRow: 0 | 1, +): HistoryScenarioPair { + const depth = (parentLevel + 1) as IncludeDepth + const branches = transitionHistoryBranches + const existingRow = otherBranch(enteringRow) + const sharedRoute = branches[existingRow].groupBase + parentLevel + const candidate = createRouteLifecycleScenario({ + depth, + branches, + descriptors: [ + { + kind: `rekey`, + level: parentLevel, + row: enteringRow, + destination: { strategy: `merge`, route: sharedRoute }, + }, + ], + }) + expectEveryRouteTransitionVisible(candidate) + + return { + control: { + depth, + steps: createInitiallySharedRoutePrefix( + depth, + parentLevel, + branches, + enteringRow, + ), + }, + candidate, + } +} + +function createSharedRouteLastSubscriberScenario( + parentLevel: 0 | 1 | 2, +): FullRowBatchScenario { + const depth = (parentLevel + 1) as IncludeDepth + const branches = transitionHistoryBranches + const childLevel = depth + const sharedRoute = branches[0].groupBase + parentLevel + const childId = rowAt(branches, 0, childLevel) + const child = { + ...batchChild(childId, sharedRoute, childId, 0), + group: branches[0].groupBase + childLevel, + } + const descriptors: ReadonlyArray = [ + { + kind: `rekey`, + level: parentLevel, + row: 0, + destination: { strategy: `split`, route: 2_100 + parentLevel }, + }, + { + kind: `rekey`, + level: parentLevel, + row: 1, + destination: { strategy: `fresh`, route: 2_200 + parentLevel }, + }, + ] + const scenario = createRouteLifecycleScenario({ + depth, + branches, + descriptors, + prefixSteps: createInitiallySharedRoutePrefix(depth, parentLevel, branches), + trailingSteps: [ + { + level: childLevel, + changes: [ + { type: `update`, value: { ...child, value: child.value + 1 } }, + ], + }, + ], + }) + expectEveryRouteTransitionVisible(scenario) + return scenario +} + +function createSnapshotOnResubscribeScenarios( + parentLevel: 0 | 1 | 2, +): HistoryScenarioPair { + const depth = (parentLevel + 1) as IncludeDepth + const branches = transitionHistoryBranches + const childLevel = depth + const originalRoute = branches[0].groupBase + parentLevel + const childId = rowAt(branches, 0, childLevel) + const child = { + ...batchChild(childId, originalRoute, childId, 0), + group: branches[0].groupBase + childLevel, + } + const updatedChild = { ...child, value: child.value + 1 } + const prefix = createConnectedBatchBranches(depth, branches) + const childUpdate: FullRowBatchStep = { + level: childLevel, + changes: [{ type: `update`, value: updatedChild }], + } + const candidate = createRouteLifecycleScenario({ + depth, + branches, + descriptors: [ + { + kind: `rekey`, + level: parentLevel, + row: 0, + destination: { strategy: `fresh`, route: 2_300 + parentLevel }, + }, + { + kind: `rekey`, + level: parentLevel, + row: 0, + destination: { strategy: `restore`, route: originalRoute }, + stepsBefore: [childUpdate], + }, + ], + }) + expectEveryRouteTransitionVisible(candidate) + const control: FullRowBatchScenario = { + depth, + steps: [...prefix, childUpdate], + } + + return { + control, + candidate, + } +} + +function createInitiallySharedRouteResubscribeScenario( + parentLevel: 0 | 1 | 2, +): HistoryScenarioPair { + const depth = (parentLevel + 1) as IncludeDepth + const branches = transitionHistoryBranches + const childLevel = depth + const sharedRoute = branches[0].groupBase + parentLevel + const childId = rowAt(branches, 0, childLevel) + const child = { + ...batchChild(childId, sharedRoute, childId, 0), + group: branches[0].groupBase + childLevel, + } + const scenario = createRouteLifecycleScenario({ + depth, + branches, + prefixSteps: createInitiallySharedRoutePrefix(depth, parentLevel, branches), + descriptors: [ + { + kind: `rekey`, + level: parentLevel, + row: 0, + destination: { strategy: `split`, route: 2_100 + parentLevel }, + }, + { + kind: `rekey`, + level: parentLevel, + row: 1, + destination: { strategy: `fresh`, route: 2_200 + parentLevel }, + }, + { + kind: `rekey`, + level: parentLevel, + row: 1, + destination: { strategy: `restore`, route: sharedRoute }, + stepsBefore: [ + { + level: childLevel, + changes: [ + { + type: `update`, + value: { ...child, value: child.value + 1 }, + }, + ], + }, + ], + }, + ], + }) + expectEveryRouteTransitionVisible(scenario) + return { + control: createSharedRouteLastSubscriberScenario(parentLevel), + candidate: scenario, + } +} + +type HistoryScenarioPair = { + control: FullRowBatchScenario + greenVariants?: ReadonlyArray + candidate: FullRowBatchScenario +} + +type RekeyRouteReuseOptions = { + depth: 2 | 3 | 4 + sourceBranch: 0 | 1 + branches: readonly [ConnectedBranch, ConnectedBranch] + rekeyGroup: number + insertedId: number + insertedValue: number + insertedPosition: number +} + +function createRekeyRouteReuseFixture({ + depth, + sourceBranch, + branches, + rekeyGroup, + insertedId, + insertedValue, + insertedPosition, +}: RekeyRouteReuseOptions): { + prefix: Array + rekey: FullRowBatchStep + reuse: FullRowBatchStep +} { + const targetLevel = (depth - 1) as IncludeDepth + assertDisjointRelationshipKeys(depth, branches, { + extraIds: [insertedId], + extraGroups: [rekeyGroup], + }) + + const prefix = createConnectedBatchBranches(depth, branches) + const source = branches[sourceBranch] + const parentGroup = source.groupBase + targetLevel - 1 + const oldGroup = source.groupBase + targetLevel + const inserted = { + ...batchChild(insertedId, parentGroup, insertedValue, insertedPosition), + group: oldGroup, + } + const insertOldRoute: FullRowBatchStep = { + level: targetLevel, + changes: [{ type: `insert`, value: inserted }], + } + const rekey: FullRowBatchStep = { + level: targetLevel, + changes: [ + { + type: `update`, + value: { + ...batchChild( + source.idBase + targetLevel, + parentGroup, + source.idBase + targetLevel, + 0, + ), + group: rekeyGroup, + }, + }, + ], + } + + return { + prefix, + rekey, + reuse: insertOldRoute, + } +} + +function createRekeyRouteReuseScenarios( + options: RekeyRouteReuseOptions, +): HistoryScenarioPair { + const { depth } = options + const { prefix, rekey, reuse } = createRekeyRouteReuseFixture(options) + + return { + control: { depth, steps: [...prefix, reuse] }, + candidate: { depth, steps: [...prefix, rekey, reuse] }, + } +} + +function createIntraBatchRekeyRouteReuseScenarios( + options: RekeyRouteReuseOptions, +): HistoryScenarioPair { + const { depth } = options + const { prefix, rekey, reuse } = createRekeyRouteReuseFixture(options) + if (rekey.level === 0 || rekey.level !== reuse.level) { + throw new Error(`Intra-batch route reuse must share a child level`) + } + + return { + control: { + depth, + steps: [ + ...prefix, + { level: rekey.level, changes: [...reuse.changes, ...rekey.changes] }, + ], + }, + candidate: { + depth, + steps: [ + ...prefix, + { level: rekey.level, changes: [...rekey.changes, ...reuse.changes] }, + ], + }, + } +} + +function rekeyRouteReuseScenarioArbitrary( + depth: 2 | 3 | 4, + sourceBranch: 0 | 1, + createScenario: ( + options: RekeyRouteReuseOptions, + ) => HistoryScenarioPair = createRekeyRouteReuseScenarios, +): fc.Arbitrary { + return fc + .record({ + ...generatedBranchArbitraries, + rekeyGroup: fc.integer({ min: 2_100, max: 2_600 }), + insertedId: fc.integer({ min: 3_000, max: 3_200 }), + insertedValue: fc.integer({ min: -3, max: 3 }), + insertedPosition: fc.integer({ min: -2, max: 2 }), + }) + .map( + ({ + leftIdBase, + leftGroupBase, + rightIdBase, + rightGroupBase, + rekeyGroup, + insertedId, + insertedValue, + insertedPosition, + }) => + createScenario({ + depth, + sourceBranch, + branches: createGeneratedBranches({ + leftIdBase, + leftGroupBase, + rightIdBase, + rightGroupBase, + }), + rekeyGroup, + insertedId, + insertedValue, + insertedPosition, + }), + ) +} + +type MovedChildReplacementOptions = { + depth: 3 | 4 + targetLevel: IncludeDepth + sourceBranch: 0 | 1 + branches: readonly [ConnectedBranch, ConnectedBranch] + insertedId: number + insertedValue: number + insertedPosition: number +} + +function createMovedChildReplacementScenarios({ + depth, + targetLevel, + sourceBranch, + branches, + insertedId, + insertedValue, + insertedPosition, +}: MovedChildReplacementOptions): HistoryScenarioPair { + if (targetLevel + 2 > depth) { + throw new Error(`Child replacement needs a visible grandchild`) + } + assertDisjointRelationshipKeys(depth, branches, { + extraIds: [insertedId], + }) + + const prefix = createConnectedBatchBranches(depth, branches) + const source = branches[sourceBranch] + const destination = branches[otherBranch(sourceBranch)] + const targetGroup = source.groupBase + targetLevel + const childLevel = (targetLevel + 1) as IncludeDepth + const childGroup = source.groupBase + childLevel + const existingChild = { + ...batchChild( + source.idBase + childLevel, + targetGroup, + source.idBase + childLevel, + 0, + ), + group: childGroup, + } + const replacementChild = { + ...batchChild(insertedId, targetGroup, insertedValue, insertedPosition), + group: childGroup, + } + const replaceChild: Array = [ + { + level: childLevel, + changes: [{ type: `delete`, value: existingChild }], + }, + { + level: childLevel, + changes: [{ type: `insert`, value: replacementChild }], + }, + ] + const reparent: FullRowBatchStep = { + level: targetLevel, + changes: [ + { + type: `update`, + value: { + ...batchChild( + source.idBase + targetLevel, + destination.groupBase + targetLevel - 1, + source.idBase + targetLevel, + 0, + ), + group: targetGroup, + }, + }, + ], + } + const atomicReplacement = ( + order: `delete-first` | `insert-first`, + ): FullRowBatchScenario => ({ + depth, + steps: [ + ...prefix, + reparent, + { + level: childLevel, + changes: + order === `delete-first` + ? [ + { type: `delete`, value: existingChild }, + { type: `insert`, value: replacementChild }, + ] + : [ + { type: `insert`, value: replacementChild }, + { type: `delete`, value: existingChild }, + ], + }, + ], + }) + return { + control: { depth, steps: [...prefix, ...replaceChild] }, + greenVariants: [ + atomicReplacement(`delete-first`), + atomicReplacement(`insert-first`), + ], + candidate: { depth, steps: [...prefix, reparent, ...replaceChild] }, + } +} + +function movedChildReplacementMirrorsArbitrary( + depth: 3 | 4, + targetLevel: IncludeDepth, + sourceBranch: 0 | 1, +): fc.Arbitrary> { + return fc + .record({ + ...generatedBranchArbitraries, + insertedId: fc.integer({ min: 3_000, max: 3_200 }), + insertedValue: fc.integer({ min: -3, max: 3 }), + insertedPosition: fc.integer({ min: -2, max: 2 }), + }) + .map( + ({ + leftIdBase, + leftGroupBase, + rightIdBase, + rightGroupBase, + insertedId, + insertedValue, + insertedPosition, + }) => { + const branches = createGeneratedBranches({ + leftIdBase, + leftGroupBase, + rightIdBase, + rightGroupBase, + }) + const createMirror = (deliveryOrder: BranchDeliveryOrder) => + createMovedChildReplacementScenarios({ + depth, + targetLevel, + sourceBranch: deliveredBranchIndex(sourceBranch, deliveryOrder), + branches: deliverBranches(branches, deliveryOrder), + insertedId, + insertedValue, + insertedPosition, + }) + + return { + forward: createMirror(`forward`), + reverse: createMirror(`reverse`), + } + }, + ) +} + +type RelationshipBatchDelivery = `split` | `atomic` +type RelationshipBatchOrder = `delete-insert` | `insert-delete` +type ReplacementPublicId = `same` | `new` +type ReplacementRoute = `handoff` | `fresh` +type AncestorUpdateShape = `route-only` | `route-and-position` + +type RelationshipBatchShape = { + delivery: RelationshipBatchDelivery + order: RelationshipBatchOrder + publicId: ReplacementPublicId + route: ReplacementRoute + ancestorUpdate: AncestorUpdateShape +} + +type RelationshipBatchShapeOptions = GeneratedBranchOptions & { + shape: RelationshipBatchShape + replacementId: number + replacementGroup: number + replacementValue: number + replacementPosition: number + movedPosition: number +} + +function createRelationshipBatchShapeScenarios({ + shape, + replacementId, + replacementGroup, + replacementValue, + replacementPosition, + movedPosition, + ...branchOptions +}: RelationshipBatchShapeOptions): HistoryScenarioPair { + const depth = 3 + const branches = createGeneratedBranches(branchOptions) + assertDisjointRelationshipKeys(depth, branches, { + extraIds: shape.publicId === `new` ? [replacementId] : [], + extraGroups: shape.route === `fresh` ? [replacementGroup] : [], + }) + + const source = branches[0] + const destination = branches[1] + const prefix = createConnectedBatchBranches(depth, branches) + const targetLevel = 1 + const childLevel = 2 + const targetId = source.idBase + targetLevel + const childId = source.idBase + childLevel + const targetGroup = source.groupBase + targetLevel + const childGroup = source.groupBase + childLevel + const currentTarget = { + ...batchChild(targetId, source.groupBase, targetId, 0), + group: targetGroup, + } + const movedTarget = { + ...currentTarget, + parentGroup: destination.groupBase, + position: + shape.ancestorUpdate === `route-and-position` + ? movedPosition + : currentTarget.position, + } + const currentChild = { + ...batchChild(childId, targetGroup, childId, 0), + group: childGroup, + } + const replacementChild = { + ...currentChild, + id: shape.publicId === `same` ? childId : replacementId, + group: shape.route === `handoff` ? childGroup : replacementGroup, + value: replacementValue, + position: replacementPosition, + } + const replacementChanges: Array> = + shape.order === `delete-insert` + ? [ + { type: `delete`, value: currentChild }, + { type: `insert`, value: replacementChild }, + ] + : [ + { type: `insert`, value: replacementChild }, + { type: `delete`, value: currentChild }, + ] + const replacementSteps: Array = + shape.delivery === `atomic` + ? [{ level: childLevel, changes: replacementChanges }] + : replacementChanges.map( + (change): FullRowBatchStep => ({ + level: childLevel, + changes: [change], + }), + ) + const positionOnlyTarget = { + ...currentTarget, + position: movedTarget.position, + } + const controlMoveSteps: Array = + shape.ancestorUpdate === `route-and-position` + ? [ + { + level: targetLevel, + changes: [{ type: `update`, value: positionOnlyTarget }], + }, + ] + : [] + + return { + // The same replacement history without the relationship move is the + // adjacent control for every shape cell. Preserve the position change so + // the only candidate difference is route membership. + control: { + depth, + steps: [...prefix, ...controlMoveSteps, ...replacementSteps], + }, + candidate: { + depth, + steps: [ + ...prefix, + { + level: targetLevel, + changes: [{ type: `update`, value: movedTarget }], + }, + ...replacementSteps, + ], + }, + } +} + +const relationshipBatchFixtureArbitrary: fc.Arbitrary< + Omit +> = fc.record({ + ...generatedBranchArbitraries, + replacementId: fc.integer({ min: 3_000, max: 3_200 }), + replacementGroup: fc.integer({ min: 2_700, max: 2_900 }), + replacementValue: fc.integer({ min: -3, max: 3 }), + replacementPosition: fc.integer({ min: -2, max: 2 }), + movedPosition: fc.constantFrom(-2, -1, 1, 2), +}) + +type RelationshipBatchShapeCell = { + shape: RelationshipBatchShape + scenarios: HistoryScenarioPair +} + +function createRelationshipBatchShapeMatrix( + fixture: Omit, + publicId: ReplacementPublicId, + route: ReplacementRoute, + ancestorUpdate: AncestorUpdateShape, +): Array { + const cells: Array = [] + + for (const delivery of [`split`, `atomic`] as const) { + for (const order of [`delete-insert`, `insert-delete`] as const) { + // Inserting the same public id before its existing row is retired is a + // duplicate-key error, not a valid alternate delivery of the same final + // state. The new-id cells exercise insert-before-delete in both forms. + if (publicId === `same` && order === `insert-delete`) continue + const shape = { + delivery, + order, + publicId, + route, + ancestorUpdate, + } satisfies RelationshipBatchShape + cells.push({ + shape, + scenarios: createRelationshipBatchShapeScenarios({ + ...fixture, + shape, + }), + }) + } + } + + return cells +} + +function createSharedRouteLifetimeScenarios( + parentLevel: 0 | 1, +): HistoryScenarioPair { + if (parentLevel === 0) { + const departed = batchRoot(100, 600, 100, 0) + const remaining = batchRoot(1_100, 600, 1_100, 1) + const child = { ...batchChild(101, 600, 101, 0), group: 601 } + const controlSteps: Array = [ + { level: 0, changes: [{ type: `insert`, value: departed }] }, + { level: 1, changes: [{ type: `insert`, value: child }] }, + { + level: 0, + changes: [{ type: `update`, value: { ...departed, group: 700 } }], + }, + { + level: 1, + changes: [ + { type: `update`, value: { ...child, value: child.value + 1 } }, + ], + }, + ] + const candidateSteps: Array = [ + { + level: 0, + changes: [ + { type: `insert`, value: departed }, + { type: `insert`, value: remaining }, + ], + }, + ...controlSteps.slice(1), + ] + + return { + control: { depth: 1, steps: controlSteps }, + candidate: { depth: 1, steps: candidateSteps }, + } + } + + const root = batchRoot(100, 600, 100, 0) + const departed = { ...batchChild(101, 600, 101, 0), group: 601 } + const remaining = { ...batchChild(1_101, 600, 1_101, 1), group: 601 } + const child = { ...batchChild(102, 601, 102, 0), group: 602 } + const controlSteps: Array = [ + { level: 0, changes: [{ type: `insert`, value: root }] }, + { level: 1, changes: [{ type: `insert`, value: departed }] }, + { level: 2, changes: [{ type: `insert`, value: child }] }, + { + level: 1, + changes: [{ type: `update`, value: { ...departed, group: 700 } }], + }, + { + level: 2, + changes: [ + { type: `update`, value: { ...child, value: child.value + 1 } }, + ], + }, + ] + const candidateSteps: Array = [ + controlSteps[0]!, + { + level: 1, + changes: [ + { type: `insert`, value: departed }, + { type: `insert`, value: remaining }, + ], + }, + ...controlSteps.slice(2), + ] + + return { + control: { depth: 2, steps: controlSteps }, + candidate: { depth: 2, steps: candidateSteps }, + } +} + +async function expectFullRowBatchScenarioMatches({ + depth, + steps, +}: FullRowBatchScenario): Promise { + await runTrace({ + steps, + driver: createFullRowBatchTraceDriver(depth), + projection: structuralProjection, + }) +} + +async function expectHistoryScenarioPairMatches({ + control, + greenVariants = [], + candidate, +}: HistoryScenarioPair): Promise { + await expectFullRowBatchScenarioMatches(control) + for (const greenVariant of greenVariants) { + await expectFullRowBatchScenarioMatches(greenVariant) + } + await expectFullRowBatchScenarioMatches(candidate) +} + +function recomputeFullRowBatchScenario( + { depth, steps }: FullRowBatchScenario, + stepCount: number, +): Array { + const roots = new Map() + const levels = Array.from({ length: 4 }, () => new Map()) + + for (const step of steps.slice(0, stepCount)) { + updateFullRowBatchModels(step, roots, levels) + } + + return recompute(roots, levels, depth) +} + +type FlatMaterialization = `array` | `concat` + +function createFlatMaterializationQuery( + materialization: FlatMaterialization, + sources: Sources, +) { + if (materialization === `array`) { + return createLiveQueryCollection((q) => + q + .from({ root: sources.roots.collection }) + .orderBy(({ root }) => root.position) + .orderBy(({ root }) => root.id) + .select(({ root }) => ({ + id: root.id, + group: root.group, + children: materialize( + q + .from({ child: sources.levels[0].collection }) + .where(({ child }) => eq(child.parentGroup, root.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ id: child.id, value: child.value })), + ), + })), + ) + } + + return createLiveQueryCollection((q) => + q + .from({ root: sources.roots.collection }) + .orderBy(({ root }) => root.position) + .orderBy(({ root }) => root.id) + .select(({ root }) => ({ + id: root.id, + group: root.group, + content: concat( + toArray( + q + .from({ child: sources.levels[0].collection }) + .where(({ child }) => eq(child.parentGroup, root.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => child.value), + ), + ), + })), + ) +} + +type FlatMaterializationContext = Omit< + StructuralTraceContext, + `incremental` +> & { + incremental: ReturnType +} + +function createFlatMaterializationDriver( + materialization: FlatMaterialization, +): TraceDriver { + return { + setup: () => { + const sources = createSources(`full`) + return { + depth: 1, + sources, + incremental: createFlatMaterializationQuery(materialization, sources), + roots: new Map(), + levels: Array.from({ length: 4 }, () => new Map()), + } + }, + start: ({ incremental }) => incremental.preload(), + apply: (step, { sources, roots, levels }) => { + if (step.level === 0) { + sources.roots.writeBatch(step.changes) + updateModel(roots, step.changes) + return + } + + if (step.level !== 1) { + throw new Error(`Flat materialization only supports depth 1`) + } + sources.levels[0].writeBatch(step.changes) + updateModel(levels[0]!, step.changes) + }, + cleanup: cleanupStructuralTrace, + } +} + +type FlatMaterializationResult = Array< + | { + id: number + group: number + children: Array<{ id: number; value: number }> + } + | { id: number; group: number; content: string } +> + +function recomputeFlatMaterialization( + materialization: FlatMaterialization, + roots: Map, + children: Map, +): FlatMaterializationResult { + return [...roots.values()].sort(compareRows).map((root) => { + const matching = [...children.values()] + .filter((child) => child.parentGroup === root.group) + .sort(compareRows) + return materialization === `array` + ? { + id: root.id, + group: root.group, + children: matching.map(({ id, value }) => ({ id, value })), + } + : { + id: root.id, + group: root.group, + content: matching.map(({ value }) => String(value)).join(``), + } + }) +} + +function flatMaterializationProjection( + materialization: FlatMaterialization, +): TraceProjection< + FlatMaterializationContext, + unknown, + FlatMaterializationResult +> { + return { + observe: ({ incremental }) => stripVirtualProperties(incremental.toArray), + recompute: ({ roots, levels }) => + recomputeFlatMaterialization(materialization, roots, levels[0]!), + assertEqual: (observed, expected) => { + expect(observed).toEqual(expected) + return undefined + }, + } +} + +const flatMaterializationScenarioArbitrary = fc + .array(fullRowBatchInputArbitrary(1, `all`), { + minLength: 1, + maxLength: 12, + }) + .map((inputs) => normalizeFullRowBatchInputs(1, inputs)) + +async function expectFlatMaterializationScenarioMatches( + materialization: FlatMaterialization, + scenario: FullRowBatchScenario, +): Promise { + await runTrace({ + steps: scenario.steps, + driver: createFlatMaterializationDriver(materialization), + projection: flatMaterializationProjection(materialization), + }) +} + +const structuralProjection: TraceProjection< + StructuralTraceContext, + unknown, + Array +> = { + observe: ({ incremental }) => stripVirtualProperties(incremental.toArray), + recompute: ({ roots, levels, depth }) => recompute(roots, levels, depth), + assertEqual: (observed, expected) => { + expect(observed).toEqual(expected) + }, +} + +async function expectScenarioMatches(scenario: Scenario): Promise { + await runTrace({ + steps: scenario.history, + driver: createStructuralTraceDriver(scenario.depth), + projection: structuralProjection, + }) +} + +function createMaterializeSources() { + return { + roots: createControlledCollection(`materialize-roots`), + middles: + createControlledCollection(`materialize-middles`), + shared: createControlledCollection(`materialize-shared`), + leaves: createControlledCollection(`materialize-leaves`), + } +} + +type MaterializeSources = ReturnType + +type MaterializeModels = { + roots: Map + middles: Map + shared: Map + leaves: Map +} + +type MaterializeTraceStep = + | { type: `insert`; insert: MaterializeInsert } + | { type: `incrementLeaf`; id: number } + | { type: `redirectMiddle`; id: number; sharedId: number } + +type MaterializeTraceContext = { + sharedIntermediate: boolean + sources: MaterializeSources + live: ReturnType + models: MaterializeModels +} + +function createMaterializeQuery(sources: MaterializeSources) { + return createLiveQueryCollection((q) => + q + .from({ root: sources.roots.collection }) + .orderBy(({ root }) => root.id) + .select(({ root }) => ({ + id: root.id, + middle: materialize( + q + .from({ middle: sources.middles.collection }) + .where(({ middle }) => eq(middle.id, root.middleId)) + .select(({ middle }) => ({ + id: middle.id, + sharedId: middle.sharedId, + shared: materialize( + q + .from({ shared: sources.shared.collection }) + .where(({ shared }) => eq(shared.id, middle.sharedId)) + .select(({ shared }) => ({ + id: shared.id, + leafId: shared.leafId, + leaf: materialize( + q + .from({ leaf: sources.leaves.collection }) + .where(({ leaf }) => eq(leaf.id, shared.leafId)) + .select(({ leaf }) => ({ + id: leaf.id, + value: leaf.value, + })) + .findOne(), + ), + })) + .findOne(), + ), + })) + .findOne(), + ), + })), + ) +} + +function recomputeMaterialize( + roots: Map, + middles: Map, + sharedRows: Map, + leaves: Map, +): Array { + return [...roots.values()] + .sort((left, right) => left.id - right.id) + .map((root) => { + const middle = middles.get(root.middleId) + if (!middle) return { id: root.id, middle: undefined } + + const shared = sharedRows.get(middle.sharedId) + return { + id: root.id, + middle: { + id: middle.id, + sharedId: middle.sharedId, + shared: shared + ? { + id: shared.id, + leafId: shared.leafId, + leaf: leaves.get(shared.leafId), + } + : undefined, + }, + } + }) +} + +function insertMaterializeRow( + insert: MaterializeInsert, + sharedIntermediate: boolean, + sources: MaterializeSources, + models: MaterializeModels, +): void { + switch (insert) { + case `root-1`: + case `root-2`: { + const row = { id: insert === `root-1` ? 1 : 2, middleId: 1 } + if (insert === `root-2`) row.middleId = 2 + sources.roots.write(`insert`, row) + models.roots.set(row.id, row) + return + } + case `middle-1`: + case `middle-2`: { + const id = insert === `middle-1` ? 1 : 2 + const row = { id, sharedId: sharedIntermediate ? 1 : id } + sources.middles.write(`insert`, row) + models.middles.set(row.id, row) + return + } + case `shared-1`: + case `shared-2`: { + const id = insert === `shared-1` ? 1 : 2 + const row = { id, leafId: id } + sources.shared.write(`insert`, row) + models.shared.set(row.id, row) + return + } + case `leaf-1`: + case `leaf-2`: { + const id = insert === `leaf-1` ? 1 : 2 + const row = { id, value: id * 10 } + sources.leaves.write(`insert`, row) + models.leaves.set(row.id, row) + } + } +} + +async function cleanupMaterializeSources(sources: MaterializeSources) { + await Promise.all( + Object.values(sources).map(({ collection }) => collection.cleanup()), + ) +} + +function createMaterializeTraceSteps( + insertOrder: Array, +): Array { + const steps: Array = insertOrder.map((insert) => ({ + type: `insert`, + insert, + })) + + for (const insert of insertOrder) { + if (insert === `leaf-1` || insert === `leaf-2`) { + steps.push({ type: `incrementLeaf`, id: insert === `leaf-1` ? 1 : 2 }) + } + } + + return steps +} + +function createMaterializeTraceDriver( + scenarioSharedIntermediate: boolean, +): TraceDriver { + return { + setup: () => { + const sources = createMaterializeSources() + return { + sharedIntermediate: scenarioSharedIntermediate, + sources, + live: createMaterializeQuery(sources), + models: { + roots: new Map(), + middles: new Map(), + shared: new Map(), + leaves: new Map(), + }, + } + }, + start: ({ live }) => live.preload(), + apply: (step, { models, sources, sharedIntermediate }) => { + if (step.type === `insert`) { + insertMaterializeRow(step.insert, sharedIntermediate, sources, models) + return + } + + if (step.type === `redirectMiddle`) { + const middle = models.middles.get(step.id) + if (!middle) throw new Error(`Missing middle ${step.id} in trace model`) + const updated = { ...middle, sharedId: step.sharedId } + sources.middles.write(`update`, updated) + models.middles.set(updated.id, updated) + return + } + + const leaf = models.leaves.get(step.id) + if (!leaf) throw new Error(`Missing leaf ${step.id} in trace model`) + const updated = { ...leaf, value: leaf.value + 1 } + sources.leaves.write(`update`, updated) + models.leaves.set(updated.id, updated) + }, + cleanup: async ({ live, sources }) => { + await live.cleanup() + await cleanupMaterializeSources(sources) + }, + } +} + +const materializeProjection: TraceProjection< + MaterializeTraceContext, + unknown, + Array +> = { + observe: ({ live }) => stripVirtualProperties(live.toArray), + recompute: ({ models }) => + recomputeMaterialize( + models.roots, + models.middles, + models.shared, + models.leaves, + ), + assertEqual: (observed, expected) => { + expect(observed).toEqual(expected) + }, +} + +async function expectMaterializeScenarioMatches({ + sharedIntermediate, + insertOrder, +}: MaterializeScenario): Promise { + await runTrace({ + steps: createMaterializeTraceSteps(insertOrder), + driver: createMaterializeTraceDriver(sharedIntermediate), + projection: materializeProjection, + }) +} + +const intraBatchChildHandOffScenario: FullRowBatchScenario = { + depth: 1, + steps: [ + { + level: 0, + changes: [ + { type: `insert`, value: batchRoot(0, 1, 0, 0) }, + { type: `insert`, value: batchRoot(1, 3, 0, 0) }, + ], + }, + { + level: 1, + changes: [ + { type: `insert`, value: batchChild(5, 3, 0, 0) }, + { type: `insert`, value: batchChild(1, 1, 0, 0) }, + ], + }, + { + level: 1, + changes: [ + { type: `update`, value: batchChild(1, 0, 0, 0) }, + { type: `update`, value: batchChild(5, 1, 0, 0) }, + ], + }, + ], +} + +function createReparentedSubtreeUpdateScenario( + depth: 3 | 4, + targetLevel: 1 | 2, +): VisibleRelationshipScenario { + return createVisibleRelationshipScenario({ + depth, + transition: `reparent`, + targetLevel, + sourceBranch: 0, + branches: [ + { idBase: 100, groupBase: 600 }, + { idBase: 1_100, groupBase: 1_600 }, + ], + noise: [targetLevel + 1, targetLevel + 2].map((level) => ({ + side: `after`, + level: level as IncludeDepth, + branch: 0, + value: 1, + position: 0, + })), + }) +} + +const minimalRekeyScenario = createVisibleRelationshipScenario({ + depth: 3, + transition: `rekey`, + targetLevel: 1, + sourceBranch: 0, + branches: [ + { idBase: 100, groupBase: 600 }, + { idBase: 1_100, groupBase: 1_600 }, + ], + rekeyGroup: 2_100, + noise: [], +}) + +const transitionHistoryBranches = [ + { idBase: 100, groupBase: 600 }, + { idBase: 1_100, groupBase: 1_600 }, +] as const + +const { + control: rekeyRouteReuseControl, + candidate: rekeyRouteResurrectionScenario, +} = createRekeyRouteReuseScenarios({ + depth: 2, + sourceBranch: 0, + branches: transitionHistoryBranches, + rekeyGroup: 2_100, + insertedId: 3_000, + insertedValue: 0, + insertedPosition: 0, +}) + +const { + control: childReplacementControl, + candidate: movedSubtreeChildReplacementScenario, +} = createMovedChildReplacementScenarios({ + depth: 3, + targetLevel: 1, + sourceBranch: 0, + branches: transitionHistoryBranches, + insertedId: 3_000, + insertedValue: 0, + insertedPosition: 0, +}) + +describe(`includes recompute oracle`, () => { + fcTest(`rejects subscriber lifecycle labels on reparent transitions`, () => { + expect(() => + createRouteLifecycleScenario({ + depth: 3, + branches: transitionHistoryBranches, + descriptors: [ + { + kind: `reparent`, + level: 2, + row: 0, + destination: { strategy: `fresh`, route: 2_100 }, + } as unknown as RouteTransitionDescriptor, + ], + }), + ).toThrow(/reparent transitions only support live merge destinations/) + }) + + fcTest(`scopes rekey route histories to their include level`, () => { + expect(() => + createRouteLifecycleScenario({ + depth: 3, + branches: transitionHistoryBranches, + descriptors: [ + { + kind: `rekey`, + level: 1, + row: 0, + destination: { strategy: `fresh`, route: 2_100 }, + }, + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `fresh`, route: 2_100 }, + }, + ], + }), + ).not.toThrow() + }) + + fcTest(`the sibling topology targets rows under one parent`, () => { + const branches = transitionHistoryBranches + const prefix = independentTransitionPrefix(`sibling`, branches) + const levelTwo = prefix.find((step) => step.level === 2) + if (!levelTwo || levelTwo.level !== 2) throw new Error(`Missing level 2`) + const first = levelTwo.changes.find( + (change) => change.value.id === rowAt(branches, 0, 2), + ) + const second = levelTwo.changes.find( + (change) => change.value.id === rowAt(branches, 1, 2), + ) + if (!first || !second) throw new Error(`Missing sibling targets`) + + expect(first.value.parentGroup).toBe(second.value.parentGroup) + }) + + fcTest(`the descendant remains attached before its ancestor moves`, () => { + const branches = transitionHistoryBranches + const scenario = createRouteLifecycleScenario({ + depth: 3, + branches, + descriptors: independentTransitionDescriptors( + `descendant-ancestor`, + branches, + [2_100, 2_500], + ), + }) + const ancestorTransitionStep = scenario.transitionStepIndexes[1] + if (ancestorTransitionStep === undefined) { + throw new Error(`Missing ancestor transition`) + } + const beforeAncestorMove = recomputeFullRowBatchScenario( + scenario, + ancestorTransitionStep, + ) + + expect( + hasDirectChild( + beforeAncestorMove, + rowAt(branches, 0, 1), + rowAt(branches, 0, 2), + ), + ).toBe(true) + }) + + fcTest( + `a root entering a live route receives its ordered snapshot`, + async () => { + const branches = transitionHistoryBranches + const prefix = createConnectedBatchBranches(1, branches) + const childStep = prefix.find((step) => step.level === 1) + if (!childStep || childStep.level !== 1) + throw new Error(`Missing children`) + const scenario: FullRowBatchScenario = { + depth: 1, + steps: [ + prefix[0]!, + { + level: 1, + changes: [ + ...childStep.changes, + { + type: `insert`, + value: { + ...batchChild(3_000, branches[0].groupBase, 3_000, -1), + group: 2_500, + }, + }, + ], + }, + { + level: 0, + changes: [ + { + type: `update`, + value: batchRoot( + branches[1].idBase, + branches[0].groupBase, + branches[1].idBase, + 0, + ), + }, + ], + }, + ], + } + + await expectFullRowBatchScenarioMatches(scenario) + }, + ) + + for (const [shapeIndex, shape] of independentTransitionShapes.entries()) { + fcTest.prop([independentTransitionScenarioArbitrary(shape)], { + numRuns: oracleRuns(4), + seed: 1734 + shapeIndex, + })( + `matches recomputation for independent ${shape} relationship targets`, + async (scenario) => { + expectEveryRouteTransitionVisible(scenario) + await expectFullRowBatchScenarioMatches(scenario) + }, + ) + } + + for (const [historyIndex, history] of destinationHistories.entries()) { + fcTest.prop([destinationHistoryScenarioArbitrary(history)], { + numRuns: oracleRuns(4), + seed: 1740 + historyIndex, + })( + `matches recomputation for the ${history} route destination history`, + async (scenario) => { + expectEveryRouteTransitionVisible(scenario) + await expectFullRowBatchScenarioMatches(scenario) + }, + ) + } + + for (const parentLevel of [0, 1, 2] as const) { + for (const enteringRow of [0, 1] as const) { + fcTest( + `matches recomputation when level-${parentLevel} row ${enteringRow} enters a live shared route`, + () => + expectHistoryScenarioPairMatches( + createMergeIntoSharedRouteScenarios(parentLevel, enteringRow), + ), + ) + } + + fcTest( + `matches recomputation when the last level-${parentLevel} shared-route subscriber leaves`, + () => + expectFullRowBatchScenarioMatches( + createSharedRouteLastSubscriberScenario(parentLevel), + ), + ) + + fcTest( + `matches recomputation after a level-${parentLevel} route resubscribes`, + () => + expectHistoryScenarioPairMatches( + createSnapshotOnResubscribeScenarios(parentLevel), + ), + ) + + fcTest( + `matches recomputation when an initially shared level-${parentLevel} route retires, changes, and resubscribes`, + () => + expectHistoryScenarioPairMatches( + createInitiallySharedRouteResubscribeScenario(parentLevel), + ), + ) + } + + fcTest(`rejects invalid route destination strategies`, () => { + const branches = transitionHistoryBranches + const ownRoute = branches[0].groupBase + 2 + const otherRoute = branches[1].groupBase + 2 + const freshRoute = 2_100 + const invalidCases: ReadonlyArray<{ + descriptors: ReadonlyArray + message: RegExp + }> = [ + { + descriptors: [ + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `fresh`, route: otherRoute }, + }, + ], + message: /fresh route must never have been used/, + }, + { + descriptors: [ + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `restore`, route: ownRoute }, + }, + ], + message: /restore route must have been retired by this row/, + }, + { + descriptors: [ + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `fresh`, route: freshRoute }, + }, + { + kind: `rekey`, + level: 2, + row: 1, + destination: { strategy: `restore`, route: ownRoute }, + }, + ], + message: /restore route must have been retired by this row/, + }, + { + descriptors: [ + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `merge`, route: freshRoute }, + }, + ], + message: /merge route must be live and different/, + }, + { + descriptors: [ + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `merge`, route: ownRoute }, + }, + ], + message: /merge route must be live and different/, + }, + { + descriptors: [ + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `split`, route: freshRoute }, + }, + ], + message: /split source route must be shared/, + }, + { + descriptors: [ + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `merge`, route: otherRoute }, + }, + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `split`, route: ownRoute }, + }, + ], + message: /split destination must be unused/, + }, + { + descriptors: [ + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `retired`, route: otherRoute }, + }, + ], + message: /retired route must have been retired by another row/, + }, + { + descriptors: [ + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `fresh`, route: freshRoute }, + }, + { + kind: `rekey`, + level: 2, + row: 0, + destination: { strategy: `retired`, route: ownRoute }, + }, + ], + message: /retired route must have been retired by another row/, + }, + ] + + for (const invalidCase of invalidCases) { + expect(() => + createRouteLifecycleScenario({ + depth: 3, + branches, + descriptors: invalidCase.descriptors, + }), + ).toThrow(invalidCase.message) + } + }) + + fcTest(`rejects overlapping visible relationship keys`, () => { + const base = { + depth: 4, + transition: `rekey`, + targetLevel: 1, + sourceBranch: 0, + noise: [], + } as const + const collisions: Array<{ + branches: readonly [ConnectedBranch, ConnectedBranch] + rekeyGroup: number + }> = [ + { + branches: [ + { idBase: 100, groupBase: 600 }, + { idBase: 102, groupBase: 1_600 }, + ], + rekeyGroup: 2_100, + }, + { + branches: [ + { idBase: 100, groupBase: 600 }, + { idBase: 1_100, groupBase: 602 }, + ], + rekeyGroup: 2_100, + }, + { + branches: [ + { idBase: 100, groupBase: 600 }, + { idBase: 1_100, groupBase: 1_600 }, + ], + rekeyGroup: 603, + }, + ] + + for (const collision of collisions) { + expect(() => + createVisibleRelationshipScenario({ + ...base, + ...collision, + }), + ).toThrow(/overlap/) + } + }) + + for (const materialization of [`array`, `concat`] as const) { + fcTest(`${materialization} follows an intra-batch child hand-off`, () => + expectFlatMaterializationScenarioMatches( + materialization, + intraBatchChildHandOffScenario, + ), + ) + } + + for (const [depth, targetLevel] of [ + [3, 1], + [4, 1], + [4, 2], + ] as const) { + fcTest( + `later updates propagate through a reparented subtree at depth ${depth}, level ${targetLevel}`, + () => + expectFullRowBatchScenarioMatches( + createReparentedSubtreeUpdateScenario(depth, targetLevel), + ), + ) + } + + fcTest(`rekeying a row detaches two descendant levels`, () => + expectFullRowBatchScenarioMatches(minimalRekeyScenario), + ) + + fcTest(`reusing a rekeyed row's old route does not resurrect its child`, () => + expectFullRowBatchScenarioMatches(rekeyRouteResurrectionScenario), + ) + + fcTest( + `matches recomputation when sharing a route without rekeying its existing row`, + () => expectFullRowBatchScenarioMatches(rekeyRouteReuseControl), + ) + + fcTest(`replacing a moved subtree child retains its grandchild`, () => + expectFullRowBatchScenarioMatches(movedSubtreeChildReplacementScenario), + ) + + fcTest( + `matches recomputation when replacing a child without reparenting its ancestor`, + () => expectFullRowBatchScenarioMatches(childReplacementControl), + ) + + for (const depth of [2, 3, 4] as const) { + for (const sourceBranch of [0, 1] as const) { + fcTest.prop([rekeyRouteReuseScenarioArbitrary(depth, sourceBranch)], { + numRuns: oracleRuns(4), + seed: 1726 + depth * 10 + sourceBranch, + })( + `reuses a retired route at depth ${depth}, branch ${sourceBranch}`, + expectHistoryScenarioPairMatches, + ) + + fcTest.prop( + [ + rekeyRouteReuseScenarioArbitrary( + depth, + sourceBranch, + createIntraBatchRekeyRouteReuseScenarios, + ), + ], + { + numRuns: oracleRuns(4), + seed: 1733 + depth * 10 + sourceBranch, + }, + )( + `handles intra-batch rekey then retired-route reuse at depth ${depth}, branch ${sourceBranch}`, + expectHistoryScenarioPairMatches, + ) + } + } + + for (const parentLevel of [0, 1] as const) { + fcTest( + `a departed level-${parentLevel} shared-route subscriber ignores later child updates`, + () => + expectHistoryScenarioPairMatches( + createSharedRouteLifetimeScenarios(parentLevel), + ), + ) + } + + for (const depth of [3, 4] as const) { + for (let targetLevel = 1; targetLevel <= depth - 2; targetLevel++) { + for (const sourceBranch of [0, 1] as const) { + fcTest.prop( + [ + movedChildReplacementMirrorsArbitrary( + depth, + targetLevel as IncludeDepth, + sourceBranch, + ), + ], + { + numRuns: oracleRuns(4), + seed: 1727 + depth * 100 + targetLevel * 10 + sourceBranch, + }, + )( + `matches forward/reverse delivery mirrors when replacing a moved child at depth ${depth}, level ${targetLevel}, source ${sourceBranch}`, + async (scenarios) => { + for (const deliveryOrder of branchDeliveryOrders) { + await expectHistoryScenarioPairMatches(scenarios[deliveryOrder]) + } + }, + ) + } + } + } + + for (const [publicIdIndex, publicId] of ( + [`same`, `new`] as const + ).entries()) { + for (const [routeIndex, route] of ( + [`handoff`, `fresh`] as const + ).entries()) { + for (const [updateIndex, ancestorUpdate] of ( + [`route-only`, `route-and-position`] as const + ).entries()) { + fcTest.prop([relationshipBatchFixtureArbitrary], { + numRuns: oracleRuns(4), + seed: 1738 + publicIdIndex * 100 + routeIndex * 10 + updateIndex, + })( + `matches the split/atomic replacement matrix for ${publicId} public id, ${route} route, ${ancestorUpdate}`, + async (fixture) => { + const cells = createRelationshipBatchShapeMatrix( + fixture, + publicId, + route, + ancestorUpdate, + ) + const finalStates = cells.map(({ scenarios: { candidate } }) => + recomputeFullRowBatchScenario(candidate, candidate.steps.length), + ) + + // Delivery boundaries and change order must not alter the final + // recompute semantics for one generated fixture. + expect(finalStates.length).toBeGreaterThan(1) + for (const finalState of finalStates.slice(1)) { + expect(finalState).toEqual(finalStates[0]) + } + + for (const { + scenarios: { control, candidate }, + } of cells) { + await expectFullRowBatchScenarioMatches(control) + await expectFullRowBatchScenarioMatches(candidate) + } + }, + ) + } + } + } + + fcTest.prop( + [ + fc.constantFrom(`array`, `concat`), + flatMaterializationScenarioArbitrary, + ], + { + numRuns: oracleRuns(30), + seed: 1721, + }, + )(`matches recomputation for flat materializations`, (kind, scenario) => + expectFlatMaterializationScenarioMatches(kind, scenario), + ) + + for (const depth of [1, 2, 3, 4] as const) { + fcTest.prop([fullRowBatchScenarioAtDepthArbitrary(depth)], { + numRuns: oracleRuns(10), + seed: 1719 + depth, + })( + `matches recomputation for visible multi-row batches at depth ${depth}`, + expectFullRowBatchScenarioMatches, + ) + + const transitions: Array = [ + `reparent`, + `rekey`, + ] + for (const transition of transitions) { + for (let targetLevel = 1; targetLevel <= depth; targetLevel++) { + fcTest.prop( + [ + visibleRelationshipScenarioArbitrary( + depth, + transition, + targetLevel as IncludeDepth, + ), + ], + { + numRuns: oracleRuns(4), + seed: 1721 + depth + targetLevel, + }, + )( + `matches recomputation for a visible ${transition} at depth ${depth}, level ${targetLevel}`, + async (scenarios) => { + for (const scenario of [ + scenarios.transitionOnly, + scenarios.stateful, + ]) { + const beforeTransition = recomputeFullRowBatchScenario( + scenario, + scenario.transitionStepIndex, + ) + const result = recomputeFullRowBatchScenario( + scenario, + scenario.transitionStepIndex + 1, + ) + + expect(result).not.toEqual(beforeTransition) + await expectFullRowBatchScenarioMatches(scenario) + } + }, + ) + } + } + } + + for (const depth of [1, 2, 3, 4] as const) { + const transitions: Array = [ + `reparent`, + `rekey`, + ] + for (const [firstIndex, firstTransition] of transitions.entries()) { + for (const [secondIndex, secondTransition] of transitions.entries()) { + if ( + transitionHistoryPlacements(depth, firstTransition, secondTransition) + .length === 0 + ) { + continue + } + + for (const sourceBranch of [0, 1] as const) { + fcTest.prop( + [ + transitionHistoryScenariosArbitrary( + depth, + firstTransition, + secondTransition, + sourceBranch, + ), + ], + { + numRuns: oracleRuns(3), + seed: + 1725 + + depth * 100 + + firstIndex * 10 + + secondIndex * 2 + + sourceBranch, + }, + )( + `matches recomputation for ${firstTransition} → ${secondTransition} histories at depth ${depth}, branch ${sourceBranch}`, + async (scenarios) => { + for (const scenario of scenarios) { + expectEveryHistoryStepVisible(scenario) + await expectFullRowBatchScenarioMatches(scenario) + } + }, + ) + } + } + } + } + + fcTest(`nested scalar materialization follows a reference update`, () => + runTrace({ + steps: [ + { type: `insert`, insert: `root-1` }, + { type: `insert`, insert: `middle-1` }, + { type: `insert`, insert: `shared-1` }, + { type: `insert`, insert: `leaf-1` }, + { type: `redirectMiddle`, id: 1, sharedId: 2 }, + ], + driver: createMaterializeTraceDriver(false), + projection: materializeProjection, + }), + ) + + fcTest(`matches recomputation for full-row sync batches`, async () => { + await runTrace({ + steps: fullRowBatchTrace, + driver: createFullRowBatchTraceDriver(1), + projection: structuralProjection, + }) + }) + + fcTest(`a reinserted parent drops its old shared route`, () => + expectFullRowBatchScenarioMatches(fullRowSharedRoutingSeed), + ) + + fcTest(`supports repeated optimistic rollbacks in one history`, async () => { + await expectScenarioMatches({ + depth: 1, + history: [ + { + type: `put`, + level: 0, + id: 0, + parentGroup: 0, + group: 0, + value: 0, + position: 0, + }, + { + type: `put`, + level: 1, + id: 0, + parentGroup: 0, + group: 0, + value: 0, + position: 0, + }, + { + type: `optimisticRollback`, + level: 1, + id: 0, + parentGroup: 0, + group: 0, + value: 1, + position: 0, + }, + { + type: `optimisticRollback`, + level: 1, + id: 0, + parentGroup: 0, + group: 0, + value: 2, + position: 0, + }, + ], + }) + }) + + fcTest.prop( + [scenarioArbitrary], + oraclePropertyOptions(40, `includes.incremental-history`), + )( + `matches naive recomputation after every incremental change`, + expectScenarioMatches, + ) + + fcTest.prop( + [ + materializeScenarioArbitrary.filter( + ({ sharedIntermediate }) => !sharedIntermediate, + ), + ], + oraclePropertyOptions(30, `includes.nested-scalar-materialization`), + )( + `matches recomputation for nested scalar materialization`, + expectMaterializeScenarioMatches, + ) + + fcTest.prop( + [ + fc.uniqueArray( + fc.record({ + id: fc.integer({ min: 0, max: 5 }), + group: fc.integer({ min: 0, max: 2 }), + value: fc.integer({ min: -3, max: 3 }), + position: fc.integer({ min: -2, max: 2 }), + }), + { selector: (row) => row.id, minLength: 1, maxLength: 5 }, + ), + fc.uniqueArray( + fc.record({ + id: fc.integer({ min: 0, max: 7 }), + parentGroup: fc.integer({ min: 0, max: 2 }), + group: fc.integer({ min: 0, max: 2 }), + value: fc.integer({ min: -3, max: 3 }), + position: fc.integer({ min: -2, max: 2 }), + }), + { selector: (row) => row.id, maxLength: 7 }, + ), + ], + oraclePropertyOptions(25, `includes.alpha-renaming`), + )( + `is unchanged by alpha-renaming, sibling declaration order, or an unrelated sibling`, + async (rootRows, childRows) => { + const roots = createControlledCollection( + `metamorphic-roots`, + rootRows, + ) + const children = createControlledCollection( + `metamorphic-children`, + childRows, + ) + const unrelated = createControlledCollection( + `metamorphic-unrelated`, + childRows.map((row) => ({ ...row, id: row.id + 100 })), + ) + + try { + const baseline = await queryOnce((q) => + q.from({ parent: roots.collection }).select(({ parent }) => ({ + id: parent.id, + group: parent.group, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .orderBy(({ child }) => child.position) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ id: child.id, value: child.value })), + ), + })), + ) + const renamed = await queryOnce((q) => + q.from({ r: roots.collection }).select(({ r }) => ({ + id: r.id, + group: r.group, + children: toArray( + q + .from({ c: children.collection }) + .where(({ c }) => eq(c.parentGroup, r.group)) + .orderBy(({ c }) => c.position) + .orderBy(({ c }) => c.id) + .select(({ c }) => ({ id: c.id, value: c.value })), + ), + })), + ) + const withUnrelatedSibling = await queryOnce((q) => + q.from({ r: roots.collection }).select(({ r }) => ({ + unrelated: toArray( + q + .from({ u: unrelated.collection }) + .where(({ u }) => eq(u.parentGroup, r.group)) + .select(({ u }) => ({ id: u.id })), + ), + id: r.id, + group: r.group, + children: toArray( + q + .from({ c: children.collection }) + .where(({ c }) => eq(c.parentGroup, r.group)) + .orderBy(({ c }) => c.position) + .orderBy(({ c }) => c.id) + .select(({ c }) => ({ id: c.id, value: c.value })), + ), + })), + ) + const withReorderedSiblings = await queryOnce((q) => + q.from({ r: roots.collection }).select(({ r }) => ({ + id: r.id, + group: r.group, + children: toArray( + q + .from({ c: children.collection }) + .where(({ c }) => eq(c.parentGroup, r.group)) + .orderBy(({ c }) => c.position) + .orderBy(({ c }) => c.id) + .select(({ c }) => ({ id: c.id, value: c.value })), + ), + unrelated: toArray( + q + .from({ u: unrelated.collection }) + .where(({ u }) => eq(u.parentGroup, r.group)) + .select(({ u }) => ({ id: u.id })), + ), + })), + ) + + expect(stripVirtualProperties(renamed)).toEqual( + stripVirtualProperties(baseline), + ) + expect( + stripVirtualProperties( + withUnrelatedSibling.map( + ({ unrelated: _unrelated, ...row }) => row, + ), + ), + ).toEqual(stripVirtualProperties(baseline)) + expect(stripVirtualProperties(withReorderedSiblings)).toEqual( + stripVirtualProperties(withUnrelatedSibling), + ) + } finally { + await Promise.all([ + roots.collection.cleanup(), + children.collection.cleanup(), + unrelated.collection.cleanup(), + ]) + } + }, + ) + + fcTest.prop( + [fc.integer({ min: -5, max: 5 }).filter((value) => value !== 0)], + oraclePropertyOptions(15, `includes.optimistic-convergence`), + )( + `optimistic updates converge to confirmed-only state`, + async (confirmedValue) => { + const roots = createControlledCollection(`convergence-roots`, [ + { id: 1, group: 1, value: 0, position: 0 }, + ]) + const children = createControlledCollection( + `convergence-children`, + [ + { + id: 1, + parentGroup: 1, + group: 1, + value: 0, + position: 0, + }, + ], + ) + const live = createLiveQueryCollection((q) => + q.from({ root: roots.collection }).select(({ root }) => ({ + id: root.id, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, root.group)) + .select(({ child }) => ({ + id: child.id, + value: child.value, + })), + ), + })), + ) + + try { + await live.preload() + const transaction = children.collection.update(1, (draft) => { + draft.value = confirmedValue + }) + expect(stripVirtualProperties(live.toArray)).toEqual([ + { id: 1, children: [{ id: 1, value: confirmedValue }] }, + ]) + + children.write(`update`, { + id: 1, + parentGroup: 1, + group: 1, + value: confirmedValue, + position: 0, + }) + children.resolveSync() + await transaction.isPersisted.promise + + expect(stripVirtualProperties(live.toArray)).toEqual([ + { id: 1, children: [{ id: 1, value: confirmedValue }] }, + ]) + } finally { + await live.cleanup() + await Promise.all([ + roots.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest.prop([fc.constant(confirmedChildReorderSeed)], { + numRuns: 1, + seed: 2051245230, + })( + `regression seed: confirmed child reorder matches recomputation`, + expectScenarioMatches, + ) + + fcTest.prop([fc.constant(sharedMaterializeSeed)], { + numRuns: 1, + seed: 1685, + })( + `shared scalar materialization preserves the deepest row`, + expectMaterializeScenarioMatches, + ) + + fcTest.prop([fc.constant(`correlation-key-update`)], { + numRuns: 1, + seed: 1658, + })(`parent correlation-key update rematerializes children`, async () => { + const roots = createControlledCollection(`correlation-seed-roots`) + const children = createControlledCollection( + `correlation-seed-children`, + ) + const live = createLiveQueryCollection((q) => + q.from({ root: roots.collection }).select(({ root }) => ({ + id: root.id, + group: root.group, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, root.group)) + .select(({ child }) => ({ id: child.id })), + ), + })), + ) + + try { + await live.preload() + children.write(`insert`, { + id: 1, + parentGroup: 0, + group: 0, + value: 0, + position: 0, + }) + children.write(`insert`, { + id: 2, + parentGroup: 1, + group: 0, + value: 0, + position: 0, + }) + roots.write(`insert`, { id: 1, group: 1, value: 0, position: 0 }) + roots.write(`update`, { id: 1, group: 0, value: 0, position: 0 }) + + expect(stripVirtualProperties(live.toArray)).toEqual([ + { id: 1, group: 0, children: [{ id: 1 }] }, + ]) + } finally { + await live.cleanup() + await Promise.all([ + roots.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }) + + fcTest.prop([fc.constant(`#1454`)], { numRuns: 1, seed: 1454 })( + `alpha-renaming a duplicate sibling alias preserves results`, + async () => { + const roots = createControlledCollection(`alias-seed-roots`, [ + { id: 1, group: 1, value: 0, position: 0 }, + ]) + const issues = createControlledCollection(`alias-seed-issues`, [ + { + id: 10, + parentGroup: 1, + group: 10, + value: 10, + position: 0, + }, + { + id: 11, + parentGroup: 1, + group: 11, + value: 99, + position: 1, + }, + ]) + const tags = createControlledCollection(`alias-seed-tags`, [ + { + id: 20, + parentGroup: 1, + group: 20, + value: 20, + position: 0, + }, + { + id: 21, + parentGroup: 1, + group: 21, + value: 99, + position: 1, + }, + ]) + + try { + const uniqueAliases = await queryOnce((q) => + q.from({ root: roots.collection }).select(({ root }) => ({ + id: root.id, + issues: toArray( + q + .from({ issue: issues.collection }) + .where(({ issue }) => eq(issue.parentGroup, root.group)) + .where(({ issue }) => eq(issue.value, 10)) + .select(({ issue }) => ({ id: issue.id })), + ), + tags: toArray( + q + .from({ tag: tags.collection }) + .where(({ tag }) => eq(tag.parentGroup, root.group)) + .where(({ tag }) => eq(tag.value, 20)) + .select(({ tag }) => ({ id: tag.id })), + ), + })), + ) + const duplicateAliases = await queryOnce((q) => + q.from({ root: roots.collection }).select(({ root }) => ({ + id: root.id, + issues: toArray( + q + .from({ item: issues.collection }) + .where(({ item }) => eq(item.parentGroup, root.group)) + .where(({ item }) => eq(item.value, 10)) + .select(({ item }) => ({ id: item.id })), + ), + tags: toArray( + q + .from({ item: tags.collection }) + .where(({ item }) => eq(item.parentGroup, root.group)) + .where(({ item }) => eq(item.value, 20)) + .select(({ item }) => ({ id: item.id })), + ), + })), + ) + + const expected = [{ id: 1, issues: [{ id: 10 }], tags: [{ id: 20 }] }] + expect(stripVirtualProperties(uniqueAliases)).toEqual(expected) + expect(stripVirtualProperties(duplicateAliases)).toEqual(expected) + } finally { + await Promise.all([ + roots.collection.cleanup(), + issues.collection.cleanup(), + tags.collection.cleanup(), + ]) + } + }, + ) + + fcTest.prop([fc.constant(`#1444`)], { numRuns: 1, seed: 1444 })( + `regression seed: optimistic child reorder matches recomputation`, + async () => { + const roots = createControlledCollection(`order-seed-roots`, [ + { id: 1, group: 1, value: 0, position: 0 }, + ]) + const children = createControlledCollection( + `order-seed-children`, + [ + { + id: 1, + parentGroup: 1, + group: 1, + value: 1, + position: 0, + }, + { + id: 2, + parentGroup: 1, + group: 1, + value: 2, + position: 1, + }, + ], + ) + const live = createLiveQueryCollection((q) => + q.from({ root: roots.collection }).select(({ root }) => ({ + id: root.id, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, root.group)) + .orderBy(({ child }) => child.position) + .select(({ child }) => ({ + id: child.id, + position: child.position, + })), + ), + })), + ) + + try { + await live.preload() + children.collection.update([1, 2], (drafts) => { + drafts[0]!.position = 1 + drafts[1]!.position = 0 + }) + + expect(stripVirtualProperties(live.toArray)).toEqual([ + { + id: 1, + children: [ + { id: 2, position: 0 }, + { id: 1, position: 1 }, + ], + }, + ]) + } finally { + await live.cleanup() + await Promise.all([ + roots.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) +}) diff --git a/packages/db/tests/query/includes-performance.bench.ts b/packages/db/tests/query/includes-performance.bench.ts new file mode 100644 index 0000000000..251a9c2635 --- /dev/null +++ b/packages/db/tests/query/includes-performance.bench.ts @@ -0,0 +1,22 @@ +import { bench, describe } from 'vitest' +import { createNestedCollectionFixture } from './includes-space-oracle-fixture.js' + +describe(`nested Collection materialization`, () => { + bench( + `constructs and preloads the 20-by-2-by-5-by-10 tree`, + async () => { + const fixture = await createNestedCollectionFixture(20) + try { + await fixture.live.preload() + } finally { + await fixture.cleanup() + } + }, + { + iterations: 10, + time: 0, + warmupIterations: 2, + warmupTime: 0, + }, + ) +}) diff --git a/packages/db/tests/query/includes-publication-oracle.test.ts b/packages/db/tests/query/includes-publication-oracle.test.ts new file mode 100644 index 0000000000..3450c8e044 --- /dev/null +++ b/packages/db/tests/query/includes-publication-oracle.test.ts @@ -0,0 +1,531 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect } from 'vitest' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { + createLiveQueryCollection, + eq, + materialize, +} from '../../src/query/index.js' +import { runTrace } from '../trace-runner.js' +import { oraclePropertyOptions } from '../oracle-config.js' +import { flushPromises, withExpectedRejection } from '../utils.js' +import { createControlledCollection } from './includes-oracle-helpers.js' +import type { TraceDriver, TraceProjection } from '../trace-runner.js' + +type ParentRow = { + id: number + group: number + value: number +} + +type ChildRow = { + id: number + parentGroup: number + value: number +} + +type MetadataRow = { + id: number + parentId: number +} + +type PublishedRow = { + item: ParentRow + children: Array + otherChildren: Array +} + +type Q2Shape = `passThrough` | `where` | `orderBy` | `select` +type Q1Shape = `direct` | `joined` + +const initialParent: ParentRow = { id: 1, group: 10, value: 0 } +const initialChild: ChildRow = { id: 100, parentGroup: 10, value: 1 } +const initialChildren: ReadonlyArray = [ + initialChild, + { id: 200, parentGroup: 20, value: 2 }, +] +const initialOtherChildren: ReadonlyArray = [ + { id: 300, parentGroup: 10, value: 3 }, + { id: 400, parentGroup: 20, value: 4 }, +] + +type PublicationAction = + | { type: `parentScalar`; value: number } + | { type: `childScalar`; value: number } + | { type: `parentRoute`; group: number } + | { type: `atomicReplace`; group: number; value: number } + | { type: `optimisticConfirm`; value: number } + | { type: `optimisticRollback`; value: number } + | { type: `parentThenChild`; parentValue: number; childValue: number } + +let nextCollectionId = 0 + +function createLayeredQuery( + parents: ReturnType>, + children: ReturnType>, + otherChildren: ReturnType>, + metadata: ReturnType>, + q1Shape: Q1Shape, + q2Shape: Q2Shape, +) { + const q1 = createLiveQueryCollection({ + id: `publication-q1-${nextCollectionId++}`, + query: (q) => { + const source = q.from({ item: parents.collection }) + const parentRows = + q1Shape === `direct` + ? source + : source.join( + { metadata: metadata.collection }, + ({ item, metadata: rowMetadata }) => + eq(item.id, rowMetadata.parentId), + `inner`, + ) + + return parentRows.select(({ item }) => ({ + item, + children: materialize( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, item.group)) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + value: child.value, + })), + ), + otherChildren: materialize( + q + .from({ otherChild: otherChildren.collection }) + .where(({ otherChild }) => eq(otherChild.parentGroup, item.group)) + .orderBy(({ otherChild }) => otherChild.id) + .select(({ otherChild }) => ({ + id: otherChild.id, + parentGroup: otherChild.parentGroup, + value: otherChild.value, + })), + ), + })) + }, + getKey: (row) => row.item.id, + }) + const id = `publication-q2-${nextCollectionId++}` + const q2 = (() => { + switch (q2Shape) { + case `passThrough`: + return createLiveQueryCollection({ + id, + query: (q) => q.from({ row: q1 }), + getKey: (row) => row.item.id, + }) + case `where`: + return createLiveQueryCollection({ + id, + query: (q) => + q + .from({ row: q1 }) + .where(({ row }) => eq(row.item.id, initialParent.id)), + getKey: (row) => row.item.id, + }) + case `orderBy`: + return createLiveQueryCollection({ + id, + query: (q) => + q.from({ row: q1 }).orderBy(({ row }) => row.item.value), + getKey: (row) => row.item.id, + }) + case `select`: + return createLiveQueryCollection({ + id, + query: (q) => + q.from({ row: q1 }).select(({ row }) => ({ + item: row.item, + children: row.children, + otherChildren: row.otherChildren, + })), + getKey: (row) => row.item.id, + }) + } + })() + return { q1, q2 } +} + +function stripVirtualProperties(value: unknown): unknown { + if (Array.isArray(value)) return value.map(stripVirtualProperties) + if (!value || typeof value !== `object`) return value + + return Object.fromEntries( + Object.entries(value) + .filter(([key]) => !key.startsWith(`$`)) + .map(([key, entry]) => [key, stripVirtualProperties(entry)]), + ) +} + +type PublicationObservation = { + q1: Array + q2: Array +} + +type PublicationContext = { + sources: { + parents: ReturnType> + children: ReturnType> + otherChildren: ReturnType> + metadata: ReturnType> + } + queries: ReturnType + model: { + parents: Map + children: Map + otherChildren: Map + } +} + +function recomputeRows(context: PublicationContext): Array { + return [...context.model.parents.values()] + .sort((left, right) => left.id - right.id) + .map((parent) => ({ + item: { ...parent }, + children: [...context.model.children.values()] + .filter((child) => child.parentGroup === parent.group) + .sort((left, right) => left.id - right.id) + .map((child) => ({ ...child })), + otherChildren: [...context.model.otherChildren.values()] + .filter((child) => child.parentGroup === parent.group) + .sort((left, right) => left.id - right.id) + .map((child) => ({ ...child })), + })) +} + +const publicationProjection: TraceProjection< + PublicationContext, + PublicationObservation +> = { + observe: ({ queries }) => ({ + q1: stripVirtualProperties(queries.q1.toArray) as Array, + q2: stripVirtualProperties(queries.q2.toArray) as Array, + }), + recompute: (context) => { + const expected = recomputeRows(context) + return { + q1: expected.map((row) => structuredClone(row)), + q2: expected.map((row) => structuredClone(row)), + } + }, + assertEqual: (observed, expected) => { + expect(observed).toEqual(expected) + return undefined + }, +} + +async function settleRollback( + rejectSync: (error: Error) => void, + persisted: Promise, +): Promise { + const message = `publication oracle rollback` + const outcome = persisted.catch(() => undefined) + await withExpectedRejection(message, async () => { + rejectSync(new Error(message)) + await outcome + await flushPromises() + }) +} + +function createPublicationDriver( + q1Shape: Q1Shape, + q2Shape: Q2Shape, + checkpointOptimistic = false, +): TraceDriver { + return { + setup: () => { + const parents = createControlledCollection(`publication-parents`, [ + initialParent, + ]) + const children = createControlledCollection( + `publication-children`, + initialChildren, + ) + const otherChildren = createControlledCollection( + `publication-other-children`, + initialOtherChildren, + ) + const metadata = createControlledCollection(`publication-metadata`, [ + { id: initialParent.id, parentId: initialParent.id }, + ]) + if (q1Shape === `joined`) { + parents.collection.createIndex((row) => row.id, { + indexType: BasicIndex, + }) + } + return { + sources: { parents, children, otherChildren, metadata }, + queries: createLayeredQuery( + parents, + children, + otherChildren, + metadata, + q1Shape, + q2Shape, + ), + model: { + parents: new Map([[initialParent.id, { ...initialParent }]]), + children: new Map( + initialChildren.map((child) => [child.id, { ...child }]), + ), + otherChildren: new Map( + initialOtherChildren.map((child) => [child.id, { ...child }]), + ), + }, + } + }, + start: async ({ queries }) => { + await queries.q1.preload() + await queries.q2.preload() + }, + apply: async (action, context, checkpoint) => { + if (action.type === `parentThenChild`) { + const parent = context.model.parents.get(initialParent.id) + const child = context.model.children.get(initialChild.id) + if (!parent || !child) throw new Error(`Missing publication fixture`) + + const nextParent = { ...parent, value: action.parentValue } + context.sources.parents.write(`update`, nextParent) + context.model.parents.set(nextParent.id, { ...nextParent }) + + checkpoint() + + const nextChild = { ...child, value: action.childValue } + context.sources.children.write(`update`, nextChild) + context.model.children.set(nextChild.id, { ...nextChild }) + return + } + + if (action.type === `childScalar`) { + const currentChild = context.model.children.get(initialChild.id) + if (!currentChild) throw new Error(`Missing publication child`) + const nextChild = { ...currentChild, value: action.value } + context.sources.children.write(`update`, nextChild) + context.model.children.set(nextChild.id, { ...nextChild }) + return + } + + const current = context.model.parents.get(initialParent.id) + if (!current) throw new Error(`Missing publication parent`) + + const next: ParentRow = { + ...current, + group: + action.type === `parentRoute` || action.type === `atomicReplace` + ? action.group + : current.group, + value: `value` in action ? action.value : current.value, + } + + if (action.type === `atomicReplace`) { + context.sources.parents.writeBatch([ + { type: `delete`, value: { ...current } }, + { type: `insert`, value: { ...next } }, + ]) + context.model.parents.set(next.id, { ...next }) + return + } + + if ( + action.type === `optimisticConfirm` || + action.type === `optimisticRollback` + ) { + const transaction = context.sources.parents.collection.update( + next.id, + (draft) => { + draft.value = next.value + }, + ) + const previous = { ...current } + context.model.parents.set(next.id, { ...next }) + + let optimisticFailure: unknown + if (checkpointOptimistic) { + try { + checkpoint() + } catch (error) { + optimisticFailure = error + } + } + + if (action.type === `optimisticConfirm`) { + context.sources.parents.write(`update`, next) + context.sources.parents.resolveSync() + await transaction.isPersisted.promise + } else { + await settleRollback( + context.sources.parents.rejectSync, + transaction.isPersisted.promise, + ) + context.model.parents.set(previous.id, previous) + } + + if (optimisticFailure) throw optimisticFailure + return + } + + context.sources.parents.write(`update`, next) + context.model.parents.set(next.id, { ...next }) + }, + cleanup: async ({ queries, sources }) => { + await queries.q2.cleanup() + await queries.q1.cleanup() + await Promise.all([ + sources.parents.collection.cleanup(), + sources.children.collection.cleanup(), + sources.otherChildren.collection.cleanup(), + sources.metadata.collection.cleanup(), + ]) + }, + } +} + +async function expectPublicationMatches( + action: PublicationAction, + checkpointOptimistic = false, + q1Shape: Q1Shape = `direct`, + q2Shape: Q2Shape = `passThrough`, +): Promise { + await runTrace({ + steps: [action], + driver: createPublicationDriver(q1Shape, q2Shape, checkpointOptimistic), + projection: publicationProjection, + }) +} + +const q2Shapes = [`passThrough`, `where`, `orderBy`, `select`] as const +const q1Shapes = [`direct`, `joined`] as const + +describe(`layered-query publication oracle`, () => { + const changedValueArbitrary = fc.oneof( + fc.integer({ min: -100, max: -1 }), + fc.integer({ min: 1, max: 100 }), + ) + const changedChildValueArbitrary = fc.oneof( + fc.integer({ min: -100, max: 0 }), + fc.integer({ min: 2, max: 100 }), + ) + + for (const q1Shape of q1Shapes) { + for (const q2Shape of q2Shapes) { + fcTest.prop( + [changedValueArbitrary], + oraclePropertyOptions( + 12, + `includes-publication.parent-scalar.${q1Shape}.${q2Shape}`, + ), + )( + `publishes parent scalar updates through a ${q1Shape} Q1 and ${q2Shape} Q2`, + async (value) => { + await expectPublicationMatches( + { type: `parentScalar`, value }, + false, + q1Shape, + q2Shape, + ) + }, + ) + + fcTest.prop( + [changedValueArbitrary, changedChildValueArbitrary], + oraclePropertyOptions( + 12, + `includes-publication.parent-then-child.${q1Shape}.${q2Shape}`, + ), + )( + `recovers a ${q1Shape} Q1 and ${q2Shape} Q2 after a child update`, + async (parentValue, childValue) => { + await expectPublicationMatches( + { type: `parentThenChild`, parentValue, childValue }, + false, + q1Shape, + q2Shape, + ) + }, + ) + + fcTest.prop( + [changedValueArbitrary], + oraclePropertyOptions( + 8, + `includes-publication.optimistic-before-confirm.${q1Shape}.${q2Shape}`, + ), + )( + `publishes optimistic state before confirmation through a ${q1Shape} Q1 and ${q2Shape} Q2`, + async (value) => { + await expectPublicationMatches( + { type: `optimisticConfirm`, value }, + true, + q1Shape, + q2Shape, + ) + }, + ) + + fcTest.prop( + [changedValueArbitrary], + oraclePropertyOptions( + 8, + `includes-publication.optimistic-after-confirm.${q1Shape}.${q2Shape}`, + ), + )( + `publishes state after optimistic confirmation through a ${q1Shape} Q1 and ${q2Shape} Q2`, + async (value) => { + await expectPublicationMatches( + { type: `optimisticConfirm`, value }, + false, + q1Shape, + q2Shape, + ) + }, + ) + } + } + + fcTest.prop( + [changedChildValueArbitrary], + oraclePropertyOptions(100, `includes-publication.child-scalar`), + )( + `publishes child-only scalar updates through both layers`, + async (value) => { + await expectPublicationMatches({ type: `childScalar`, value }) + }, + ) + + fcTest.prop( + [fc.constantFrom(20, 30)], + oraclePropertyOptions(100, `includes-publication.parent-route`), + )(`compares route transitions at both query layers`, async (group) => { + await expectPublicationMatches({ type: `parentRoute`, group }) + }) + + fcTest.prop( + [ + fc.record({ + group: fc.constantFrom(10, 20, 30), + value: changedValueArbitrary, + }), + ], + oraclePropertyOptions( + 100, + `includes-publication.atomic-parent-replacement`, + ), + )(`compares atomic parent replacements at both query layers`, async (row) => { + await expectPublicationMatches({ type: `atomicReplace`, ...row }) + }) + + fcTest.prop( + [changedValueArbitrary], + oraclePropertyOptions(100, `includes-publication.optimistic-rollback`), + )(`publishes restored state after optimistic rollback`, async (value) => { + await expectPublicationMatches({ + type: `optimisticRollback`, + value, + }) + }) +}) diff --git a/packages/db/tests/query/includes-query-shape-oracle.test.ts b/packages/db/tests/query/includes-query-shape-oracle.test.ts new file mode 100644 index 0000000000..cbf342402c --- /dev/null +++ b/packages/db/tests/query/includes-query-shape-oracle.test.ts @@ -0,0 +1,453 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect } from 'vitest' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { + createLiveQueryCollection, + eq, + materialize, +} from '../../src/query/index.js' +import { runTrace } from '../trace-runner.js' +import { oracleRuns } from '../oracle-config.js' +import { createControlledCollection } from './includes-oracle-helpers.js' +import type { TraceDriver, TraceProjection } from '../trace-runner.js' + +function rowsById(rows: Array): Map { + return new Map(rows.map((row) => [row.id, row])) +} + +function stripVirtualProperties(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map(stripVirtualProperties) + } + if (!value || typeof value !== `object`) { + return value + } + + return Object.fromEntries( + Object.entries(value) + .filter(([key]) => !key.startsWith(`$`)) + .map(([key, entry]) => [key, stripVirtualProperties(entry)]), + ) +} + +function assertRowsEqual(observed: unknown, expected: unknown): undefined { + expect(observed).toEqual(expected) + return undefined +} + +type Cleanable = { cleanup: () => Promise } + +async function cleanupQuery( + live: Cleanable, + sources: Array<{ collection: Cleanable }>, +): Promise { + await live.cleanup() + await Promise.all(sources.map(({ collection }) => collection.cleanup())) +} + +type ParentRow = { id: number } +type ChildRow = { id: number; parentId: number } + +type MultiplicitySources = ReturnType + +function createChildren(childCount: number): Array { + return Array.from({ length: childCount }, (_, index) => ({ + id: index + 1, + parentId: 1, + })) +} + +function createMultiplicitySources(childCount: number) { + const sources = { + parents: createControlledCollection(`join-parents`, [{ id: 1 }]), + children: createControlledCollection( + `join-children`, + createChildren(childCount), + ), + } + sources.parents.collection.createIndex((row) => row.id, { + indexType: BasicIndex, + }) + sources.children.collection.createIndex((row) => row.parentId, { + indexType: BasicIndex, + }) + return sources +} + +function createMultiplicityQuery(sources: MultiplicitySources) { + return createLiveQueryCollection({ + query: (q) => + q + .from({ parent: sources.parents.collection }) + .innerJoin( + { child: sources.children.collection }, + ({ parent, child }) => eq(parent.id, child.parentId), + ) + .select(({ parent }) => ({ id: parent.id })), + getKey: (row) => row.id, + }) +} + +type MultiplicityContext = { + sources: MultiplicitySources + live: ReturnType + parents: Map + children: Map +} + +function createMultiplicityDriver( + childCount: number, +): TraceDriver { + return { + setup: () => { + const parents = [{ id: 1 }] + const children = createChildren(childCount) + const sources = createMultiplicitySources(childCount) + return { + sources, + live: createMultiplicityQuery(sources), + parents: rowsById(parents), + children: rowsById(children), + } + }, + start: ({ live }) => live.preload(), + apply: (childId, { children, sources }) => { + const child = children.get(childId) + if (!child) throw new Error(`Missing child ${childId}`) + sources.children.write(`delete`, child) + children.delete(childId) + }, + cleanup: ({ live, sources }) => cleanupQuery(live, Object.values(sources)), + } +} + +const multiplicityProjection: TraceProjection< + MultiplicityContext, + unknown, + Array +> = { + observe: ({ live }) => stripVirtualProperties(live.toArray), + recompute: ({ children, parents }) => + [...parents.values()] + .filter((parent) => + [...children.values()].some((child) => child.parentId === parent.id), + ) + .sort((left, right) => left.id - right.id), + assertEqual: assertRowsEqual, +} + +type PartRow = { id: number } +type OrderRow = { id: number; partId: number } +type ProductionRow = { id: number; orderId: number } +type CorrelationTarget = `source` | `joined` + +function createCorrelationSources(correlationId: number, productionId: number) { + const sources = { + parts: createControlledCollection(`correlation-parts`, [ + { id: correlationId }, + ]), + orders: createControlledCollection(`correlation-orders`, [ + { id: correlationId, partId: correlationId }, + ]), + productions: createControlledCollection( + `correlation-productions`, + [{ id: productionId, orderId: correlationId }], + ), + } + sources.orders.collection.createIndex((row) => row.id, { + indexType: BasicIndex, + }) + sources.orders.collection.createIndex((row) => row.partId, { + indexType: BasicIndex, + }) + sources.productions.collection.createIndex((row) => row.orderId, { + indexType: BasicIndex, + }) + return sources +} + +type CorrelationSources = ReturnType + +function createCorrelationQuery( + sources: CorrelationSources, + target: CorrelationTarget, +) { + return createLiveQueryCollection((q) => + q + .from({ part: sources.parts.collection }) + .orderBy(({ part }) => part.id) + .select(({ part }) => { + const joined = q + .from({ production: sources.productions.collection }) + .innerJoin( + { order: sources.orders.collection }, + ({ production, order }) => eq(production.orderId, order.id), + ) + const correlated = + target === `joined` + ? joined.where(({ order }) => eq(order.partId, part.id)) + : joined.where(({ production }) => eq(production.orderId, part.id)) + + return { + id: part.id, + productions: materialize( + correlated + .orderBy(({ production }) => production.id) + .select(({ production }) => ({ + id: production.id, + orderId: production.orderId, + })), + ), + } + }), + ) +} + +type CorrelationContext = { + target: CorrelationTarget + sources: CorrelationSources + live: ReturnType + parts: Map + orders: Map + productions: Map +} + +function createCorrelationDriver( + target: CorrelationTarget, + correlationId: number, + productionId: number, +): TraceDriver { + return { + setup: () => { + const part = { id: correlationId } + const order = { id: correlationId, partId: correlationId } + const production = { id: productionId, orderId: correlationId } + const sources = createCorrelationSources(correlationId, productionId) + return { + target, + sources, + live: createCorrelationQuery(sources, target), + parts: rowsById([part]), + orders: rowsById([order]), + productions: rowsById([production]), + } + }, + start: ({ live }) => live.preload(), + apply: () => undefined, + cleanup: ({ live, sources }) => cleanupQuery(live, Object.values(sources)), + } +} + +type CorrelationResult = Array<{ + id: number + productions: Array +}> + +const correlationProjection: TraceProjection< + CorrelationContext, + unknown, + CorrelationResult +> = { + observe: ({ live }) => stripVirtualProperties(live.toArray), + recompute: ({ orders, parts, productions, target }) => + [...parts.values()] + .sort((left, right) => left.id - right.id) + .map((part) => ({ + id: part.id, + productions: [...productions.values()] + .filter((production) => { + const order = orders.get(production.orderId) + if (!order) return false + return target === `joined` + ? order.partId === part.id + : production.orderId === part.id + }) + .sort((left, right) => left.id - right.id), + })), + assertEqual: assertRowsEqual, +} + +type AuthorRow = { id: number; name: string } +type PostRow = { id: number; authorId: number | null } + +function createNullableSources( + authors: Array, + posts: Array, +) { + return { + authors: createControlledCollection(`nullable-authors`, authors), + posts: createControlledCollection(`nullable-posts`, posts), + } +} + +type NullableSources = ReturnType + +function createNullableQuery(sources: NullableSources) { + return createLiveQueryCollection((q) => + q + .from({ post: sources.posts.collection }) + .orderBy(({ post }) => post.id) + .select(({ post }) => ({ + id: post.id, + author: materialize( + q + .from({ author: sources.authors.collection }) + .where(({ author }) => eq(author.id, post.authorId)) + .select(({ author }) => ({ + id: author.id, + name: author.name, + })) + .findOne(), + ), + })), + ) +} + +type NullableContext = { + sources: NullableSources + live: ReturnType + authors: Map + posts: Map +} + +function createNullableDriver( + authors: Array, + posts: Array, +): TraceDriver { + return { + setup: () => { + const sources = createNullableSources( + authors.map((author) => ({ ...author })), + posts.map((post) => ({ ...post })), + ) + return { + sources, + live: createNullableQuery(sources), + authors: rowsById(authors), + posts: rowsById(posts), + } + }, + start: ({ live }) => live.preload(), + apply: (post, context) => { + context.sources.posts.write( + context.posts.has(post.id) ? `update` : `insert`, + { ...post }, + ) + context.posts.set(post.id, post) + }, + cleanup: ({ live, sources }) => cleanupQuery(live, Object.values(sources)), + } +} + +type NullableResult = Array<{ + id: number + author: AuthorRow | undefined +}> + +const nullableProjection: TraceProjection< + NullableContext, + unknown, + NullableResult +> = { + observe: ({ live }) => stripVirtualProperties(live.toArray), + recompute: ({ authors, posts }) => + [...posts.values()] + .sort((left, right) => left.id - right.id) + .map((post) => ({ + id: post.id, + author: post.authorId === null ? undefined : authors.get(post.authorId), + })), + assertEqual: assertRowsEqual, +} + +describe(`includes query-shape recompute oracle`, () => { + fcTest.prop([fc.integer({ min: 2, max: 5 })], { + numRuns: oracleRuns(12), + seed: 1703, + })( + `deleting one joined contributor preserves remaining multiplicity (#1703)`, + async (childCount) => { + await runTrace({ + steps: [1], + driver: createMultiplicityDriver(childCount), + projection: multiplicityProjection, + }) + }, + ) + + fcTest( + `matches recomputation when the final joined contributor is deleted`, + () => + runTrace({ + steps: [1], + driver: createMultiplicityDriver(1), + projection: multiplicityProjection, + }), + ) + + fcTest.prop( + [ + fc.record({ + correlationId: fc.integer({ min: 1, max: 100 }), + productionId: fc.integer({ min: 101, max: 200 }), + }), + ], + { numRuns: oracleRuns(12), seed: 1704 }, + )( + `materialization follows correlation through a joined alias (#1704)`, + async ({ correlationId, productionId }) => { + await runTrace({ + steps: [], + driver: createCorrelationDriver(`joined`, correlationId, productionId), + projection: correlationProjection, + }) + }, + ) + + fcTest( + `matches recomputation when materialization correlates through its source alias`, + () => + runTrace({ + steps: [], + driver: createCorrelationDriver(`source`, 1, 101), + projection: correlationProjection, + }), + ) + + fcTest.prop([fc.integer({ min: 1, max: 100 })], { + numRuns: oracleRuns(12), + seed: 1706, + })( + `findOne maps a null correlation key to undefined (#1706)`, + async (postId) => { + await runTrace({ + steps: [], + driver: createNullableDriver([], [{ id: postId, authorId: null }]), + projection: nullableProjection, + }) + }, + ) + + fcTest( + `matches recomputation for an unmatched non-null correlation key`, + () => + runTrace({ + steps: [], + driver: createNullableDriver([], [{ id: 1, authorId: 999 }]), + projection: nullableProjection, + }), + ) + + fcTest( + `matches recomputation when an existing correlation key becomes null`, + () => + runTrace({ + steps: [{ id: 1, authorId: null }], + driver: createNullableDriver( + [{ id: 1, name: `Ada` }], + [{ id: 1, authorId: 1 }], + ), + projection: nullableProjection, + }), + ) +}) diff --git a/packages/db/tests/query/includes-space-oracle-fixture.ts b/packages/db/tests/query/includes-space-oracle-fixture.ts new file mode 100644 index 0000000000..cc52961b73 --- /dev/null +++ b/packages/db/tests/query/includes-space-oracle-fixture.ts @@ -0,0 +1,105 @@ +import { createCollection } from '../../src/collection/index.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { localOnlyCollectionOptions } from '../../src/local-only.js' +import { createLiveQueryCollection, eq } from '../../src/query/index.js' + +type RootRow = { id: string } +type BranchRow = { id: string; rootId: string } +type TwigRow = { id: string; branchId: string } +type LeafRow = { id: string; twigId: string } + +let fixtureId = 0 + +function createRows(rootCount: number) { + const roots: Array = [] + const branches: Array = [] + const twigs: Array = [] + const leaves: Array = [] + + for (let rootIndex = 0; rootIndex < rootCount; rootIndex++) { + const rootId = `root-${rootIndex}` + roots.push({ id: rootId }) + for (let branchIndex = 0; branchIndex < 2; branchIndex++) { + const branchId = `branch-${rootIndex}-${branchIndex}` + branches.push({ id: branchId, rootId }) + for (let twigIndex = 0; twigIndex < 5; twigIndex++) { + const twigId = `twig-${rootIndex}-${branchIndex}-${twigIndex}` + twigs.push({ id: twigId, branchId }) + for (let leafIndex = 0; leafIndex < 10; leafIndex++) { + leaves.push({ + id: `leaf-${rootIndex}-${branchIndex}-${twigIndex}-${leafIndex}`, + twigId, + }) + } + } + } + } + + return { roots, branches, twigs, leaves } +} + +function source(name: string, rows: Array) { + return createCollection( + localOnlyCollectionOptions({ + id: `includes-space-${fixtureId}-${name}`, + getKey: (row: T) => row.id, + initialData: rows, + }), + ) +} + +export async function createNestedCollectionFixture(rootCount: number) { + fixtureId++ + const rows = createRows(rootCount) + const sources = { + roots: source(`roots`, rows.roots), + branches: source(`branches`, rows.branches), + twigs: source(`twigs`, rows.twigs), + leaves: source(`leaves`, rows.leaves), + } + + await Promise.all( + Object.values(sources).map((collection) => collection.preload()), + ) + sources.branches.createIndex((row) => row.rootId, { indexType: BTreeIndex }) + sources.twigs.createIndex((row) => row.branchId, { indexType: BTreeIndex }) + sources.leaves.createIndex((row) => row.twigId, { indexType: BTreeIndex }) + + const live = createLiveQueryCollection((q) => + q.from({ root: sources.roots }).select(({ root }) => ({ + id: root.id, + branches: q + .from({ branch: sources.branches }) + .where(({ branch }) => eq(branch.rootId, root.id)) + .select(({ branch }) => ({ + id: branch.id, + twigs: q + .from({ twig: sources.twigs }) + .where(({ twig }) => eq(twig.branchId, branch.id)) + .select(({ twig }) => ({ + id: twig.id, + leaves: q + .from({ leaf: sources.leaves }) + .where(({ leaf }) => eq(leaf.twigId, twig.id)) + .select(({ leaf }) => ({ id: leaf.id })), + })), + })), + })), + ) + + return { + live, + expectedFacadeCount: rootCount + rootCount * 2 + rootCount * 2 * 5, + cleanup: async () => { + const results = await Promise.allSettled([ + live.cleanup(), + ...Object.values(sources).map((collection) => collection.cleanup()), + ]) + const rejection = results.find( + (result): result is PromiseRejectedResult => + result.status === `rejected`, + ) + if (rejection) throw rejection.reason + }, + } +} diff --git a/packages/db/tests/query/includes-space-oracle.test.ts b/packages/db/tests/query/includes-space-oracle.test.ts new file mode 100644 index 0000000000..e1873d94b7 --- /dev/null +++ b/packages/db/tests/query/includes-space-oracle.test.ts @@ -0,0 +1,56 @@ +import { describe, expect, it, vi } from 'vitest' +import { BucketFacadeAdapter } from '../../src/query/live/bucket-facade-adapter.js' +import { createNestedCollectionFixture } from './includes-space-oracle-fixture.js' + +// Inspect retained adapter state only in tests; diagnostics need no runtime API. +type AdapterState = { + getEntry: (...args: Array) => object + entries: Map> + retiredEntries: Map> +} + +function countEntries(entries: AdapterState[`entries`]): number { + return [...entries.values()].reduce( + (total, buckets) => total + buckets.size, + 0, + ) +} + +describe(`nested Collection materialization space oracle`, () => { + it(`constructs exactly one facade per reachable bucket`, async () => { + const entries = vi.spyOn( + BucketFacadeAdapter.prototype as unknown as AdapterState, + `getEntry`, + ) + const fixture = await createNestedCollectionFixture(20) + try { + await fixture.live.preload() + + const adapters = new Set(entries.mock.contexts as Array) + const created = new Set( + entries.mock.results + .filter((result) => result.type === `return`) + .map((result) => result.value), + ) + expect(created.size).toBe(fixture.expectedFacadeCount) + expect( + [...adapters].reduce( + (n, adapter) => n + countEntries(adapter.entries), + 0, + ), + ).toBe(fixture.expectedFacadeCount) + expect( + [...adapters].reduce( + (n, adapter) => n + countEntries(adapter.retiredEntries), + 0, + ), + ).toBe(0) + } finally { + try { + await fixture.cleanup() + } finally { + entries.mockRestore() + } + } + }) +}) diff --git a/packages/db/tests/query/includes-temporal-oracle.test.ts b/packages/db/tests/query/includes-temporal-oracle.test.ts new file mode 100644 index 0000000000..e54c146196 --- /dev/null +++ b/packages/db/tests/query/includes-temporal-oracle.test.ts @@ -0,0 +1,1570 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { extractSimpleComparisons } from '../../src/query/expression-helpers.js' +import { SubsetDemandController } from '../../src/query/live/subset-demand-controller.js' +import { DeduplicatedLoadSubset } from '../../src/query/subset-dedupe.js' +import { + createLiveQueryCollection, + eq, + materialize, + toArray, +} from '../../src/query/index.js' +import { runTrace } from '../trace-runner.js' +import { oraclePropertyOptions } from '../oracle-config.js' +import { flushPromises } from '../utils.js' +import type { Collection } from '../../src/collection/index.js' +import type { Deferred } from '../../src/deferred.js' +import type { LoadSubsetOptions } from '../../src/types.js' +import type { LazyDemandPlan } from '../../src/query/compiler/joins.js' +import type { TraceDriver, TraceProjection } from '../trace-runner.js' +import type { Scheduler } from 'fast-check' + +type Post = { + id: number + authorId: string + title: string +} + +type Comment = { + id: number + postId: number + body: string +} + +type User = { + id: number + name: string +} + +type ProgressivePost = { + id: number + userId: number + title: string +} + +let collectionId = 0 + +function nextCollectionId(prefix: string): string { + collectionId += 1 + return `${prefix}-${collectionId}` +} + +type PreloadState = { + preloadFailure?: { error: unknown } + preloadOutcome?: Promise + preloadSettled: boolean +} + +function startPreload( + live: ReturnType, + state: PreloadState, +): Promise { + const preload = live.preload() + state.preloadOutcome = preload.then( + () => { + state.preloadSettled = true + }, + (error) => { + state.preloadFailure = { error } + state.preloadSettled = true + }, + ) + return preload +} + +async function finishPreload(state: PreloadState): Promise { + await state.preloadOutcome + if (state.preloadFailure) throw state.preloadFailure.error +} + +function correlationKeys( + loads: ReadonlyArray, + field: string, +): Array { + return [ + ...new Set( + loads.flatMap((load) => + extractSimpleComparisons(load.where).flatMap((filter) => { + if (filter.field[0] !== field) return [] + if (filter.operator === `eq` && typeof filter.value === `number`) { + return [filter.value] + } + if (filter.operator !== `in` || !Array.isArray(filter.value)) { + return [] + } + return filter.value.filter( + (value): value is number => typeof value === `number`, + ) + }), + ), + ), + ].sort((left, right) => left - right) +} + +function createColdPosts(initial: ReadonlyArray): { + collection: Collection + loaded: Deferred +} { + const loaded = createDeferred() + const collection = createCollection({ + id: nextCollectionId(`temporal-posts`), + getKey: (post) => post.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: () => { + begin() + for (const post of initial) { + write({ type: `insert`, value: post }) + } + commit() + markReady() + loaded.resolve() + return Promise.resolve() + }, + }), + }, + }) + return { collection, loaded } +} + +function createColdComments(): { + collection: Collection + loads: Array +} { + const loads: Array = [] + const comments: Array = [ + { id: 100, postId: 1, body: `one` }, + { id: 200, postId: 2, body: `two` }, + ] + const collection = createCollection({ + id: nextCollectionId(`temporal-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: (options) => { + loads.push(options) + const requested = new Set(correlationKeys([options], `postId`)) + begin() + for (const comment of comments) { + if (requested.has(comment.postId)) { + write({ type: `insert`, value: comment }) + } + } + commit() + markReady() + return Promise.resolve() + }, + }), + }, + }) + return { collection, loads } +} + +it.each( + ([`array`, `materialized`] as const).flatMap((form) => + ([`expression`, `functional`] as const).map((projection) => ({ + form, + projection, + })), + ), +)( + `$form / $projection preserves child demand and applied settlement across projection`, + async ({ form, projection }) => { + const posts = createColdPosts([{ id: 1, authorId: `one`, title: `post` }]) + const started = createDeferred() + const release = createDeferred() + const loads: Array = [] + const comments = createCollection({ + id: nextCollectionId(`projection-pending-comments`), + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: (options) => { + loads.push(options) + const keys = correlationKeys([options], `postId`) + started.resolve() + return release.promise.then(async () => { + if (options.signal?.aborted) return + begin() + if (keys.includes(1)) + write({ + type: `insert`, + value: { id: 100, postId: 1, body: `one` }, + }) + await commit() + markReady() + }) + }, + }), + }, + }) + const live = createLiveQueryCollection((q) => { + const included = q.from({ post: posts.collection }).select(({ post }) => { + const childRows = q + .from({ comment: comments }) + .where(({ comment }) => eq(comment.postId, post.id)) + return { + id: post.id, + comments: + form === `array` ? toArray(childRows) : materialize(childRows), + count: 0, + } + }) + const outer = q.from({ row: included }) + return projection === `expression` + ? outer.select(({ row }) => row) + : outer.fn.select(({ row }) => { + expect + .soft( + Array.isArray(row.comments), + `callback receives an inline value`, + ) + .toBe(true) + return { + id: row.id, + comments: row.comments, + count: Array.isArray(row.comments) ? row.comments.length : -1, + } + }) + }) + let settled = false + const preload = live.preload() + const observed = preload.then( + () => { + settled = true + }, + () => { + settled = true + }, + ) + try { + await Promise.race([started.promise, preload]) + expect(loads).toHaveLength(1) + expect(correlationKeys(loads, `postId`)).toEqual([1]) + expect(settled).toBe(false) + release.resolve() + await preload + expect(live.toArray).toHaveLength(1) + // Observe the runtime boundary: a broken projection can omit this value. + const publishedComments = live.toArray[0]?.comments as unknown as + | Array + | undefined + expect( + publishedComments?.map(({ id, postId, body }) => ({ + id, + postId, + body, + })), + ).toEqual([{ id: 100, postId: 1, body: `one` }]) + if (projection === `functional`) expect(live.toArray[0]?.count).toBe(1) + } finally { + release.resolve() + await live.cleanup() + await observed + await posts.collection.cleanup() + await comments.cleanup() + } + }, +) + +type ReadinessObservation = { + ready: boolean + preloadSettled: boolean + rowCount: number + childLoadCount: number + loadedPostIds: Array +} + +type ReadinessContext = { + posts: Collection + comments: Collection + live: ReturnType + loads: Array + preload: PreloadState + parentLoaded: Deferred + expected: ReadinessObservation +} + +function createReadinessDriver( + initialPosts: ReadonlyArray, +): TraceDriver { + return { + setup: () => { + const { collection: postCollection, loaded: parentLoaded } = + createColdPosts(initialPosts) + const { collection: comments, loads } = createColdComments() + const live = createLiveQueryCollection((q) => + q + .from({ post: postCollection }) + .where(({ post }) => eq(post.authorId, `selected`)) + .select(({ post }) => ({ + id: post.id, + comments: toArray( + q + .from({ comment: comments }) + .where(({ comment }) => eq(comment.postId, post.id)), + ), + })), + ) + + return { + posts: postCollection, + comments, + live, + loads, + preload: { preloadSettled: false }, + parentLoaded, + expected: { + ready: true, + preloadSettled: true, + rowCount: initialPosts.length, + childLoadCount: initialPosts.length === 0 ? 0 : 1, + loadedPostIds: initialPosts.map(({ id }) => id), + }, + } + }, + start: async (context) => { + const preload = startPreload(context.live, context.preload) + await context.parentLoaded.promise + await preload + }, + apply: () => undefined, + cleanup: async ({ posts, comments, live, preload }) => { + await live.cleanup() + await finishPreload(preload) + await Promise.all([posts.cleanup(), comments.cleanup()]) + }, + } +} + +const readinessProjection: TraceProjection< + ReadinessContext, + ReadinessObservation +> = { + observe: ({ live, loads, preload }) => ({ + ready: live.isReady(), + preloadSettled: preload.preloadSettled, + rowCount: live.size, + childLoadCount: loads.length, + loadedPostIds: correlationKeys(loads, `postId`), + }), + recompute: ({ expected }) => expected, + assertEqual: (observed, expected) => { + expect(observed).toEqual(expected) + return undefined + }, +} + +async function expectReadinessMatches( + posts: ReadonlyArray, +): Promise { + await runTrace({ + steps: [], + driver: createReadinessDriver(posts), + projection: readinessProjection, + }) +} + +type DemandCancellationObservation = { + ready: boolean + rowCount: number + childLoadStarted: boolean + childLoadPending: boolean +} + +type DemandCancellationContext = { + posts: Collection + comments: Collection + live: ReturnType + removePost: () => void + childLoad: ReturnType> + childLoadStarted: Deferred + preload: PreloadState + expected: DemandCancellationObservation +} + +function createRemovablePost(): { + collection: Collection + remove: () => void + add: () => void +} { + const post: Post = { + id: 1, + authorId: `selected`, + title: `selected`, + } + let remove: () => void = () => { + throw new Error(`Post collection has not started`) + } + let add: () => void = () => { + throw new Error(`Post collection has not started`) + } + const collection = createCollection({ + id: nextCollectionId(`temporal-removable-post`), + getKey: (row) => row.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: post }) + commit() + markReady() + remove = () => { + begin() + write({ type: `delete`, value: post }) + commit() + } + add = () => { + begin() + write({ type: `insert`, value: post }) + commit() + } + }, + }, + }) + return { collection, remove: () => remove(), add: () => add() } +} + +function createDemandCancellationDriver(): TraceDriver< + `remove-parent`, + DemandCancellationContext +> { + return { + setup: () => { + const { collection: posts, remove } = createRemovablePost() + const childLoad = createDeferred() + const childLoadStarted = createDeferred() + const comments = createCollection({ + id: nextCollectionId(`temporal-pending-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + sync: { + sync: () => ({ + loadSubset: () => { + childLoadStarted.resolve() + return childLoad.promise + }, + }), + }, + }) + const live = createLiveQueryCollection((q) => + q.from({ post: posts }).select(({ post }) => ({ + id: post.id, + comments: toArray( + q + .from({ comment: comments }) + .where(({ comment }) => eq(comment.postId, post.id)), + ), + })), + ) + return { + posts, + comments, + live, + removePost: remove, + childLoad, + childLoadStarted, + preload: { preloadSettled: false }, + expected: { + ready: false, + rowCount: 1, + childLoadStarted: true, + childLoadPending: true, + }, + } + }, + start: async (context) => { + startPreload(context.live, context.preload) + await context.childLoadStarted.promise + }, + apply: (_step, context) => { + context.removePost() + context.expected = { + ready: true, + rowCount: 0, + childLoadStarted: true, + childLoadPending: true, + } + }, + cleanup: async ({ posts, comments, live, childLoad, preload }) => { + childLoad.resolve() + await live.cleanup() + await finishPreload(preload) + await Promise.all([posts.cleanup(), comments.cleanup()]) + }, + } +} + +const demandCancellationProjection: TraceProjection< + DemandCancellationContext, + DemandCancellationObservation +> = { + observe: ({ live, childLoadStarted, childLoad }) => ({ + ready: live.isReady(), + rowCount: live.size, + childLoadStarted: !childLoadStarted.isPending(), + childLoadPending: childLoad.isPending(), + }), + recompute: ({ expected }) => expected, + assertEqual: (observed, expected) => { + expect(observed).toEqual(expected) + return undefined + }, +} + +async function expectObsoleteDemandDoesNotBlockReadiness(): Promise { + await runTrace({ + steps: [`remove-parent`], + driver: createDemandCancellationDriver(), + projection: demandCancellationProjection, + }) +} + +async function expectObsoleteDemandCannotPublishAfterReactivation(): Promise { + const { collection: posts, remove, add } = createRemovablePost() + const requests: Array<{ + deferred: Deferred + outcome: Promise + signal: AbortSignal | undefined + }> = [] + const comments = createCollection({ + id: nextCollectionId(`temporal-generation-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: (options) => { + const requestIndex = requests.length + const deferred = createDeferred() + const signal = options.signal + const outcome = deferred.promise.then(() => { + if (signal?.aborted) return + begin() + write({ + type: `insert`, + value: + requestIndex === 0 + ? { id: 100, postId: 1, body: `obsolete` } + : { id: 200, postId: 1, body: `current` }, + }) + commit() + markReady() + }) + requests.push({ deferred, outcome, signal }) + return outcome + }, + }), + }, + }) + const live = createLiveQueryCollection((q) => + q.from({ post: posts }).select(({ post }) => ({ + id: post.id, + comments: toArray( + q + .from({ comment: comments }) + .where(({ comment }) => eq(comment.postId, post.id)) + .select(({ comment }) => ({ + id: comment.id, + body: comment.body, + })), + ), + })), + ) + + const preload = live.preload() + try { + await flushPromises() + expect(requests).toHaveLength(1) + + remove() + await preload + expect(live.size).toBe(0) + + add() + await flushPromises() + expect(requests).toHaveLength(2) + + requests[1]!.deferred.resolve() + await requests[1]!.outcome + await flushPromises() + expect(live.get(1)?.comments).toEqual([{ id: 200, body: `current` }]) + + requests[0]!.deferred.resolve() + await requests[0]!.outcome + await flushPromises() + expect(live.get(1)?.comments).toEqual([{ id: 200, body: `current` }]) + expect(requests[0]!.signal?.aborted).toBe(true) + } finally { + for (const request of requests) request.deferred.resolve() + await Promise.allSettled(requests.map(({ outcome }) => outcome)) + await live.cleanup() + await Promise.all([posts.cleanup(), comments.cleanup()]) + } +} + +async function expectScheduledDemandCompletionsStayGenerationSafe( + scheduler: Scheduler, +): Promise { + const { collection: posts, remove, add } = createRemovablePost() + const requests: Array<{ + outcome: Promise + signal: AbortSignal | undefined + }> = [] + const comments = createCollection({ + id: nextCollectionId(`temporal-scheduled-generation-comments`), + getKey: (comment) => comment.id, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: (options) => { + const requestIndex = requests.length + const signal = options.signal + const outcome = scheduler + .schedule(Promise.resolve(), `demand-${requestIndex}`) + .then(() => { + if (signal?.aborted) return + begin() + write({ + type: `insert`, + value: { + id: requestIndex === 0 ? 100 : 200, + postId: 1, + body: requestIndex === 0 ? `obsolete` : `current`, + }, + }) + commit() + markReady() + }) + requests.push({ outcome, signal }) + return outcome + }, + }), + }, + }) + const live = createPostsWithCommentsLive(posts, comments) + const preload = live.preload() + + try { + await flushPromises() + expect(requests).toHaveLength(1) + + remove() + await preload + add() + await flushPromises() + expect(requests).toHaveLength(2) + expect(requests[0]!.signal?.aborted).toBe(true) + + await scheduler.waitAll() + await Promise.all(requests.map(({ outcome }) => outcome)) + await flushPromises() + + expect(live.isReady()).toBe(true) + expect(live.get(1)?.comments.map(({ id, body }) => ({ id, body }))).toEqual( + [{ id: 200, body: `current` }], + ) + } finally { + if (scheduler.count() > 0) await scheduler.waitAll() + await Promise.allSettled(requests.map(({ outcome }) => outcome)) + await live.cleanup() + await Promise.all([posts.cleanup(), comments.cleanup()]) + } +} + +function createMutablePosts( + initial: ReadonlyArray, + options: { markReadyInitially?: boolean } = {}, +): { + collection: Collection + write: (type: `insert` | `delete`, post: Post) => void + markReady: () => void +} { + let writePost: (type: `insert` | `delete`, post: Post) => void = () => { + throw new Error(`Post collection has not started`) + } + let markPostsReady: () => void = () => { + throw new Error(`Post collection has not started`) + } + const collection = createCollection({ + id: nextCollectionId(`temporal-mutable-posts`), + getKey: (post) => post.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const post of initial) write({ type: `insert`, value: post }) + commit() + if (options.markReadyInitially !== false) markReady() + writePost = (type, post) => { + begin() + write({ type, value: post }) + commit() + } + markPostsReady = markReady + }, + }, + }) + return { + collection, + write: (type, post) => writePost(type, post), + markReady: () => markPostsReady(), + } +} + +function createPendingComments(): { + collection: Collection + requests: Array<{ + deferred: Deferred + outcome: Promise + keys: Array + signal: AbortSignal | undefined + }> +} { + const requests: Array<{ + deferred: Deferred + outcome: Promise + keys: Array + signal: AbortSignal | undefined + }> = [] + const collection = createCollection({ + id: nextCollectionId(`temporal-pending-coverage-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => ({ + loadSubset: (options) => { + const deferred = createDeferred() + const outcome = deferred.promise.then(() => { + if (!options.signal?.aborted) markReady() + }) + requests.push({ + deferred, + outcome, + keys: correlationKeys([options], `postId`), + signal: options.signal, + }) + return outcome + }, + }), + }, + }) + return { collection, requests } +} + +function createPostsWithCommentsLive( + posts: Collection, + comments: Collection, +) { + return createLiveQueryCollection((q) => + q.from({ post: posts }).select(({ post }) => ({ + id: post.id, + comments: toArray( + q + .from({ comment: comments }) + .where(({ comment }) => eq(comment.postId, post.id)), + ), + })), + ) +} + +async function expectRetainedDemandBlocksReadiness(): Promise { + const firstPost = { id: 1, authorId: `selected`, title: `one` } + const secondPost = { id: 2, authorId: `selected`, title: `two` } + const posts = createMutablePosts([firstPost]) + const { collection: comments, requests } = createPendingComments() + const live = createPostsWithCommentsLive(posts.collection, comments) + const preload: PreloadState = { preloadSettled: false } + startPreload(live, preload) + + try { + await flushPromises() + expect(requests.map(({ keys }) => keys)).toEqual([[1]]) + + posts.write(`insert`, secondPost) + await flushPromises() + expect(requests.map(({ keys }) => keys)).toEqual([[1], [2]]) + + requests[1]!.deferred.resolve() + await requests[1]!.outcome + await flushPromises() + expect(preload.preloadSettled).toBe(false) + expect(live.isReady()).toBe(false) + + requests[0]!.deferred.resolve() + await requests[0]!.outcome + await finishPreload(preload) + expect(live.isReady()).toBe(true) + } finally { + for (const request of requests) request.deferred.resolve() + await Promise.allSettled(requests.map(({ outcome }) => outcome)) + await live.cleanup() + await Promise.all([posts.collection.cleanup(), comments.cleanup()]) + } +} + +async function expectObsoleteDemandCannotSettleReactivatedDemand(): Promise { + const post = { id: 1, authorId: `selected`, title: `one` } + const posts = createMutablePosts([post], { markReadyInitially: false }) + const { collection: comments, requests } = createPendingComments() + const live = createPostsWithCommentsLive(posts.collection, comments) + const preload: PreloadState = { preloadSettled: false } + startPreload(live, preload) + + try { + await flushPromises() + expect(requests).toHaveLength(1) + + posts.write(`delete`, post) + posts.write(`insert`, post) + await flushPromises() + expect(requests).toHaveLength(2) + expect(requests[0]!.signal?.aborted).toBe(true) + + requests[0]!.deferred.resolve() + await requests[0]!.outcome + posts.markReady() + await flushPromises() + expect(preload.preloadSettled).toBe(false) + expect(live.isReady()).toBe(false) + + requests[1]!.deferred.resolve() + await requests[1]!.outcome + await finishPreload(preload) + } finally { + for (const request of requests) request.deferred.resolve() + await Promise.allSettled(requests.map(({ outcome }) => outcome)) + await live.cleanup() + await Promise.all([posts.collection.cleanup(), comments.cleanup()]) + } +} + +async function expectRejectedDemandEntersError(): Promise { + const posts = createMutablePosts([ + { id: 1, authorId: `selected`, title: `one` }, + ]) + let loadCount = 0 + let shouldReject = true + const childLoadError = new Error(`child load failed`) + const comments = createCollection({ + id: nextCollectionId(`temporal-rejected-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => ({ + loadSubset: () => { + loadCount += 1 + if (shouldReject) { + return Promise.reject(childLoadError) + } + markReady() + return true + }, + }), + }, + }) + const live = createPostsWithCommentsLive(posts.collection, comments) + const preload: PreloadState = { preloadSettled: false } + const consoleError = vi.spyOn(console, `error`).mockImplementation(() => {}) + startPreload(live, preload) + + try { + await flushPromises() + expect(loadCount).toBe(1) + expect(live.status).toBe(`error`) + expect(preload.preloadSettled).toBe(true) + expect(preload.preloadFailure?.error).toBe(childLoadError) + + await live.cleanup() + await preload.preloadOutcome + shouldReject = false + await live.preload() + expect(loadCount).toBe(2) + expect(live.isReady()).toBe(true) + } finally { + await live.cleanup() + await preload.preloadOutcome + await Promise.all([posts.collection.cleanup(), comments.cleanup()]) + consoleError.mockRestore() + } +} + +async function expectFailedDemandRetriesSameCoverage(): Promise { + let loadCount = 0 + let shouldReject = true + const comments = createCollection({ + id: nextCollectionId(`temporal-demand-retry-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: ({ markReady }) => ({ + loadSubset: () => { + loadCount += 1 + if (shouldReject) { + return Promise.reject(new Error(`child load failed`)) + } + markReady() + return true + }, + }), + }, + }) + comments.createIndex((comment) => comment.postId) + const subscription = comments.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const controller = new SubsetDemandController() + const plan: LazyDemandPlan = { + id: `same-coverage-retry`, + path: [`postId`], + collectionId: comments.id, + initialKeys: new Set(), + } + + try { + const first = controller.setDemand(subscription, plan, new Set([1])) + expect(first.ready).toBeInstanceOf(Promise) + if (!(first.ready instanceof Promise)) { + throw new Error(`Expected failed demand to be asynchronous`) + } + await expect(first.ready).rejects.toThrow(`child load failed`) + + shouldReject = false + const retry = controller.setDemand(subscription, plan, new Set([1])) + expect(retry.changed).toBe(true) + expect(loadCount).toBe(2) + if (retry.ready instanceof Promise) await retry.ready + } finally { + controller.clear() + subscription.unsubscribe() + await comments.cleanup() + } +} + +async function expectDemandReactivationRetriesAfterReleaseFailure( + keys: ReadonlyArray, +): Promise { + let loadCount = 0 + let allowUnload = false + const releaseError = new Error(`child release failed`) + const comments = createCollection({ + id: nextCollectionId(`temporal-release-retry-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: ({ markReady }) => ({ + loadSubset: () => { + loadCount += 1 + markReady() + return true + }, + unloadSubset: () => { + if (!allowUnload) throw releaseError + }, + }), + }, + }) + comments.createIndex((comment) => comment.postId) + const subscription = comments.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const controller = new SubsetDemandController() + const plan: LazyDemandPlan = { + id: `release-failure-retry`, + path: [`postId`], + collectionId: comments.id, + initialKeys: new Set(), + } + + try { + expect( + controller.setDemand(subscription, plan, new Set(keys)), + ).toMatchObject({ changed: true, empty: false }) + expect(loadCount).toBe(1) + + const retired = controller.setDemand(subscription, plan, new Set()) + expect(retired).toMatchObject({ changed: true, empty: true }) + + const reactivated = controller.setDemand(subscription, plan, new Set(keys)) + expect(reactivated).toMatchObject({ changed: true, empty: false }) + expect(loadCount).toBe(2) + } finally { + allowUnload = true + controller.clear() + subscription.unsubscribe() + await comments.cleanup() + } +} + +async function expectRetiredDemandStaysNonfatalAfterReleaseFailure(): Promise { + const post = { id: 1, authorId: `selected`, title: `one` } + const posts = createMutablePosts([post]) + let loadCount = 0 + let allowUnload = false + const releaseError = new Error(`child release failed`) + const comments = createCollection({ + id: nextCollectionId(`temporal-retired-release-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: ({ markReady }) => ({ + loadSubset: () => { + loadCount += 1 + markReady() + return true + }, + unloadSubset: () => { + if (!allowUnload) throw releaseError + }, + }), + }, + }) + const live = createPostsWithCommentsLive(posts.collection, comments) + const consoleError = vi.spyOn(console, `error`).mockImplementation(() => {}) + + try { + await live.preload() + expect(loadCount).toBe(1) + expect(live.status).toBe(`ready`) + + posts.write(`delete`, post) + await flushPromises() + expect(live.size).toBe(0) + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBe(releaseError) + + posts.write(`insert`, post) + await flushPromises() + expect(loadCount).toBe(2) + expect(live.status).toBe(`ready`) + } finally { + allowUnload = true + await live.cleanup() + await Promise.all([posts.collection.cleanup(), comments.cleanup()]) + consoleError.mockRestore() + } +} + +async function expectFailedReplayStopsGatingAfterLastDemandRetires(): Promise { + const post = { id: 1, authorId: `selected`, title: `one` } + const posts = createMutablePosts([post]) + const replay = createDeferred() + let begin!: () => void + let write!: (message: { type: `insert`; value: Comment }) => void + let commit!: () => true | Promise + let truncate!: () => void + let loadCount = 0 + const comments = createCollection({ + id: nextCollectionId(`temporal-retired-replay-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount += 1 + if (loadCount === 1) { + begin() + write({ + type: `insert`, + value: { id: 10, postId: post.id, body: `old` }, + }) + commit() + return true + } + begin() + write({ + type: `insert`, + value: { id: 20, postId: post.id, body: `private replacement` }, + }) + commit() + return replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createPostsWithCommentsLive(posts.collection, comments) + const publications: Array> = [] + const subscription = live.subscribeChanges( + () => publications.push(live.toArray.map(({ id }) => id)), + { includeInitialState: false }, + ) + const consoleError = vi.spyOn(console, `error`).mockImplementation(() => {}) + + try { + await live.preload() + expect(live.get(post.id)?.comments.map(({ id }) => id)).toEqual([10]) + publications.length = 0 + + begin() + truncate() + commit() + await flushPromises() + expect(loadCount).toBe(2) + + replay.reject(new Error(`replacement failed`)) + await flushPromises() + expect(live.get(post.id)?.comments.map(({ id }) => id)).toEqual([10]) + expect(publications).toEqual([]) + + posts.write(`delete`, post) + await flushPromises() + + // Once the parent retires the last child demand, its failed replay can no + // longer gate unrelated parent changes in the shared graph. + expect(live.size).toBe(0) + expect(publications).toEqual([[]]) + } finally { + replay.resolve() + subscription.unsubscribe() + await live.cleanup() + await Promise.all([posts.collection.cleanup(), comments.cleanup()]) + consoleError.mockRestore() + } +} + +async function expectSynchronousEmptyDemandIsReady(): Promise { + const posts = createMutablePosts([ + { id: 1, authorId: `selected`, title: `one` }, + ]) + let loadCount = 0 + const comments = createCollection({ + id: nextCollectionId(`temporal-empty-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + sync: { + sync: () => ({ + loadSubset: () => { + loadCount += 1 + return true + }, + }), + }, + }) + const live = createPostsWithCommentsLive(posts.collection, comments) + + try { + await live.preload() + expect(loadCount).toBe(1) + expect(live.isReady()).toBe(true) + expect(live.get(1)?.comments).toEqual([]) + } finally { + await live.cleanup() + await Promise.all([posts.collection.cleanup(), comments.cleanup()]) + } +} + +async function expectPartialShrinkRetainsCoverage(): Promise { + const firstPost = { id: 1, authorId: `selected`, title: `one` } + const secondPost = { id: 2, authorId: `selected`, title: `two` } + const posts = createMutablePosts([firstPost, secondPost]) + const initialLoad = createDeferred() + const installed = new Map() + let begin: () => void + let write: (change: { type: `insert` | `delete`; value: Comment }) => void + let commit: () => void + let markReady: () => void + let deduped: DeduplicatedLoadSubset + const unloads: Array> = [] + const comments = createCollection({ + id: nextCollectionId(`temporal-shrink-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + sync: { + sync: (methods) => { + ;({ begin, write, commit, markReady } = methods) + deduped = new DeduplicatedLoadSubset({ + loadSubset: (options) => + initialLoad.promise.then(() => { + const keys = correlationKeys([options], `postId`) + begin() + for (const postId of keys) { + const comment = { id: postId * 100, postId, body: `${postId}` } + installed.set(postId, comment) + write({ type: `insert`, value: comment }) + } + commit() + markReady() + }), + }) + return { + loadSubset: (options) => deduped.loadSubset(options), + unloadSubset: (options) => { + const keys = correlationKeys([options], `postId`) + unloads.push(keys) + begin() + for (const postId of keys) { + const comment = installed.get(postId) + if (comment) write({ type: `delete`, value: comment }) + installed.delete(postId) + } + commit() + }, + } + }, + }, + }) + const live = createPostsWithCommentsLive(posts.collection, comments) + const preload = live.preload() + + try { + await flushPromises() + initialLoad.resolve() + await preload + expect(live.get(1)?.comments).toHaveLength(1) + + posts.write(`delete`, secondPost) + await flushPromises() + expect(live.get(1)?.comments).toHaveLength(1) + expect(unloads).toEqual([]) + + posts.write(`delete`, firstPost) + await flushPromises() + expect(unloads).toEqual([[1, 2]]) + } finally { + initialLoad.resolve() + await live.cleanup() + await Promise.all([posts.collection.cleanup(), comments.cleanup()]) + } +} + +type FastPathEvent = { + phase: `fast` | `late` + keys: Array +} + +type ProgressiveObservation = { + events: Array + ready: boolean + preloadSettled: boolean +} + +type ProgressiveStep = `release-parent` + +type ProgressiveContext = { + users: Collection | undefined + posts: Collection + live: ReturnType + events: Array + closeWindow: () => void + releaseParent: (() => void) | undefined + startReached: Deferred + parentDelivery: Promise | undefined + preload: PreloadState + expected: ProgressiveObservation +} + +function createProgressivePosts(): { + collection: Collection + events: Array + closeWindow: () => void + syncStarted: Deferred +} { + let windowOpen = true + const events: Array = [] + const syncStarted = createDeferred() + const collection = createCollection({ + id: nextCollectionId(`temporal-progressive-posts`), + getKey: (post) => post.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, commit, markReady }) => { + syncStarted.resolve() + begin() + commit() + markReady() + return { + loadSubset: (options) => { + events.push({ + phase: windowOpen ? `fast` : `late`, + keys: correlationKeys([options], `userId`), + }) + return Promise.resolve() + }, + } + }, + }, + }) + return { + collection, + events, + syncStarted, + closeWindow: () => { + windowOpen = false + }, + } +} + +function createGatedUsers(): { + collection: Collection + release: () => void + started: Deferred + delivery: Promise +} { + const gate = createDeferred() + const started = createDeferred() + const delivery = createDeferred() + const collection = createCollection({ + id: nextCollectionId(`temporal-users`), + getKey: (user) => user.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + started.resolve() + gate.promise.then( + () => { + begin() + write({ type: `insert`, value: { id: 2, name: `selected` } }) + commit() + markReady() + delivery.resolve() + }, + (error) => delivery.reject(error), + ) + }, + }, + }) + return { + collection, + release: () => gate.resolve(), + started, + delivery: delivery.promise, + } +} + +function createProgressiveDriver( + mode: `direct` | `nested`, +): TraceDriver { + return { + setup: () => { + const { + collection: posts, + events, + closeWindow, + syncStarted, + } = createProgressivePosts() + + if (mode === `direct`) { + const live = createLiveQueryCollection((q) => + q.from({ post: posts }).where(({ post }) => eq(post.userId, 2)), + ) + return { + users: undefined, + posts, + live, + events, + closeWindow, + releaseParent: undefined, + startReached: syncStarted, + parentDelivery: undefined, + preload: { preloadSettled: false }, + expected: { + events: [{ phase: `fast`, keys: [2] }], + ready: true, + preloadSettled: true, + }, + } + } + + const { + collection: users, + release, + started, + delivery, + } = createGatedUsers() + const live = createLiveQueryCollection((q) => + q + .from({ user: users }) + .where(({ user }) => eq(user.id, 2)) + .select(({ user }) => ({ + id: user.id, + posts: toArray( + q + .from({ post: posts }) + .where(({ post }) => eq(post.userId, user.id)), + ), + })), + ) + return { + users, + posts, + live, + events, + closeWindow, + releaseParent: release, + startReached: started, + parentDelivery: delivery, + preload: { preloadSettled: false }, + expected: { + events: [{ phase: `fast`, keys: [2] }], + ready: false, + preloadSettled: false, + }, + } + }, + start: async (context) => { + const preload = startPreload(context.live, context.preload) + await context.startReached.promise + context.closeWindow() + if (mode === `direct`) await preload + }, + apply: async (_step, context) => { + context.releaseParent?.() + context.expected = { + events: [{ phase: `fast`, keys: [2] }], + ready: true, + preloadSettled: true, + } + await finishPreload(context.preload) + }, + cleanup: async ({ + users, + posts, + live, + releaseParent, + parentDelivery, + preload, + }) => { + releaseParent?.() + await parentDelivery + await live.cleanup() + await finishPreload(preload) + await Promise.all([users?.cleanup(), posts.cleanup()]) + }, + } +} + +const progressiveProjection: TraceProjection< + ProgressiveContext, + ProgressiveObservation +> = { + observe: ({ events, live, preload }) => ({ + events: [...events], + ready: live.isReady(), + preloadSettled: preload.preloadSettled, + }), + recompute: ({ expected }) => expected, + assertEqual: (observed, expected) => { + expect(observed).toEqual(expected) + return undefined + }, +} + +async function expectProgressiveTraceMatches( + mode: `direct` | `nested`, +): Promise { + await runTrace({ + steps: mode === `nested` ? [`release-parent`] : [], + driver: createProgressiveDriver(mode), + projection: progressiveProjection, + }) +} + +describe(`includes temporal oracle`, () => { + it(`an empty outer does not wait for an undemanded child`, () => + expectReadinessMatches([])) + + it(`loads a demanded child before becoming ready`, async () => { + await expectReadinessMatches([ + { id: 1, authorId: `selected`, title: `one` }, + { id: 2, authorId: `selected`, title: `two` }, + ]) + }) + + it( + `obsolete child demand does not block readiness`, + expectObsoleteDemandDoesNotBlockReadiness, + ) + + it( + `obsolete child demand cannot publish after the route is reactivated`, + expectObsoleteDemandCannotPublishAfterReactivation, + ) + + fcTest.prop( + [fc.scheduler()], + oraclePropertyOptions(20, `includes-temporal.demand-scheduling`), + )( + `obsolete and current demand completions are generation-safe in either order`, + expectScheduledDemandCompletionsStayGenerationSafe, + ) + + it( + `retained pending demand blocks readiness after demand expands`, + expectRetainedDemandBlocksReadiness, + ) + + it( + `obsolete demand cannot settle a reactivated demand incarnation`, + expectObsoleteDemandCannotSettleReactivatedDemand, + ) + + it(`rejected demand enters error`, expectRejectedDemandEntersError) + + it( + `failed demand retries the same coverage`, + expectFailedDemandRetriesSameCoverage, + ) + + it(`reactivated demand retries after its prior release fails`, () => + expectDemandReactivationRetriesAfterReleaseFailure([1])) + + fcTest.prop( + [ + fc.uniqueArray(fc.integer({ min: -3, max: 3 }), { + minLength: 1, + maxLength: 5, + }), + ], + oraclePropertyOptions(20, `includes-temporal.release-reentry`), + )( + `failed release never suppresses a later demand incarnation`, + expectDemandReactivationRetriesAfterReleaseFailure, + ) + + it( + `failed release retires an empty live-query demand without poisoning reentry`, + expectRetiredDemandStaysNonfatalAfterReleaseFailure, + ) + + it( + `failed replay stops gating after its last demand retires`, + expectFailedReplayStopsGatingAfterLastDemandRetires, + ) + + it( + `a synchronous empty demand can establish ready coverage`, + expectSynchronousEmptyDemandIsReady, + ) + + it( + `partially shrinking demand retains established coverage`, + expectPartialShrinkRetainsCoverage, + ) + + it(`loads a direct progressive subset inside the fast-path window`, async () => { + await expectProgressiveTraceMatches(`direct`) + }) + + it(`a nested progressive subset loads inside the fast-path window`, () => + expectProgressiveTraceMatches(`nested`)) +}) diff --git a/packages/db/tests/query/includes-work-counter-oracle.test.ts b/packages/db/tests/query/includes-work-counter-oracle.test.ts new file mode 100644 index 0000000000..78beb5ca3d --- /dev/null +++ b/packages/db/tests/query/includes-work-counter-oracle.test.ts @@ -0,0 +1,455 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { beforeAll, describe, expect, it } from 'vitest' +import { oracleRuns } from '../oracle-config.js' +import { createCollection } from '../../src/collection/index.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { localOnlyCollectionOptions } from '../../src/local-only.js' +import { + createLiveQueryCollection, + eq, + materialize, +} from '../../src/query/index.js' +import type { Collection } from '../../src/collection/index.js' + +let nextCollectionId = 0 + +type TermRow = { id: string; text: string } +type MeaningRow = { id: string; termId: string } +type GroupRow = { id: string; meaningId: string } +type LinkRow = { id: string; groupId: string; targetId: string } + +type SourceRows = { + terms: Array + meanings: Array + groups: Array + links: Array +} + +type FillerCounts = { + terms: number + meanings: number + groups: number + links: number +} + +type WorkScenario = { + filler: FillerCounts + joinTargets: boolean +} + +type WorkCount = { + delivered: number + examined: number +} + +type SourceWork = { + terms: WorkCount + meanings: WorkCount + groups: WorkCount + links: WorkCount +} + +type LinkObservation = + | { id: string; text: string } + | { id: string; targetId: string } + +type WorkObservation = { + result: Array<{ + id: string + meanings: Array<{ + id: string + groups: Array<{ + id: string + links: Array + }> + }> + }> + sourceWork: SourceWork +} + +async function runCleanups( + cleanups: ReadonlyArray<() => void | Promise>, +): Promise { + const results = await Promise.allSettled( + cleanups.map(async (cleanup) => cleanup()), + ) + const firstRejection = results.find( + (result): result is PromiseRejectedResult => result.status === `rejected`, + ) + if (firstRejection !== undefined) throw firstRejection.reason +} + +const noFillers: FillerCounts = { + terms: 0, + meanings: 0, + groups: 0, + links: 0, +} + +function createSourceCollection( + name: string, + initialData: Array, +) { + return createCollection( + localOnlyCollectionOptions({ + id: `${name}-${nextCollectionId++}`, + getKey: (row) => row.id, + initialData, + }), + ) +} + +// Count both sides of the source boundary named in #1709's profile. Delivered +// rows show what enters the dataflow graph. entries() visits capture scans and +// get() calls capture keyed reads, so examined work cannot hide behind a filter. +function countSourceWork(collection: Collection) { + let deliveredRows = 0 + let examinedRows = 0 + let readingEntry = false + const originalSubscribeChanges = collection.subscribeChanges.bind(collection) + const originalEntries = collection.entries.bind(collection) + const originalGet = collection.get.bind(collection) + + collection.subscribeChanges = (callback, options) => { + return originalSubscribeChanges((changes) => { + deliveredRows += changes.length + callback(changes) + }, options) + } + + collection.get = (key) => { + if (!readingEntry) examinedRows++ + return originalGet(key) + } + + collection.entries = function* () { + const entries = originalEntries() + const readNext = () => { + readingEntry = true + try { + return entries.next() + } finally { + readingEntry = false + } + } + + for (let next = readNext(); !next.done; next = readNext()) { + examinedRows++ + yield next.value + } + } + + return (): WorkCount => ({ delivered: deliveredRows, examined: examinedRows }) +} + +function createFixtureRows(filler: FillerCounts): SourceRows { + return { + terms: [ + { id: `term-0`, text: `selected term` }, + { id: `term-1`, text: `first target` }, + { id: `term-2`, text: `second target` }, + ...Array.from({ length: filler.terms }, (_, index) => ({ + id: `term-filler-${index}`, + text: `irrelevant target ${index}`, + })), + ], + meanings: [ + { id: `meaning-0`, termId: `term-0` }, + ...Array.from({ length: filler.meanings }, (_, index) => ({ + id: `meaning-filler-${index}`, + termId: `term-never-selected`, + })), + ], + groups: [ + { id: `group-0`, meaningId: `meaning-0` }, + ...Array.from({ length: filler.groups }, (_, index) => ({ + id: `group-filler-${index}`, + meaningId: `meaning-never-selected`, + })), + ], + links: [ + // Keep filler links on one existing target key. Only the left-side input + // grows; the term-filler control probes right-side input growth separately. + { id: `link-0`, groupId: `group-0`, targetId: `term-1` }, + { + id: `link-1`, + groupId: `group-0`, + targetId: `term-2`, + }, + ...Array.from({ length: filler.links }, (_, index) => ({ + id: `link-filler-${index}`, + groupId: `group-never-selected`, + targetId: `term-1`, + })), + ], + } +} + +function observeLink(link: LinkObservation): LinkObservation { + if (`text` in link) return { id: link.id, text: link.text } + if (`targetId` in link) return { id: link.id, targetId: link.targetId } + + const exhaustive: never = link + return exhaustive +} + +async function observeWork({ + filler, + joinTargets, +}: WorkScenario): Promise { + const rows = createFixtureRows(filler) + const sources = { + terms: createSourceCollection(`work-terms`, rows.terms), + meanings: createSourceCollection(`work-meanings`, rows.meanings), + groups: createSourceCollection(`work-groups`, rows.groups), + links: createSourceCollection(`work-links`, rows.links), + } + let cleanupLive: (() => Promise) | undefined + + try { + await Promise.all(Object.values(sources).map((source) => source.preload())) + + // Match #1709's reproduction: load first, then add a B-tree index on each + // correlation and join column before constructing the live query. + sources.terms.createIndex((row) => row.id, { indexType: BTreeIndex }) + sources.meanings.createIndex((row) => row.termId, { + indexType: BTreeIndex, + }) + sources.groups.createIndex((row) => row.meaningId, { + indexType: BTreeIndex, + }) + sources.links.createIndex((row) => row.groupId, { indexType: BTreeIndex }) + sources.links.createIndex((row) => row.targetId, { + indexType: BTreeIndex, + }) + + const counters = { + terms: countSourceWork(sources.terms), + meanings: countSourceWork(sources.meanings), + groups: countSourceWork(sources.groups), + links: countSourceWork(sources.links), + } + + const live = createLiveQueryCollection((q) => + q + .from({ term: sources.terms }) + .where(({ term }) => eq(term.id, `term-0`)) + .select(({ term }) => ({ + id: term.id, + meanings: materialize( + q + .from({ meaning: sources.meanings }) + .where(({ meaning }) => eq(meaning.termId, term.id)) + .select(({ meaning }) => ({ + id: meaning.id, + groups: materialize( + q + .from({ group: sources.groups }) + .where(({ group }) => eq(group.meaningId, meaning.id)) + .select(({ group }) => { + const selectedLinks = q + .from({ link: sources.links }) + .where(({ link }) => eq(link.groupId, group.id)) + + return { + id: group.id, + links: joinTargets + ? materialize( + selectedLinks + .innerJoin( + { target: sources.terms }, + ({ link, target }) => + eq(link.targetId, target.id), + ) + .select(({ link, target }) => ({ + id: link.id, + text: target.text, + })), + ) + : materialize( + selectedLinks.select(({ link }) => ({ + id: link.id, + targetId: link.targetId, + })), + ), + } + }), + ), + })), + ), + })), + ) + cleanupLive = () => live.cleanup() + + await live.preload() + const root = live.toArray[0]! + return { + result: [ + { + id: root.id, + meanings: root.meanings.map((meaning) => ({ + id: meaning.id, + groups: meaning.groups.map((group) => ({ + id: group.id, + links: group.links.map(observeLink), + })), + })), + }, + ], + sourceWork: { + terms: counters.terms(), + meanings: counters.meanings(), + groups: counters.groups(), + links: counters.links(), + }, + } + } finally { + await runCleanups([ + async () => cleanupLive?.(), + ...Object.values(sources).map((source) => async () => source.cleanup()), + ]) + } +} + +function expectedResult({ + joinTargets, +}: Pick): WorkObservation[`result`] { + return [ + { + id: `term-0`, + meanings: [ + { + id: `meaning-0`, + groups: [ + { + id: `group-0`, + links: [ + joinTargets + ? { id: `link-0`, text: `first target` } + : { id: `link-0`, targetId: `term-1` }, + joinTargets + ? { + id: `link-1`, + text: `second target`, + } + : { + id: `link-1`, + targetId: `term-2`, + }, + ], + }, + ], + }, + ], + }, + ] +} + +const joinedBaselineWork: SourceWork = { + terms: { delivered: 3, examined: 3 }, + meanings: { delivered: 1, examined: 1 }, + groups: { delivered: 1, examined: 1 }, + links: { delivered: 2, examined: 2 }, +} + +const joinFreeBaselineWork: SourceWork = { + terms: { delivered: 1, examined: 1 }, + meanings: { delivered: 1, examined: 1 }, + groups: { delivered: 1, examined: 1 }, + links: { delivered: 2, examined: 2 }, +} + +let joinedBaselineObservation: WorkObservation +let joinFreeBaselineObservation: WorkObservation + +async function expectCorrelatedJoinWorkBound( + fillerCount: number, +): Promise { + const baseline = joinedBaselineObservation + const scaled = await observeWork({ + filler: { + terms: 0, + meanings: 0, + groups: 0, + links: fillerCount, + }, + joinTargets: true, + }) + expect(baseline.result).toEqual(expectedResult({ joinTargets: true })) + expect(scaled.result).toEqual(baseline.result) + expect(baseline.sourceWork).toEqual(joinedBaselineWork) + + expect(scaled.sourceWork).toEqual(baseline.sourceWork) +} + +describe(`includes deterministic work-counter oracle`, () => { + beforeAll(async () => { + const [joinedBaseline, joinFreeBaseline] = await Promise.all([ + observeWork({ filler: noFillers, joinTargets: true }), + observeWork({ filler: noFillers, joinTargets: false }), + ]) + joinedBaselineObservation = joinedBaseline + joinFreeBaselineObservation = joinFreeBaseline + }) + + it.each([1, 2, 3])( + `pins the #1709 work bound at the small filler boundary (%i)`, + expectCorrelatedJoinWorkBound, + ) + + fcTest.prop([fc.integer({ min: 1, max: 24 })], { + numRuns: oracleRuns(6), + seed: 1709, + })( + `a join preserves correlated source pushdown (#1709)`, + expectCorrelatedJoinWorkBound, + ) + + fcTest.prop([fc.integer({ min: 1, max: 24 })], { + numRuns: oracleRuns(6), + seed: 170_900, + })( + `indexed join-target growth keeps source work flat (#1709 direction control)`, + async (fillerCount) => { + const baseline = joinedBaselineObservation + const scaled = await observeWork({ + filler: { + terms: fillerCount, + meanings: 0, + groups: 0, + links: 0, + }, + joinTargets: true, + }) + + expect(baseline.result).toEqual(expectedResult({ joinTargets: true })) + expect(scaled.result).toEqual(baseline.result) + expect(baseline.sourceWork).toEqual(joinedBaselineWork) + expect(scaled.sourceWork).toEqual(baseline.sourceWork) + }, + ) + + fcTest.prop([fc.integer({ min: 1, max: 24 })], { + numRuns: oracleRuns(6), + seed: 17_090, + })( + `join-free correlated includes keep source work flat (#1709 control)`, + async (fillerCount) => { + const baseline = joinFreeBaselineObservation + const scaled = await observeWork({ + filler: { + terms: fillerCount, + meanings: fillerCount, + groups: fillerCount, + links: fillerCount, + }, + joinTargets: false, + }) + + expect(baseline.result).toEqual(expectedResult({ joinTargets: false })) + expect(scaled.result).toEqual(baseline.result) + expect(baseline.sourceWork).toEqual(joinFreeBaselineWork) + expect(scaled.sourceWork).toEqual(baseline.sourceWork) + }, + ) +}) diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index 0124ebbeac..85f6d6d4e7 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -6,13 +6,19 @@ import { count, createLiveQueryCollection, eq, + gte, materialize, toArray, } from '../../src/query/index.js' import { createCollection } from '../../src/collection/index.js' -import { CleanupQueue } from '../../src/collection/cleanup-queue.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' import { localOnlyCollectionOptions } from '../../src/local-only.js' -import { mockSyncCollectionOptions, stripVirtualProps } from '../utils.js' +import { + flushPromises, + mockSyncCollectionOptions, + resetCleanupQueue, + stripVirtualProps, +} from '../utils.js' import type { SyncConfig } from '../../src/types.js' type Project = { @@ -322,6 +328,71 @@ describe(`includes subqueries`, () => { }, ]) }) + + it(`publishes a non-empty child Collection as ready with its rows`, async () => { + const collection = buildIncludesQuery() + const publications: Array<{ ready: boolean; issueIds: Array }> = + [] + const subscription = collection.subscribeChanges( + () => { + const project = collection.get(1) + if (!project) return + publications.push({ + ready: project.issues.isReady(), + issueIds: [...project.issues.values()].map((issue) => issue.id), + }) + }, + { includeInitialState: true }, + ) + + try { + await collection.preload() + expect(publications[0]).toEqual({ ready: true, issueIds: [10, 11] }) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`gives an active null correlation an empty child Collection`, async () => { + const parents = createCollection( + mockSyncCollectionOptions<{ id: number; groupId: number | null }>({ + id: `includes-null-correlation-parents`, + getKey: (parent) => parent.id, + initialData: [{ id: 1, groupId: null }], + }), + ) + const children = createCollection( + mockSyncCollectionOptions<{ id: number; groupId: number }>({ + id: `includes-null-correlation-children`, + getKey: (child) => child.id, + initialData: [{ id: 10, groupId: 1 }], + }), + ) + const collection = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + items: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + .select(({ child }) => ({ id: child.id })), + })), + ) + + await collection.preload() + + const items = collection.get(1)!.items + expect(items).toBeDefined() + expect(plainRows(items)).toEqual([]) + expect(items.isReady()).toBe(true) + let preloadSettled = false + const preload = items.preload().then(() => { + preloadSettled = true + }) + await flushPromises() + expect(preloadSettled).toBe(true) + await preload + }) }) describe(`reactivity`, () => { @@ -390,7 +461,8 @@ describe(`includes subqueries`, () => { const collection = buildIncludesQuery() await collection.preload() - expect(childItems((collection.get(1) as any).issues)).toHaveLength(2) + const originalIssues = (collection.get(1) as any).issues + expect(childItems(originalIssues)).toHaveLength(2) // Remove project Alpha projects.utils.begin() @@ -401,6 +473,7 @@ describe(`includes subqueries`, () => { projects.utils.commit() expect(collection.get(1)).toBeUndefined() + expect(childItems(originalIssues)).toEqual([]) // Re-add project Alpha — should get a fresh child collection projects.utils.begin() @@ -412,6 +485,8 @@ describe(`includes subqueries`, () => { const alpha = collection.get(1) as any expect(alpha).toMatchObject({ id: 1, name: `Alpha Reborn` }) + expect(alpha.issues).not.toBe(originalIssues) + expect(childItems(originalIssues)).toEqual([]) expect(childItems(alpha.issues)).toEqual([ { id: 10, title: `Bug in Alpha` }, { id: 11, title: `Feature for Alpha` }, @@ -567,6 +642,60 @@ describe(`includes subqueries`, () => { }) describe(`change propagation`, () => { + it(`Collection includes: joined child update does not duplicate insert into child collection`, async () => { + type LineItem = { id: number; productId: number; qty: number } + type Product = { id: number; categoryId: number; name: string } + + const lineItems = createCollection( + mockSyncCollectionOptions({ + id: `includes-line-items`, + getKey: (lineItem) => lineItem.id, + initialData: [{ id: 1, productId: 10, qty: 1 }], + }), + ) + const products = createCollection( + mockSyncCollectionOptions({ + id: `includes-products`, + getKey: (product) => product.id, + initialData: [{ id: 10, categoryId: 1, name: `Widget` }], + }), + ) + + const collection = createLiveQueryCollection((q) => + q.from({ lineItem: lineItems }).select(({ lineItem }) => ({ + id: lineItem.id, + product: q + .from({ product: products }) + .where(({ product }) => eq(product.id, lineItem.productId)) + .select(({ product }) => ({ + id: product.id, + categoryId: product.categoryId, + name: product.name, + })), + })), + ) + await collection.preload() + + lineItems.utils.begin() + expect(() => { + lineItems.utils.write({ + type: `delete`, + value: { id: 1, productId: 10, qty: 1 }, + }) + lineItems.utils.write({ + type: `insert`, + value: { id: 1, productId: 10, qty: 2 }, + }) + }).not.toThrow() + lineItems.utils.commit() + + await vi.waitFor(() => { + expect(childItems((collection.get(1) as any).product)).toEqual([ + { id: 10, categoryId: 1, name: `Widget` }, + ]) + }) + }) + it(`Collection includes: child change does not re-emit the parent row`, async () => { const collection = buildIncludesQuery() await collection.preload() @@ -764,6 +893,37 @@ describe(`includes subqueries`, () => { { id: 11, title: `Feature for Alpha` }, ]) }) + + it(`order-only child changes update Collection layout`, async () => { + const collection = createLiveQueryCollection((q) => + q.from({ p: projects }).select(({ p }) => ({ + id: p.id, + issues: q + .from({ i: issues }) + .where(({ i }) => eq(i.projectId, p.id)) + .orderBy(({ i }) => i.title, `asc`) + .select(({ i }) => ({ id: i.id })), + })), + ) + + await collection.preload() + expect(plainRows((collection.get(1) as any).issues)).toEqual([ + { id: 10 }, + { id: 11 }, + ]) + + issues.utils.begin() + issues.utils.write({ + type: `update`, + value: { id: 11, projectId: 1, title: `A Feature for Alpha` }, + }) + issues.utils.commit() + + expect(plainRows((collection.get(1) as any).issues)).toEqual([ + { id: 11 }, + { id: 10 }, + ]) + }) }) describe(`ordered child queries with limit`, () => { @@ -1002,8 +1162,9 @@ describe(`includes subqueries`, () => { await collection.preload() // Both Frontend and Backend share departmentId 100 - expect(childItems((collection.get(1) as any).members)).toHaveLength(2) - expect(childItems((collection.get(2) as any).members)).toHaveLength(2) + const sharedMembers = (collection.get(1) as any).members + expect((collection.get(2) as any).members).toBe(sharedMembers) + expect(childItems(sharedMembers)).toHaveLength(2) // Delete the Frontend team teams.utils.begin() @@ -1016,235 +1177,687 @@ describe(`includes subqueries`, () => { expect(collection.get(1)).toBeUndefined() // Backend should still have its child collection with all members + expect((collection.get(2) as any).members).toBe(sharedMembers) expect(childItems((collection.get(2) as any).members)).toEqual([ { id: 10, name: `Alice` }, { id: 11, name: `Bob` }, ]) + + // Rejoining the still-active route reuses its shared facade. + teams.utils.begin() + teams.utils.write({ type: `insert`, value: sampleTeams[0]! }) + teams.utils.commit() + expect((collection.get(1) as any).members).toBe(sharedMembers) }) - it(`correlation field does not need to be in the parent select`, async () => { - const teams = createTeamsCollection() - const members = createMembersCollection() + it(`publishes a parent route move and its child facades coherently`, async () => { + type Parent = { id: number; groupId: number } + type Child = { id: number; groupId: number } - // departmentId is used for correlation but NOT selected in the parent output + const parents = createCollection( + mockSyncCollectionOptions({ + id: `coherent-route-parents`, + getKey: (parent) => parent.id, + initialData: [{ id: 1, groupId: 1 }], + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `coherent-route-children`, + getKey: (child) => child.id, + initialData: [ + { id: 10, groupId: 1 }, + { id: 20, groupId: 2 }, + ], + }), + ) const collection = createLiveQueryCollection((q) => - q.from({ t: teams }).select(({ t }) => ({ - id: t.id, - name: t.name, - members: q - .from({ m: members }) - .where(({ m }) => eq(m.departmentId, t.departmentId)) - .select(({ m }) => ({ - id: m.id, - name: m.name, - })), + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + groupId: parent.groupId, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + .select(({ child }) => ({ id: child.id })), })), ) await collection.preload() - expect(toTree(collection)).toEqual([ - { - id: 1, - name: `Frontend`, - members: [ - { id: 10, name: `Alice` }, - { id: 11, name: `Bob` }, - ], - }, - { - id: 2, - name: `Backend`, - members: [ - { id: 10, name: `Alice` }, - { id: 11, name: `Bob` }, - ], + const oldFacade = collection.get(1)!.children + const observations: Array<{ + groupId: number + sameFacade: boolean + oldRows: Array<{ id: number }> + currentRows: Array<{ id: number }> + }> = [] + const facadeSubscription = oldFacade.subscribeChanges( + () => { + const current = collection.get(1)! + observations.push({ + groupId: current.groupId, + sameFacade: current.children === oldFacade, + oldRows: plainRows(oldFacade), + currentRows: plainRows(current.children), + }) }, - { - id: 3, - name: `Marketing`, - members: [{ id: 20, name: `Charlie` }], + { includeInitialState: false }, + ) + const rootObservations: Array<{ + groupId: number + oldRows: Array<{ id: number }> + currentRows: Array<{ id: number }> + }> = [] + const rootSubscription = collection.subscribeChanges( + () => { + const current = collection.get(1)! + rootObservations.push({ + groupId: current.groupId, + oldRows: plainRows(oldFacade), + currentRows: plainRows(current.children), + }) }, - ]) + { includeInitialState: false }, + ) + + try { + parents.utils.begin() + parents.utils.write({ + type: `update`, + value: { id: 1, groupId: 2 }, + previousValue: { id: 1, groupId: 1 }, + }) + parents.utils.commit() + + expect(observations).toEqual([ + { + groupId: 2, + sameFacade: false, + oldRows: [], + currentRows: [{ id: 20 }], + }, + ]) + expect(rootObservations).toEqual([ + { + groupId: 2, + oldRows: [], + currentRows: [{ id: 20 }], + }, + ]) + } finally { + facadeSubscription.unsubscribe() + rootSubscription.unsubscribe() + } }) - }) - // Nested includes: two-level parent → child → grandchild (Project → Issue → Comment). - // Each level (Issue/Comment) can be materialized as a live Collection or a plain array (via toArray). - // We test all four combinations: - // Collection → Collection — both levels are live Collections - // Collection → toArray — issues are Collections, comments are arrays - // toArray → Collection — issues are arrays, comments are Collections - // toArray → toArray — both levels are plain arrays - describe(`nested includes: Collection → Collection`, () => { - function buildNestedQuery() { - return createLiveQueryCollection((q) => - q.from({ p: projects }).select(({ p }) => ({ - id: p.id, - name: p.name, - issues: q - .from({ i: issues }) - .where(({ i }) => eq(i.projectId, p.id)) - .select(({ i }) => ({ - id: i.id, - title: i.title, - comments: q - .from({ c: comments }) - .where(({ c }) => eq(c.issueId, i.id)) - .select(({ c }) => ({ - id: c.id, - body: c.body, - })), - })), + it(`replays existing bucket rows when their parent route becomes active`, async () => { + type Parent = { id: number; groupId: number } + type Child = { id: number; groupId: number } + + const parents = createCollection( + mockSyncCollectionOptions({ + id: `late-route-parents`, + getKey: (parent) => parent.id, + initialData: [], + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `late-route-children`, + getKey: (child) => child.id, + initialData: [{ id: 10, groupId: 1 }], + }), + ) + const collection = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + .select(({ child }) => ({ id: child.id })), })), ) - } - it(`supports two levels of includes`, async () => { - const collection = buildNestedQuery() await collection.preload() + parents.utils.begin() + parents.utils.write({ + type: `insert`, + value: { id: 1, groupId: 1 }, + }) + parents.utils.commit() - expect(toTree(collection)).toEqual([ - { - id: 1, - name: `Alpha`, - issues: [ - { - id: 10, - title: `Bug in Alpha`, - comments: [ - { id: 100, body: `Looks bad` }, - { id: 101, body: `Fixed it` }, - ], - }, - { - id: 11, - title: `Feature for Alpha`, - comments: [], - }, + expect(childItems(collection.get(1)!.children)).toEqual([{ id: 10 }]) + }) + + it(`replays existing bucket rows when a parent enters a limited result`, async () => { + type Parent = { id: number; rank: number; groupId: number } + type Child = { id: number; groupId: number } + + const parents = createCollection( + mockSyncCollectionOptions({ + id: `limited-late-route-parents`, + getKey: (parent) => parent.id, + initialData: [ + { id: 1, rank: 1, groupId: 1 }, + { id: 2, rank: 2, groupId: 2 }, ], - }, - { - id: 2, - name: `Beta`, - issues: [ - { - id: 20, - title: `Bug in Beta`, - comments: [{ id: 200, body: `Same bug` }], - }, + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `limited-late-route-children`, + getKey: (child) => child.id, + initialData: [ + { id: 10, groupId: 1 }, + { id: 20, groupId: 2 }, ], - }, - { - id: 3, - name: `Gamma`, - issues: [], - }, - ]) - }) + }), + ) + const collection = createLiveQueryCollection((q) => + q + .from({ parent: parents }) + .orderBy(({ parent }) => parent.rank) + .limit(1) + .select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + .select(({ child }) => ({ id: child.id })), + })), + ) - it(`adding a grandchild (comment) updates the nested child collection`, async () => { - const collection = buildNestedQuery() await collection.preload() + expect(childItems(collection.get(1)!.children)).toEqual([{ id: 10 }]) - // Issue 11 (Feature for Alpha) has no comments initially - const alpha = collection.get(1) as any - const issue11 = alpha.issues.get(11) - expect(childItems(issue11.comments)).toEqual([]) - - // Add a comment to issue 11 — no issue or project changes - comments.utils.begin() - comments.utils.write({ - type: `insert`, - value: { id: 110, issueId: 11, body: `Great feature` }, + parents.utils.begin() + parents.utils.write({ + type: `delete`, + value: { id: 1, rank: 1, groupId: 1 }, }) - comments.utils.commit() + parents.utils.commit() - const issue11After = (collection.get(1) as any).issues.get(11) - expect(childItems(issue11After.comments)).toEqual([ - { id: 110, body: `Great feature` }, - ]) + expect(collection.get(1)).toBeUndefined() + expect(childItems(collection.get(2)!.children)).toEqual([{ id: 20 }]) }) - it(`removing a grandchild (comment) updates the nested child collection`, async () => { - const collection = buildNestedQuery() - await collection.preload() - - // Issue 10 (Bug in Alpha) has 2 comments - const issue10 = (collection.get(1) as any).issues.get(10) - expect(childItems(issue10.comments)).toHaveLength(2) + it(`keeps a limited child facade complete as the window widens and receives later changes`, async () => { + type Parent = { id: number; rank: number; groupId: number } + type Child = { id: number; groupId: number; label: string } - // Remove one comment - comments.utils.begin() - comments.utils.write({ + const parents = createCollection( + mockSyncCollectionOptions({ + id: `widened-limited-facade-parents`, + getKey: (parent) => parent.id, + initialData: [ + { id: 1, rank: 1, groupId: 1 }, + { id: 2, rank: 2, groupId: 2 }, + ], + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `widened-limited-facade-children`, + getKey: (child) => child.id, + initialData: [ + { id: 10, groupId: 1, label: `first` }, + { id: 20, groupId: 2, label: `preloaded` }, + ], + }), + ) + const buildQuery = () => + createLiveQueryCollection((q) => + q + .from({ parent: parents }) + // Keep this as orderBy + limit: parent keys branch before top-K, + // so the second bucket's initial rows arrive while it is inactive. + .orderBy(({ parent }) => parent.rank) + .limit(1) + .select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + .select(({ child }) => ({ + id: child.id, + label: child.label, + })), + })), + ) + const collection = buildQuery() + + await collection.preload() + const windowResult = collection.utils.setWindow({ offset: 0, limit: 2 }) + if (windowResult instanceof Promise) { + await windowResult + } + + const secondFacade = collection.get(2)!.children + expect(secondFacade.status).toBe(`ready`) + expect(secondFacade.isReady()).toBe(true) + expect(plainRows(secondFacade)).toEqual([{ id: 20, label: `preloaded` }]) + + children.utils.begin() + children.utils.write({ + type: `insert`, + value: { id: 21, groupId: 2, label: `fresh` }, + }) + children.utils.commit() + children.utils.begin() + children.utils.write({ + type: `update`, + value: { id: 20, groupId: 2, label: `updated` }, + previousValue: { id: 20, groupId: 2, label: `preloaded` }, + }) + children.utils.commit() + + expect(plainRows(secondFacade)).toEqual([ + { id: 20, label: `updated` }, + { id: 21, label: `fresh` }, + ]) + + parents.utils.begin() + parents.utils.write({ type: `delete`, - value: sampleComments.find((c) => c.id === 100)!, + value: { id: 1, rank: 1, groupId: 1 }, }) - comments.utils.commit() + parents.utils.commit() + await collection.cleanup() + + const replayed = buildQuery() + await replayed.preload() + expect(plainRows(replayed.get(2)!.children)).toEqual([ + { id: 20, label: `updated` }, + { id: 21, label: `fresh` }, + ]) + await replayed.cleanup() + }) - const issue10After = (collection.get(1) as any).issues.get(10) - expect(childItems(issue10After.comments)).toEqual([ - { id: 101, body: `Fixed it` }, + it(`replays existing child rows when a parent where predicate becomes true`, async () => { + type Parent = { id: number; active: boolean; groupId: number } + type Child = { id: number; groupId: number } + + const parents = createCollection( + mockSyncCollectionOptions({ + id: `where-activated-facade-parents`, + getKey: (parent) => parent.id, + initialData: [{ id: 1, active: false, groupId: 1 }], + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `where-activated-facade-children`, + getKey: (child) => child.id, + initialData: [{ id: 10, groupId: 1 }], + }), + ) + const collection = createLiveQueryCollection((q) => + q + .from({ parent: parents }) + .where(({ parent }) => eq(parent.active, true)) + .select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)), + })), + ) + + await collection.preload() + expect(collection.size).toBe(0) + + parents.utils.begin() + parents.utils.write({ + type: `update`, + value: { id: 1, active: true, groupId: 1 }, + previousValue: { id: 1, active: false, groupId: 1 }, + }) + parents.utils.commit() + + expect(plainRows(collection.get(1)!.children)).toEqual([ + { id: 10, groupId: 1 }, ]) }) - it(`adding an issue (middle-level insert) creates a child with empty comments`, async () => { - const collection = buildNestedQuery() + it(`does not publish facade changes when root publication fails`, async () => { + type Parent = { id: number; groupId: number } + type Child = { id: number; groupId: number } + + const parents = createCollection( + mockSyncCollectionOptions({ + id: `failed-publication-parents`, + getKey: (parent) => parent.id, + initialData: [{ id: 1, groupId: 1 }], + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `failed-publication-children`, + getKey: (child) => child.id, + initialData: [ + { id: 10, groupId: 1 }, + { id: 20, groupId: 2 }, + ], + }), + ) + let keyReads = 0 + let failAt: number | undefined + const collection = createLiveQueryCollection({ + query: (q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + groupId: parent.groupId, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + .select(({ child }) => ({ id: child.id })), + })), + getKey: (row) => { + keyReads += 1 + if (keyReads === failAt) throw new Error(`root publication failed`) + return row.id + }, + }) + await collection.preload() + const oldFacade = collection.get(1)!.children + const facadeChanges = vi.fn() + const subscription = oldFacade.subscribeChanges(facadeChanges, { + includeInitialState: false, + }) - issues.utils.begin() - issues.utils.write({ - type: `insert`, - value: { id: 30, projectId: 3, title: `Gamma issue` }, + try { + keyReads = 0 + failAt = 3 + expect(() => { + parents.utils.begin() + parents.utils.write({ + type: `update`, + previousValue: { id: 1, groupId: 1 }, + value: { id: 1, groupId: 2 }, + }) + parents.utils.commit() + }).toThrow(`root publication failed`) + + expect(collection.get(1)!.groupId).toBe(1) + expect(collection.get(1)!.children).toBe(oldFacade) + expect(childItems(oldFacade)).toEqual([{ id: 10 }]) + expect(facadeChanges).not.toHaveBeenCalled() + } finally { + subscription.unsubscribe() + } + }) + + it(`publishes coherent includes through immediate ordered load-more passes`, async () => { + type Parent = { id: number; groupId: number; rank: number } + type Child = { id: number; groupId: number } + + const sourceRows: Array = [ + { id: 1, groupId: 1, rank: 1 }, + { id: 2, groupId: 2, rank: 2 }, + { id: 3, groupId: 3, rank: 3 }, + ] + let nextRow = 0 + let loadCount = 0 + const parents = createCollection({ + id: `ordered-publication-parents`, + getKey: (parent) => parent.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: (options) => { + // The current tie class is already present. A boundary probe + // must not consume the next page of source rows. + if (options.where) return true + loadCount += 1 + const row = sourceRows[nextRow++] + if (row) { + begin() + write({ type: `insert`, value: row }) + commit() + } + markReady() + return true + }, + }), + }, }) - issues.utils.commit() + const children = createCollection( + mockSyncCollectionOptions({ + id: `ordered-publication-children`, + getKey: (child) => child.id, + initialData: [ + { id: 10, groupId: 1 }, + { id: 20, groupId: 2 }, + { id: 30, groupId: 3 }, + ], + }), + ) + const collection = createLiveQueryCollection((q) => + q + .from({ parent: parents }) + .orderBy(({ parent }) => parent.rank) + .limit(3) + .select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + .select(({ child }) => ({ id: child.id })), + })), + ) + const observations: Array< + Array<{ id: number; childIds: Array }> + > = [] + const readObservation = () => + [...collection.values()].map((parent) => ({ + id: parent.id, + childIds: plainRows(parent.children).map((child) => child.id), + })) + const subscription = collection.subscribeChanges( + () => { + observations.push(readObservation()) + }, + { includeInitialState: false }, + ) + + try { + await collection.preload() + // Bounded tie probes carry `where` and are not counted as page loads. + expect(loadCount).toBe(sourceRows.length) + for (const observation of observations) { + for (const parent of observation) { + expect(parent.childIds).toEqual([parent.id * 10]) + } + } + expect(readObservation()).toEqual([ + { id: 1, childIds: [10] }, + { id: 2, childIds: [20] }, + { id: 3, childIds: [30] }, + ]) + } finally { + subscription.unsubscribe() + } + }) + + it(`retires a shared facade after all parent routes publish`, async () => { + type Parent = { id: number; groupId: number } + type Child = { id: number; groupId: number } + + const initialParents: Array = [ + { id: 1, groupId: 1 }, + { id: 2, groupId: 1 }, + ] + const parents = createCollection( + mockSyncCollectionOptions({ + id: `coherent-shared-route-parents`, + getKey: (parent) => parent.id, + initialData: initialParents, + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `coherent-shared-route-children`, + getKey: (child) => child.id, + initialData: [ + { id: 10, groupId: 1 }, + { id: 20, groupId: 2 }, + { id: 30, groupId: 3 }, + ], + }), + ) + const collection = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + groupId: parent.groupId, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + .select(({ child }) => ({ id: child.id })), + })), + ) + + await collection.preload() + + const sharedFacade = collection.get(1)!.children + expect(collection.get(2)!.children).toBe(sharedFacade) + const observations: Array<{ + oldRows: Array<{ id: number }> + parents: Array<{ + id: number + groupId: number + rows: Array<{ id: number }> + }> + }> = [] + const subscription = sharedFacade.subscribeChanges( + () => { + observations.push({ + oldRows: plainRows(sharedFacade), + parents: [1, 2].map((id) => { + const current = collection.get(id)! + return { + id, + groupId: current.groupId, + rows: plainRows(current.children), + } + }), + }) + }, + { includeInitialState: false }, + ) + + try { + parents.utils.begin() + parents.utils.write({ + type: `update`, + value: { id: 1, groupId: 2 }, + previousValue: initialParents[0]!, + }) + parents.utils.write({ + type: `update`, + value: { id: 2, groupId: 3 }, + previousValue: initialParents[1]!, + }) + parents.utils.commit() + + expect(observations).toEqual([ + { + oldRows: [], + parents: [ + { id: 1, groupId: 2, rows: [{ id: 20 }] }, + { id: 2, groupId: 3, rows: [{ id: 30 }] }, + ], + }, + ]) + } finally { + subscription.unsubscribe() + } + }) + + it(`correlation field does not need to be in the parent select`, async () => { + const teams = createTeamsCollection() + const members = createMembersCollection() + + // departmentId is used for correlation but NOT selected in the parent output + const collection = createLiveQueryCollection((q) => + q.from({ t: teams }).select(({ t }) => ({ + id: t.id, + name: t.name, + members: q + .from({ m: members }) + .where(({ m }) => eq(m.departmentId, t.departmentId)) + .select(({ m }) => ({ + id: m.id, + name: m.name, + })), + })), + ) + + await collection.preload() expect(toTree(collection)).toEqual([ { id: 1, - name: `Alpha`, - issues: [ - { - id: 10, - title: `Bug in Alpha`, - comments: [ - { id: 100, body: `Looks bad` }, - { id: 101, body: `Fixed it` }, - ], - }, - { id: 11, title: `Feature for Alpha`, comments: [] }, + name: `Frontend`, + members: [ + { id: 10, name: `Alice` }, + { id: 11, name: `Bob` }, ], }, { id: 2, - name: `Beta`, - issues: [ - { - id: 20, - title: `Bug in Beta`, - comments: [{ id: 200, body: `Same bug` }], - }, + name: `Backend`, + members: [ + { id: 10, name: `Alice` }, + { id: 11, name: `Bob` }, ], }, { id: 3, - name: `Gamma`, - issues: [{ id: 30, title: `Gamma issue`, comments: [] }], + name: `Marketing`, + members: [{ id: 20, name: `Charlie` }], }, ]) }) + }) - it(`removing an issue (middle-level delete) removes it from the parent`, async () => { + // Nested includes: two-level parent → child → grandchild (Project → Issue → Comment). + // Each level (Issue/Comment) can be materialized as a live Collection or a plain array (via toArray). + // We test all four combinations: + // Collection → Collection — both levels are live Collections + // Collection → toArray — issues are Collections, comments are arrays + // toArray → Collection — issues are arrays, comments are Collections + // toArray → toArray — both levels are plain arrays + describe(`nested includes: Collection → Collection`, () => { + function buildNestedQuery() { + return createLiveQueryCollection((q) => + q.from({ p: projects }).select(({ p }) => ({ + id: p.id, + name: p.name, + issues: q + .from({ i: issues }) + .where(({ i }) => eq(i.projectId, p.id)) + .select(({ i }) => ({ + id: i.id, + title: i.title, + comments: q + .from({ c: comments }) + .where(({ c }) => eq(c.issueId, i.id)) + .select(({ c }) => ({ + id: c.id, + body: c.body, + })), + })), + })), + ) + } + + it(`supports two levels of includes`, async () => { const collection = buildNestedQuery() await collection.preload() - issues.utils.begin() - issues.utils.write({ - type: `delete`, - value: sampleIssues.find((i) => i.id === 11)!, - }) - issues.utils.commit() - expect(toTree(collection)).toEqual([ { id: 1, @@ -1258,6 +1871,11 @@ describe(`includes subqueries`, () => { { id: 101, body: `Fixed it` }, ], }, + { + id: 11, + title: `Feature for Alpha`, + comments: [], + }, ], }, { @@ -1279,16 +1897,152 @@ describe(`includes subqueries`, () => { ]) }) - it(`updating an issue title (middle-level update) reflects in the parent`, async () => { + it(`adding a grandchild (comment) updates the nested child collection`, async () => { const collection = buildNestedQuery() await collection.preload() - issues.utils.begin() - issues.utils.write({ - type: `update`, - value: { id: 10, projectId: 1, title: `Renamed Bug` }, + // Issue 11 (Feature for Alpha) has no comments initially + const alpha = collection.get(1) as any + const issue11 = alpha.issues.get(11) + expect(childItems(issue11.comments)).toEqual([]) + + // Add a comment to issue 11 — no issue or project changes + comments.utils.begin() + comments.utils.write({ + type: `insert`, + value: { id: 110, issueId: 11, body: `Great feature` }, }) - issues.utils.commit() + comments.utils.commit() + + const issue11After = (collection.get(1) as any).issues.get(11) + expect(childItems(issue11After.comments)).toEqual([ + { id: 110, body: `Great feature` }, + ]) + }) + + it(`removing a grandchild (comment) updates the nested child collection`, async () => { + const collection = buildNestedQuery() + await collection.preload() + + // Issue 10 (Bug in Alpha) has 2 comments + const issue10 = (collection.get(1) as any).issues.get(10) + expect(childItems(issue10.comments)).toHaveLength(2) + + // Remove one comment + comments.utils.begin() + comments.utils.write({ + type: `delete`, + value: sampleComments.find((c) => c.id === 100)!, + }) + comments.utils.commit() + + const issue10After = (collection.get(1) as any).issues.get(10) + expect(childItems(issue10After.comments)).toEqual([ + { id: 101, body: `Fixed it` }, + ]) + }) + + it(`adding an issue (middle-level insert) creates a child with empty comments`, async () => { + const collection = buildNestedQuery() + await collection.preload() + + issues.utils.begin() + issues.utils.write({ + type: `insert`, + value: { id: 30, projectId: 3, title: `Gamma issue` }, + }) + issues.utils.commit() + + expect(toTree(collection)).toEqual([ + { + id: 1, + name: `Alpha`, + issues: [ + { + id: 10, + title: `Bug in Alpha`, + comments: [ + { id: 100, body: `Looks bad` }, + { id: 101, body: `Fixed it` }, + ], + }, + { id: 11, title: `Feature for Alpha`, comments: [] }, + ], + }, + { + id: 2, + name: `Beta`, + issues: [ + { + id: 20, + title: `Bug in Beta`, + comments: [{ id: 200, body: `Same bug` }], + }, + ], + }, + { + id: 3, + name: `Gamma`, + issues: [{ id: 30, title: `Gamma issue`, comments: [] }], + }, + ]) + }) + + it(`removing an issue (middle-level delete) removes it from the parent`, async () => { + const collection = buildNestedQuery() + await collection.preload() + + issues.utils.begin() + issues.utils.write({ + type: `delete`, + value: sampleIssues.find((i) => i.id === 11)!, + }) + issues.utils.commit() + + expect(toTree(collection)).toEqual([ + { + id: 1, + name: `Alpha`, + issues: [ + { + id: 10, + title: `Bug in Alpha`, + comments: [ + { id: 100, body: `Looks bad` }, + { id: 101, body: `Fixed it` }, + ], + }, + ], + }, + { + id: 2, + name: `Beta`, + issues: [ + { + id: 20, + title: `Bug in Beta`, + comments: [{ id: 200, body: `Same bug` }], + }, + ], + }, + { + id: 3, + name: `Gamma`, + issues: [], + }, + ]) + }) + + it(`updating an issue title (middle-level update) reflects in the parent`, async () => { + const collection = buildNestedQuery() + await collection.preload() + + issues.utils.begin() + issues.utils.write({ + type: `update`, + value: { id: 10, projectId: 1, title: `Renamed Bug` }, + }) + issues.utils.commit() expect(toTree(collection)).toEqual([ { @@ -2230,6 +2984,10 @@ describe(`includes subqueries`, () => { ], }, ]) + + const alphaIssues = collection.get(1)!.issues + expect([...alphaIssues.keys()]).toEqual([10]) + expect(alphaIssues.get(10)?.$key).toBe(10) }) it(`reacts to parent field change`, async () => { @@ -2535,6 +3293,271 @@ describe(`includes subqueries`, () => { items: [], }, ]) + + const aliceItems = collection.get(1)!.items + const bobItems = collection.get(2)!.items + expect([...aliceItems.keys()]).toEqual([10]) + expect(aliceItems.get(10)?.$key).toBe(10) + expect([...bobItems.keys()]).toEqual([]) + }) + + it(`keeps the same child public key in distinct parent-context buckets`, async () => { + type ScoreParent = { id: number; groupId: number; minimumScore: number } + type ScoreChild = { + id: number + groupId: number + score: number + label: string + } + + const parents = createCollection( + mockSyncCollectionOptions({ + id: `same-child-context-parents`, + getKey: (parent) => parent.id, + initialData: [ + { id: 1, groupId: 1, minimumScore: 10 }, + { id: 2, groupId: 1, minimumScore: 20 }, + ], + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `same-child-context-children`, + getKey: (child) => child.id, + initialData: [{ id: 7, groupId: 1, score: 30, label: `seven` }], + }), + ) + + const collection = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + children: materialize( + q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + .where(({ child }) => gte(child.score, parent.minimumScore)) + .select(({ child }) => ({ id: child.id, score: child.score })), + ), + firstChild: materialize( + q + .from({ firstChild: children }) + .where(({ firstChild }) => eq(firstChild.groupId, parent.groupId)) + .where(({ firstChild }) => + gte(firstChild.score, parent.minimumScore), + ) + .select(({ firstChild }) => ({ id: firstChild.id })) + .findOne(), + ), + labels: concat( + toArray( + q + .from({ labelChild: children }) + .where(({ labelChild }) => + eq(labelChild.groupId, parent.groupId), + ) + .where(({ labelChild }) => + gte(labelChild.score, parent.minimumScore), + ) + .select(({ labelChild }) => labelChild.label), + ), + ), + })), + ) + + await collection.preload() + + expect(toTree(collection)).toEqual([ + { + id: 1, + children: [{ id: 7, score: 30 }], + firstChild: { id: 7 }, + labels: `seven`, + }, + { + id: 2, + children: [{ id: 7, score: 30 }], + firstChild: { id: 7 }, + labels: `seven`, + }, + ]) + + parents.utils.begin() + parents.utils.write({ + type: `update`, + previousValue: { id: 2, groupId: 1, minimumScore: 20 }, + value: { id: 2, groupId: 2, minimumScore: 20 }, + }) + parents.utils.commit() + children.utils.begin() + children.utils.write({ + type: `update`, + previousValue: { id: 7, groupId: 1, score: 30, label: `seven` }, + value: { id: 7, groupId: 2, score: 30, label: `seven` }, + }) + children.utils.commit() + + expect(toTree(collection)).toEqual([ + { id: 1, children: [], firstChild: undefined, labels: `` }, + { + id: 2, + children: [{ id: 7, score: 30 }], + firstChild: { id: 7 }, + labels: `seven`, + }, + ]) + }) + + it(`uses canonical identity for non-JSON parent context values`, async () => { + type TaggedParent = { id: number; groupId: number; tag: bigint } + type TaggedChild = { + id: number | string + groupId: number + tag: bigint + metadataId: number + } + type Metadata = { id: number; groupId: number; tag: bigint } + + const parents = createCollection( + mockSyncCollectionOptions({ + id: `bigint-context-parents`, + getKey: (parent) => parent.id, + initialData: [{ id: 1, groupId: 1, tag: 1n }], + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `bigint-context-children`, + getKey: (child) => child.id, + initialData: [ + { id: 1, groupId: 1, tag: 1n, metadataId: 101 }, + { id: `1`, groupId: 1, tag: 1n, metadataId: 102 }, + ], + }), + ) + const metadataRows = createCollection( + mockSyncCollectionOptions({ + id: `bigint-context-metadata`, + getKey: (row) => row.id, + initialData: [ + { id: 101, groupId: 1, tag: 1n }, + { id: 102, groupId: 1, tag: 1n }, + ], + }), + ) + + const collection = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + children: materialize( + q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + .where(({ child }) => eq(child.tag, parent.tag)) + .select(({ child }) => ({ id: child.id })), + ), + joinedChildren: materialize( + q + .from({ joinedChild: children }) + .join( + { metadata: metadataRows }, + ({ joinedChild, metadata }) => + eq(joinedChild.metadataId, metadata.id), + `inner`, + ) + .where(({ metadata }) => eq(metadata.groupId, parent.groupId)) + .where(({ metadata }) => eq(metadata.tag, parent.tag)) + .select(({ joinedChild }) => ({ id: joinedChild.id })), + ), + })), + ) + + await collection.preload() + const [row] = toTree(collection) + const typedIds = (values: Array<{ id: number | string }>) => + values.map(({ id }) => `${typeof id}:${id}`).sort() + expect(row!.id).toBe(1) + expect(typedIds(row!.children)).toEqual([`number:1`, `string:1`]) + expect(typedIds(row!.joinedChildren)).toEqual([`number:1`, `string:1`]) + }) + + it(`keeps relation-local identity through Collection and nested includes`, async () => { + type Parent = { id: number; groupId: number; minimumScore: number } + type Child = { id: number; groupId: number; score: number } + type Note = { id: number; childId: number } + + const parents = createCollection( + mockSyncCollectionOptions({ + id: `nested-context-parents`, + getKey: (parent) => parent.id, + initialData: [ + { id: 1, groupId: 1, minimumScore: 10 }, + { id: 2, groupId: 1, minimumScore: 20 }, + ], + }), + ) + const children = createCollection( + mockSyncCollectionOptions({ + id: `nested-context-children`, + getKey: (child) => child.id, + initialData: [{ id: 7, groupId: 1, score: 30 }], + }), + ) + const notes = createCollection( + mockSyncCollectionOptions({ + id: `nested-context-notes`, + getKey: (note) => note.id, + initialData: [{ id: 70, childId: 7 }], + }), + ) + + const collection = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + .where(({ child }) => gte(child.score, parent.minimumScore)) + .select(({ child }) => ({ + id: child.id, + notes: materialize( + q + .from({ note: notes }) + .where(({ note }) => eq(note.childId, child.id)) + .select(({ note }) => ({ id: note.id })), + ), + })), + })), + ) + + await collection.preload() + + const firstChildren = collection.get(1)!.children + const secondChildren = collection.get(2)!.children + expect([...firstChildren.keys()]).toEqual([7]) + expect([...secondChildren.keys()]).toEqual([7]) + expect(stripVirtualProps(firstChildren.get(7))).toEqual({ + id: 7, + notes: [{ id: 70 }], + }) + expect(stripVirtualProps(secondChildren.get(7))).toEqual({ + id: 7, + notes: [{ id: 70 }], + }) + + children.utils.begin() + children.utils.write({ + type: `update`, + previousValue: { id: 7, groupId: 1, score: 30 }, + value: { id: 7, groupId: 1, score: 15 }, + }) + children.utils.commit() + + expect([...firstChildren.keys()]).toEqual([7]) + expect([...secondChildren.keys()]).toEqual([]) + expect(stripVirtualProps(firstChildren.get(7))).toEqual({ + id: 7, + notes: [{ id: 70 }], + }) }) it(`shared correlation key with parent filter + orderBy + limit`, async () => { @@ -4021,48 +5044,63 @@ describe(`includes subqueries`, () => { describe(`child collection garbage collection`, () => { beforeEach(() => { vi.useFakeTimers() - CleanupQueue.resetInstance() + resetCleanupQueue() }) afterEach(() => { vi.useRealTimers() - CleanupQueue.resetInstance() + resetCleanupQueue() }) - it(`child collections should not be garbage collected when external subscribers unmount`, async () => { + it(`retains child facades while their root is subscribed and releases them after root GC`, async () => { const collection = buildIncludesQuery() - await collection.preload() + const rootSub = collection.subscribeChanges(() => {}) + try { + await collection.preload() - // Verify child data exists - const alpha = collection.get(1) as any - expect(childItems(alpha.issues)).toEqual([ - { id: 10, title: `Bug in Alpha` }, - { id: 11, title: `Feature for Alpha` }, - ]) + // Verify child data exists + const alpha = collection.get(1)! + expect(childItems(alpha.issues)).toEqual([ + { id: 10, title: `Bug in Alpha` }, + { id: 11, title: `Feature for Alpha` }, + ]) - const beta = collection.get(2) as any - expect(childItems(beta.issues)).toEqual([ - { id: 20, title: `Bug in Beta` }, - ]) + const beta = collection.get(2)! + expect(childItems(beta.issues)).toEqual([ + { id: 20, title: `Bug in Beta` }, + ]) - // Simulate what useLiveQuery does in React: subscribe to child collection, - // then unsubscribe when the component unmounts (e.g., virtual table scroll) - const childSub = alpha.issues.subscribeChanges(() => {}) - childSub.unsubscribe() + // Simulate what useLiveQuery does in React: subscribe to child collection, + // then unsubscribe when the component unmounts (e.g., virtual table scroll) + const childSub = alpha.issues.subscribeChanges(() => {}) + childSub.unsubscribe() - // Advance well past the default gcTime (5 minutes = 300,000ms) - await vi.advanceTimersByTimeAsync(600_000) + // Advance well past the default gcTime (5 minutes = 300,000ms) + await vi.advanceTimersByTimeAsync(600_000) - // Child collection data should still be intact — the includes system - // owns these collections and manages their lifecycle via flushIncludesState. - // External GC must not destroy them. - expect(childItems(alpha.issues)).toEqual([ - { id: 10, title: `Bug in Alpha` }, - { id: 11, title: `Feature for Alpha` }, - ]) - expect(childItems(beta.issues)).toEqual([ - { id: 20, title: `Bug in Beta` }, - ]) + // Parent routes own these facades even when no child consumer remains. + expect(collection.status).toBe(`ready`) + expect(childItems(alpha.issues)).toEqual([ + { id: 10, title: `Bug in Alpha` }, + { id: 11, title: `Feature for Alpha` }, + ]) + expect(childItems(beta.issues)).toEqual([ + { id: 20, title: `Bug in Beta` }, + ]) + + rootSub.unsubscribe() + await vi.advanceTimersByTimeAsync(5_001) + + expect(collection.status).toBe(`cleaned-up`) + expect(childItems(alpha.issues)).toEqual([]) + expect(childItems(beta.issues)).toEqual([]) + } finally { + rootSub.unsubscribe() + await collection.cleanup() + await projects.cleanup() + await issues.cleanup() + await comments.cleanup() + } }) }) @@ -4688,219 +5726,1579 @@ describe(`includes subqueries`, () => { })), }) - await timeline.preload() - - const data = () => timeline.get(TIMELINE_KEY) as any - - runs.insert({ key: `run-1`, _seq: 1, status: `started` }) - texts.insert({ - key: `text-1`, - run_id: `run-1`, - _seq: 2, - status: `streaming`, - }) - await new Promise((r) => setTimeout(r, 100)) - - expect(data().runs).toHaveLength(1) - expect(data().runs[0].texts).toHaveLength(1) - expect(data().runs[0].texts[0].text).toBe(``) + await timeline.preload() + + const data = () => timeline.get(TIMELINE_KEY) as any + + runs.insert({ key: `run-1`, _seq: 1, status: `started` }) + texts.insert({ + key: `text-1`, + run_id: `run-1`, + _seq: 2, + status: `streaming`, + }) + await new Promise((r) => setTimeout(r, 100)) + + expect(data().runs).toHaveLength(1) + expect(data().runs[0].texts).toHaveLength(1) + expect(data().runs[0].texts[0].text).toBe(``) + + textDeltas.insert({ + key: `td-1`, + text_id: `text-1`, + run_id: `run-1`, + _seq: 3, + delta: `Hello`, + }) + await new Promise((r) => setTimeout(r, 100)) + expect(data().runs[0].texts[0].text).toBe(`Hello`) + + textDeltas.insert({ + key: `td-2`, + text_id: `text-1`, + run_id: `run-1`, + _seq: 4, + delta: ` world`, + }) + await new Promise((r) => setTimeout(r, 100)) + expect(data().runs[0].texts[0].text).toBe(`Hello world`) + }) + + it.each([0, 1])( + `deep buffer change for run %i leaves its sibling's values and notifications unchanged`, + async (changedIndex) => { + const siblingIndex = 1 - changedIndex + const TIMELINE_KEY = `tl-spurious` + + type Seed = { key: string } + type Run = { key: string; _seq: number; status: string } + type Text = { + key: string + run_id: string + _seq: number + status: string + } + type TextDelta = { + key: string + text_id: string + run_id: string + _seq: number + delta: string + } + + const seed = createCollection( + localOnlyCollectionOptions({ + id: `spurious-seed`, + getKey: (s) => s.key, + initialData: [{ key: TIMELINE_KEY }], + }), + ) + + const runs = createCollection( + localOnlyCollectionOptions({ + id: `spurious-runs`, + getKey: (r) => r.key, + initialData: [], + }), + ) + + const texts = createCollection( + localOnlyCollectionOptions({ + id: `spurious-texts`, + getKey: (t) => t.key, + initialData: [], + }), + ) + + const textDeltas = createCollection( + localOnlyCollectionOptions({ + id: `spurious-deltas`, + getKey: (d) => d.key, + initialData: [], + }), + ) + + const runsLive = createLiveQueryCollection({ + id: `spurious-runs-live`, + query: (q) => + q.from({ run: runs }).select(({ run }) => ({ + timelineKey: TIMELINE_KEY, + key: run.key, + order: coalesce(run._seq, -1), + status: run.status, + })), + }) + + const textsLive = createLiveQueryCollection({ + id: `spurious-texts-live`, + query: (q) => + q.from({ text: texts }).select(({ text }) => ({ + timelineKey: TIMELINE_KEY, + key: text.key, + run_id: text.run_id, + order: coalesce(text._seq, -1), + status: text.status, + })), + }) + + const textDeltasLive = createLiveQueryCollection({ + id: `spurious-deltas-live`, + query: (q) => + q.from({ delta: textDeltas }).select(({ delta }) => ({ + timelineKey: TIMELINE_KEY, + key: delta.key, + text_id: delta.text_id, + run_id: delta.run_id, + order: coalesce(delta._seq, -1), + delta: delta.delta, + })), + }) + + const timeline = createLiveQueryCollection({ + id: `spurious-timeline`, + query: (q) => + q.from({ s: seed }).select(({ s }) => ({ + key: s.key, + runs: toArray( + q + .from({ run: runsLive }) + .where(({ run }) => eq(run.timelineKey, s.key)) + .orderBy(({ run }) => run.order) + .select(({ run }) => ({ + key: run.key, + order: run.order, + status: run.status, + texts: toArray( + q + .from({ text: textsLive }) + .where(({ text }) => eq(text.run_id, run.key)) + .orderBy(({ text }) => text.order) + .select(({ text }) => ({ + key: text.key, + run_id: text.run_id, + order: text.order, + status: text.status, + text: concat( + toArray( + q + .from({ delta: textDeltasLive }) + .where(({ delta }) => + eq(delta.text_id, text.key), + ) + .orderBy(({ delta }) => delta.order) + .select(({ delta }) => delta.delta), + ), + ), + })), + ), + })), + ), + })), + }) + + await timeline.preload() + + const data = () => timeline.get(TIMELINE_KEY) as any + + runs.insert({ key: `run-1`, _seq: 1, status: `started` }) + runs.insert({ key: `run-2`, _seq: 2, status: `started` }) + texts.insert({ + key: `text-1`, + run_id: `run-1`, + _seq: 3, + status: `streaming`, + }) + texts.insert({ + key: `text-2`, + run_id: `run-2`, + _seq: 4, + status: `streaming`, + }) + await new Promise((r) => setTimeout(r, 100)) + + expect(data().runs).toHaveLength(2) + expect(data().runs[0].texts[0].text).toBe(``) + expect(data().runs[1].texts[0].text).toBe(``) + + const timelineRowBefore = data() + const siblingTextsBefore = timelineRowBefore.runs[siblingIndex].texts + const sibling = createLiveQueryCollection({ + query: (q) => + q.from({ row: timeline }).fn.select(({ row }) => ({ + key: row.key, + texts: row.runs[siblingIndex]!.texts, + })), + getKey: (row) => row.key, + }) + await sibling.preload() + const siblingEvents = vi.fn() + const siblingSubscription = sibling.subscribeChanges(siblingEvents, { + includeInitialState: false, + }) + const updateEvents = vi.fn() + const timelineSubscription = timeline.subscribeChanges(updateEvents, { + includeInitialState: false, + }) + + try { + textDeltas.insert({ + key: `td-1`, + text_id: `text-${changedIndex + 1}`, + run_id: `run-${changedIndex + 1}`, + _seq: 5, + delta: `Hello`, + }) + await new Promise((r) => setTimeout(r, 100)) + + expect(data().runs[changedIndex].texts[0].text).toBe(`Hello`) + expect(data().runs[siblingIndex].texts[0].text).toBe(``) + + expect(updateEvents).toHaveBeenCalledTimes(1) + expect(updateEvents.mock.calls[0]![0]).toMatchObject([ + { type: `update`, key: TIMELINE_KEY, value: data() }, + ]) + expect(data().runs[siblingIndex].texts).toEqual(siblingTextsBefore) + expect(timelineRowBefore.runs[changedIndex].texts[0].text).toBe(``) + expect(siblingEvents).not.toHaveBeenCalled() + } finally { + timelineSubscription.unsubscribe() + siblingSubscription.unsubscribe() + await sibling.cleanup() + } + }, + ) + + // Three collection levels (products -> priceRanges -> region). When two + // price ranges in different parent groups point at the same deepest + // correlation key (regionId 1, one under each product), each must still + // resolve its own copy of the nested `region` array. + it(`resolves nested grandchildren for sibling groups sharing a correlation key`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 2, productId: 1, regionId: 2 }, + { id: 3, productId: 2, regionId: 1 }, // same regionId as priceRange 1 + ], + }), + ) + + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + + await collection.preload() + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 1, + regionId: 1, + region: [{ id: 1, name: `Europe` }], + }, + { + id: 2, + regionId: 2, + region: [{ id: 2, name: `North America` }], + }, + ], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { + id: 3, + regionId: 1, + region: [{ id: 1, name: `Europe` }], + }, + ], + }, + ]) + }) + + // When a second parent group starts referencing a deepest correlation key + // that another group already resolved (the sibling price range is inserted + // after the initial load), the newly inserted group must also receive the + // nested grandchildren. + it(`fans nested grandchildren out to a sibling group inserted after load`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-incremental-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-incremental-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-incremental-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-incremental-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + // Insert a second price range under a different product, sharing regionId 1. + priceRanges.insert({ id: 3, productId: 2, regionId: 1 }) + await new Promise((r) => setTimeout(r, 50)) + + const tree = toTree(collection) + const tshirt = tree.find((p: any) => p.title === `T-Shirt`) + const hoodie = tree.find((p: any) => p.title === `Hoodie`) + expect(tshirt.priceRanges.find((pr: any) => pr.id === 1).region).toEqual([ + { id: 1, name: `Europe` }, + ]) + expect(hoodie.priceRanges.find((pr: any) => pr.id === 3).region).toEqual([ + { id: 1, name: `Europe` }, + ]) + }) + + it(`keeps deeper nested includes reactive for a sibling group added after load`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string; countryId: number } + type Country = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-late-sibling-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-late-sibling-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-late-sibling-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe`, countryId: 1 }], + }), + ) + const countries = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-late-sibling-countries`, + getKey: (c) => c.id, + initialData: [{ id: 1, name: `France` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + countries.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-late-sibling-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ + id: r.id, + name: r.name, + country: toArray( + q + .from({ c: countries }) + .where(({ c }) => eq(c.id, r.countryId)) + .select(({ c }) => ({ id: c.id, name: c.name })), + ), + })), + ), + })), + ), + })), + }) + await collection.preload() + + priceRanges.insert({ id: 2, productId: 2, regionId: 1 }) + await flushPromises() + + priceRanges.delete(1) + await flushPromises() + + countries.update(1, (draft) => { + draft.name = `Renamed France` + }) + await flushPromises() + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { + id: 2, + regionId: 1, + region: [ + { + id: 1, + name: `Europe`, + country: [{ id: 1, name: `Renamed France` }], + }, + ], + }, + ], + }, + ]) + }) + + it(`keeps shared nested includes reactive for sibling groups added after load at depth 3`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-3-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-3-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-3-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-depth-3-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + priceRanges.insert({ id: 2, productId: 2, regionId: 1 }) + await flushPromises() + + regions.update(1, (draft) => { + draft.name = `Renamed Europe` + }) + await flushPromises() + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [{ id: 1, region: [{ id: 1, name: `Renamed Europe` }] }], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [{ id: 2, region: [{ id: 1, name: `Renamed Europe` }] }], + }, + ]) + }) + + it(`keeps shared nested includes reactive for sibling groups added after load at depth 4`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string; countryId: number } + type Country = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-4-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-4-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-4-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe`, countryId: 1 }], + }), + ) + const countries = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-4-countries`, + getKey: (c) => c.id, + initialData: [{ id: 1, name: `France` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + countries.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-depth-4-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ + id: r.id, + name: r.name, + country: toArray( + q + .from({ c: countries }) + .where(({ c }) => eq(c.id, r.countryId)) + .select(({ c }) => ({ id: c.id, name: c.name })), + ), + })), + ), + })), + ), + })), + }) + await collection.preload() + + priceRanges.insert({ id: 2, productId: 2, regionId: 1 }) + await flushPromises() + + countries.update(1, (draft) => { + draft.name = `Renamed France` + }) + await flushPromises() + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 1, + region: [ + { + id: 1, + name: `Europe`, + country: [{ id: 1, name: `Renamed France` }], + }, + ], + }, + ], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { + id: 2, + region: [ + { + id: 1, + name: `Europe`, + country: [{ id: 1, name: `Renamed France` }], + }, + ], + }, + ], + }, + ]) + }) + + it(`keeps shared nested includes reactive for sibling groups added after load at depth 5`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string; countryId: number } + type Country = { id: number; name: string; zoneId: number } + type Zone = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-5-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-5-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-5-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe`, countryId: 1 }], + }), + ) + const countries = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-5-countries`, + getKey: (c) => c.id, + initialData: [{ id: 1, name: `France`, zoneId: 1 }], + }), + ) + const zones = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-5-zones`, + getKey: (z) => z.id, + initialData: [{ id: 1, name: `Eurozone` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + countries.preload(), + zones.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-depth-5-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ + id: r.id, + name: r.name, + country: toArray( + q + .from({ c: countries }) + .where(({ c }) => eq(c.id, r.countryId)) + .select(({ c }) => ({ + id: c.id, + name: c.name, + zone: toArray( + q + .from({ z: zones }) + .where(({ z }) => eq(z.id, c.zoneId)) + .select(({ z }) => ({ + id: z.id, + name: z.name, + })), + ), + })), + ), + })), + ), + })), + ), + })), + }) + await collection.preload() + + priceRanges.insert({ id: 2, productId: 2, regionId: 1 }) + await flushPromises() + + zones.update(1, (draft) => { + draft.name = `Renamed Eurozone` + }) + await flushPromises() + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 1, + region: [ + { + id: 1, + name: `Europe`, + country: [ + { + id: 1, + name: `France`, + zone: [{ id: 1, name: `Renamed Eurozone` }], + }, + ], + }, + ], + }, + ], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { + id: 2, + region: [ + { + id: 1, + name: `Europe`, + country: [ + { + id: 1, + name: `France`, + zone: [{ id: 1, name: `Renamed Eurozone` }], + }, + ], + }, + ], + }, + ], + }, + ]) + }) + + // When two parent groups share a deepest correlation key and one of them is + // deleted, the surviving group must keep its nested grandchildren. + it(`resolves two nested includes on the same child independently when they share a correlation value`, async () => { + type Product = { id: number; title: string } + type PriceRange = { + id: number + productId: number + regionId: number + currencyId: number + } + type Region = { id: number; name: string } + type Currency = { id: number; code: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-value-products`, + getKey: (p) => p.id, + initialData: [{ id: 1, title: `T-Shirt` }], + }), + ) + // The price range points at region 1 and currency 1: both nested includes + // correlate on the same value. + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-value-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1, currencyId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-value-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], + }), + ) + const currencies = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-value-currencies`, + getKey: (c) => c.id, + initialData: [{ id: 1, code: `EUR` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + currencies.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-value-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + currency: toArray( + q + .from({ c: currencies }) + .where(({ c }) => eq(c.id, pr.currencyId)) + .select(({ c }) => ({ id: c.id, code: c.code })), + ), + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + // Re-point only the region include; the currency include still resolves 1. + priceRanges.update(1, (draft) => { + draft.regionId = 2 + }) + await new Promise((r) => setTimeout(r, 50)) + + // A later currency change must still reach the currency include. + currencies.update(1, (draft) => { + draft.code = `USD` + }) + await new Promise((r) => setTimeout(r, 50)) + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 1, + currency: [{ id: 1, code: `USD` }], + region: [{ id: 2, name: `North America` }], + }, + ], + }, + ]) + }) + + it(`isolates a nested correlation-key update from a second nested include on the same child`, async () => { + type Product = { id: number; title: string } + type PriceRange = { + id: number + productId: number + regionId: number + currencyId: number + } + type Region = { id: number; name: string } + type Currency = { id: number; code: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `temp2-products`, + getKey: (p) => p.id, + initialData: [{ id: 1, title: `T-Shirt` }], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `temp2-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1, currencyId: 9 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `temp2-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], + }), + ) + const currencies = createCollection( + localOnlyCollectionOptions({ + id: `temp2-currencies`, + getKey: (c) => c.id, + initialData: [{ id: 9, code: `EUR` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + currencies.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `temp2-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + currency: toArray( + q + .from({ c: currencies }) + .where(({ c }) => eq(c.id, pr.currencyId)) + .select(({ c }) => ({ id: c.id, code: c.code })), + ), + })), + ), + })), + }) + await collection.preload() + + // Change ONLY regionId; currency must still resolve, and a later currency + // rename must still reach this price range. + priceRanges.update(1, (draft) => { + draft.regionId = 2 + }) + await new Promise((r) => setTimeout(r, 50)) + + currencies.update(9, (draft) => { + draft.code = `USD` + }) + await new Promise((r) => setTimeout(r, 50)) + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 1, + region: [{ id: 2, name: `North America` }], + currency: [{ id: 9, code: `USD` }], + }, + ], + }, + ]) + }) + + it(`keeps the survivor's data when a child changes its nested key then a sibling sharing the old key is deleted`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `temp-upd-products`, + getKey: (p) => p.id, + initialData: [{ id: 1, title: `T-Shirt` }], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `temp-upd-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 2, productId: 1, regionId: 1 }, + ], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `temp-upd-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `temp-upd-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + // pr_1 moves from region 1 to region 2 (both pr_1, pr_2 started at region 1) + priceRanges.update(1, (draft) => { + draft.regionId = 2 + }) + await new Promise((r) => setTimeout(r, 50)) + + // delete pr_2 (the remaining referencer of region 1) + priceRanges.delete(2) + await new Promise((r) => setTimeout(r, 50)) + + // rename region 1 — nothing references it anymore, must NOT affect pr_1 + regions.update(1, (draft) => { + draft.name = `Renamed Europe` + }) + await new Promise((r) => setTimeout(r, 50)) + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { id: 1, regionId: 2, region: [{ id: 2, name: `North America` }] }, + ], + }, + ]) + }) + + it(`keeps grandchildren on the surviving sibling after the other is deleted`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-delete-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-delete-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 3, productId: 2, regionId: 1 }, + ], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-delete-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-delete-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + // Delete the Hoodie's price range (the sibling sharing regionId 1). + priceRanges.delete(3) + await new Promise((r) => setTimeout(r, 50)) + + const tree = toTree(collection) + const tshirt = tree.find((p: any) => p.title === `T-Shirt`) + const hoodie = tree.find((p: any) => p.title === `Hoodie`) + expect(tshirt.priceRanges.find((pr: any) => pr.id === 1).region).toEqual([ + { id: 1, name: `Europe` }, + ]) + expect(hoodie.priceRanges).toEqual([]) + }) + + it(`keeps routing when one of multiple same-parent siblings sharing a nested key is deleted`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-same-parent-products`, + getKey: (p) => p.id, + initialData: [{ id: 1, title: `T-Shirt` }], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-same-parent-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 2, productId: 1, regionId: 1 }, + ], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-same-parent-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-same-parent-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + priceRanges.delete(1) + await new Promise((r) => setTimeout(r, 50)) - textDeltas.insert({ - key: `td-1`, - text_id: `text-1`, - run_id: `run-1`, - _seq: 3, - delta: `Hello`, + regions.update(1, (draft) => { + draft.name = `Renamed Europe` }) - await new Promise((r) => setTimeout(r, 100)) - expect(data().runs[0].texts[0].text).toBe(`Hello`) + await new Promise((r) => setTimeout(r, 50)) - textDeltas.insert({ - key: `td-2`, - text_id: `text-1`, - run_id: `run-1`, - _seq: 4, - delta: ` world`, - }) - await new Promise((r) => setTimeout(r, 100)) - expect(data().runs[0].texts[0].text).toBe(`Hello world`) + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 2, + regionId: 1, + region: [{ id: 1, name: `Renamed Europe` }], + }, + ], + }, + ]) }) - it(`deep buffer change for one parent does not emit spurious update for sibling parent`, async () => { - const TIMELINE_KEY = `tl-spurious` - - type Seed = { key: string } - type Run = { key: string; _seq: number; status: string } - type Text = { - key: string - run_id: string - _seq: number - status: string - } - type TextDelta = { - key: string - text_id: string - run_id: string - _seq: number - delta: string - } - - const seed = createCollection( - localOnlyCollectionOptions({ - id: `spurious-seed`, - getKey: (s) => s.key, - initialData: [{ key: TIMELINE_KEY }], + // The shared-correlation-key routing is independent of how each level is + // materialized, so the same guarantee must hold when the nested levels are + // left as live Collections (no toArray/materialize wrapper). + it(`resolves nested grandchildren for sibling groups when levels stay Collections`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-collection-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], }), ) - - const runs = createCollection( - localOnlyCollectionOptions({ - id: `spurious-runs`, - getKey: (r) => r.key, - initialData: [], + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-collection-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 2, productId: 1, regionId: 2 }, + { id: 3, productId: 2, regionId: 1 }, + ], }), ) - - const texts = createCollection( - localOnlyCollectionOptions({ - id: `spurious-texts`, - getKey: (t) => t.key, - initialData: [], + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-collection-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], }), ) - const textDeltas = createCollection( - localOnlyCollectionOptions({ - id: `spurious-deltas`, - getKey: (d) => d.key, - initialData: [], - }), - ) + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) - const runsLive = createLiveQueryCollection({ - id: `spurious-runs-live`, + const collection = createLiveQueryCollection({ + id: `shared-corr-collection-live`, query: (q) => - q.from({ run: runs }).select(({ run }) => ({ - timelineKey: TIMELINE_KEY, - key: run.key, - order: coalesce(run._seq, -1), - status: run.status, + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + })), })), }) + await collection.preload() - const textsLive = createLiveQueryCollection({ - id: `spurious-texts-live`, - query: (q) => - q.from({ text: texts }).select(({ text }) => ({ - timelineKey: TIMELINE_KEY, - key: text.key, - run_id: text.run_id, - order: coalesce(text._seq, -1), - status: text.status, - })), - }) + // toTree recursively unwraps the nested live Collections into arrays. + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { id: 1, regionId: 1, region: [{ id: 1, name: `Europe` }] }, + { + id: 2, + regionId: 2, + region: [{ id: 2, name: `North America` }], + }, + ], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { id: 3, regionId: 1, region: [{ id: 1, name: `Europe` }] }, + ], + }, + ]) + }) - const textDeltasLive = createLiveQueryCollection({ - id: `spurious-deltas-live`, - query: (q) => - q.from({ delta: textDeltas }).select(({ delta }) => ({ - timelineKey: TIMELINE_KEY, - key: delta.key, - text_id: delta.text_id, - run_id: delta.run_id, - order: coalesce(delta._seq, -1), - delta: delta.delta, - })), - }) + // Same guarantee for materialize(), which produces array/singleton + // snapshots through the same nested-includes routing. + it(`resolves nested grandchildren for sibling groups with materialize()`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } - const timeline = createLiveQueryCollection({ - id: `spurious-timeline`, + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-materialize-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-materialize-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 2, productId: 1, regionId: 2 }, + { id: 3, productId: 2, regionId: 1 }, + ], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-materialize-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-materialize-live`, query: (q) => - q.from({ s: seed }).select(({ s }) => ({ - key: s.key, - runs: toArray( + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: materialize( q - .from({ run: runsLive }) - .where(({ run }) => eq(run.timelineKey, s.key)) - .orderBy(({ run }) => run.order) - .select(({ run }) => ({ - key: run.key, - order: run.order, - status: run.status, - texts: toArray( + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: materialize( q - .from({ text: textsLive }) - .where(({ text }) => eq(text.run_id, run.key)) - .orderBy(({ text }) => text.order) - .select(({ text }) => ({ - key: text.key, - run_id: text.run_id, - order: text.order, - status: text.status, - text: concat( - toArray( - q - .from({ delta: textDeltasLive }) - .where(({ delta }) => eq(delta.text_id, text.key)) - .orderBy(({ delta }) => delta.order) - .select(({ delta }) => delta.delta), - ), - ), - })), + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), ), })), ), })), }) + await collection.preload() - await timeline.preload() - - const data = () => timeline.get(TIMELINE_KEY) as any - - runs.insert({ key: `run-1`, _seq: 1, status: `started` }) - runs.insert({ key: `run-2`, _seq: 2, status: `started` }) - texts.insert({ - key: `text-1`, - run_id: `run-1`, - _seq: 3, - status: `streaming`, - }) - texts.insert({ - key: `text-2`, - run_id: `run-2`, - _seq: 4, - status: `streaming`, - }) - await new Promise((r) => setTimeout(r, 100)) - - expect(data().runs).toHaveLength(2) - expect(data().runs[0].texts[0].text).toBe(``) - expect(data().runs[1].texts[0].text).toBe(``) - - const timelineRowBefore = data() - const run1TextsBefore = timelineRowBefore.runs[0].texts - const updateEvents: Array = [] - timeline.subscribeChanges((changes) => { - for (const change of changes) { - if (change.type === `update`) { - updateEvents.push(change) - } - } - }) - - textDeltas.insert({ - key: `td-1`, - text_id: `text-2`, - run_id: `run-2`, - _seq: 5, - delta: `Hello`, - }) - await new Promise((r) => setTimeout(r, 100)) + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { id: 1, regionId: 1, region: [{ id: 1, name: `Europe` }] }, + { + id: 2, + regionId: 2, + region: [{ id: 2, name: `North America` }], + }, + ], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { id: 3, regionId: 1, region: [{ id: 1, name: `Europe` }] }, + ], + }, + ]) - expect(data().runs[1].texts[0].text).toBe(`Hello`) - expect(data().runs[0].texts[0].text).toBe(``) + // Post-load: insert a price range under Hoodie that references regionId 2, + // a correlation key already materialized for T-Shirt at load. This drives + // the late-arrival snapshot re-emit path through materialize() — the new + // sibling group must be seeded with the already-drained North America row + // without disturbing T-Shirt's existing nested rows. + priceRanges.insert({ id: 4, productId: 2, regionId: 2 }) + await new Promise((r) => setTimeout(r, 50)) - expect(data().runs[0].texts).toBe(run1TextsBefore) + const tree = toTree(collection) + const tshirt = tree.find((p: any) => p.title === `T-Shirt`) + const hoodie = tree.find((p: any) => p.title === `Hoodie`) + expect(tshirt.priceRanges.find((pr: any) => pr.id === 2).region).toEqual([ + { id: 2, name: `North America` }, + ]) + expect(hoodie.priceRanges.find((pr: any) => pr.id === 4).region).toEqual([ + { id: 2, name: `North America` }, + ]) }) }) @@ -5127,6 +7525,70 @@ describe(`includes subqueries`, () => { }) describe(`materialize`, () => { + it(`uses the same public-key tie-breaker for Collection and inline includes`, async () => { + type OrderingParent = { id: number } + type OrderingChild = { + id: number + parentId: number + label: string + } + + const orderingParents = createCollection( + mockSyncCollectionOptions({ + id: `includes-ordering-parents`, + getKey: (parent) => parent.id, + initialData: [{ id: 1 }], + }), + ) + const orderingChildren = createCollection( + mockSyncCollectionOptions({ + id: `includes-ordering-children`, + getKey: (child) => child.id, + autoIndex: `eager`, + initialData: [ + { id: 2, parentId: 1, label: `two` }, + { id: 3, parentId: 1, label: `three` }, + { id: 10, parentId: 1, label: `ten` }, + ], + }), + ) + const collection = createLiveQueryCollection((q) => { + return q.from({ parent: orderingParents }).select(({ parent }) => { + const childRows = () => + q + .from({ child: orderingChildren }) + .where(({ child }) => eq(child.parentId, parent.id)) + .select(({ child }) => ({ id: child.id, label: child.label })) + + return { + id: parent.id, + facade: childRows(), + array: toArray(childRows()), + joined: concat( + toArray( + q + .from({ child: orderingChildren }) + .where(({ child }) => eq(child.parentId, parent.id)) + .select(({ child }) => child.label), + ), + ), + first: materialize(childRows().findOne()), + materialized: materialize(childRows()), + } + }) + }) + await collection.preload() + + const result = collection.get(1)! + const facadeIds = result.facade.toArray.map((child) => child.id) + + expect(facadeIds).toEqual([2, 3, 10]) + expect(result.array.map((child) => child.id)).toEqual(facadeIds) + expect(result.materialized.map((child) => child.id)).toEqual(facadeIds) + expect(result.first?.id).toBe(facadeIds[0]) + expect(result.joined).toBe(`twothreeten`) + }) + // For singleton behavior we look up each issue's parent project. // Each issue references exactly one project via projectId. function buildSingletonQuery() { diff --git a/packages/db/tests/query/ir-stable-identity.test.ts b/packages/db/tests/query/ir-stable-identity.test.ts new file mode 100644 index 0000000000..fed74b9f72 --- /dev/null +++ b/packages/db/tests/query/ir-stable-identity.test.ts @@ -0,0 +1,1604 @@ +import { describe, expect, it, vi } from 'vitest' +import { fc, test as fcTest } from '@fast-check/vitest' +import { Temporal } from 'temporal-polyfill' +import { CollectionImpl } from '../../src/collection/index.js' +import { Query, getQueryIR } from '../../src/query/builder/index.js' +import { + add, + and, + avg, + caseWhen, + coalesce, + concat, + count, + eq, + gt, + gte, + inArray, + isNull, + isUndefined, + length, + like, + lower, + lt, + max, + not, + or, + subtract, + sum, + upper, +} from '../../src/query/builder/functions.js' +import { + UnhashableQueryIRError, + getLoadSubsetDemandKey, + getQueryIdentity, + getStableQueryIRHash, + getStableValueHash, +} from '../../src/query/ir-stable-identity.js' +import { + CollectionRef, + Func, + IncludesSubquery, + PropRef, + QueryRef, + UnionAll, + Value, +} from '../../src/query/ir.js' +import { + compileExpression, + toBooleanPredicate, +} from '../../src/query/compiler/evaluators.js' +import { + createRuntimeReferenceIdentityFactory, + getRuntimeReferenceIdentity, +} from '../../src/query/runtime-reference-identity.js' +import { createValueIdentity } from '../../src/query/equality-value-identity.js' +import type { BasicExpression, QueryIR } from '../../src/query/ir.js' +import type { LoadSubsetOptions } from '../../src/types.js' + +interface User { + id: number + name: string + email?: string | null + active: boolean + age: number + salary: number + status: `active` | `inactive` + teamId: string + departmentId: number | null + createdAt: Date + profile?: { + skills: Array + experience: { + years: number + } + } + blob?: Uint8Array + largeViewCount?: bigint +} + +function getProjectedExpressionIdentity(expression: BasicExpression): string { + return getQueryIdentity({ + ...getQueryIR(new Query().from({ user: usersCollection })), + select: { value: expression }, + }) +} + +const referenceSemanticPairArbitrary = fc.oneof( + fc + .array(fc.integer()) + .map((values): [unknown, unknown] => [[...values], [...values]]), + fc + .dictionary(fc.string(), fc.integer()) + .map((value): [unknown, unknown] => [{ ...value }, { ...value }]), + fc + .array(fc.tuple(fc.string(), fc.integer())) + .map((entries): [unknown, unknown] => [new Map(entries), new Map(entries)]), + fc + .array(fc.integer()) + .map((values): [unknown, unknown] => [new Set(values), new Set(values)]), + fc + .int16Array() + .map((value): [unknown, unknown] => [ + new Int16Array(value), + new Int16Array(value), + ]), +) + +const outputExpressionPairArbitrary: fc.Arbitrary<{ + first: BasicExpression + second: BasicExpression +}> = fc.oneof( + fc.integer().map((value) => ({ + first: new Value(value), + second: new Value(value), + })), + fc.string().map((value) => ({ + first: new Func(`concat`, [new Value(value)]), + second: new Func(`concat`, [new Value(value)]), + })), + fc.uint8Array({ minLength: 1, maxLength: 8 }).map((value) => ({ + first: new Func(`concat`, [new Value(Buffer.from(value))]), + second: new Func(`concat`, [new Value(new Uint8Array(value))]), + })), + fc.constant({ + first: new Value(-0), + second: new Value(0), + }), +) + +interface Post { + id: number + userId: number + title: string + published: boolean + views: number + createdAt: Date +} + +const usersCollection = new CollectionImpl({ + id: `users`, + getKey: (item) => item.id, + sync: { sync: () => {} }, +}) + +describe(`stable runtime value hashing`, () => { + it(`normalizes object key order`, () => { + expect(getStableValueHash([`todos`, { status: `open`, page: 1 }])).toBe( + getStableValueHash([`todos`, { page: 1, status: `open` }]), + ) + }) + + it(`reports the path of an unhashable query key value`, () => { + expect(() => + getStableValueHash([`todos`, { predicate: () => true }], `queryKey`), + ).toThrow(/queryKey\[1\]\.predicate/) + }) +}) + +describe(`semantic expression identity`, () => { + const age = new PropRef([`user`, `age`]) + const active = new PropRef([`user`, `active`]) + type EquivalentExpressionPair = { + original: BasicExpression + equivalent: BasicExpression + } + + const comparisonPairArbitrary: fc.Arbitrary = fc + .record({ + operator: fc.constantFrom<`gt` | `gte` | `lt` | `lte`>( + `gt`, + `gte`, + `lt`, + `lte`, + ), + threshold: fc.integer(), + }) + .map(({ operator, threshold }) => { + const inverse: Record<`gt` | `gte` | `lt` | `lte`, string> = { + gt: `lt`, + gte: `lte`, + lt: `gt`, + lte: `gte`, + } + return { + original: new Func(operator, [age, new Value(threshold)]), + equivalent: new Func(inverse[operator], [ + new Value(threshold), + age, + ]), + } + }) + const equalityPairArbitrary: fc.Arbitrary = fc + .boolean() + .map((value) => ({ + original: new Func(`eq`, [active, new Value(value)]), + equivalent: new Func(`eq`, [new Value(value), active]), + })) + const membershipPairArbitrary: fc.Arbitrary = fc + .uniqueArray(fc.integer(), { minLength: 1, maxLength: 8 }) + .map((values) => ({ + original: new Func(`in`, [age, new Value(values)]), + equivalent: new Func(`in`, [ + age, + new Value([...values].reverse().concat(values[0]!)), + ]), + })) + const atomicExpressionPairArbitrary = fc.oneof( + comparisonPairArbitrary, + equalityPairArbitrary, + membershipPairArbitrary, + ) + const equivalentExpressionPairArbitrary = fc.oneof( + { weight: 3, arbitrary: atomicExpressionPairArbitrary }, + { + weight: 2, + arbitrary: fc + .tuple( + fc.constantFrom(`and`, `or`), + atomicExpressionPairArbitrary, + atomicExpressionPairArbitrary, + ) + .map(([operator, left, right]) => ({ + original: new Func(operator, [ + left.original, + new Func(operator, [right.original, left.original]), + ]), + equivalent: new Func(operator, [ + right.equivalent, + left.equivalent, + ]), + })), + }, + ) + + it(`normalizes associative, commutative, and idempotent boolean forms`, () => { + const adult = new Func(`gte`, [age, new Value(18)]) + const enabled = new Func(`eq`, [active, new Value(true)]) + const nested = new Func(`and`, [ + enabled, + new Func(`and`, [adult, enabled]), + ]) + const flat = new Func(`and`, [adult, enabled]) + + expect(getProjectedExpressionIdentity(nested)).toBe( + getProjectedExpressionIdentity(flat), + ) + expect(getProjectedExpressionIdentity(new Func(`or`, [adult, adult]))).toBe( + getProjectedExpressionIdentity(new Func(`or`, [adult])), + ) + }) + + it(`keeps a boolean wrapper when duplicate operands coerce their result`, () => { + const bareAge = new PropRef([`user`, `age`]) + const duplicateAnd = new Func(`and`, [bareAge, bareAge]) + const row = { user: { age: 18 } } + + expect(toBooleanPredicate(compileExpression(bareAge)(row))).toBe(false) + expect(toBooleanPredicate(compileExpression(duplicateAnd)(row))).toBe(true) + expect(getProjectedExpressionIdentity(duplicateAnd)).not.toBe( + getProjectedExpressionIdentity(bareAge), + ) + }) + + it(`normalizes equality and reversed inequalities`, () => { + expect( + getProjectedExpressionIdentity(new Func(`eq`, [age, new Value(18)])), + ).toBe(getProjectedExpressionIdentity(new Func(`eq`, [new Value(18), age]))) + expect( + getProjectedExpressionIdentity(new Func(`gt`, [age, new Value(18)])), + ).toBe(getProjectedExpressionIdentity(new Func(`lt`, [new Value(18), age]))) + }) + + it(`preserves order-sensitive function arguments`, () => { + expect( + getProjectedExpressionIdentity(new Func(`subtract`, [age, new Value(1)])), + ).not.toBe( + getProjectedExpressionIdentity(new Func(`subtract`, [new Value(1), age])), + ) + }) + + fcTest.prop([ + equivalentExpressionPairArbitrary, + fc.record({ age: fc.integer(), active: fc.boolean() }), + ])(`canonical expression grammar preserves semantics`, (pair, sample) => { + const row = { user: sample } + + expect(compileExpression(pair.original)(row)).toBe( + compileExpression(pair.equivalent)(row), + ) + expect(getProjectedExpressionIdentity(pair.original)).toBe( + getProjectedExpressionIdentity(pair.equivalent), + ) + }) + + fcTest.prop([referenceSemanticPairArbitrary])( + `keeps reference-semantic values distinct across expression and demand identity`, + ([first, second]) => { + const value = new PropRef([`row`, `value`]) + const firstPredicate = new Func(`eq`, [value, new Value(first)]) + const secondPredicate = new Func(`eq`, [ + value, + new Value(second), + ]) + const row = { row: { value: first } } + + expect(compileExpression(firstPredicate)(row)).toBe(true) + expect(compileExpression(secondPredicate)(row)).toBe(false) + expect(getProjectedExpressionIdentity(firstPredicate)).not.toBe( + getProjectedExpressionIdentity(secondPredicate), + ) + expect( + getLoadSubsetDemandKey({ where: firstPredicate, limit: 1 }), + ).not.toBe(getLoadSubsetDemandKey({ where: secondPredicate, limit: 1 })) + }, + ) + + it(`does not initialize runtime reference identities during module evaluation`, async () => { + const getRandomValues = vi.fn((values: Uint32Array) => values) + vi.stubGlobal(`crypto`, { getRandomValues }) + vi.resetModules() + + try { + const { getRuntimeReferenceIdentity: getFreshRuntimeReferenceIdentity } = + await import(`../../src/query/runtime-reference-identity.js`) + + expect(getRandomValues).not.toHaveBeenCalled() + + getFreshRuntimeReferenceIdentity({}) + getFreshRuntimeReferenceIdentity({}) + + expect(getRandomValues).toHaveBeenCalledOnce() + } finally { + vi.unstubAllGlobals() + } + }) + + it(`does not reuse reference identities across runtimes`, () => { + const firstRuntime = createRuntimeReferenceIdentityFactory() + const secondRuntime = createRuntimeReferenceIdentityFactory() + + expect(firstRuntime({ a: 1 })).not.toEqual(secondRuntime({ b: 2 })) + }) + + it(`allocates runtime entropy only when the first identity is requested`, () => { + const getRandomValues = vi.fn((values: Uint32Array) => values) + vi.stubGlobal(`crypto`, { getRandomValues }) + try { + const runtime = createRuntimeReferenceIdentityFactory() + expect(getRandomValues).not.toHaveBeenCalled() + + runtime({}) + runtime({}) + expect(getRandomValues).toHaveBeenCalledOnce() + } finally { + vi.unstubAllGlobals() + } + }) + + it(`keeps symbol identities stable and distinct`, () => { + const first = Symbol(`value`) + const second = Symbol(`value`) + + expect(getRuntimeReferenceIdentity(first)).toEqual( + getRuntimeReferenceIdentity(first), + ) + expect(getRuntimeReferenceIdentity(first)).not.toEqual( + getRuntimeReferenceIdentity(second), + ) + }) + + it(`does not retain symbols in strong identity maps when weak symbol keys are supported`, () => { + const NativeMap = Map + const stronglyStoredSymbols = new Set() + class TrackingMap extends NativeMap { + override set(key: K, value: V): this { + if (typeof key === `symbol`) stronglyStoredSymbols.add(key) + return super.set(key, value) + } + } + const local = Symbol(`local`) + const registered = Symbol.for( + `tanstack-db-runtime-reference-test-${Date.now()}`, + ) + + vi.stubGlobal(`Map`, TrackingMap) + try { + const runtime = createRuntimeReferenceIdentityFactory() + runtime(local) + runtime(registered) + } finally { + vi.unstubAllGlobals() + } + + expect(stronglyStoredSymbols).not.toContain(local) + expect(stronglyStoredSymbols).not.toContain(registered) + }) + + it(`keeps correct symbol identity when weak symbol keys are unavailable`, () => { + const NativeWeakMap = WeakMap + class ObjectOnlyWeakMap extends NativeWeakMap { + override set(key: K, value: V): this { + if (typeof key === `symbol`) { + throw new TypeError(`Symbols cannot be weak keys`) + } + return super.set(key, value) + } + } + const first = Symbol(`value`) + const second = Symbol(`value`) + + vi.stubGlobal(`WeakMap`, ObjectOnlyWeakMap) + try { + const runtime = createRuntimeReferenceIdentityFactory() + + expect(runtime(first)).toEqual(runtime(first)) + expect(runtime(first)).not.toEqual(runtime(second)) + } finally { + vi.unstubAllGlobals() + } + }) + + it(`scopes opaque value identities to their owner`, () => { + const firstScope = createValueIdentity() + const secondScope = createValueIdentity() + const first = Symbol(`value`) + const second = Symbol(`value`) + + expect(firstScope.equality(first)).toEqual(firstScope.equality(first)) + expect(firstScope.equality(first)).not.toEqual(firstScope.equality(second)) + expect(firstScope.equality(first)).not.toEqual(secondScope.equality(first)) + }) + + it(`falls back when the runtime crypto object lacks getRandomValues`, () => { + vi.stubGlobal(`crypto`, {}) + try { + const runtime = createRuntimeReferenceIdentityFactory() + + expect(runtime({ a: 1 })).toEqual([ + `runtimeReference`, + expect.any(String), + 1, + ]) + } finally { + vi.unstubAllGlobals() + } + }) + + fcTest.prop([ + fc.uniqueArray(fc.oneof(fc.integer(), fc.string(), fc.boolean()), { + minLength: 1, + maxLength: 8, + }), + ])(`treats IN candidates as a set`, (candidates) => { + const value = new PropRef([`row`, `value`]) + const ordered = new Func(`in`, [value, new Value(candidates)]) + const reordered = new Func(`in`, [ + value, + new Value([...candidates].reverse().concat(candidates[0]!)), + ]) + + for (const candidate of candidates) { + const row = { row: { value: candidate } } + expect(compileExpression(ordered)(row)).toBe( + compileExpression(reordered)(row), + ) + } + expect(getProjectedExpressionIdentity(ordered)).toBe( + getProjectedExpressionIdentity(reordered), + ) + expect(getLoadSubsetDemandKey({ where: ordered })).toBe( + getLoadSubsetDemandKey({ where: reordered }), + ) + }) +}) + +describe(`loadSubset demand identity`, () => { + const id = new PropRef([`id`]) + const group = new PropRef([`group`]) + const first = new Func(`eq`, [id, new Value(`a`)]) + const second = new Func(`eq`, [group, new Value(`x`)]) + const orderBy: NonNullable = [ + { + expression: id, + compareOptions: { direction: `asc`, nulls: `first` }, + }, + { + expression: group, + compareOptions: { direction: `desc`, nulls: `last` }, + }, + ] + + it(`includes the exact requested window`, () => { + const narrow = { where: first, orderBy, limit: 10, offset: 5 } + const wide = { where: first, orderBy, limit: 20, offset: 0 } + + expect(getLoadSubsetDemandKey(narrow)).not.toBe( + getLoadSubsetDemandKey(wide), + ) + }) + + it(`normalizes predicates but preserves orderBy sequence`, () => { + const left = new Func(`and`, [first, second]) + const right = new Func(`and`, [second, first]) + + expect(getLoadSubsetDemandKey({ where: left, orderBy })).toBe( + getLoadSubsetDemandKey({ where: right, orderBy }), + ) + expect(getLoadSubsetDemandKey({ where: left, orderBy })).not.toBe( + getLoadSubsetDemandKey({ where: right, orderBy: [...orderBy].reverse() }), + ) + }) + + it(`includes cursor shape and excludes runtime owners`, () => { + const cursor = { + whereFrom: new Func(`gt`, [id, new Value(`a`)]), + whereCurrent: first, + lastKey: `a`, + } + const firstOwner = new AbortController() + const secondOwner = new AbortController() + const subscription = {} as NonNullable + + expect( + getLoadSubsetDemandKey({ + where: first, + cursor, + signal: firstOwner.signal, + subscription, + }), + ).toBe( + getLoadSubsetDemandKey({ + where: first, + cursor, + signal: secondOwner.signal, + }), + ) + expect(getLoadSubsetDemandKey({ where: first, cursor })).not.toBe( + getLoadSubsetDemandKey({ + where: first, + cursor: { ...cursor, lastKey: `b` }, + }), + ) + }) + + it(`uses the base query key for an unconstrained owner-only demand`, () => { + expect(getLoadSubsetDemandKey({})).toBeUndefined() + expect(getLoadSubsetDemandKey({ offset: 0 })).toBeUndefined() + expect( + getLoadSubsetDemandKey({ signal: new AbortController().signal }), + ).toBeUndefined() + expect(getLoadSubsetDemandKey({ where: first, offset: 0 })).toBe( + getLoadSubsetDemandKey({ where: first }), + ) + }) + + it.each([ + [`function`, () => `value`, () => `value`], + [`symbol`, Symbol(`value`), Symbol(`value`)], + ])(`uses runtime reference identity for %s demand values`, (_name, a, b) => { + const field = new PropRef([`row`, `value`]) + const demand = (value: unknown): LoadSubsetOptions => ({ + where: new Func(`eq`, [field, new Value(value)]), + }) + + expect(getLoadSubsetDemandKey(demand(a))).toBe( + getLoadSubsetDemandKey(demand(a)), + ) + expect(getLoadSubsetDemandKey(demand(a))).not.toBe( + getLoadSubsetDemandKey(demand(b)), + ) + }) + + it.each([ + [`signed zero`, -0, 0], + [`invalid Date`, new Date(Number.NaN), new Date(Number.NaN)], + [ + `Temporal.PlainDate`, + Temporal.PlainDate.from(`2024-01-15`), + Temporal.PlainDate.from(`2024-01-15`), + ], + [ + `Temporal.Duration`, + Temporal.Duration.from(`PT1H`), + Temporal.Duration.from(`PT1H`), + ], + [ + `large cross-constructor binary`, + new Uint8Array(129).fill(7), + Buffer.from(new Uint8Array(129).fill(7)), + ], + ])( + `uses comparison semantics for equivalent %s values`, + (_label, firstValue, secondValue) => { + const value = new PropRef([`row`, `value`]) + const firstPredicate = new Func(`eq`, [ + value, + new Value(firstValue), + ]) + const secondPredicate = new Func(`eq`, [ + value, + new Value(secondValue), + ]) + + expect( + compileExpression(firstPredicate)({ row: { value: secondValue } }), + ).toBe(true) + expect(getProjectedExpressionIdentity(firstPredicate)).toBe( + getProjectedExpressionIdentity(secondPredicate), + ) + expect(getLoadSubsetDemandKey({ where: firstPredicate })).toBe( + getLoadSubsetDemandKey({ where: secondPredicate }), + ) + expect(getQueryIdentity(createProfileValueQuery(firstValue))).toBe( + getQueryIdentity(createProfileValueQuery(secondValue)), + ) + }, + ) +}) + +const postsCollection = new CollectionImpl({ + id: `posts`, + getKey: (item) => item.id, + sync: { sync: () => {} }, +}) + +function createProfileValueQuery(value: unknown): QueryIR { + return { + ...getQueryIR(new Query().from({ user: usersCollection })), + where: [ + new Func(`eq`, [ + new PropRef([`user`, `profile`]), + new Value(value), + ]), + ], + } +} + +function createAlphaRenamedJoinQuery( + userAlias: string, + postAlias: string, +): QueryIR { + return { + from: new CollectionRef( + usersCollection as unknown as ConstructorParameters< + typeof CollectionRef + >[0], + userAlias, + ), + join: [ + { + type: `inner`, + from: new CollectionRef( + postsCollection as unknown as ConstructorParameters< + typeof CollectionRef + >[0], + postAlias, + ), + left: new PropRef([userAlias, `id`]), + right: new PropRef([postAlias, `userId`]), + }, + ], + where: [ + new Func(`eq`, [new PropRef([postAlias, `published`]), new Value(true)]), + ], + select: { + userId: new PropRef([userAlias, `id`]), + postTitle: new PropRef([postAlias, `title`]), + }, + orderBy: [ + { + expression: new PropRef([postAlias, `createdAt`]), + compareOptions: { direction: `desc`, nulls: `last` }, + }, + ], + } +} + +function createAlphaRenamedImplicitJoinQuery( + userAlias: string, + postAlias: string, +): QueryIR { + return { + from: new CollectionRef( + usersCollection as unknown as ConstructorParameters< + typeof CollectionRef + >[0], + userAlias, + ), + join: [ + { + type: `inner`, + from: new CollectionRef( + postsCollection as unknown as ConstructorParameters< + typeof CollectionRef + >[0], + postAlias, + ), + left: new PropRef([userAlias, `id`]), + right: new PropRef([postAlias, `userId`]), + }, + ], + } +} + +function createProjectedExpressionQuery(expression: BasicExpression): QueryIR { + return { + from: new CollectionRef( + usersCollection as unknown as ConstructorParameters< + typeof CollectionRef + >[0], + `user`, + ), + select: { value: expression }, + } +} + +function createAlphaRenamedNestedQuery( + innerAlias: string, + outerAlias: string, +): QueryIR { + const inner: QueryIR = { + from: new CollectionRef( + usersCollection as unknown as ConstructorParameters< + typeof CollectionRef + >[0], + innerAlias, + ), + select: { + id: new PropRef([innerAlias, `id`]), + status: new PropRef([innerAlias, `status`]), + }, + } + return { + from: new QueryRef(inner, outerAlias), + where: [ + new Func(`eq`, [ + new PropRef([outerAlias, `status`]), + new Value(`active`), + ]), + ], + select: { id: new PropRef([outerAlias, `id`]) }, + } +} + +function createUnionDerivedNestedQuery(outerAlias: string): QueryIR { + const users = new Query() + .from({ user: usersCollection }) + .select(({ user }) => ({ id: user.id, kind: user.status })) + const posts = new Query() + .from({ post: postsCollection }) + .select(({ post }) => ({ id: post.id, kind: post.title })) + const union = new Query() + .unionAll(users, posts) + .where(({ kind }) => eq(kind, `active`)) + + return getQueryIR( + new Query().from({ [outerAlias]: union } as Record), + ) +} + +function createUnionDerivedNestedOutputQuery(outerAlias: string): QueryIR { + const users = new Query() + .from({ user: usersCollection }) + .select(({ user }) => ({ profile: { id: user.id } })) + const posts = new Query() + .from({ post: postsCollection }) + .select(({ post }) => ({ profile: { id: post.id } })) + const union = new Query() + .unionAll(users, posts) + .where(({ profile }) => eq(profile.id, 1)) + + return getQueryIR( + new Query().from({ [outerAlias]: union } as Record), + ) +} + +function createUnionDerivedIncludesQuery(parentAlias: string): QueryIR { + const firstPosts = new Query() + .from({ post: postsCollection }) + .select(({ post }) => ({ id: post.id, userId: post.userId })) + const secondPosts = new Query() + .from({ otherPost: postsCollection }) + .select(({ otherPost }) => ({ + id: otherPost.id, + userId: otherPost.userId, + })) + const childQuery = getQueryIR(new Query().unionAll(firstPosts, secondPosts)) + const posts = new IncludesSubquery( + childQuery, + new PropRef([parentAlias, `id`]), + new PropRef([`userId`]), + `posts`, + undefined, + undefined, + `array`, + ) + + return { + from: new CollectionRef( + usersCollection as unknown as ConstructorParameters< + typeof CollectionRef + >[0], + parentAlias, + ), + select: { + id: new PropRef([parentAlias, `id`]), + posts, + }, + } +} + +function createCorrelatedUnionIncludesQuery(parentAlias: string): QueryIR { + const createBranch = (childAlias: string): QueryIR => ({ + from: new CollectionRef( + postsCollection as unknown as ConstructorParameters< + typeof CollectionRef + >[0], + childAlias, + ), + select: { + profile: { id: new PropRef([childAlias, `id`]) }, + userId: new PropRef([childAlias, `userId`]), + parentAge: new PropRef([parentAlias, `age`]), + }, + }) + const childQuery: QueryIR = { + from: new UnionAll([createBranch(`firstPost`), createBranch(`secondPost`)]), + where: [new Func(`eq`, [new PropRef([`profile`, `id`]), new Value(1)])], + } + const posts = new IncludesSubquery( + childQuery, + new PropRef([parentAlias, `id`]), + new PropRef([`userId`]), + `posts`, + undefined, + [new PropRef([parentAlias, `age`])], + `array`, + ) + + return { + from: new CollectionRef( + usersCollection as unknown as ConstructorParameters< + typeof CollectionRef + >[0], + parentAlias, + ), + select: { + id: new PropRef([parentAlias, `id`]), + posts, + }, + } +} + +function createAlphaRenamedIncludesQuery( + parentAlias: string, + childAlias: string, +): QueryIR { + const childQuery: QueryIR = { + from: new CollectionRef( + postsCollection as unknown as ConstructorParameters< + typeof CollectionRef + >[0], + childAlias, + ), + select: { + id: new PropRef([childAlias, `id`]), + title: new PropRef([childAlias, `title`]), + }, + } + const posts = new IncludesSubquery( + childQuery, + new PropRef([parentAlias, `id`]), + new PropRef([childAlias, `userId`]), + `posts`, + [ + new Func(`eq`, [ + new PropRef([parentAlias, `status`]), + new Value(`active`), + ]), + ], + [new PropRef([parentAlias, `id`])], + `array`, + ) + return { + from: new CollectionRef( + usersCollection as unknown as ConstructorParameters< + typeof CollectionRef + >[0], + parentAlias, + ), + select: { + id: new PropRef([parentAlias, `id`]), + posts, + }, + } +} + +const structuredQueries: Array<[string, () => QueryIR]> = [ + [ + `basic collection source`, + () => getQueryIR(new Query().from({ user: usersCollection })), + ], + [ + `captured primitive where value`, + () => { + const status = `active` as const + return getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => eq(user.status, status)), + ) + }, + ], + [ + `boolean expression tree`, + () => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => + and( + eq(user.active, true), + or(gt(user.age, 30), not(isNull(user.email))), + ), + ), + ), + ], + [ + `array membership and undefined checks`, + () => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => + and( + inArray(user.teamId, [`eng`, `design`]), + not(isUndefined(user.profile)), + ), + ), + ), + ], + [ + `date bigint and typed array values`, + () => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => + and( + gte(user.createdAt, new Date(`2024-01-01T00:00:00.000Z`)), + gt(user.largeViewCount, 9007199254740993n), + eq(user.blob, new Uint8Array([1, 2, 3])), + ), + ), + ), + ], + [ + `plain object values`, + () => + getQueryIR( + new Query().from({ user: usersCollection }).where(({ user }) => + eq(user.profile, { + experience: { years: 5 }, + skills: [`ts`, `db`], + }), + ), + ), + ], + [ + `nested select and computed expressions`, + () => + getQueryIR( + new Query().from({ user: usersCollection }).select(({ user }) => ({ + id: user.id, + displayName: concat(upper(user.name), ` <`, lower(user.email), `>`), + score: add(user.salary, 1000), + fallbackEmail: coalesce(user.email, `missing@example.com`), + meta: { + active: user.active, + nameLength: length(user.name), + }, + })), + ), + ], + [ + `conditional projection select`, + () => + getQueryIR( + new Query().from({ user: usersCollection }).select(({ user }) => ({ + id: user.id, + profile: caseWhen( + gt(user.age, 18), + { + label: `adult`, + email: user.email, + }, + { + label: `minor`, + email: null, + }, + ), + })), + ), + ], + [ + `top-level alias spread select`, + () => + getQueryIR( + new Query().from({ user: usersCollection }).select(({ user }) => user), + ), + ], + [ + `locale orderBy options`, + () => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .orderBy(({ user }) => user.name, { + direction: `asc`, + nulls: `last`, + stringSort: `locale`, + locale: `en-US`, + localeOptions: { sensitivity: `base`, numeric: true }, + }), + ), + ], + [ + `groupBy aggregates and selected orderBy`, + () => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .groupBy(({ user }) => user.teamId) + .select(({ user }) => ({ + teamId: user.teamId, + userCount: count(user.id), + avgAge: avg(user.age), + totalSalary: sum(user.salary), + latestSignup: max(user.createdAt), + })) + .having(({ $selected }) => gt($selected.userCount, 1)) + .orderBy(({ $selected }) => $selected.avgAge, `desc`), + ), + ], + [ + `join query`, + () => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .join( + { post: postsCollection }, + ({ user, post }) => eq(user.id, post.userId), + `left`, + ) + .where(({ post }) => eq(post.published, true)) + .select(({ user, post }) => ({ + userId: user.id, + postTitle: post.title, + })), + ), + ], + [ + `subquery join`, + () => + getQueryIR( + new Query() + .from({ + post: new Query() + .from({ post: postsCollection }) + .where(({ post }) => gt(post.views, 100)), + }) + .join( + { + activeUser: new Query() + .from({ user: usersCollection }) + .where(({ user }) => eq(user.status, `active`)), + }, + ({ post, activeUser }) => eq(post.userId, activeUser.id), + `inner`, + ), + ), + ], + [ + `unioned source object`, + () => + getQueryIR( + new Query().unionAll({ user: usersCollection, post: postsCollection }), + ), + ], + [ + `unioned query branches`, + () => + getQueryIR( + new Query().unionAll( + new Query().from({ user: usersCollection }).select(({ user }) => ({ + id: user.id, + label: user.name, + })), + new Query().from({ post: postsCollection }).select(({ post }) => ({ + id: post.id, + label: post.title, + })), + ), + ), + ], + [ + `includes subquery`, + () => + getQueryIR( + new Query().from({ user: usersCollection }).select(({ user }) => ({ + id: user.id, + posts: new Query() + .from({ post: postsCollection }) + .where(({ post }) => eq(post.userId, user.id)) + .select(({ post }) => ({ + id: post.id, + title: post.title, + })), + })), + ), + ], + [ + `pagination shape`, + () => + getQueryIR( + new Query() + .from({ post: postsCollection }) + .where(({ post }) => like(post.title, `%db%`)) + .orderBy(({ post }) => post.createdAt, `desc`) + .offset(20) + .limit(10), + ), + ], +] + +describe(`stable QueryIR identity smoke test`, () => { + it(`can derive identity for representative structured query shapes`, () => { + expect(structuredQueries).toHaveLength(17) + + const hashes = structuredQueries.map(([name, createQuery]) => { + const hash = getStableQueryIRHash(createQuery()) + expect(hash, name).toContain(`"type":"query"`) + expect(() => JSON.parse(hash), name).not.toThrow() + return hash + }) + + expect(new Set(hashes).size).toBe(hashes.length) + }) + + it(`does not depend on collection object identity when ids match`, () => { + const otherUsersCollection = new CollectionImpl({ + id: `users`, + getKey: (item) => item.id, + sync: { sync: () => {} }, + }) + + const createQuery = (collection: CollectionImpl) => + getQueryIR( + new Query() + .from({ user: collection }) + .where(({ user }) => eq(user.status, `active`)), + ) + + expect(getStableQueryIRHash(createQuery(usersCollection))).toBe( + getStableQueryIRHash(createQuery(otherUsersCollection)), + ) + }) + + fcTest.prop([ + fc.uniqueArray(fc.stringMatching(/^[a-z][a-z0-9]{0,8}$/), { + minLength: 4, + maxLength: 4, + }), + ])(`does not depend on lexical source aliases`, (aliases) => { + const [firstUser, firstPost, secondUser, secondPost] = aliases + + expect( + getQueryIdentity(createAlphaRenamedJoinQuery(firstUser!, firstPost!)), + ).toBe( + getQueryIdentity(createAlphaRenamedJoinQuery(secondUser!, secondPost!)), + ) + expect( + getQueryIdentity(createAlphaRenamedNestedQuery(firstUser!, firstPost!)), + ).toBe( + getQueryIdentity(createAlphaRenamedNestedQuery(secondUser!, secondPost!)), + ) + expect( + getQueryIdentity(createAlphaRenamedIncludesQuery(firstUser!, firstPost!)), + ).toBe( + getQueryIdentity( + createAlphaRenamedIncludesQuery(secondUser!, secondPost!), + ), + ) + }) + + it(`keeps aliases that define an implicit joined result shape`, () => { + expect( + getQueryIdentity(createAlphaRenamedImplicitJoinQuery(`user`, `post`)), + ).not.toBe( + getQueryIdentity( + createAlphaRenamedImplicitJoinQuery(`account`, `article`), + ), + ) + }) + + it(`keeps aliases that define an implicit union-source result shape`, () => { + const usersAndPosts = getQueryIR( + new Query().unionAll({ + user: usersCollection, + post: postsCollection, + }), + ) + const accountsAndArticles = getQueryIR( + new Query().unionAll({ + account: usersCollection, + article: postsCollection, + }), + ) + + expect(usersAndPosts.from.type).toBe(`unionFrom`) + expect(getQueryIdentity(usersAndPosts)).not.toBe( + getQueryIdentity(accountsAndArticles), + ) + }) + + it(`keeps aliases when an empty groupBy still selects a namespaced row`, () => { + const createQuery = (alias: string) => + getQueryIR( + new Query() + .from({ [alias]: usersCollection } as Record< + string, + typeof usersCollection + >) + .groupBy(() => []), + ) + + expect(getQueryIdentity(createQuery(`user`))).not.toBe( + getQueryIdentity(createQuery(`account`)), + ) + }) + + it(`keeps output-producing runtime values exact`, () => { + const bufferExpression = new Func(`concat`, [new Value(Buffer.from([65]))]) + const uint8Expression = new Func(`concat`, [ + new Value(new Uint8Array([65])), + ]) + + expect(compileExpression(bufferExpression)({})).toBe(`A`) + expect(compileExpression(uint8Expression)({})).toBe(`65`) + expect( + getQueryIdentity(createProjectedExpressionQuery(bufferExpression)), + ).not.toBe( + getQueryIdentity(createProjectedExpressionQuery(uint8Expression)), + ) + + expect( + getQueryIdentity(createProjectedExpressionQuery(new Value(-0))), + ).not.toBe(getQueryIdentity(createProjectedExpressionQuery(new Value(0)))) + + const firstObject = { value: 1 } + const secondObject = { value: 1 } + expect( + getQueryIdentity(createProjectedExpressionQuery(new Value(firstObject))), + ).not.toBe( + getQueryIdentity(createProjectedExpressionQuery(new Value(secondObject))), + ) + expect(compileExpression(new Value(firstObject))({})).toBe(firstObject) + expect(compileExpression(new Value(secondObject))({})).toBe(secondObject) + }) + + fcTest.prop([outputExpressionPairArbitrary])( + `equal query identities imply equal projected expression results`, + ({ first, second }) => { + const firstIdentity = getQueryIdentity( + createProjectedExpressionQuery(first), + ) + const secondIdentity = getQueryIdentity( + createProjectedExpressionQuery(second), + ) + + if (firstIdentity === secondIdentity) { + expect( + Object.is( + compileExpression(first)({}), + compileExpression(second)({}), + ), + ).toBe(true) + } + }, + ) + + fcTest.prop([ + fc + .stringMatching(/^[a-z][a-z0-9]{0,8}$/) + .filter( + (alias) => + alias !== `kind` && alias !== `profile` && alias !== `userId`, + ), + ])(`does not bind union-derived output fields to outer aliases`, (alias) => { + expect(getQueryIdentity(createUnionDerivedNestedQuery(`kind`))).toBe( + getQueryIdentity(createUnionDerivedNestedQuery(alias)), + ) + expect( + getQueryIdentity(createUnionDerivedNestedOutputQuery(`profile`)), + ).toBe(getQueryIdentity(createUnionDerivedNestedOutputQuery(alias))) + expect(getQueryIdentity(createUnionDerivedIncludesQuery(`userId`))).toBe( + getQueryIdentity(createUnionDerivedIncludesQuery(alias)), + ) + expect( + getQueryIdentity(createCorrelatedUnionIncludesQuery(`profile`)), + ).toBe(getQueryIdentity(createCorrelatedUnionIncludesQuery(alias))) + }) + + it(`shares identity across equivalent predicate formulations`, () => { + const left = getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => and(eq(user.status, `active`), gt(user.age, 18))), + ) + const right = getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => and(lt(18, user.age), eq(`active`, user.status))), + ) + + expect(getQueryIdentity(left)).toBe(getQueryIdentity(right)) + }) + + it(`normalizes the implicit conjunction order of repeated where clauses`, () => { + const left = getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => eq(user.status, `active`)) + .where(({ user }) => gt(user.age, 18)), + ) + const right = getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => gt(user.age, 18)) + .where(({ user }) => eq(user.status, `active`)), + ) + const duplicate = getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => eq(user.status, `active`)) + .where(({ user }) => gt(user.age, 18)) + .where(({ user }) => eq(user.status, `active`)), + ) + + expect(getQueryIdentity(left)).toBe(getQueryIdentity(right)) + expect(getQueryIdentity(left)).toBe(getQueryIdentity(duplicate)) + }) + + it(`normalizes the implicit conjunction order of repeated having clauses`, () => { + const left = getQueryIR( + new Query() + .from({ user: usersCollection }) + .groupBy(({ user }) => user.teamId) + .select(({ user }) => ({ + teamId: user.teamId, + userCount: count(user.id), + averageAge: avg(user.age), + })) + .having(({ $selected }) => gt($selected.userCount, 1)) + .having(({ $selected }) => gt($selected.averageAge, 18)), + ) + const right = getQueryIR( + new Query() + .from({ user: usersCollection }) + .groupBy(({ user }) => user.teamId) + .select(({ user }) => ({ + teamId: user.teamId, + userCount: count(user.id), + averageAge: avg(user.age), + })) + .having(({ $selected }) => gt($selected.averageAge, 18)) + .having(({ $selected }) => gt($selected.userCount, 1)), + ) + + expect(getQueryIdentity(left)).toBe(getQueryIdentity(right)) + }) + + it(`includes a query plan's result window`, () => { + const createQuery = (limit: number) => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .orderBy(({ user }) => user.age) + .limit(limit), + ) + + expect(getQueryIdentity(createQuery(10))).not.toBe( + getQueryIdentity(createQuery(20)), + ) + }) + + it(`elides the default query offset`, () => { + const base = getQueryIR(new Query().from({ user: usersCollection })) + const offsetZero = getQueryIR( + new Query().from({ user: usersCollection }).offset(0), + ) + + expect(getQueryIdentity(base)).toBe(getQueryIdentity(offsetZero)) + }) + + it(`preserves function-argument and orderBy-clause order`, () => { + const subtractAge = getQueryIR( + new Query() + .from({ user: usersCollection }) + .orderBy(({ user }) => subtract(user.age, 1)) + .orderBy(({ user }) => user.name), + ) + const subtractFromOne = getQueryIR( + new Query() + .from({ user: usersCollection }) + .orderBy(({ user }) => subtract(1, user.age)) + .orderBy(({ user }) => user.name), + ) + const reversedClauses = getQueryIR( + new Query() + .from({ user: usersCollection }) + .orderBy(({ user }) => user.name) + .orderBy(({ user }) => subtract(user.age, 1)), + ) + + expect(getQueryIdentity(subtractAge)).not.toBe( + getQueryIdentity(subtractFromOne), + ) + expect(getQueryIdentity(subtractAge)).not.toBe( + getQueryIdentity(reversedClauses), + ) + }) + + it(`preserves semantically significant union source ordering`, () => { + const usersThenPosts = getQueryIR( + new Query().unionAll({ user: usersCollection, post: postsCollection }), + ) + const postsThenUsers = getQueryIR( + new Query().unionAll({ post: postsCollection, user: usersCollection }), + ) + + expect(getStableQueryIRHash(usersThenPosts)).not.toBe( + getStableQueryIRHash(postsThenUsers), + ) + }) + + it(`changes identity when captured structured values change`, () => { + const createQuery = (status: User[`status`]) => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => eq(user.status, status)), + ) + + expect(getStableQueryIRHash(createQuery(`active`))).not.toBe( + getStableQueryIRHash(createQuery(`inactive`)), + ) + }) + + fcTest.prop([referenceSemanticPairArbitrary])( + `keeps queries distinct when captured values compare by reference`, + ([first, second]) => { + const firstQuery = createProfileValueQuery(first) + const secondQuery = createProfileValueQuery(second) + const firstPredicate = firstQuery.where![0] as BasicExpression + const secondPredicate = secondQuery.where![0] as BasicExpression + const row = { user: { profile: first } } + + expect(compileExpression(firstPredicate)(row)).toBe(true) + expect(compileExpression(secondPredicate)(row)).toBe(false) + expect(getQueryIdentity(firstQuery)).not.toBe( + getQueryIdentity(secondQuery), + ) + }, + ) + + it(`uses evaluator semantics for invalid Date and Temporal values`, () => { + expect(getQueryIdentity(createProfileValueQuery(new Date(`invalid`)))).toBe( + getQueryIdentity(createProfileValueQuery(new Date(`also invalid`))), + ) + expect( + getQueryIdentity( + createProfileValueQuery(Temporal.PlainDate.from(`2026-08-24`)), + ), + ).toBe( + getQueryIdentity( + createProfileValueQuery(Temporal.PlainDate.from(`2026-08-24`)), + ), + ) + }) + + it(`normalizes object property ordering in structural value hashes`, () => { + expect( + getStableValueHash({ + skills: [`ts`, `db`], + experience: { years: 5 }, + }), + ).toBe( + getStableValueHash({ + experience: { years: 5 }, + skills: [`ts`, `db`], + }), + ) + }) + + it(`keeps runtime values disjoint from internal identity tags`, () => { + const createQuery = (value: unknown) => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => eq(user.profile, value as never)), + ) + + const hashes = [ + undefined, + { type: `undefined` }, + [`undefined`], + Number.NaN, + { type: `number`, value: `NaN` }, + new Date(`2024-01-01T00:00:00.000Z`), + { type: `Date`, value: `2024-01-01T00:00:00.000Z` }, + ].map((value) => getStableQueryIRHash(createQuery(value))) + + expect(new Set(hashes).size).toBe(hashes.length) + }) + + it(`preserves __proto__ as a normal object key`, () => { + const withProtoKey = JSON.parse(`{"__proto__":{"value":true}}`) as object + const withoutProtoKey = {} + const createQuery = (value: object) => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => eq(user.profile, value as never)), + ) + + expect(getStableQueryIRHash(createQuery(withProtoKey))).not.toBe( + getStableQueryIRHash(createQuery(withoutProtoKey)), + ) + }) + + it(`rejects functional query variants`, () => { + const queries = [ + getQueryIR( + new Query() + .from({ user: usersCollection }) + .fn.where(({ user }) => user.active), + ), + getQueryIR( + new Query() + .from({ user: usersCollection }) + .fn.select(({ user }) => ({ id: user.id })), + ), + getQueryIR( + new Query() + .from({ user: usersCollection }) + .groupBy(({ user }) => user.teamId) + .select(({ user }) => ({ + teamId: user.teamId, + userCount: count(user.id), + })) + .fn.having(({ $selected }) => $selected.userCount > 1), + ), + ] + + for (const query of queries) { + expect(() => getStableQueryIRHash(query)).toThrow(UnhashableQueryIRError) + } + }) + + it.each([ + [`function`, () => `Tanner`, () => `Tanner`], + [`symbol`, Symbol(`name`), Symbol(`name`)], + ])(`keeps %s query values distinct by reference`, (_name, a, b) => { + const query = (value: unknown) => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => eq(user.name, value as never)), + ) + + expect(getStableQueryIRHash(query(a))).toBe(getStableQueryIRHash(query(a))) + expect(getStableQueryIRHash(query(a))).not.toBe( + getStableQueryIRHash(query(b)), + ) + }) + + it(`accepts opaque object values by reference`, () => { + const circularValue: Record = {} + circularValue.self = circularValue + + class OpaqueValue { + value = `Tanner` + } + + expect(() => + getQueryIdentity(createProfileValueQuery(circularValue)), + ).not.toThrow() + expect(() => + getQueryIdentity(createProfileValueQuery(new OpaqueValue())), + ).not.toThrow() + }) +}) diff --git a/packages/db/tests/query/join-subquery.test.ts b/packages/db/tests/query/join-subquery.test.ts index 093d6e62ae..c14e66b113 100644 --- a/packages/db/tests/query/join-subquery.test.ts +++ b/packages/db/tests/query/join-subquery.test.ts @@ -1,6 +1,7 @@ -import { beforeEach, describe, expect, test } from 'vitest' +import { afterEach, beforeEach, describe, expect, test, vi } from 'vitest' import { createLiveQueryCollection, eq, gt } from '../../src/query/index.js' import { createCollection } from '../../src/collection/index.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' import { mockSyncCollectionOptions, stripVirtualProps } from '../utils.js' // Sample data types for join-subquery testing @@ -475,7 +476,7 @@ function createJoinSubqueryTests(autoIndex: `off` | `eager`): void { }) }) - test(`should use subquery in LEFT JOIN clause - left join with ordered subquery with limit`, () => { + test(`should use subquery in LEFT JOIN clause - left join with ordered subquery with limit`, async () => { const joinSubquery = createLiveQueryCollection({ query: (q) => { return q @@ -497,6 +498,9 @@ function createJoinSubqueryTests(autoIndex: `off` | `eager`): void { startSync: true, }) + // Initial ordered refinement may hold publication beyond startSync. + await joinSubquery.preload() + expect(joinSubquery.isReady()).toBe(true) const results = joinSubquery.toArray.map((row) => ({ ...stripVirtualProps(row), issue: stripVirtualProps(row.issue), @@ -516,7 +520,7 @@ function createJoinSubqueryTests(autoIndex: `off` | `eager`): void { ]) }) - test(`should use subquery in RIGHT JOIN clause - left join with ordered subquery with limit`, () => { + test(`should use subquery in RIGHT JOIN clause - left join with ordered subquery with limit`, async () => { const joinSubquery = createLiveQueryCollection({ query: (q) => { return q @@ -538,6 +542,8 @@ function createJoinSubqueryTests(autoIndex: `off` | `eager`): void { startSync: true, }) + await joinSubquery.preload() + expect(joinSubquery.isReady()).toBe(true) const results = joinSubquery.toArray.map((row) => ({ ...stripVirtualProps(row), issue: stripVirtualProps(row.issue), @@ -873,3 +879,202 @@ describe(`Join with Subqueries`, () => { createJoinSubqueryTests(`off`) createJoinSubqueryTests(`eager`) }) + +describe(`Lazy join: subquery whose join key resolves to an indexed collection`, () => { + type Team = { id: string } + type Member = { id: string; teamId: string } + + // `teams.id` is indexed; `members` has no index. + const makeTeamsCollection = () => + createCollection( + mockSyncCollectionOptions({ + id: `lazy-join-teams`, + getKey: (r) => r.id, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + initialData: [{ id: `t1` }], + }), + ) + const makeMembersCollection = () => + createCollection( + mockSyncCollectionOptions({ + id: `lazy-join-members`, + getKey: (r) => r.id, + autoIndex: `off`, + initialData: [{ id: `m1`, teamId: `t1` }], + }), + ) + + let teams: ReturnType + let members: ReturnType + let warnSpy: ReturnType + + beforeEach(() => { + teams = makeTeamsCollection() + members = makeMembersCollection() + warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) + }) + + afterEach(() => { + warnSpy.mockRestore() + }) + + // When a subquery used in a JOIN clause selects its join key from the + // *joined* side of the subquery (here `team.id`) rather than from its own + // FROM side (`member`), the outer join key resolves to `teams.id`, which is + // indexed. The lazy-join loader should therefore load through that index and + // must not emit a "Join requires an index" warning that points at the + // already-indexed `teams` collection. + test(`does not warn about an index that the resolved collection already has`, () => { + const joinQuery = createLiveQueryCollection({ + startSync: true, + query: (q) => { + const teamByMember = q + .from({ member: members }) + .leftJoin({ team: teams }, ({ team, member }) => + eq(team.id, member.teamId), + ) + .select(({ team }) => ({ teamId: team.id })) + + return q + .from({ m: members }) + .leftJoin({ memberTeam: teamByMember }, ({ m, memberTeam }) => + eq(memberTeam.teamId, m.teamId), + ) + .select(({ m }) => ({ id: m.id })) + }, + }) + + // Data flows correctly regardless (via fallback full-load today). + expect(joinQuery.toArray.map((r) => r.id)).toEqual([`m1`]) + + const indexWarnings = warnSpy.mock.calls + .map((c) => String(c[0])) + .filter((m) => m.includes(`Join requires an index`)) + + // `teams.id` is already indexed, so no warning should advise indexing it. + expect(indexWarnings.filter((m) => m.includes(`lazy-join-teams`))).toEqual( + [], + ) + }) +}) + +describe(`Lazy join index availability`, () => { + test(`uses an auto-index with omitted locale options`, async () => { + type Team = { id: string } + type Member = { id: string; teamId: string } + const teams = createCollection( + mockSyncCollectionOptions({ + id: `lazy-default-collation-teams`, + getKey: (team) => team.id, + initialData: [{ id: `t1` }], + }), + ) + const members = createCollection( + mockSyncCollectionOptions({ + id: `lazy-default-collation-members`, + getKey: (member) => member.id, + initialData: [{ id: `m1`, teamId: `t1` }], + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + defaultStringCollation: { + stringSort: `locale`, + localeOptions: { sensitivity: undefined }, + }, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `m1`, teamId: `t1` } }) + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }), + ) + const warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const live = createLiveQueryCollection((q) => + q + .from({ team: teams }) + .leftJoin({ member: members }, ({ team, member }) => + eq(team.id, member.teamId), + ) + .select(({ team, member }) => ({ + id: team.id, + memberId: member.id, + })), + ) + + try { + await live.preload() + expect(live.toArray.map(stripVirtualProps)).toEqual([ + { id: `t1`, memberId: `m1` }, + ]) + expect(members.indexes.size).toBe(1) + expect(warnSpy).not.toHaveBeenCalledWith( + expect.stringContaining(`Join requires an index`), + ) + } finally { + warnSpy.mockRestore() + await Promise.all([live.cleanup(), teams.cleanup(), members.cleanup()]) + } + }) + + test(`warns when demand falls back to a full local scan`, async () => { + type Team = { id: string } + type Member = { id: string; teamId: string } + const teams = createCollection( + mockSyncCollectionOptions({ + id: `lazy-fallback-teams`, + getKey: (team) => team.id, + initialData: [{ id: `t1` }], + }), + ) + const members = createCollection( + mockSyncCollectionOptions({ + id: `lazy-fallback-members`, + getKey: (member) => member.id, + initialData: [{ id: `m1`, teamId: `t1` }], + syncMode: `on-demand`, + autoIndex: `off`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `m1`, teamId: `t1` } }) + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }), + ) + const warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const live = createLiveQueryCollection((q) => + q + .from({ team: teams }) + .leftJoin({ member: members }, ({ team, member }) => + eq(team.id, member.teamId), + ) + .select(({ team, member }) => ({ + id: team.id, + memberId: member.id, + })), + ) + + try { + await live.preload() + expect(live.toArray.map(stripVirtualProps)).toEqual([ + { id: `t1`, memberId: `m1` }, + ]) + expect(warnSpy).toHaveBeenCalledWith( + expect.stringContaining( + `[lazy-fallback-members] Join requires an index on "teamId"`, + ), + ) + } finally { + warnSpy.mockRestore() + await Promise.all([live.cleanup(), teams.cleanup(), members.cleanup()]) + } + }) +}) diff --git a/packages/db/tests/query/live-query-collection.test.ts b/packages/db/tests/query/live-query-collection.test.ts index 8b2b609016..2ef16a0db1 100644 --- a/packages/db/tests/query/live-query-collection.test.ts +++ b/packages/db/tests/query/live-query-collection.test.ts @@ -18,8 +18,13 @@ import { } from '../utils.js' import { createDeferred } from '../../src/deferred' import { BTreeIndex } from '../../src/indexes/btree-index' +import { createFilterFunctionFromExpression } from '../../src/collection/change-events' import { Func, Value } from '../../src/query/ir.js' -import type { ChangeMessage, LoadSubsetOptions } from '../../src/types.js' +import type { + ChangeMessage, + LoadSubsetOptions, + SyncConfig, +} from '../../src/types.js' // Sample user type for tests type User = { @@ -449,6 +454,21 @@ describe(`createLiveQueryCollection`, () => { }) }) + it(`should forward an explicit gcTime of 0 (disable GC) instead of coercing it to the default`, () => { + const options = liveQueryCollectionOptions({ + query: (q) => + q + .from({ user: usersCollection }) + .where(({ user }) => eq(user.active, true)), + gcTime: 0, + }) + + // gcTime: 0 disables garbage collection. A `|| 5000` fallback treats the + // explicit 0 as unset and silently replaces it with the 5s default, so the + // collection is garbage collected instead of being kept alive. + expect(options.gcTime).toBe(0) + }) + it(`should not reuse finalized graph after GC cleanup (resubscribe is safe)`, async () => { const liveQuery = createLiveQueryCollection({ query: (q) => @@ -594,6 +614,33 @@ describe(`createLiveQueryCollection`, () => { finalSubscription.unsubscribe() }) + it(`loads its data again when preloaded after the live query and its source collection were cleaned up`, async () => { + const activeUsers = createLiveQueryCollection({ + query: (q) => + q + .from({ user: usersCollection }) + .where(({ user }) => eq(user.active, true)), + }) + + await activeUsers.preload() + expect(activeUsers.status).toBe(`ready`) + expect(activeUsers.size).toBe(2) + + // Tear down the source collection and the live query, e.g. when switching + // to a different data set at runtime. Cleaning up a source collection puts + // the dependent live query into an error state. + await usersCollection.cleanup() + expect(activeUsers.status).toBe(`error`) + + await activeUsers.cleanup() + expect(activeUsers.status).toBe(`cleaned-up`) + + // Preloading again restarts sync and resolves once the data is loaded. + await activeUsers.preload() + expect(activeUsers.status).toBe(`ready`) + expect(activeUsers.size).toBe(2) + }) + it(`should handle temporal values correctly in live queries`, async () => { // Define a type with temporal values type Task = { @@ -1393,199 +1440,1543 @@ describe(`createLiveQueryCollection`, () => { expect(liveQuery.isLoadingSubset).toBe(false) }) - it(`concurrent live queries should each track loading state independently`, async () => { - // This tests the fix for the !wasLoadingBefore bug: - // When multiple live queries subscribe to the same source collection, - // each must independently track when loading finishes. - // Previously, only the first live query would track loading because - // wasLoadingBefore was true for subsequent queries. - - let resolveLoadSubset: () => void - const loadSubsetPromise = new Promise((resolve) => { - resolveLoadSubset = resolve - }) - - const sourceCollection = createCollection<{ id: number; value: number }>({ - id: `source-concurrent-lq`, - getKey: (item) => item.id, + it(`releases an ordered source when initial live-query loading throws`, async () => { + const failure = new Error(`initial ordered live-query load failed`) + const source = createCollection({ + id: `initial-ordered-live-query-error`, + getKey: (user) => user.id, syncMode: `on-demand`, - startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, sync: { - sync: ({ markReady, begin, write, commit }) => { - begin() - write({ type: `insert`, value: { id: 1, value: 10 } }) - commit() + sync: ({ markReady }) => { markReady() - return { - loadSubset: () => loadSubsetPromise, + loadSubset: () => { + throw failure + }, } }, }, }) + const live = createLiveQueryCollection((q) => + q + .from({ user: source }) + .orderBy(({ user }) => user.name, `asc`) + .limit(1), + ) - // Create TWO live queries that subscribe to the same source collection - const liveQuery1 = createLiveQueryCollection({ - query: (q) => q.from({ item: sourceCollection }), - startSync: true, - }) - - const liveQuery2 = createLiveQueryCollection({ - query: (q) => q.from({ item: sourceCollection }), - startSync: true, - }) - - // Wait for both subscriptions to start and trigger loadSubset - await flushPromises() - await new Promise((resolve) => setTimeout(resolve, 10)) - - // Source should be ready - expect(sourceCollection.isReady()).toBe(true) - - // Both live queries should be loading (not ready yet) - // KEY ASSERTION: Without the fix, liveQuery2 would be 'ready' here - // because it skipped tracking when wasLoadingBefore was true - expect(liveQuery1.status).toBe(`loading`) - expect(liveQuery2.status).toBe(`loading`) - - // Resolve the loadSubset promise - resolveLoadSubset!() - await flushPromises() - await new Promise((resolve) => setTimeout(resolve, 10)) + await expect(Promise.resolve().then(() => live.preload())).rejects.toBe( + failure, + ) + expect(source.subscriberCount).toBe(0) - // Now both should be ready - expect(liveQuery1.status).toBe(`ready`) - expect(liveQuery2.status).toBe(`ready`) + await Promise.all([live.cleanup(), source.cleanup()]) }) - }) - describe(`move functionality`, () => { - it(`should support moving orderBy window past current window using move function`, async () => { - // Create a collection with more users for testing window movement - const extendedUsers = createCollection( + it(`releases earlier live-query sources when initial lazy demand throws`, async () => { + type Issue = { id: number; userId: number } + const failure = new Error(`initial live-query lazy demand failed`) + const users = createCollection( mockSyncCollectionOptions({ - id: `extended-users`, + id: `partial-live-query-users`, getKey: (user) => user.id, - initialData: [ - { id: 1, name: `Alice`, active: true }, - { id: 2, name: `Bob`, active: true }, - { id: 3, name: `Charlie`, active: true }, - { id: 4, name: `David`, active: true }, - { id: 5, name: `Eve`, active: true }, - { id: 6, name: `Frank`, active: true }, - ], + initialData: [sampleUsers[0]!], }), ) - - const activeUsers = createLiveQueryCollection((q) => + const issues = createCollection({ + id: `partial-live-query-issues`, + getKey: (issue) => issue.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + throw failure + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => q - .from({ user: extendedUsers }) - .where(({ user }) => eq(user.active, true)) - .orderBy(({ user }) => user.name, `desc`) - .limit(3) - .offset(0), + .from({ user: users }) + .leftJoin({ issue: issues }, ({ user, issue }) => + eq(user.id, issue.userId), + ) + .select(({ user, issue }) => ({ + id: user.id, + issueId: issue.id, + })), ) - await activeUsers.preload() - - // Initial result should have first 3 users (Alice, Bob, Charlie) - expect(activeUsers.size).toBe(3) - const initialResults = activeUsers.toArray - expect(initialResults.map((r) => r.name)).toEqual([ - `Frank`, - `Eve`, - `David`, - ]) - - // Move the window to show users David, Eve, Frank (offset: 3, limit: 3) - activeUsers.utils.setWindow({ offset: 3, limit: 3 }) - - // Wait for the move to take effect - await new Promise((resolve) => setTimeout(resolve, 10)) + await expect(Promise.resolve().then(() => live.preload())).rejects.toBe( + failure, + ) + expect(users.subscriberCount).toBe(0) + expect(issues.subscriberCount).toBe(0) - const moveResults = activeUsers.toArray - expect(moveResults.map((r) => r.name)).toEqual([ - `Charlie`, - `Bob`, - `Alice`, - ]) + await Promise.all([live.cleanup(), users.cleanup(), issues.cleanup()]) }) - it(`should support moving orderBy window before current window using move function`, async () => { - const extendedUsers = createCollection( + it(`isolates synchronous lazy-demand failure from an established source commit`, async () => { + type Issue = { id: number; userId: number } + const failure = new Error(`incremental live-query lazy demand failed`) + const users = createCollection( mockSyncCollectionOptions({ - id: `extended-users-before`, + id: `incremental-live-query-users`, getKey: (user) => user.id, - initialData: [ - { id: 1, name: `Alice`, active: true }, - { id: 2, name: `Bob`, active: true }, - { id: 3, name: `Charlie`, active: true }, - { id: 4, name: `David`, active: true }, - { id: 5, name: `Eve`, active: true }, - { id: 6, name: `Frank`, active: true }, - ], + initialData: [], }), ) - - const activeUsers = createLiveQueryCollection((q) => + const issues = createCollection({ + id: `incremental-live-query-issues`, + getKey: (issue) => issue.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + throw failure + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => q - .from({ user: extendedUsers }) - .where(({ user }) => eq(user.active, true)) - .orderBy(({ user }) => user.name, `asc`) - .limit(3) - .offset(3), + .from({ user: users }) + .leftJoin({ issue: issues }, ({ user, issue }) => + eq(user.id, issue.userId), + ) + .select(({ user, issue }) => ({ + id: user.id, + issueId: issue.id, + })), ) - await activeUsers.preload() - - // Initial result should have users David, Eve, Frank - expect(activeUsers.size).toBe(3) - const initialResults = activeUsers.toArray - expect(initialResults.map((r) => r.name)).toEqual([ - `David`, - `Eve`, - `Frank`, - ]) - - // Move the window to show users Alice, Bob, Charlie (offset: 0, limit: 3) - activeUsers.utils.setWindow({ offset: 0, limit: 3 }) - - // Wait for the move to take effect - await new Promise((resolve) => setTimeout(resolve, 10)) + try { + await live.preload() + expect(live.status).toBe(`ready`) + + let commitError: unknown + try { + users.utils.begin() + users.utils.write({ type: `insert`, value: sampleUsers[0]! }) + users.utils.commit() + } catch (error) { + commitError = error + } - const moveResults = activeUsers.toArray - expect(moveResults.map((r) => r.name)).toEqual([ - `Alice`, - `Bob`, - `Charlie`, - ]) + expect(commitError).toBeUndefined() + expect(live.status).toBe(`error`) + expect(live.utils.lastSubsetError).toBe(failure) + } finally { + await Promise.all([live.cleanup(), users.cleanup(), issues.cleanup()]) + } }) - it(`should support moving offset while keeping limit constant`, async () => { - const extendedUsers = createCollection( - mockSyncCollectionOptions({ - id: `extended-users-offset`, - getKey: (user) => user.id, - initialData: [ - { id: 1, name: `Alice`, active: true }, - { id: 2, name: `Bob`, active: true }, - { id: 3, name: `Charlie`, active: true }, - { id: 4, name: `David`, active: true }, - { id: 5, name: `Eve`, active: true }, - ], - }), - ) + it.each([`throw`, `reject`] as const)( + `propagates lazy child demand failure from a window change ($0)`, + async (delivery) => { + type Parent = { id: number; rank: number } + type Child = { id: number; parentId: number } + const failure = new Error(`window child demand failed`) + const loadedParents = new Set() + let parentLoadCount = 0 + const parents = createCollection({ + id: `window-lazy-demand-parents`, + getKey: (parent) => parent.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + // Boundary refinement asks only for rows tied with rank 1. + // This source has already supplied that whole tie class. + if (options.where) return Promise.resolve() + parentLoadCount++ + begin() + const candidates: Array = [ + { id: 1, rank: 1 }, + { id: 2, rank: 2 }, + ] + candidates.slice(0, parentLoadCount).forEach((parent) => { + if (loadedParents.has(parent.id)) return + loadedParents.add(parent.id) + write({ type: `insert`, value: parent }) + }) + commit() + return Promise.resolve() + }, + } + }, + }, + }) + let childLoadCount = 0 + const children = createCollection({ + id: `window-lazy-demand-children`, + getKey: (child) => child.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: () => { + childLoadCount++ + if (childLoadCount > 1) { + if (delivery === `throw`) throw failure + return Promise.reject(failure) + } + begin() + write({ type: `insert`, value: { id: 10, parentId: 1 } }) + commit() + return Promise.resolve() + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents }) + .leftJoin({ child: children }, ({ parent, child }) => + eq(parent.id, child.parentId), + ) + .orderBy(({ parent }) => parent.rank, `asc`) + .limit(1) + .select(({ parent, child }) => ({ + id: parent.id, + childId: child.id, + })), + ) - const activeUsers = createLiveQueryCollection((q) => - q - .from({ user: extendedUsers }) - .where(({ user }) => eq(user.active, true)) - .orderBy(({ user }) => user.name, `asc`) - .limit(2) - .offset(0), - ) + try { + await live.preload() + expect(live.status).toBe(`ready`) - await activeUsers.preload() + const setWindow = async () => { + const result = live.utils.setWindow({ offset: 0, limit: 2 }) + if (result !== true) await result + } + await expect(setWindow()).rejects.toBe(failure) + expect(live.utils.lastSubsetError).toBe(failure) + } finally { + await Promise.all([ + live.cleanup(), + parents.cleanup(), + children.cleanup(), + ]) + } + }, + ) + + it(`retries the same ordered refill after a transient rejection`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`ordered refill failed`) + let loadCount = 0 + const acquisitions: Array = [] + const source = createCollection({ + id: `ordered-refill-retry-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + acquisitions.push(options) + loadCount++ + if (loadCount === 3) return Promise.reject(failure) + const deliver = (row: Row) => { + begin() + write({ type: `insert`, value: row }) + commit() + } + if (loadCount === 1) { + deliver({ id: 1, rank: 1 }) + return true + } + if (loadCount === 2 || options.where) return true + return Promise.resolve().then(() => deliver({ id: 2, rank: 2 })) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + expect(loadCount).toBe(2) + + const failedWindow = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(failedWindow).toBeInstanceOf(Promise) + await expect(failedWindow).rejects.toBe(failure) + expect(live.utils.lastSubsetError).toBe(failure) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + + const retry = live.utils.setWindow({ offset: 0, limit: 2 }) + if (retry !== true) await retry + // Recovery loads the full source once; it needs no tie-boundary probe. + expect(loadCount).toBe(4) + const recovery = acquisitions[3]! + expect(recovery.where).toBeUndefined() + expect(recovery.orderBy).toBeUndefined() + expect(recovery.limit).toBeUndefined() + expect(recovery.offset).toBeUndefined() + expect(recovery.cursor).toBeUndefined() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`retries a failed full-source window refinement`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`full-source refinement failed`) + let loadCount = 0 + let syncOps!: Parameters[`sync`]>[0] + const acquisitions: Array = [] + const releases: Array = [] + const source = createCollection({ + id: `ordered-full-source-retry-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + const { begin, write, commit, markReady } = operations + markReady() + return { + loadSubset: (options) => { + loadCount++ + acquisitions.push(options) + begin() + write({ + type: `insert`, + value: { id: loadCount, rank: loadCount }, + }) + commit(options.signal) + return loadCount === 1 + ? Promise.reject(failure) + : Promise.resolve() + }, + unloadSubset: (options) => { + releases.push(options) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(0) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + .distinct(), + ) + + try { + await live.preload() + await expect( + live.utils.setWindow({ offset: 0, limit: 2 }), + ).rejects.toBe(failure) + expect(Array.from(live.values())).toEqual([]) + + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(loadCount).toBe(2) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await flushPromises() + await flushPromises() + expect(loadCount).toBe(3) + + await live.cleanup() + expect(releases).toHaveLength(acquisitions.length) + for (const [index, acquisition] of acquisitions.entries()) { + expect(releases[index]).toBe(acquisition) + } + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`publishes a window after its failed full-source demand replays successfully`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`full-source refinement failed`) + let loadCount = 0 + let syncOps!: Parameters[`sync`]>[0] + const publications: Array> = [] + const source = createCollection({ + id: `ordered-full-source-replay-recovery-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.markReady() + return { + loadSubset: (options) => { + loadCount++ + operations.begin() + operations.write({ + type: `insert`, + value: { id: 1, rank: 1 }, + }) + if (loadCount > 1) { + operations.write({ + type: `insert`, + value: { id: 2, rank: 2 }, + }) + } + operations.commit(options.signal) + return loadCount === 1 + ? Promise.reject(failure) + : Promise.resolve() + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(0) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + .distinct(), + ) + const subscription = live.subscribeChanges(() => { + publications.push(Array.from(live.values(), ({ id }) => id)) + }) + + try { + await live.preload() + await expect( + live.utils.setWindow({ offset: 0, limit: 2 }), + ).rejects.toBe(failure) + expect(Array.from(live.values())).toEqual([]) + + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await flushPromises() + await flushPromises() + expect(loadCount).toBe(2) + expect(Array.from(live.values())).toEqual([]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 0 }) + expect(publications).toEqual([]) + + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(publications).toEqual([[1, 2]]) + } finally { + subscription.unsubscribe() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`waits for an active replay before settling a window move`, async () => { + type Row = { id: number; rank: number } + const replayGate = createDeferred() + let recovering = false + let syncOps!: Parameters[`sync`]>[0] + const source = createCollection({ + id: `ordered-window-during-replay-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.begin() + for (let id = 1; id <= 4; id++) { + operations.write({ type: `insert`, value: { id, rank: id } }) + } + operations.commit() + operations.markReady() + return { + loadSubset: (options) => { + if (!recovering) return true + operations.begin() + for (let id = 5; id <= 8; id++) { + operations.write({ type: `insert`, value: { id, rank: id } }) + } + operations.commit(options.signal) + return replayGate.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + + try { + await live.preload() + await live.utils.setWindow({ offset: 0, limit: 4 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2, 3, 4]) + + recovering = true + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await flushPromises() + + const move = live.utils.setWindow({ offset: 0, limit: 3 }) + expect(move).toBeInstanceOf(Promise) + let settled = false + void Promise.resolve(move).then( + () => { + settled = true + }, + () => { + settled = true + }, + ) + await flushPromises() + expect(settled).toBe(false) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2, 3, 4]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) + + replayGate.resolve() + await move + expect(Array.from(live.values(), ({ id }) => id)).toEqual([5, 6, 7]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + } finally { + replayGate.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`rejects a replay-blocked window move when cleanup abandons it`, async () => { + type Row = { id: number; rank: number } + const replayGate = createDeferred() + let recovering = false + let syncOps!: Parameters[`sync`]>[0] + const publications: Array> = [] + const source = createCollection({ + id: `ordered-replay-window-cleanup-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, rank: 1 } }) + operations.write({ type: `insert`, value: { id: 2, rank: 2 } }) + operations.commit() + operations.markReady() + return { + loadSubset: () => (recovering ? replayGate.promise : true), + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + const subscription = live.subscribeChanges(() => { + publications.push(Array.from(live.values(), ({ id }) => id)) + }) + + try { + await live.preload() + publications.length = 0 + recovering = true + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await flushPromises() + + const move = live.utils.setWindow({ offset: 0, limit: 3 }) + expect(move).toBeInstanceOf(Promise) + let moveError: unknown + let settled = false + void Promise.resolve(move).then( + () => { + settled = true + }, + (error) => { + moveError = error + settled = true + }, + ) + await flushPromises() + expect(settled).toBe(false) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(publications).toEqual([]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + + await live.cleanup() + await flushPromises() + expect(settled).toBe(true) + expect(moveError).toMatchObject({ name: `AbortError` }) + expect(publications).toEqual([]) + expect(live.status).toBe(`cleaned-up`) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + } finally { + subscription.unsubscribe() + replayGate.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`rejects a window move while source recovery is failed`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`ordered source replay failed`) + let recovering = false + let recoveryLoads = 0 + let syncOps!: Parameters[`sync`]>[0] + const source = createCollection({ + id: `ordered-window-after-failed-replay-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, rank: 1 } }) + operations.write({ type: `insert`, value: { id: 2, rank: 2 } }) + operations.commit() + operations.markReady() + return { + loadSubset: () => { + if (!recovering) return true + recoveryLoads++ + return Promise.reject(failure) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + + try { + await live.preload() + recovering = true + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await vi.waitFor(() => expect(live.utils.lastSubsetError).toBe(failure)) + const loadsAfterFailure = recoveryLoads + + await expect( + live.utils.setWindow({ offset: 0, limit: 3 }), + ).rejects.toBe(failure) + expect(recoveryLoads).toBe(loadsAfterFailure) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it.each( + ( + [ + { label: `Error`, value: new Error(`replay failed`) }, + { label: `undefined`, value: undefined }, + { label: `NaN`, value: Number.NaN }, + { label: `false`, value: false }, + { label: `object`, value: { reason: `replay failed` } }, + ] as const + ).flatMap(({ label, value }) => + ([`throw`, `reject`] as const).flatMap((delivery) => + ([`retained`, `new`] as const).map((demand) => ({ + delivery, + demand, + label, + value, + })), + ), + ), + )( + `scopes a normalized $delivery replay failure with $label to its $demand demand`, + async ({ delivery, demand, value }) => { + type Row = { id: number; rank: number } + const replayGate = createDeferred() + let recovering = false + let failedReplayCalls = 0 + let syncOps!: Parameters[`sync`]>[0] + const source = createCollection({ + id: `ordered-normalized-${delivery}-${String(value)}-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, rank: 1 } }) + operations.commit() + operations.markReady() + return { + loadSubset: (options) => { + if (!recovering) return true + // Choose by request shape, not callback order: a startup + // throw rolls back new demand but retains a replayed owner. + const target = + demand === `retained` + ? options.limit !== undefined + : options.limit === undefined && + options.where === undefined + if (!target || failedReplayCalls > 0) + return replayGate.promise + failedReplayCalls++ + if (delivery === `throw`) throw value + return Promise.reject(value) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + recovering = true + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await vi.waitFor(() => + expect(live.utils.lastSubsetError).toBeInstanceOf(Error), + ) + const reportedError = live.utils.lastSubsetError + expect(failedReplayCalls).toBe(1) + + const windowMove = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(windowMove).toBeInstanceOf(Promise) + replayGate.resolve() + const settlement = await Promise.resolve(windowMove).then( + () => ({ status: `fulfilled` as const }), + (error: unknown) => ({ status: `rejected` as const, error }), + ) + if (demand === `new` && delivery === `throw`) { + expect(settlement.status).toBe(`fulfilled`) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + } else { + expect(settlement.status).toBe(`rejected`) + if (settlement.status !== `rejected`) + throw new Error(`Expected replay rejection`) + expect(settlement.error).toBe(reportedError) + expect(settlement.error).toBeInstanceOf(Error) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + } + expect(live.utils.lastSubsetError).toBe(reportedError) + } finally { + replayGate.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }, + ) + + it(`ignores queued replay setup after cleanup`, async () => { + type Row = { id: number; rank: number } + let syncOps!: Parameters[`sync`]>[0] + const source = createCollection({ + id: `ordered-replay-success-after-cleanup-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, rank: 1 } }) + operations.commit() + operations.markReady() + return { loadSubset: () => true } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + const queued: Array<() => void> = [] + + try { + await live.preload() + const queueSpy = vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => queued.push(callback)) + + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + const replaySetup = queued.splice(0) + expect(replaySetup.length).toBeGreaterThan(0) + + await live.cleanup() + for (const callback of replaySetup) expect(callback).not.toThrow() + for (const callback of queued.splice(0)) expect(callback).not.toThrow() + queueSpy.mockRestore() + } finally { + vi.restoreAllMocks() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`resolves omitted window fields from the last requested window`, async () => { + type Row = { id: number; rank: number } + const source = createCollection({ + id: `ordered-partial-window-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (let id = 1; id <= 5; id++) { + write({ type: `insert`, value: { id, rank: id } }) + } + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + + try { + await live.preload() + const requestedWindow = { offset: 2 } + await live.utils.setWindow(requestedWindow) + requestedWindow.offset = 4 + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([3, 4]) + expect(live.utils.getWindow()).toEqual({ offset: 2, limit: 2 }) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`rejects a pending window move when cleanup abandons it`, async () => { + type Row = { id: number; rank: number } + const gate = createDeferred() + let loadCount = 0 + const source = createCollection({ + id: `ordered-cleanup-window-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + commit() + markReady() + return { + loadSubset: () => { + loadCount++ + return loadCount === 3 ? gate.promise : true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + const move = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(move).toBeInstanceOf(Promise) + const rejection = expect(move).rejects.toMatchObject({ + name: `AbortError`, + }) + + await live.cleanup() + await rejection + expect(live.status).toBe(`cleaned-up`) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + } finally { + gate.resolve() + await source.cleanup() + } + }) + + it(`keeps window generations distinct across immediate cleanup and restart`, async () => { + type Row = { id: number; rank: number } + const oldGate = createDeferred() + const newGate = createDeferred() + let limitFourCalls = 0 + const source = createCollection({ + id: `ordered-window-restart-generation-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + operations.begin() + for (let id = 1; id <= 6; id++) { + operations.write({ type: `insert`, value: { id, rank: id } }) + } + operations.commit() + operations.markReady() + return { + loadSubset: (options) => { + if (options.where || options.limit !== 4) return true + limitFourCalls++ + if (limitFourCalls === 1) { + options.signal?.addEventListener( + `abort`, + () => + oldGate.reject(new DOMException(`aborted`, `AbortError`)), + { once: true }, + ) + return oldGate.promise + } + return newGate.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + const abandoned = live.utils.setWindow({ offset: 2, limit: 2 }) + expect(abandoned).toBeInstanceOf(Promise) + const abandonedRejection = expect(abandoned).rejects.toMatchObject({ + name: `AbortError`, + }) + + const cleanup = live.cleanup() + const preload = live.preload() + const replacement = live.utils.setWindow({ offset: 2, limit: 2 }) + expect(replacement).toBeInstanceOf(Promise) + await Promise.all([cleanup, preload, abandonedRejection]) + + newGate.resolve() + await replacement + await live.utils.setWindow({ limit: 1 }) + expect(live.utils.getWindow()).toEqual({ offset: 2, limit: 1 }) + } finally { + oldGate.resolve() + newGate.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`keeps the last complete window when a required tie boundary rejects`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`ordered boundary failed`) + let loadCount = 0 + const source = createCollection({ + id: `ordered-boundary-rollback-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + commit(options.signal) + return true + } + if (loadCount === 2) return true + if (loadCount === 3) { + begin() + write({ type: `insert`, value: { id: 3, rank: 2 } }) + commit(options.signal) + return Promise.resolve() + } + if (loadCount === 4) return Promise.reject(failure) + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + const publications: Array> = [] + const subscription = live.subscribeChanges(() => { + publications.push(Array.from(live.values(), ({ id }) => id)) + }) + + const result = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(result).toBeInstanceOf(Promise) + await expect(result).rejects.toBe(failure) + await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(publications).toEqual([]) + subscription.unsubscribe() + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`settles a superseding window only after that window is visible`, async () => { + type Row = { id: number; rank: number } + const gate = createDeferred() + let loadCount = 0 + const source = createCollection({ + id: `ordered-superseding-window-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + write({ type: `insert`, value: { id: 2, rank: 2 } }) + commit() + markReady() + return { + loadSubset: () => { + loadCount++ + return loadCount <= 2 ? true : gate.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + const first = live.utils.setWindow({ offset: 0, limit: 3 }) + expect(first).toBeInstanceOf(Promise) + const second = live.utils.setWindow({ offset: 1, limit: 1 }) + expect(second).toBeInstanceOf(Promise) + + let secondSettled = false + void Promise.resolve(second).then(() => { + secondSettled = true + }) + await flushPromises() + expect(secondSettled).toBe(false) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + + gate.resolve() + await Promise.all([first, second]) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([2]) + } finally { + gate.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`keeps the restarted session's settled window after a failed move`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`restarted ordered page failed`) + let failPage = false + const source = createCollection({ + id: `ordered-window-restart-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + write({ type: `insert`, value: { id: 2, rank: 2 } }) + commit() + markReady() + return { + loadSubset: (options) => { + if (failPage && !options.where) throw failure + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + + await live.cleanup() + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + + failPage = true + await expect( + Promise.resolve().then(() => + live.utils.setWindow({ offset: 0, limit: 3 }), + ), + ).rejects.toBe(failure) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`does not publish a row that leaves and re-enters during a failed window move`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`offset page failed`) + let failPage = false + const source = createCollection({ + id: `ordered-window-offset-rollback-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + write({ type: `insert`, value: { id: 2, rank: 2 } }) + commit() + markReady() + return { + loadSubset: (options) => { + if (failPage && !options.where) throw failure + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + + try { + await live.preload() + const publications: Array> = [] + const subscription = live.subscribeChanges((changes) => { + publications.push(changes.map(({ type, key }) => ({ type, key }))) + }) + + failPage = true + await expect( + Promise.resolve().then(() => + live.utils.setWindow({ offset: 1, limit: 2 }), + ), + ).rejects.toBe(failure) + await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(publications).toEqual([]) + subscription.unsubscribe() + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`keeps partial ordered source work private when later refinement rejects`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`ordered boundary failed`) + let loadCount = 0 + const source = createCollection({ + id: `ordered-window-partial-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + commit(options.signal) + return true + } + if (loadCount === 2) return true + if (loadCount === 3) { + begin() + // Fulfill the requested continuation so its new boundary + // needs refinement. Also deliver a live insert before the + // cursor: it would replace the old top-one result if leaked. + write({ type: `insert`, value: { id: 2, rank: 2 } }) + write({ type: `insert`, value: { id: 0, rank: 0 } }) + commit(options.signal) + return Promise.resolve() + } + if (loadCount === 4) return Promise.reject(failure) + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + const publications: Array> = [] + const subscription = live.subscribeChanges((changes) => { + publications.push(changes.map(({ type, key }) => ({ type, key }))) + }) + + await expect( + live.utils.setWindow({ offset: 0, limit: 2 }), + ).rejects.toBe(failure) + await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + expect(publications).toEqual([]) + + await live.utils.setWindow({ offset: 0, limit: 2 }) + await flushPromises() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([0, 1]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(publications).toHaveLength(1) + subscription.unsubscribe() + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`copies a settled window instead of retaining caller-owned options`, async () => { + type Row = { id: number; rank: number } + const source = createCollection({ + id: `ordered-window-options-copy-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + write({ type: `insert`, value: { id: 2, rank: 2 } }) + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + const requestedWindow = { offset: 0, limit: 2 } + await live.utils.setWindow(requestedWindow) + requestedWindow.limit = 1 + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`concurrent live queries should each track loading state independently`, async () => { + // This tests the fix for the !wasLoadingBefore bug: + // When multiple live queries subscribe to the same source collection, + // each must independently track when loading finishes. + // Previously, only the first live query would track loading because + // wasLoadingBefore was true for subsequent queries. + + let resolveLoadSubset: () => void + const loadSubsetPromise = new Promise((resolve) => { + resolveLoadSubset = resolve + }) + + const sourceCollection = createCollection<{ id: number; value: number }>({ + id: `source-concurrent-lq`, + getKey: (item) => item.id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ markReady, begin, write, commit }) => { + begin() + write({ type: `insert`, value: { id: 1, value: 10 } }) + commit() + markReady() + + return { + loadSubset: () => loadSubsetPromise, + } + }, + }, + }) + + // Create TWO live queries that subscribe to the same source collection + const liveQuery1 = createLiveQueryCollection({ + query: (q) => q.from({ item: sourceCollection }), + startSync: true, + }) + + const liveQuery2 = createLiveQueryCollection({ + query: (q) => q.from({ item: sourceCollection }), + startSync: true, + }) + + // Wait for both subscriptions to start and trigger loadSubset + await flushPromises() + await new Promise((resolve) => setTimeout(resolve, 10)) + + // Source should be ready + expect(sourceCollection.isReady()).toBe(true) + + // Both live queries should be loading (not ready yet) + // KEY ASSERTION: Without the fix, liveQuery2 would be 'ready' here + // because it skipped tracking when wasLoadingBefore was true + expect(liveQuery1.status).toBe(`loading`) + expect(liveQuery2.status).toBe(`loading`) + + // Resolve the loadSubset promise + resolveLoadSubset!() + await flushPromises() + await new Promise((resolve) => setTimeout(resolve, 10)) + + // Now both should be ready + expect(liveQuery1.status).toBe(`ready`) + expect(liveQuery2.status).toBe(`ready`) + }) + }) + + describe(`move functionality`, () => { + it(`should support moving orderBy window past current window using move function`, async () => { + // Create a collection with more users for testing window movement + const extendedUsers = createCollection( + mockSyncCollectionOptions({ + id: `extended-users`, + getKey: (user) => user.id, + initialData: [ + { id: 1, name: `Alice`, active: true }, + { id: 2, name: `Bob`, active: true }, + { id: 3, name: `Charlie`, active: true }, + { id: 4, name: `David`, active: true }, + { id: 5, name: `Eve`, active: true }, + { id: 6, name: `Frank`, active: true }, + ], + }), + ) + + const activeUsers = createLiveQueryCollection((q) => + q + .from({ user: extendedUsers }) + .where(({ user }) => eq(user.active, true)) + .orderBy(({ user }) => user.name, `desc`) + .limit(3) + .offset(0), + ) + + await activeUsers.preload() + + // Initial result should have first 3 users (Alice, Bob, Charlie) + expect(activeUsers.size).toBe(3) + const initialResults = activeUsers.toArray + expect(initialResults.map((r) => r.name)).toEqual([ + `Frank`, + `Eve`, + `David`, + ]) + + // Move the window to show users David, Eve, Frank (offset: 3, limit: 3) + activeUsers.utils.setWindow({ offset: 3, limit: 3 }) + + // Wait for the move to take effect + await new Promise((resolve) => setTimeout(resolve, 10)) + + const moveResults = activeUsers.toArray + expect(moveResults.map((r) => r.name)).toEqual([ + `Charlie`, + `Bob`, + `Alice`, + ]) + }) + + it(`should support moving orderBy window before current window using move function`, async () => { + const extendedUsers = createCollection( + mockSyncCollectionOptions({ + id: `extended-users-before`, + getKey: (user) => user.id, + initialData: [ + { id: 1, name: `Alice`, active: true }, + { id: 2, name: `Bob`, active: true }, + { id: 3, name: `Charlie`, active: true }, + { id: 4, name: `David`, active: true }, + { id: 5, name: `Eve`, active: true }, + { id: 6, name: `Frank`, active: true }, + ], + }), + ) + + const activeUsers = createLiveQueryCollection((q) => + q + .from({ user: extendedUsers }) + .where(({ user }) => eq(user.active, true)) + .orderBy(({ user }) => user.name, `asc`) + .limit(3) + .offset(3), + ) + + await activeUsers.preload() + + // Initial result should have users David, Eve, Frank + expect(activeUsers.size).toBe(3) + const initialResults = activeUsers.toArray + expect(initialResults.map((r) => r.name)).toEqual([ + `David`, + `Eve`, + `Frank`, + ]) + + // Move the window to show users Alice, Bob, Charlie (offset: 0, limit: 3) + activeUsers.utils.setWindow({ offset: 0, limit: 3 }) + + // Wait for the move to take effect + await new Promise((resolve) => setTimeout(resolve, 10)) + + const moveResults = activeUsers.toArray + expect(moveResults.map((r) => r.name)).toEqual([ + `Alice`, + `Bob`, + `Charlie`, + ]) + }) + + it(`should support moving offset while keeping limit constant`, async () => { + const extendedUsers = createCollection( + mockSyncCollectionOptions({ + id: `extended-users-offset`, + getKey: (user) => user.id, + initialData: [ + { id: 1, name: `Alice`, active: true }, + { id: 2, name: `Bob`, active: true }, + { id: 3, name: `Charlie`, active: true }, + { id: 4, name: `David`, active: true }, + { id: 5, name: `Eve`, active: true }, + ], + }), + ) + + const activeUsers = createLiveQueryCollection((q) => + q + .from({ user: extendedUsers }) + .where(({ user }) => eq(user.active, true)) + .orderBy(({ user }) => user.name, `asc`) + .limit(2) + .offset(0), + ) + + await activeUsers.preload() // Initial result should have first 2 users (Alice, Bob) expect(activeUsers.size).toBe(2) @@ -2012,6 +3403,37 @@ describe(`createLiveQueryCollection`, () => { expect(result).toBe(true) }) + it(`does not wait for subset work that predates the window operation`, async () => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `window-with-unrelated-load`, + getKey: (user) => user.id, + initialData: sampleUsers, + autoIndex: `eager`, + }), + ) + const live = createLiveQueryCollection((q) => + q + .from({ user: source }) + .orderBy(({ user }) => user.name, `asc`) + .limit(1), + ) + let resolveUnrelated: () => void + const unrelated = new Promise((resolve) => { + resolveUnrelated = resolve + }) + + try { + await live.preload() + live._sync.trackLoadPromise(unrelated) + + expect(live.utils.setWindow({ offset: 0, limit: 2 })).toBe(true) + } finally { + resolveUnrelated!() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + it(`setWindow returns and resolves a Promise when async loading is triggered`, async () => { // This is an integration test that validates the full async flow: // 1. setWindow triggers loading more data @@ -2053,7 +3475,10 @@ describe(`createLiveQueryCollection`, () => { return true } - // Second call (triggered by setWindow) returns a promise + // The second call closes the initial ordered boundary. + if (loadSubsetCallCount === 2) return true + + // The later call triggered by setWindow returns a promise. const loadPromise = new Promise((resolve) => { // Simulate async data loading with a delay setTimeout(() => { @@ -2089,7 +3514,7 @@ describe(`createLiveQueryCollection`, () => { // Initial state: should have 2 items (values 1, 2) expect(liveQuery.size).toBe(2) expect(liveQuery.isLoadingSubset).toBe(false) - expect(loadSubsetCallCount).toBe(1) + expect(loadSubsetCallCount).toBe(2) // Move window to offset 3, which requires loading more data // This should trigger loadSubset and return a Promise @@ -2119,8 +3544,16 @@ describe(`createLiveQueryCollection`, () => { expect(promiseResolved).toBe(false) expect(liveQuery.isLoadingSubset).toBe(true) - // Now advance time to complete the loading (50ms total from loadSubset call) + // Complete the page request. The operation must remain pending while + // the loader closes the ordering boundary so equal sort values cannot + // be omitted from later window moves. await vi.advanceTimersByTimeAsync(40) + expect(loadSubsetCallCount).toBe(4) + expect(promiseResolved).toBe(false) + expect(liveQuery.isLoadingSubset).toBe(true) + + // Complete the boundary request as well. + await vi.advanceTimersByTimeAsync(50) // Wait for the promise to resolve if (result !== true) { @@ -2140,6 +3573,139 @@ describe(`createLiveQueryCollection`, () => { } }) + it(`does not settle a synchronous ordered window before loading its tie boundary`, async () => { + type Row = { id: number; rank: number } + + const remote: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 0 }, + { id: 3, rank: 1 }, + { id: 4, rank: 1 }, + ] + const delivered = new Set() + let calls = 0 + + const source = createCollection({ + id: `sync-ordered-boundary-settlement`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + calls++ + const filter = options.where + ? createFilterFunctionFromExpression(options.where) + : () => true + const candidates = remote + .filter(filter) + .filter(({ id }) => !delivered.has(id)) + .sort( + (left, right) => + left.rank - right.rank || right.id - left.id, + ) + const selected = + options.limit === undefined + ? candidates + : candidates.slice(0, options.limit) + + if (selected.length > 0) { + begin() + for (const row of selected) { + delivered.add(row.id) + write({ type: `insert`, value: row }) + } + commit(options.signal) + } + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + expect(calls).toBe(2) + expect(live.toArray.map(({ id }) => id)).toEqual([1]) + + const settled = live.utils.setWindow({ offset: 2, limit: 1 }) + if (settled !== true) await settled + + expect(calls).toBe(4) + expect(live.toArray.map(({ id }) => id)).toEqual([3]) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it.each([ + { primary: `sync`, boundary: `sync` }, + { primary: `sync`, boundary: `async` }, + { primary: `async`, boundary: `sync` }, + { primary: `async`, boundary: `async` }, + ] as const)( + `rejects initial preload when a required $boundary tie-boundary load fails after a $primary primary load`, + async ({ primary, boundary }) => { + type Row = { id: number; rank: number } + + const failure = new Error(`ordered boundary failed`) + let calls = 0 + const source = createCollection({ + id: `initial-${primary}-${boundary}-ordered-boundary-failure`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + calls++ + if (options.where) { + if (boundary === `async`) return Promise.reject(failure) + throw failure + } + + begin() + write({ type: `insert`, value: { id: 2, rank: 0 } }) + commit(options.signal) + return primary === `async` ? Promise.resolve() : true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await expect(live.preload()).rejects.toBe(failure) + expect(calls).toBe(2) + expect(live.status).toBe(`error`) + expect(live.utils.lastSubsetError).toBe(failure) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }, + ) + it(`advances offset when async loadSubset fills an initially empty window`, async () => { type Item = { id: number; value: number } const remoteData: Array = [ @@ -2162,6 +3728,10 @@ describe(`createLiveQueryCollection`, () => { markReady() return { loadSubset: (options: LoadSubsetOptions) => { + // The last loaded boundary row is already present. Respect + // the exact tie predicate instead of treating it as an + // unbounded offset request. + if (options.where) return Promise.resolve() loadOffsets.push(options.offset) return new Promise((resolve) => { setTimeout(() => { @@ -2204,7 +3774,7 @@ describe(`createLiveQueryCollection`, () => { expect(liveQuery.toArray.map((item) => item.value)).toEqual([3, 4]) }) - it(`requests new offsets when window moves across identical orderBy values`, async () => { + it(`loads an identical orderBy tie class before later window moves`, async () => { type Item = { id: number; rank: number } const remoteData: Array = [ { id: 1, rank: 1 }, @@ -2263,7 +3833,7 @@ describe(`createLiveQueryCollection`, () => { await moveFirst } await flushPromises() - expect(loadOffsets).toEqual([0, 2]) + expect(loadOffsets).toEqual([0, undefined]) expect(liveQuery.toArray.map((item) => item.id)).toEqual([3, 4]) const moveSecond = liveQuery.utils.setWindow({ offset: 4, limit: 2 }) @@ -2271,7 +3841,7 @@ describe(`createLiveQueryCollection`, () => { await moveSecond } await flushPromises() - expect(loadOffsets).toEqual([0, 2, 4]) + expect(loadOffsets).toEqual([0, undefined]) expect(liveQuery.toArray.map((item) => item.id)).toEqual([5, 6]) }) }) @@ -2365,14 +3935,12 @@ describe(`createLiveQueryCollection`, () => { commentsOptions.utils.commit() await new Promise((resolve) => setTimeout(resolve, 10)) } catch (error: any) { - expect(error.message).toContain(`already exists in the collection`) - expect(error.message).toContain(`custom getKey`) - expect(error.message).toContain(`joined queries`) - expect(error.message).toContain(`composite key`) + expect(error.message).toContain(`public key "user1"`) + expect(error.message).toContain(`not congruent`) return } - throw new Error(`Expected DuplicateKeySyncError to be thrown`) + throw new Error(`Expected duplicate public-key invariant to be thrown`) }) }) diff --git a/packages/db/tests/query/load-subset-join-dedupe.test.ts b/packages/db/tests/query/load-subset-join-dedupe.test.ts new file mode 100644 index 0000000000..54427f5920 --- /dev/null +++ b/packages/db/tests/query/load-subset-join-dedupe.test.ts @@ -0,0 +1,165 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { extractSimpleComparisons } from '../../src/query/expression-helpers.js' +import { createLiveQueryCollection, eq } from '../../src/query/index.js' +import { expectAssertionFailure } from '../expected-failure.js' +import { TraceAssertionError } from '../trace-runner.js' +import { flushPromises } from '../utils.js' +import type { + ChangeMessageOrDeleteKeyMessage, + LoadSubsetOptions, +} from '../../src/types.js' + +type Parent = { id: number; name: string } +type Child = { id: number; parentId: number; title: string } + +const parents = [ + { id: 1, name: `A` }, + { id: 2, name: `B` }, + { id: 3, name: `C` }, +] +const children = [ + { id: 10, parentId: 1, title: `A1` }, + { id: 11, parentId: 1, title: `A2` }, + { id: 20, parentId: 2, title: `B1` }, +] + +let sequence = 0 +const cleanups: Array<() => void | Promise> = [] + +function createParents() { + let begin!: () => void + let write!: (message: ChangeMessageOrDeleteKeyMessage) => void + let commit!: () => void + const collection = createCollection({ + id: `join-dedupe-parents-${sequence++}`, + getKey: (parent) => parent.id, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + begin() + for (const parent of parents) write({ type: `insert`, value: parent }) + commit() + params.markReady() + }, + }, + }) + cleanups.push(() => collection.cleanup()) + return { + collection, + insert: (parent: Parent) => { + begin() + write({ type: `insert`, value: parent }) + commit() + }, + } +} + +function createChildren() { + const loads: Array = [] + const collection = createCollection({ + id: `join-dedupe-children-${sequence++}`, + getKey: (child) => child.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const child of children) write({ type: `insert`, value: child }) + commit() + markReady() + return { + loadSubset: vi.fn((options: LoadSubsetOptions) => { + loads.push(options) + return Promise.resolve() + }), + } + }, + }, + }) + cleanups.push(() => collection.cleanup()) + return { collection, loads } +} + +function createJoinedQuery( + parentCollection: ReturnType[`collection`], + childCollection: ReturnType[`collection`], +) { + const live = createLiveQueryCollection((query) => + query + .from({ parent: parentCollection }) + .join({ child: childCollection }, ({ parent, child }) => + eq(child.parentId, parent.id), + ), + ) + cleanups.push(() => live.cleanup()) + return live +} + +describe(`loadSubset join-key deduplication`, () => { + afterEach(async () => { + for (const cleanup of cleanups.splice(0).reverse()) await cleanup() + }) + + it( + `discovered trace: a second live query reuses its loaded join predicate`, + expectAssertionFailure( + async () => { + const { collection: parentCollection } = createParents() + const { collection: childCollection, loads } = createChildren() + const firstLive = createJoinedQuery(parentCollection, childCollection) + + await firstLive.preload() + const loadCount = loads.length + expect(loadCount).toBeGreaterThan(0) + + const secondLive = createJoinedQuery(parentCollection, childCollection) + await secondLive.preload() + try { + expect(loads).toHaveLength(loadCount) + } catch (error) { + throw new TraceAssertionError(0, error) + } + }, + { + checkpoint: 0, + classify: ({ actual, expected }) => actual === 2 && expected === 1, + }, + ), + ) + + it(`requests only a newly inserted join key`, async () => { + const { collection: parentCollection, insert } = createParents() + const { collection: childCollection, loads } = createChildren() + const live = createJoinedQuery(parentCollection, childCollection) + + await live.preload() + const loadCount = loads.length + + insert({ id: 4, name: `D` }) + await flushPromises() + + const newLoads = loads.slice(loadCount) + expect(newLoads).toHaveLength(1) + + const [load] = newLoads + if (!load) { + throw new Error(`Expected one child transport load`) + } + expect(load).toEqual({ + where: expect.anything(), + orderBy: undefined, + limit: undefined, + signal: expect.any(AbortSignal), + subscription: expect.anything(), + }) + expect(load.where).toBeDefined() + expect(extractSimpleComparisons(load.where)).toEqual([ + { field: [`parentId`], operator: `in`, value: [4] }, + ]) + }) +}) diff --git a/packages/db/tests/query/load-subset-oracle.property.test.ts b/packages/db/tests/query/load-subset-oracle.property.test.ts new file mode 100644 index 0000000000..186186819a --- /dev/null +++ b/packages/db/tests/query/load-subset-oracle.property.test.ts @@ -0,0 +1,1222 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { createOptimisticAction } from '../../src/optimistic-action.js' +import { createLiveQueryCollection, eq } from '../../src/query/index.js' +import { Func, PropRef, Value } from '../../src/query/ir.js' +import { DeduplicatedLoadSubset } from '../../src/query/subset-dedupe.js' +import { createTransaction } from '../../src/transactions.js' +import { expectAssertionFailure } from '../expected-failure.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { + oracleRandomParameters, + readOracleRunConfig, +} from '../oracle-config.js' +import { TraceAssertionError } from '../trace-runner.js' +import type { + LoadSubsetOptions, + LoadSubsetRequestResult, + SyncAppliedReceipt, +} from '../../src/types.js' + +type PersistedLoadRow = { + id: string + projectId: string +} + +type OptimisticDerivedRow = { + id: string + value: string +} + +type ExactDemand = { + values: ReadonlyArray + orderField: `rank` | `score` + direction: `asc` | `desc` + nulls: `first` | `last` + stringSort: `lexical` | `locale` + offset: number + limit: number | undefined + cursorBoundary: number | undefined +} + +type ConcurrentExactScenario = { + trace: ReadonlyArray + settlementOrder: `forward` | `reverse` +} + +const rankRef = new PropRef([`rank`]) +const scoreRef = new PropRef([`score`]) + +function requirePendingAppliedReceipt( + receipt: LoadSubsetRequestResult, +): Promise { + if (receipt === true) { + throw new Error(`Expected an asynchronous subset load`) + } + return receipt +} + +const exactDemandArbitrary: fc.Arbitrary = fc + .record({ + values: fc.uniqueArray(fc.integer({ min: -3, max: 3 }), { + minLength: 1, + maxLength: 5, + }), + orderField: fc.constantFrom(`rank` as const, `score` as const), + direction: fc.constantFrom(`asc` as const, `desc` as const), + nulls: fc.constantFrom(`first` as const, `last` as const), + stringSort: fc.constantFrom(`lexical` as const, `locale` as const), + offset: fc.integer({ min: 0, max: 4 }), + limit: fc.option(fc.integer({ min: 0, max: 5 }), { nil: undefined }), + cursorBoundary: fc.option(fc.integer({ min: -3, max: 3 }), { + nil: undefined, + }), + }) + .map((demand) => ({ + ...demand, + values: [...demand.values].sort((left, right) => left - right), + })) + +function exactDemandFingerprint(demand: ExactDemand): string { + return JSON.stringify(demand) +} + +const exactDemandTraceArbitrary = fc + .uniqueArray(exactDemandArbitrary, { + minLength: 1, + maxLength: 6, + selector: exactDemandFingerprint, + }) + .chain((pool) => + fc + .array(fc.integer({ min: 0, max: pool.length - 1 }), { + minLength: 1, + maxLength: 20, + }) + .map((indices) => indices.map((index) => pool[index]!)), + ) + +const concurrentExactScenarioArbitrary: fc.Arbitrary = + exactDemandTraceArbitrary.map((trace) => ({ + trace, + settlementOrder: trace.length % 2 === 0 ? `forward` : `reverse`, + })) + +function toLoadSubsetOptions(demand: ExactDemand): LoadSubsetOptions { + const orderRef = demand.orderField === `rank` ? rankRef : scoreRef + return { + where: new Func(`in`, [scoreRef, new Value([...demand.values])]), + orderBy: [ + { + expression: orderRef, + compareOptions: { + direction: demand.direction, + nulls: demand.nulls, + stringSort: demand.stringSort, + }, + }, + ], + offset: demand.offset, + limit: demand.limit, + cursor: + demand.cursorBoundary === undefined + ? undefined + : { + whereFrom: new Func(demand.direction === `asc` ? `gt` : `lt`, [ + orderRef, + new Value(demand.cursorBoundary), + ]), + whereCurrent: new Func(`eq`, [ + orderRef, + new Value(demand.cursorBoundary), + ]), + lastKey: demand.cursorBoundary, + }, + } +} + +function assertCompletedExactDemandTrace( + trace: ReadonlyArray, +): void { + let starts = 0 + const completed = new Set() + let expectedStart: LoadSubsetOptions | undefined + const dedupe = new DeduplicatedLoadSubset({ + loadSubset: (options) => { + expect(options).toEqual(expectedStart) + starts++ + return true + }, + }) + + for (const demand of trace) { + const startsBefore = starts + expectedStart = toLoadSubsetOptions(demand) + const result = dedupe.loadSubset(expectedStart) + const fingerprint = exactDemandFingerprint(demand) + expect(result).toBe(true) + expect(starts - startsBefore).toBe(completed.has(fingerprint) ? 0 : 1) + completed.add(fingerprint) + } +} + +async function assertConcurrentExactDemandTrace({ + trace, + settlementOrder, +}: ConcurrentExactScenario): Promise { + const transports: Array<{ + deferred: ReturnType> + promise: Promise + }> = [] + const promisesByDemand = new Map>() + const dedupe = new DeduplicatedLoadSubset({ + loadSubset: () => { + const deferred = createDeferred() + const transport = { deferred, promise: deferred.promise } + transports.push(transport) + return transport.promise + }, + }) + + const callers = trace.map((demand) => { + const fingerprint = exactDemandFingerprint(demand) + const startsBefore = transports.length + const result = dedupe.loadSubset(toLoadSubsetOptions(demand)) + if (!(result instanceof Promise)) { + throw new Error(`A new in-flight demand must return a promise`) + } + const existing = promisesByDemand.get(fingerprint) + if (existing) { + expect(transports).toHaveLength(startsBefore) + expect(result).toBe(existing) + } else { + expect(transports).toHaveLength(startsBefore + 1) + promisesByDemand.set(fingerprint, result) + } + return result + }) + + const observed = Promise.allSettled(callers) + const settlement = + settlementOrder === `forward` ? transports : [...transports].reverse() + for (const transport of settlement) transport.deferred.resolve() + expect((await observed).every(({ status }) => status === `fulfilled`)).toBe( + true, + ) + + const startsAfterSettlement = transports.length + for (const demand of trace) { + expect(dedupe.loadSubset(toLoadSubsetOptions(demand))).toBe(true) + } + expect(transports).toHaveLength(startsAfterSettlement) + + dedupe.reset() + const restarted = dedupe.loadSubset(toLoadSubsetOptions(trace[0]!)) + expect(restarted).toBeInstanceOf(Promise) + expect(transports).toHaveLength(startsAfterSettlement + 1) + transports.at(-1)!.deferred.resolve() + await restarted +} + +async function expectExactWaitersShareRejection(): Promise { + const deferred = createDeferred() + void deferred.promise.catch(() => undefined) + const dedupe = new DeduplicatedLoadSubset({ + loadSubset: () => deferred.promise, + }) + const demand: ExactDemand = { + values: [1, 2], + orderField: `rank`, + direction: `asc`, + nulls: `last`, + stringSort: `lexical`, + offset: 0, + limit: 2, + cursorBoundary: undefined, + } + + const first = dedupe.loadSubset(toLoadSubsetOptions(demand)) + const second = dedupe.loadSubset(toLoadSubsetOptions(demand)) + expect(first).toBeInstanceOf(Promise) + expect(second).toBe(first) + + const outcomes = Promise.allSettled([first, second]) + deferred.reject(new Error(`transport failed`)) + expect((await outcomes).map(({ status }) => status)).toEqual([ + `rejected`, + `rejected`, + ]) + + const retry = dedupe.loadSubset(toLoadSubsetOptions(demand)) + expect(retry).toBeInstanceOf(Promise) + await expect(retry).rejects.toThrow(`transport failed`) +} + +const { multiplier, ...replay } = readOracleRunConfig() +const exactScenarioRuns = 40 * multiplier + +let collectionSequence = 0 + +async function expectPersistingLoadIsApplied( + persisting: boolean, + delivery: `synchronous` | `asynchronous` = `synchronous`, + transactionStart: `during-load` | `before-load` = `during-load`, +) { + const rows: Array = [ + { id: `r1`, projectId: `p1` }, + { id: `r2`, projectId: `p1` }, + ] + let loadCalls = 0 + const source = createCollection({ + id: `load-subset-applied-oracle-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + if (transactionStart === `before-load`) begin() + markReady() + return { + loadSubset: () => { + loadCalls += 1 + const applyRows = () => { + if (transactionStart === `during-load`) begin() + for (const row of rows) { + write({ type: `insert`, value: { ...row } }) + } + return commit() + } + if (delivery === `synchronous`) { + return applyRows() + } + return Promise.resolve().then(async () => { + const applied = applyRows() + if (applied !== true) await applied + }) + }, + } + }, + }, + }) + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + if (persisting) { + transaction.mutate(() => source.insert({ id: `other`, projectId: `p2` })) + expect(transaction.state).toBe(`persisting`) + } + const live = createLiveQueryCollection((query) => + query.from({ row: source }).where(({ row }) => eq(row.projectId, `p1`)), + ) + + try { + const ready = live.toArrayWhenReady() + if (persisting) { + let settled = false + void ready.then(() => { + settled = true + }) + await Promise.resolve() + await Promise.resolve() + + expect(settled).toBe(false) + expect(source.get(`r1`)).toBeUndefined() + expect(source.get(`r2`)).toBeUndefined() + + persistence.resolve() + await transaction.isPersisted.promise + } + + const result = await ready + expect(loadCalls).toBe(1) + try { + expect(result.map(({ id }) => id).sort()).toEqual([`r1`, `r2`]) + } catch (error) { + throw new TraceAssertionError(0, error) + } + } finally { + if (persisting) { + persistence.resolve() + await transaction.isPersisted.promise + } + await live.cleanup() + await source.cleanup() + } +} + +async function expectAppliedReceiptTiming( + gate: `free` | `parked`, + delivery: `synchronous` | `asynchronous`, +): Promise { + const source = createCollection({ + id: `load-subset-applied-timing-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: () => { + const applyRow = () => { + begin() + write({ + type: `insert`, + value: { id: `remote`, projectId: `p1` }, + }) + return commit() + } + + return delivery === `synchronous` + ? applyRow() + : Promise.resolve().then(async () => { + const applied = applyRow() + if (applied !== true) await applied + }) + }, + } + }, + }, + }) + source.startSyncImmediate() + + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + if (gate === `parked`) { + transaction.mutate(() => source.insert({ id: `local`, projectId: `p2` })) + expect(transaction.state).toBe(`persisting`) + } + + const receipt = source._sync.loadSubset({}) + + try { + if (gate === `free` && delivery === `synchronous`) { + expect(receipt).toBe(true) + expect(source.get(`remote`)).toEqual( + expect.objectContaining({ id: `remote`, projectId: `p1` }), + ) + return + } + + const pending = requirePendingAppliedReceipt(receipt) + let settled = false + let visibleWhenSettled = false + void pending.then(() => { + settled = true + visibleWhenSettled = source.get(`remote`)?.id === `remote` + }) + + expect(settled).toBe(false) + expect(source.get(`remote`)).toBeUndefined() + await Promise.resolve() + await Promise.resolve() + + if (gate === `parked`) { + expect(settled).toBe(false) + expect(source.get(`remote`)).toBeUndefined() + persistence.resolve() + await transaction.isPersisted.promise + } + + await pending + expect(settled).toBe(true) + expect(visibleWhenSettled).toBe(true) + expect(source.get(`remote`)).toEqual( + expect.objectContaining({ id: `remote`, projectId: `p1` }), + ) + } finally { + persistence.resolve() + if (gate === `parked`) { + await transaction.isPersisted.promise.catch(() => undefined) + } + await source.cleanup() + } +} + +async function expectAppliedLoadDoesNotFlushEarlierParkedSync() { + const rows: Array = [ + { id: `r1`, projectId: `p1` }, + { id: `r2`, projectId: `p1` }, + ] + let publishUnrelated!: () => void + const source = createCollection({ + id: `load-subset-applied-order-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + publishUnrelated = () => { + begin() + write({ + type: `insert`, + value: { id: `unrelated`, projectId: `p2` }, + }) + commit() + } + markReady() + return { + loadSubset: () => { + begin() + for (const row of rows) { + write({ type: `insert`, value: { ...row } }) + } + return commit() + }, + } + }, + }, + }) + source.startSyncImmediate() + + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => source.insert({ id: `other`, projectId: `p2` })) + expect(transaction.state).toBe(`persisting`) + publishUnrelated() + + const live = createLiveQueryCollection((query) => + query.from({ row: source }).where(({ row }) => eq(row.projectId, `p1`)), + ) + + try { + const ready = live.toArrayWhenReady() + let settled = false + void ready.then(() => { + settled = true + }) + await Promise.resolve() + await Promise.resolve() + + expect(settled).toBe(false) + expect(source.get(`unrelated`)).toBeUndefined() + + persistence.resolve() + await transaction.isPersisted.promise + await expect(ready).resolves.toEqual( + expect.arrayContaining([ + expect.objectContaining({ id: `r1` }), + expect.objectContaining({ id: `r2` }), + ]), + ) + } finally { + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await live.cleanup() + await source.cleanup() + } +} + +async function expectCompletionWaitsForAppliedRows() { + let publishUnrelated!: () => void + let transportCalls = 0 + const source = createCollection({ + id: `load-subset-applied-coverage-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + publishUnrelated = () => { + begin() + write({ + type: `insert`, + value: { id: `unrelated`, projectId: `p2` }, + }) + commit() + } + const deduplicated = new DeduplicatedLoadSubset({ + loadSubset: () => { + transportCalls += 1 + begin() + write({ + type: `insert`, + value: { id: `r1`, projectId: `p1` }, + }) + return commit() + }, + }) + markReady() + return { loadSubset: deduplicated.loadSubset } + }, + }, + }) + source.startSyncImmediate() + + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => source.insert({ id: `other`, projectId: `p2` })) + publishUnrelated() + + try { + const first = source._sync.loadSubset({}) + expect(first).toBeInstanceOf(Promise) + await Promise.resolve() + await Promise.resolve() + + const concurrent = source._sync.loadSubset({}) + expect(concurrent).toBe(first) + expect(transportCalls).toBe(1) + expect(source.get(`r1`)).toBeUndefined() + + persistence.resolve() + await transaction.isPersisted.promise + await Promise.all([first, concurrent]) + expect(source.get(`r1`)).toEqual( + expect.objectContaining({ id: `r1`, projectId: `p1` }), + ) + expect(source._sync.loadSubset({})).toBe(true) + } finally { + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await source.cleanup() + } +} + +async function expectConcurrentStreamCommitStaysParked() { + let publishUnrelated!: () => void + let publishSubset!: () => void + const source = createCollection({ + id: `load-subset-applied-concurrent-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + publishUnrelated = () => { + begin() + write({ + type: `insert`, + value: { id: `unrelated`, projectId: `p2` }, + }) + commit() + } + markReady() + return { + loadSubset: () => + new Promise((resolve) => { + publishSubset = () => { + begin() + write({ + type: `insert`, + value: { id: `r1`, projectId: `p1` }, + }) + const applied = commit() + if (applied === true) { + resolve() + } else { + void applied.then(resolve) + } + } + }), + } + }, + }, + }) + source.startSyncImmediate() + + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => source.insert({ id: `other`, projectId: `p2` })) + + const load = requirePendingAppliedReceipt(source._sync.loadSubset({})) + publishUnrelated() + publishSubset() + + try { + let settled = false + void load.then(() => { + settled = true + }) + await Promise.resolve() + await Promise.resolve() + + expect(settled).toBe(false) + expect(source.get(`unrelated`)).toBeUndefined() + expect(source.get(`r1`)).toBeUndefined() + persistence.resolve() + await transaction.isPersisted.promise + await load + expect(source.get(`unrelated`)).toEqual( + expect.objectContaining({ id: `unrelated`, projectId: `p2` }), + ) + expect(source.get(`r1`)).toEqual( + expect.objectContaining({ id: `r1`, projectId: `p1` }), + ) + } finally { + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await source.cleanup() + } +} + +async function expectLaterImmediateCommitSettlesAppliedSubset() { + let publishLater!: () => SyncAppliedReceipt + const source = createCollection({ + id: `load-subset-applied-priority-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { id: `initial`, projectId: `p0` }, + }) + void commit() + publishLater = () => { + begin({ immediate: true }) + write({ + type: `insert`, + value: { id: `later`, projectId: `p2` }, + }) + return commit() + } + markReady() + return { + loadSubset: () => { + begin() + write({ + type: `insert`, + value: { id: `subset`, projectId: `p1` }, + }) + return commit() + }, + } + }, + }, + }) + source.startSyncImmediate() + + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => source.insert({ id: `local`, projectId: `p3` })) + + const load = requirePendingAppliedReceipt(source._sync.loadSubset({})) + const later = publishLater() + + try { + let loadSettled = false + let subsetVisibleWhenSettled = false + void load.then(() => { + loadSettled = true + subsetVisibleWhenSettled = source.get(`subset`)?.id === `subset` + }) + await later + await load + + expect(loadSettled).toBe(true) + expect(subsetVisibleWhenSettled).toBe(true) + expect(source.get(`subset`)).toEqual( + expect.objectContaining({ id: `subset` }), + ) + expect(source.get(`later`)).toEqual( + expect.objectContaining({ id: `later` }), + ) + expect(source.get(`initial`)).toEqual( + expect.objectContaining({ id: `initial` }), + ) + + persistence.resolve() + await transaction.isPersisted.promise + await Promise.all([load, later]) + + expect(source.get(`subset`)).toEqual( + expect.objectContaining({ id: `subset` }), + ) + } finally { + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await source.cleanup() + } +} + +async function expectAbortedReceiptDoesNotSettleDemand( + abortPhase: `before-commit` | `while-parked`, +) { + let transportCalls = 0 + const committed = createDeferred() + const deduplicated = new DeduplicatedLoadSubset({ + loadSubset: async ({ signal }) => { + transportCalls += 1 + if (abortPhase === `before-commit`) { + // Give cancellation a chance to revoke this request before its + // request-scoped rows enter the collection transaction. + await Promise.resolve() + if (signal?.aborted) { + return + } + } + begin() + write({ + type: `insert`, + value: { id: `row`, projectId: `p1` }, + }) + const applied = commit(signal) + committed.resolve() + if (applied !== true) await applied + }, + }) + let begin!: () => void + let write!: (message: { type: `insert`; value: PersistedLoadRow }) => void + let commit!: (signal?: AbortSignal) => SyncAppliedReceipt + const source = createCollection({ + id: `load-subset-applied-abort-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { loadSubset: deduplicated.loadSubset } + }, + }, + }) + source.startSyncImmediate() + + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => source.insert({ id: `local`, projectId: `p2` })) + const controller = new AbortController() + const first = requirePendingAppliedReceipt( + source._sync.loadSubset({ signal: controller.signal }), + ) + if (abortPhase === `while-parked`) { + await committed.promise + } + controller.abort() + + try { + persistence.resolve() + await transaction.isPersisted.promise + if (abortPhase === `while-parked`) { + await expect(first).rejects.toMatchObject({ name: `AbortError` }) + } else { + await first + } + expect(transportCalls).toBe(1) + expect(source.get(`row`)).toBeUndefined() + + const retry = source._sync.loadSubset({}) + if (retry !== true) await retry + expect(transportCalls).toBe(2) + expect(source.get(`row`)).toEqual(expect.objectContaining({ id: `row` })) + } finally { + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await source.cleanup() + } +} + +async function expectAbortDuringPublicationDoesNotCancelReceipt() { + const controller = new AbortController() + const source = createCollection({ + id: `load-subset-applied-publication-abort-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: ({ signal }) => { + begin() + write({ + type: `insert`, + value: { id: `row`, projectId: `p1` }, + }) + return commit(signal) + }, + } + }, + }, + }) + source.startSyncImmediate() + + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => source.insert({ id: `local`, projectId: `pending` })) + const subscription = source.subscribeChanges((changes) => { + if (changes.some((change) => change.key === `row`)) { + controller.abort() + } + }) + const load = requirePendingAppliedReceipt( + source._sync.loadSubset({ signal: controller.signal }), + ) + + try { + persistence.resolve() + await transaction.isPersisted.promise + await expect(load).resolves.toBeUndefined() + expect(controller.signal.aborted).toBe(true) + expect(source.get(`row`)).toEqual(expect.objectContaining({ id: `row` })) + } finally { + subscription.unsubscribe() + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await source.cleanup() + } +} + +async function expectCanceledReceiptReleasesOnlyItsSuppression() { + let begin!: () => void + let write!: (message: { type: `update`; value: PersistedLoadRow }) => void + let commit!: (signal?: AbortSignal) => SyncAppliedReceipt + const source = createCollection({ + id: `load-subset-cancel-suppression-${collectionSequence++}`, + getKey: (row) => row.id, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + begin() + write({ type: `update`, value: { id: `first`, projectId: `old` } }) + write({ type: `update`, value: { id: `second`, projectId: `old` } }) + commit() + params.markReady() + }, + }, + }) + await source.preload() + await Promise.resolve() + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => source.insert({ id: `local`, projectId: `pending` })) + expect(transaction.state).toBe(`persisting`) + try { + begin() + write({ type: `update`, value: { id: `first`, projectId: `new` } }) + const canceled = commit() + const canceledTransaction = source._state.pendingSyncedTransactions.at(-1)! + expect(source._state.pendingSyncedTransactions).toHaveLength(1) + begin() + write({ type: `update`, value: { id: `second`, projectId: `new` } }) + expect(source._state.pendingSyncedTransactions).toHaveLength(2) + + source._state.capturePreSyncVisibleState() + expect(source._state.recentlySyncedKeys).toEqual( + new Set([`first`, `second`]), + ) + + source._state.cancelPendingSyncedTransaction(canceledTransaction) + expect(source._state.pendingSyncedTransactions).toHaveLength(1) + expect(source._state.recentlySyncedKeys).toEqual(new Set([`second`])) + expect(source._state.preSyncVisibleState.has(`first`)).toBe(false) + expect(source._state.preSyncVisibleState.has(`second`)).toBe(true) + if (canceled !== true) { + await expect(canceled).rejects.toMatchObject({ name: `AbortError` }) + } + } finally { + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await source.cleanup() + } +} + +async function expectCleanupRejectsDemandOnce() { + let receipt!: Promise + let transportCalls = 0 + const deduplicated = new DeduplicatedLoadSubset({ + loadSubset: () => { + transportCalls += 1 + begin() + write({ + type: `insert`, + value: { id: `row`, projectId: `p1` }, + }) + const applied = commit() + if (transportCalls === 1) { + if (applied === true) { + throw new Error(`Expected the subset transaction to remain parked`) + } + receipt = applied + } + return applied + }, + }) + let begin!: () => void + let write!: (message: { type: `insert`; value: PersistedLoadRow }) => void + let commit!: () => SyncAppliedReceipt + const source = createCollection({ + id: `load-subset-applied-cleanup-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: deduplicated.loadSubset, + cleanup: () => deduplicated.reset(), + } + }, + }, + }) + source.startSyncImmediate() + + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => source.insert({ id: `local`, projectId: `p2` })) + const load = requirePendingAppliedReceipt(source._sync.loadSubset({})) + let settlements = 0 + void receipt.then( + () => { + settlements += 1 + }, + () => { + settlements += 1 + }, + ) + + await source.cleanup() + await expect(load).rejects.toMatchObject({ name: `AbortError` }) + await expect(receipt).rejects.toMatchObject({ name: `AbortError` }) + expect(settlements).toBe(1) + + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await Promise.resolve() + expect(settlements).toBe(1) + + // Restarting installs fresh sync controls. Reacquisition must both perform + // transport work and publish its rows; stale callbacks cannot prove either. + source.startSyncImmediate() + const retry = source._sync.loadSubset({}) + if (retry !== true) await retry + expect(transportCalls).toBe(2) + expect(source.get(`row`)).toEqual(expect.objectContaining({ id: `row` })) + + await source.cleanup() +} + +async function expectDerivedSyncDuringOptimisticMutation(): Promise { + let begin!: () => void + let write!: (message: { type: `insert`; value: OptimisticDerivedRow }) => void + let commit!: () => void + const source = createCollection({ + id: `optimistic-derived-source-${collectionSequence++}`, + getKey: (row) => row.id, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + }, + }, + }) + const derived = createLiveQueryCollection({ + query: (query) => + query + .from({ row: source }) + .select(({ row }) => ({ id: row.id, value: row.value })), + getKey: (row) => row.id, + startSync: true, + }) + const persistence = createDeferred() + // Query collections currently expose read-side virtual properties in their + // insert input type even though the runtime accepts the plain selected row. + const insertDerived = derived.insert.bind(derived) as unknown as ( + row: OptimisticDerivedRow, + ) => ReturnType + const insertOptimistically = createOptimisticAction({ + onMutate: insertDerived, + mutationFn: () => persistence.promise, + }) + + await derived.preload() + const transaction = insertOptimistically({ + id: `optimistic`, + value: `optimistic`, + }) + try { + begin() + write({ type: `insert`, value: { id: `synced`, value: `synced` } }) + commit() + + try { + expect([...derived.keys()].sort()).toEqual([`optimistic`, `synced`]) + } catch (error) { + throw new TraceAssertionError(0, error) + } + } finally { + persistence.resolve() + await transaction.isPersisted.promise + await derived.cleanup() + await source.cleanup() + } +} + +describe(`exact loadSubset demand oracle`, () => { + it(`uses SQL unknown for nullish comparisons in the independent model`, () => { + const missing = new PropRef([`missing`]) + + expect( + evaluateReferenceExpression( + new Func(`lte`, [missing, new Value(null)]), + {}, + ), + ).toBeNull() + expect( + evaluateReferenceExpression(new Func(`lt`, [missing, new Value(0)]), {}), + ).toBeNull() + }) + + it(`generates repeated, cursor, empty, and unbounded exact demands`, () => { + const traces = fc.sample(exactDemandTraceArbitrary, { + seed: 1656, + numRuns: 200, + }) + const demands = traces.flat() + + expect( + traces.some( + (trace) => + new Set(trace.map(exactDemandFingerprint)).size < trace.length, + ), + ).toBe(true) + expect( + demands.some(({ cursorBoundary }) => cursorBoundary !== undefined), + ).toBe(true) + expect(demands.some(({ limit }) => limit === 0)).toBe(true) + expect(demands.some(({ limit }) => limit === undefined)).toBe(true) + expect(new Set(demands.map(({ offset }) => offset)).size).toBeGreaterThan(1) + }) + + fcTest.prop([exactDemandTraceArbitrary], { + numRuns: exactScenarioRuns, + seed: 1657, + })( + `starts each completed exact demand once for a fixed seed`, + assertCompletedExactDemandTrace, + ) + + fcTest.prop( + [exactDemandTraceArbitrary], + oracleRandomParameters( + exactScenarioRuns, + replay, + `load-subset.exact-completion`, + ), + )( + `starts each completed exact demand once for a random or replayed seed`, + assertCompletedExactDemandTrace, + ) + + fcTest.prop([concurrentExactScenarioArbitrary], { + numRuns: exactScenarioRuns, + seed: 1661, + })( + `shares only identical in-flight demands for a fixed seed`, + assertConcurrentExactDemandTrace, + ) + + fcTest.prop( + [concurrentExactScenarioArbitrary], + oracleRandomParameters( + exactScenarioRuns, + replay, + `load-subset.exact-inflight`, + ), + )( + `shares only identical in-flight demands for a random or replayed seed`, + assertConcurrentExactDemandTrace, + ) + + it(`reports one rejection to every exact waiter and then retries`, async () => { + await expectExactWaitersShareRejection() + }) +}) + +describe(`loadSubset application and cancellation`, () => { + it(`applies loaded rows when no mutation is persisting`, async () => { + await expectPersistingLoadIsApplied(false) + }) + + it(`applies loaded rows before resolving readiness behind a persisting mutation`, async () => { + await expectPersistingLoadIsApplied(true) + }) + + it(`applies asynchronously delivered rows before resolving readiness`, async () => { + await expectPersistingLoadIsApplied(true, `asynchronous`) + }) + + it(`applies a transaction opened before its subset demand`, async () => { + await expectPersistingLoadIsApplied(true, `synchronous`, `before-load`) + }) + + it.each([ + [`free`, `synchronous`], + [`free`, `asynchronous`], + [`parked`, `synchronous`], + [`parked`, `asynchronous`], + ] as const)( + `preserves applied-receipt timing with a %s gate and %s delivery`, + expectAppliedReceiptTiming, + ) + + it(`does not flush earlier parked sync work to apply a subset load`, async () => { + await expectAppliedLoadDoesNotFlushEarlierParkedSync() + }) + + it(`settles a demand only after its rows apply`, async () => { + await expectCompletionWaitsForAppliedRows() + }) + + it(`keeps an unrelated stream commit parked during a subset acquisition`, async () => { + await expectConcurrentStreamCommitStaysParked() + }) + + it(`settles a subset receipt after a later immediate commit applies it`, async () => { + await expectLaterImmediateCommitSettlesAppliedSubset() + }) + + it.each([`before-commit`, `while-parked`] as const)( + `does not settle a demand when its parked receipt is aborted %s`, + expectAbortedReceiptDoesNotSettleDemand, + ) + + it(`ignores an abort raised after application starts publishing`, async () => { + await expectAbortDuringPublicationDoesNotCancelReceipt() + }) + + it(`releases only a canceled receipt's event suppression`, async () => { + await expectCanceledReceiptReleasesOnlyItsSuppression() + }) + + it(`rejects an abandoned demand once`, async () => { + await expectCleanupRejectsDemandOnce() + }) + + it(`publishes synced source rows while a derived mutation persists`, async () => { + await expectAssertionFailure(expectDerivedSyncDuringOptimisticMutation, { + checkpoint: 0, + classify: ({ actual, expected }) => + Array.isArray(actual) && + actual.join(`,`) === `optimistic` && + Array.isArray(expected) && + expected.join(`,`) === `optimistic,synced`, + })() + }) +}) diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts new file mode 100644 index 0000000000..1637094b3e --- /dev/null +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -0,0 +1,878 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { + createLiveQueryCollection, + eq, + toArray, +} from '../../src/query/index.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { flushPromises } from '../utils.js' +import type { + ChangeMessageOrDeleteKeyMessage, + LoadSubsetOptions, + SyncConfig, +} from '../../src/types.js' + +type Row = { id: string; version: number } +type ObservedRow = { sourceId: string; rowKey: string; version: number } + +describe(`loadSubset replay refinement`, () => { + // A direct subscriber survives source cleanup. A dependent live query enters + // a terminal error instead; restarting only its source must not revive it. + it.each( + ([`direct`, `live`] as const).flatMap((consumer) => + ([`resolve`, `reject`] as const).map((outcome) => ({ + consumer, + outcome, + })), + ), + )( + `separates direct restart from fatal live source cleanup: %j`, + async ({ consumer, outcome }) => { + let operations!: Parameters[`sync`]>[0] + let loads = 0 + const pending = createDeferred() + void pending.promise.catch(() => undefined) + const initial = [{ id: `row`, version: 1 }] + const replacement = [{ id: `row`, version: 2 }] + const source = createCollection({ + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (next) => { + operations = next + next.markReady() + return { + loadSubset: () => { + if (++loads > 1) return pending.promise + next.begin() + next.write({ type: `insert`, value: initial[0]! }) + next.commit() + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = + consumer === `live` + ? createLiveQueryCollection((q) => q.from({ row: source })) + : undefined + const visible = new Map() + const rows = (values: ReadonlyArray) => + values.map(({ id, version }) => ({ id, version })) + const readEvents = () => rows([...visible.values()]) + const read = () => (live ? rows(live.toArray) : readEvents()) + const publications: Array> = [] + const subscription = (live ?? source).subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(String(change.key)) + else visible.set(String(change.key), { ...change.value }) + } + publications.push(readEvents()) + }, + { includeInitialState: consumer === `live` }, + ) + + try { + if (live) await live.preload() + else subscription.requestSnapshot({}) + await flushPromises() + expect(loads).toBe(1) + expect(read()).toEqual(initial) + expect(readEvents()).toEqual(initial) + publications.length = 0 + + await source.cleanup() + if (live) expect(live.status).toBe(`error`) + source.startSyncImmediate() + await flushPromises() + expect(loads).toBe(2) + expect(read()).toEqual(initial) + // Cleanup may change row metadata without changing the public data. + for (const publication of publications) + expect(publication).toEqual(initial) + + operations.begin() + operations.write({ type: `insert`, value: replacement[0]! }) + await operations.commit() + await flushPromises() + expect(rows(source.toArray)).toEqual(replacement) + expect(read()).toEqual(initial) + expect(readEvents()).toEqual(initial) + for (const publication of publications) + expect(publication).toEqual(initial) + publications.length = 0 + + if (outcome === `resolve`) pending.resolve() + else pending.reject(new Error(`restart failed`)) + await flushPromises() + const publishes = consumer === `direct` && outcome === `resolve` + const expected = publishes ? replacement : initial + expect(read()).toEqual(expected) + expect(readEvents()).toEqual(expected) + expect(publications).toEqual(publishes ? [replacement] : []) + if (live) expect(live.status).toBe(`error`) + } finally { + pending.resolve() + subscription.unsubscribe() + await live?.cleanup() + await source.cleanup() + } + }, + ) + + it(`publishes a successful sibling after a settled failed include route retires`, async () => { + type Parent = { id: string; left: number | null; right: number } + type Child = { id: number; version: number } + let parentSync!: Parameters[`sync`]>[0] + let childSync!: Parameters[`sync`]>[0] + const failed = createDeferred() + const successful = createDeferred() + const loads: Array<{ options: LoadSubsetOptions; ids: Array }> = [] + const unloads: Array = [] + const parents = createCollection({ + id: `settled-peer-parent`, + getKey: ({ id }) => id, + sync: { + sync: (operations) => { + parentSync = operations + operations.begin() + operations.write({ + type: `insert`, + value: { id: `parent`, left: 1, right: 2 }, + }) + operations.commit() + operations.markReady() + }, + }, + }) + const children = createCollection({ + id: `settled-peer-children`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: (operations) => { + childSync = operations + operations.markReady() + return { + loadSubset: (options) => { + const rows = [1, 2] + .map((id) => ({ id, version: loads.length < 2 ? 1 : 2 })) + .filter( + (row) => + !options.where || + evaluateReferenceExpression(options.where, row) === true, + ) + loads.push({ options, ids: rows.map(({ id }) => id) }) + operations.begin() + for (const value of rows) + operations.write({ type: `insert`, value }) + operations.commit() + if (loads.length <= 2) return true + return rows.some(({ id }) => id === 1) + ? failed.promise + : successful.promise + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + left: toArray( + q + .from({ leftChild: children }) + .where(({ leftChild }) => eq(leftChild.id, parent.left)), + ), + right: toArray( + q + .from({ rightChild: children }) + .where(({ rightChild }) => eq(rightChild.id, parent.right)), + ), + })), + ) + const read = () => + live.toArray.map(({ id, left, right }) => ({ + id, + left: left.map(({ id: key, version }) => ({ id: key, version })), + right: right.map(({ id: key, version }) => ({ id: key, version })), + })) + const publications: Array> = [] + const subscription = live.subscribeChanges( + () => publications.push(read()), + { includeInitialState: false }, + ) + try { + await live.preload() + expect(loads.map(({ ids }) => ids)).toEqual([[1], [2]]) + const initial = [ + { + id: `parent`, + left: [{ id: 1, version: 1 }], + right: [{ id: 2, version: 1 }], + }, + ] + expect(read()).toEqual(initial) + publications.length = 0 + childSync.begin() + childSync.truncate() + childSync.commit() + await flushPromises() + expect(loads.slice(2).map(({ ids }) => ids)).toEqual([[1], [2]]) + failed.reject(new Error(`left replay failed`)) + successful.resolve() + await flushPromises() + expect(read()).toEqual(initial) + expect(publications).toEqual([]) + parentSync.begin() + parentSync.write({ + type: `update`, + value: { id: `parent`, left: null, right: 2 }, + }) + parentSync.commit() + await flushPromises() + expect(read()).toEqual([ + { id: `parent`, left: [], right: [{ id: 2, version: 2 }] }, + ]) + expect(publications).toEqual([ + [{ id: `parent`, left: [], right: [{ id: 2, version: 2 }] }], + ]) + expect(loads).toHaveLength(4) + expect(unloads).toContain(loads[2]!.options) + expect(loads[2]!.options.signal?.aborted).toBe(true) + expect(loads[3]!.options.signal?.aborted).toBe(false) + childSync.begin() + childSync.write({ type: `update`, value: { id: 2, version: 3 } }) + childSync.commit() + await flushPromises() + expect(read()).toEqual([ + { id: `parent`, left: [], right: [{ id: 2, version: 3 }] }, + ]) + expect(publications).toHaveLength(2) + } finally { + failed.resolve() + successful.resolve() + subscription.unsubscribe() + await live.cleanup() + await Promise.all([parents.cleanup(), children.cleanup()]) + } + expect(unloads).toHaveLength(loads.length) + for (const { options } of loads) { + expect(unloads.filter((unloaded) => unloaded === options)).toHaveLength(1) + } + }) + + function createHarness( + sourceId: string, + initialRows: ReadonlyArray = [{ id: `row`, version: 1 }], + ) { + let begin!: () => void + let write!: (message: ChangeMessageOrDeleteKeyMessage) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const pending: Array<{ + options: LoadSubsetOptions + deferred: ReturnType> + }> = [] + const batches: Array< + Array<{ + type: `insert` | `update` | `delete` + row: { sourceId: string; rowKey: string; version: number } + previousVersion?: number + }> + > = [] + const source = createCollection({ + id: sourceId, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: (options) => { + loadCount++ + if (loadCount === 1) { + begin() + for (const value of initialRows) { + write({ type: `insert`, value }) + } + commit() + return true + } + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const downstream = createLiveQueryCollection({ + id: `${sourceId}-downstream`, + query: (q) => + q.from({ row: source }).select(({ row }) => ({ + id: row.id, + version: row.version, + })), + startSync: true, + }) + const callbackReads: Array> = [] + const subscription = downstream.subscribeChanges( + (changes) => { + const batch = changes.map((change) => ({ + type: change.type, + row: { + sourceId, + rowKey: String(change.key), + version: change.value.version, + }, + ...(change.previousValue === undefined + ? {} + : { previousVersion: change.previousValue.version }), + })) + if (batch.length > 0) { + batches.push(batch) + callbackReads.push( + downstream.toArray.map(({ id, version }) => ({ + sourceId, + rowKey: id, + version, + })), + ) + } + }, + { includeInitialState: true }, + ) + + const replaceCore = (version: number) => { + begin() + write({ type: `insert`, value: { id: `row`, version } }) + commit() + } + const updateCore = (previousVersion: number, version: number) => { + begin() + write({ + type: `update`, + value: { id: `row`, version }, + previousValue: { id: `row`, version: previousVersion }, + }) + commit() + } + const applyCore = ( + changes: ReadonlyArray>, + ) => { + begin() + for (const change of changes) write(change) + commit() + } + const startReplay = async () => { + begin() + truncate() + commit() + await flushPromises() + } + const coreRows = () => + source.toArray.map(({ id, version }) => ({ + sourceId, + rowKey: id, + version, + })) + const visibleRows = () => + downstream.toArray.map(({ id, version }) => ({ + sourceId, + rowKey: id, + version, + })) + + return { + source, + downstream, + subscription, + pending, + batches, + callbackReads, + replaceCore, + updateCore, + applyCore, + startReplay, + coreRows, + visibleRows, + } + } + + it(`retains the last complete publication when replay fails after writing`, async () => { + const sourceId = `replay-refinement-failure` + const row = (version: number) => ({ + sourceId, + rowKey: `row`, + version, + }) + const harness = createHarness(sourceId) + + try { + await harness.downstream.preload() + await harness.startReplay() + + harness.replaceCore(2) + harness.pending[0]?.deferred.reject(new Error(`replay failed`)) + await flushPromises() + + expect(harness.coreRows()).toEqual([row(2)]) + expect(harness.visibleRows()).toEqual([row(1)]) + expect(harness.batches).toEqual([[{ type: `insert`, row: row(1) }]]) + expect(harness.callbackReads).toEqual([[row(1)]]) + } finally { + harness.subscription.unsubscribe() + await Promise.all([ + harness.downstream.cleanup(), + harness.source.cleanup(), + ]) + } + }) + + it(`publishes a replay replacement before its source reports ready`, async () => { + const replay = createDeferred() + let loadCount = 0 + let begin!: () => void + let write!: (message: ChangeMessageOrDeleteKeyMessage) => void + let commit!: () => void + let truncate!: () => void + let sourceSubscription: LoadSubsetOptions[`subscription`] + const source = createCollection({ + id: `replay-ready-publication-source`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + sourceSubscription = options.subscription + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: `row`, version: 1 } }) + commit() + return true + } + return replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q.from({ row: source }).select(({ row }) => ({ + id: row.id, + version: row.version, + })), + ) + const readVersions = () => live.toArray.map(({ version }) => version) + const readyReads: Array> = [] + + try { + await live.preload() + expect(readVersions()).toEqual([1]) + sourceSubscription!.on(`status:ready`, () => { + readyReads.push(readVersions()) + }) + + begin() + truncate() + commit() + await flushPromises() + begin() + write({ type: `insert`, value: { id: `row`, version: 2 } }) + commit() + + replay.resolve() + await flushPromises() + + expect(readyReads).toEqual([[2]]) + expect(readVersions()).toEqual([2]) + } finally { + replay.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`keeps a failed replay private until a later authoritative replay`, async () => { + const sourceId = `replay-refinement-failure-liveness` + const row = (version: number) => ({ sourceId, rowKey: `row`, version }) + const harness = createHarness(sourceId) + + try { + await harness.downstream.preload() + expect(harness.visibleRows().map(({ version }) => version)).toEqual([1]) + + await harness.startReplay() + harness.replaceCore(2) + harness.pending[0]!.deferred.reject(new Error(`replay failed`)) + await flushPromises() + + expect(harness.visibleRows()).toEqual([row(1)]) + expect(harness.downstream.status).toBe(`ready`) + expect(harness.batches).toEqual([[{ type: `insert`, row: row(1) }]]) + + harness.updateCore(2, 3) + await flushPromises() + + expect(harness.visibleRows()).toEqual([row(1)]) + expect(harness.batches).toEqual([[{ type: `insert`, row: row(1) }]]) + expect(harness.callbackReads).toEqual([[row(1)]]) + + await harness.startReplay() + harness.replaceCore(4) + harness.pending[1]!.deferred.resolve() + await flushPromises() + + expect(harness.visibleRows()).toEqual([row(4)]) + expect(harness.batches).toEqual([ + [{ type: `insert`, row: row(1) }], + [{ type: `update`, row: row(4), previousVersion: 1 }], + ]) + expect(harness.callbackReads).toEqual([[row(1)], [row(4)]]) + } finally { + for (const replay of harness.pending) replay.deferred.resolve() + harness.subscription.unsubscribe() + await Promise.all([ + harness.downstream.cleanup(), + harness.source.cleanup(), + ]) + } + }) + + it(`replaces a multi-row failed replay only with later authoritative state`, async () => { + const sourceId = `replay-refinement-multi-row-failure` + const observed = (id: string, version: number) => ({ + sourceId, + rowKey: id, + version, + }) + const harness = createHarness(sourceId, [ + { id: `a`, version: 1 }, + { id: `b`, version: 1 }, + { id: `c`, version: 1 }, + ]) + const sortedVisible = () => + harness + .visibleRows() + .sort((left, right) => left.rowKey.localeCompare(right.rowKey)) + const sortedCore = () => + harness + .coreRows() + .sort((left, right) => left.rowKey.localeCompare(right.rowKey)) + + try { + await harness.downstream.preload() + expect(sortedVisible()).toEqual([ + observed(`a`, 1), + observed(`b`, 1), + observed(`c`, 1), + ]) + const publishedBatches = harness.batches.length + + await harness.startReplay() + harness.applyCore([ + { type: `insert`, value: { id: `a`, version: 2 } }, + { type: `insert`, value: { id: `d`, version: 1 } }, + ]) + harness.pending[0]!.deferred.reject(new Error(`partial replay failed`)) + await flushPromises() + + harness.applyCore([ + { + type: `update`, + value: { id: `a`, version: 3 }, + previousValue: { id: `a`, version: 2 }, + }, + { type: `delete`, key: `d` }, + { type: `insert`, value: { id: `e`, version: 1 } }, + ]) + await flushPromises() + + expect(sortedCore()).toEqual([observed(`a`, 3), observed(`e`, 1)]) + expect(sortedVisible()).toEqual([ + observed(`a`, 1), + observed(`b`, 1), + observed(`c`, 1), + ]) + expect(harness.batches).toHaveLength(publishedBatches) + + await harness.startReplay() + harness.applyCore([ + { type: `insert`, value: { id: `a`, version: 4 } }, + { type: `insert`, value: { id: `b`, version: 1 } }, + { type: `insert`, value: { id: `e`, version: 2 } }, + ]) + harness.pending[1]!.deferred.resolve() + await flushPromises() + + expect(sortedVisible()).toEqual([ + observed(`a`, 4), + observed(`b`, 1), + observed(`e`, 2), + ]) + expect(harness.batches).toHaveLength(publishedBatches + 1) + expect(harness.batches.at(-1)).toEqual([ + { + type: `update`, + row: observed(`a`, 4), + previousVersion: 1, + }, + { type: `delete`, row: observed(`c`, 1) }, + { type: `insert`, row: observed(`e`, 2) }, + ]) + expect( + harness.callbackReads + .at(-1) + ?.sort((left, right) => left.rowKey.localeCompare(right.rowKey)), + ).toEqual([observed(`a`, 4), observed(`b`, 1), observed(`e`, 2)]) + } finally { + for (const replay of harness.pending) replay.deferred.resolve() + harness.subscription.unsubscribe() + await Promise.all([ + harness.downstream.cleanup(), + harness.source.cleanup(), + ]) + } + }) + + it(`waits for every overlapping replay before publishing the newest success`, async () => { + const sourceId = `replay-refinement-overlap` + const row = (version: number) => ({ + sourceId, + rowKey: `row`, + version, + }) + const harness = createHarness(sourceId) + + try { + await harness.downstream.preload() + await harness.startReplay() + await harness.startReplay() + + expect(harness.pending[0]?.options.signal?.aborted).toBe(true) + harness.replaceCore(3) + harness.pending[1]?.deferred.resolve() + await flushPromises() + + expect(harness.visibleRows()).toEqual([row(1)]) + expect(harness.batches).toEqual([[{ type: `insert`, row: row(1) }]]) + expect(harness.callbackReads).toEqual([[row(1)]]) + + harness.pending[0]?.deferred.reject( + new DOMException(`obsolete`, `AbortError`), + ) + await flushPromises() + + expect(harness.coreRows()).toEqual([row(3)]) + expect(harness.visibleRows()).toEqual([row(3)]) + expect(harness.batches).toEqual([ + [{ type: `insert`, row: row(1) }], + [{ type: `update`, row: row(3), previousVersion: 1 }], + ]) + expect(harness.callbackReads).toEqual([[row(1)], [row(3)]]) + } finally { + for (const replay of harness.pending) replay.deferred.resolve() + harness.subscription.unsubscribe() + await Promise.all([ + harness.downstream.cleanup(), + harness.source.cleanup(), + ]) + } + }) + + it(`waits for every recovering source before publishing a joined replacement`, async () => { + type Primary = { id: string; joinKey: string; version: number } + type Secondary = { id: string; joinKey: string; version: number } + + const createSource = (id: string) => { + let begin!: () => void + let write!: (message: { type: `insert`; value: T }) => void + let commit!: () => true | Promise + let truncate!: () => void + const pending: Array>> = [] + const collection = createCollection({ + id, + getKey: ({ id: key }) => key, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + const request = createDeferred() + pending.push(request) + return request.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + return { + collection, + pending, + async apply(row: T) { + begin() + write({ type: `insert`, value: row }) + const receipt = commit() + if (receipt !== true) await receipt + }, + replay() { + begin() + truncate() + return commit() + }, + } + } + + const primary = createSource(`joined-replay-primary`) + const secondary = createSource(`joined-replay-secondary`) + const live = createLiveQueryCollection((q) => + q + .from({ primary: primary.collection }) + .innerJoin( + { secondary: secondary.collection }, + ({ primary: left, secondary: right }) => + eq(left.joinKey, right.joinKey), + ) + .orderBy(({ primary: row }) => row.version) + .limit(1) + .select(({ primary: left, secondary: right }) => ({ + id: left.id, + secondaryId: right.id, + primaryVersion: left.version, + secondaryVersion: right.version, + })), + ) + const read = () => + live.toArray.map( + ({ id, secondaryId, primaryVersion, secondaryVersion }) => ({ + id, + secondaryId, + primaryVersion, + secondaryVersion, + }), + ) + const publications: Array> = [] + let subscription: ReturnType | undefined + let primaryReplay: true | Promise = true + let secondaryReplay: true | Promise = true + + try { + const preload = live.preload() + await flushPromises() + expect(primary.pending).toHaveLength(1) + await primary.apply({ id: `p`, joinKey: `shared`, version: 1 }) + primary.pending[0]!.resolve() + await flushPromises() + expect(secondary.pending).toHaveLength(1) + await secondary.apply({ id: `s`, joinKey: `shared`, version: 1 }) + secondary.pending[0]!.resolve() + await flushPromises() + for (const request of primary.pending.slice(1)) request.resolve() + await preload + expect(read()).toEqual([ + { + id: `p`, + secondaryId: `s`, + primaryVersion: 1, + secondaryVersion: 1, + }, + ]) + + subscription = live.subscribeChanges(() => publications.push(read()), { + includeInitialState: false, + }) + const initialPrimaryLoads = primary.pending.length + const initialSecondaryLoads = secondary.pending.length + primaryReplay = primary.replay() + secondaryReplay = secondary.replay() + await flushPromises() + expect(primary.pending.length).toBeGreaterThan(initialPrimaryLoads) + expect(secondary.pending.length).toBeGreaterThan(initialSecondaryLoads) + + await primary.apply({ id: `p`, joinKey: `shared`, version: 2 }) + await secondary.apply({ id: `s`, joinKey: `shared`, version: 2 }) + for (const request of primary.pending.slice(initialPrimaryLoads)) { + request.resolve() + } + await flushPromises() + + expect(read()).toEqual([ + { + id: `p`, + secondaryId: `s`, + primaryVersion: 1, + secondaryVersion: 1, + }, + ]) + expect(publications).toEqual([]) + + for (const request of secondary.pending.slice(initialSecondaryLoads)) { + request.resolve() + } + await Promise.all([primaryReplay, secondaryReplay]) + await flushPromises() + + expect(read()).toEqual([ + { + id: `p`, + secondaryId: `s`, + primaryVersion: 2, + secondaryVersion: 2, + }, + ]) + expect(publications).toEqual([ + [ + { + id: `p`, + secondaryId: `s`, + primaryVersion: 2, + secondaryVersion: 2, + }, + ], + ]) + } finally { + for (const request of [...primary.pending, ...secondary.pending]) { + request.resolve() + } + subscription?.unsubscribe() + await Promise.all([ + Promise.resolve(primaryReplay).catch(() => undefined), + Promise.resolve(secondaryReplay).catch(() => undefined), + live.cleanup(), + primary.collection.cleanup(), + secondary.collection.cleanup(), + ]) + } + }) +}) diff --git a/packages/db/tests/query/load-subset-source-readiness-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-source-readiness-refinement-oracle.test.ts new file mode 100644 index 0000000000..1cc4b5962e --- /dev/null +++ b/packages/db/tests/query/load-subset-source-readiness-refinement-oracle.test.ts @@ -0,0 +1,348 @@ +import { expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { BTreeIndex } from '../../src/index.js' +import { extractSimpleComparisons } from '../../src/query/expression-helpers.js' +import { + createLiveQueryCollection, + eq, + toArray, +} from '../../src/query/index.js' +import { flushPromises } from '../utils.js' +import type { LoadSubsetOptions } from '../../src/types.js' + +type Row = { id: string; group: string } + +it.each([ + { oldOutcome: `resolve`, settlementOrder: `old-first` }, + { oldOutcome: `reject`, settlementOrder: `old-first` }, + { oldOutcome: `resolve`, settlementOrder: `fresh-first` }, + { oldOutcome: `reject`, settlementOrder: `fresh-first` }, +] as const)( + `fences a retired source-demand attempt across $settlementOrder $oldOutcome settlement`, + async ({ oldOutcome, settlementOrder }) => { + type Parent = { id: string; group: string } + type Child = { id: string; group: string } + type PendingRequest = { + options: LoadSubsetOptions + rows: ReturnType>> + } + const caseId = `${oldOutcome}-${settlementOrder}` + const parentId = `readiness-generation-parent-${caseId}` + const childId = `readiness-generation-child-${caseId}` + let parentBegin!: () => void + let parentWrite!: (message: { + type: `update` + value: Parent + previousValue: Parent + }) => void + let parentCommit!: () => true | Promise + const oldParent: Parent = { id: `parent`, group: `old` } + const freshParent: Parent = { ...oldParent, group: `fresh` } + const parent = createCollection({ + id: parentId, + getKey: (row) => row.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + parentBegin = begin + parentWrite = write + parentCommit = commit + begin() + write({ type: `insert`, value: oldParent }) + commit() + markReady() + }, + }, + }) + let childBegin!: () => void + let childWrite!: (message: { type: `insert`; value: Child }) => void + let childCommit!: () => true | Promise + const pending: Array = [] + const unloads: Array<{ + options: LoadSubsetOptions + abortedAtUnload: boolean | undefined + }> = [] + const child = createCollection({ + id: childId, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + childBegin = begin + childWrite = write + childCommit = commit + markReady() + return { + loadSubset: (options) => { + const rows = createDeferred>() + pending.push({ options, rows }) + return rows.promise.then(async (acquiredRows) => { + if (acquiredRows.length > 0) { + childBegin() + for (const row of acquiredRows) { + childWrite({ type: `insert`, value: row }) + } + const applied = childCommit() + if (applied !== true) await applied + } + return + }) + }, + unloadSubset: (options) => { + unloads.push({ + options, + abortedAtUnload: options.signal?.aborted, + }) + }, + } + }, + }, + }) + const live = createLiveQueryCollection({ + id: `readiness-generation-live-${caseId}`, + query: (q) => + q.from({ parent }).select(({ parent: parentRow }) => ({ + id: parentRow.id, + children: toArray( + q + .from({ child }) + .where(({ child: childRow }) => + eq(childRow.group, parentRow.group), + ), + ), + })), + startSync: true, + }) + let preloadState: `pending` | `resolved` | `rejected` = `pending` + const preload = live.preload() + void preload.then( + () => { + preloadState = `resolved` + }, + () => { + preloadState = `rejected` + }, + ) + const requestedGroups = (options: LoadSubsetOptions): Array => + extractSimpleComparisons(options.where).flatMap((comparison) => { + if (comparison.field.join(`.`) !== `group`) return [] + if (comparison.operator === `eq`) { + return typeof comparison.value === `string` ? [comparison.value] : [] + } + if (comparison.operator !== `in` || !Array.isArray(comparison.value)) { + return [] + } + return comparison.value.filter( + (value): value is string => typeof value === `string`, + ) + }) + const expectUnloads = ( + ...expectedOptions: ReadonlyArray + ): void => { + expect(unloads).toHaveLength(expectedOptions.length) + for (const [index, options] of expectedOptions.entries()) { + expect(unloads[index]!.options).toBe(options) + expect(unloads[index]!.abortedAtUnload).toBe(true) + } + } + let liveCleaned = false + + try { + await flushPromises() + expect(pending).toHaveLength(1) + expect(requestedGroups(pending[0]!.options)).toEqual([`old`]) + expect(live.status).toBe(`loading`) + expect(preloadState).toBe(`pending`) + + parentBegin() + parentWrite({ + type: `update`, + value: freshParent, + previousValue: oldParent, + }) + const parentApplied = parentCommit() + if (parentApplied !== true) await parentApplied + await flushPromises() + + expect(pending).toHaveLength(2) + expect(requestedGroups(pending[0]!.options)).toEqual([`old`]) + expect(requestedGroups(pending[1]!.options)).toEqual([`fresh`]) + expect(pending[0]!.options.signal?.aborted).toBe(true) + expect(pending[1]!.options.signal?.aborted).toBe(false) + expectUnloads(pending[0]!.options) + expect(live.status).toBe(`loading`) + expect(preloadState).toBe(`pending`) + + const freshChild: Child = { id: `fresh-child`, group: `fresh` } + const freshSettlement = { settled: false } + const settleOld = async () => { + if (oldOutcome === `resolve`) { + pending[0]!.rows.resolve([]) + } else { + pending[0]!.rows.reject(new Error(`retired source demand failed`)) + } + await flushPromises() + } + const settleFresh = async () => { + expect(child.get(freshChild.id)).toBeUndefined() + pending[1]!.rows.resolve([freshChild]) + await flushPromises() + freshSettlement.settled = true + } + const settlements = + settlementOrder === `old-first` + ? [settleOld, settleFresh] + : [settleFresh, settleOld] + for (const settle of settlements) { + await settle() + expect(live.status).toBe(freshSettlement.settled ? `ready` : `loading`) + expect(preloadState).toBe( + freshSettlement.settled ? `resolved` : `pending`, + ) + expect(live.utils.lastSubsetError).toBeUndefined() + } + + await preload + await flushPromises() + + expect(live.status).toBe(`ready`) + expect(preloadState).toBe(`resolved`) + expect(live.utils.lastSubsetError).toBeUndefined() + expect(child.get(freshChild.id)).toEqual( + expect.objectContaining(freshChild), + ) + expect(live.toArray).toEqual([ + expect.objectContaining({ + id: `parent`, + children: [expect.objectContaining({ id: `fresh-child` })], + }), + ]) + expect(pending[1]!.options.signal?.aborted).toBe(false) + expectUnloads(pending[0]!.options) + + await live.cleanup() + liveCleaned = true + expect(pending[1]!.options.signal?.aborted).toBe(true) + expectUnloads(pending[0]!.options, pending[1]!.options) + } finally { + for (const request of pending) { + request.rows.resolve([]) + } + await Promise.all([ + preload.catch(() => undefined), + liveCleaned ? Promise.resolve() : live.cleanup(), + ]) + await Promise.all([parent.cleanup(), child.cleanup()]) + } + }, +) + +it.each([`resolve`, `reject`, `cleanup`] as const)( + `matches cross-source initial readiness through %s`, + async (secondOutcome) => { + const leftId = `readiness-left-${secondOutcome}` + const rightId = `readiness-right-${secondOutcome}` + const leftDelivery = createDeferred() + const rightDelivery = createDeferred() + const createSource = ( + id: string, + row: Row, + delivery: ReturnType>, + ) => + createCollection({ + id, + getKey: (value) => value.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: () => + delivery.promise.then(async () => { + begin() + write({ type: `insert`, value: row }) + const applied = commit() + if (applied !== true) await applied + return + }), + unloadSubset: () => {}, + } + }, + }, + }) + const left = createSource( + leftId, + { id: `left`, group: `shared` }, + leftDelivery, + ) + const right = createSource( + rightId, + { id: `right`, group: `shared` }, + rightDelivery, + ) + const live = createLiveQueryCollection({ + id: `readiness-live-${secondOutcome}`, + query: (q) => + q + .from({ left }) + .innerJoin({ right }, ({ left: leftRow, right: rightRow }) => + eq(leftRow.group, rightRow.group), + ) + .select(({ left: leftRow, right: rightRow }) => ({ + leftId: leftRow.id, + rightId: rightRow.id, + })), + startSync: true, + }) + const preload = live.preload() + void preload.catch(() => undefined) + + try { + expect(live.status).toBe(`loading`) + + leftDelivery.resolve() + await flushPromises() + + expect(live.status).toBe(`loading`) + expect(live.toArray).toEqual([]) + + if (secondOutcome === `cleanup`) { + await live.cleanup() + expect(live.status).toBe(`cleaned-up`) + + rightDelivery.resolve() + await flushPromises() + + expect(live.status).toBe(`cleaned-up`) + expect(live.toArray).toEqual([]) + return + } else if (secondOutcome === `resolve`) { + rightDelivery.resolve() + } else { + rightDelivery.reject(new Error(`right source failed`)) + } + await flushPromises() + + expect(live.status).toBe(secondOutcome === `resolve` ? `ready` : `error`) + if (secondOutcome === `resolve`) { + await expect(preload).resolves.toBeUndefined() + expect(live.toArray).toEqual([ + expect.objectContaining({ leftId: `left`, rightId: `right` }), + ]) + } else { + await expect(preload).rejects.toThrow(`right source failed`) + } + } finally { + leftDelivery.resolve() + rightDelivery.resolve() + await live.cleanup() + await Promise.all([left.cleanup(), right.cleanup()]) + } + }, +) diff --git a/packages/db/tests/query/load-subset-subquery.test.ts b/packages/db/tests/query/load-subset-subquery.test.ts index 27ce7207c7..ead6b921d3 100644 --- a/packages/db/tests/query/load-subset-subquery.test.ts +++ b/packages/db/tests/query/load-subset-subquery.test.ts @@ -1,10 +1,18 @@ -import { beforeEach, describe, expect, it, vi } from 'vitest' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { createCollection } from '../../src/collection/index.js' import { and, + coalesce, createLiveQueryCollection, eq, + gt, gte, + inArray, + isNull, + lt, + lte, + not, + or, } from '../../src/query/index.js' import { PropRef, Value } from '../../src/query/ir.js' import type { Collection } from '../../src/collection/index.js' @@ -13,7 +21,8 @@ import type { NonSingleResult, UtilsRecord, } from '../../src/types.js' -import type { OrderBy } from '../../src/query/ir.js' +import type { BasicExpression, OrderBy } from '../../src/query/ir.js' +import type { Ref } from '../../src/query/index.js' // Sample types for testing type Order = { @@ -74,8 +83,95 @@ type OrdersCollection = Collection< > & NonSingleResult +type ForwardingCase = { + name: string + build: (order: Ref) => BasicExpression + expected: BasicExpression +} + +const forwardingCases: ReadonlyArray = [ + { + name: `equality`, + build: (order) => eq(order.status, `queued`), + expected: eq(new PropRef([`status`]), new Value(`queued`)), + }, + { + name: `greater than`, + build: (order) => gt(order.id, 1), + expected: gt(new PropRef([`id`]), new Value(1)), + }, + { + name: `greater than or equal`, + build: (order) => gte(order.id, 1), + expected: gte(new PropRef([`id`]), new Value(1)), + }, + { + name: `less than`, + build: (order) => lt(order.id, 3), + expected: lt(new PropRef([`id`]), new Value(3)), + }, + { + name: `less than or equal`, + build: (order) => lte(order.id, 3), + expected: lte(new PropRef([`id`]), new Value(3)), + }, + { + name: `IN`, + build: (order) => inArray(order.id, [1, 2, 3]), + expected: inArray(new PropRef([`id`]), [1, 2, 3]), + }, + { + name: `NOT`, + build: (order) => not(eq(order.status, `completed`)), + expected: not(eq(new PropRef([`status`]), new Value(`completed`))), + }, + { + name: `IS NULL`, + build: (order) => isNull(order.status), + expected: isNull(new PropRef([`status`])), + }, + { + name: `OR`, + build: (order) => + or(eq(order.status, `queued`), eq(order.status, `completed`)), + expected: or( + eq(new PropRef([`status`]), new Value(`queued`)), + eq(new PropRef([`status`]), new Value(`completed`)), + ), + }, + { + name: `nested AND/OR`, + build: (order) => + and( + gt(order.id, 1), + or(eq(order.status, `queued`), eq(order.status, `completed`)), + ), + expected: and( + gt(new PropRef([`id`]), new Value(1)), + or( + eq(new PropRef([`status`]), new Value(`queued`)), + eq(new PropRef([`status`]), new Value(`completed`)), + ), + ), + }, +] + describe(`loadSubset with subqueries`, () => { let chargesCollection: ChargersCollection + const cleanups: Array<{ cleanup: () => void | Promise }> = [] + + afterEach(async () => { + const results = await Promise.allSettled( + cleanups + .splice(0) + .reverse() + .map((value) => value.cleanup()), + ) + const failure = results.find( + (result): result is PromiseRejectedResult => result.status === `rejected`, + ) + if (failure) throw failure.reason + }) beforeEach(() => { // Create charges collection @@ -93,6 +189,7 @@ describe(`loadSubset with subqueries`, () => { }, }, }) + cleanups.push(chargesCollection) }) function createOrdersCollectionWithTracking(): { @@ -126,6 +223,24 @@ describe(`loadSubset with subqueries`, () => { return { collection, loadSubsetCalls } } + it.each(forwardingCases)( + `forwards the $name predicate exactly once`, + async ({ build, expected }) => { + const { collection: ordersCollection, loadSubsetCalls } = + createOrdersCollectionWithTracking() + const query = createLiveQueryCollection((q) => + q.from({ order: ordersCollection }).where(({ order }) => build(order)), + ) + cleanups.push(ordersCollection, query) + + await query.preload() + expect(loadSubsetCalls).toHaveLength(1) + expect(loadSubsetCalls[0]?.where).toEqual(expected) + expect(loadSubsetCalls[0]?.orderBy).toBeUndefined() + expect(loadSubsetCalls[0]?.limit).toBeUndefined() + }, + ) + it(`should call loadSubset with where clause for direct query`, async () => { const today = `2024-01-12` const { collection: ordersCollection, loadSubsetCalls } = @@ -137,6 +252,7 @@ describe(`loadSubset with subqueries`, () => { .where(({ order }) => gte(order.scheduled_at, today)) .where(({ order }) => eq(order.status, `queued`)), ) + cleanups.push(ordersCollection, directQuery) await directQuery.preload() @@ -175,6 +291,7 @@ describe(`loadSubset with subqueries`, () => { eq(charge.address_id, prepaidOrder.address_id), ) }) + cleanups.push(ordersCollection, subqueryQuery) await subqueryQuery.preload() @@ -204,26 +321,30 @@ describe(`loadSubset with subqueries`, () => { .orderBy(({ order }) => order.scheduled_at, `desc`) .limit(2), ) + cleanups.push(ordersCollection, directQuery) await directQuery.preload() // Verify loadSubset was called expect(loadSubsetCalls.length).toBeGreaterThan(0) - // Verify the last call has the orderBy clause and limit - const lastCall = loadSubsetCalls[loadSubsetCalls.length - 1] - expect(lastCall).toBeDefined() - expect(lastCall!.orderBy).toBeDefined() - expect(lastCall!.limit).toBe(2) - const expectedOrderBy: OrderBy = [ { expression: new PropRef([`scheduled_at`]), - compareOptions: { direction: `desc`, nulls: `first` }, + compareOptions: { + direction: `desc`, + nulls: `first`, + stringSort: `locale`, + }, }, ] - expect(lastCall!.orderBy).toEqual(expectedOrderBy) + const orderedCalls = loadSubsetCalls.filter(({ orderBy }) => orderBy) + expect(orderedCalls).not.toHaveLength(0) + for (const { orderBy, limit } of orderedCalls) { + expect(orderBy).toEqual(expectedOrderBy) + expect(limit).toBe(2) + } }) it(`should call loadSubset with orderBy clause for subquery`, async () => { @@ -244,25 +365,61 @@ describe(`loadSubset with subqueries`, () => { eq(charge.address_id, prepaidOrder.address_id), ) }) + cleanups.push(ordersCollection, subqueryQuery) await subqueryQuery.preload() // Verify loadSubset was called for the orders collection expect(loadSubsetCalls.length).toBeGreaterThan(0) - // Verify the last call has the orderBy clause and limit - const lastCall = loadSubsetCalls[loadSubsetCalls.length - 1] - expect(lastCall).toBeDefined() - expect(lastCall!.orderBy).toBeDefined() - expect(lastCall!.limit).toBe(2) - const expectedOrderBy: OrderBy = [ { expression: new PropRef([`scheduled_at`]), - compareOptions: { direction: `desc`, nulls: `first` }, + compareOptions: { + direction: `desc`, + nulls: `first`, + stringSort: `locale`, + }, }, ] - expect(lastCall!.orderBy).toEqual(expectedOrderBy) + const orderedCalls = loadSubsetCalls.filter(({ orderBy }) => orderBy) + expect(orderedCalls).not.toHaveLength(0) + for (const { orderBy, limit } of orderedCalls) { + expect(orderBy).toEqual(expectedOrderBy) + expect(limit).toBe(2) + } + }) + + it(`does not forward a computed subquery order to loadSubset`, async () => { + const { collection: ordersCollection, loadSubsetCalls } = + createOrdersCollectionWithTracking() + + const query = createLiveQueryCollection((q) => { + const orderedOrders = q + .from({ order: ordersCollection }) + .select(({ order }) => ({ + address_id: order.address_id, + sortKey: coalesce(order.scheduled_at, `1970-01-01`), + })) + .orderBy(({ $selected }) => $selected.sortKey, `desc`) + .limit(2) + + return q + .from({ charge: chargesCollection }) + .fullJoin({ order: orderedOrders }, ({ charge, order }) => + eq(charge.address_id, order.address_id), + ) + }) + cleanups.push(ordersCollection, query) + + await query.preload() + + expect(loadSubsetCalls).not.toHaveLength(0) + expect( + loadSubsetCalls.every( + ({ orderBy, limit }) => orderBy === undefined && limit === undefined, + ), + ).toBe(true) }) }) diff --git a/packages/db/tests/query/load-subset-transaction-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-transaction-refinement-oracle.test.ts new file mode 100644 index 0000000000..f92a867d1b --- /dev/null +++ b/packages/db/tests/query/load-subset-transaction-refinement-oracle.test.ts @@ -0,0 +1,86 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { createTransaction } from '../../src/transactions.js' + +type Row = { id: string; group: string } + +describe(`loadSubset transaction refinement`, () => { + it.each([`at-commit`, `while-parked`, `after-publication-starts`] as const)( + `matches the independent receipt and publication model when aborting %s`, + async (abortPhase) => { + const sourceId = `transaction-refinement-${abortPhase}` + const remoteRow: Row = { id: `remote`, group: `requested` } + const controller = new AbortController() + const persistence = createDeferred() + const publishedBatches: Array> = [] + const callbackReads: Array> = [] + const source = createCollection({ + id: sourceId, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: ({ signal }) => { + begin() + write({ type: `insert`, value: remoteRow }) + if (abortPhase === `at-commit`) controller.abort() + return commit(signal) + }, + } + }, + }, + }) + source.startSyncImmediate() + const blocker = createTransaction({ + mutationFn: () => persistence.promise, + }) + blocker.mutate(() => + source.insert({ id: `local`, group: `outside-request` }), + ) + const subscription = source.subscribeChanges( + (changes) => { + const remoteKeys = changes + .filter((change) => change.key === remoteRow.id) + .map((change) => String(change.key)) + if (remoteKeys.length === 0) return + publishedBatches.push(remoteKeys) + callbackReads.push(source.has(remoteRow.id) ? [remoteRow.id] : []) + if (abortPhase === `after-publication-starts`) { + controller.abort() + } + }, + { includeInitialState: false }, + ) + const load = source._sync.loadSubset({ signal: controller.signal }) + expect(load).toBeInstanceOf(Promise) + + try { + if (abortPhase === `while-parked`) { + controller.abort() + } + + persistence.resolve() + await blocker.isPersisted.promise + + if (abortPhase !== `after-publication-starts`) { + await expect(load).rejects.toMatchObject({ name: `AbortError` }) + } else { + await expect(load).resolves.toBeUndefined() + } + + const published = abortPhase === `after-publication-starts` + expect(source.has(remoteRow.id)).toBe(published) + expect(publishedBatches).toEqual(published ? [[remoteRow.id]] : []) + expect(callbackReads).toEqual(published ? [[remoteRow.id]] : []) + } finally { + persistence.resolve() + await blocker.isPersisted.promise.catch(() => undefined) + subscription.unsubscribe() + await source.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/query/order-by.test.ts b/packages/db/tests/query/order-by.test.ts index 0b45fb351e..ba55fbc209 100644 --- a/packages/db/tests/query/order-by.test.ts +++ b/packages/db/tests/query/order-by.test.ts @@ -255,10 +255,6 @@ function createEmployeesWithNullableCollection( function createOrderByTests(autoIndex: `off` | `eager`): void { describe(`with autoIndex ${autoIndex}`, () => { - // Some tests require an index for incremental updates (loadMoreIfNeeded). - // These only work with autoIndex: 'eager' which auto-creates the needed indexes. - const itWhenAutoIndexEager = autoIndex === `eager` ? it : it.skip - let employeesCollection: ReturnType let departmentsCollection: ReturnType @@ -620,59 +616,56 @@ function createOrderByTests(autoIndex: `off` | `eager`): void { ]) }) - itWhenAutoIndexEager( - `applies incremental insert of a new row inside the topK but after max sent value correctly`, - async () => { - const collection = createLiveQueryCollection((q) => - q - .from({ employees: employeesCollection }) - .orderBy(({ employees }) => employees.salary, `asc`) - .offset(1) - .limit(10) - .select(({ employees }) => ({ - id: employees.id, - name: employees.name, - salary: employees.salary, - })), - ) - await collection.preload() + it(`applies incremental insert of a new row inside the topK but after max sent value correctly`, async () => { + const collection = createLiveQueryCollection((q) => + q + .from({ employees: employeesCollection }) + .orderBy(({ employees }) => employees.salary, `asc`) + .offset(1) + .limit(10) + .select(({ employees }) => ({ + id: employees.id, + name: employees.name, + salary: employees.salary, + })), + ) + await collection.preload() - const results = Array.from(collection.values()) + const results = Array.from(collection.values()) - expect(results.map((r) => r.salary)).toEqual([ - 52_000, 55_000, 60_000, 65_000, - ]) + expect(results.map((r) => r.salary)).toEqual([ + 52_000, 55_000, 60_000, 65_000, + ]) - // Now insert a new employee with highest salary - // this should now become part of the topK because - // the topK isn't full yet, so even though it's after the max sent value - // it should still be part of the topK - const newEmployee = { - id: 6, - name: `George`, - department_id: 1, - salary: 72_000, - hire_date: `2023-01-01`, - } - - employeesCollection.utils.begin() - employeesCollection.utils.write({ - type: `insert`, - value: newEmployee, - }) - employeesCollection.utils.commit() - - const newResults = Array.from(collection.values()) - - expect(newResults.map((r) => [r.id, r.salary])).toEqual([ - [5, 52_000], - [3, 55_000], - [2, 60_000], - [4, 65_000], - [6, 72_000], - ]) - }, - ) + // Now insert a new employee with highest salary + // this should now become part of the topK because + // the topK isn't full yet, so even though it's after the max sent value + // it should still be part of the topK + const newEmployee = { + id: 6, + name: `George`, + department_id: 1, + salary: 72_000, + hire_date: `2023-01-01`, + } + + employeesCollection.utils.begin() + employeesCollection.utils.write({ + type: `insert`, + value: newEmployee, + }) + employeesCollection.utils.commit() + + const newResults = Array.from(collection.values()) + + expect(newResults.map((r) => [r.id, r.salary])).toEqual([ + [5, 52_000], + [3, 55_000], + [2, 60_000], + [4, 65_000], + [6, 72_000], + ]) + }) it(`applies incremental insert of a new row after the topK correctly`, async () => { const collection = createLiveQueryCollection((q) => @@ -800,40 +793,37 @@ function createOrderByTests(autoIndex: `off` | `eager`): void { ]) }) - itWhenAutoIndexEager( - `handles deletion from partial page with limit larger than data`, - async () => { - const collection = createLiveQueryCollection((q) => - q - .from({ employees: employeesCollection }) - .orderBy(({ employees }) => employees.salary, `desc`) - .limit(20) // Limit larger than number of employees (5) - .select(({ employees }) => ({ - id: employees.id, - name: employees.name, - salary: employees.salary, - })), - ) - await collection.preload() + it(`handles deletion from partial page with limit larger than data`, async () => { + const collection = createLiveQueryCollection((q) => + q + .from({ employees: employeesCollection }) + .orderBy(({ employees }) => employees.salary, `desc`) + .limit(20) // Limit larger than number of employees (5) + .select(({ employees }) => ({ + id: employees.id, + name: employees.name, + salary: employees.salary, + })), + ) + await collection.preload() - const results = Array.from(collection.values()) - expect(results).toHaveLength(5) - expect(results[0]!.name).toBe(`Diana`) - - // Delete Diana (the highest paid employee, first in DESC order) - const dianaData = employeeData.find((e) => e.id === 4)! - employeesCollection.utils.begin() - employeesCollection.utils.write({ - type: `delete`, - value: dianaData, - }) - employeesCollection.utils.commit() - - const newResults = Array.from(collection.values()) - expect(newResults).toHaveLength(4) - expect(newResults[0]!.name).toBe(`Bob`) - }, - ) + const results = Array.from(collection.values()) + expect(results).toHaveLength(5) + expect(results[0]!.name).toBe(`Diana`) + + // Delete Diana (the highest paid employee, first in DESC order) + const dianaData = employeeData.find((e) => e.id === 4)! + employeesCollection.utils.begin() + employeesCollection.utils.write({ + type: `delete`, + value: dianaData, + }) + employeesCollection.utils.commit() + + const newResults = Array.from(collection.values()) + expect(newResults).toHaveLength(4) + expect(newResults[0]!.name).toBe(`Bob`) + }) }) describe(`OrderBy with Joins`, () => { @@ -1851,141 +1841,172 @@ function createOrderByTests(autoIndex: `off` | `eager`): void { }) describe(`OrderBy Optimization Tests`, () => { - const itWhenAutoIndex = autoIndex === `eager` ? it : it.skip - - itWhenAutoIndex( - `optimizes single-column orderBy when passed as single value`, - async () => { - // Patch getConfig to expose the builder on the returned config for test access - const { CollectionConfigBuilder } = await import( - `../../src/query/live/collection-config-builder.js` + it(`optimizes single-column orderBy when passed as single value`, async () => { + // Patch getConfig to expose the builder on the returned config for test access + const { CollectionConfigBuilder } = await import( + `../../src/query/live/collection-config-builder.js` + ) + const originalGetConfig = CollectionConfigBuilder.prototype.getConfig + + CollectionConfigBuilder.prototype.getConfig = function (this: any) { + const cfg = originalGetConfig.call(this) + ;(cfg as any).__builder = this + return cfg + } + + try { + const collection = createLiveQueryCollection((q) => + q + .from({ employees: employeesCollection }) + .orderBy(({ employees }) => employees.salary, `desc`) + .limit(3) + .select(({ employees }) => ({ + id: employees.id, + name: employees.name, + salary: employees.salary, + })), ) - const originalGetConfig = CollectionConfigBuilder.prototype.getConfig - - CollectionConfigBuilder.prototype.getConfig = function (this: any) { - const cfg = originalGetConfig.call(this) - ;(cfg as any).__builder = this - return cfg - } - - try { - const collection = createLiveQueryCollection((q) => - q - .from({ employees: employeesCollection }) - .orderBy(({ employees }) => employees.salary, `desc`) - .limit(3) - .select(({ employees }) => ({ - id: employees.id, - name: employees.name, - salary: employees.salary, - })), - ) - await collection.preload() + await collection.preload() - const builder = (collection as any).config.__builder - expect(builder).toBeTruthy() - expect( - Object.keys(builder.optimizableOrderByCollections), - ).toContain(employeesCollection.id) - } finally { - CollectionConfigBuilder.prototype.getConfig = originalGetConfig - } - }, - ) - - itWhenAutoIndex( - `optimizes orderBy with alias paths in joins`, - async () => { - // Patch getConfig to expose the builder on the returned config for test access - const { CollectionConfigBuilder } = await import( - `../../src/query/live/collection-config-builder.js` + const builder = (collection as any).config.__builder + expect(builder).toBeTruthy() + const orderByInfo = Object.values( + builder.optimizableOrderByCollections, + )[0] as any + const orderedSource = builder.collectionSources.find( + (source: { alias: string }) => source.alias === `employees`, ) - const originalGetConfig = CollectionConfigBuilder.prototype.getConfig - - CollectionConfigBuilder.prototype.getConfig = function (this: any) { - const cfg = originalGetConfig.call(this) - ;(cfg as any).__builder = this - return cfg - } - - try { - const collection = createLiveQueryCollection((q) => - q - .from({ employees: employeesCollection }) - .join( - { departments: departmentsCollection }, - ({ employees, departments }) => - eq(employees.department_id, departments.id), - ) - .orderBy(({ departments }) => departments.name, `asc`) - .limit(5) - .select(({ employees, departments }) => ({ - employeeId: employees.id, - employeeName: employees.name, - departmentName: departments.name, - })), - ) + expect(orderByInfo.sourceId).toBe(orderedSource.sourceId) + } finally { + CollectionConfigBuilder.prototype.getConfig = originalGetConfig + } + }) - await collection.preload() - - const builder = (collection as any).config.__builder - expect(builder).toBeTruthy() - - // Verify that the order-by optimization is scoped to the departments alias - const orderByInfo = Object.values( - builder.optimizableOrderByCollections, - )[0] as any - expect(orderByInfo).toBeDefined() - expect(orderByInfo.alias).toBe(`departments`) - expect(orderByInfo.offset).toBe(0) - expect(orderByInfo.limit).toBe(5) - } finally { - CollectionConfigBuilder.prototype.getConfig = originalGetConfig - } - }, - ) - - itWhenAutoIndex( - `optimizes single-column orderBy when passed as array with single element`, - async () => { - // Patch getConfig to expose the builder on the returned config for test access - const { CollectionConfigBuilder } = await import( - `../../src/query/live/collection-config-builder.js` + it(`optimizes orderBy with alias paths in joins`, async () => { + // Patch getConfig to expose the builder on the returned config for test access + const { CollectionConfigBuilder } = await import( + `../../src/query/live/collection-config-builder.js` + ) + const originalGetConfig = CollectionConfigBuilder.prototype.getConfig + + CollectionConfigBuilder.prototype.getConfig = function (this: any) { + const cfg = originalGetConfig.call(this) + ;(cfg as any).__builder = this + return cfg + } + + try { + const collection = createLiveQueryCollection((q) => + q + .from({ employees: employeesCollection }) + .join( + { departments: departmentsCollection }, + ({ employees, departments }) => + eq(employees.department_id, departments.id), + ) + .orderBy(({ departments }) => departments.name, `asc`) + .limit(5) + .select(({ employees, departments }) => ({ + employeeId: employees.id, + employeeName: employees.name, + departmentName: departments.name, + })), + ) + + await collection.preload() + + const builder = (collection as any).config.__builder + expect(builder).toBeTruthy() + + // Verify that the order-by optimization is scoped to the departments alias + const orderByInfo = Object.values( + builder.optimizableOrderByCollections, + )[0] as any + const orderedSource = builder.collectionSources.find( + (source: { alias: string }) => source.alias === `departments`, ) - const originalGetConfig = CollectionConfigBuilder.prototype.getConfig - - CollectionConfigBuilder.prototype.getConfig = function (this: any) { - const cfg = originalGetConfig.call(this) - ;(cfg as any).__builder = this - return cfg - } - - try { - const collection = createLiveQueryCollection((q) => - q - .from({ employees: employeesCollection }) - .orderBy(({ employees }) => [employees.salary], `desc`) - .limit(3) - .select(({ employees }) => ({ - id: employees.id, - name: employees.name, - salary: employees.salary, - })), + expect(orderByInfo).toBeDefined() + expect(orderByInfo.alias).toBe(`departments`) + expect(orderByInfo.sourceId).toBe(orderedSource.sourceId) + expect(orderByInfo.offset).toBe(0) + expect(orderByInfo.limit).toBe(5) + } finally { + CollectionConfigBuilder.prototype.getConfig = originalGetConfig + } + }) + + it(`loads an ordered self-join through the ordered alias`, async () => { + const collection = createLiveQueryCollection((q) => + q + .from({ employee: employeesCollection }) + .join({ manager: employeesCollection }, ({ employee, manager }) => + eq(employee.id, manager.id), ) + .orderBy(({ manager }) => manager.name, `asc`) + .limit(3) + .select(({ employee, manager }) => ({ + id: employee.id, + employeeName: employee.name, + managerName: manager.name, + })), + ) - await collection.preload() + await collection.preload() - const builder = (collection as any).config.__builder - expect(builder).toBeTruthy() - expect( - Object.keys(builder.optimizableOrderByCollections), - ).toContain(employeesCollection.id) - } finally { - CollectionConfigBuilder.prototype.getConfig = originalGetConfig - } - }, - ) + expect( + Array.from(collection.values()).map((row) => [ + row.employeeName, + row.managerName, + ]), + ).toEqual([ + [`Alice`, `Alice`], + [`Bob`, `Bob`], + [`Charlie`, `Charlie`], + ]) + }) + + it(`optimizes single-column orderBy when passed as array with single element`, async () => { + // Patch getConfig to expose the builder on the returned config for test access + const { CollectionConfigBuilder } = await import( + `../../src/query/live/collection-config-builder.js` + ) + const originalGetConfig = CollectionConfigBuilder.prototype.getConfig + + CollectionConfigBuilder.prototype.getConfig = function (this: any) { + const cfg = originalGetConfig.call(this) + ;(cfg as any).__builder = this + return cfg + } + + try { + const collection = createLiveQueryCollection((q) => + q + .from({ employees: employeesCollection }) + .orderBy(({ employees }) => [employees.salary], `desc`) + .limit(3) + .select(({ employees }) => ({ + id: employees.id, + name: employees.name, + salary: employees.salary, + })), + ) + + await collection.preload() + + const builder = (collection as any).config.__builder + expect(builder).toBeTruthy() + const orderByInfo = Object.values( + builder.optimizableOrderByCollections, + )[0] as any + const orderedSource = builder.collectionSources.find( + (source: { alias: string }) => source.alias === `employees`, + ) + expect(orderByInfo.sourceId).toBe(orderedSource.sourceId) + } finally { + CollectionConfigBuilder.prototype.getConfig = originalGetConfig + } + }) }) describe(`String Comparison Tests`, () => { @@ -2631,7 +2652,7 @@ describe(`OrderBy with duplicate values`, () => { ]) // Now move to next page (offset 5, limit 5) - collection.utils.setWindow({ offset: 5, limit: 5 }) + await collection.utils.setWindow({ offset: 5, limit: 5 }) await collection.stateWhenReady() // Second page should return items 6-10 (all with value 5) @@ -2648,7 +2669,7 @@ describe(`OrderBy with duplicate values`, () => { // Now move to third page (offset 10, limit 5) // It should advance past the duplicate 5s - collection.utils.setWindow({ offset: 10, limit: 5 }) + await collection.utils.setWindow({ offset: 10, limit: 5 }) await collection.stateWhenReady() // Third page should return items 11-13 (the items after the duplicate 5s) @@ -2665,7 +2686,7 @@ describe(`OrderBy with duplicate values`, () => { ]) // Verify we can continue to next page - collection.utils.setWindow({ offset: 15, limit: 5 }) + await collection.utils.setWindow({ offset: 15, limit: 5 }) await collection.stateWhenReady() // Should be empty since we've exhausted all items @@ -2850,16 +2871,17 @@ describe(`OrderBy with duplicate values`, () => { { id: 4, a: 4, keep: true }, { id: 5, a: 5, keep: true }, ]) - expect(loadSubsetCallCount).toBe(1) + expect(loadSubsetCallCount).toBeGreaterThanOrEqual(1) + expect(loadSubsetCallCount).toBeLessThanOrEqual(2) // First loadSubset call (initial page at offset 0) has no cursor expect(loadSubsetCursors[0]).toBeUndefined() + const initialLoadSubsetCallCount = loadSubsetCallCount // Now move to next page (offset 5, limit 5) - this should trigger loadSubset with a cursor const moveToSecondPage = collection.utils.setWindow({ offset: 5, limit: 5, }) - expect(moveToSecondPage).toBeInstanceOf(Promise) await moveToSecondPage // Second page should return items 6-10 (all with value 5, loaded from sync layer) @@ -2873,12 +2895,9 @@ describe(`OrderBy with duplicate values`, () => { { id: 9, a: 5, keep: true }, { id: 10, a: 5, keep: true }, ]) - // we expect 1 new loadSubset call (cursor expressions for whereFrom/whereCurrent are now combined in single call) - expect(loadSubsetCallCount).toBe(2) - // Second loadSubset call (pagination) has a cursor with whereFrom and whereCurrent - expect(loadSubsetCursors[1]).toBeDefined() - expect(loadSubsetCursors[1]).toHaveProperty(`whereFrom`) - expect(loadSubsetCursors[1]).toHaveProperty(`whereCurrent`) + // Initial tie expansion already loaded this page; reuse it without a fetch. + expect(loadSubsetCallCount).toBe(initialLoadSubsetCallCount) + const secondPageLoadSubsetCallCount = loadSubsetCallCount // Now move to third page (offset 10, limit 5) // It should advance past the duplicate 5s @@ -2887,11 +2906,7 @@ describe(`OrderBy with duplicate values`, () => { limit: 5, }) - // Now it is `true` because we already have that page - // because when we loaded the 2nd page we loaded all the duplicate 5s and then we loaded - // values > 5 with limit 5 but since the entire 2nd page is filled with the duplicate 5s - // we in fact already loaded the third page so it is immediately available here - expect(moveToThirdPage).toBe(true) + await moveToThirdPage // Third page should return items 11-13 (the items after the duplicate 5s) // The bug would cause this to stall and return empty or get stuck @@ -2905,10 +2920,12 @@ describe(`OrderBy with duplicate values`, () => { { id: 14, a: 14, keep: true }, { id: 15, a: 15, keep: true }, ]) - // We expect no more loadSubset calls because when we loaded the previous page - // we asked for all data equal to max value and LIMIT values greater than max value - // and the LIMIT values greater than max value already loaded the next page - expect(loadSubsetCallCount).toBe(2) + expect(loadSubsetCallCount).toBeGreaterThan( + secondPageLoadSubsetCallCount, + ) + expect(loadSubsetCallCount).toBeLessThanOrEqual( + secondPageLoadSubsetCallCount + 2, + ) }) it(`should correctly advance window when there are duplicate values loaded from both local collection and sync layer`, async () => { @@ -3086,16 +3103,17 @@ describe(`OrderBy with duplicate values`, () => { { id: 4, a: 4, keep: true }, { id: 5, a: 5, keep: true }, ]) - expect(loadSubsetCallCount).toBe(1) + expect(loadSubsetCallCount).toBeGreaterThanOrEqual(1) + expect(loadSubsetCallCount).toBeLessThanOrEqual(2) // First loadSubset call (initial page at offset 0) has no cursor expect(loadSubsetCursors[0]).toBeUndefined() + const initialLoadSubsetCallCount = loadSubsetCallCount // Now move to next page (offset 5, limit 5) - this should trigger loadSubset with a cursor const moveToSecondPage = collection.utils.setWindow({ offset: 5, limit: 5, }) - expect(moveToSecondPage).toBeInstanceOf(Promise) await moveToSecondPage // Second page should return items 6-10 (all with value 5, loaded from sync layer) @@ -3109,12 +3127,9 @@ describe(`OrderBy with duplicate values`, () => { { id: 9, a: 5, keep: true }, { id: 10, a: 5, keep: true }, ]) - // we expect 1 new loadSubset call (cursor expressions for whereFrom/whereCurrent are now combined in single call) - expect(loadSubsetCallCount).toBe(2) - // Second loadSubset call (pagination) has a cursor with whereFrom and whereCurrent - expect(loadSubsetCursors[1]).toBeDefined() - expect(loadSubsetCursors[1]).toHaveProperty(`whereFrom`) - expect(loadSubsetCursors[1]).toHaveProperty(`whereCurrent`) + // Initial tie expansion already loaded this page; reuse it without a fetch. + expect(loadSubsetCallCount).toBe(initialLoadSubsetCallCount) + const secondPageLoadSubsetCallCount = loadSubsetCallCount // Now move to third page (offset 10, limit 5) // It should advance past the duplicate 5s @@ -3123,11 +3138,7 @@ describe(`OrderBy with duplicate values`, () => { limit: 5, }) - // Now it is `true` because we already have that page - // because when we loaded the 2nd page we loaded all the duplicate 5s and then we loaded - // values > 5 with limit 5 but since the entire 2nd page is filled with the duplicate 5s - // we in fact already loaded the third page so it is immediately available here - expect(moveToThirdPage).toBe(true) + await moveToThirdPage // Third page should return items 11-13 (the items after the duplicate 5s) // The bug would cause this to stall and return empty or get stuck @@ -3141,10 +3152,12 @@ describe(`OrderBy with duplicate values`, () => { { id: 14, a: 14, keep: true }, { id: 15, a: 15, keep: true }, ]) - // We expect no more loadSubset calls because when we loaded the previous page - // we asked for all data equal to max value and LIMIT values greater than max value - // and the LIMIT values greater than max value already loaded the next page - expect(loadSubsetCallCount).toBe(2) + expect(loadSubsetCallCount).toBeGreaterThan( + secondPageLoadSubsetCallCount, + ) + expect(loadSubsetCallCount).toBeLessThanOrEqual( + secondPageLoadSubsetCallCount + 2, + ) }) }) } @@ -3188,9 +3201,10 @@ describe(`OrderBy with Date values and precision differences`, () => { const initialData = testData.slice(0, 5) - // Track the cursor expressions sent to loadSubset - // Note: cursor expressions are now passed separately from where (whereFrom/whereCurrent/lastKey) + // Track both forms used by ordered loading: page cursors and boundary + // predicates. const loadSubsetCursors: Array = [] + const loadSubsetWheres: Array = [] const sourceCollection = createCollection( mockSyncCollectionOptions({ @@ -3212,6 +3226,7 @@ describe(`OrderBy with Date values and precision differences`, () => { loadSubset: (options) => { // Capture the cursor for inspection (now contains whereFrom/whereCurrent/lastKey) loadSubsetCursors.push(options.cursor) + loadSubsetWheres.push(options.where) return new Promise((resolve) => { setTimeout(() => { @@ -3317,21 +3332,28 @@ describe(`OrderBy with Date values and precision differences`, () => { // Find the cursor that contains the "whereCurrent" expression (the minValue query) // With the fix, whereCurrent should be: and(gte(createdAt, baseTime), lt(createdAt, baseTime+1ms)) // Without the fix, this would be: eq(createdAt, baseTime) - const cursorWithDateRange = loadSubsetCursors.find((cursor) => { - if (!cursor?.whereCurrent) return false - const whereCurrent = cursor.whereCurrent - // Check if whereCurrent is an 'and' with 'gte' and 'lt' (the fix) - if (whereCurrent.name === `and` && whereCurrent.args?.length === 2) { - const [first, second] = whereCurrent.args - return first?.name === `gte` && second?.name === `lt` + const findDateRange = (expression: any): any => { + if (!expression) return undefined + if (expression.name === `and` && expression.args?.length === 2) { + const [first, second] = expression.args + if (first?.name === `gte` && second?.name === `lt`) { + return expression + } } - return false - }) + return expression.args + ?.map((argument: any) => findDateRange(argument)) + .find(Boolean) + } + const equalValuesQuery = [ + ...loadSubsetWheres, + ...loadSubsetCursors.map((cursor) => cursor?.whereCurrent), + ] + .map(findDateRange) + .find(Boolean) // The fix should produce a range query (and(gte, lt)) for Date values // instead of an exact equality query (eq) - expect(cursorWithDateRange).toBeDefined() - const equalValuesQuery = cursorWithDateRange.whereCurrent + expect(equalValuesQuery).toBeDefined() expect(equalValuesQuery.name).toBe(`and`) expect(equalValuesQuery.args[0].name).toBe(`gte`) expect(equalValuesQuery.args[1].name).toBe(`lt`) diff --git a/packages/db/tests/query/ordered-default-work.test.ts b/packages/db/tests/query/ordered-default-work.test.ts new file mode 100644 index 0000000000..d346e21197 --- /dev/null +++ b/packages/db/tests/query/ordered-default-work.test.ts @@ -0,0 +1,231 @@ +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createLiveQueryCollection } from '../../src/query/index.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { flushPromises } from '../utils.js' +import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' + +type Row = { id: number; rank: number } + +async function setup( + indexed: boolean, + multi: boolean, + evict = false, + syncFailure = false, +) { + const truth = Array.from({ length: 20 }, (_, id) => ({ + id: id + 1, + rank: id + 1, + })) + const installed = new Map() + const calls: Array = [] + const active = new Set() + const owned = new Map>() + let sync!: Parameters[`sync`]>[0] + let failFull = 0 + const failure = new Error(`transient full-source failure`) + const source = createCollection({ + getKey: (row) => row.id, + syncMode: `on-demand`, + ...(indexed + ? { autoIndex: `eager` as const, defaultIndexType: BTreeIndex } + : {}), + sync: { + sync: (operations) => { + sync = operations + sync.markReady() + return { + loadSubset: (options) => { + calls.push(options) + if (failFull && !options.orderBy && !options.where) { + failFull-- + if (syncFailure) throw failure + active.add(options) + return Promise.reject(failure) + } + active.add(options) + const rows = truth + .filter( + (row) => + (!options.where || + evaluateReferenceExpression(options.where, row) === true) && + (!options.cursor || + evaluateReferenceExpression( + options.cursor.whereFrom, + row, + ) === true), + ) + .sort((a, b) => a.rank - b.rank || a.id - b.id) + const offset = options.cursor ? 0 : (options.offset ?? 0) + const selected = rows.slice( + offset, + options.limit === undefined ? undefined : offset + options.limit, + ) + owned.set(options, new Set(selected.map((row) => row.id))) + sync.begin() + for (const row of selected) { + if (installed.get(row.id) === row) continue + sync.write({ + type: installed.has(row.id) ? `update` : `insert`, + value: row, + }) + installed.set(row.id, row) + } + return Promise.resolve(sync.commit()).then(() => {}) + }, + unloadSubset: (options) => { + expect(active.delete(options)).toBe(true) + const released = owned.get(options) + owned.delete(options) + if (!evict || !released) return + sync.begin() + for (const id of released) { + const row = installed.get(id) + if (row && ![...owned.values()].some((keys) => keys.has(id))) { + installed.delete(id) + sync.write({ type: `delete`, value: row }) + } + } + void sync.commit() + }, + } + }, + }, + }) + const createQuery = () => + createLiveQueryCollection((q) => { + const sorted = q.from({ row: source }).orderBy(({ row }) => row.rank) + return (multi ? sorted.orderBy(({ row }) => row.id) : sorted) + .limit(2) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + }) + const live = createQuery() + await live.preload() + return { + live, + createQuery, + source, + calls, + active, + failure, + failNextFull: (count = 1) => { + failFull = count + }, + insert: (row: Row) => { + truth.push(row) + installed.set(row.id, row) + sync.begin() + sync.write({ type: `insert`, value: row }) + return sync.commit() + }, + remove: (id: number) => { + const index = truth.findIndex((row) => row.id === id) + const [row] = truth.splice(index, 1) + installed.delete(id) + sync.begin() + sync.write({ type: `delete`, value: row! }) + return sync.commit() + }, + cleanup: async () => { + await live.cleanup() + await source.cleanup() + }, + } +} + +describe(`Ordered source work across default and indexed plans`, () => { + it.each( + [false, true].flatMap((indexed) => + [false, true].map((multi) => ({ indexed, multi })), + ), + )( + `does not reacquire a full window for out-of-window inserts: %j`, + async ({ indexed, multi }) => { + const h = await setup(indexed, multi) + try { + const calls = h.calls.length + const active = h.active.size + for (let id = 100; id < 103; id++) { + await h.insert({ id, rank: id }) + await flushPromises() + } + expect(h.live.toArray.map((row) => row.id)).toEqual([1, 2]) + expect.soft(h.calls.length - calls).toBe(0) + expect(h.active.size).toBe(active) + } finally { + await h.cleanup() + } + }, + ) + + it.each( + [false, true].flatMap((multi) => + [false, true].map((peer) => ({ multi, peer })), + ), + )( + `retires a replaced prefix without evicting current or peer rows: %j`, + async ({ multi, peer }) => { + const h = await setup(false, multi, true) + const other = peer ? h.createQuery() : undefined + try { + await other?.preload() + for (const limit of [3, 4, 5]) await h.live.utils.setWindow({ limit }) + expect(h.live.toArray.map((row) => row.id)).toEqual([1, 2, 3, 4, 5]) + expect([...h.active].filter((options) => options.orderBy)).toHaveLength( + peer ? 2 : 1, + ) + if (other) expect(other.toArray.map((row) => row.id)).toEqual([1, 2]) + } finally { + await other?.cleanup() + await h.cleanup() + } + }, + ) + + it.each( + [`success`, `exhausted`, `cleanup`].flatMap((outcome) => + [false, true].map((syncFailure) => ({ outcome, syncFailure })), + ), + )( + `bounds automatic repair with stale rows retained: %j`, + async ({ outcome, syncFailure }) => { + const h = await setup(true, false, false, syncFailure) + vi.useFakeTimers({ toFake: [`setTimeout`, `clearTimeout`] }) + try { + h.failNextFull(outcome === `success` ? 1 : 3) + await h.remove(1) + await vi.advanceTimersByTimeAsync(0) + const afterFailure = h.calls.length + expect(h.live.utils.lastSubsetError).toBe(h.failure) + expect(h.live.status).toBe(`ready`) + expect(h.live.toArray.map((row) => row.id)).toEqual([1, 2]) + if (outcome === `cleanup`) await h.live.cleanup() + await vi.advanceTimersByTimeAsync(249) + expect(h.calls).toHaveLength(afterFailure) + await vi.advanceTimersByTimeAsync(751) + expect(h.calls.length - afterFailure).toBe( + outcome === `cleanup` ? 0 : outcome === `success` ? 1 : 2, + ) + if (outcome === `cleanup`) return + expect(h.live.status).toBe(`ready`) + expect(h.live.toArray.map((row) => row.id)).toEqual( + outcome === `success` ? [2, 3] : [1, 2], + ) + const settledCalls = h.calls.length + await h.insert({ id: 50, rank: 0 }) + await vi.advanceTimersByTimeAsync(0) + await vi.advanceTimersByTimeAsync(10000) + expect(h.calls).toHaveLength(settledCalls) + if (outcome === `exhausted`) { + expect(h.live.toArray.map((row) => row.id)).toEqual([1, 2]) + await h.live.utils.setWindow({ limit: 2 }) + } + expect(h.live.toArray.map((row) => row.id)).toEqual([50, 2]) + } finally { + await h.cleanup() + vi.useRealTimers() + } + }, + ) +}) diff --git a/packages/db/tests/query/ordered-demand-retirement.test.ts b/packages/db/tests/query/ordered-demand-retirement.test.ts new file mode 100644 index 0000000000..cd31c8c828 --- /dev/null +++ b/packages/db/tests/query/ordered-demand-retirement.test.ts @@ -0,0 +1,128 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createLiveQueryCollection } from '../../src/query/index.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { flushPromises } from '../utils.js' +import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' + +type Row = { id: number; rank: number } + +describe(`Ordered demand retirement`, () => { + it.each([false, true])( + `replays only authoritative demand after repair, with peer=%s`, + async (withPeer) => { + const truth: Array = [1, 2, 3, 4].map((id) => ({ id, rank: id })) + const active = new Set() + const calls: Array = [] + const installed = new Map() + let transferred = 0 + let sync!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + sync.markReady() + return { + loadSubset: async (options) => { + active.add(options) + calls.push(options) + await Promise.resolve() + if (options.signal?.aborted) return + const rows = truth + .filter( + (row) => + (!options.where || + evaluateReferenceExpression(options.where, row) === + true) && + (!options.cursor || + evaluateReferenceExpression( + options.cursor.whereFrom, + row, + ) === true), + ) + .sort((a, b) => a.rank - b.rank) + const offset = options.cursor ? 0 : (options.offset ?? 0) + const selected = rows.slice( + offset, + options.limit === undefined + ? undefined + : offset + options.limit, + ) + transferred += selected.length + sync.begin() + for (const value of selected) { + if (installed.get(value.id) === value) continue + sync.write({ + type: installed.has(value.id) ? `update` : `insert`, + value, + }) + installed.set(value.id, value) + } + await sync.commit() + }, + unloadSubset: (options) => { + expect(active.delete(options)).toBe(true) + }, + } + }, + }, + }) + const query = () => + createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2) + .select(({ row }) => ({ id: row.id, rank: row.rank })), + ) + const live = query() + const peer = withPeer ? query() : undefined + try { + await live.preload() + await peer?.preload() + const owners = withPeer ? 2 : 1 + expect(active.size).toBe(owners * 2) + truth[0] = { id: 1, rank: 10 } + installed.set(1, truth[0]) + sync.begin() + sync.write({ type: `update`, value: truth[0] }) + await sync.commit() + await flushPromises() + expect(live.toArray.map((row) => row.id)).toEqual([2, 3]) + expect.soft(active.size).toBe(owners) + expect + .soft( + [...active].every((options) => !options.orderBy && !options.where), + ) + .toBe(true) + + const beforeCalls = calls.length + const beforeRows = transferred + installed.clear() + sync.begin() + sync.truncate() + await sync.commit() + await flushPromises() + expect.soft(calls.length - beforeCalls).toBe(owners) + expect.soft(transferred - beforeRows).toBe(owners * truth.length) + expect(live.toArray.map((row) => row.id)).toEqual([2, 3]) + expect(live.isReady()).toBe(true) + await live.cleanup() + expect.soft(active.size).toBe(withPeer ? 1 : 0) + if (peer) { + expect(peer.toArray.map((row) => row.id)).toEqual([2, 3]) + expect(peer.isReady()).toBe(true) + } + } finally { + await live.cleanup() + await peer?.cleanup() + await source.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts b/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts new file mode 100644 index 0000000000..eec402036d --- /dev/null +++ b/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts @@ -0,0 +1,542 @@ +import { isDeepStrictEqual } from 'node:util' +import { describe, expect, it } from 'vitest' +import { fc, test as fcTest } from '@fast-check/vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createLiveQueryCollection } from '../../src/query/index.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { flushPromises } from '../utils.js' +import { + oracleRandomParameters, + readOracleRunConfig, +} from '../oracle-config.js' +import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' + +type Row = { id: number; rank: number; version: number } +type Route = `page` | `prefix` | `boundary` | `full-source` +type Scenario = { + route: Route + delivery: `before-settlement` | `after-success` + window: `keep` | `widen` + outcome: `resolve` | `reject` | `abort-error` + session: `retain` | `restart` + barrier: `initial` | `replay` + rankOffset?: number + rankStep?: number +} + +const routes: ReadonlyArray = [ + `page`, + `prefix`, + `boundary`, + `full-source`, +] + +async function observeHistory(scenario: Scenario) { + const mismatches: Array<{ law: string; actual: unknown; expected: unknown }> = + [] + const check = (law: string, actual: unknown, expected: unknown) => { + if (!isDeepStrictEqual(actual, expected)) + mismatches.push({ law, actual, expected }) + } + type Sync = Parameters[`sync`]>[0] + const truth: Array = [1, 2, 3, 4, 5].map((id) => ({ + id, + version: 1, + rank: + (scenario.rankOffset ?? 0) + + (scenario.rankStep ?? 1) * + (scenario.route === `boundary` && id === 2 ? 1 : id), + })) + const referenceWindow = (limit: number) => truth.slice(0, limit) + const gate = createDeferred() + const failure = + scenario.outcome === `abort-error` + ? Object.assign(new Error(`target canceled`), { name: `AbortError` }) + : new Error(`target rejected`) + const requests: Array<{ + options: LoadSubsetOptions + session: number + ids: Array + indexed: boolean + applied: boolean + }> = [] + const released: Array = [] + const sourceCleanups: Array = [] + const publications: Array> = [] + const deliveredRows = new Map() + let generation = 0 + let activeSync!: Sync + let activeInstalled!: Set + let targetOutcome: string | undefined + let target: (typeof requests)[number] | undefined + let targetWrites = 0 + let appliedBeforeSettlement = false + let replayStarted = false + let allowTarget = scenario.barrier === `initial` + const source = createCollection({ + id: `ordered-history-source-${JSON.stringify(scenario)}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: scenario.route === `prefix` ? `off` : `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (sync: Sync) => { + activeSync = sync + const session = ++generation + const installed = new Set() + activeInstalled = installed + sync.markReady() + return { + loadSubset: (options) => { + if (requests.length >= 30) + throw new Error(`ordered history exceeded source work bound`) + let rows = truth.filter( + (row) => + !options.where || + evaluateReferenceExpression(options.where, row) === true, + ) + if (options.cursor) + rows = rows.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + const offset = options.cursor ? 0 : (options.offset ?? 0) + rows = rows.slice( + offset, + options.limit === undefined ? undefined : offset + options.limit, + ) + const request = { + options, + session, + ids: rows.map(({ id }) => id), + indexed: source.indexes.size > 0, + applied: false, + } + requests.push(request) + const matchesRoute = + scenario.route === `boundary` + ? options.orderBy === undefined && options.where !== undefined + : scenario.route === `full-source` + ? options.limit === undefined && options.where === undefined + : options.orderBy !== undefined && options.limit !== undefined + const gated = allowTarget && !target && matchesRoute + if (gated) target = request + const apply = async () => { + if (options.signal?.aborted || session !== generation) return + request.applied = true + const fresh = rows.filter(({ id }) => !installed.has(id)) + if (fresh.length === 0) return + sync.begin() + for (const value of fresh) { + installed.add(value.id) + sync.write({ type: `insert`, value }) + if (gated) targetWrites++ + } + const receipt = sync.commit() + if (receipt !== true) await receipt + } + return (async () => { + if (gated && scenario.delivery === `before-settlement`) + await apply() + if (gated) + await gate.promise.then( + () => { + targetOutcome = `resolve` + }, + (error: unknown) => { + targetOutcome = + error === failure ? scenario.outcome : `unexpected` + throw error + }, + ) + if (!gated || scenario.delivery === `after-success`) await apply() + })() + }, + unloadSubset: (options) => { + released.push(options) + }, + cleanup: () => { + sourceCleanups.push(session) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => { + const from = q.from({ row: source }) + return (scenario.route === `full-source` ? from.distinct() : from) + .orderBy(({ row }) => row.rank) + .limit(1) + .select(({ row }) => ({ + id: row.id, + rank: row.rank, + version: row.version, + })) + }) + const read = () => + live.toArray.map(({ id, rank, version }) => ({ id, rank, version })) + const subscription = live.subscribeChanges( + (batch) => { + // subscribeChanges also sends an empty initial-snapshot completion callback. + // Count row publications only when there are row deltas. + if (batch.length === 0) return + for (const change of batch) { + const value = { + id: change.value.id, + rank: change.value.rank, + version: change.value.version, + } + if (change.type === `delete`) { + check(`delete-payload`, value, deliveredRows.get(change.key)) + deliveredRows.delete(change.key) + } else { + if (change.type === `update`) + check( + `update-previous`, + change.previousValue && { + id: change.previousValue.id, + rank: change.previousValue.rank, + version: change.previousValue.version, + }, + deliveredRows.get(change.key), + ) + else check(`insert-new-key`, deliveredRows.has(change.key), false) + deliveredRows.set(change.key, value) + } + } + const byId = (rows: Array) => + rows.slice().sort((a, b) => a.id - b.id) + check(`message-snapshot`, byId([...deliveredRows.values()]), byId(read())) + publications.push(read()) + }, + { includeInitialState: false }, + ) + const observe = (promise: Promise | true) => { + const state: { settled: boolean; error?: unknown } = { settled: false } + const done = Promise.resolve(promise).then( + () => { + state.settled = true + }, + (error: unknown) => { + state.settled = true + state.error = error + }, + ) + return { state, done } + } + const preload = observe(live.preload()) + let move: ReturnType | undefined + let baseline: Array = [] + try { + if (scenario.barrier === `replay`) { + await preload.done + expect(preload.state).toEqual({ settled: true }) + expect(read()).toEqual(referenceWindow(1)) + baseline = read() + publications.length = 0 + allowTarget = true + for (let index = 0; index < truth.length; index++) + truth[index] = { ...truth[index]!, version: 2 } + activeInstalled.clear() + activeSync.begin() + activeSync.truncate() + activeSync.commit() + replayStarted = true + } + for (let turn = 0; turn < 8 && !target; turn++) await flushPromises() + expect( + target, + JSON.stringify({ + scenario, + requests: requests.map(({ ids, options }) => ({ + ids, + limit: options.limit, + ordered: options.orderBy !== undefined, + filtered: options.where !== undefined, + })), + }), + ).toBeDefined() + expect(target!.session).toBe(1) + expect(target!.ids.length).toBeGreaterThan(0) + await flushPromises() + if (scenario.barrier === `initial`) + expect(preload.state.settled).toBe(false) + expect(read()).toEqual(baseline) + check(`pending-window`, live.utils.getWindow(), { offset: 0, limit: 1 }) + expect(publications).toEqual([]) + expect(target!.applied).toBe(scenario.delivery === `before-settlement`) + appliedBeforeSettlement = target!.applied + if (scenario.delivery === `before-settlement`) { + // A replay peer may have installed the same rows already. The provider + // still completed this read; its whole selected subset must be present. + expect(target!.ids.every((id) => source.has(id))).toBe(true) + } else expect(targetWrites).toBe(0) + if (scenario.window === `widen`) { + move = observe(live.utils.setWindow({ offset: 0, limit: 3 })) + await flushPromises() + expect(move.state.settled).toBe(false) + check(`pending-move-window`, live.utils.getWindow(), { + offset: 0, + limit: 1, + }) + expect(publications).toEqual([]) + } + if (scenario.session === `restart`) { + allowTarget = false + await live.cleanup() + await source.cleanup() + expect(target!.options.signal?.aborted).toBe(true) + expect(sourceCleanups).toEqual([1]) + await preload.done + if (scenario.barrier === `initial`) + check( + `cleanup-preload`, + preload.state.error instanceof Error + ? preload.state.error.name + : `resolved`, + `AbortError`, + ) + if (move) { + await move.done + check( + `cleanup-window`, + move.state.error instanceof Error + ? move.state.error.name + : `resolved`, + `AbortError`, + ) + } + await live.preload() + expect(generation).toBe(2) + check(`restarted-window`, read(), referenceWindow(1)) + check(`restarted-window-options`, live.utils.getWindow(), { + offset: 0, + limit: 1, + }) + } + const prior = read() + const priorStatus = live.status + const priorError = live.utils.lastSubsetError + const callbacksBeforeSettlement = publications.length + if (scenario.outcome === `resolve`) gate.resolve() + else gate.reject(failure) + for (let turn = 0; turn < 8; turn++) await flushPromises() + expect(targetOutcome).toBe(scenario.outcome) + // Deferred application happens only after a live attempt succeeds. Failure + // and old-session success must not apply its rows through this provider. + expect(target!.applied).toBe( + scenario.delivery === `before-settlement` || + (scenario.outcome === `resolve` && scenario.session === `retain`), + ) + if (scenario.session === `restart`) { + check(`obsolete-status`, live.status, priorStatus) + check(`obsolete-error`, live.utils.lastSubsetError === priorError, true) + check(`obsolete-rows`, read(), prior) + check( + `obsolete-publication`, + publications.length, + callbacksBeforeSettlement, + ) + truth[0] = { ...truth[0]!, rank: truth[0]!.rank - 1 } + activeSync.begin() + activeSync.write({ type: `update`, value: truth[0] }) + activeSync.commit() + for (let turn = 0; turn < 4; turn++) await flushPromises() + check(`restart-reactivity`, read(), referenceWindow(1)) + check( + `restart-callback`, + publications.length, + callbacksBeforeSettlement + 1, + ) + } else if (scenario.outcome === `resolve`) { + check(`success-preload`, preload.state, { settled: true }) + check(`success-window-options`, live.utils.getWindow(), { + offset: 0, + limit: scenario.window === `widen` ? 3 : 1, + }) + if (move) check(`success-window`, move.state, { settled: true }) + check( + `success-rows`, + read(), + referenceWindow(scenario.window === `widen` ? 3 : 1), + ) + const finalWindow = referenceWindow(scenario.window === `widen` ? 3 : 1) + // A move queued behind replay may follow publication of the complete old + // window, or coalesce with it. Neither path may expose a partial window. + const legalPublications = [[finalWindow]] + if (scenario.barrier === `replay` && scenario.window === `widen`) { + legalPublications.push([referenceWindow(1), finalWindow]) + } + check( + `success-publication`, + legalPublications.some((trace) => + isDeepStrictEqual(publications, trace), + ) + ? `valid` + : publications, + `valid`, + ) + } else { + if (scenario.barrier === `initial`) + check( + `failure-preload`, + { + settled: preload.state.settled, + error: + preload.state.error === failure + ? `target` + : preload.state.error === undefined + ? `none` + : `other`, + }, + { settled: true, error: `target` }, + ) + else check(`replay-error`, live.utils.lastSubsetError === failure, true) + if (move) + check( + `failure-window`, + { + settled: move.state.settled, + error: + move.state.error === failure + ? `target` + : move.state.error === undefined + ? `none` + : `other`, + }, + { settled: true, error: `target` }, + ) + check(`failure-rows`, read(), baseline) + check(`failed-window-options`, live.utils.getWindow(), { + offset: 0, + limit: 1, + }) + check(`failure-publication`, publications, []) + } + } finally { + allowTarget = false + gate.resolve() + subscription.unsubscribe() + await live.cleanup() + await source.cleanup() + await preload.done + if (move) await move.done + } + expect(new Set(released).size).toBe(released.length) + for (const { options } of requests) + expect(released.filter((release) => release === options)).toHaveLength(1) + expect(sourceCleanups).toEqual(scenario.session === `restart` ? [1, 2] : [1]) + expect(requests.every(({ options }) => options.signal?.aborted)).toBe(true) + const route = + target!.options.orderBy !== undefined + ? target!.indexed + ? `page` + : `prefix` + : target!.options.where !== undefined + ? `boundary` + : `full-source` + return { + route, + authority: + target!.options.limit === undefined && target!.options.where === undefined + ? `full` + : `finite`, + generation, + coordinates: [ + route, + appliedBeforeSettlement ? `before-settlement` : `after-success`, + move ? `widen` : `keep`, + targetOutcome, + generation === 2 ? `restart` : `retain`, + replayStarted ? `replay` : `initial`, + ], + mismatches, + } +} + +async function assertHistory(scenario: Scenario) { + const result = await observeHistory(scenario) + expect(result.route).toBe(scenario.route) + expect(result.authority).toBe( + scenario.route === `full-source` ? `full` : `finite`, + ) + expect(result.generation).toBe(scenario.session === `restart` ? 2 : 1) + expect(result.mismatches).toEqual([]) + return result +} + +describe(`ordered lifecycle product`, () => { + const observed = new Set() + const cells: Array = routes.flatMap((route) => + ([`before-settlement`, `after-success`] as const).flatMap((delivery) => + ([`keep`, `widen`] as const).flatMap((window) => + ([`resolve`, `reject`, `abort-error`] as const).flatMap((outcome) => + ([`retain`, `restart`] as const).flatMap((session) => + ([`initial`, `replay`] as const).map((barrier) => ({ + route, + delivery, + window, + outcome, + session, + barrier, + })), + ), + ), + ), + ), + ) + it(`keeps all 192 declared histories distinct`, () => { + expect(cells).toHaveLength(192) + expect(new Set(cells.map((cell) => JSON.stringify(cell))).size).toBe(192) + }) + it.each(cells)( + `$route / $delivery / $window / $outcome / $session / $barrier`, + async (scenario) => { + const result = await assertHistory(scenario) + observed.add(JSON.stringify(result.coordinates)) + }, + ) + it(`reaches all 192 histories through physical work and terminal cleanup`, () => { + expect(observed.size).toBe(192) + }) + const arbitrary = fc.record({ + route: fc.constantFrom(...routes), + delivery: fc.constantFrom( + `before-settlement` as const, + `after-success` as const, + ), + window: fc.constantFrom(`keep` as const, `widen` as const), + outcome: fc.constantFrom( + `resolve` as const, + `reject` as const, + `abort-error` as const, + ), + session: fc.constantFrom(`retain` as const, `restart` as const), + barrier: fc.constantFrom(`initial` as const, `replay` as const), + rankOffset: fc.integer({ min: -1000, max: 1000 }), + rankStep: fc.integer({ min: 1, max: 10 }), + }) + const { multiplier, ...replay } = readOracleRunConfig() + fcTest.prop([arbitrary], { numRuns: 20 * multiplier, seed: 93471 })( + `matches the ordered lifecycle for a fixed seed`, + async (scenario) => { + await assertHistory(scenario) + }, + Math.max(10000, multiplier * 1500), + ) + fcTest.prop( + [arbitrary], + oracleRandomParameters(20 * multiplier, replay, `ordered-work.lifecycle`), + )( + `matches the ordered lifecycle for a random or replayed seed`, + async (scenario) => { + await assertHistory(scenario) + }, + Math.max(10000, multiplier * 1500), + ) +}) diff --git a/packages/db/tests/query/ordered-source-loader-state.test.ts b/packages/db/tests/query/ordered-source-loader-state.test.ts new file mode 100644 index 0000000000..4f1cc78e11 --- /dev/null +++ b/packages/db/tests/query/ordered-source-loader-state.test.ts @@ -0,0 +1,835 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createLiveQueryCollection } from '../../src/query/index.js' +import { OrderedSourceLoader } from '../../src/query/live/ordered-source-loader.js' +import { PropRef } from '../../src/query/ir.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { flushPromises } from '../utils.js' +import type { + CollectionSubscription, + ReleaseLoadSubset, +} from '../../src/collection/subscription.js' +import type { OrderByOptimizationInfo } from '../../src/query/compiler/order-by.js' +import type { + LoadSubsetOptions, + LoadSubsetRequestResult, +} from '../../src/types.js' + +type RequestOptions = LoadSubsetOptions & { + minValues?: Array + onLoadSubsetResult?: ( + result: LoadSubsetRequestResult, + acquisition: LoadSubsetOptions, + release: ReleaseLoadSubset, + ) => void +} + +function createDeferred() { + let resolve!: () => void + let reject!: (error: unknown) => void + const promise = new Promise((done, fail) => { + resolve = done + reject = fail + }) + return { promise, resolve, reject } +} + +function createOrderByInfo( + overrides: Partial = {}, +): OrderByOptimizationInfo { + return { + sourceId: `source`, + alias: `row`, + orderBy: [ + { + expression: new PropRef([`row`, `rank`]), + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `lexical`, + }, + }, + ], + offset: 0, + limit: 1, + comparator: (left, right) => + (left?.rank as number) - (right?.rank as number), + valueExtractorForRawRow: (row) => row.rank, + index: {} as NonNullable, + dataNeeded: () => 1, + requiresFullSource: false, + ...overrides, + } +} + +type Observed = { + method: `limited` | `snapshot` + options: RequestOptions + acquisition: LoadSubsetOptions + deferred: ReturnType +} + +function fakeSubscription( + requests: Array, + releases: Array, + onRelease: () => void = () => {}, +) { + const request = (method: Observed[`method`], options: RequestOptions) => { + const acquisition: LoadSubsetOptions = { + orderBy: options.orderBy, + limit: options.limit, + where: options.where, + } + const deferred = createDeferred() + requests.push({ method, options, acquisition, deferred }) + options.onLoadSubsetResult?.(deferred.promise, acquisition, () => { + releases.push(acquisition) + onRelease() + }) + } + return { + readOrderedSnapshot: () => [], + setOrderByIndex: () => {}, + requestLimitedSnapshot: (options: RequestOptions) => + request(`limited`, options), + requestSnapshot: (options: RequestOptions) => request(`snapshot`, options), + } as unknown as CollectionSubscription +} + +describe(`Ordered source request ownership`, () => { + it.each([`success`, `failure`] as const)( + `keeps a failed public window private when its older tie request ends in %s`, + async (olderOutcome) => { + type Row = { id: number; rank: number } + const truth: Array = [1, 2, 3].map((id) => ({ id, rank: id })) + const requests: Array<{ + kind: `page` | `boundary` | `full` + options: LoadSubsetOptions + gate: ReturnType + }> = [] + const releases: Array = [] + let hold = false + let update!: (row: Row) => void + const source = createCollection({ + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (sync) => { + const installed = new Map() + update = (row) => { + installed.set(row.id, row) + sync.begin() + sync.write({ type: `update`, value: row }) + sync.commit() + } + sync.markReady() + return { + loadSubset: async (options) => { + const kind = options.orderBy + ? `page` + : options.where + ? `boundary` + : `full` + const gate = createDeferred() + requests.push({ kind, options, gate }) + if (!hold || kind === `page`) gate.resolve() + await gate.promise + if (options.signal?.aborted) return + const rows = truth + .filter( + (row) => + (!options.where || + evaluateReferenceExpression(options.where, row) === + true) && + (!options.cursor || + evaluateReferenceExpression( + options.cursor.whereFrom, + row, + ) === true), + ) + .sort((a, b) => a.rank - b.rank) + const offset = options.cursor ? 0 : (options.offset ?? 0) + const selected = rows.slice( + offset, + options.limit === undefined + ? undefined + : offset + options.limit, + ) + sync.begin() + for (const row of selected) { + if (installed.get(row.id) === row) continue + sync.write({ + type: installed.has(row.id) ? `update` : `insert`, + value: row, + }) + installed.set(row.id, row) + } + await sync.commit() + }, + unloadSubset: (options) => { + releases.push(options) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1) + .select(({ row }) => ({ id: row.id, rank: row.rank })), + ) + const visibleRows = () => + live.toArray.map(({ id, rank }) => ({ id, rank })) + const publications: Array> = [] + live.subscribeChanges( + (batch) => { + if (batch.length) publications.push(visibleRows()) + }, + { includeInitialState: false }, + ) + try { + await live.preload() + expect(visibleRows()).toEqual([{ id: 1, rank: 1 }]) + publications.length = 0 + hold = true + const move = Promise.resolve(live.utils.setWindow({ limit: 2 })).then( + () => ({ status: `fulfilled` as const }), + (error) => ({ status: `rejected` as const, error }), + ) + await flushPromises() + const boundary = requests.at(-1)! + expect(boundary.kind).toBe(`boundary`) + truth[0] = { id: 1, rank: 10 } + update(truth[0]) + await flushPromises() + const full = requests.at(-1)! + expect(full.kind).toBe(`full`) + const count = requests.length + const failure = new Error(`authoritative repair failed`) + full.gate.reject(failure) + // The window still waits for its older publication participant to settle. + await flushPromises() + if (olderOutcome === `failure`) + boundary.gate.reject(new Error(`older tie failed`)) + else boundary.gate.resolve() + await flushPromises() + expect(requests).toHaveLength(count) + expect(await move).toEqual({ status: `rejected`, error: failure }) + expect(releases).toEqual([]) + expect(publications).toEqual([]) + expect(visibleRows()).toEqual([{ id: 1, rank: 1 }]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + + hold = false + await live.utils.setWindow({ limit: 2 }) + expect(requests).toHaveLength(count + 1) + expect(releases).toHaveLength(count) + expect(new Set(releases)).toEqual( + new Set(requests.slice(0, count).map(({ options }) => options)), + ) + expect(visibleRows()).toEqual([ + { id: 2, rank: 2 }, + { id: 3, rank: 3 }, + ]) + expect(publications).toEqual([ + [ + { id: 2, rank: 2 }, + { id: 3, rank: 3 }, + ], + ]) + } finally { + await live.cleanup() + for (const request of requests) request.gate.resolve() + await source.cleanup() + } + }, + ) + + // Finite success is not authority to repair a newer full-source failure. + // Cross request kind with settlement order instead of testing each alone. + it.each( + ([`page`, `boundary`] as const).flatMap((olderKind) => + ([`older-first`, `full-first`] as const).flatMap((order) => + ([`success`, `failure`] as const).flatMap((outcome) => + ([`success`, `failure`] as const).map((olderOutcome) => ({ + olderKind, + order, + outcome, + olderOutcome, + })), + ), + ), + ), + )( + `keeps full-source recovery authoritative across overlap: %j`, + async ({ olderKind, order, outcome, olderOutcome }) => { + const requests: Array = [] + const releases: Array = [] + const participants: Array> = [] + const subscription = fakeSubscription(requests, releases) + subscription.readOrderedSnapshot = () => [ + { type: `insert`, key: 1, value: { id: 1, rank: 1 } }, + ] + const loader = new OrderedSourceLoader( + createOrderByInfo(), + subscription, + `row`, + (result) => { + if (result instanceof Promise) participants.push(result) + }, + ) + try { + loader.start() + if (olderKind === `boundary`) { + requests[0]!.deferred.resolve() + await participants[0] + expect(requests).toHaveLength(2) + expect(requests[1]!.options.where).toBeDefined() + } + const olderIndex = requests.length - 1 + loader.invalidateSourceOrdering() + loader.loadMore() + const fullIndex = olderIndex + 1 + const count = fullIndex + 1 + expect(requests).toHaveLength(count) + expect(requests[fullIndex]!.options.orderBy).toBeUndefined() + expect(requests[fullIndex]!.options.where).toBeUndefined() + const failure = new Error(`full-source failed`) + const olderFailure = new Error(`older request failed`) + const settleOlder = async () => { + if (olderOutcome === `failure`) { + requests[olderIndex]!.deferred.reject(olderFailure) + await expect(participants[olderIndex]).rejects.toBe(olderFailure) + } else { + requests[olderIndex]!.deferred.resolve() + await participants[olderIndex] + } + expect(requests).toHaveLength(count) + } + const settleFull = async () => { + if (outcome === `failure`) { + requests[fullIndex]!.deferred.reject(failure) + await expect(participants[fullIndex]).rejects.toBe(failure) + } else { + requests[fullIndex]!.deferred.resolve() + await participants[fullIndex] + } + expect(requests).toHaveLength(count) + } + if (order === `older-first`) { + await settleOlder() + await settleFull() + } else { + await settleFull() + await settleOlder() + } + loader.loadMore() + expect(requests).toHaveLength(count) + const successfulFinite = + outcome === `success` + ? requests + .slice(0, fullIndex) + .filter( + (_, index) => + index !== olderIndex || olderOutcome === `success`, + ) + .map(({ acquisition }) => acquisition) + : [] + expect(releases).toEqual(successfulFinite) + + loader.loadMore(1) + const failedIndices = ( + order === `older-first` + ? [olderIndex, fullIndex] + : [fullIndex, olderIndex] + ).filter( + (index) => + (index === olderIndex ? olderOutcome : outcome) === `failure`, + ) + expect(releases).toEqual([ + ...successfulFinite, + ...failedIndices.map((index) => requests[index]!.acquisition), + ]) + if (outcome === `failure`) { + expect(requests).toHaveLength(count + 1) + loader.loadMore(1) + expect(requests).toHaveLength(count + 1) + requests[count]!.deferred.resolve() + await participants[count] + } else expect(requests).toHaveLength(count) + } finally { + loader.dispose() + for (const request of requests) request.deferred.resolve() + await Promise.allSettled(participants) + } + }, + ) + + it.each( + [false, true].flatMap((fullFirst) => + [false, true].flatMap((replayed) => + [false, true].map((releaseThrows) => ({ + fullFirst, + replayed, + releaseThrows, + })), + ), + ), + )( + `retains each failed release while replay repairs only full-source work: %j`, + async ({ fullFirst, replayed, releaseThrows }) => { + const requests: Array = [] + const releases: Array = [] + const participants: Array> = [] + const cleanupError = new Error(`release failed`) + const loader = new OrderedSourceLoader( + createOrderByInfo(), + fakeSubscription(requests, releases, () => { + if (releaseThrows) throw cleanupError + }), + `row`, + (result) => { + if (result instanceof Promise) participants.push(result) + }, + ) + try { + loader.start() + loader.invalidateSourceOrdering() + loader.loadMore() + expect(requests).toHaveLength(2) + const order = fullFirst ? [1, 0] : [0, 1] + for (const index of order) { + const failure = new Error(`request ${index} failed`) + requests[index]!.deferred.reject(failure) + await expect(participants[index]).rejects.toBe(failure) + } + if (replayed) loader.settleFullSourceReplay() + if (releaseThrows) + expect(() => loader.loadMore(1)).toThrow(cleanupError) + else loader.loadMore(1) + const expected = order + .filter((index) => !replayed || index === 0) + .map((index) => requests[index]!.acquisition) + expect(releases).toEqual(expected) + loader.loadMore(2) + expect(releases).toEqual(expected) + expect(requests).toHaveLength(replayed ? 2 : 3) + } finally { + loader.dispose() + for (const request of requests) request.deferred.resolve() + await Promise.allSettled(participants) + } + }, + ) + + it(`a page failure while a full-source demand is held releases only the page`, async () => { + const requests: Array = [] + const releases: Array = [] + const loader = new OrderedSourceLoader( + createOrderByInfo(), + fakeSubscription(requests, releases), + `row`, + ) + loader.start() + const page = ( + loader as unknown as { pending: Promise | undefined } + ).pending! + expect(requests.map(({ method }) => method)).toEqual([`limited`]) + + // A delete during the in-flight page requires authoritative repair. + loader.invalidateSourceOrdering() + loader.loadMore() + expect(requests.map(({ method }) => method)).toEqual([ + `limited`, + `snapshot`, + ]) + expect(requests[1]!.options.limit).toBeUndefined() + const fullSource = ( + loader as unknown as { pending: Promise | undefined } + ).pending! + expect(fullSource).not.toBe(page) + + const failure = new Error(`page rejected`) + requests[0]!.deferred.reject(failure) + await expect(page).rejects.toBe(failure) + + // Blocked automatic retry; explicit retry releases the page only and must + // not issue a duplicate full-source demand while one is already held. + expect(loader.loadMore()).toBe(fullSource) + expect(requests).toHaveLength(2) + expect(releases).toEqual([]) + expect(loader.loadMore(1)).toBe(fullSource) + expect(releases).toEqual([requests[0]!.acquisition]) + expect(requests).toHaveLength(2) + + requests[1]!.deferred.resolve() + await fullSource + expect(loader.loadMore(2)).toBeUndefined() + expect(requests).toHaveLength(2) + expect(releases).toEqual([requests[0]!.acquisition]) + loader.dispose() + }) + + it(`sync full-source failure retains no demand: replay settle is a no-op and retry reissues once`, () => { + const releases: Array = [] + const methods: Array = [] + const failure = new Error(`full-source threw after callback`) + const acquisition: LoadSubsetOptions = {} + let fail = true + const subscription = { + setOrderByIndex: () => {}, + requestSnapshot: (options: RequestOptions) => { + methods.push(`snapshot`) + if (!fail) return + fail = false + options.onLoadSubsetResult?.(true, acquisition, () => + releases.push(acquisition), + ) + throw failure + }, + } as unknown as CollectionSubscription + const loader = new OrderedSourceLoader( + createOrderByInfo({ requiresFullSource: true }), + subscription, + `row`, + ) + expect(() => loader.start()).toThrow(failure) + expect(releases).toEqual([acquisition]) + expect(methods).toEqual([`snapshot`]) + + loader.settleFullSourceReplay() + expect(loader.loadMore()).toBeUndefined() + expect(methods).toEqual([`snapshot`]) + + loader.loadMore(1) + expect(methods).toEqual([`snapshot`, `snapshot`]) + expect(releases).toEqual([acquisition]) + loader.dispose() + }) + + it(`replay repairs a failed full-source demand: a later explicit retry releases nothing`, async () => { + const requests: Array = [] + const releases: Array = [] + const loader = new OrderedSourceLoader( + createOrderByInfo({ requiresFullSource: true }), + fakeSubscription(requests, releases), + `row`, + ) + loader.start() + const pending = ( + loader as unknown as { pending: Promise | undefined } + ).pending! + const failure = new Error(`full-source rejected`) + requests[0]!.deferred.reject(failure) + await expect(pending).rejects.toBe(failure) + expect(requests).toHaveLength(1) + + loader.settleFullSourceReplay() + expect(loader.loadMore(1)).toBeUndefined() + expect(releases).toEqual([]) + expect(requests).toHaveLength(1) + expect(loader.loadMore(2)).toBeUndefined() + expect(requests).toHaveLength(1) + loader.dispose() + }) + + it(`without replay the explicit retry releases and reissues exactly once`, async () => { + const requests: Array = [] + const releases: Array = [] + const loader = new OrderedSourceLoader( + createOrderByInfo({ requiresFullSource: true }), + fakeSubscription(requests, releases), + `row`, + ) + loader.start() + const pending = ( + loader as unknown as { pending: Promise | undefined } + ).pending! + requests[0]!.deferred.reject(new Error(`full-source rejected`)) + await expect(pending).rejects.toThrow(`full-source rejected`) + + loader.loadMore(1) + expect(releases).toEqual([requests[0]!.acquisition]) + expect(requests).toHaveLength(2) + loader.loadMore(1) + expect(requests).toHaveLength(2) + loader.dispose() + }) + + it(`a zero window opening with an offset requests the whole prefix from zero`, () => { + const requests: Array = [] + const releases: Array = [] + const info = createOrderByInfo({ offset: 2, limit: 0 }) + // Production dataNeeded is limit - topK size; it never adds the offset. + info.dataNeeded = () => info.limit + const loader = new OrderedSourceLoader( + info, + fakeSubscription(requests, releases), + `row`, + ) + loader.start() + expect(requests).toHaveLength(0) + + info.limit = 3 + loader.loadMore(1) + expect(requests).toHaveLength(1) + expect(requests[0]!.method).toBe(`limited`) + expect(requests[0]!.options.limit).toBe(5) + expect(requests[0]!.options.offset).toBe(0) + expect(requests[0]!.options.minValues).toBeUndefined() + loader.dispose() + }) + + it(`a failed window move keeps the snapshot; the retry loads the source once`, async () => { + type Row = { id: number; rank: number } + const truth: Array = [1, 2, 3, 4, 5, 6].map((id) => ({ id, rank: id })) + const failure = new Error(`page rejected`) + const requests: Array<{ kind: string; options: LoadSubsetOptions }> = [] + const unloads: Array = [] + let failNextCursor = false + const kindOf = (options: LoadSubsetOptions) => + options.orderBy !== undefined + ? options.cursor + ? `cursor-page` + : `page` + : options.where !== undefined + ? `boundary` + : `full` + const source = createCollection({ + id: `cut-c-probe-source`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (sync) => { + const installed = new Set() + sync.markReady() + return { + loadSubset: (options) => { + requests.push({ kind: kindOf(options), options }) + if (options.cursor && failNextCursor) { + failNextCursor = false + return Promise.reject(failure) + } + let rows = truth.filter( + (row) => + !options.where || + evaluateReferenceExpression(options.where, row) === true, + ) + if (options.cursor) + rows = rows.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + const offset = options.cursor ? 0 : (options.offset ?? 0) + rows = rows.slice( + offset, + options.limit === undefined + ? undefined + : offset + options.limit, + ) + return (async () => { + await Promise.resolve() + const fresh = rows.filter(({ id }) => !installed.has(id)) + if (fresh.length === 0) return + sync.begin() + for (const value of fresh) { + installed.add(value.id) + sync.write({ type: `insert`, value }) + } + const receipt = sync.commit() + if (receipt !== true) await receipt + })() + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2) + .select(({ row }) => ({ id: row.id, rank: row.rank })), + ) + const publications: Array> = [] + live.subscribeChanges( + (batch) => { + if (batch.length > 0) + publications.push(live.toArray.map(({ id }) => id)) + }, + { includeInitialState: false }, + ) + try { + await live.preload() + expect(live.toArray.map(({ id }) => id)).toEqual([1, 2]) + expect(requests.map(({ kind }) => kind)).toEqual([`page`, `boundary`]) + expect(unloads).toEqual([]) + const initialRequests = requests.length + publications.length = 0 + + failNextCursor = true + const move = live.utils.setWindow({ limit: 4 }) + expect(move).not.toBe(true) + await expect(move).rejects.toBe(failure) + await flushPromises() + // Rows: last settled snapshot; events: none; wait: rejected; requests: one. + expect(live.toArray.map(({ id }) => id)).toEqual([1, 2]) + expect(publications).toEqual([]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(requests.slice(initialRequests).map(({ kind }) => kind)).toEqual([ + `cursor-page`, + ]) + expect(unloads).toEqual([]) + expect(live.status).not.toBe(`error`) + + const retry = live.utils.setWindow({ limit: 4 }) + expect(retry).not.toBe(true) + await retry + await flushPromises() + expect(live.toArray.map(({ id }) => id)).toEqual([1, 2, 3, 4]) + expect(publications).toEqual([[1, 2, 3, 4]]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) + expect(requests.slice(initialRequests).map(({ kind }) => kind)).toEqual([ + `cursor-page`, + `full`, + ]) + // Retry retires the failed page; successful repair then retires earlier + // finite demands while keeping the new authoritative demand. + expect(unloads).toEqual([ + requests[initialRequests]!.options, + ...requests.slice(0, initialRequests).map(({ options }) => options), + ]) + + await flushPromises() + expect(requests).toHaveLength(initialRequests + 2) + } finally { + await live.cleanup() + await source.cleanup() + } + }) +}) + +describe(`Successful finite demand retirement`, () => { + it.each([`none`, `throw`, `truncate`, `dispose`] as const)( + `attempts releases once across %s reentry`, + async (action) => { + const requests: Array = [] + const releases: Array = [] + const participants: Array> = [] + const failure = new Error(`release failed`) + let first = true + const subscription = fakeSubscription(requests, releases, () => { + if (!first) return + first = false + if (action === `throw`) throw failure + if (action === `truncate`) loader.resetCursor() + if (action === `dispose`) loader.dispose() + }) + subscription.readOrderedSnapshot = () => [ + { type: `insert`, key: 1, value: { id: 1, rank: 1 } }, + ] + const loader = new OrderedSourceLoader( + createOrderByInfo({ dataNeeded: () => 0 }), + subscription, + `row`, + (result) => { + if (result instanceof Promise) participants.push(result) + }, + ) + try { + loader.start() + requests[0]!.deferred.resolve() + await participants[0] + requests[1]!.deferred.resolve() + await participants[1] + loader.invalidateSourceOrdering() + loader.loadMore() + requests[2]!.deferred.resolve() + if (action === `throw`) + await expect(participants[2]).rejects.toBe(failure) + else await participants[2] + expect(releases).toEqual( + requests + .slice(0, action === `truncate` || action === `dispose` ? 1 : 2) + .map(({ acquisition }) => acquisition), + ) + if (action === `truncate`) { + loader.settleFullSourceReplay() + expect(releases).toEqual( + requests.slice(0, 2).map(({ acquisition }) => acquisition), + ) + } + loader.loadMore(1) + expect(new Set(releases).size).toBe(releases.length) + expect(requests).toHaveLength(3) + } finally { + loader.dispose() + for (const request of requests) request.deferred.resolve() + await Promise.allSettled(participants) + } + }, + ) + + it.each([`before-replay`, `after-replay`] as const)( + `keeps unfinished physical work observed when it settles %s`, + async (when) => { + const requests: Array = [] + const releases: Array = [] + const participants: Array> = [] + const subscription = fakeSubscription(requests, releases) + let replaying = false + Object.defineProperty(subscription, `hasPendingTruncateReplacement`, { + get: () => replaying, + }) + const loader = new OrderedSourceLoader( + createOrderByInfo(), + subscription, + `row`, + (result) => { + if (result instanceof Promise) participants.push(result) + }, + ) + try { + loader.start() + replaying = true + loader.resetCursor() + loader.loadFullSource() + requests[1]!.deferred.resolve() + await participants[1] + expect(releases).toEqual([]) + const settlePage = async () => { + requests[0]!.deferred.resolve() + await participants[0] + } + if (when === `before-replay`) { + await settlePage() + expect(releases).toEqual([]) + } + replaying = false + loader.settleFullSourceReplay() + if (when === `after-replay`) { + expect(releases).toEqual([]) + await settlePage() + } + expect(releases).toEqual([requests[0]!.acquisition]) + } finally { + loader.dispose() + for (const request of requests) request.deferred.resolve() + await Promise.allSettled(participants) + } + }, + ) +}) diff --git a/packages/db/tests/query/ordered-source-loader.test.ts b/packages/db/tests/query/ordered-source-loader.test.ts new file mode 100644 index 0000000000..c08093a0db --- /dev/null +++ b/packages/db/tests/query/ordered-source-loader.test.ts @@ -0,0 +1,979 @@ +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { OrderedSourceLoader } from '../../src/query/live/ordered-source-loader.js' +import { Func, PropRef, Value } from '../../src/query/ir.js' +import type { + CollectionSubscription, + ReleaseLoadSubset, +} from '../../src/collection/subscription.js' +import type { OrderByOptimizationInfo } from '../../src/query/compiler/order-by.js' +import type { + LoadSubsetOptions, + LoadSubsetRequestResult, +} from '../../src/types.js' + +const pendingPromise = (loader: OrderedSourceLoader) => + (loader as unknown as { pending: Promise | undefined }).pending + +type RequestOptions = LoadSubsetOptions & { + minValues?: Array + onLoadSubsetResult?: ( + result: LoadSubsetRequestResult, + acquisition: LoadSubsetOptions, + release: ReleaseLoadSubset, + ) => void +} + +function createDeferred() { + let resolve!: () => void + let reject!: (error: unknown) => void + const promise = new Promise((done, fail) => { + resolve = done + reject = fail + }) + return { promise, resolve, reject } +} + +function createOrderByInfo( + overrides: Partial = {}, +): OrderByOptimizationInfo { + return { + sourceId: `source`, + alias: `row`, + orderBy: [ + { + expression: new PropRef([`row`, `rank`]), + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `lexical`, + }, + }, + ], + offset: 0, + limit: 1, + comparator: (left, right) => + (left?.rank as number) - (right?.rank as number), + valueExtractorForRawRow: (row) => row.rank, + index: {} as NonNullable, + dataNeeded: () => 1, + requiresFullSource: false, + ...overrides, + } +} + +describe(`OrderedSourceLoader`, () => { + it(`settles a larger prefix after an older lease release throws`, async () => { + const failure = new Error(`old prefix release failed`) + const requests: Array = [] + const source = createCollection<{ id: number; rank: number }>({ + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (let id = 1; id <= 3; id++) + write({ type: `insert`, value: { id, rank: id } }) + commit() + markReady() + return { + loadSubset: (options) => { + requests.push(options) + return Promise.resolve() + }, + unloadSubset: (options) => { + if (options === requests[0]) throw failure + }, + } + }, + }, + }) + const subscription = source.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const reads = vi.spyOn(subscription, `readOrderedSnapshot`) + const info = createOrderByInfo({ index: undefined, dataNeeded: () => 0 }) + const loader = new OrderedSourceLoader( + info, + subscription as unknown as CollectionSubscription, + `row`, + ) + try { + loader.start() + await pendingPromise(loader) + reads.mockClear() + info.limit = 2 + await expect(loader.loadMore(1)).rejects.toBe(failure) + expect(reads).toHaveBeenCalledWith(expect.objectContaining({ limit: 2 })) + expect(subscription.lastError).toBe(failure) + info.limit = 3 + await loader.loadMore(2) + expect(requests.map((request) => request.limit)).toEqual([ + 1, + undefined, + 2, + undefined, + 3, + undefined, + ]) + expect( + requests + .filter((request) => request.limit === undefined) + .every((request) => request.where !== undefined), + ).toBe(true) + expect(source.size).toBe(3) + } finally { + loader.dispose() + subscription.unsubscribe() + await source.cleanup() + } + }) + + const syncRouteCells = ( + [`page`, `prefix`, `boundary`, `full-source`] as const + ).flatMap((route) => + ([`success`, `throw`, `callback-then-throw`] as const).map((outcome) => ({ + route, + outcome, + })), + ) + + it.each(syncRouteCells)( + `preserves $route request semantics with synchronous $outcome`, + async ({ route, outcome }) => { + const requests: Array<{ method: string; options: RequestOptions }> = [] + const released: Array = [] + const failure = new Error(`target request failed`) + const waiting = createDeferred() + let boundaryReads = 0 + const targetIndex = route === `boundary` ? 1 : 0 + const request = (method: string, options: RequestOptions) => { + const index = requests.length + requests.push({ method, options }) + if (index !== targetIndex) { + // Bootstrap the boundary case; leave later refinement/retry in flight. + options.onLoadSubsetResult?.( + index < targetIndex ? true : waiting.promise, + options, + () => {}, + ) + return + } + if (outcome !== `throw`) { + options.onLoadSubsetResult?.(true, options, () => + released.push(options), + ) + } + if (outcome !== `success`) throw failure + } + const subscription = { + setOrderByIndex: () => {}, + readOrderedSnapshot: () => { + boundaryReads++ + return [{ value: { rank: 1 } }] + }, + requestLimitedSnapshot: (options: RequestOptions) => + request(`limited`, options), + requestSnapshot: (options: RequestOptions) => + request(`snapshot`, options), + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ + dataNeeded: () => 0, + ...(route === `prefix` ? { index: undefined } : {}), + requiresFullSource: route === `full-source`, + }), + subscription as unknown as CollectionSubscription, + `row`, + ) + try { + if (outcome !== `success` && route !== `boundary`) { + expect(() => loader.start()).toThrow(failure) + } else { + loader.start() + if (outcome === `success`) await pendingPromise(loader) + else await expect(pendingPromise(loader)).rejects.toBe(failure) + } + // Drain the synchronous boundary's own settlement as well as its parent. + await Promise.resolve() + const target = requests[targetIndex]! + expect(target.method).toBe(route === `page` ? `limited` : `snapshot`) + expect(target.options.limit).toBe( + route === `page` || route === `prefix` ? 1 : undefined, + ) + expect(Boolean(target.options.where)).toBe(route === `boundary`) + expect(released).toEqual( + outcome === `callback-then-throw` ? [target.options] : [], + ) + if (outcome === `success`) { + // Ordered loads establish a cursor and refine ties; neither a tie + // load nor a full-source load may restart that refinement step. + expect(boundaryReads).toBe(route === `full-source` ? 0 : 1) + expect(requests).toHaveLength(route === `full-source` ? 1 : 2) + if (route === `page` || route === `prefix`) { + expect(requests[1]!.options.where).toBeDefined() + expect(requests[1]!.options.orderBy).toBeUndefined() + } + } else { + const count = requests.length + loader.loadMore() + expect(requests).toHaveLength(count) + loader.loadMore(1) + expect(requests).toHaveLength(count + 1) + const retry = requests.at(-1)! + expect(retry.method).toBe(`snapshot`) + expect(retry.options.orderBy).toBeUndefined() + expect(retry.options.where).toBeUndefined() + expect(retry.options.limit).toBeUndefined() + } + } finally { + loader.dispose() + waiting.resolve() + await Promise.resolve() + } + }, + ) + + it(`recovers authoritatively when reading a settled boundary fails`, async () => { + const failure = new Error(`boundary read failed`) + const requests: Array<{ method: string; options: RequestOptions }> = [] + const released: Array = [] + const request = (method: string, options: RequestOptions) => { + requests.push({ method, options }) + options.onLoadSubsetResult?.(Promise.resolve(), options, () => + released.push(options), + ) + } + const subscription = { + setOrderByIndex: () => {}, + readOrderedSnapshot: () => { + throw failure + }, + requestLimitedSnapshot: (options: RequestOptions) => + request(`limited`, options), + requestSnapshot: (options: RequestOptions) => + request(`snapshot`, options), + } + const loader = new OrderedSourceLoader( + createOrderByInfo(), + subscription as unknown as CollectionSubscription, + `row`, + ) + loader.start() + await expect(pendingPromise(loader)).rejects.toBe(failure) + loader.loadMore() + expect(requests).toHaveLength(1) + await loader.loadMore(1) + expect(released).toEqual([requests[0]!.options]) + expect(requests.map(({ method }) => method)).toEqual([ + `limited`, + `snapshot`, + ]) + expect(requests[1]!.options.limit).toBeUndefined() + loader.dispose() + }) + + const asyncRouteCells = ( + [`page`, `prefix`, `boundary`, `full-source`] as const + ).flatMap((route) => + ( + [ + `resolve`, + `reject`, + `abort`, + `dispose-resolve`, + `dispose-reject`, + ] as const + ).map((outcome) => ({ route, outcome })), + ) + + it.each(asyncRouteCells)( + `keeps the $route acquisition lifecycle exact for $outcome`, + async ({ route, outcome }) => { + type ObservedRequest = { + method: `limited` | `snapshot` + options: RequestOptions + acquisition: LoadSubsetOptions + controller: AbortController + deferred: ReturnType + } + const requests: Array = [] + const releases: Array = [] + const request = ( + method: ObservedRequest[`method`], + options: RequestOptions, + ) => { + const controller = new AbortController() + const acquisition: LoadSubsetOptions = { + signal: controller.signal, + orderBy: options.orderBy, + limit: options.limit, + } + const deferred = createDeferred() + requests.push({ method, options, acquisition, controller, deferred }) + options.onLoadSubsetResult?.(deferred.promise, acquisition, () => + releases.push(acquisition), + ) + } + const subscription = { + readOrderedSnapshot: () => + route === `boundary` ? [{ value: { rank: 1 } }] : [], + setOrderByIndex: () => {}, + requestLimitedSnapshot: (options: RequestOptions) => + request(`limited`, options), + requestSnapshot: (options: RequestOptions) => + request(`snapshot`, options), + } + const info = createOrderByInfo( + route === `prefix` + ? { index: undefined } + : route === `full-source` + ? { requiresFullSource: true } + : {}, + ) + const loader = new OrderedSourceLoader( + info, + subscription as unknown as CollectionSubscription, + `row`, + ) + + loader.start() + if (route === `boundary`) { + expect(requests.map(({ method }) => method)).toEqual([`limited`]) + requests[0]!.deferred.resolve() + await Promise.resolve() + await Promise.resolve() + expect(requests.map(({ method }) => method)).toEqual([ + `limited`, + `snapshot`, + ]) + } + const target = requests.at(-1)! + const targetSettlement = pendingPromise(loader)! + const failure = + outcome === `abort` + ? new DOMException(`${route} canceled`, `AbortError`) + : new Error(`${route} rejected`) + + if (outcome === `dispose-resolve` || outcome === `dispose-reject`) { + loader.dispose() + if (outcome === `dispose-resolve`) target.deferred.resolve() + else target.deferred.reject(failure) + await targetSettlement + expect(requests.at(-1)).toBe(target) + expect(releases).toEqual([]) + return + } + + if (outcome === `resolve`) { + target.deferred.resolve() + await targetSettlement + expect(target.controller.signal.aborted).toBe(false) + expect(releases).toEqual([]) + } else { + if (outcome === `abort`) target.controller.abort() + target.deferred.reject(failure) + await expect(targetSettlement).rejects.toBe(failure) + expect(target.controller.signal.aborted).toBe(outcome === `abort`) + + const requestCount = requests.length + expect(loader.loadMore()).toBeUndefined() + expect(requests).toHaveLength(requestCount) + + loader.loadMore(1) + expect(releases).toEqual([target.acquisition]) + expect(requests).toHaveLength(requestCount + 1) + const retry = requests.at(-1)! + expect(retry.method).toBe(`snapshot`) + retry.deferred.resolve() + await pendingPromise(loader) + expect(releases).toEqual([ + target.acquisition, + ...(route === `boundary` ? [requests[0]!.acquisition] : []), + ]) + } + + loader.dispose() + }, + ) + + it.each( + ([`reset`, `dispose`] as const).flatMap((lifecycle) => + ([`resolve`, `reject`, `abort`] as const).map((outcome) => ({ + lifecycle, + outcome, + })), + ), + )( + `preserves replacement ownership after $lifecycle and obsolete $outcome`, + async ({ lifecycle, outcome }) => { + const requests: Array<{ + method: string + options: RequestOptions + deferred: ReturnType + }> = [] + const releases: Array = [] + const request = (method: string, options: RequestOptions) => { + const deferred = createDeferred() + requests.push({ method, options, deferred }) + options.onLoadSubsetResult?.(deferred.promise, options, () => + releases.push(options), + ) + } + const subscription = { + setOrderByIndex: () => {}, + readOrderedSnapshot: () => [], + requestLimitedSnapshot: (options: RequestOptions) => + request(`page`, options), + requestSnapshot: (options: RequestOptions) => + request(`full-source`, options), + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ dataNeeded: () => 0 }), + subscription as unknown as CollectionSubscription, + `row`, + ) + try { + loader.start() + const obsolete = pendingPromise(loader)! + if (lifecycle === `reset`) loader.resetCursor() + else loader.dispose() + const replacement = loader.loadMore(1) + expect(requests.map(({ method }) => method)).toEqual( + lifecycle === `reset` ? [`page`, `page`] : [`page`], + ) + if (lifecycle === `reset`) { + expect(replacement).toBeInstanceOf(Promise) + expect(requests[1]!.options.offset).toBe(0) + expect(requests[1]!.options.minValues).toBeUndefined() + } + + if (outcome === `resolve`) requests[0]!.deferred.resolve() + else { + requests[0]!.deferred.reject( + outcome === `abort` + ? new DOMException(`obsolete request canceled`, `AbortError`) + : new Error(`obsolete request failed`), + ) + } + await obsolete + expect(pendingPromise(loader)).toBe(replacement) + expect(releases).toEqual([]) + + if (lifecycle === `reset`) { + requests[1]!.deferred.resolve() + await replacement + // A successful finite replacement cannot prove that partial writes + // from the obsolete failure were repaired. Success and repair debt + // coexist; only an authoritative full-source request clears it. + loader.loadMore(2) + expect(requests.map(({ method }) => method)).toEqual( + outcome === `resolve` + ? [`page`, `page`] + : [`page`, `page`, `full-source`], + ) + if (outcome !== `resolve`) { + expect(requests[2]!.options.orderBy).toBeUndefined() + expect(requests[2]!.options.limit).toBeUndefined() + requests[2]!.deferred.resolve() + await pendingPromise(loader) + loader.loadMore(3) + expect(requests).toHaveLength(3) + } + } else { + expect(loader.loadMore(2)).toBeUndefined() + expect(requests).toHaveLength(1) + } + } finally { + loader.dispose() + requests.forEach(({ deferred }) => deferred.resolve()) + } + }, + ) + + it.each([ + { label: `undefined`, value: undefined, continuation: `full-source` }, + { label: `null`, value: null, continuation: `full-source` }, + { label: `zero`, value: 0, continuation: `tie` }, + { label: `false`, value: false, continuation: `tie` }, + { label: `empty string`, value: ``, continuation: `tie` }, + ])( + `uses $continuation for a $label boundary`, + async ({ value, continuation }) => { + const methods: Array = [] + let needed = 0 + const request = (method: string, options: RequestOptions) => { + methods.push(method) + options.onLoadSubsetResult?.(true, options, () => {}) + } + const subscription = { + setOrderByIndex: () => {}, + readOrderedSnapshot: () => [{ value: { rank: value } }], + requestLimitedSnapshot: (options: RequestOptions) => + request(`page`, options), + requestSnapshot: (options: RequestOptions) => { + const kind = options.where ? `tie` : `full-source` + expect(kind).toBe(continuation) + request(kind, options) + }, + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ dataNeeded: () => needed, comparator: () => 0 }), + subscription as unknown as CollectionSubscription, + `row`, + ) + + loader.start() + await pendingPromise(loader) + await pendingPromise(loader) + expect(methods).toEqual([`page`, continuation]) + + needed = 2 + loader.loadMore(1) + await pendingPromise(loader) + expect(methods).toEqual( + continuation === `tie` + ? [`page`, `tie`, `page`] + : [`page`, `full-source`], + ) + loader.dispose() + }, + ) + + it(`retains only bounded promise state during a long refinement chain`, async () => { + let biggest: { rank: number } | undefined + const requests: Array> = [] + const tracked: Array<{ settled: boolean }> = [] + const request = (options: RequestOptions) => { + const next = createDeferred() + requests.push(next) + options.onLoadSubsetResult?.( + next.promise, + { + orderBy: options.orderBy, + limit: options.limit, + }, + () => {}, + ) + } + const subscription = { + readOrderedSnapshot: () => (biggest ? [{ value: biggest }] : []), + setOrderByIndex: () => {}, + requestLimitedSnapshot: request, + requestSnapshot: request, + } + const info = createOrderByInfo() + const loader = new OrderedSourceLoader( + info, + subscription as unknown as CollectionSubscription, + `row`, + (promise) => { + if (!(promise instanceof Promise)) return + const participant = { settled: false } + tracked.push(participant) + void promise.then( + () => { + participant.settled = true + }, + () => { + participant.settled = true + }, + ) + }, + ) + + loader.start() + for (let step = 0; step < 20; step++) { + expect(requests[step]).toBeDefined() + if (step % 2 === 0) biggest = { rank: step / 2 } + requests[step]!.resolve() + await Promise.resolve() + await Promise.resolve() + await Promise.resolve() + } + + // One request is active and its predecessor may still be settling during + // the handoff. Earlier ancestors must already be collectible. + expect( + tracked.filter(({ settled }) => !settled).length, + ).toBeLessThanOrEqual(2) + loader.dispose() + }) + + it.each([ + { + name: `page`, + info: createOrderByInfo(), + expectedMethod: `limited`, + }, + { + name: `prefix`, + info: createOrderByInfo({ index: undefined }), + expectedMethod: `snapshot`, + }, + { + name: `full source`, + info: createOrderByInfo({ requiresFullSource: true }), + expectedMethod: `snapshot`, + }, + ])( + `keeps a callback-before-throw $name request failed until a later operation`, + async ({ info, expectedMethod }) => { + const failure = new Error(`${expectedMethod} request failed`) + const methods: Array = [] + let fail = true + const request = ( + method: string, + options: { + onLoadSubsetResult?: ( + result: true, + acquisition: LoadSubsetOptions, + release: ReleaseLoadSubset, + ) => void + }, + ) => { + methods.push(method) + if (!fail) return + fail = false + options.onLoadSubsetResult?.(true, {}, () => + subscription.releaseLoadSubset({}), + ) + loader.loadMore() + throw failure + } + const subscription = { + setOrderByIndex: () => {}, + releaseLoadSubset: (_options: LoadSubsetOptions) => {}, + requestLimitedSnapshot: (options: RequestOptions) => + request(`limited`, options), + requestSnapshot: (options: RequestOptions) => + request(`snapshot`, options), + } + const loader = new OrderedSourceLoader( + info, + subscription as unknown as CollectionSubscription, + `row`, + ) + + expect(() => loader.start()).toThrow(failure) + await Promise.resolve() + await Promise.resolve() + expect(methods).toEqual([expectedMethod]) + expect(loader.loadMore()).toBeUndefined() + expect(methods).toEqual([expectedMethod]) + + loader.loadMore(1) + expect(methods).toEqual([expectedMethod, `snapshot`]) + loader.dispose() + }, + ) + + it(`blocks retry reentered from provisional acquisition cleanup`, () => { + const failure = new Error(`prefix request failed`) + const methods: Array = [] + let fail = true + const subscription = { + setOrderByIndex: () => {}, + releaseLoadSubset: (_options: LoadSubsetOptions) => { + loader.loadMore(1) + }, + requestSnapshot: (options: RequestOptions) => { + methods.push(`snapshot`) + if (!fail) return + fail = false + options.onLoadSubsetResult?.(true, {}, () => + subscription.releaseLoadSubset({}), + ) + throw failure + }, + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ index: undefined }), + subscription as unknown as CollectionSubscription, + `row`, + ) + + expect(() => loader.start()).toThrow(failure) + expect(methods).toEqual([`snapshot`]) + + loader.loadMore(2) + expect(methods).toEqual([`snapshot`, `snapshot`]) + loader.dispose() + }) + + it(`preserves the request failure when provisional cleanup also throws`, async () => { + const requestFailure = new Error(`snapshot publication failed`) + const cleanupFailure = new Error(`provisional cleanup failed`) + const reported: Array = [] + const loads: Array = [] + const unloads: Array = [] + let failedReleaseAttempts = 0 + const source = createCollection<{ id: number; rank: number }>({ + id: `ordered-provisional-cleanup-error`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + commit() + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[1] && ++failedReleaseAttempts === 1) { + throw cleanupFailure + } + }, + } + }, + }, + }) + const subscription = source.subscribeChanges( + (changes) => { + if (changes.length > 0) throw requestFailure + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => reported.push(error)) + const loader = new OrderedSourceLoader( + createOrderByInfo({ index: undefined }), + subscription as unknown as CollectionSubscription, + `row`, + ) + + try { + subscription.requestSnapshot({ + where: new Func(`eq`, [new PropRef([`id`]), new Value(`unrelated`)]), + optimizedOnly: false, + }) + expect(() => loader.start()).toThrow(requestFailure) + expect(subscription.lastError).toBe(requestFailure) + expect(reported).toEqual([requestFailure]) + expect(unloads).toEqual([loads[1]]) + + loader.dispose() + subscription.unsubscribe() + expect(unloads).toEqual([loads[1], loads[0]]) + expect(subscription.lastError).toBe(requestFailure) + expect(reported).toEqual([requestFailure]) + + subscription.unsubscribe() + expect(unloads).toEqual([loads[1], loads[0]]) + } finally { + loader.dispose() + subscription.unsubscribe() + await source.cleanup() + } + }) + + it.each([ + [`string`, `snapshot publication failed`], + [`undefined`, undefined], + ] as const)( + `normalizes a %s provisional failure once for every observer`, + async (_label, thrownValue) => { + const cleanupFailure = new Error(`provisional cleanup failed`) + const reported: Array = [] + let unloads = 0 + const source = createCollection<{ id: number; rank: number }>({ + id: `ordered-provisional-non-error-${String(thrownValue)}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + commit() + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloads++ + if (unloads === 1) throw cleanupFailure + }, + } + }, + }, + }) + const subscription = source.subscribeChanges( + (changes) => { + if (changes.length > 0) throw thrownValue + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => reported.push(error)) + const loader = new OrderedSourceLoader( + createOrderByInfo({ index: undefined }), + subscription as unknown as CollectionSubscription, + `row`, + ) + const notCaught = Symbol(`not caught`) + let caught: unknown = notCaught + + try { + loader.start() + } catch (error) { + caught = error + } + + expect(caught).not.toBe(notCaught) + expect(caught).toBeInstanceOf(Error) + expect((caught as Error).message).toBe(String(thrownValue)) + expect(subscription.lastError).toBe(caught) + expect(reported).toEqual([caught]) + + loader.dispose() + subscription.unsubscribe() + await source.cleanup() + }, + ) + + it(`retires an acquisition when its internal result observer throws`, async () => { + const observerFailure = new Error(`ordered result observer failed`) + const acquisition: LoadSubsetOptions = {} + const methods: Array = [] + const releases: Array = [] + let failObserver = true + const subscription = { + setOrderByIndex: () => {}, + releaseLoadSubset: (options: LoadSubsetOptions) => { + releases.push(options) + loader.loadMore(1) + }, + requestSnapshot: (options: RequestOptions) => { + methods.push(`snapshot`) + options.onLoadSubsetResult?.(true, acquisition, () => + subscription.releaseLoadSubset(acquisition), + ) + }, + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ index: undefined }), + subscription as unknown as CollectionSubscription, + `row`, + () => { + if (!failObserver) return + failObserver = false + throw observerFailure + }, + ) + + expect(() => loader.start()).toThrow(observerFailure) + await Promise.resolve() + await Promise.resolve() + expect(releases).toEqual([acquisition]) + expect(methods).toEqual([`snapshot`]) + + expect(loader.loadMore()).toBeUndefined() + expect(methods).toEqual([`snapshot`]) + + loader.loadMore(1) + expect(methods).toEqual([`snapshot`, `snapshot`]) + loader.dispose() + }) + + it(`does not replace a failed acquisition while its release is running`, async () => { + const requestFailure = new Error(`ordered acquisition rejected`) + const releaseFailure = new Error(`ordered acquisition release failed`) + const acquisition: LoadSubsetOptions = {} + const methods: Array = [] + let firstRequest = true + const subscription = { + setOrderByIndex: () => {}, + releaseLoadSubset: (_options: LoadSubsetOptions) => { + loader.loadMore(2) + throw releaseFailure + }, + requestSnapshot: (options: RequestOptions) => { + methods.push(`snapshot`) + if (!firstRequest) return + firstRequest = false + options.onLoadSubsetResult?.( + Promise.reject(requestFailure), + acquisition, + () => subscription.releaseLoadSubset(acquisition), + ) + }, + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ index: undefined }), + subscription as unknown as CollectionSubscription, + `row`, + ) + + loader.start() + await expect(pendingPromise(loader)).rejects.toBe(requestFailure) + expect(() => loader.loadMore(1)).toThrow(releaseFailure) + expect(methods).toEqual([`snapshot`]) + + loader.loadMore(3) + expect(methods).toEqual([`snapshot`, `snapshot`]) + loader.dispose() + }) + + it(`blocks a reentrant boundary retry until a later operation`, async () => { + const failure = new Error(`boundary request failed`) + const methods: Array = [] + let failBoundary = true + const subscription = { + readOrderedSnapshot: () => [{ value: { rank: 1 } }], + setOrderByIndex: () => {}, + releaseLoadSubset: (_options: LoadSubsetOptions) => {}, + requestLimitedSnapshot: (options: RequestOptions) => { + methods.push(`limited`) + options.onLoadSubsetResult?.( + true, + { + orderBy: options.orderBy, + limit: options.limit, + }, + () => + subscription.releaseLoadSubset({ + orderBy: options.orderBy, + limit: options.limit, + }), + ) + }, + requestSnapshot: (options: { + onLoadSubsetResult?: ( + result: true, + acquisition: LoadSubsetOptions, + release: ReleaseLoadSubset, + ) => void + }) => { + methods.push(`snapshot`) + if (!failBoundary) return + failBoundary = false + options.onLoadSubsetResult?.(true, {}, () => + subscription.releaseLoadSubset({}), + ) + loader.loadMore() + throw failure + }, + } + const loader = new OrderedSourceLoader( + createOrderByInfo(), + subscription as unknown as CollectionSubscription, + `row`, + ) + + loader.start() + const initial = pendingPromise(loader) + await expect(initial).rejects.toBe(failure) + expect(methods).toEqual([`limited`, `snapshot`]) + expect(loader.loadMore()).toBeUndefined() + expect(methods).toEqual([`limited`, `snapshot`]) + + loader.loadMore(1) + expect(methods).toEqual([`limited`, `snapshot`, `snapshot`]) + loader.dispose() + }) +}) diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts new file mode 100644 index 0000000000..c10075a5bd --- /dev/null +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -0,0 +1,1960 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createEffect } from '../../src/query/effect.js' +import { getLoadSubsetDemandKey } from '../../src/query/ir-stable-identity.js' +import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' +import { eq, gte } from '../../src/query/builder/functions.js' +import { + oracleRandomParameters, + readOracleRunConfig, +} from '../oracle-config.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { flushPromises } from '../utils.js' +import type { InitialQueryBuilder } from '../../src/query/builder/index.js' +import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' + +type Row = { + id: number + rank: number + eligible: boolean + label: string +} + +type Marker = { id: number; rowId: number } + +type Scenario = { + middleCount: number + middleEligible: boolean + lastEligible: boolean + tied: boolean + direction: `asc` | `desc` +} + +type RequestObservation = { + kind: `page` | `boundary` + key: string | undefined + hasCursor: boolean + limit: number | undefined + offset: number | undefined + lastKey: string | number | undefined +} + +type ConsumerObservation = { + rows: Array + requests: Array + compareRequestTrace: boolean + publications: Array> + errors: Array + live: boolean +} + +const scenarioArbitrary: fc.Arbitrary = fc.record({ + middleCount: fc.constantFrom(0 as const, 1 as const, 2 as const, 3 as const), + middleEligible: fc.boolean(), + lastEligible: fc.boolean(), + tied: fc.boolean(), + direction: fc.constantFrom(`asc` as const, `desc` as const), +}) + +const exhaustiveScenarios: ReadonlyArray = ( + [0, 1, 2, 3] as const +).flatMap((middleCount) => + [false, true].flatMap((middleEligible) => + [false, true].flatMap((lastEligible) => + [false, true].flatMap((tied) => + ([`asc`, `desc`] as const).map((direction) => ({ + middleCount, + middleEligible, + lastEligible, + tied, + direction, + })), + ), + ), + ), +) + +function compareRows(direction: Scenario[`direction`]) { + return (left: Row, right: Row): number => { + const rank = left.rank - right.rank + return (direction === `asc` ? rank : -rank) || left.id - right.id + } +} + +function rowsForScenario(scenario: Scenario): Array { + return [ + { id: 1, rank: 0, eligible: true, label: `first` }, + ...Array.from({ length: scenario.middleCount }, (_, index) => ({ + id: index + 3, + rank: scenario.tied ? 0 : index + 1, + eligible: scenario.middleEligible, + label: `middle-${index}`, + })), + { + id: 2, + rank: scenario.middleCount + 1, + eligible: scenario.lastEligible, + label: `last`, + }, + ] +} + +let harnessId = 0 + +async function observeConsumer( + kind: `collection` | `effect`, + scenario: Scenario, + joinedOnlyPredicate = false, +): Promise { + type Sync = Parameters[`sync`]>[0] + const truth = rowsForScenario(scenario).sort(compareRows(scenario.direction)) + const sourceSize = truth.length + const eligibleTruth = truth.filter(({ eligible }) => eligible) + const rowToDelete = + eligibleTruth.length >= 3 && + eligibleTruth[0]!.rank !== eligibleTruth[1]!.rank + ? eligibleTruth[0] + : undefined + const delivered = new Set() + const requests: Array = [] + const errors: Array = [] + const effectRows = new Map() + let sync!: Sync + + const apply = async (rows: ReadonlyArray) => { + const fresh = rows.filter((row) => !delivered.has(row.id)) + if (fresh.length === 0) return + sync.begin() + for (const row of fresh) { + delivered.add(row.id) + sync.write({ type: `insert`, value: { ...row } }) + } + const receipt = sync.commit() + if (receipt !== true) await receipt + } + + const source = createCollection({ + id: `ordered-consumer-${kind}-${harnessId++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: async (options) => { + const isPage = options.orderBy !== undefined + requests.push({ + kind: isPage ? `page` : `boundary`, + key: getLoadSubsetDemandKey(options), + hasCursor: options.cursor !== undefined, + limit: options.limit, + offset: options.offset, + lastKey: options.cursor?.lastKey, + }) + if (requests.length > truth.length * 3 + 4) { + throw new Error(`ordered loading did not reach a fixed point`) + } + + const matching = options.where + ? truth.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === true, + ) + : truth + + if (!isPage) { + await apply(matching.filter((row) => !delivered.has(row.id))) + return + } + + const start = + options.cursor?.lastKey === undefined + ? (options.offset ?? 0) + : matching.findIndex( + ({ id }) => id === options.cursor?.lastKey, + ) + 1 + const page = matching.slice(start).slice(0, options.limit) + if (page.length > 0) { + await apply(page) + } + }, + unloadSubset: () => {}, + } + }, + }, + }) + const markers = truth + .filter(({ eligible }) => eligible) + .map(({ id }) => ({ id, rowId: id })) + const markerSource = createCollection({ + id: `ordered-marker-${kind}-${harnessId++}`, + getKey: ({ id }) => id, + syncMode: `eager`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const marker of markers) { + write({ type: `insert`, value: marker }) + } + commit() + markReady() + }, + }, + }) + + let live: ReturnType | undefined + let effect: ReturnType | undefined + const publications: Array> = [] + const query = (q: InitialQueryBuilder) => { + const ordered = q + .from({ row: source }) + .leftJoin({ marker: markerSource }, ({ row, marker }) => + eq(row.id, marker.rowId), + ) + .where(({ row, marker }) => + joinedOnlyPredicate ? gte(marker.rowId, 0) : eq(row.id, marker.rowId), + ) + .orderBy(({ row }) => row.rank, scenario.direction) + return (rowToDelete ? ordered.orderBy(({ row }) => row.id, `asc`) : ordered) + .limit(2) + .select(({ row }) => ({ + id: row.id, + rank: row.rank, + eligible: row.eligible, + label: row.label, + })) + } + + const visibleRows = () => + (live ? [...live.values()] : [...effectRows.values()]) + .map(({ id, rank, eligible, label }) => ({ id, rank, eligible, label })) + .sort(compareRows(scenario.direction)) + + try { + if (kind === `collection`) { + live = createLiveQueryCollection(query) + live.subscribeChanges(() => { + publications.push(visibleRows()) + }) + await live.preload() + } else { + effect = createEffect({ + query, + onBatch: (events) => { + for (const event of events) { + if (event.type === `exit`) effectRows.delete(event.key) + else effectRows.set(event.key, { ...event.value }) + } + publications.push(visibleRows()) + }, + onSourceError: (error) => errors.push(error.message), + }) + } + + for (let turn = 0; turn < truth.length * 3 + 6; turn++) { + await flushPromises() + } + + const rows = visibleRows() + const expected = eligibleTruth.slice(0, 2) + expect(rows, JSON.stringify({ kind, scenario, requests })).toEqual(expected) + for (const publication of publications) { + expect(publication).toEqual(expected.slice(0, publication.length)) + } + const semanticPublications = publications.filter( + (publication, index) => + index === 0 || + JSON.stringify(publication) !== JSON.stringify(publications[index - 1]), + ) + for (let index = 1; index < semanticPublications.length; index++) { + expect(semanticPublications[index]!.length).toBeGreaterThan( + semanticPublications[index - 1]!.length, + ) + } + // Single-term bootstrap demand should be identical across entry points. + // Multi-term loading may schedule a different bounded number of prefix + // and tie refinements, so compare that path by rows and work bounds. + let finalRows = rows + const publicationsBeforeMutation = publications.length + if (rowToDelete) { + truth.splice(truth.indexOf(rowToDelete), 1) + delivered.delete(rowToDelete.id) + sync.begin({ immediate: true }) + sync.write({ type: `delete`, value: { ...rowToDelete } }) + const receipt = sync.commit() + if (receipt !== true) await receipt + for (let turn = 0; turn < sourceSize * 3 + 6; turn++) { + await flushPromises() + } + finalRows = visibleRows() + expect(finalRows, JSON.stringify({ kind, scenario, requests })).toEqual( + truth.filter(({ eligible }) => eligible).slice(0, 2), + ) + expect( + publications.length - publicationsBeforeMutation, + ).toBeLessThanOrEqual(1) + } + expect(publications.at(-1) ?? []).toEqual(finalRows) + // The explicit source deletion can publish without another provider call. + expect(publications.length).toBeLessThanOrEqual( + requests.length + 1 + Number(rowToDelete !== undefined), + ) + expect(requests.length).toBeLessThanOrEqual(sourceSize * 3 + 2) + expect( + requests.every( + (request) => request.kind === `boundary` || request.limit !== undefined, + ), + ).toBe(true) + + return { + rows: finalRows, + requests, + compareRequestTrace: rowToDelete === undefined, + publications, + errors, + live: live ? live.status === `ready` : effect?.disposed === false, + } + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await markerSource.cleanup() + await source.cleanup() + } +} + +async function assertConsumerParity(scenario: Scenario): Promise { + const [collection, effect] = await Promise.all([ + observeConsumer(`collection`, scenario), + observeConsumer(`effect`, scenario), + ]) + expect(effect.rows).toEqual(collection.rows) + expect(effect.errors).toEqual(collection.errors) + expect(effect.live).toBe(collection.live) + const semanticRequests = (requests: ReadonlyArray) => + requests.map(({ kind, hasCursor, limit, offset }) => ({ + kind, + hasCursor, + limit, + // Once a cursor is present, the original offset no longer changes the + // provider slice. Live collections retain it in the exact demand while + // Effects omit it, so compare the adapter-visible operation instead. + offset: hasCursor ? 0 : offset, + })) + if (effect.compareRequestTrace && collection.compareRequestTrace) { + expect(semanticRequests(effect.requests)).toEqual( + semanticRequests(collection.requests), + ) + } +} + +async function observeLaterOrderTermMutation( + kind: `collection` | `effect`, +): Promise<{ rows: Array; requests: Array }> { + const truth: Array = [ + { id: 1, rank: 0, eligible: true, label: `a` }, + { id: 2, rank: 0, eligible: true, label: `b` }, + { id: 3, rank: 0, eligible: true, label: `c` }, + ] + const delivered = new Set() + const requests: Array = [] + const effectRows = new Map() + let sync!: Parameters[`sync`]>[0] + + const source = createCollection({ + id: `ordered-later-term-${kind}-${harnessId++}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: async (options) => { + requests.push(getLoadSubsetDemandKey(options)) + let selected = options.where + ? truth.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === true, + ) + : [...truth] + selected.sort( + (left, right) => + left.rank - right.rank || + left.label.localeCompare(right.label) || + left.id - right.id, + ) + if (options.cursor) { + selected = selected.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + } else if (options.offset) { + selected = selected.slice(options.offset) + } + if (options.limit !== undefined) { + selected = selected.slice(0, options.limit) + } + const fresh = selected.filter(({ id }) => !delivered.has(id)) + if (fresh.length === 0) return + operations.begin() + for (const row of fresh) { + delivered.add(row.id) + operations.write({ type: `insert`, value: { ...row } }) + } + const receipt = operations.commit() + if (receipt !== true) await receipt + }, + unloadSubset: () => {}, + } + }, + }, + }) + const query = (q: InitialQueryBuilder) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .orderBy(({ row }) => row.label) + .limit(2) + const live = + kind === `collection` ? createLiveQueryCollection({ query }) : undefined + const effect = + kind === `effect` + ? createEffect({ + query, + onBatch: (events) => { + for (const event of events) { + if (event.type === `exit`) effectRows.delete(event.key) + else effectRows.set(event.key, { ...event.value }) + } + }, + }) + : undefined + + const visibleIds = () => + (live ? live.toArray : [...effectRows.values()]) + .sort( + (left, right) => + left.rank - right.rank || + left.label.localeCompare(right.label) || + left.id - right.id, + ) + .map(({ id }) => id) + + try { + if (live) await live.preload() + else await vi.waitFor(() => expect(visibleIds()).toEqual([1, 2])) + expect(visibleIds()).toEqual([1, 2]) + + const first = { ...source.get(1)!, label: `z` } + sync.begin({ immediate: true }) + sync.write({ type: `update`, value: { ...first } }) + const receipt = sync.commit() + if (receipt !== true) await receipt + await vi.waitFor(() => expect(visibleIds()).toEqual([2, 3])) + + expect(requests.length).toBeLessThanOrEqual(6) + return { rows: visibleIds(), requests } + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await source.cleanup() + } +} + +async function observeFinitePrefixMutation( + kind: `collection` | `effect`, + mutation: `move` | `delete`, +): Promise<{ + rows: Array + requests: number + publications: Array> +}> { + const truth = new Map([ + [1, { id: 1, rank: 0, eligible: true, label: `visible` }], + [2, { id: 2, rank: 1, eligible: true, label: `hidden` }], + ]) + const delivered = new Set() + const effectRows = new Map() + const publications: Array> = [] + let requests = 0 + let sync!: Parameters[`sync`]>[0] + + const source = createCollection({ + id: `ordered-finite-prefix-${kind}-${harnessId++}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: async (options) => { + requests++ + let selected = [...truth.values()].sort( + (left, right) => left.rank - right.rank || left.id - right.id, + ) + if (options.where) { + selected = selected.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === true, + ) + } + if (options.cursor) { + selected = selected.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + } else if (options.offset) { + selected = selected.slice(options.offset) + } + if (options.limit !== undefined) { + selected = selected.slice(0, options.limit) + } + const fresh = selected.filter(({ id }) => !delivered.has(id)) + if (fresh.length === 0) return + operations.begin() + for (const row of fresh) { + delivered.add(row.id) + operations.write({ type: `insert`, value: { ...row } }) + } + const receipt = operations.commit() + if (receipt !== true) await receipt + }, + unloadSubset: () => {}, + } + }, + }, + }) + const query = (q: InitialQueryBuilder) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1) + const live = + kind === `collection` ? createLiveQueryCollection({ query }) : undefined + const effect = + kind === `effect` + ? createEffect({ + query, + onBatch: (events) => { + for (const event of events) { + if (event.type === `exit`) effectRows.delete(event.key) + else effectRows.set(event.key, { ...event.value }) + } + publications.push( + [...effectRows.values()] + .sort((left, right) => left.rank - right.rank) + .map(({ id }) => id), + ) + }, + }) + : undefined + const visibleIds = () => + (live ? live.toArray : [...effectRows.values()]) + .sort((left, right) => left.rank - right.rank) + .map(({ id }) => id) + + try { + if (live) { + live.subscribeChanges(() => publications.push(visibleIds())) + await live.preload() + } else { + await vi.waitFor(() => expect(visibleIds()).toEqual([1])) + } + const requestsBeforeMutation = requests + const moved = { ...truth.get(1)!, rank: 10 } + if (mutation === `delete`) truth.delete(1) + else truth.set(1, moved) + sync.begin({ immediate: true }) + sync.write({ + type: mutation === `delete` ? `delete` : `update`, + value: { ...moved }, + }) + const receipt = sync.commit() + if (receipt !== true) await receipt + await vi.waitFor(() => + expect( + visibleIds(), + JSON.stringify({ kind, requests, source: source.toArray }), + ).toEqual([2]), + ) + + expect(requests).toBeGreaterThan(requestsBeforeMutation) + return { rows: visibleIds(), requests, publications } + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await source.cleanup() + } +} + +describe(`ordered source work oracle`, () => { + it(`keeps later order-term invalidation equal across consumers`, async () => { + const [collection, effect] = await Promise.all([ + observeLaterOrderTermMutation(`collection`), + observeLaterOrderTermMutation(`effect`), + ]) + expect(effect.rows).toEqual(collection.rows) + // The two graph entry points may take a different bounded number of + // refinement passes, but they must exercise the same demand forms. + expect(new Set(effect.requests)).toEqual(new Set(collection.requests)) + }) + + it.each([`move`, `delete`] as const)( + `recovers a finite source prefix equally across consumers after %s`, + async (mutation) => { + const [collection, effect] = await Promise.all([ + observeFinitePrefixMutation(`collection`, mutation), + observeFinitePrefixMutation(`effect`, mutation), + ]) + + expect(effect.rows).toEqual(collection.rows) + expect(effect.publications.at(-1)).toEqual(collection.publications.at(-1)) + }, + ) + + it(`loads each source of a filtered join once`, async () => { + type Order = { + id: number + scheduledAt: string + status: string + addressId: number + } + type Charge = { id: number; addressId: number } + let orderLoads = 0 + let chargeLoads = 0 + const orders = createCollection({ + id: `ordered-filtered-join-orders`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { + id: 1, + scheduledAt: `2024-01-15`, + status: `queued`, + addressId: 1, + }, + }) + write({ + type: `insert`, + value: { + id: 2, + scheduledAt: `2024-01-10`, + status: `queued`, + addressId: 2, + }, + }) + commit() + markReady() + return { + loadSubset: () => { + orderLoads++ + return true + }, + } + }, + }, + }) + const charges = createCollection({ + id: `ordered-filtered-join-charges`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 10, addressId: 1 } }) + write({ type: `insert`, value: { id: 20, addressId: 2 } }) + commit() + markReady() + return { + loadSubset: () => { + chargeLoads++ + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ order: orders }) + .where(({ order }) => gte(order.scheduledAt, `2024-01-12`)) + .where(({ order }) => eq(order.status, `queued`)) + .innerJoin({ charge: charges }, ({ order, charge }) => + eq(order.addressId, charge.addressId), + ), + ) + + try { + await live.preload() + expect( + [...live.values()].map(({ order, charge }) => [order.id, charge.id]), + ).toEqual([[1, 10]]) + expect(orderLoads).toBe(1) + expect(chargeLoads).toBe(1) + } finally { + await Promise.all([live.cleanup(), orders.cleanup(), charges.cleanup()]) + } + }) + + it.each([`collection`, `effect`] as const)( + `does no source work for a zero-sized %s window`, + async (consumer) => { + let loads = 0 + const source = createCollection({ + id: `ordered-zero-window`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + } + }, + }, + }) + const query = (q: InitialQueryBuilder) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(0) + const live = + consumer === `collection` + ? createLiveQueryCollection({ + id: `ordered-zero-window-live`, + query, + startSync: true, + }) + : undefined + const effect = + consumer === `effect` + ? createEffect({ query, onBatch: () => {} }) + : undefined + + try { + if (live) await live.preload() + else await flushPromises() + expect(loads).toBe(0) + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await source.cleanup() + } + }, + ) + + it.each( + ([`collection`, `effect`] as const).flatMap((consumer) => + ([`eager`, `off`] as const).map((autoIndex) => ({ + consumer, + autoIndex, + })), + ), + )( + `does no source work for a joined $consumer with a zero-sized $autoIndex window`, + async ({ consumer, autoIndex }) => { + let rowLoads = 0 + let markerLoads = 0 + const source = createCollection({ + id: `ordered-zero-window-unindexed-${consumer}-source`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex, + defaultIndexType: autoIndex === `eager` ? BTreeIndex : undefined, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + rowLoads++ + return true + }, + } + }, + }, + }) + const markers = createCollection({ + id: `ordered-zero-window-unindexed-${consumer}-marker`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex, + defaultIndexType: autoIndex === `eager` ? BTreeIndex : undefined, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + markerLoads++ + return true + }, + } + }, + }, + }) + const query = (q: InitialQueryBuilder) => + q + .from({ row: source }) + .innerJoin({ marker: markers }, ({ row, marker }) => + eq(row.id, marker.rowId), + ) + .orderBy(({ row }) => row.rank) + .limit(0) + const live = + consumer === `collection` + ? createLiveQueryCollection({ query, startSync: true }) + : undefined + const effect = + consumer === `effect` + ? createEffect({ query, onBatch: () => {} }) + : undefined + + try { + if (live) await live.preload() + else await flushPromises() + expect({ rowLoads, markerLoads }).toEqual({ + rowLoads: 0, + markerLoads: 0, + }) + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await Promise.all([source.cleanup(), markers.cleanup()]) + } + }, + ) + + it.each( + ([`collection`, `effect`] as const).flatMap((consumer) => + [2, 5].flatMap((limit) => + ([`first`, `last`] as const).flatMap((position) => + ([`asc`, `desc`] as const).map((direction) => ({ + consumer, + limit, + position, + direction, + })), + ), + ), + ), + )( + `does not refetch when a visible row changes outside the ordering key: %j`, + async ({ consumer, limit, position, direction }) => { + let sync!: Parameters[`sync`]>[0] + let loads = 0 + const rows = rowsForScenario({ + middleCount: 1, + middleEligible: true, + lastEligible: true, + tied: false, + direction, + }) + const source = createCollection({ + id: `ordered-value-update`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: async () => { + loads++ + if (loads > 1) return + operations.begin() + for (const row of rows) { + operations.write({ type: `insert`, value: { ...row } }) + } + const receipt = operations.commit() + if (receipt !== true) await receipt + }, + } + }, + }, + }) + const query = (q: InitialQueryBuilder) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, direction) + .limit(limit) + const effectRows = new Map() + const live = + consumer === `collection` ? createLiveQueryCollection(query) : undefined + const effect = + consumer === `effect` + ? createEffect({ + query, + onBatch: (events) => { + for (const event of events) { + if (event.type === `exit`) effectRows.delete(event.key) + else effectRows.set(event.key, event.value) + } + }, + }) + : undefined + const readRows = () => + (live ? [...live.values()] : [...effectRows.values()]) + .map(({ id, rank, eligible, label }) => ({ + id, + rank, + eligible, + label, + })) + .sort(compareRows(direction)) + + try { + if (live) await live.preload() + await flushPromises() + const expected = [...rows].sort(compareRows(direction)).slice(0, limit) + expect(readRows()).toEqual(expected) + const loadCount = loads + const row = position === `first` ? expected[0]! : expected.at(-1)! + sync.begin({ immediate: true }) + sync.write({ type: `update`, value: { ...row, label: `changed` } }) + sync.commit() + await flushPromises() + + expect(readRows()).toEqual( + expected.map((value) => + value.id === row.id ? { ...value, label: `changed` } : value, + ), + ) + expect(loads).toBe(loadCount) + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await source.cleanup() + } + }, + ) + + it.each([ + { + name: `matching`, + secondaryRows: [ + { id: `c-child`, joinKey: `c` }, + { id: `d-child`, joinKey: `d` }, + ], + expected: [`c:c-child`, `d:d-child`], + }, + { + name: `empty`, + secondaryRows: [] as Array<{ id: string; joinKey: string }>, + expected: [] as Array, + }, + ])( + `waits for a $name joined source after exhausting tied ordered rows`, + async ({ name, secondaryRows, expected }) => { + type Primary = { id: string; rank: number; joinKey: string } + type Secondary = { id: string; joinKey: string } + const primaryRows: Array = [`a`, `b`, `c`, `d`].map((id) => ({ + id, + rank: 0, + joinKey: id, + })) + const deliveredPrimary = new Set() + const deliveredSecondary = new Set() + const secondaryLoads: Array<{ + options: LoadSubsetOptions + gate: ReturnType> + }> = [] + let primaryExhausted = false + + const primary = createCollection({ + id: `ordered-late-join-primary-${name}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: async (options) => { + const rows = options.orderBy + ? [ + primaryRows[ + options.cursor?.lastKey + ? primaryRows.findIndex( + ({ id }) => id === options.cursor?.lastKey, + ) + 1 + : 0 + ], + ].filter((row): row is Primary => row !== undefined) + : primaryRows.filter( + (row) => + !options.where || + evaluateReferenceExpression(options.where, row) === + true, + ) + const fresh = rows.filter(({ id }) => !deliveredPrimary.has(id)) + if (fresh.length > 0) { + begin() + for (const row of fresh) { + deliveredPrimary.add(row.id) + write({ type: `insert`, value: row }) + } + const receipt = commit(options.signal) + if (receipt !== true) await receipt + } + primaryExhausted = deliveredPrimary.size === primaryRows.length + }, + } + }, + }, + }) + const secondary = createCollection({ + id: `ordered-late-join-secondary-${name}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: async (options) => { + const gate = createDeferred() + secondaryLoads.push({ options, gate }) + await gate.promise + const fresh = secondaryRows.filter( + (row) => + !deliveredSecondary.has(row.id) && + (!options.where || + evaluateReferenceExpression(options.where, row) === true), + ) + for (const row of fresh) { + deliveredSecondary.add(row.id) + begin() + write({ type: `insert`, value: row }) + const receipt = commit(options.signal) + if (receipt !== true) await receipt + } + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ primaryRow: primary }) + .innerJoin( + { secondaryRow: secondary }, + ({ primaryRow, secondaryRow }) => + eq(primaryRow.joinKey, secondaryRow.joinKey), + ) + .orderBy(({ primaryRow }) => primaryRow.rank) + .limit(2), + ) + + try { + const preload = live.preload() + let settled = false + void preload.finally(() => { + settled = true + }) + await vi.waitFor(() => + expect( + primaryExhausted, + JSON.stringify({ + deliveredPrimary: [...deliveredPrimary], + secondaryLoads: secondaryLoads.length, + }), + ).toBe(true), + ) + expect(secondaryLoads.length).toBeGreaterThan(0) + expect(settled).toBe(false) + + for (const load of [...secondaryLoads].reverse()) { + load.gate.resolve() + await flushPromises() + } + await preload + + expect( + live.toArray.map( + ({ primaryRow, secondaryRow }) => + `${primaryRow.id}:${secondaryRow.id}`, + ), + ).toEqual(expected) + expect(live.isLoadingSubset).toBe(false) + expect(live.utils.lastSubsetError).toBeUndefined() + } finally { + for (const { gate } of secondaryLoads) gate.resolve() + await Promise.all([ + live.cleanup(), + primary.cleanup(), + secondary.cleanup(), + ]) + } + }, + ) + + it(`keeps independent joined loads isolated when they settle in reverse`, async () => { + type Primary = { id: string; rank: number; joinKey: string } + type Secondary = { id: string; joinKey: string } + const pending: Array<{ + options: LoadSubsetOptions + gate: ReturnType> + }> = [] + const completionOrder: Array = [] + const primary = createCollection({ + id: `ordered-independent-primary`, + getKey: ({ id }) => id, + syncMode: `eager`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `a`, rank: 1, joinKey: `a` } }) + write({ type: `insert`, value: { id: `b`, rank: 2, joinKey: `b` } }) + commit() + markReady() + }, + }, + }) + const secondaryRows: Array = [ + { id: `a-child`, joinKey: `a` }, + { id: `b-child`, joinKey: `b` }, + ] + const delivered = new Set() + const secondary = createCollection({ + id: `ordered-independent-secondary`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: async (options) => { + const index = pending.length + const gate = createDeferred() + pending.push({ options, gate }) + await gate.promise + const rows = secondaryRows.filter( + (candidate) => + !delivered.has(candidate.id) && + (!options.where || + evaluateReferenceExpression(options.where, candidate) === + true), + ) + for (const row of rows) { + delivered.add(row.id) + begin() + write({ type: `insert`, value: row }) + const receipt = commit(options.signal) + if (receipt !== true) await receipt + } + completionOrder.push(index) + }, + } + }, + }, + }) + const createJoined = (id: `a` | `b`) => + createLiveQueryCollection((q) => + q + .from({ primaryRow: primary }) + .where(({ primaryRow }) => eq(primaryRow.id, id)) + .innerJoin( + { secondaryRow: secondary }, + ({ primaryRow, secondaryRow }) => + eq(primaryRow.joinKey, secondaryRow.joinKey), + ) + .orderBy(({ primaryRow }) => primaryRow.rank) + .limit(1), + ) + const first = createJoined(`a`) + const second = createJoined(`b`) + + try { + const firstPreload = first.preload() + await vi.waitFor(() => expect(pending).toHaveLength(1)) + const secondPreload = second.preload() + await vi.waitFor(() => expect(pending).toHaveLength(2)) + let firstSettled = false + void firstPreload.finally(() => { + firstSettled = true + }) + + pending[1]!.gate.resolve() + await secondPreload + await flushPromises() + expect(firstSettled).toBe(false) + expect( + second.toArray.map( + ({ primaryRow, secondaryRow }) => + `${primaryRow.id}:${secondaryRow.id}`, + ), + ).toEqual([`b:b-child`]) + + pending[0]!.gate.resolve() + await firstPreload + expect(completionOrder).toEqual([1, 0]) + expect( + first.toArray.map( + ({ primaryRow, secondaryRow }) => + `${primaryRow.id}:${secondaryRow.id}`, + ), + ).toEqual([`a:a-child`]) + expect(first.utils.lastSubsetError).toBeUndefined() + expect(second.utils.lastSubsetError).toBeUndefined() + } finally { + for (const { gate } of pending) gate.resolve() + await Promise.all([ + first.cleanup(), + second.cleanup(), + primary.cleanup(), + secondary.cleanup(), + ]) + } + }) + + it(`keeps one source's replay from suppressing another source's recovery`, async () => { + type Primary = { id: number; rank: number } + type Secondary = { id: number; primaryId: number } + const primaryTruth = new Map([ + [1, { id: 1, rank: 0 }], + [2, { id: 2, rank: 1 }], + ]) + const deliveredPrimary = new Set() + const secondaryReplay = createDeferred() + let primaryRequests = 0 + let replayingSecondary = false + let primarySync!: Parameters[`sync`]>[0] + let secondarySync!: Parameters[`sync`]>[0] + let secondaryReplayCalls = 0 + + const primary = createCollection({ + id: `ordered-independent-recovery-primary`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + primarySync = operations + operations.markReady() + return { + loadSubset: async (options) => { + primaryRequests++ + let selected = [...primaryTruth.values()].sort( + (left, right) => left.rank - right.rank || left.id - right.id, + ) + if (options.where) { + selected = selected.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === true, + ) + } + if (options.cursor) { + selected = selected.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + } else if (options.offset) { + selected = selected.slice(options.offset) + } + if (options.limit !== undefined) { + selected = selected.slice(0, options.limit) + } + const fresh = selected.filter( + ({ id }) => !deliveredPrimary.has(id), + ) + if (fresh.length === 0) return + operations.begin() + for (const row of fresh) { + deliveredPrimary.add(row.id) + operations.write({ type: `insert`, value: { ...row } }) + } + const receipt = operations.commit(options.signal) + if (receipt !== true) await receipt + }, + unloadSubset: () => {}, + } + }, + }, + }) + const secondary = createCollection({ + id: `ordered-independent-recovery-secondary`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + secondarySync = operations + operations.markReady() + return { + loadSubset: async (options) => { + if (replayingSecondary) { + secondaryReplayCalls++ + await secondaryReplay.promise + } + operations.begin() + operations.write({ + type: `insert`, + value: { id: 1, primaryId: 1 }, + }) + const receipt = operations.commit(options.signal) + if (receipt !== true) await receipt + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: primary }) + .leftJoin({ child: secondary }, ({ row, child }) => + eq(row.id, child.primaryId), + ) + .orderBy(({ row }) => row.rank) + .limit(1) + .select(({ row, child }) => ({ + id: row.id, + rank: row.rank, + childId: child.id, + })), + ) + + try { + await live.preload() + expect(live.toArray.map(({ id }) => id)).toEqual([1]) + + replayingSecondary = true + secondarySync.begin() + secondarySync.truncate() + const truncateReceipt = secondarySync.commit() + if (truncateReceipt !== true) await truncateReceipt + await vi.waitFor(() => expect(secondaryReplayCalls).toBeGreaterThan(0)) + + const requestsBeforeMutation = primaryRequests + const moved = { id: 1, rank: 10 } + primaryTruth.set(1, moved) + primarySync.begin({ immediate: true }) + primarySync.write({ type: `update`, value: moved }) + const mutationReceipt = primarySync.commit() + if (mutationReceipt !== true) await mutationReceipt + await vi.waitFor(() => + expect(primaryRequests).toBeGreaterThan(requestsBeforeMutation), + ) + expect(live.toArray.map(({ id }) => id)).toEqual([1]) + + secondaryReplay.resolve() + await vi.waitFor(() => + expect(live.toArray.map(({ id }) => id)).toEqual([2]), + ) + expect(primaryRequests).toBeGreaterThan(requestsBeforeMutation) + } finally { + secondaryReplay.resolve() + await Promise.all([ + live.cleanup(), + primary.cleanup(), + secondary.cleanup(), + ]) + } + }) + + it(`publishes one complete batch after an indexed loader fills a window`, async () => { + const remoteRows: ReadonlyArray = [ + { id: 1, rank: 1, eligible: true, label: `one` }, + { id: 2, rank: 2, eligible: true, label: `two` }, + ] + const batches: Array> = [] + const callbackReads: Array> = [] + let loads = 0 + const delivered = new Set() + let sync!: Parameters[`sync`]>[0] + const source = createCollection({ + id: `ordered-atomic-indexed-window`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + loads++ + const row = remoteRows.find( + (candidate) => + !delivered.has(candidate.id) && + (!options.where || + evaluateReferenceExpression(options.where, candidate) === + true), + ) + if (!row) return true + delivered.add(row.id) + sync.begin() + sync.write({ type: `insert`, value: row }) + const receipt = sync.commit() + if (receipt !== true) { + throw new Error(`Expected synchronous source application`) + } + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(0), + ) + const readIds = () => live.toArray.map(({ id }) => id) + const subscription = live.subscribeChanges( + (changes) => { + batches.push(changes.map(({ key }) => Number(key)).sort()) + callbackReads.push(readIds()) + }, + { includeInitialState: false }, + ) + + try { + await live.preload() + await live.utils.setWindow({ offset: 0, limit: 2 }) + await flushPromises() + + // Each page turn is followed by a tie-boundary request. The source + // returns one row at a time while honoring both predicates. + expect(loads).toBe(4) + expect(readIds()).toEqual([1, 2]) + expect(batches).toEqual([[1, 2]]) + expect(callbackReads).toEqual([[1, 2]]) + } finally { + subscription.unsubscribe() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`keeps live collections and Effects equal across the exhaustive small domain`, async () => { + for (const scenario of exhaustiveScenarios) { + await assertConsumerParity(scenario) + } + }) + + it(`fills ordered windows filtered only through a left-joined alias`, async () => { + for (const scenario of exhaustiveScenarios) { + const [collection, effect] = await Promise.all([ + observeConsumer(`collection`, scenario, true), + observeConsumer(`effect`, scenario, true), + ]) + expect(collection.rows).toEqual(effect.rows) + expect(collection.errors).toEqual([]) + expect(effect.errors).toEqual([]) + } + }) + + it.each( + [0, 1, 2, 3, 4].flatMap((middleCount) => + ([`asc`, `desc`] as const).flatMap((direction) => + [false, true].map((tied) => ({ middleCount, direction, tied })), + ), + ), + )( + `settles an underfilled source without repeating a continuation: %j`, + async ({ middleCount, direction, tied }) => { + const scenario: Scenario = { + middleCount, + middleEligible: false, + lastEligible: false, + tied, + direction, + } + const [collection, effect] = await Promise.all([ + observeConsumer(`collection`, scenario), + observeConsumer(`effect`, scenario), + ]) + + expect(collection.rows.map(({ id }) => id)).toEqual([1]) + expect(effect.rows).toEqual(collection.rows) + expect(effect.errors).toEqual(collection.errors) + expect(effect.live).toBe(collection.live) + for (const observation of [collection, effect]) { + expect(observation.errors).toEqual([]) + expect(observation.live).toBe(true) + // At most one page and one tie-boundary load per source row, including + // the final empty page. A fixed cap mistakes longer finite walks for loops. + const sourceSize = rowsForScenario(scenario).length + expect( + observation.requests.length, + JSON.stringify(observation.requests), + ).toBeLessThanOrEqual(2 * sourceSize) + expect( + observation.requests.filter(({ kind }) => kind === `page`).length, + ).toBeLessThanOrEqual(sourceSize) + expect( + observation.requests.filter(({ kind }) => kind === `boundary`).length, + ).toBeLessThanOrEqual(sourceSize) + expect( + new Set(observation.requests.map(({ key }) => key)).size, + JSON.stringify(observation.requests), + ).toBe(observation.requests.length) + } + }, + ) + + it(`replaces an ordered snapshot after truncate without repeating void loads`, async () => { + const initial: ReadonlyArray = [ + { id: 1, rank: 1, eligible: true, label: `old-one` }, + { id: 2, rank: 2, eligible: true, label: `old-two` }, + ] + const replacement: ReadonlyArray = [ + { id: 3, rank: 3, eligible: true, label: `new-three` }, + { id: 4, rank: 4, eligible: true, label: `new-four` }, + ] + let truth = initial + let sync!: Parameters[`sync`]>[0] + let loads = 0 + const installed = new Set() + const source = createCollection({ + id: `ordered-void-truncate`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: async (options) => { + loads++ + if (loads > 12) { + throw new Error( + `ordered void loading did not reach a fixed point`, + ) + } + const matching = options.where + ? truth.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === true, + ) + : truth + const rows = matching + .slice( + options.offset ?? 0, + options.limit === undefined + ? undefined + : (options.offset ?? 0) + options.limit, + ) + .filter(({ id }) => !installed.has(id)) + if (rows.length === 0) return + sync.begin() + for (const row of rows) { + installed.add(row.id) + sync.write({ type: `insert`, value: row }) + } + const receipt = sync.commit() + if (receipt !== true) await receipt + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + + try { + await live.preload() + expect(live.toArray.map(({ id }) => id)).toEqual([1, 2]) + + truth = replacement + installed.clear() + sync.begin() + sync.truncate() + const receipt = sync.commit() + if (receipt !== true) await receipt + await flushPromises() + + expect(live.toArray.map(({ id }) => id)).toEqual([3, 4]) + expect(loads).toBeLessThanOrEqual(8) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it.each([false, true])( + `cancels queued ordered recovery on cleanup (restart=%s)`, + async (restart) => { + const seedRow: Row = { id: 1, rank: 1, eligible: true, label: `retained` } + let sync!: Parameters[`sync`]>[0] + let installed = false + const requests: Array = [] + const source = createCollection({ + id: `queued-recovery-cleanup-${harnessId++}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + requests.push(options) + if (installed) return true + installed = true + operations.begin() + operations.write({ type: `insert`, value: seedRow }) + return operations.commit() + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + try { + await live.preload() + const initialRequests = requests.length + sync.begin() + sync.truncate() + installed = false + expect(sync.commit()).toBe(true) + await live.cleanup() + await flushPromises() + expect(requests).toHaveLength(initialRequests) + if (restart) { + await live.preload() + expect(live.toArray.map(({ id }) => id)).toEqual([1]) + expect(live.utils.lastSubsetError).toBeUndefined() + expect(requests.length).toBeGreaterThan(initialRequests) + } + expect( + requests.filter( + ({ where, limit }) => where === undefined && limit === undefined, + ), + ).toEqual([]) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }, + ) + + it.each([ + { name: `until full-source recovery settles`, failure: undefined }, + { + name: `when full-source recovery throws synchronously`, + failure: { + mode: `sync` as const, + attempts: 1, + error: new Error(`full-source recovery failed`), + }, + }, + { + name: `after two synchronous full-source recovery failures`, + failure: { + mode: `sync` as const, + attempts: 2, + error: new Error(`full-source recovery failed twice`), + }, + }, + { + name: `after asynchronous full-source recovery retries`, + failure: { + mode: `async` as const, + attempts: 1, + error: new Error(`full-source recovery rejected`), + }, + }, + { + name: `after two asynchronous full-source recovery failures`, + failure: { + mode: `async` as const, + attempts: 2, + error: new Error(`full-source recovery rejected twice`), + }, + }, + ])(`keeps an ordered snapshot unchanged $name`, async ({ failure }) => { + const makeRows = (ranks: ReadonlyArray): Array => + ranks.map((rank, index) => ({ + id: index + 1, + rank, + eligible: true, + label: `row-${rank}`, + })) + let truth = makeRows([1, 2, 3, 4, 5]) + let sync!: Parameters[`sync`]>[0] + let recovering = false + let fullSourceRequests = 0 + const fullSource = createDeferred() + const installed = new Set() + const publications: Array> = [] + const escapedErrors: Array = [] + const acquisitions: Array = [] + const releases: Array = [] + const enqueueMicrotask = globalThis.queueMicrotask.bind(globalThis) + const queueMicrotaskSpy = + failure?.mode === `sync` + ? vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => + enqueueMicrotask(() => { + try { + callback() + } catch (error) { + escapedErrors.push(error) + } + }), + ) + : undefined + + const source = createCollection({ + id: `delayed-full-source-recovery`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + const isFullSource = + options.where === undefined && options.limit === undefined + if (recovering && isFullSource) { + fullSourceRequests++ + if (failure && fullSourceRequests <= failure.attempts) { + if (failure.mode === `sync`) throw failure.error + acquisitions.push(options) + return Promise.reject(failure.error) + } + } + acquisitions.push(options) + + return (async () => { + if (recovering && isFullSource && !failure) { + await fullSource.promise + } + + let selected = options.where + ? truth.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === + true, + ) + : [...truth] + if (options.cursor) { + selected = selected.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + } + selected.sort((left, right) => left.rank - right.rank) + if (!options.cursor && options.offset) { + selected = selected.slice(options.offset) + } + if (options.limit !== undefined) { + selected = selected.slice(0, options.limit) + } + + const fresh = selected.filter(({ id }) => !installed.has(id)) + if (fresh.length === 0) return + sync.begin() + for (const row of fresh) { + installed.add(row.id) + sync.write({ type: `insert`, value: row }) + } + const receipt = sync.commit() + if (receipt !== true) await receipt + })() + }, + unloadSubset: (options) => { + releases.push(options) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + const subscription = live.subscribeChanges(() => { + publications.push(live.toArray.map(({ rank }) => rank)) + }) + + try { + await live.preload() + await live.utils.setWindow({ offset: 0, limit: 4 }) + await flushPromises() + expect(live.toArray.map(({ rank }) => rank)).toEqual([1, 2, 3, 4]) + const publicationCount = publications.length + + truth = makeRows([0, 0.5, 1, 1.5, 2, 3]) + installed.clear() + recovering = true + sync.begin() + sync.truncate() + const receipt = sync.commit() + if (receipt !== true) await receipt + await vi.waitFor(() => expect(fullSourceRequests).toBe(1)) + for (let index = 0; index < 4; index++) await flushPromises() + + expect(live.toArray.map(({ rank }) => rank)).toEqual([1, 2, 3, 4]) + expect(publications).toHaveLength(publicationCount) + + if (failure) { + expect(live.utils.lastSubsetError).toBe(failure.error) + expect(escapedErrors).toEqual([]) + truth[0] = { ...truth[0]!, label: `updated during failed recovery` } + sync.begin() + sync.write({ type: `update`, value: truth[0] }) + const updateReceipt = sync.commit() + if (updateReceipt !== true) await updateReceipt + await flushPromises() + expect(live.toArray.map(({ rank }) => rank)).toEqual([1, 2, 3, 4]) + expect(live.get(1)?.label).toBe(`row-1`) + expect(publications).toHaveLength(publicationCount) + expect(fullSourceRequests).toBe(1) + const retryCount = failure.attempts + for (let retry = 0; retry < retryCount; retry++) { + installed.clear() + sync.begin() + sync.truncate() + const retryReceipt = sync.commit() + if (retryReceipt !== true) await retryReceipt + await vi.waitFor(() => expect(fullSourceRequests).toBe(retry + 2)) + if (retry + 1 < retryCount) { + for (let index = 0; index < 4; index++) await flushPromises() + expect(live.toArray.map(({ rank }) => rank)).toEqual([1, 2, 3, 4]) + expect(publications).toHaveLength(publicationCount) + expect(live.utils.lastSubsetError).toBe(failure.error) + expect(escapedErrors).toEqual([]) + } + } + await vi.waitFor(() => + expect(source.toArray.map(({ rank }) => rank).sort()).toEqual([ + 0, 0.5, 1, 1.5, 2, 3, + ]), + ) + await vi.waitFor(() => + expect(live.toArray.map(({ rank }) => rank)).toEqual([ + 0, 0.5, 1, 1.5, + ]), + ) + expect(fullSourceRequests).toBe(retryCount + 1) + expect(live.get(1)?.label).toBe(`updated during failed recovery`) + } else { + fullSource.resolve() + await flushPromises() + expect(live.toArray.map(({ rank }) => rank)).toEqual([0, 0.5, 1, 1.5]) + } + expect(publications.slice(publicationCount)).toEqual([[0, 0.5, 1, 1.5]]) + expect(escapedErrors).toEqual([]) + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(live.toArray.map(({ rank }) => rank)).toEqual([0, 0.5]) + const loadsBeforeReplay = fullSourceRequests + installed.clear() + sync.begin() + sync.truncate() + const nextReplay = sync.commit() + if (nextReplay !== true) await nextReplay + await flushPromises() + expect(fullSourceRequests).toBeGreaterThan(loadsBeforeReplay) + expect(live.toArray.map(({ rank }) => rank)).toEqual([0, 0.5]) + } finally { + queueMicrotaskSpy?.mockRestore() + fullSource.resolve() + subscription.unsubscribe() + await Promise.all([live.cleanup(), source.cleanup()]) + } + expect(releases).toHaveLength(acquisitions.length) + for (const acquisition of acquisitions) { + expect( + releases.filter((release) => release === acquisition), + ).toHaveLength(1) + } + }) + + const { multiplier, ...replay } = readOracleRunConfig() + const runs = 20 * multiplier + + fcTest.prop([scenarioArbitrary], { numRuns: runs, seed: 17801 })( + `keeps rows, request traces, batches, errors, and liveness equal for a fixed seed`, + assertConsumerParity, + ) + + fcTest.prop( + [scenarioArbitrary], + oracleRandomParameters(runs, replay, `ordered-work.consumer-parity`), + )( + `keeps rows, request traces, batches, errors, and liveness equal for a random or replayed seed`, + assertConsumerParity, + ) +}) diff --git a/packages/db/tests/query/pagination-oracle.property.test.ts b/packages/db/tests/query/pagination-oracle.property.test.ts new file mode 100644 index 0000000000..ea9638f13b --- /dev/null +++ b/packages/db/tests/query/pagination-oracle.property.test.ts @@ -0,0 +1,4939 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { BTreeIndex } from '../../src/index.js' +import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' +import { eq } from '../../src/query/builder/functions.js' +import { PropRef } from '../../src/query/ir.js' +import { makeComparator } from '../../src/utils/comparison.js' +import { + oracleRandomParameters, + readOracleRunConfig, +} from '../oracle-config.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { TraceAssertionError } from '../trace-runner.js' +import { flushPromises, mockSyncCollectionOptions } from '../utils.js' +import type { Deferred } from '../../src/deferred.js' +import type { + ChangeMessage, + LoadSubsetOptions, + SyncConfig, +} from '../../src/types.js' + +type PageRow = { + id: number + rank: number + keep?: boolean +} + +type PublicPageRow = Pick +type PublicPageChange = { + type: `insert` | `update` | `delete` + key: number + value: PublicPageRow + previousValue?: PublicPageRow +} + +type MultiOrderRow = { + id: number + primary: number | null + secondary: number | null +} + +type MultiOrderTerm = { + direction: `asc` | `desc` + nulls: `first` | `last` +} + +type MultiOrderScenario = { + rows: ReadonlyArray + primary: MultiOrderTerm + secondary: MultiOrderTerm + limit: number +} + +type NullableCursorRow = { + id: number + rank: number | null +} + +type LocaleCursorRow = { + id: number + label: string +} + +type AdversarialOrderedRow = { + id: number + rank: number | null | object + label: string +} + +type NullableCursorScenario = { + rank: number + direction: `asc` | `desc` +} + +type PaginationWindow = { + offset: number + limit: number +} + +type PaginationScenario = { + ranks: ReadonlyArray + keeps?: ReadonlyArray + direction: `asc` | `desc` + windows: ReadonlyArray + explicitPublicKeyOrder?: boolean + includeFilter?: boolean + reverseInsertion?: boolean + reverseProviderTies?: boolean + localRowsBeforeFirstRequest?: ReadonlyArray +} + +type PaginationAction = + | ({ type: `window` } & PaginationWindow) + | { type: `put`; id: number; rank: number; keep?: boolean } + | { type: `delete`; id: number } + +type PaginationStateScenario = { + ranks: ReadonlyArray + keeps?: ReadonlyArray + direction: `asc` | `desc` + initialWindow: PaginationWindow + actions: ReadonlyArray + explicitPublicKeyOrder?: boolean + includeFilter?: boolean + reverseInsertion?: boolean +} + +type PaginationStructure = Pick< + PaginationScenario, + `explicitPublicKeyOrder` | `includeFilter` | `reverseInsertion` +> + +type PendingCursorLoad = { + options: LoadSubsetOptions + deferred: ReturnType> + settled?: boolean +} + +type PendingMutation = + | { type: `insert`; row: PageRow } + | { type: `delete`; id: number } + | { type: `update`; row: PageRow } + +type PendingMutationScenario = { + ranks: ReadonlyArray + direction: `asc` | `desc` + limit: number + mutation: PendingMutation + responseOutcome: `resolve` | `reject` +} + +class DeliveredRowsTraceAssertionError extends TraceAssertionError { + constructor( + cause: unknown, + readonly deliveredRows: ReadonlyArray, + ) { + super(0, cause) + if (cause instanceof Error) { + this.message += `: ${cause.message}; delivered=${JSON.stringify(deliveredRows)}` + } + } +} + +class PendingMutationTraceAssertionError extends DeliveredRowsTraceAssertionError {} + +class PendingHistoryTraceAssertionError extends DeliveredRowsTraceAssertionError {} + +type PendingHistoryScenario = { + ranks: ReadonlyArray + direction: `asc` | `desc` + initialLimit: number + narrowLimit: number + wideLimit: number + firstRank: number + secondRank: number +} + +const initialRowsArbitrary = fc.array( + fc.record({ + rank: fc.integer({ min: -2, max: 2 }), + keep: fc.boolean(), + }), + { + minLength: 1, + maxLength: 12, + }, +) + +const scenarioPayloadArbitrary: fc.Arbitrary = fc + .record({ + rows: initialRowsArbitrary, + direction: fc.constantFrom(`asc` as const, `desc` as const), + reverseProviderTies: fc.boolean(), + windows: fc.array( + fc.record({ + offset: fc.integer({ min: 0, max: 12 }), + limit: fc.integer({ min: 0, max: 8 }), + }), + { minLength: 1, maxLength: 12 }, + ), + }) + .map(({ rows, ...scenario }) => ({ + ...scenario, + ranks: rows.map(({ rank }) => rank), + keeps: rows.map(({ keep }) => keep), + })) + +const paginationStructures: ReadonlyArray = [ + ...[false, true].flatMap((explicitPublicKeyOrder) => + [false, true].flatMap((includeFilter) => + [false, true].map((reverseInsertion) => ({ + explicitPublicKeyOrder, + includeFilter, + reverseInsertion, + })), + ), + ), +] + +const paginationStructureArbitrary: fc.Arbitrary = + fc.record({ + explicitPublicKeyOrder: fc.boolean(), + includeFilter: fc.boolean(), + reverseInsertion: fc.boolean(), + }) + +const scenarioArbitrary: fc.Arbitrary = fc + .tuple(scenarioPayloadArbitrary, paginationStructureArbitrary) + .map(([scenario, structure]) => ({ ...scenario, ...structure })) + +const windowArbitrary: fc.Arbitrary = fc.record({ + offset: fc.integer({ min: 0, max: 12 }), + limit: fc.integer({ min: 0, max: 8 }), +}) + +const paginationActionArbitrary: fc.Arbitrary = fc.oneof( + { + weight: 2, + arbitrary: windowArbitrary.map((window) => ({ + type: `window` as const, + ...window, + })), + }, + { + weight: 3, + arbitrary: fc.record({ + type: fc.constant(`put` as const), + id: fc.integer({ min: 1, max: 16 }), + rank: fc.integer({ min: -2, max: 2 }), + keep: fc.boolean(), + }), + }, + { + weight: 2, + arbitrary: fc.record({ + type: fc.constant(`delete` as const), + id: fc.integer({ min: 1, max: 16 }), + }), + }, +) + +const stateScenarioPayloadArbitrary: fc.Arbitrary = fc + .record({ + rows: initialRowsArbitrary, + direction: fc.constantFrom(`asc` as const, `desc` as const), + initialWindow: windowArbitrary, + actions: fc.array(paginationActionArbitrary, { + minLength: 1, + maxLength: 20, + }), + }) + .map(({ rows, ...scenario }) => ({ + ...scenario, + ranks: rows.map(({ rank }) => rank), + keeps: rows.map(({ keep }) => keep), + })) + +const stateScenarioArbitrary: fc.Arbitrary = fc + .tuple(stateScenarioPayloadArbitrary, paginationStructureArbitrary) + .map(([scenario, structure]) => ({ ...scenario, ...structure })) + +const pendingMutationScenarioArbitrary: fc.Arbitrary = + fc + .record({ + ranks: fc.array(fc.integer({ min: -2, max: 2 }), { + minLength: 3, + maxLength: 8, + }), + direction: fc.constantFrom(`asc` as const, `desc` as const), + requestedLimit: fc.integer({ min: 1, max: 8 }), + mutationKind: fc.constantFrom( + `insert` as const, + `update` as const, + `delete` as const, + ), + responseOutcome: fc.constantFrom(`resolve` as const, `reject` as const), + targetIndex: fc.nat({ max: 7 }), + rank: fc.integer({ min: -2, max: 2 }), + }) + .map( + ({ + ranks, + direction, + requestedLimit, + mutationKind, + responseOutcome, + targetIndex, + rank, + }) => { + const id = (targetIndex % ranks.length) + 1 + const previousRank = ranks[id - 1]! + const changedRank = + rank === previousRank ? (rank === 2 ? -2 : rank + 1) : rank + const mutation: PendingMutation = + mutationKind === `insert` + ? { type: `insert`, row: { id: ranks.length + 1, rank } } + : mutationKind === `update` + ? { type: `update`, row: { id, rank: changedRank } } + : { type: `delete`, id } + return { + ranks, + direction, + limit: Math.min(requestedLimit, ranks.length - 2), + mutation, + responseOutcome, + } + }, + ) + +const pendingHistoryScenarioArbitrary: fc.Arbitrary = fc + .record({ + ranks: fc.array(fc.integer({ min: -2, max: 2 }), { + minLength: 4, + maxLength: 8, + }), + direction: fc.constantFrom(`asc` as const, `desc` as const), + requestedInitialLimit: fc.integer({ min: 2, max: 7 }), + requestedNarrowLimit: fc.integer({ min: 1, max: 6 }), + requestedWideLimit: fc.integer({ min: 3, max: 8 }), + firstRank: fc.integer({ min: -2, max: 2 }), + secondRank: fc.integer({ min: -2, max: 2 }), + }) + .map( + ({ + ranks, + direction, + requestedInitialLimit, + requestedNarrowLimit, + requestedWideLimit, + firstRank, + secondRank, + }) => { + const initialLimit = Math.min(requestedInitialLimit, ranks.length - 1) + return { + ranks, + direction, + initialLimit, + narrowLimit: Math.min(requestedNarrowLimit, initialLimit - 1), + wideLimit: Math.max( + initialLimit + 1, + Math.min(requestedWideLimit, ranks.length), + ), + firstRank, + secondRank, + } + }, + ) + +const responseTimingArbitrary = fc.constantFrom( + `before-response` as const, + `after-response` as const, +) + +const nullableNumberArbitrary = fc.option(fc.integer({ min: -2, max: 2 }), { + nil: null, +}) + +const multiOrderTermArbitrary: fc.Arbitrary = fc.record({ + direction: fc.constantFrom(`asc` as const, `desc` as const), + nulls: fc.constantFrom(`first` as const, `last` as const), +}) + +const multiOrderScenarioArbitrary: fc.Arbitrary = fc + .record({ + rows: fc.uniqueArray( + fc.record({ + id: fc.integer({ min: 1, max: 12 }), + primary: nullableNumberArbitrary, + secondary: nullableNumberArbitrary, + }), + { + minLength: 2, + maxLength: 10, + selector: ({ id }) => id, + }, + ), + primary: multiOrderTermArbitrary, + secondary: multiOrderTermArbitrary, + requestedLimit: fc.integer({ min: 1, max: 10 }), + }) + .filter(({ rows }) => + rows.some( + ({ primary, secondary }) => primary === null || secondary === null, + ), + ) + .map(({ rows, primary, secondary, requestedLimit }) => ({ + rows, + primary, + secondary, + limit: Math.min(requestedLimit, rows.length), + })) + +const nullableCursorScenarioArbitrary: fc.Arbitrary = + fc.record({ + rank: fc.integer({ min: -2, max: 2 }), + direction: fc.constantFrom(`asc` as const, `desc` as const), + }) + +type CleanupTarget = { + cleanup: () => unknown +} + +async function cleanupAll( + ...targets: ReadonlyArray +): Promise { + const results = await Promise.allSettled( + targets.map((target) => Promise.resolve().then(() => target.cleanup())), + ) + const rejection = results.find( + (result): result is PromiseRejectedResult => result.status === `rejected`, + ) + if (rejection) throw rejection.reason +} + +const { multiplier, ...replay } = readOracleRunConfig() +const orderedScenarioRuns = 12 * multiplier +const transitionScenarioRuns = 8 * multiplier + +let collectionSequence = 0 + +function referenceWindow( + rows: ReadonlyArray, + direction: `asc` | `desc`, + window: PaginationWindow, +): Array { + return referenceWindowRows(rows, direction, window).map(({ id }) => id) +} + +function referenceWindowRows( + rows: ReadonlyArray, + direction: `asc` | `desc`, + window: PaginationWindow, +): Array { + return [...rows] + .sort( + (left, right) => + (left.rank - right.rank) * (direction === `asc` ? 1 : -1) || + left.id - right.id, + ) + .slice(window.offset, window.offset + window.limit) + .map(({ id, rank }) => ({ id, rank })) +} + +function projectPageRow(row: PageRow): PublicPageRow { + return { id: row.id, rank: row.rank } +} + +function normalizePageChanges( + changes: ReadonlyArray>, +): Array { + return changes + .map((change) => ({ + type: change.type, + key: change.key, + value: projectPageRow(change.value), + ...(change.previousValue !== undefined + ? { previousValue: projectPageRow(change.previousValue) } + : {}), + })) + .sort((left, right) => left.key - right.key) +} + +function expectedPageChanges( + before: ReadonlyArray, + after: ReadonlyArray, +): Array { + const beforeById = new Map(before.map((row) => [row.id, row])) + const afterById = new Map(after.map((row) => [row.id, row])) + const changes: Array = [] + + for (const row of before) { + const next = afterById.get(row.id) + if (!next) { + changes.push({ type: `delete`, key: row.id, value: row }) + } else if (next.rank !== row.rank) { + changes.push({ + type: `update`, + key: row.id, + value: next, + previousValue: row, + }) + } + } + for (const row of after) { + if (!beforeById.has(row.id)) { + changes.push({ type: `insert`, key: row.id, value: row }) + } + } + return changes.sort((left, right) => left.key - right.key) +} + +function isKeptRow(id: number): boolean { + return id % 3 !== 0 +} + +function visibleRows( + rows: ReadonlyArray, + includeFilter: boolean | undefined, +): Array { + return includeFilter ? rows.filter(({ keep }) => keep) : [...rows] +} + +function rowsForLoadSubset( + rows: ReadonlyArray, + options: LoadSubsetOptions, +): Array { + const matchingRows = options.where + ? rows.filter( + (row) => evaluateReferenceExpression(options.where!, row) === true, + ) + : rows + if (!options.cursor) { + const start = options.offset ?? 0 + const end = + options.limit === undefined ? matchingRows.length : start + options.limit + return matchingRows.slice(start, end) + } + + const current = matchingRows.filter((row) => + Boolean(evaluateReferenceExpression(options.cursor!.whereCurrent, row)), + ) + const from = matchingRows.filter((row) => + Boolean(evaluateReferenceExpression(options.cursor!.whereFrom, row)), + ) + const limitedFrom = + options.limit === undefined ? from : from.slice(0, options.limit) + const requested = new Map() + for (const row of [...current, ...limitedFrom]) requested.set(row.id, row) + return [...requested.values()] +} + +function createConformingOrderedSource( + id: string, + rows: ReadonlyArray, + autoIndex: `eager` | `off` = `eager`, +) { + const requests: Array = [] + const delivered = new Set() + const source = createCollection({ + id, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + const requested = rowsForLoadSubset(rows, options) + begin() + for (const row of requested) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: row }) + } + const receipt = commit(options.signal) + return receipt === true ? Promise.resolve() : receipt + }, + } + }, + }, + }) + + return { requests, source } +} + +async function runPaginationScenario( + scenario: PaginationScenario, +): Promise { + const rows = scenario.ranks.map((rank, index) => ({ + id: index + 1, + rank, + keep: scenario.keeps?.[index] ?? isKeptRow(index + 1), + })) + const initialRows = scenario.reverseInsertion ? [...rows].reverse() : rows + const expectedRows = visibleRows(rows, scenario.includeFilter) + const initialWindow = scenario.windows[0]! + const source = createCollection( + mockSyncCollectionOptions({ + id: `pagination-oracle-source-${collectionSequence++}`, + initialData: initialRows.map((row) => ({ ...row })), + getKey: (row: PageRow) => row.id, + autoIndex: `eager`, + }), + ) + const live = createLiveQueryCollection((query) => { + const from = query.from({ row: source }) + const filtered = scenario.includeFilter + ? from.where(({ row }) => eq(row.keep, true)) + : from + const ordered = filtered.orderBy(({ row }) => row.rank, scenario.direction) + return ( + scenario.explicitPublicKeyOrder === false + ? ordered + : ordered.orderBy(({ row }) => row.id, `asc`) + ) + .offset(initialWindow.offset) + .limit(initialWindow.limit) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + }) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id, rank }) => ({ id, rank }))).toEqual( + referenceWindowRows(expectedRows, scenario.direction, initialWindow), + ) + + for (const window of scenario.windows.slice(1)) { + const result = live.utils.setWindow(window) + if (result instanceof Promise) await result + + expect( + Array.from(live.values(), ({ id, rank }) => ({ id, rank })), + ).toEqual(referenceWindowRows(expectedRows, scenario.direction, window)) + } + } finally { + await cleanupAll(live, source) + } +} + +async function expectMultiOrderBoundaryMatches(): Promise { + await runMultiOrderScenario({ + rows: [ + { id: 1, primary: 0, secondary: 2 }, + { id: 2, primary: 0, secondary: 0 }, + { id: 3, primary: 0, secondary: 1 }, + { id: 4, primary: 1, secondary: 1 }, + { id: 5, primary: 1, secondary: 0 }, + { id: 6, primary: 2, secondary: 0 }, + ], + primary: { direction: `asc`, nulls: `first` }, + secondary: { direction: `asc`, nulls: `first` }, + limit: 4, + }) +} + +function compareNullableNumber( + left: number | null, + right: number | null, + term: MultiOrderTerm, +): number { + if (left === null || right === null) { + if (left === right) return 0 + return left === null + ? term.nulls === `first` + ? -1 + : 1 + : term.nulls === `first` + ? 1 + : -1 + } + const compared = left === right ? 0 : left < right ? -1 : 1 + return term.direction === `asc` ? compared : -compared +} + +function referenceMultiOrder(scenario: MultiOrderScenario): Array { + return [...scenario.rows] + .sort( + (left, right) => + compareNullableNumber(left.primary, right.primary, scenario.primary) || + compareNullableNumber( + left.secondary, + right.secondary, + scenario.secondary, + ) || + left.id - right.id, + ) + .slice(0, scenario.limit) + .map(({ id }) => id) +} + +async function runMultiOrderScenario( + scenario: MultiOrderScenario, +): Promise { + const source = createCollection( + mockSyncCollectionOptions({ + id: `pagination-multi-order-oracle-source-${collectionSequence++}`, + initialData: scenario.rows.map((row) => ({ ...row })), + getKey: (row: MultiOrderRow) => row.id, + autoIndex: `eager`, + }), + ) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.primary, scenario.primary) + .orderBy(({ row }) => row.secondary, scenario.secondary) + .orderBy(({ row }) => row.id, `asc`) + .limit(scenario.limit) + .select(({ row }) => ({ id: row.id })), + ) + + try { + await live.preload() + try { + expect(Array.from(live.values(), ({ id }) => id)).toEqual( + referenceMultiOrder(scenario), + ) + } catch (error) { + throw new TraceAssertionError(0, error) + } + } finally { + await cleanupAll(live, source) + } +} + +async function runNullableCursorScenario( + scenario: NullableCursorScenario, +): Promise { + const rows: Array = [ + { id: 1, rank: null }, + { id: 2, rank: scenario.rank }, + ] + const orderedRows = [...rows].sort( + (left, right) => + compareNullableNumber(left.rank, right.rank, { + direction: scenario.direction, + nulls: `first`, + }) || left.id - right.id, + ) + const pending: Array = [] + const delivered = new Set() + let begin!: () => void + let write!: (message: { type: `insert`; value: NullableCursorRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-nullable-cursor-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, { + direction: scenario.direction, + nulls: `first`, + }) + .orderBy(({ row }) => row.id, `asc`) + .limit(1), + ) + + try { + const preload = live.preload() + expect(pending.length).toBeGreaterThan(0) + // Settling one request can append its boundary-refinement request. + // eslint-disable-next-line @typescript-eslint/prefer-for-of + for (let index = 0; index < pending.length; index++) { + const request = pending[index]! + begin() + for (const row of rowsForLoadSubset(orderedRows, request.options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.settled = true + request.deferred.resolve() + await flushPromises() + } + await preload + + try { + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + } catch (error) { + throw new TraceAssertionError(0, error) + } + } finally { + for (const request of pending) request.deferred.resolve() + await cleanupAll(live, source) + } +} + +async function runPaginationStateScenario( + scenario: PaginationStateScenario, +): Promise { + const rows = new Map( + scenario.ranks.map((rank, index) => [ + index + 1, + { + id: index + 1, + rank, + keep: scenario.keeps?.[index] ?? isKeptRow(index + 1), + }, + ]), + ) + const initialRows = [...rows.values()] + if (scenario.reverseInsertion) initialRows.reverse() + let currentWindow = scenario.initialWindow + const sourceOptions = mockSyncCollectionOptions({ + id: `pagination-state-oracle-source-${collectionSequence++}`, + initialData: initialRows.map((row) => ({ ...row })), + getKey: (row: PageRow) => row.id, + autoIndex: `eager` as const, + }) + const source = createCollection(sourceOptions) + const live = createLiveQueryCollection((query) => { + const from = query.from({ row: source }) + const filtered = scenario.includeFilter + ? from.where(({ row }) => eq(row.keep, true)) + : from + const ordered = filtered.orderBy(({ row }) => row.rank, scenario.direction) + return ( + scenario.explicitPublicKeyOrder === false + ? ordered + : ordered.orderBy(({ row }) => row.id, `asc`) + ) + .offset(currentWindow.offset) + .limit(currentWindow.limit) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + }) + const publications: Array<{ + changes: Array + rows: Array + }> = [] + let publicationSubscription: + | ReturnType + | undefined + + const readCurrentWindow = () => + Array.from(live.values(), ({ id, rank }) => ({ id, rank })) + + const expectCurrentWindow = (checkpoint: number) => { + try { + expect(readCurrentWindow()).toEqual( + referenceWindowRows( + visibleRows([...rows.values()], scenario.includeFilter), + scenario.direction, + currentWindow, + ), + ) + } catch (error) { + throw new TraceAssertionError(checkpoint, error) + } + } + + try { + await live.preload() + expectCurrentWindow(0) + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBeUndefined() + publicationSubscription = live.subscribeChanges( + (changes) => + publications.push({ + changes: normalizePageChanges( + changes as Array>, + ), + rows: readCurrentWindow(), + }), + { includeInitialState: false }, + ) + + for (const [index, action] of scenario.actions.entries()) { + const beforeRows = readCurrentWindow() + const publicationCount = publications.length + if (action.type === `window`) { + currentWindow = { offset: action.offset, limit: action.limit } + const result = live.utils.setWindow(currentWindow) + if (result instanceof Promise) await result + } else if (action.type === `put`) { + const row = { + id: action.id, + rank: action.rank, + keep: action.keep ?? isKeptRow(action.id), + } + const type = rows.has(action.id) ? `update` : `insert` + rows.set(action.id, row) + sourceOptions.utils.begin() + sourceOptions.utils.write({ type, value: { ...row } }) + sourceOptions.utils.commit() + } else { + const row = rows.get(action.id) + if (row) { + rows.delete(action.id) + sourceOptions.utils.begin() + sourceOptions.utils.write({ type: `delete`, value: { ...row } }) + sourceOptions.utils.commit() + } + } + expectCurrentWindow(index + 1) + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBeUndefined() + const afterRows = readCurrentWindow() + const expectedChanges = expectedPageChanges(beforeRows, afterRows) + expect(publications.slice(publicationCount)).toEqual( + expectedChanges.length > 0 + ? [{ changes: expectedChanges, rows: afterRows }] + : [], + ) + } + } finally { + publicationSubscription?.unsubscribe() + await cleanupAll(live, source) + } +} + +async function runOnDemandPaginationScenario( + scenario: PaginationScenario, + assertLoads?: (loads: ReadonlyArray) => void, +): Promise { + const authoritativeRows = scenario.ranks.map((rank, index) => ({ + id: index + 1, + rank, + keep: scenario.keeps?.[index] ?? isKeptRow(index + 1), + })) + const expectedRows = visibleRows(authoritativeRows, scenario.includeFilter) + const directionFactor = scenario.direction === `asc` ? 1 : -1 + const orderedRows = [...authoritativeRows].sort( + (left, right) => + (left.rank - right.rank) * directionFactor || + (left.id - right.id) * + (scenario.explicitPublicKeyOrder === false && + scenario.reverseProviderTies + ? -1 + : 1), + ) + const deliveredIds = new Set() + const loads: Array = [] + const initialWindow = scenario.windows[0]! + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + + const source = createCollection({ + id: `pagination-on-demand-oracle-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + const { markReady } = operations + markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + loads.push({ ...options }) + const requested = rowsForLoadSubset(orderedRows, options) + const delivered = scenario.reverseInsertion + ? [...requested].reverse() + : requested + + const settled = new Promise((resolve) => { + queueMicrotask(() => { + begin() + for (const row of delivered) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + resolve() + }) + }) + return settled + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => { + const from = query.from({ row: source }) + const filtered = scenario.includeFilter + ? from.where(({ row }) => eq(row.keep, true)) + : from + const ordered = filtered.orderBy(({ row }) => row.rank, scenario.direction) + return ( + scenario.explicitPublicKeyOrder === false + ? ordered + : ordered.orderBy(({ row }) => row.id, `asc`) + ) + .offset(initialWindow.offset) + .limit(initialWindow.limit) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + }) + const publications: Array<{ + changes: Array + rows: Array + status: string + }> = [] + const publicationSubscription = live.subscribeChanges( + (changes) => { + const rows = Array.from(live.values(), ({ id, rank }) => ({ id, rank })) + publications.push({ + changes: normalizePageChanges( + changes as Array>, + ), + rows, + status: live.status, + }) + }, + { includeInitialState: false }, + ) + + try { + const preloadPublicationCount = publications.length + const preload = live.preload() + expect(Array.from(live.values())).toHaveLength(0) + await preload + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBeUndefined() + if (initialWindow.limit > 0) { + expect(loads.length).toBeGreaterThan(0) + } + try { + expect( + Array.from(live.values(), ({ id, rank }) => ({ id, rank })), + ).toEqual( + referenceWindowRows(expectedRows, scenario.direction, initialWindow), + ) + } catch (error) { + throw new TraceAssertionError(0, error) + } + const initialExpected = referenceWindowRows( + expectedRows, + scenario.direction, + initialWindow, + ) + expect(publications.slice(preloadPublicationCount)).toEqual([ + ...(initialExpected.length > 0 + ? [ + { + changes: expectedPageChanges([], initialExpected), + rows: initialExpected, + status: `loading`, + }, + ] + : []), + // A real source acquisition uses one empty batch to wake subscriptions + // when the initial source set becomes ready, even if it produced no + // visible rows. A zero window needs no acquisition or wake-up. + ...(initialWindow.limit > 0 + ? [{ changes: [], rows: initialExpected, status: `ready` }] + : []), + ]) + + if (scenario.localRowsBeforeFirstRequest) { + expect(loads).toHaveLength(0) + const publicationCount = publications.length + begin() + for (const row of scenario.localRowsBeforeFirstRequest) { + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + expect(publications).toHaveLength(publicationCount) + expect(Array.from(live.values())).toHaveLength(0) + } + + for (const [index, window] of scenario.windows.slice(1).entries()) { + const before = Array.from(live.values(), ({ id, rank }) => ({ id, rank })) + const publicationCount = publications.length + const result = live.utils.setWindow(window) + if (result instanceof Promise) { + expect( + Array.from(live.values(), ({ id, rank }) => ({ id, rank })), + ).toEqual(before) + } + if (result instanceof Promise) await result + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBeUndefined() + + try { + expect( + Array.from(live.values(), ({ id, rank }) => ({ id, rank })), + ).toEqual(referenceWindowRows(expectedRows, scenario.direction, window)) + } catch (error) { + throw new TraceAssertionError(index + 1, error) + } + const after = referenceWindowRows( + expectedRows, + scenario.direction, + window, + ) + const expectedChanges = expectedPageChanges(before, after) + expect(publications.slice(publicationCount)).toEqual( + expectedChanges.length > 0 + ? [{ changes: expectedChanges, rows: after, status: `ready` }] + : [], + ) + } + + const expectedOrderBy = [ + { + expression: new PropRef([`rank`]), + compareOptions: { direction: scenario.direction, nulls: `first` }, + }, + ...(scenario.explicitPublicKeyOrder === false + ? [] + : [ + { + expression: new PropRef([`id`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ]), + ] + for (const load of loads) { + if (load.orderBy) { + expect(load.orderBy).toMatchObject(expectedOrderBy) + } else if (load.where) { + // Boundary refinement asks for the complete tie class with an exact + // predicate. Prefix and cursor requests still carry the source order. + expect(load.limit).toBeUndefined() + } else { + // If the same finite prefix cannot fill the local window, one + // unbounded request safely establishes the remaining source rows. + expect(load.cursor).toBeUndefined() + expect(load.limit).toBeUndefined() + expect(load.offset).toBeUndefined() + } + } + expect( + loads.length, + JSON.stringify( + loads.map(({ limit, offset, cursor, where }) => ({ + limit, + offset, + cursor, + where, + })), + ), + ).toBeLessThanOrEqual(scenario.windows.length * (expectedRows.length + 2)) + assertLoads?.(loads) + } finally { + publicationSubscription.unsubscribe() + await cleanupAll(live, source) + } +} + +async function expectOnDemandWindowsAreCompletionOrderIndependent( + deliveryOrder: `forward` | `reverse`, +): Promise { + const authoritativeRows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + { id: 3, rank: 2 }, + { id: 4, rank: 3 }, + ] + const pending: Array = [] + const deliveredIds = new Set([1]) + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + const apply = (options: LoadSubsetOptions) => { + begin() + for (const row of rowsForLoadSubset(authoritativeRows, options)) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + } + const source = createCollection({ + id: `pagination-completion-order-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + begin() + write({ type: `insert`, value: { ...authoritativeRows[0]! } }) + commit() + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + } + }, + }, + }) + const createLive = (limit: number) => + createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .orderBy(({ row }) => row.id, `asc`) + .limit(limit), + ) + const firstLive = createLive(2) + const secondLive = createLive(3) + + try { + const first = firstLive.preload() + const second = secondLive.preload() + expect(pending).toHaveLength(2) + + const indices = deliveryOrder === `forward` ? [0, 1] : [1, 0] + for (const index of indices) { + const request = pending[index]! + apply(request.options) + request.deferred.resolve() + await flushPromises() + } + for (let index = 2; index < pending.length; index++) { + const request = pending[index]! + apply(request.options) + request.deferred.resolve() + await flushPromises() + } + await first + await second + + expect(Array.from(firstLive.values(), ({ id }) => id)).toEqual([1, 2]) + expect(Array.from(secondLive.values(), ({ id }) => id)).toEqual([1, 2, 3]) + } finally { + for (const request of pending) request.deferred.resolve() + await cleanupAll(firstLive, secondLive, source) + } +} + +async function runAdversarialOrderedProviderScenario(options: { + providerRows: ReadonlyArray + initialRows?: ReadonlyArray + order: + | { kind: `rank`; direction: `asc` | `desc`; nulls: `first` | `last` } + | { + kind: `reference` + direction?: `asc` | `desc` + nulls?: `first` | `last` + } + | { kind: `locale` } + limit: number + expectedIds: ReadonlyArray + useOffsetWhenAvailable?: boolean +}): Promise> { + const loads: Array = [] + const delivered = new Set(options.initialRows?.map(({ id }) => id) ?? []) + const source = createCollection({ + id: `pagination-adversarial-order-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + if (options.initialRows?.length) { + begin() + for (const row of options.initialRows) { + write({ type: `insert`, value: { ...row } }) + } + commit() + } + markReady() + return { + loadSubset: (loadOptions: LoadSubsetOptions) => { + loads.push(loadOptions) + if (loads.length > options.providerRows.length * 4 + 4) { + throw new Error( + `Ordered refinement exceeded its finite source work bound: ${JSON.stringify( + loads.map(({ limit, offset, cursor }) => ({ + limit, + offset, + lastKey: cursor?.lastKey, + })), + )}`, + ) + } + const providerRows = loadOptions.where + ? options.providerRows.filter( + (row) => + evaluateReferenceExpression(loadOptions.where!, row) === + true, + ) + : options.providerRows + const providerMatch = options.useOffsetWhenAvailable + ? providerRows.slice( + loadOptions.offset ?? 0, + loadOptions.limit === undefined + ? undefined + : (loadOptions.offset ?? 0) + loadOptions.limit, + ) + : rowsForLoadSubset(options.providerRows, loadOptions) + const requested = providerMatch + begin() + for (const row of requested) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + const receipt = commit() + return receipt === true ? Promise.resolve() : receipt + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => { + const from = query.from({ row: source }) + const ordered = + options.order.kind === `locale` + ? from.orderBy(({ row }) => row.label, { + direction: `asc`, + nulls: `first`, + stringSort: `locale`, + locale: `en-US`, + localeOptions: { numeric: true }, + }) + : from.orderBy( + ({ row }) => row.rank, + options.order.kind === `reference` + ? { + direction: options.order.direction ?? `asc`, + nulls: options.order.nulls ?? `first`, + } + : { + direction: options.order.direction, + nulls: options.order.nulls, + }, + ) + return ordered.limit(options.limit).select(({ row }) => ({ id: row.id })) + }) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual( + options.expectedIds, + ) + // Snapshot observations before cleanup. Teardown must not create fresh + // source demand, and callers must not mistake such work for the scenario's + // final refinement request. + return [...loads] + } finally { + await cleanupAll(live, source) + } +} + +async function runPendingMutationScenario( + scenario: PendingMutationScenario, + timing: `before-response` | `after-response`, + finalLimitAfterMutation?: number, + explicitPublicKeyOrder = true, + transport: `cursor` | `offset` | `key` = `cursor`, +): Promise { + const rows = new Map( + scenario.ranks.map((rank, index) => [index + 1, { id: index + 1, rank }]), + ) + const firstDelivered = referenceWindowRows( + [...rows.values()], + scenario.direction, + { offset: 0, limit: 1 }, + )[0]! + const pending: Array = [] + const deliveredIds = new Set([firstDelivered.id]) + // A rejected initial subset load is fatal. Establish a ready baseline first + // so reject scenarios exercise subscription-scoped window recovery. + let capturePending = scenario.responseOutcome === `resolve` + let begin!: () => void + let write!: (message: { + type: `insert` | `update` | `delete` + value: PageRow + }) => void + let commit!: () => void + + const source = createCollection({ + id: `pagination-event-order-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + begin() + write({ type: `insert`, value: { ...firstDelivered } }) + commit() + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + if (!capturePending) return true + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => { + const ordered = query + .from({ row: source }) + .orderBy(({ row }) => row.rank, scenario.direction) + return ( + explicitPublicKeyOrder + ? ordered.orderBy(({ row }) => row.id, `asc`) + : ordered + ).limit(scenario.responseOutcome === `reject` ? 1 : scenario.limit) + }) + const outstanding: Array> = [] + + const applyMutation = () => { + const { mutation } = scenario + begin() + if (mutation.type === `delete`) { + const row = rows.get(mutation.id) + if (!row) throw new Error(`Cannot delete missing authoritative row`) + rows.delete(mutation.id) + deliveredIds.delete(mutation.id) + write({ type: `delete`, value: { ...row } }) + } else { + rows.set(mutation.row.id, { ...mutation.row }) + deliveredIds.add(mutation.row.id) + write({ type: mutation.type, value: { ...mutation.row } }) + } + commit() + } + + const settlePending = async () => { + // Settling one request can append its boundary-refinement request. + // eslint-disable-next-line @typescript-eslint/prefer-for-of + for (let index = 0; index < pending.length; index++) { + const request = pending[index]! + if (request.settled) continue + request.settled = true + const orderedRows = referenceWindowRows( + [...rows.values()], + scenario.direction, + { + offset: 0, + limit: rows.size, + }, + ) + const options = { ...request.options } + if (transport !== `cursor`) { + // Model providers whose opaque continuation token is indexed by the + // last fetched row key, rather than by the predicate expression. + if (transport === `key` && options.cursor) { + const boundary = orderedRows.findIndex( + ({ id }) => id === options.cursor!.lastKey, + ) + expect(boundary).toBeGreaterThanOrEqual(0) + options.offset = boundary + 1 + } + options.cursor = undefined + } + begin() + for (const row of rowsForLoadSubset(orderedRows, options)) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.deferred.resolve() + await flushPromises() + } + } + + try { + const preload = live.preload() + outstanding.push(preload) + let finalLimit = scenario.limit + if (scenario.responseOutcome === `resolve`) { + expect(pending).toHaveLength(1) + if (timing === `before-response`) applyMutation() + await settlePending() + await preload + if (timing === `after-response`) { + applyMutation() + await flushPromises() + } + if (finalLimitAfterMutation !== undefined) { + finalLimit = finalLimitAfterMutation + const widened = live.utils.setWindow({ + offset: 0, + limit: finalLimit, + }) + if (widened instanceof Promise) outstanding.push(widened) + } + await settlePending() + await Promise.all(outstanding) + } else { + await preload + await flushPromises() + capturePending = true + expect(pending).toHaveLength(0) + finalLimit += 1 + const failedWindow = live.utils.setWindow({ + offset: 0, + limit: finalLimit, + }) + expect(failedWindow).toBeInstanceOf(Promise) + const cursorError = new Error(`cursor failed`) + const observedFailure = (failedWindow as Promise).then( + () => undefined, + (error: unknown) => error, + ) + outstanding.push((failedWindow as Promise).catch(() => {})) + expect(pending).toHaveLength(1) + if (timing === `before-response`) applyMutation() + pending[0]!.settled = true + pending[0]!.deferred.reject(cursorError) + if (timing === `after-response`) applyMutation() + await flushPromises() + await settlePending() + expect(await observedFailure).toBe(cursorError) + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBe(cursorError) + + const retry = live.utils.setWindow({ offset: 0, limit: finalLimit }) + const observedRetry = + retry instanceof Promise + ? retry.then(undefined, (error: unknown) => { + throw error + }) + : undefined + await flushPromises() + await settlePending() + await flushPromises() + if (observedRetry) { + outstanding.push(observedRetry) + await observedRetry + } + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBe(cursorError) + } + + try { + expect( + Array.from(live.values(), ({ id, rank }) => ({ id, rank })), + ).toEqual( + referenceWindowRows([...rows.values()], scenario.direction, { + offset: 0, + limit: finalLimit, + }), + ) + } catch (error) { + throw new PendingMutationTraceAssertionError( + error, + referenceWindowRows( + [...rows.values()].filter(({ id }) => deliveredIds.has(id)), + scenario.direction, + { offset: 0, limit: deliveredIds.size }, + ), + ) + } + } finally { + for (const request of pending) request.deferred.resolve() + await Promise.allSettled(outstanding) + await cleanupAll(live, source) + } +} + +async function runRejectedCursorRetryAfterMutation(): Promise { + const rows = new Map([ + [1, { id: 1, rank: 0 }], + [2, { id: 2, rank: 1 }], + [3, { id: 3, rank: 2 }], + [4, { id: 4, rank: 3 }], + ]) + const pending: Array = [] + const deliveredIds = new Set([1]) + // Keep the rejected cursor in the incremental path rather than failing the + // live query's initial preload. + let capturePending = false + let begin!: () => void + let write!: (message: { type: `insert` | `update`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-rejected-cursor-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + begin() + write({ type: `insert`, value: { ...rows.get(1)! } }) + commit() + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + if (!capturePending) return true + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .orderBy(({ row }) => row.id, `asc`) + .limit(1), + ) + + const settle = async (request: PendingCursorLoad): Promise => { + request.settled = true + begin() + const orderedRows = referenceWindowRows([...rows.values()], `asc`, { + offset: 0, + limit: rows.size, + }) + for (const row of rowsForLoadSubset(orderedRows, request.options)) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.deferred.resolve() + await flushPromises() + } + + try { + await live.preload() + await flushPromises() + capturePending = true + expect(pending).toHaveLength(0) + + const failedWindow = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(failedWindow).toBeInstanceOf(Promise) + const cursorError = new Error(`cursor failed`) + const observedFailure = (failedWindow as Promise).then( + () => undefined, + (error: unknown) => error, + ) + expect(pending).toHaveLength(1) + + rows.set(1, { id: 1, rank: 3 }) + begin() + write({ type: `update`, value: { id: 1, rank: 3 } }) + commit() + + pending[0]!.settled = true + pending[0]!.deferred.reject(cursorError) + await flushPromises() + for (let index = 1; index < pending.length; index++) { + if (!pending[index]!.settled) await settle(pending[index]!) + } + expect(await observedFailure).toBe(cursorError) + + const retry = live.utils.setWindow({ offset: 0, limit: 3 }) + for (let index = 1; index < pending.length; index++) { + if (!pending[index]!.settled) await settle(pending[index]!) + } + if (retry instanceof Promise) await retry + + try { + expect(Array.from(live.values(), ({ id }) => id)).toEqual([2, 3, 1]) + } catch (error) { + throw new TraceAssertionError(0, error) + } + } finally { + for (const request of pending) request.deferred.resolve() + await cleanupAll(live, source) + } +} + +async function runPendingHistoryScenario( + scenario: PendingHistoryScenario, +): Promise { + const rows = new Map( + scenario.ranks.map((rank, index) => [index + 1, { id: index + 1, rank }]), + ) + const firstDelivered = referenceWindowRows( + [...rows.values()], + scenario.direction, + { offset: 0, limit: 1 }, + )[0]! + const pending: Array = [] + const deliveredIds = new Set([firstDelivered.id]) + const outstanding: Array> = [] + let begin!: () => void + let write!: (message: { type: `insert` | `update`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-pending-history-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + begin() + write({ type: `insert`, value: { ...firstDelivered } }) + commit() + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, scenario.direction) + .orderBy(({ row }) => row.id, `asc`) + .limit(scenario.initialLimit), + ) + + const updateFirstDelivered = (rank: number): void => { + const previous = rows.get(firstDelivered.id)! + const changedRank = changedRankValue(previous.rank, rank) + const next = { ...previous, rank: changedRank } + rows.set(next.id, next) + begin() + write({ type: `update`, value: { ...next } }) + commit() + } + + const settle = async (request: PendingCursorLoad): Promise => { + request.settled = true + const orderedRows = referenceWindowRows( + [...rows.values()], + scenario.direction, + { offset: 0, limit: rows.size }, + ) + begin() + for (const row of rowsForLoadSubset(orderedRows, request.options)) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.deferred.resolve() + await flushPromises() + } + + const track = (result: true | Promise): void => { + if (result instanceof Promise) outstanding.push(result) + } + + try { + outstanding.push(live.preload()) + expect(pending).toHaveLength(1) + + updateFirstDelivered(scenario.firstRank) + track(live.utils.setWindow({ offset: 0, limit: scenario.narrowLimit })) + track(live.utils.setWindow({ offset: 0, limit: scenario.wideLimit })) + expect(pending.length).toBeGreaterThan(0) + updateFirstDelivered(scenario.secondRank) + + await settle(pending[0]!) + for (let index = 1; index < pending.length; index++) { + if (index > rows.size * 4) { + throw new Error( + `Ordered continuation exceeded its finite source work bound: ${JSON.stringify( + pending.map(({ options }) => ({ + limit: options.limit, + offset: options.offset, + lastKey: options.cursor?.lastKey, + })), + )}`, + ) + } + await settle(pending[index]!) + } + await Promise.all(outstanding) + + try { + const actual = Array.from(live.values(), ({ id, rank }) => ({ id, rank })) + const expected = referenceWindowRows( + [...rows.values()], + scenario.direction, + { offset: 0, limit: scenario.wideLimit }, + ) + expect(actual).toEqual(expected) + } catch (error) { + throw new PendingHistoryTraceAssertionError( + error, + referenceWindowRows( + [...rows.values()].filter(({ id }) => deliveredIds.has(id)), + scenario.direction, + { offset: 0, limit: scenario.wideLimit }, + ), + ) + } + } finally { + for (const request of pending) request.deferred.resolve() + await Promise.allSettled(outstanding) + await cleanupAll(live, source) + } +} + +function changedRankValue(previous: number, requested: number): number { + return requested === previous + ? requested === 2 + ? -2 + : requested + 1 + : requested +} + +async function expectInflightRequestFillsNewWindow(): Promise { + const rows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + { id: 3, rank: 2 }, + { id: 4, rank: 3 }, + ] + const pending: Array = [] + const deliveredIds = new Set([1]) + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-late-window-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + begin() + write({ type: `insert`, value: { ...rows[0]! } }) + commit() + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .orderBy(({ row }) => row.id, `asc`) + .limit(2), + ) + + const settle = async (request: PendingCursorLoad) => { + begin() + for (const row of rowsForLoadSubset(rows, request.options)) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.deferred.resolve() + await Promise.resolve() + } + + try { + const preload = live.preload() + expect(pending).toHaveLength(1) + const setWindow = live.utils.setWindow({ offset: 2, limit: 2 }) + expect(setWindow).toBeInstanceOf(Promise) + await flushPromises() + expect(pending.length).toBeGreaterThan(0) + + for (let index = 0; index < pending.length; index++) { + if (index > rows.length * 2) { + throw new Error(`Ordered continuation exceeded its work bound`) + } + await settle(pending[index]!) + await flushPromises() + } + await preload + if (setWindow instanceof Promise) await setWindow + + try { + expect(Array.from(live.values(), ({ id }) => id)).toEqual([3, 4]) + } catch (error) { + throw new TraceAssertionError(0, error) + } + } finally { + for (const request of pending) request.deferred.resolve() + await cleanupAll(live, source) + } +} + +describe(`pagination recomputation oracle`, () => { + it.each([ + { label: `Error`, reason: new Error(`first cleanup failed`) }, + { label: `undefined`, reason: undefined }, + { label: `null`, reason: null }, + { label: `false`, reason: false }, + { label: `zero`, reason: 0 }, + { label: `NaN`, reason: Number.NaN }, + { label: `empty string`, reason: `` }, + ])( + `observes $label cleanup failure after every teardown settles`, + async ({ reason: firstFailure }) => { + const secondFailure = new Error(`second cleanup failed`) + const firstFailureRelease = createDeferred() + const lastCleanupRelease = createDeferred() + const repeatedFirstFailureRelease = createDeferred() + const repeatedLastCleanupRelease = createDeferred() + const events: Array = [] + const unhandled: Array = [] + const recordUnhandled = (reason: unknown) => unhandled.push(reason) + const createTargets = ( + firstRelease: Deferred, + lastRelease: Deferred, + ): ReadonlyArray => [ + { + cleanup: async () => { + events.push(`first`) + await firstRelease.promise + throw firstFailure + }, + }, + { + cleanup: () => { + events.push(`second`) + throw secondFailure + }, + }, + { + cleanup: async () => { + events.push(`third`) + await lastRelease.promise + }, + }, + ] + const observeFirstFailure = (cleanup: Promise) => + cleanup.then( + () => { + throw new Error(`expected cleanup to reject`) + }, + (error: unknown) => expect(error).toBe(firstFailure), + ) + let cleanupFinished = false + process.on(`unhandledRejection`, recordUnhandled) + + try { + const cleanup = cleanupAll( + ...createTargets(firstFailureRelease, lastCleanupRelease), + ).finally(() => { + cleanupFinished = true + }) + const observedFailure = observeFirstFailure(cleanup) + + await flushPromises() + expect(events).toEqual([`first`, `second`, `third`]) + expect(cleanupFinished).toBe(false) + expect(unhandled).toEqual([]) + + firstFailureRelease.resolve() + await flushPromises() + expect(cleanupFinished).toBe(false) + expect(unhandled).toEqual([]) + + lastCleanupRelease.resolve() + await observedFailure + await flushPromises() + expect(cleanupFinished).toBe(true) + expect(unhandled).toEqual([]) + + let repeatedCleanupFinished = false + const repeatedCleanup = cleanupAll( + ...createTargets( + repeatedFirstFailureRelease, + repeatedLastCleanupRelease, + ), + ).finally(() => { + repeatedCleanupFinished = true + }) + const repeatedObservedFailure = observeFirstFailure(repeatedCleanup) + await flushPromises() + expect(events).toEqual([ + `first`, + `second`, + `third`, + `first`, + `second`, + `third`, + ]) + expect(repeatedCleanupFinished).toBe(false) + expect(unhandled).toEqual([]) + + repeatedFirstFailureRelease.resolve() + await flushPromises() + expect(repeatedCleanupFinished).toBe(false) + expect(unhandled).toEqual([]) + + repeatedLastCleanupRelease.resolve() + await repeatedObservedFailure + await flushPromises() + expect(repeatedCleanupFinished).toBe(true) + expect(unhandled).toEqual([]) + } finally { + firstFailureRelease.resolve() + lastCleanupRelease.resolve() + repeatedFirstFailureRelease.resolve() + repeatedLastCleanupRelease.resolve() + process.off(`unhandledRejection`, recordUnhandled) + } + }, + ) + + it(`refills a joined result window through a contract-compliant source`, async () => { + type ParentRow = { id: number; rank: number; groupId: number } + type ChildRow = { id: number; groupId: number } + const parents = [ + { id: 1, rank: 0, groupId: 1 }, + { id: 2, rank: 1, groupId: 2 }, + { id: 3, rank: 2, groupId: 3 }, + { id: 4, rank: 3, groupId: 4 }, + ] satisfies ReadonlyArray + const { requests, source: parentSource } = createConformingOrderedSource( + `pagination-joined-underfill-source-${collectionSequence++}`, + parents, + ) + const childSource = createCollection( + mockSyncCollectionOptions({ + id: `pagination-joined-underfill-child-${collectionSequence++}`, + initialData: [ + { id: 20, groupId: 2 }, + { id: 30, groupId: 3 }, + { id: 40, groupId: 4 }, + ] satisfies ReadonlyArray, + getKey: (row: ChildRow) => row.id, + }), + ) + const live = createLiveQueryCollection((query) => + query + .from({ parent: parentSource }) + .innerJoin({ child: childSource }, ({ parent, child }) => + eq(parent.groupId, child.groupId), + ) + .orderBy(({ parent }) => parent.rank, `asc`) + .orderBy(({ parent }) => parent.id, `asc`) + .limit(2) + .select(({ parent }) => ({ id: parent.id })), + ) + + try { + await live.preload() + await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([2, 3]) + expect(requests).toHaveLength(1) + expect(requests[0]?.limit).toBeUndefined() + } finally { + await cleanupAll(live, childSource, parentSource) + } + }) + + it(`loads the full ordered source when no continuation index exists`, async () => { + type ParentRow = { id: number; rank: number; groupId: number } + type ChildRow = { id: number; groupId: number } + const parents = [ + { id: 1, rank: 0, groupId: 1 }, + { id: 2, rank: 1, groupId: 2 }, + { id: 3, rank: 2, groupId: 3 }, + { id: 4, rank: 3, groupId: 4 }, + ] satisfies ReadonlyArray + const { requests, source: parentSource } = createConformingOrderedSource( + `pagination-no-index-underfill-source-${collectionSequence++}`, + parents, + `off`, + ) + const childSource = createCollection( + mockSyncCollectionOptions({ + id: `pagination-no-index-underfill-child-${collectionSequence++}`, + initialData: [ + { id: 20, groupId: 2 }, + { id: 30, groupId: 3 }, + { id: 40, groupId: 4 }, + ] satisfies ReadonlyArray, + getKey: (row: ChildRow) => row.id, + }), + ) + const live = createLiveQueryCollection((query) => + query + .from({ parent: parentSource }) + .innerJoin({ child: childSource }, ({ parent, child }) => + eq(parent.groupId, child.groupId), + ) + .orderBy(({ parent }) => parent.rank, `asc`) + .orderBy(({ parent }) => parent.id, `asc`) + .limit(2) + .select(({ parent }) => ({ id: parent.id })), + ) + + try { + await live.preload() + await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([2, 3]) + expect(requests).toHaveLength(1) + expect(requests[0]?.limit).toBeUndefined() + } finally { + await cleanupAll(live, childSource, parentSource) + } + }) + + it(`refines a joined foreign order term through the source tie class`, async () => { + type ParentRow = { id: number; sourceRank: number; childId: number } + type ChildRow = { id: number; score: number } + const parents = [ + { id: 1, sourceRank: 0, childId: 1 }, + { id: 2, sourceRank: 0, childId: 2 }, + { id: 3, sourceRank: 0, childId: 3 }, + { id: 4, sourceRank: 0, childId: 4 }, + ] satisfies ReadonlyArray + const { requests, source: parentSource } = createConformingOrderedSource( + `pagination-joined-foreign-order-source-${collectionSequence++}`, + parents, + ) + const childSource = createCollection( + mockSyncCollectionOptions({ + id: `pagination-joined-foreign-order-child-${collectionSequence++}`, + initialData: [ + { id: 1, score: 10 }, + { id: 2, score: 20 }, + { id: 3, score: 0 }, + { id: 4, score: 30 }, + ] satisfies ReadonlyArray, + getKey: (row: ChildRow) => row.id, + }), + ) + const live = createLiveQueryCollection((query) => + query + .from({ parent: parentSource }) + .leftJoin({ child: childSource }, ({ parent, child }) => + eq(parent.childId, child.id), + ) + .orderBy(({ parent }) => parent.sourceRank, `asc`) + .orderBy(({ child }) => child.score, `asc`) + .orderBy(({ parent }) => parent.id, `asc`) + .limit(2) + .select(({ parent }) => ({ id: parent.id })), + ) + + try { + await live.preload() + await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([3, 1]) + expect(requests).toHaveLength(1) + expect(requests[0]?.limit).toBeUndefined() + } finally { + await cleanupAll(live, childSource, parentSource) + } + }) + + it(`materializes an empty source window`, async () => { + await runPaginationScenario({ + ranks: [], + direction: `asc`, + windows: [{ offset: 0, limit: 3 }], + }) + }) + + it(`does not refetch when live insertion fills a settled empty window`, async () => { + let sync!: Parameters[`sync`]>[0] + const requests: Array = [] + const source = createCollection({ + id: `settled-empty-window`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + requests.push(options) + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + try { + await live.preload() + expect(live.toArray).toEqual([]) + expect(requests).toHaveLength(1) + + sync.begin() + sync.write({ type: `insert`, value: { id: 1, rank: 1 } }) + const receipt = sync.commit() + if (receipt !== true) await receipt + await flushPromises() + + expect(live.toArray.map(({ id, rank }) => ({ id, rank }))).toEqual([ + { id: 1, rank: 1 }, + ]) + expect(live.utils.lastSubsetError).toBeUndefined() + // Correct rows alone would miss a repeated prefix and boundary fetch. + expect( + requests.map(({ limit, offset, orderBy, where, cursor }) => ({ + limit, + offset, + ordered: Boolean(orderBy), + filtered: Boolean(where), + cursor: Boolean(cursor), + })), + ).toEqual([ + { limit: 1, offset: 0, ordered: true, filtered: false, cursor: false }, + ]) + } finally { + await cleanupAll(live, source) + } + }) + + it(`materializes an offset past the final row`, async () => { + await runPaginationScenario({ + ranks: [0, 1], + direction: `asc`, + windows: [{ offset: 4, limit: 2 }], + }) + }) + + it(`materializes an initially empty zero-limit window`, async () => { + await runPaginationScenario({ + ranks: [0, 1, 2], + direction: `asc`, + windows: [{ offset: 0, limit: 0 }], + }) + }) + + it(`clears and restores a nonempty window across a zero limit`, async () => { + await runPaginationScenario({ + ranks: [0, 1, 2], + direction: `asc`, + windows: [ + { offset: 0, limit: 2 }, + { offset: 0, limit: 0 }, + { offset: 1, limit: 1 }, + ], + }) + }) + + it(`advances past an implicit public-key tie class`, async () => { + await runPaginationScenario({ + ranks: [1, 2, 3, 4, 5, 5, 5, 5, 5, 5, 11, 12, 13, 14, 15, 16], + direction: `asc`, + explicitPublicKeyOrder: false, + includeFilter: true, + reverseInsertion: true, + windows: [ + { offset: 0, limit: 5 }, + { offset: 5, limit: 5 }, + { offset: 10, limit: 5 }, + ], + }) + }) + + it(`keeps implicit ties stable across filtered source mutations`, async () => { + await runPaginationStateScenario({ + ranks: [0, 0, 0, 1, 1, 1], + direction: `asc`, + explicitPublicKeyOrder: false, + includeFilter: true, + reverseInsertion: true, + initialWindow: { offset: 0, limit: 3 }, + actions: [ + { type: `put`, id: 7, rank: 0 }, + { type: `delete`, id: 2 }, + { type: `window`, offset: 1, limit: 3 }, + ], + }) + }) + + it.each([ + { + name: `enters the filter`, + keeps: [false, true], + action: { type: `put` as const, id: 1, rank: 0, keep: true }, + expected: [1, 2], + }, + { + name: `leaves the filter`, + keeps: [true, true], + action: { type: `put` as const, id: 1, rank: 0, keep: false }, + expected: [2], + }, + ])(`updates a row that $name`, async ({ keeps, action, expected }) => { + const scenario: PaginationStateScenario = { + ranks: [0, 1], + keeps, + direction: `asc`, + includeFilter: true, + explicitPublicKeyOrder: true, + reverseInsertion: false, + initialWindow: { offset: 0, limit: 2 }, + actions: [action], + } + await runPaginationStateScenario(scenario) + + const finalRows = scenario.ranks.map((rank, index) => ({ + id: index + 1, + rank, + keep: index === 0 ? action.keep : keeps[index], + })) + expect( + referenceWindow( + visibleRows(finalRows, true), + scenario.direction, + scenario.initialWindow, + ), + ).toEqual(expected) + }) + + it(`discovered trace: loads an on-demand window after a zero limit`, async () => { + await runOnDemandPaginationScenario({ + ranks: [0, 0], + direction: `asc`, + windows: [ + { offset: 1, limit: 0 }, + { offset: 0, limit: 2 }, + ], + }) + }) + + it(`widens an offset on-demand window after starting at zero limit`, async () => { + await runOnDemandPaginationScenario({ + ranks: [-1, 0, 0, 0, -1, 0], + direction: `asc`, + windows: [ + { offset: 1, limit: 0 }, + { offset: 4, limit: 1 }, + { offset: 4, limit: 2 }, + { offset: 0, limit: 0 }, + ], + }) + }) + + it(`starts an on-demand source prefix after opening a zero window with a local row`, async () => { + await runOnDemandPaginationScenario( + { + ranks: [0, 1], + direction: `asc`, + explicitPublicKeyOrder: false, + windows: [ + { offset: 0, limit: 0 }, + { offset: 0, limit: 1 }, + ], + localRowsBeforeFirstRequest: [{ id: 2, rank: 1 }], + }, + (loads) => { + expect(loads[0]?.cursor).toBeUndefined() + expect(loads[0]?.offset).toBe(0) + expect(loads[0]?.limit).toBe(1) + }, + ) + }) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + ([`sync`, `async`] as const).map((replayDelivery) => ({ + direction, + replayDelivery, + })), + ), + )( + `keeps a failed $direction window private after $replayDelivery source replay until explicit retry`, + async ({ direction, replayDelivery }) => { + const rows: Array = [ + { id: 1, rank: 1 }, + { id: 2, rank: 2 }, + ] + const failure = new Error(`window acquisition failed`) + const replayGate = createDeferred() + let operations!: Parameters[`sync`]>[0] + let loads = 0 + const source = createCollection({ + id: `pagination-failed-window-replay-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (sync) => { + operations = sync + sync.markReady() + return { + loadSubset: (options) => { + loads++ + sync.begin() + for (const row of loads === 1 ? rows.slice(0, 1) : rows) { + sync.write({ type: `insert`, value: { ...row } }) + } + const receipt = sync.commit(options.signal) + if (loads === 1) return Promise.reject(failure) + if (replayDelivery === `sync`) return receipt + return replayGate.promise.then(async () => { + if (receipt !== true) await receipt + }) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, direction) + .limit(0) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + .distinct(), + ) + const publications: Array> = [] + const subscriber = live.subscribeChanges(() => { + publications.push(Array.from(live.values(), ({ id }) => id)) + }) + const assertHeld = () => { + expect(Array.from(live.values())).toEqual([]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 0 }) + expect(publications).toEqual([]) + } + try { + await live.preload() + await expect( + live.utils.setWindow({ offset: 0, limit: 2 }), + ).rejects.toBe(failure) + assertHeld() + operations.begin() + operations.truncate() + const receipt = operations.commit() + if (receipt !== true) await receipt + await flushPromises() + assertHeld() + replayGate.resolve() + await flushPromises() + await flushPromises() + expect(loads).toBe(2) + assertHeld() + + await live.utils.setWindow({ offset: 0, limit: 2 }) + const expected = referenceWindow(rows, direction, { + offset: 0, + limit: 2, + }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual(expected) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(publications).toEqual([expected]) + } finally { + replayGate.resolve() + subscriber.unsubscribe() + await cleanupAll(live, source) + } + }, + ) + + it.each([ + { offset: 0, limit: 1, failureKind: `error` as const }, + { offset: 2, limit: 1, failureKind: `error` as const }, + { offset: 0, limit: 1, failureKind: `abort` as const }, + { offset: 2, limit: 1, failureKind: `abort` as const }, + ])( + `recovers the first $failureKind-rejected ordered request for window $offset:$limit from the full source`, + async ({ failureKind, ...window }) => { + const authoritativeRows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + { id: 3, rank: 2 }, + { id: 4, rank: 3 }, + ] + const requests: Array = [] + const firstRequest = createDeferred() + const deliveredIds = new Set([4]) + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-rejected-first-prefix-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + if (requests.length === 1) return firstRequest.promise + + begin() + for (const row of rowsForLoadSubset( + authoritativeRows, + options, + )) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(0), + ) + + try { + await live.preload() + begin() + write({ type: `insert`, value: { ...authoritativeRows[3]! } }) + commit() + + const requestedPrefix = window.offset + window.limit + const failed = live.utils.setWindow(window) + expect(failed).toBeInstanceOf(Promise) + expect(requests[0]).toMatchObject({ + offset: 0, + limit: requestedPrefix, + }) + expect(requests[0]?.cursor).toBeUndefined() + const failure = + failureKind === `abort` + ? new DOMException(`first ordered request canceled`, `AbortError`) + : new Error(`first ordered request failed`) + firstRequest.reject(failure) + await expect(failed).rejects.toBe(failure) + + const retry = live.utils.setWindow(window) + if (retry instanceof Promise) await retry + expect(requests[1]?.limit).toBeUndefined() + expect(requests[1]?.offset).toBeUndefined() + expect(requests[1]?.cursor).toBeUndefined() + expect(requests).toHaveLength(2) + expect(Array.from(live.values(), ({ id }) => id)).toEqual( + referenceWindow(authoritativeRows, `asc`, window), + ) + } finally { + firstRequest.resolve() + await cleanupAll(live, source) + } + }, + ) + + it.each([`error`, `AbortError`] as const)( + `does not derive a retry cursor from rows written by a %s request`, + async (failureKind) => { + const authoritativeRows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + { id: 3, rank: 2 }, + { id: 4, rank: 99 }, + ] + const requests: Array = [] + const unloaded: Array = [] + const deliveredIds = new Set() + const rejectedPage = createDeferred() + let rejectNextPage = false + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + let truncate!: () => void + const source = createCollection({ + id: `pagination-rejected-partial-page-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + if (rejectNextPage) { + rejectNextPage = false + begin() + deliveredIds.add(4) + write({ type: `insert`, value: { ...authoritativeRows[3]! } }) + commit() + return rejectedPage.promise + } + + begin() + for (const row of rowsForLoadSubset( + authoritativeRows, + options, + )) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + unloadSubset: (options) => unloaded.push(options), + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + const initialRequestCount = requests.length + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + + rejectNextPage = true + const failed = live.utils.setWindow({ offset: 0, limit: 4 }) + expect(failed).toBeInstanceOf(Promise) + const failure = + failureKind === `AbortError` + ? new DOMException(`partial ordered request canceled`, `AbortError`) + : new Error(`partial ordered request failed`) + rejectedPage.reject(failure) + await expect(failed).rejects.toBe(failure) + await flushPromises() + expect(requests).toHaveLength(initialRequestCount + 1) + + // Replay can replace the failed request's physical options object + // before the explicit retry retires its logical demand. + const beforeFailedReplay = requests.length + deliveredIds.clear() + begin() + truncate() + commit() + await flushPromises() + const failedReplayRequests = requests.slice(beforeFailedReplay) + // The settled first row permits a three-row continuation. Replay + // must preserve that exact demand even after its first attempt fails. + const replayedFailedRequest = failedReplayRequests.find( + ({ limit, cursor }) => limit === 3 && cursor !== undefined, + ) + expect(replayedFailedRequest).toBeDefined() + expect(replayedFailedRequest).toMatchObject({ offset: 1, limit: 3 }) + expect(replayedFailedRequest?.cursor).toEqual( + requests[initialRequestCount]?.cursor, + ) + + const releasesBeforeRetry = unloaded.length + const requestsBeforeRetry = requests.length + const retry = live.utils.setWindow({ offset: 0, limit: 2 }) + if (retry instanceof Promise) await retry + expect(unloaded.slice(releasesBeforeRetry)).toEqual([ + replayedFailedRequest, + ]) + expect(requests).toHaveLength(requestsBeforeRetry) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + + const beforeWiden = requests.length + const widen = live.utils.setWindow({ offset: 0, limit: 3 }) + if (widen instanceof Promise) await widen + expect(requests).toHaveLength(beforeWiden) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2, 3]) + + const beforeReplay = requests.length + deliveredIds.clear() + begin() + truncate() + commit() + await flushPromises() + expect( + requests.slice(beforeReplay).every(({ cursor }) => !cursor), + ).toBe(true) + } finally { + rejectedPage.resolve() + await cleanupAll(live, source) + } + }, + ) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + ([`throw`, `reject`] as const).map((delivery) => ({ + direction, + delivery, + })), + ), + )( + `holds a $direction page and concurrent live insert when its boundary refinement fails by $delivery`, + async ({ direction, delivery }) => { + const sign = direction === `asc` ? 1 : -1 + const rows: Array = [ + { id: 1, rank: sign }, + { id: 2, rank: 2 * sign }, + ] + const liveInsert: PageRow = { id: 0, rank: 0 } + const delivered = new Set() + const failure = new Error(`later boundary failed`) + let widening = false + let failedBoundary: LoadSubsetOptions | undefined + let suppliedPage: Array | undefined + const source = createCollection({ + id: `pagination-boundary-publication-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + const selected = rowsForLoadSubset(rows, options) + if ( + widening && + options.where && + !options.orderBy && + selected.some(({ id }) => id === 2) && + !failedBoundary + ) { + failedBoundary = options + if (delivery === `throw`) throw failure + return Promise.reject(failure) + } + const newRows = selected.filter(({ id }) => !delivered.has(id)) + begin() + for (const row of newRows) { + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + if (widening && options.orderBy && !suppliedPage) { + suppliedPage = newRows + rows.push(liveInsert) + delivered.add(liveInsert.id) + write({ type: `insert`, value: { ...liveInsert } }) + } + const receipt = commit(options.signal) + // Make the page asynchronous so the failure is in its later + // refinement, not the synchronous setWindow call stack. + return Promise.resolve(receipt).then(() => undefined) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, direction) + .limit(1), + ) + const publications: Array> = [] + const subscription = live.subscribeChanges(() => { + publications.push(Array.from(live.values(), ({ id }) => id)) + }) + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + publications.length = 0 + widening = true + await expect( + live.utils.setWindow({ offset: 0, limit: 2 }), + ).rejects.toBe(failure) + expect(suppliedPage?.map(({ id }) => id)).toEqual([2]) + expect(failedBoundary).toBeDefined() + expect( + rowsForLoadSubset(rows, failedBoundary!).map(({ id }) => id), + ).toEqual([2]) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + expect(publications).toEqual([]) + await live.utils.setWindow({ offset: 0, limit: 2 }) + const expected = referenceWindow(rows, direction, { + offset: 0, + limit: 2, + }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual(expected) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(publications).toEqual([expected]) + } finally { + subscription.unsubscribe() + await cleanupAll(live, source) + } + }, + ) + + it(`recovers a failed tie boundary from the authoritative full source`, async () => { + const authoritativeRows: Array = [ + { id: 1, rank: -1 }, + // A provider may return equal-order rows in any order. The local public + // key tie-breaker must choose id 2 after boundary refinement. + { id: 4, rank: 0 }, + { id: 3, rank: 0 }, + { id: 2, rank: 0 }, + { id: 6, rank: 1 }, + { id: 5, rank: 99 }, + ] + const deliveredIds = new Set() + const requests: Array = [] + const failedPage = createDeferred() + let rejectNextPage = false + let begin!: () => void + let write!: (message: { type: `insert` | `delete`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-recovered-prefix-tie-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + if (rejectNextPage) { + rejectNextPage = false + begin() + deliveredIds.add(5) + write({ type: `insert`, value: { id: 5, rank: 99 } }) + commit() + return failedPage.promise + } + + begin() + for (const row of rowsForLoadSubset(authoritativeRows, options)) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + + rejectNextPage = true + const failed = live.utils.setWindow({ offset: 0, limit: 3 }) + expect(failed).toBeInstanceOf(Promise) + failedPage.reject(new Error(`later page failed`)) + await expect(failed).rejects.toThrow(`later page failed`) + + const retry = live.utils.setWindow({ offset: 0, limit: 2 }) + if (retry instanceof Promise) await retry + const recoveryRequest = requests.at(-1) + expect(recoveryRequest?.limit).toBeUndefined() + expect(recoveryRequest?.offset).toBeUndefined() + expect(recoveryRequest?.cursor).toBeUndefined() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + + const deleted = authoritativeRows.filter(({ id }) => + [1, 2, 3].includes(id), + ) + for (const row of deleted) { + authoritativeRows.splice(authoritativeRows.indexOf(row), 1) + deliveredIds.delete(row.id) + } + begin() + for (const row of deleted) write({ type: `delete`, value: { ...row } }) + commit() + await flushPromises() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([4, 6]) + } finally { + failedPage.resolve() + await cleanupAll(live, source) + } + }) + + it(`rejects a reentrant window move when an ordered request writes and then throws`, async () => { + const authoritativeRows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + { id: 3, rank: 2 }, + ] + const requests: Array = [] + const deliveredIds = new Set() + const failure = new Error(`ordered request threw after writing`) + let reentrantError: unknown + let throwNextPage = false + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-synchronous-partial-page-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + if (throwNextPage) { + throwNextPage = false + begin() + deliveredIds.add(3) + write({ type: `insert`, value: { ...authoritativeRows[2]! } }) + commit() + try { + live.utils.setWindow({ offset: 0, limit: 3 }) + } catch (error) { + reentrantError = error + } + throw failure + } + + begin() + for (const row of rowsForLoadSubset(authoritativeRows, options)) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + const initialRequestCount = requests.length + throwNextPage = true + + expect(() => live.utils.setWindow({ offset: 0, limit: 2 })).toThrow( + failure, + ) + expect(reentrantError).toMatchObject({ name: `SetWindowReentrancyError` }) + expect(requests).toHaveLength(initialRequestCount + 1) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + + const retry = live.utils.setWindow({ offset: 0, limit: 2 }) + if (retry instanceof Promise) await retry + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + } finally { + await cleanupAll(live, source) + } + }) + + it.each( + ([`sync`, `async`] as const).flatMap((delivery) => + [1, 2].map((requestNumber) => ({ delivery, requestNumber })), + ), + )( + `rejects a window move reentered from startup request $requestNumber after $delivery delivery`, + async ({ delivery, requestNumber }) => { + const authoritativeRows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + ] + const delivered = new Set() + let nestedResult: true | Promise | undefined + let nestedError: unknown + let requests = 0 + let firstRequestSettled = false + function createWindowedQuery() { + return createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + } + const source = createCollection({ + id: `pagination-initial-request-reentrancy-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + requests++ + if (requests === requestNumber) { + if (delivery === `async` && requestNumber === 2) { + expect(firstRequestSettled).toBe(true) + } + try { + nestedResult = live.utils.setWindow({ offset: 0, limit: 2 }) + } catch (error) { + nestedError = error + } + } + const fresh = rowsForLoadSubset( + authoritativeRows, + options, + ).filter(({ id }) => !delivered.has(id)) + if (fresh.length === 0) return true + begin() + for (const row of fresh) { + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return delivery === `async` + ? Promise.resolve().then(() => { + firstRequestSettled = true + }) + : true + }, + } + }, + }, + }) + const live = createWindowedQuery() + + try { + await live.preload() + expect(requests).toBeGreaterThanOrEqual(requestNumber) + expect(nestedResult).toBeUndefined() + expect(nestedError).toMatchObject({ name: `SetWindowReentrancyError` }) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + } finally { + await cleanupAll(live, source) + } + }, + ) + + it(`rejects a window move reentered from a public change callback`, async () => { + const authoritativeRows: Array = [ + { id: 1, rank: 0, keep: true }, + { id: 2, rank: 1, keep: true }, + ] + const delivered = new Set() + let begin!: () => void + let write!: (message: { + type: `update` + value: PageRow + previousValue: PageRow + }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-publication-reentrancy-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options) => { + const fresh = rowsForLoadSubset( + authoritativeRows, + options, + ).filter(({ id }) => !delivered.has(id)) + if (fresh.length === 0) return true + begin() + for (const row of fresh) { + delivered.add(row.id) + operations.write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + let nestedResult: true | Promise | undefined + let nestedError: unknown + + try { + await live.preload() + const subscription = live.subscribeChanges(() => { + try { + nestedResult = live.utils.setWindow({ offset: 0, limit: 2 }) + } catch (error) { + nestedError = error + } + }) + const previous = authoritativeRows[0]! + const current = { ...previous, keep: false } + authoritativeRows[0] = current + begin() + write({ type: `update`, value: current, previousValue: previous }) + commit() + subscription.unsubscribe() + + expect(nestedResult).toBeUndefined() + expect(nestedError).toMatchObject({ name: `SetWindowReentrancyError` }) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + } finally { + await cleanupAll(live, source) + } + }) + + it.each([`return-only`, `write-after-cleanup`])( + `does not settle a window move after its sync session is cleaned up: %s`, + async (delivery) => { + const authoritativeRows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + ] + const delivered = new Set() + let cleanUpDuringNextRequest = false + let cleanupPromise: Promise | undefined + function createWindowedQuery() { + return createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + } + const source = createCollection({ + id: `pagination-window-cleanup-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + if (cleanUpDuringNextRequest) { + cleanUpDuringNextRequest = false + cleanupPromise = live.cleanup() + if (delivery === `return-only`) return true + } + const fresh = rowsForLoadSubset( + authoritativeRows, + options, + ).filter(({ id }) => !delivered.has(id)) + if (fresh.length === 0) return true + begin() + for (const row of fresh) { + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + } + }, + }, + }) + const live = createWindowedQuery() + + try { + await live.preload() + cleanUpDuringNextRequest = true + const move = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(cleanUpDuringNextRequest).toBe(false) + expect(cleanupPromise).toBeInstanceOf(Promise) + await cleanupPromise + + expect(move).toBeInstanceOf(Promise) + await expect(move).rejects.toMatchObject({ name: `AbortError` }) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + } finally { + await cleanupAll(live, source) + } + }, + ) + + it(`tracks an asynchronous prefix refresh after synchronous satisfaction`, async () => { + const rows: Array = [ + { id: 1, rank: 1 }, + { id: 2, rank: 2 }, + { id: 3, rank: 3 }, + ] + const requests: Array = [] + const delivered = new Set() + const refinement = createDeferred() + let deferLoads = false + const source = createCollection({ + id: `pagination-async-refinement-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + const publish = (options: LoadSubsetOptions) => { + begin() + for (const row of rowsForLoadSubset(rows, options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + } + + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + if (!deferLoads) { + publish(options) + return true + } + + return refinement.promise.then(() => publish(options)) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .orderBy(({ row }) => row.id, `asc`) + .limit(1), + ) + + try { + await live.preload() + await flushPromises() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + const initialRequestCount = requests.length + expect(initialRequestCount).toBeGreaterThan(0) + expect( + requests.every( + ({ limit, where }) => limit !== undefined || where !== undefined, + ), + ).toBe(true) + deferLoads = true + + const widened = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(widened).toBeInstanceOf(Promise) + await flushPromises() + expect(requests.length).toBeGreaterThan(initialRequestCount) + const widenedRequest = requests + .slice(initialRequestCount) + .find(({ limit }) => limit === 2) + expect(widenedRequest).toBeDefined() + expect(widenedRequest?.offset).toBeUndefined() + expect(widenedRequest?.cursor).toBeUndefined() + const settledBeforeRefinement = await Promise.race([ + Promise.resolve(widened).then(() => true), + new Promise((resolve) => setTimeout(() => resolve(false), 10)), + ]) + expect(settledBeforeRefinement).toBe(false) + + refinement.resolve() + if (widened instanceof Promise) await widened + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + } finally { + await cleanupAll(live, source) + } + }) + + it(`refines locale-ordered continuations locally when predicate IR cannot express the collation`, async () => { + const rows: Array = [ + { id: 1, label: `item2` }, + { id: 2, label: `item10` }, + { id: 3, label: `item11` }, + ] + const pending: Array = [] + const delivered = new Set() + let begin!: () => void + let write!: (message: { type: `insert`; value: LocaleCursorRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-locale-cursor-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.label, { + direction: `asc`, + nulls: `first`, + stringSort: `locale`, + locale: `en-US`, + localeOptions: { numeric: true }, + }) + .orderBy(({ row }) => row.id, `asc`) + .limit(1), + ) + + try { + const preload = live.preload() + expect(pending).toHaveLength(1) + // Settling one request can append its boundary-refinement request. + // eslint-disable-next-line @typescript-eslint/prefer-for-of + for (let index = 0; index < pending.length; index++) { + const request = pending[index]! + begin() + for (const row of rowsForLoadSubset(rows, request.options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.deferred.resolve() + await flushPromises() + } + await preload + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(pending.length).toBeLessThanOrEqual(rows.length * 2) + expect( + pending.some( + ({ options }) => + options.limit === undefined && options.where === undefined, + ), + ).toBe(true) + + const transportCount = pending.length + const widened = live.utils.setWindow({ offset: 0, limit: 2 }) + for (let index = transportCount; index < pending.length; index++) { + const request = pending[index]! + begin() + for (const row of rowsForLoadSubset(rows, request.options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.deferred.resolve() + await flushPromises() + } + if (widened instanceof Promise) await widened + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(pending.length).toBeLessThanOrEqual(rows.length * 3) + } finally { + for (const request of pending) request.deferred.resolve() + await cleanupAll(live, source) + } + }) + + const nullableBoundaryRows: ReadonlyArray = [ + { id: 1, primary: null, secondary: 2 }, + { id: 2, primary: null, secondary: 0 }, + { id: 3, primary: null, secondary: 1 }, + { id: 4, primary: 1, secondary: null }, + { id: 5, primary: 1, secondary: 0 }, + { id: 6, primary: 2, secondary: 0 }, + ] + + it.each([ + [ + `discovered trace: orders an ascending nullable boundary by its second term`, + { + rows: nullableBoundaryRows, + primary: { direction: `asc`, nulls: `first` }, + secondary: { direction: `asc`, nulls: `first` }, + limit: 1, + }, + ], + [ + `orders a descending nullable boundary by its second term`, + { + rows: nullableBoundaryRows, + primary: { direction: `desc`, nulls: `first` }, + secondary: { direction: `desc`, nulls: `first` }, + limit: 1, + }, + ], + [ + `orders an ascending and descending mixed nullable boundary`, + { + rows: nullableBoundaryRows, + primary: { direction: `asc`, nulls: `first` }, + secondary: { direction: `desc`, nulls: `first` }, + limit: 1, + }, + ], + [ + `discovered trace: orders a descending and ascending mixed nullable boundary`, + { + rows: nullableBoundaryRows, + primary: { direction: `desc`, nulls: `first` }, + secondary: { direction: `asc`, nulls: `first` }, + limit: 1, + }, + ], + [ + `uses the public key to break a complete tuple tie`, + { + rows: [ + { id: 2, primary: 0, secondary: 0 }, + { id: 1, primary: 0, secondary: 0 }, + ], + primary: { direction: `asc`, nulls: `last` }, + secondary: { direction: `asc`, nulls: `last` }, + limit: 1, + }, + ], + [ + `discovered trace: places nulls last in an ascending nullable boundary`, + { + rows: nullableBoundaryRows, + primary: { direction: `asc`, nulls: `last` }, + secondary: { direction: `asc`, nulls: `last` }, + limit: 1, + }, + ], + [ + `places nulls last in a descending nullable boundary`, + { + rows: nullableBoundaryRows, + primary: { direction: `desc`, nulls: `last` }, + secondary: { direction: `desc`, nulls: `last` }, + limit: 1, + }, + ], + ] satisfies ReadonlyArray)( + `%s`, + async (_name, scenario) => runMultiOrderScenario(scenario), + ) + + fcTest.prop([multiOrderScenarioArbitrary], { + numRuns: orderedScenarioRuns, + seed: 1663, + })( + `matches multi-column nullable ordering for a fixed seed`, + runMultiOrderScenario, + ) + + fcTest.prop( + [multiOrderScenarioArbitrary], + oracleRandomParameters( + orderedScenarioRuns, + replay, + `pagination.multi-order`, + ), + )( + `matches multi-column nullable ordering for a random or replayed seed`, + runMultiOrderScenario, + ) + + fcTest.prop([nullableCursorScenarioArbitrary], { + numRuns: transitionScenarioRuns, + seed: 1665, + })( + `matches nullable cursor ordering while an async response is pending for a fixed seed`, + runNullableCursorScenario, + ) + + fcTest.prop( + [nullableCursorScenarioArbitrary], + oracleRandomParameters( + transitionScenarioRuns, + replay, + `pagination.nullable-cursor`, + ), + )( + `matches nullable cursor ordering while an async response is pending for a random or replayed seed`, + runNullableCursorScenario, + ) + + it.each([ + [`boundary insert`, { type: `insert`, row: { id: 5, rank: 0.5 } }], + [`visible delete`, { type: `delete`, id: 1 }], + [ + `boundary-crossing rank update`, + { type: `update`, row: { id: 4, rank: 0.5 } }, + ], + ] satisfies ReadonlyArray)( + `%s converges before and after a pending response`, + async (_name, mutation) => { + const scenario: PendingMutationScenario = { + ranks: [0, 1, 2, 3], + direction: `asc`, + limit: 3, + mutation, + responseOutcome: `resolve`, + } + await runPendingMutationScenario(scenario, `before-response`) + await runPendingMutationScenario(scenario, `after-response`) + }, + ) + + it.each([ + [`insert`, { type: `insert`, row: { id: 9, rank: 0.5 } }], + [`delete`, { type: `delete`, id: 1 }], + [`rank update`, { type: `update`, row: { id: 2, rank: 10 } }], + ] satisfies ReadonlyArray)( + `revalidates a finite ordered prefix after a settled SSE %s`, + async (_name, mutation) => { + await runPendingMutationScenario( + { + ranks: [0, 1, 2, 3, 4, 5, 6, 7], + direction: `asc`, + limit: 2, + mutation, + responseOutcome: `resolve`, + }, + `after-response`, + ) + }, + ) + + it(`retains a finite inactive prefix across shrink, SSE, and re-expansion`, async () => { + const rows = new Map( + Array.from({ length: 5 }, (_, index) => [ + index + 1, + { id: index + 1, rank: index + 1 }, + ]), + ) + const delivered = new Set() + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-retained-prefix-live-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const ordered = [...rows.values()].sort( + (left, right) => left.rank - right.rank || left.id - right.id, + ) + const requested = rowsForLoadSubset(ordered, options) + begin() + for (const row of requested) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + const receipt = commit() + return Promise.resolve(receipt) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .orderBy(({ row }) => row.id, `asc`) + .limit(3), + ) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2, 3]) + + await live.utils.setWindow({ offset: 0, limit: 1 }) + const inserted = { id: 9, rank: 2.5 } + rows.set(inserted.id, inserted) + delivered.add(inserted.id) + begin() + write({ type: `insert`, value: inserted }) + commit() + await flushPromises() + + await live.utils.setWindow({ offset: 0, limit: 3 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2, 9]) + } finally { + await cleanupAll(live, source) + } + }) + + it.each([`asc`, `desc`] as const)( + `refreshes from the start when one SSE batch moves the retained prefix (%s)`, + async (direction) => { + const rows = new Map( + Array.from({ length: 6 }, (_, index) => [ + index + 1, + { id: index + 1, rank: index + 1 }, + ]), + ) + const delivered = new Set() + const loads: Array = [] + let begin!: () => void + let write!: (message: { + type: `insert` | `update` + value: PageRow + }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-batch-prefix-refresh-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + loads.push(options) + const settled = new Promise((resolve) => { + queueMicrotask(() => { + const ordered = referenceWindowRows( + [...rows.values()], + direction, + { offset: 0, limit: rows.size }, + ) + begin() + for (const row of rowsForLoadSubset(ordered, options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + resolve() + }) + }) + return settled + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, direction) + .orderBy(({ row }) => row.id, `asc`) + .limit(2), + ) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual( + direction === `asc` ? [1, 2] : [6, 5], + ) + + begin() + const movedIds = direction === `asc` ? [1, 2, 3, 4] : [3, 4, 5, 6] + for (const id of movedIds) { + const row = { + id, + rank: direction === `asc` ? 100 + id : -100 - id, + } + rows.set(id, row) + write({ type: `update`, value: { ...row } }) + } + commit() + for (let index = 0; index < 5; index++) await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual( + direction === `asc` ? [5, 6] : [2, 1], + ) + expect( + loads.some( + ({ limit, cursor }) => limit === 2 && cursor === undefined, + ), + ).toBe(true) + } finally { + await cleanupAll(live, source) + } + }, + ) + + it.each([ + [`insert`, { type: `insert`, row: { id: 7, rank: 0 } }, [7, 1], [7, 1, 2]], + [`update`, { type: `update`, row: { id: 2, rank: -1 } }, [2, 1], [2, 1, 3]], + [`delete`, { type: `delete`, id: 1 }, [2, 3], [2, 3, 4]], + ] satisfies ReadonlyArray< + readonly [ + string, + PendingMutation, + ReadonlyArray, + ReadonlyArray, + ] + >)( + `keeps an SSE %s that arrives during boundary refinement`, + async (_name, mutation, expectedIds, expectedWideIds) => { + const rows = new Map( + Array.from({ length: 6 }, (_, index) => [ + index + 1, + { id: index + 1, rank: index + 1 }, + ]), + ) + const delivered = new Set() + const pending: Array = [] + let begin!: () => void + let write!: (message: { + type: `insert` | `update` | `delete` + value: PageRow + }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-pending-refinement-sse-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .orderBy(({ row }) => row.id, `asc`) + .limit(2), + ) + + const settle = async (request: PendingCursorLoad) => { + const ordered = referenceWindowRows([...rows.values()], `asc`, { + offset: 0, + limit: rows.size, + }) + begin() + for (const row of rowsForLoadSubset(ordered, request.options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.deferred.resolve() + await flushPromises() + } + + try { + const preload = live.preload() + expect(pending).toHaveLength(1) + await settle(pending[0]!) + expect(pending).toHaveLength(2) + + begin() + if (mutation.type === `delete`) { + const row = rows.get(mutation.id)! + rows.delete(mutation.id) + delivered.delete(mutation.id) + write({ type: `delete`, value: { ...row } }) + } else { + rows.set(mutation.row.id, { ...mutation.row }) + delivered.add(mutation.row.id) + write({ type: mutation.type, value: { ...mutation.row } }) + } + commit() + + await settle(pending[1]!) + expect( + pending.some( + ({ options }) => + options.limit === 2 && options.cursor === undefined, + ), + ).toBe(true) + for (let index = 2; index < pending.length; index++) { + await settle(pending[index]!) + } + await preload + + expect(Array.from(live.values(), ({ id }) => id)).toEqual(expectedIds) + + const pendingBeforeWiden = pending.length + const widened = live.utils.setWindow({ offset: 0, limit: 3 }) + await flushPromises() + if (mutation.type === `insert`) { + expect(pending.length).toBeGreaterThan(pendingBeforeWiden) + expect( + pending + .slice(pendingBeforeWiden) + .some(({ options }) => options.limit === 3), + ).toBe(true) + } else { + expect( + pending.some( + ({ options }) => + options.limit === undefined && + options.where === undefined && + options.cursor === undefined, + ), + ).toBe(true) + expect(widened).toBe(true) + expect(pending).toHaveLength(pendingBeforeWiden) + } + for (let index = pendingBeforeWiden; index < pending.length; index++) { + await settle(pending[index]!) + } + if (widened instanceof Promise) await widened + expect(Array.from(live.values(), ({ id }) => id)).toEqual( + expectedWideIds, + ) + } finally { + for (const request of pending) request.deferred.resolve() + await cleanupAll(live, source) + } + }, + ) + + it(`does not use a new row beyond finite coverage as a widening boundary`, async () => { + await runPendingMutationScenario( + { + ranks: [0, 1, 2, 3, 4, 5, 6, 7], + direction: `asc`, + limit: 2, + mutation: { type: `insert`, row: { id: 9, rank: 4.5 } }, + responseOutcome: `resolve`, + }, + `after-response`, + 5, + ) + }) + + it(`discovered trace: a settled rank update refreshes top-k membership`, async () => { + const scenario: PendingMutationScenario = { + ranks: [0, 0, 1], + direction: `desc`, + limit: 1, + mutation: { type: `update`, row: { id: 3, rank: 0 } }, + responseOutcome: `resolve`, + } + await runPendingMutationScenario(scenario, `after-response`) + }) + + it(`a rejected cursor does not treat a live insert as remote coverage`, async () => { + const scenario: PendingMutationScenario = { + ranks: [0, -1, 0], + direction: `asc`, + limit: 1, + mutation: { type: `insert`, row: { id: 4, rank: 0 } }, + responseOutcome: `reject`, + } + + await runPendingMutationScenario(scenario, `before-response`) + }) + + fcTest.prop([pendingMutationScenarioArbitrary, responseTimingArbitrary], { + numRuns: transitionScenarioRuns, + seed: 1660, + })( + `matches recomputation when source mutations cross a pending cursor response for a fixed seed`, + runPendingMutationScenario, + ) + + it.each( + ([`insert`, `update`, `delete`] as const).flatMap((mutationKind) => + ([`resolve`, `reject`] as const).flatMap((responseOutcome) => + ([`before-response`, `after-response`] as const).map( + (timing) => [mutationKind, responseOutcome, timing] as const, + ), + ), + ), + )( + `covers pending %s with a %s response %s deterministically`, + async (mutationKind, responseOutcome, timing) => { + const mutation: PendingMutation = + mutationKind === `insert` + ? { type: `insert`, row: { id: 5, rank: -1 } } + : mutationKind === `update` + ? { type: `update`, row: { id: 2, rank: -1 } } + : { type: `delete`, id: 2 } + await runPendingMutationScenario( + { + ranks: [0, 1, 2, 3], + direction: `asc`, + limit: 2, + mutation, + responseOutcome, + }, + timing, + ) + }, + ) + + fcTest.prop( + [pendingMutationScenarioArbitrary, responseTimingArbitrary], + oracleRandomParameters( + transitionScenarioRuns, + replay, + `pagination.pending-mutation`, + ), + )( + `matches recomputation when source mutations cross a pending cursor response for a random or replayed seed`, + runPendingMutationScenario, + ) + + it( + `discovered trace: retries a rejected cursor after a source and window transition`, + runRejectedCursorRetryAfterMutation, + ) + + fcTest.prop([pendingHistoryScenarioArbitrary], { + numRuns: transitionScenarioRuns, + seed: 1664, + })( + `matches recomputation across multi-action pending histories for a fixed seed`, + runPendingHistoryScenario, + ) + + fcTest.prop( + [pendingHistoryScenarioArbitrary], + oracleRandomParameters( + transitionScenarioRuns, + replay, + `pagination.pending-history`, + ), + )( + `matches recomputation across multi-action pending histories for a random or replayed seed`, + runPendingHistoryScenario, + ) + + it( + `discovered trace: an in-flight request does not underfill a new window`, + expectInflightRequestFillsNewWindow, + ) + + it(`discovered trace: a row moving across an offset window must refill its boundary`, async () => { + const scenario: PaginationStateScenario = { + ranks: [0, 0, 0], + direction: `desc`, + initialWindow: { offset: 1, limit: 1 }, + actions: [{ type: `put`, id: 1, rank: -1 }], + } + await runPaginationStateScenario(scenario) + }) + + it(`retains authoritative rows when a later window admits a prior insert`, async () => { + const scenario: PaginationStateScenario = { + ranks: [0, 0], + direction: `asc`, + initialWindow: { offset: 0, limit: 1 }, + actions: [ + { type: `put`, id: 3, rank: 1 }, + { type: `window`, offset: 0, limit: 3 }, + ], + } + await runPaginationStateScenario(scenario) + }) + + it(`restores an out-of-window insert when a later offset selects it`, async () => { + const scenario: PaginationStateScenario = { + ranks: [0, 0], + direction: `asc`, + initialWindow: { offset: 0, limit: 1 }, + actions: [ + { type: `put`, id: 3, rank: 1 }, + { type: `window`, offset: 2, limit: 1 }, + ], + } + await runPaginationStateScenario(scenario) + }) + + it(`restores an out-of-window rank update when a later offset selects it`, async () => { + const scenario: PaginationStateScenario = { + ranks: [0, 0, 0], + direction: `asc`, + initialWindow: { offset: 0, limit: 1 }, + actions: [ + { type: `put`, id: 2, rank: 1 }, + { type: `window`, offset: 2, limit: 1 }, + ], + } + await runPaginationStateScenario(scenario) + }) + + it(`discovered trace: inserting at an empty offset boundary refills the window`, async () => { + const scenario: PaginationStateScenario = { + ranks: [0, 0], + direction: `asc`, + initialWindow: { offset: 0, limit: 1 }, + actions: [ + { type: `put`, id: 3, rank: 1 }, + { type: `window`, offset: 3, limit: 1 }, + { type: `put`, id: 4, rank: 1 }, + ], + } + await runPaginationStateScenario(scenario) + }) + + it(`discovered trace: an insert before a later offset does not skip its new boundary`, async () => { + const scenario: PaginationStateScenario = { + ranks: [-1, -1, 0, -1, 0, -1, 0, 1, 1], + direction: `asc`, + initialWindow: { offset: 0, limit: 3 }, + actions: [ + { type: `put`, id: 10, rank: 0 }, + { type: `window`, offset: 8, limit: 1 }, + ], + } + + await runPaginationStateScenario(scenario) + }) + + it(`discovered trace: an async cursor loads the full offset window`, async () => { + const scenario: PaginationScenario = { + ranks: [0, 0, 0, 0, 0, 0, 1], + direction: `asc`, + windows: [ + { offset: 0, limit: 1 }, + { offset: 2, limit: 5 }, + ], + } + await runOnDemandPaginationScenario(scenario) + }) + + it(`discovered trace: an async cursor crosses an offset before filling one row`, async () => { + const scenario: PaginationScenario = { + ranks: [0, 0, -1], + direction: `asc`, + windows: [ + { offset: 0, limit: 1 }, + { offset: 2, limit: 1 }, + ], + } + await runOnDemandPaginationScenario(scenario) + }) + + it.each( + [`asc`, `desc`].flatMap((direction) => + [false, true].flatMap((explicitPublicKeyOrder) => + [false, true].map((includeFilter) => ({ + direction: direction as `asc` | `desc`, + explicitPublicKeyOrder, + includeFilter, + })), + ), + ), + )( + `bounds requests for an underfilled source: $direction, explicit key=$explicitPublicKeyOrder, filter=$includeFilter`, + async (structure) => { + await runOnDemandPaginationScenario({ + ...structure, + ranks: [0, 0], + keeps: [true, false], + windows: [{ offset: 0, limit: 3 }], + }) + }, + ) + + it.each( + paginationStructures.map((structure, index) => ({ + name: `key=${structure.explicitPublicKeyOrder ? `explicit` : `implicit`}, filter=${structure.includeFilter ? `on` : `off`}, insertion=${structure.reverseInsertion ? `reverse` : `forward`}`, + structure, + index, + })), + )(`covers $name`, async ({ structure, index }) => { + const cellRuns = Math.max(1, Math.ceil(transitionScenarioRuns / 8)) + await fc.assert( + fc.asyncProperty(scenarioPayloadArbitrary, async (scenario) => { + const complete = { ...scenario, ...structure } + await runPaginationScenario(complete) + await runOnDemandPaginationScenario(complete) + }), + { numRuns: cellRuns, seed: 16_570 + index }, + ) + await fc.assert( + fc.asyncProperty(stateScenarioPayloadArbitrary, async (scenario) => { + await runPaginationStateScenario({ ...scenario, ...structure }) + }), + { numRuns: cellRuns, seed: 16_580 + index }, + ) + }) + + fcTest.prop([scenarioArbitrary], { + numRuns: orderedScenarioRuns, + seed: 1657, + })( + `matches full recomputation across ordered windows for a fixed seed`, + runPaginationScenario, + ) + + fcTest.prop( + [scenarioArbitrary], + oracleRandomParameters( + orderedScenarioRuns, + replay, + `pagination.ordered-window`, + ), + )( + `matches full recomputation across ordered windows for a random or replayed seed`, + runPaginationScenario, + ) + + fcTest.prop([stateScenarioArbitrary], { + numRuns: transitionScenarioRuns, + seed: 1658, + })( + `matches full recomputation across source and window transitions for a fixed seed`, + runPaginationStateScenario, + ) + + fcTest.prop( + [stateScenarioArbitrary], + oracleRandomParameters( + transitionScenarioRuns, + replay, + `pagination.window-transition`, + ), + )( + `matches full recomputation across source and window transitions for a random or replayed seed`, + runPaginationStateScenario, + ) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + ([`pages`, `widen`] as const).flatMap((mode) => + [3, 10].map((pageSize) => ({ direction, mode, pageSize })), + ), + ), + )( + `fetches linear row volume while traversing settled pages: %j`, + async ({ direction, mode, pageSize }) => { + const pageCount = 10 + const rows = Array.from({ length: pageCount * pageSize }, (_, rank) => ({ + id: rank + 1, + rank, + })) + const ordered = direction === `asc` ? rows : [...rows].reverse() + const { source, requests } = createConformingOrderedSource( + `pagination-transfer-${collectionSequence++}`, + ordered, + ) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, direction) + .limit(pageSize), + ) + try { + await live.preload() + for (let page = 0; page < pageCount; page++) { + const offset = mode === `pages` ? page * pageSize : 0 + const limit = mode === `pages` ? pageSize : (page + 1) * pageSize + if (page > 0) await live.utils.setWindow({ offset, limit }) + expect([...live.values()].map(projectPageRow)).toEqual( + ordered.slice(offset, offset + limit), + ) + } + // Count every provider-returned row, including duplicates and tie + // probes. Request counts alone cannot detect repeated growing prefixes. + const returnedRows = requests.reduce( + (total, request) => + total + rowsForLoadSubset(ordered, request).length, + 0, + ) + expect(returnedRows).toBeLessThanOrEqual(rows.length + 2 * pageCount) + expect(requests.some((request) => request.cursor !== undefined)).toBe( + true, + ) + } finally { + await live.cleanup() + await source.cleanup() + } + }, + ) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + [false, true].flatMap((explicitPublicKeyOrder) => + [false, true].flatMap((tied) => + [1, 2].map((limit) => ({ + direction, + explicitPublicKeyOrder, + tied, + limit, + })), + ), + ), + ), + )( + `loads the source prefix when moving past an intervening insert: %j`, + async ({ direction, explicitPublicKeyOrder, tied, limit }) => { + const sign = direction === `asc` ? 1 : -1 + await runPaginationStateScenario({ + direction, + initialWindow: { offset: 0, limit: 1 }, + actions: [ + { type: `put`, id: 3, rank: sign * (tied ? 1 : 2), keep: false }, + { type: `window`, offset: 1, limit }, + { type: `put`, id: 1, rank: 0, keep: false }, + ], + ranks: [0, sign], + keeps: [false, false], + explicitPublicKeyOrder, + includeFilter: false, + reverseInsertion: false, + }) + }, + ) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + ([`before-response`, `after-response`] as const).flatMap((timing) => + [0.5, 100].flatMap((rank) => + ([`cursor`, `offset`, `key`] as const).map((transport) => ({ + direction, + timing, + rank, + transport, + })), + ), + ), + ), + )( + `keeps live observations separate from a settled acquisition: %j`, + async ({ direction, timing, rank, transport }) => { + const sign = direction === `asc` ? 1 : -1 + await runPendingMutationScenario( + { + ranks: [0, sign, 2 * sign, 3 * sign], + direction, + limit: 1, + mutation: { type: `insert`, row: { id: 9, rank: sign * rank } }, + responseOutcome: `resolve`, + }, + timing, + 3, + false, + transport, + ) + }, + ) + + it(`discovered trace: a rank update must refill a top-1 window`, async () => { + const scenario: PaginationStateScenario = { + ranks: [0, 0], + direction: `asc`, + initialWindow: { offset: 0, limit: 1 }, + actions: [{ type: `put`, id: 1, rank: 1 }], + } + await runPaginationStateScenario(scenario) + }) + + it(`refills an implicit tie window when a visible row moves below it`, async () => { + await runPaginationStateScenario({ + ranks: [0, 0, 0, -1, 0], + direction: `desc`, + explicitPublicKeyOrder: false, + includeFilter: false, + reverseInsertion: false, + initialWindow: { offset: 0, limit: 4 }, + actions: [{ type: `put`, id: 1, rank: -2, keep: false }], + }) + }) + + it.each([`resolve`, `reject`] as const)( + `keeps the last complete implicit window while background recovery %ss`, + async (settlement) => { + const authoritativeRows = new Map([ + [1, { id: 1, rank: 0, keep: true }], + [2, { id: 2, rank: 1, keep: true }], + ]) + const recovery = createDeferred() + const recoveryError = new Error(`background recovery failed`) + const loads: Array = [] + const delivered = new Set() + let recovering = false + let begin!: () => void + let write!: (message: { + type: `insert` | `update` + value: PageRow + }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-background-prefix-recovery-${collectionSequence++}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + loads.push(options) + const isFullSource = + options.where === undefined && + options.limit === undefined && + options.cursor === undefined + const applyRows = () => { + const rows = rowsForLoadSubset( + [...authoritativeRows.values()], + options, + ) + begin() + for (const row of rows) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + } + if (!recovering || !isFullSource) { + applyRows() + return true + } + return recovery.promise.then(() => { + if (settlement === `reject`) throw recoveryError + applyRows() + }) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1) + .select(({ row }) => ({ + id: row.id, + rank: row.rank, + keep: row.keep, + })), + ) + const publications: Array> = [] + const subscription = live.subscribeChanges( + () => publications.push(live.toArray.map(projectPageRow)), + { includeInitialState: false }, + ) + + try { + await live.preload() + expect(live.toArray.map(projectPageRow)).toEqual([{ id: 1, rank: 0 }]) + publications.length = 0 + const loadsBeforeMutation = loads.length + + recovering = true + const moved = { id: 1, rank: 10, keep: true } + authoritativeRows.set(1, moved) + begin() + write({ type: `update`, value: { ...moved } }) + commit() + await flushPromises() + + const recoveryLoads = loads.slice(loadsBeforeMutation) + expect(recoveryLoads).toHaveLength(1) + expect(recoveryLoads[0]?.where).toBeUndefined() + expect(recoveryLoads[0]?.limit).toBeUndefined() + expect(recoveryLoads[0]?.cursor).toBeUndefined() + expect(live.toArray.map(projectPageRow)).toEqual([{ id: 1, rank: 0 }]) + expect(publications).toEqual([]) + + recovery.resolve() + await flushPromises() + + if (settlement === `resolve`) { + expect(live.toArray.map(projectPageRow)).toEqual([{ id: 2, rank: 1 }]) + expect(publications).toEqual([[{ id: 2, rank: 1 }]]) + expect(live.utils.lastSubsetError).toBeUndefined() + } else { + expect(live.toArray.map(projectPageRow)).toEqual([{ id: 1, rank: 0 }]) + expect(publications).toEqual([]) + expect(live.utils.lastSubsetError).toBe(recoveryError) + } + } finally { + recovery.resolve() + subscription.unsubscribe() + await cleanupAll(live, source) + } + }, + ) + + it(`does not recover the full source when a visible row keeps its order`, async () => { + const loads: Array = [] + let begin!: () => void + let write!: (message: { type: `insert` | `update`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-stable-order-update-${collectionSequence++}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + loads.push(options) + begin() + write({ + type: `insert`, + value: { id: 1, rank: 0, keep: true }, + }) + commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + const loadsBeforeMutation = loads.length + begin() + write({ type: `update`, value: { id: 1, rank: 0, keep: false } }) + commit() + await flushPromises() + + expect( + live.toArray.map(({ id, rank, keep }) => ({ id, rank, keep })), + ).toEqual([{ id: 1, rank: 0, keep: false }]) + expect(loads).toHaveLength(loadsBeforeMutation) + } finally { + await cleanupAll(live, source) + } + }) + + it.each([ + [`top-one`, [0, 0], { offset: 0, limit: 1 }, 1, 1], + [`offset`, [0, 0, 1], { offset: 1, limit: 1 }, 2, 2], + ] as const)( + `refills an implicit %s tie window after a rank update`, + async (_name, ranks, initialWindow, id, rank) => { + await runPaginationStateScenario({ + ranks: [...ranks], + direction: `asc`, + explicitPublicKeyOrder: false, + includeFilter: false, + reverseInsertion: false, + initialWindow, + actions: [{ type: `put`, id, rank, keep: false }], + }) + }, + ) + + it(`opens an implicit tie window from zero at the lowest public key`, async () => { + await runPaginationStateScenario({ + ranks: [0], + direction: `asc`, + explicitPublicKeyOrder: false, + includeFilter: false, + reverseInsertion: false, + initialWindow: { offset: 0, limit: 0 }, + actions: [ + { type: `put`, id: 2, rank: 0, keep: false }, + { type: `window`, offset: 0, limit: 1 }, + { type: `delete`, id: 1 }, + ], + }) + }) + + it(`ignores an out-of-window insert when refilling after a delete`, async () => { + const scenario: PaginationStateScenario = { + ranks: [100, 90, 80, 70], + direction: `desc`, + initialWindow: { offset: 0, limit: 3 }, + actions: [ + { type: `put`, id: 5, rank: 10 }, + { type: `delete`, id: 2 }, + ], + } + await runPaginationStateScenario(scenario) + }) + + it(`ignores an out-of-window rank update when refilling after a delete`, async () => { + const scenario: PaginationStateScenario = { + ranks: [0, 0, 0], + direction: `asc`, + initialWindow: { offset: 0, limit: 1 }, + actions: [ + { type: `put`, id: 2, rank: 1 }, + { type: `delete`, id: 1 }, + ], + } + + await runPaginationStateScenario(scenario) + }) + + it(`ignores an out-of-window rank update when the visible row leaves`, async () => { + const scenario: PaginationStateScenario = { + ranks: [0, 0, 0], + direction: `asc`, + initialWindow: { offset: 0, limit: 1 }, + actions: [ + { type: `put`, id: 2, rank: 1 }, + { type: `put`, id: 1, rank: 2 }, + ], + } + + await runPaginationStateScenario(scenario) + }) + + it(`refills untouched rows when widening after an out-of-window rank update`, async () => { + const scenario: PaginationStateScenario = { + ranks: [0, 0, 0], + direction: `desc`, + initialWindow: { offset: 0, limit: 1 }, + actions: [ + { type: `put`, id: 2, rank: -1 }, + { type: `window`, offset: 0, limit: 3 }, + ], + } + + await runPaginationStateScenario(scenario) + }) + + it(`rebuilds the full boundary when widening after an out-of-window rank update`, async () => { + const scenario: PaginationStateScenario = { + ranks: [1, 0, 1, 0, 1], + direction: `desc`, + initialWindow: { offset: 0, limit: 1 }, + actions: [ + { type: `put`, id: 3, rank: 0 }, + { type: `window`, offset: 0, limit: 4 }, + ], + } + + await runPaginationStateScenario(scenario) + }) + + it(`ignores an out-of-window insert when widening a tied window`, async () => { + const scenario: PaginationStateScenario = { + ranks: [0, 0], + direction: `asc`, + initialWindow: { offset: 0, limit: 1 }, + actions: [ + { type: `put`, id: 3, rank: 0 }, + { type: `window`, offset: 0, limit: 2 }, + ], + } + await runPaginationStateScenario(scenario) + }) + + it(`expands a multi-column boundary before choosing top-K`, async () => { + await expectMultiOrderBoundaryMatches() + }) + + it(`expands a provider tie before applying the public-key tie-breaker`, async () => { + const loads = await runAdversarialOrderedProviderScenario({ + providerRows: [ + { id: 2, rank: 0, label: `second` }, + { id: 1, rank: 0, label: `first` }, + { id: 3, rank: 1, label: `third` }, + ], + order: { kind: `rank`, direction: `asc`, nulls: `first` }, + limit: 1, + expectedIds: [1], + }) + + expect(loads).toHaveLength(2) + expect(loads[1]?.where).toBeDefined() + expect(loads[1]?.cursor).toBeUndefined() + }) + + it(`does not derive an ordered boundary from another demand's local row`, async () => { + const unrelated = { id: 100, rank: 100, label: `unrelated` } + const loads = await runAdversarialOrderedProviderScenario({ + providerRows: [ + { id: 1, rank: 1, label: `first` }, + { id: 2, rank: 2, label: `second` }, + unrelated, + ], + initialRows: [unrelated], + order: { kind: `rank`, direction: `asc`, nulls: `first` }, + limit: 1, + expectedIds: [1], + }) + + expect(loads[0]?.offset).toBe(0) + expect(loads[0]?.cursor).toBeUndefined() + }) + + it(`refines an initial locale window without trusting provider collation`, async () => { + const loads = await runAdversarialOrderedProviderScenario({ + // Lexical provider order disagrees with locale numeric order. + providerRows: [ + { id: 2, rank: 0, label: `item10` }, + { id: 1, rank: 0, label: `item2` }, + ], + order: { kind: `locale` }, + limit: 1, + expectedIds: [1], + useOffsetWhenAvailable: true, + }) + + expect(loads).toHaveLength(2) + expect(loads[1]?.limit).toBeUndefined() + expect(loads[1]?.offset).toBeUndefined() + expect(loads[1]?.cursor).toBeUndefined() + }) + + it(`refines an initial reference-ordered window locally`, async () => { + const first = { value: `first` } + const second = { value: `second` } + // Fix their runtime reference order before the provider returns the + // opposite prefix. + makeComparator({ direction: `asc`, nulls: `first` })(first, second) + + const loads = await runAdversarialOrderedProviderScenario({ + providerRows: [ + { id: 2, rank: second, label: `second` }, + { id: 1, rank: first, label: `first` }, + ], + order: { kind: `reference` }, + limit: 1, + expectedIds: [1], + useOffsetWhenAvailable: true, + }) + + expect(loads).toHaveLength(2) + expect(loads[1]?.limit).toBeUndefined() + expect(loads[1]?.offset).toBeUndefined() + }) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + ([`first`, `last`] as const).map((nulls) => ({ direction, nulls })), + ), + )( + `refines invalid Date ties with an unbounded local-order request ($direction, nulls $nulls)`, + async ({ direction, nulls }) => { + const invalid = new Date(Number.NaN) + const loads = await runAdversarialOrderedProviderScenario({ + providerRows: [ + { id: 2, rank: invalid, label: `second` }, + { id: 1, rank: invalid, label: `first` }, + ], + order: { kind: `reference`, direction, nulls }, + limit: 1, + expectedIds: [1], + useOffsetWhenAvailable: true, + }) + + expect(loads).toHaveLength(2) + expect(loads[1]?.limit).toBeUndefined() + expect(loads[1]?.offset).toBeUndefined() + }, + ) + + it(`uses an ascending index for a bounded descending demand`, async () => { + const rows: Array = [ + { id: 3, rank: 1 }, + { id: 1, rank: 0 }, + { id: 2, rank: 0 }, + ] + const loads: Array = [] + const loaded = new Set() + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-reversed-index-ties-${collectionSequence++}`, + getKey: (row: PageRow) => row.id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + begin() + for (const row of rowsForLoadSubset(rows, options)) { + if (loaded.has(row.id)) continue + loaded.add(row.id) + write({ type: `insert`, value: row }) + } + commit() + return true + }, + } + }, + }, + }) + source.createIndex((row) => row.rank, { + indexType: BTreeIndex, + options: { + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `locale`, + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `desc`) + .limit(2), + ) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([3, 1]) + expect(loads.length).toBeGreaterThan(0) + expect( + loads.every( + ({ limit, where }) => limit !== undefined || where !== undefined, + ), + JSON.stringify( + loads.map(({ limit, offset, cursor, orderBy, where }) => ({ + limit, + offset, + cursor: cursor !== undefined, + orderBy: orderBy !== undefined, + where: where !== undefined, + })), + ), + ).toBe(true) + } finally { + await cleanupAll(live, source) + } + }) + + it.each([{ ids: [1, Number.NaN] }, { ids: [Number.NaN, 1] }])( + `keeps finite public keys before NaN across insertion order`, + async ({ ids }) => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `pagination-nan-key-order-${collectionSequence++}`, + initialData: ids.map((id) => ({ id, rank: 0 })), + getKey: (row: PageRow) => row.id, + autoIndex: `eager`, + }), + ) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + } finally { + await cleanupAll(live, source) + } + }, + ) + + it(`stabilizes an on-demand window with a NaN public-key tie`, async () => { + const loads = await runAdversarialOrderedProviderScenario({ + providerRows: [ + { id: 1, rank: 0, label: `finite` }, + { id: Number.NaN, rank: 0, label: `nan` }, + ], + order: { kind: `rank`, direction: `asc`, nulls: `first` }, + limit: 1, + expectedIds: [1], + }) + + expect(loads).toHaveLength(2) + }) + + it.each([ + { direction: `asc`, nulls: `first`, expectedIds: [1, 2] }, + { direction: `asc`, nulls: `last`, expectedIds: [2, 3] }, + { direction: `desc`, nulls: `first`, expectedIds: [1, 3] }, + { direction: `desc`, nulls: `last`, expectedIds: [3, 2] }, + ] as const)( + `keeps null placement and $direction across source refinement ($nulls)`, + async ({ direction, nulls, expectedIds }) => { + const providerRows = [ + { id: 1, rank: null, label: `null` }, + { id: 2, rank: 0, label: `zero` }, + { id: 3, rank: 1, label: `one` }, + ].sort( + (left, right) => + compareNullableNumber(left.rank, right.rank, { direction, nulls }) || + left.id - right.id, + ) + await runAdversarialOrderedProviderScenario({ + providerRows, + order: { kind: `rank`, direction, nulls }, + limit: 2, + expectedIds, + }) + }, + ) + + fcTest.prop([scenarioArbitrary], { + numRuns: transitionScenarioRuns, + seed: 1659, + })( + `matches full recomputation when exact async cursor loads widen ordered coverage for a fixed seed`, + runOnDemandPaginationScenario, + ) + + fcTest.prop( + [scenarioArbitrary], + oracleRandomParameters( + transitionScenarioRuns, + replay, + `pagination.async-cursor`, + ), + )( + `matches full recomputation when exact async cursor loads widen ordered coverage for a random or replayed seed`, + runOnDemandPaginationScenario, + ) + + it.each([`forward`, `reverse`] as const)( + `keeps concurrent on-demand windows correct under %s completion`, + expectOnDemandWindowsAreCompletionOrderIndependent, + ) +}) diff --git a/packages/db/tests/query/predicate-utils.test.ts b/packages/db/tests/query/predicate-utils.test.ts deleted file mode 100644 index 1f47eef23b..0000000000 --- a/packages/db/tests/query/predicate-utils.test.ts +++ /dev/null @@ -1,1453 +0,0 @@ -import { describe, expect, it } from 'vitest' -import { - isLimitSubset, - isOffsetLimitSubset, - isOrderBySubset, - isPredicateSubset, - isWhereSubset, - minusWherePredicates, - unionWherePredicates, -} from '../../src/query/predicate-utils' -import { Func, PropRef, Value } from '../../src/query/ir' -import type { - BasicExpression, - OrderBy, - OrderByClause, -} from '../../src/query/ir' -import type { LoadSubsetOptions } from '../../src/types' - -// Helper functions to build expressions more easily -function ref(path: string | Array): PropRef { - return new PropRef(typeof path === `string` ? [path] : path) -} - -function val(value: any): Value { - return new Value(value) -} - -function func(name: string, ...args: Array): Func { - return new Func(name, args) -} - -function eq(left: BasicExpression, right: BasicExpression): Func { - return func(`eq`, left, right) -} - -function gt(left: BasicExpression, right: BasicExpression): Func { - return func(`gt`, left, right) -} - -function gte(left: BasicExpression, right: BasicExpression): Func { - return func(`gte`, left, right) -} - -function lt(left: BasicExpression, right: BasicExpression): Func { - return func(`lt`, left, right) -} - -function lte(left: BasicExpression, right: BasicExpression): Func { - return func(`lte`, left, right) -} - -function and(...args: Array): Func { - return func(`and`, ...args) -} - -function or(...args: Array): Func { - return func(`or`, ...args) -} - -function inOp(left: BasicExpression, values: Array): Func { - return func(`in`, left, val(values)) -} - -function orderByClause( - expression: BasicExpression, - direction: `asc` | `desc` = `asc`, -): OrderByClause { - return { - expression, - compareOptions: { - direction, - nulls: `last`, - stringSort: `lexical`, - }, - } -} - -describe(`isWhereSubset`, () => { - describe(`basic cases`, () => { - it(`should return true for both undefined (all data is subset of all data)`, () => { - expect(isWhereSubset(undefined, undefined)).toBe(true) - }) - - it(`should return false for undefined subset with constrained superset`, () => { - // Requesting ALL data but only loaded SOME data = NOT subset - expect(isWhereSubset(undefined, gt(ref(`age`), val(10)))).toBe(false) - }) - - it(`should return true for constrained subset with undefined superset`, () => { - // Loaded ALL data, so any constrained subset is covered - expect(isWhereSubset(gt(ref(`age`), val(20)), undefined)).toBe(true) - }) - - it(`should return true for identical expressions`, () => { - const expr = gt(ref(`age`), val(10)) - expect(isWhereSubset(expr, expr)).toBe(true) - }) - - it(`should return true for structurally equal expressions`, () => { - expect( - isWhereSubset(gt(ref(`age`), val(10)), gt(ref(`age`), val(10))), - ).toBe(true) - }) - - it(`should return true when subset is false`, () => { - // When subset is false the result will always be the empty set - // and the empty set is a subset of any set - expect(isWhereSubset(val(false), gt(ref(`age`), val(10)))).toBe(true) - }) - }) - - describe(`comparison operators`, () => { - it(`should handle gt: age > 20 is subset of age > 10`, () => { - expect( - isWhereSubset(gt(ref(`age`), val(20)), gt(ref(`age`), val(10))), - ).toBe(true) - }) - - it(`should handle gt: age > 10 is NOT subset of age > 20`, () => { - expect( - isWhereSubset(gt(ref(`age`), val(10)), gt(ref(`age`), val(20))), - ).toBe(false) - }) - - it(`should handle gte: age >= 20 is subset of age >= 10`, () => { - expect( - isWhereSubset(gte(ref(`age`), val(20)), gte(ref(`age`), val(10))), - ).toBe(true) - }) - - it(`should handle lt: age < 10 is subset of age < 20`, () => { - expect( - isWhereSubset(lt(ref(`age`), val(10)), lt(ref(`age`), val(20))), - ).toBe(true) - }) - - it(`should handle lt: age < 20 is NOT subset of age < 10`, () => { - expect( - isWhereSubset(lt(ref(`age`), val(20)), lt(ref(`age`), val(10))), - ).toBe(false) - }) - - it(`should handle lte: age <= 10 is subset of age <= 20`, () => { - expect( - isWhereSubset(lte(ref(`age`), val(10)), lte(ref(`age`), val(20))), - ).toBe(true) - }) - - it(`should handle eq: age = 15 is subset of age > 10`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(15)), gt(ref(`age`), val(10))), - ).toBe(true) - }) - - it(`should handle eq: age = 5 is NOT subset of age > 10`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(5)), gt(ref(`age`), val(10))), - ).toBe(false) - }) - - it(`should handle eq: age = 15 is subset of age >= 15`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(15)), gte(ref(`age`), val(15))), - ).toBe(true) - }) - - it(`should handle eq: age = 15 is subset of age < 20`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(15)), lt(ref(`age`), val(20))), - ).toBe(true) - }) - - it(`should handle mixed operators: gt vs gte`, () => { - expect( - isWhereSubset(gt(ref(`age`), val(10)), gte(ref(`age`), val(10))), - ).toBe(true) - }) - - it(`should handle mixed operators: gte vs gt`, () => { - expect( - isWhereSubset(gte(ref(`age`), val(11)), gt(ref(`age`), val(10))), - ).toBe(true) - expect( - isWhereSubset(gte(ref(`age`), val(10)), gt(ref(`age`), val(10))), - ).toBe(false) - }) - }) - - describe(`IN operator`, () => { - it(`should handle eq vs in: age = 5 is subset of age IN [5, 10, 15]`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(5)), inOp(ref(`age`), [5, 10, 15])), - ).toBe(true) - }) - - it(`should handle eq vs in: age = 20 is NOT subset of age IN [5, 10, 15]`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(20)), inOp(ref(`age`), [5, 10, 15])), - ).toBe(false) - }) - - it(`should handle in vs in: [5, 10] is subset of [5, 10, 15]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [5, 10]), inOp(ref(`age`), [5, 10, 15])), - ).toBe(true) - }) - - it(`should handle in vs in: [5, 20] is NOT subset of [5, 10, 15]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [5, 20]), inOp(ref(`age`), [5, 10, 15])), - ).toBe(false) - }) - - it(`should handle empty IN array: age IN [] is subset of age IN []`, () => { - expect(isWhereSubset(inOp(ref(`age`), []), inOp(ref(`age`), []))).toBe( - true, - ) - }) - - it(`should handle empty IN array: age IN [] is subset of age IN [5, 10]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), []), inOp(ref(`age`), [5, 10])), - ).toBe(true) - }) - - it(`should handle empty IN array: age IN [5, 10] is NOT subset of age IN []`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [5, 10]), inOp(ref(`age`), [])), - ).toBe(false) - }) - - it(`should handle singleton IN array: age = 5 is subset of age IN [5]`, () => { - expect(isWhereSubset(eq(ref(`age`), val(5)), inOp(ref(`age`), [5]))).toBe( - true, - ) - }) - - it(`should handle singleton IN array: age = 10 is NOT subset of age IN [5]`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(10)), inOp(ref(`age`), [5])), - ).toBe(false) - }) - - it(`should handle singleton IN array: age IN [5] is subset of age IN [5, 10, 15]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [5]), inOp(ref(`age`), [5, 10, 15])), - ).toBe(true) - }) - - it(`should handle singleton IN array: age IN [20] is NOT subset of age IN [5, 10, 15]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [20]), inOp(ref(`age`), [5, 10, 15])), - ).toBe(false) - }) - - it(`should handle singleton IN array: age IN [5, 10, 15] is NOT subset of age IN [5]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [5, 10, 15]), inOp(ref(`age`), [5])), - ).toBe(false) - }) - }) - - describe(`AND combinations`, () => { - it(`should handle AND in subset: (A AND B) is subset of A`, () => { - expect( - isWhereSubset( - and(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - gt(ref(`age`), val(10)), - ), - ).toBe(true) - }) - - it(`should handle AND in subset: (A AND B) is NOT subset of C (different field)`, () => { - expect( - isWhereSubset( - and(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - eq(ref(`name`), val(`John`)), - ), - ).toBe(false) - }) - - it(`should handle AND in superset: A is subset of (A AND B) is false (superset is more restrictive)`, () => { - expect( - isWhereSubset( - gt(ref(`age`), val(10)), - and(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - ), - ).toBe(false) - }) - - it(`should handle AND in both: (age > 20 AND status = 'active') is subset of (age > 10 AND status = 'active')`, () => { - expect( - isWhereSubset( - and(gt(ref(`age`), val(20)), eq(ref(`status`), val(`active`))), - and(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - ), - ).toBe(true) - }) - }) - - describe(`OR combinations`, () => { - it(`should handle OR in superset: A is subset of (A OR B)`, () => { - expect( - isWhereSubset( - gt(ref(`age`), val(10)), - or(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - ), - ).toBe(true) - }) - - it(`should return false when subset doesn't imply any branch of OR superset`, () => { - expect( - isWhereSubset( - eq(ref(`age`), val(10)), - or(gt(ref(`age`), val(10)), lt(ref(`age`), val(5))), - ), - ).toBe(false) - }) - - it(`should handle OR in subset: (A OR B) is subset of C only if both A and B are subsets of C`, () => { - expect( - isWhereSubset( - or(gt(ref(`age`), val(20)), gt(ref(`age`), val(30))), - gt(ref(`age`), val(10)), - ), - ).toBe(true) - }) - - it(`should handle OR in both: (age > 20 OR status = 'active') is subset of (age > 10 OR status = 'active')`, () => { - expect( - isWhereSubset( - or(gt(ref(`age`), val(20)), eq(ref(`status`), val(`active`))), - or(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - ), - ).toBe(true) - }) - - it(`should handle OR in subset: (A OR B) is NOT subset of C if either is not a subset`, () => { - expect( - isWhereSubset( - or(gt(ref(`age`), val(20)), lt(ref(`age`), val(5))), - gt(ref(`age`), val(10)), - ), - ).toBe(false) - }) - }) - - describe(`AND subset with OR superset`, () => { - it(`should recognize and(eq, isNull) as subset of or(and(eq, isNull), and(eq, isNull))`, () => { - const projectX = `4e164373-31b4-4b42-95c9-9c395cfb4916` - const projectY = `2fd4c147-2547-4b02-9554-9cd067187409` - - const queryX = and( - eq(ref(`project_id`), val(projectX)), - func(`isNull`, ref(`soft_deleted_at`)), - ) - const queryY = and( - eq(ref(`project_id`), val(projectY)), - func(`isNull`, ref(`soft_deleted_at`)), - ) - - const unionPredicate = or(queryX, queryY) - - expect(isWhereSubset(queryX, unionPredicate)).toBe(true) - expect(isWhereSubset(queryY, unionPredicate)).toBe(true) - }) - - it(`should recognize and(A, B) as subset of or(and(A, B), and(C, D))`, () => { - const subsetExpr = and(eq(ref(`id`), val(1)), gt(ref(`age`), val(20))) - const supersetExpr = or( - and(eq(ref(`id`), val(1)), gt(ref(`age`), val(20))), - and(eq(ref(`id`), val(2)), gt(ref(`age`), val(30))), - ) - expect(isWhereSubset(subsetExpr, supersetExpr)).toBe(true) - }) - - it(`should return false when and(A, B) matches no disjunct`, () => { - const subsetExpr = and(eq(ref(`id`), val(3)), gt(ref(`age`), val(20))) - const supersetExpr = or( - and(eq(ref(`id`), val(1)), gt(ref(`age`), val(20))), - and(eq(ref(`id`), val(2)), gt(ref(`age`), val(30))), - ) - expect(isWhereSubset(subsetExpr, supersetExpr)).toBe(false) - }) - }) - - describe(`isNull predicates`, () => { - it(`should return true for identical isNull expressions`, () => { - const a = func(`isNull`, ref(`deleted_at`)) - const b = func(`isNull`, ref(`deleted_at`)) - expect(isWhereSubset(a, b)).toBe(true) - }) - - it(`should return false for isNull on different fields`, () => { - const a = func(`isNull`, ref(`deleted_at`)) - const b = func(`isNull`, ref(`created_at`)) - expect(isWhereSubset(a, b)).toBe(false) - }) - - it(`should return true for and(eq, isNull) subset of identical and(eq, isNull)`, () => { - const subset = and( - eq(ref(`project_id`), val(`abc`)), - func(`isNull`, ref(`soft_deleted_at`)), - ) - const superset = and( - eq(ref(`project_id`), val(`abc`)), - func(`isNull`, ref(`soft_deleted_at`)), - ) - expect(isWhereSubset(subset, superset)).toBe(true) - }) - }) - - describe(`different fields`, () => { - it(`should return false for different fields with no relationship`, () => { - expect( - isWhereSubset(gt(ref(`age`), val(20)), gt(ref(`salary`), val(1000))), - ).toBe(false) - }) - }) - - describe(`Date support`, () => { - const date1 = new Date(`2024-01-01`) - const date2 = new Date(`2024-01-15`) - const date3 = new Date(`2024-02-01`) - - it(`should handle Date equality`, () => { - expect( - isWhereSubset( - eq(ref(`createdAt`), val(date2)), - eq(ref(`createdAt`), val(date2)), - ), - ).toBe(true) - }) - - it(`should handle Date range comparisons: date > 2024-01-15 is subset of date > 2024-01-01`, () => { - expect( - isWhereSubset( - gt(ref(`createdAt`), val(date2)), - gt(ref(`createdAt`), val(date1)), - ), - ).toBe(true) - }) - - it(`should handle Date range comparisons: date < 2024-01-15 is subset of date < 2024-02-01`, () => { - expect( - isWhereSubset( - lt(ref(`createdAt`), val(date2)), - lt(ref(`createdAt`), val(date3)), - ), - ).toBe(true) - }) - - it(`should handle Date equality vs range: date = 2024-01-15 is subset of date > 2024-01-01`, () => { - expect( - isWhereSubset( - eq(ref(`createdAt`), val(date2)), - gt(ref(`createdAt`), val(date1)), - ), - ).toBe(true) - }) - - it(`should handle Date equality vs IN: date = 2024-01-15 is subset of date IN [2024-01-01, 2024-01-15, 2024-02-01]`, () => { - expect( - isWhereSubset( - eq(ref(`createdAt`), val(date2)), - inOp(ref(`createdAt`), [date1, date2, date3]), - ), - ).toBe(true) - }) - - it(`should handle Date IN subset: date IN [2024-01-01, 2024-01-15] is subset of date IN [2024-01-01, 2024-01-15, 2024-02-01]`, () => { - expect( - isWhereSubset( - inOp(ref(`createdAt`), [date1, date2]), - inOp(ref(`createdAt`), [date1, date2, date3]), - ), - ).toBe(true) - }) - - it(`should return false when Date not in IN set`, () => { - expect( - isWhereSubset( - eq(ref(`createdAt`), val(date1)), - inOp(ref(`createdAt`), [date2, date3]), - ), - ).toBe(false) - }) - }) -}) - -describe(`unionWherePredicates`, () => { - describe(`basic cases`, () => { - it(`should return false for empty array`, () => { - const result = unionWherePredicates([]) - expect(result.type).toBe(`val`) - expect((result as Value).value).toBe(false) - }) - - it(`should return the single predicate as-is`, () => { - const pred = gt(ref(`age`), val(10)) - const result = unionWherePredicates([pred]) - expect(result).toBe(pred) - }) - }) - - describe(`same field comparisons`, () => { - it(`should take least restrictive for gt: age > 10 OR age > 20 → age > 10`, () => { - const result = unionWherePredicates([ - gt(ref(`age`), val(10)), - gt(ref(`age`), val(20)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`gt`) - const field = (result as Func).args[1] as Value - expect(field.value).toBe(10) - }) - - it(`should take least restrictive for gte: age >= 10 OR age >= 20 → age >= 10`, () => { - const result = unionWherePredicates([ - gte(ref(`age`), val(10)), - gte(ref(`age`), val(20)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`gte`) - const field = (result as Func).args[1] as Value - expect(field.value).toBe(10) - }) - - it(`should take least restrictive for lt: age < 20 OR age < 10 → age < 20`, () => { - const result = unionWherePredicates([ - lt(ref(`age`), val(20)), - lt(ref(`age`), val(10)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`lt`) - const field = (result as Func).args[1] as Value - expect(field.value).toBe(20) - }) - - it(`should combine eq into IN: age = 5 OR age = 10 → age IN [5, 10]`, () => { - const result = unionWherePredicates([ - eq(ref(`age`), val(5)), - eq(ref(`age`), val(10)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`in`) - const values = ((result as Func).args[1] as Value).value - expect(values).toContain(5) - expect(values).toContain(10) - expect(values.length).toBe(2) - }) - - it(`should fold IN and equality into single IN: age IN [1,2] OR age = 3 → age IN [1,2,3]`, () => { - const result = unionWherePredicates([ - inOp(ref(`age`), [1, 2]), - eq(ref(`age`), val(3)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`in`) - const values = ((result as Func).args[1] as Value).value - expect(values).toContain(1) - expect(values).toContain(2) - expect(values).toContain(3) - expect(values.length).toBe(3) - }) - - it(`should handle gte and gt together: age > 10 OR age >= 15 → age > 10`, () => { - const result = unionWherePredicates([ - gt(ref(`age`), val(10)), - gte(ref(`age`), val(15)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`gt`) - const field = (result as Func).args[1] as Value - expect(field.value).toBe(10) - }) - }) - - describe(`different fields`, () => { - it(`should combine with OR: age > 10 OR status = 'active'`, () => { - const result = unionWherePredicates([ - gt(ref(`age`), val(10)), - eq(ref(`status`), val(`active`)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`or`) - expect((result as Func).args.length).toBe(2) - }) - }) - - describe(`flatten OR`, () => { - it(`should flatten nested ORs`, () => { - const result = unionWherePredicates([ - or(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - eq(ref(`name`), val(`John`)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`or`) - expect((result as Func).args.length).toBe(3) - }) - }) - - describe(`Date support`, () => { - const date1 = new Date(`2024-01-01`) - const date2 = new Date(`2024-01-15`) - const date3 = new Date(`2024-02-01`) - - it(`should combine Date equalities into IN: date = date1 OR date = date2 → date IN [date1, date2]`, () => { - const result = unionWherePredicates([ - eq(ref(`createdAt`), val(date1)), - eq(ref(`createdAt`), val(date2)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`in`) - const values = ((result as Func).args[1] as Value).value - expect(values.length).toBe(2) - expect(values).toContainEqual(date1) - expect(values).toContainEqual(date2) - }) - - it(`should fold Date IN and equality: date IN [date1,date2] OR date = date3 → date IN [date1,date2,date3]`, () => { - const result = unionWherePredicates([ - inOp(ref(`createdAt`), [date1, date2]), - eq(ref(`createdAt`), val(date3)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`in`) - const values = ((result as Func).args[1] as Value).value - expect(values.length).toBe(3) - expect(values).toContainEqual(date1) - expect(values).toContainEqual(date2) - expect(values).toContainEqual(date3) - }) - }) -}) - -describe(`isOrderBySubset`, () => { - it(`should return true for undefined subset`, () => { - const orderBy: OrderBy = [orderByClause(ref(`age`), `asc`)] - expect(isOrderBySubset(undefined, orderBy)).toBe(true) - expect(isOrderBySubset([], orderBy)).toBe(true) - }) - - it(`should return false for undefined superset with non-empty subset`, () => { - const orderBy: OrderBy = [orderByClause(ref(`age`), `asc`)] - expect(isOrderBySubset(orderBy, undefined)).toBe(false) - expect(isOrderBySubset(orderBy, [])).toBe(false) - }) - - it(`should return true for identical orderBy`, () => { - const orderBy: OrderBy = [orderByClause(ref(`age`), `asc`)] - expect(isOrderBySubset(orderBy, orderBy)).toBe(true) - }) - - it(`should return true when subset is prefix of superset`, () => { - const subset: OrderBy = [orderByClause(ref(`age`), `asc`)] - const superset: OrderBy = [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - ] - expect(isOrderBySubset(subset, superset)).toBe(true) - }) - - it(`should return false when subset is not a prefix`, () => { - const subset: OrderBy = [orderByClause(ref(`name`), `desc`)] - const superset: OrderBy = [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - ] - expect(isOrderBySubset(subset, superset)).toBe(false) - }) - - it(`should return false when directions differ`, () => { - const subset: OrderBy = [orderByClause(ref(`age`), `desc`)] - const superset: OrderBy = [orderByClause(ref(`age`), `asc`)] - expect(isOrderBySubset(subset, superset)).toBe(false) - }) - - it(`should return false when subset is longer than superset`, () => { - const subset: OrderBy = [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - orderByClause(ref(`status`), `asc`), - ] - const superset: OrderBy = [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - ] - expect(isOrderBySubset(subset, superset)).toBe(false) - }) -}) - -describe(`isLimitSubset`, () => { - it(`should return false for undefined subset with limited superset (requesting all data but only have limited)`, () => { - expect(isLimitSubset(undefined, 10)).toBe(false) - }) - - it(`should return true for undefined subset with undefined superset (requesting all data and have all data)`, () => { - expect(isLimitSubset(undefined, undefined)).toBe(true) - }) - - it(`should return true for undefined superset`, () => { - expect(isLimitSubset(10, undefined)).toBe(true) - }) - - it(`should return true when subset <= superset`, () => { - expect(isLimitSubset(10, 20)).toBe(true) - expect(isLimitSubset(10, 10)).toBe(true) - }) - - it(`should return false when subset > superset`, () => { - expect(isLimitSubset(20, 10)).toBe(false) - }) -}) - -describe(`isOffsetLimitSubset`, () => { - it(`should return true when subset range is within superset range (same offset)`, () => { - expect( - isOffsetLimitSubset({ offset: 0, limit: 5 }, { offset: 0, limit: 10 }), - ).toBe(true) - expect( - isOffsetLimitSubset({ offset: 0, limit: 10 }, { offset: 0, limit: 10 }), - ).toBe(true) - }) - - it(`should return true when subset starts later but is still within superset range`, () => { - // superset loads rows [0, 10), subset loads rows [5, 10) - subset is within superset - expect( - isOffsetLimitSubset({ offset: 5, limit: 5 }, { offset: 0, limit: 10 }), - ).toBe(true) - }) - - it(`should return false when subset extends beyond superset range`, () => { - // superset loads rows [0, 10), subset loads rows [5, 15) - subset extends beyond - expect( - isOffsetLimitSubset({ offset: 5, limit: 10 }, { offset: 0, limit: 10 }), - ).toBe(false) - }) - - it(`should return false when subset is completely outside superset range`, () => { - // superset loads rows [0, 10), subset loads rows [20, 30) - no overlap - expect( - isOffsetLimitSubset({ offset: 20, limit: 10 }, { offset: 0, limit: 10 }), - ).toBe(false) - }) - - it(`should return false when superset starts after subset`, () => { - // superset loads rows [10, 20), subset loads rows [0, 10) - superset starts too late - expect( - isOffsetLimitSubset({ offset: 0, limit: 10 }, { offset: 10, limit: 10 }), - ).toBe(false) - }) - - it(`should return true when superset is unlimited`, () => { - expect(isOffsetLimitSubset({ offset: 0, limit: 10 }, { offset: 0 })).toBe( - true, - ) - expect(isOffsetLimitSubset({ offset: 20, limit: 10 }, { offset: 0 })).toBe( - true, - ) - }) - - it(`should return false when superset is unlimited but starts after subset`, () => { - // superset loads rows [10, ∞), subset loads rows [0, 10) - superset starts too late - expect(isOffsetLimitSubset({ offset: 0, limit: 10 }, { offset: 10 })).toBe( - false, - ) - }) - - it(`should return false when subset is unlimited but superset has a limit`, () => { - expect(isOffsetLimitSubset({ offset: 0 }, { offset: 0, limit: 10 })).toBe( - false, - ) - }) - - it(`should return true when both are unlimited and superset starts at or before subset`, () => { - expect(isOffsetLimitSubset({ offset: 10 }, { offset: 0 })).toBe(true) - expect(isOffsetLimitSubset({ offset: 10 }, { offset: 10 })).toBe(true) - }) - - it(`should return false when both are unlimited but superset starts after subset`, () => { - expect(isOffsetLimitSubset({ offset: 0 }, { offset: 10 })).toBe(false) - }) - - it(`should default offset to 0 when undefined`, () => { - expect(isOffsetLimitSubset({ limit: 5 }, { limit: 10 })).toBe(true) - expect(isOffsetLimitSubset({ offset: 0, limit: 5 }, { limit: 10 })).toBe( - true, - ) - expect(isOffsetLimitSubset({ limit: 5 }, { offset: 0, limit: 10 })).toBe( - true, - ) - }) -}) - -describe(`isPredicateSubset`, () => { - it(`should check all components for unlimited superset`, () => { - // For unlimited supersets, where-subset logic applies - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(20)), - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 10, - } - const superset: LoadSubsetOptions = { - where: gt(ref(`age`), val(10)), - orderBy: [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - ], - // No limit - unlimited superset - } - expect(isPredicateSubset(subset, superset)).toBe(true) - }) - - it(`should require equal where clauses for limited supersets`, () => { - // For limited supersets, where clauses must be EQUAL - const sameWhere = gt(ref(`age`), val(10)) - - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 5, - } - const superset: LoadSubsetOptions = { - where: sameWhere, // Same where clause - orderBy: [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - ], - limit: 20, - } - expect(isPredicateSubset(subset, superset)).toBe(true) - }) - - it(`should return false for limited superset with different where clause`, () => { - // Even if subset's where is more restrictive, it can't be a subset - // of a limited superset with a different where clause. - // The top N items of "age > 20" may not be in the top M items of "age > 10" - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(20)), // More restrictive - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 5, - } - const superset: LoadSubsetOptions = { - where: gt(ref(`age`), val(10)), // Less restrictive but LIMITED - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 20, - } - // This should be FALSE because the top 5 of "age > 20" - // might include items outside the top 20 of "age > 10" - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return false for limited superset with no where vs subset with where`, () => { - // This is the reported bug case: pagination with search filter - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(20)), // Has a filter - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 10, - } - const superset: LoadSubsetOptions = { - where: undefined, // No filter but LIMITED - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 10, - } - // The filtered results might include items outside the unfiltered top 10 - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return false if where is not subset`, () => { - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(5)), - limit: 10, - } - const superset: LoadSubsetOptions = { - where: gt(ref(`age`), val(10)), - limit: 20, - } - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return false if orderBy is not subset`, () => { - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(20)), - orderBy: [orderByClause(ref(`name`), `desc`)], - } - const superset: LoadSubsetOptions = { - where: gt(ref(`age`), val(10)), - orderBy: [orderByClause(ref(`age`), `asc`)], - } - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return false if limit is not subset`, () => { - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(20)), - limit: 30, - } - const superset: LoadSubsetOptions = { - where: gt(ref(`age`), val(10)), - limit: 20, - } - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - describe(`with offset`, () => { - it(`should return true when subset offset+limit is within superset range`, () => { - const sameWhere = gt(ref(`age`), val(10)) - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 5, - limit: 5, - } - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 10, - } - // subset loads rows [5, 10), superset loads rows [0, 10) - subset is within - expect(isPredicateSubset(subset, superset)).toBe(true) - }) - - it(`should return false when subset is at different offset outside superset range`, () => { - const sameWhere = gt(ref(`age`), val(10)) - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 20, - limit: 10, - } - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 10, - } - // subset loads rows [20, 30), superset loads rows [0, 10) - no overlap - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return false when subset extends beyond superset even with same where`, () => { - const sameWhere = gt(ref(`age`), val(10)) - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 5, - limit: 10, - } - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 10, - } - // subset loads rows [5, 15), superset loads rows [0, 10) - subset extends beyond - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return true for unlimited superset with any subset offset`, () => { - const sameWhere = gt(ref(`age`), val(10)) - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 100, - limit: 10, - } - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - // No limit - unlimited - } - expect(isPredicateSubset(subset, superset)).toBe(true) - }) - - it(`should return false when superset has offset that starts after subset needs`, () => { - const sameWhere = gt(ref(`age`), val(10)) - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 10, - } - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 5, - limit: 10, - } - // subset needs rows [0, 10), superset only has rows [5, 15) - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should handle pagination correctly - page 2 not subset of page 1`, () => { - const sameWhere = gt(ref(`age`), val(10)) - // Page 1: offset 0, limit 10 - const page1: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 10, - } - // Page 2: offset 10, limit 10 - const page2: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 10, - limit: 10, - } - // Page 2 is NOT a subset of page 1 (different rows) - expect(isPredicateSubset(page2, page1)).toBe(false) - // Page 1 is NOT a subset of page 2 (different rows) - expect(isPredicateSubset(page1, page2)).toBe(false) - }) - - it(`should return true when superset covers multiple pages`, () => { - const sameWhere = gt(ref(`age`), val(10)) - // Superset: offset 0, limit 30 (covers pages 1-3) - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 30, - } - // Page 2: offset 10, limit 10 - const page2: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 10, - limit: 10, - } - // Page 2 IS a subset of superset (rows 10-19 within 0-29) - expect(isPredicateSubset(page2, superset)).toBe(true) - }) - }) -}) - -describe(`minusWherePredicates`, () => { - describe(`basic cases`, () => { - it(`should return original predicate when nothing to subtract`, () => { - const pred = gt(ref(`age`), val(10)) - const result = minusWherePredicates(pred, undefined) - - expect(result).toEqual(pred) - }) - - it(`should return null when from is undefined (can't simplify NOT(B))`, () => { - const subtract = gt(ref(`age`), val(10)) - const result = minusWherePredicates(undefined, subtract) - - expect(result).toEqual({ - type: `func`, - name: `not`, - args: [subtract], - }) - }) - - it(`should return empty set when from is subset of subtract`, () => { - const from = gt(ref(`age`), val(20)) // age > 20 - const subtract = gt(ref(`age`), val(10)) // age > 10 - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ type: `val`, value: false }) - }) - - it(`should return null when predicates are on different fields`, () => { - const from = gt(ref(`age`), val(10)) - const subtract = eq(ref(`status`), val(`active`)) - const result = minusWherePredicates(from, subtract) - - expect(result).toBeNull() - }) - }) - - describe(`IN minus IN`, () => { - it(`should compute set difference: IN [A,B,C,D] - IN [B,C] = IN [A,D]`, () => { - const from = inOp(ref(`status`), [`A`, `B`, `C`, `D`]) - const subtract = inOp(ref(`status`), [`B`, `C`]) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `in`, - args: [ref(`status`), val([`A`, `D`])], - }) - }) - - it(`should return empty set when all values are subtracted`, () => { - const from = inOp(ref(`status`), [`A`, `B`]) - const subtract = inOp(ref(`status`), [`A`, `B`]) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ type: `val`, value: false }) - }) - - it(`should return original when no overlap`, () => { - const from = inOp(ref(`status`), [`A`, `B`]) - const subtract = inOp(ref(`status`), [`C`, `D`]) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual(from) - }) - - it(`should collapse to equality when one value remains`, () => { - const from = inOp(ref(`status`), [`A`, `B`]) - const subtract = inOp(ref(`status`), [`B`]) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `eq`, - args: [ref(`status`), val(`A`)], - }) - }) - }) - - describe(`IN minus equality`, () => { - it(`should remove value from IN: IN [A,B,C] - eq(B) = IN [A,C]`, () => { - const from = inOp(ref(`status`), [`A`, `B`, `C`]) - const subtract = eq(ref(`status`), val(`B`)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `in`, - args: [ref(`status`), val([`A`, `C`])], - }) - }) - - it(`should collapse to equality when one value remains`, () => { - const from = inOp(ref(`status`), [`A`, `B`]) - const subtract = eq(ref(`status`), val(`A`)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `eq`, - args: [ref(`status`), val(`B`)], - }) - }) - - it(`should return empty set when removing last value`, () => { - const from = inOp(ref(`status`), [`A`]) - const subtract = eq(ref(`status`), val(`A`)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ type: `val`, value: false }) - }) - }) - - describe(`equality minus equality`, () => { - it(`should return empty set when same value`, () => { - const from = eq(ref(`age`), val(15)) - const subtract = eq(ref(`age`), val(15)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ type: `val`, value: false }) - }) - - it(`should return original when different values`, () => { - const from = eq(ref(`age`), val(15)) - const subtract = eq(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual(from) - }) - }) - - describe(`range minus range - gt/gte`, () => { - it(`should compute difference: age > 10 - age > 20 = (age > 10 AND age <= 20)`, () => { - const from = gt(ref(`age`), val(10)) - const subtract = gt(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gt(ref(`age`), val(10)), lte(ref(`age`), val(20))], - }) - }) - - it(`should return original when no overlap: age > 20 - age > 10`, () => { - const from = gt(ref(`age`), val(20)) - const subtract = gt(ref(`age`), val(10)) - const result = minusWherePredicates(from, subtract) - - // age > 20 is subset of age > 10, so result is empty - expect(result).toEqual({ type: `val`, value: false }) - }) - - it(`should compute difference: age >= 10 - age >= 20 = (age >= 10 AND age < 20)`, () => { - const from = gte(ref(`age`), val(10)) - const subtract = gte(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gte(ref(`age`), val(10)), lt(ref(`age`), val(20))], - }) - }) - - it(`should compute difference: age > 10 - age >= 20 = (age > 10 AND age < 20)`, () => { - const from = gt(ref(`age`), val(10)) - const subtract = gte(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gt(ref(`age`), val(10)), lt(ref(`age`), val(20))], - }) - }) - - it(`should compute difference: age >= 10 - age > 20 = (age >= 10 AND age <= 20)`, () => { - const from = gte(ref(`age`), val(10)) - const subtract = gt(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gte(ref(`age`), val(10)), lte(ref(`age`), val(20))], - }) - }) - }) - - describe(`range minus range - lt/lte`, () => { - it(`should compute difference: age < 30 - age < 20 = (age >= 20 AND age < 30)`, () => { - const from = lt(ref(`age`), val(30)) - const subtract = lt(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gte(ref(`age`), val(20)), lt(ref(`age`), val(30))], - }) - }) - - it(`should return original when no overlap: age < 20 - age < 30`, () => { - const from = lt(ref(`age`), val(20)) - const subtract = lt(ref(`age`), val(30)) - const result = minusWherePredicates(from, subtract) - - // age < 20 is subset of age < 30, so result is empty - expect(result).toEqual({ type: `val`, value: false }) - }) - - it(`should compute difference: age <= 30 - age <= 20 = (age > 20 AND age <= 30)`, () => { - const from = lte(ref(`age`), val(30)) - const subtract = lte(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gt(ref(`age`), val(20)), lte(ref(`age`), val(30))], - }) - }) - - it(`should compute difference: age < 30 - age <= 20 = (age > 20 AND age < 30)`, () => { - const from = lt(ref(`age`), val(30)) - const subtract = lte(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gt(ref(`age`), val(20)), lt(ref(`age`), val(30))], - }) - }) - - it(`should compute difference: age <= 30 - age < 20 = (age >= 20 AND age <= 30)`, () => { - const from = lte(ref(`age`), val(30)) - const subtract = lt(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gte(ref(`age`), val(20)), lte(ref(`age`), val(30))], - }) - }) - }) - - describe(`common conditions`, () => { - it(`should handle common conditions: (age > 10 AND status = 'active') - (age > 20 AND status = 'active') = (age > 10 AND age <= 20 AND status = 'active')`, () => { - const from = and( - gt(ref(`age`), val(10)), - eq(ref(`status`), val(`active`)), - ) - const subtract = and( - gt(ref(`age`), val(20)), - eq(ref(`status`), val(`active`)), - ) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [ - eq(ref(`status`), val(`active`)), // common condition - gt(ref(`age`), val(10)), - lte(ref(`age`), val(20)), - ], - }) - }) - - it(`should handle multiple common conditions`, () => { - const from = and( - gt(ref(`age`), val(10)), - eq(ref(`status`), val(`active`)), - eq(ref(`department`), val(`engineering`)), - ) - const subtract = and( - gt(ref(`age`), val(20)), - eq(ref(`status`), val(`active`)), - eq(ref(`department`), val(`engineering`)), - ) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [ - eq(ref(`status`), val(`active`)), // common condition - eq(ref(`department`), val(`engineering`)), // common condition - gt(ref(`age`), val(10)), - lte(ref(`age`), val(20)), - ], - }) - }) - - it(`should handle IN with common conditions: (age IN [10,20,30] AND status = 'active') - (age IN [20,30] AND status = 'active') = (age IN [10] AND status = 'active')`, () => { - const from = and( - inOp(ref(`age`), [10, 20, 30]), - eq(ref(`status`), val(`active`)), - ) - const subtract = and( - inOp(ref(`age`), [20, 30]), - eq(ref(`status`), val(`active`)), - ) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [ - eq(ref(`status`), val(`active`)), // common condition - { - type: `func`, - name: `eq`, - args: [ref(`age`), val(10)], - }, - ], - }) - }) - - it(`should return null when common conditions exist but remaining difference cannot be simplified`, () => { - const from = and( - gt(ref(`age`), val(10)), - eq(ref(`status`), val(`active`)), - ) - const subtract = and( - gt(ref(`name`), val(`Z`)), - eq(ref(`status`), val(`active`)), - ) - const result = minusWherePredicates(from, subtract) - - // Can't simplify age > 10 - name > 'Z' (different fields), so returns null - expect(result).toBeNull() - }) - }) - - describe(`Date support`, () => { - it(`should handle Date IN minus Date IN`, () => { - const date1 = new Date(`2024-01-01`) - const date2 = new Date(`2024-01-15`) - const date3 = new Date(`2024-02-01`) - - const from = inOp(ref(`createdAt`), [date1, date2, date3]) - const subtract = inOp(ref(`createdAt`), [date2]) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `in`, - args: [ref(`createdAt`), val([date1, date3])], - }) - }) - - it(`should handle Date range difference: date > 2024-01-01 - date > 2024-01-15`, () => { - const date1 = new Date(`2024-01-01`) - const date15 = new Date(`2024-01-15`) - - const from = gt(ref(`createdAt`), val(date1)) - const subtract = gt(ref(`createdAt`), val(date15)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [ - gt(ref(`createdAt`), val(date1)), - lte(ref(`createdAt`), val(date15)), - ], - }) - }) - }) - - describe(`real-world sync scenarios`, () => { - it(`should compute missing data range: need age > 10, already have age > 20`, () => { - const requested = gt(ref(`age`), val(10)) - const alreadyLoaded = gt(ref(`age`), val(20)) - const needToFetch = minusWherePredicates(requested, alreadyLoaded) - - // Need to fetch: 10 < age <= 20 - expect(needToFetch).toEqual({ - type: `func`, - name: `and`, - args: [gt(ref(`age`), val(10)), lte(ref(`age`), val(20))], - }) - }) - - it(`should compute missing IDs: need IN [1..100], already have IN [50..100]`, () => { - const allIds = Array.from({ length: 100 }, (_, i) => i + 1) - const loadedIds = Array.from({ length: 51 }, (_, i) => i + 50) - - const requested = inOp(ref(`id`), allIds) - const alreadyLoaded = inOp(ref(`id`), loadedIds) - const needToFetch = minusWherePredicates(requested, alreadyLoaded) - - // Need to fetch: ids 1..49 - const expectedIds = Array.from({ length: 49 }, (_, i) => i + 1) - expect(needToFetch).toEqual({ - type: `func`, - name: `in`, - args: [ref(`id`), val(expectedIds)], - }) - }) - - it(`should return empty when all requested data is already loaded`, () => { - const requested = gt(ref(`age`), val(20)) - const alreadyLoaded = gt(ref(`age`), val(10)) - const needToFetch = minusWherePredicates(requested, alreadyLoaded) - - // Requested is subset of already loaded - nothing more to fetch - expect(needToFetch).toEqual({ type: `val`, value: false }) - }) - }) -}) diff --git a/packages/db/tests/query/public-container-copy.test.ts b/packages/db/tests/query/public-container-copy.test.ts new file mode 100644 index 0000000000..e29c5942bf --- /dev/null +++ b/packages/db/tests/query/public-container-copy.test.ts @@ -0,0 +1,195 @@ +import { expect, it } from 'vitest' +import { transformPublicContainers } from '../../src/query/compiler/route-metadata.js' +import { + createLiveQueryCollection, + eq, + materialize, + toArray, +} from '../../src/query/index.js' +import { createControlledCollection } from './includes-oracle-helpers.js' + +it.each( + ([`object`, `array`] as const).flatMap((kind) => + [false, true].flatMap((ordered) => + [1, NaN].map((code) => ({ kind, ordered, code })), + ), + ), +)( + `preserves $kind reference-key matches through an ordered=$ordered projected source with code=$code`, + async ({ kind, ordered, code }) => { + const makeKey = (value: number): object => + kind === `object` ? { code: value } : [value] + const key = makeKey(code) + const other = makeKey(2) + const parents = createControlledCollection(`copy-parents`, [ + { id: 1, group: 1, key }, + ]) + const children = createControlledCollection(`copy-children`, [ + { id: 10, group: 1, key }, + { id: 20, group: 1, key: makeKey(code) }, + { id: 30, group: 1, key: other }, + ]) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + const filtered = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.group, parent.group)) + const source = ( + ordered ? filtered.orderBy(({ child }) => child.id) : filtered + ).select(({ child }) => ({ id: child.id, key: child.key })) + const matches = q + .from({ inner: source }) + .where(({ inner }) => eq(inner.key, parent.key)) + .select(({ inner }) => ({ id: inner.id })) + return { + id: parent.id, + collection: matches, + array: toArray(matches), + materialized: materialize(matches), + } + }), + ) + const check = (expected: Array) => { + const row = live.get(1)! + for (const values of [ + row.collection.toArray, + row.array, + row.materialized, + ]) { + expect(values.map(({ id }) => id).sort((a, b) => a - b)).toEqual( + expected, + ) + } + } + try { + await live.preload() + check([10]) + parents.write(`update`, { id: 1, group: 1, key: other }) + check([30]) + children.write(`update`, { id: 10, group: 1, key: other }) + check([10, 30]) + children.write(`delete`, { id: 30, group: 1, key: other }) + check([10]) + } finally { + await live.cleanup() + await Promise.all([ + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, +) + +it.each([NaN, -0, 0, undefined, null, Infinity])( + `preserves every reference under an identity transform of %s`, + (value) => { + const key = { value } + const array = [value, key] + const input = { key, array, self: undefined as unknown } + input.self = input + expect(transformPublicContainers(input, (leaf) => leaf, new Set())).toBe( + input, + ) + expect(transformPublicContainers(array, (leaf) => leaf, new Set())).toBe( + array, + ) + expect(transformPublicContainers(key, (leaf) => leaf, new Set())).toBe(key) + }, +) + +it(`preserves a signed-zero replacement at the root and in nested containers`, () => { + const transform = (value: unknown) => (Object.is(value, -0) ? 0 : value) + expect(transformPublicContainers(-0, transform, new Set())).toBe(0) + const input = { key: [-0] } + const result = transformPublicContainers( + input, + transform, + new Set(), + ) as typeof input + expect(result).not.toBe(input) + expect(result.key[0]).toBe(0) + expect(input.key[0]).toBe(-0) +}) + +it.each([false, true])( + `copies public descriptors with null prototype=%s`, + (nullPrototype) => { + const privateKey = Symbol(`private`) + const publicKey = Symbol(`public`) + const opaque = new Date(0) + const replacement = new Map() + const reference = { token: true } + const child = { [privateKey]: true, value: 1 } + const input = Object.create( + nullPrototype ? null : Object.prototype, + ) as Record + let reads = 0 + const getter = () => { + reads++ + return 7 + } + Object.defineProperties(input, { + child: { value: child, enumerable: true, writable: false }, + alias: { value: child, enumerable: true }, + leaf: { value: reference, enumerable: true }, + opaque: { value: opaque, enumerable: true }, + hidden: { value: 4, enumerable: false }, + accessor: { get: getter, enumerable: true }, + [`__proto__`]: { value: `user property`, enumerable: true }, + [publicKey]: { value: child, enumerable: true }, + [privateKey]: { value: true }, + self: { value: input, enumerable: true }, + }) + const result = transformPublicContainers( + input, + (value) => (value === reference ? replacement : value), + new Set([privateKey]), + ) as typeof input + expect(reads).toBe(0) + expect(Object.getPrototypeOf(result)).toBe(Object.getPrototypeOf(input)) + expect(Reflect.ownKeys(result)).toEqual( + Reflect.ownKeys(input).filter((key) => key !== privateKey), + ) + expect(result.child).toEqual({ value: 1 }) + expect(result.alias).toBe(result.child) + expect(result[publicKey]).toBe(result.child) + expect(result.self).toBe(result) + expect(result.leaf).toBe(replacement) + expect(result.opaque).toBe(opaque) + expect(result[`__proto__`]).toBe(`user property`) + expect(Object.getOwnPropertyDescriptor(result, `child`)).toEqual({ + value: result.child, + enumerable: true, + writable: false, + configurable: false, + }) + expect(Object.getOwnPropertyDescriptor(result, `accessor`)?.get).toBe( + getter, + ) + expect(Object.getOwnPropertyDescriptor(result, `hidden`)).toEqual( + Object.getOwnPropertyDescriptor(input, `hidden`), + ) + expect(child[privateKey]).toBe(true) + expect(input.self).toBe(input) + }, +) + +it(`preserves sparse arrays and locked lengths while removing private keys`, () => { + const privateKey = Symbol(`private`) + const input: Array> = new Array(4) + input[2] = { value: 2, [privateKey]: true } + Object.defineProperty(input, `length`, { writable: false }) + const result = transformPublicContainers( + input, + (value) => value, + new Set([privateKey]), + ) as Array + const expected = new Array(4) + expected[2] = { value: 2 } + expect(result).toEqual(expected) + expect(Object.hasOwn(result, 0)).toBe(false) + expect(Object.getOwnPropertyDescriptor(result, `length`)).toEqual( + Object.getOwnPropertyDescriptor(input, `length`), + ) + expect(input[2][privateKey]).toBe(true) +}) diff --git a/packages/db/tests/query/replay-failure-boundary.test.ts b/packages/db/tests/query/replay-failure-boundary.test.ts new file mode 100644 index 0000000000..afbebf8bb8 --- /dev/null +++ b/packages/db/tests/query/replay-failure-boundary.test.ts @@ -0,0 +1,216 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection' +import { createDeferred } from '../../src/deferred' +import { BasicIndex } from '../../src/indexes/basic-index' +import { createLiveQueryCollection, eq } from '../../src/query' +import { PropRef } from '../../src/query/ir' +import { evaluateReferenceExpression } from '../reference-expression' +import { flushPromises } from '../utils' +import type { SyncConfig } from '../../src/types' + +type Row = { id: number; version: number } + +describe.each([`direct`, `query`] as const)( + `failed replay publication and recovery for %s`, + (consumer) => { + it.each( + ([`throw`, `reject`] as const).flatMap((failureMode) => + [false, true].map((partialWrite) => ({ failureMode, partialWrite })), + ), + )( + `keeps peers and retained results sound: %j`, + async ({ failureMode, partialWrite }) => { + const failure = new Error(`replacement failed`) + const pending = createDeferred() + let phase: `initial` | `failed` | `recovered` = `initial` + let sync!: Parameters[`sync`]>[0] + let childSync!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: (operations) => { + sync = operations + operations.begin() + for (const id of [1, 2]) + operations.write({ type: `insert`, value: { id, version: 1 } }) + operations.commit() + operations.markReady() + return { + loadSubset: (options) => { + const ids = [1, 2].filter( + (id) => + !options.where || + evaluateReferenceExpression(options.where, { + id, + version: 1, + }), + ) + if (phase === `initial`) return true + for (const id of ids) { + if (phase === `failed` && id === 1 && !partialWrite) + continue + operations.begin() + operations.write({ + type: source.has(id) ? `update` : `insert`, + value: { id, version: phase === `recovered` ? 4 : 2 }, + }) + operations.commit() + } + if (phase === `failed` && ids.includes(1)) { + if (failureMode === `throw`) throw failure + return pending.promise + } + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const children = createCollection({ + getKey: (row) => row.id, + sync: { + sync: (operations) => { + childSync = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, version: 1 } }) + operations.commit() + operations.markReady() + }, + }, + }) + const makeLive = () => + createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => eq(row.id, 1)) + .orderBy(({ row }) => row.id) + .limit(1) + .select(({ row }) => ({ + id: row.id, + version: row.version, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.id, row.id)), + })), + ) + const live = consumer === `query` ? makeLive() : undefined + const peer = createLiveQueryCollection((q) => + q.from({ row: source }).where(({ row }) => eq(row.id, 2)), + ) + const visible = new Map() + const direct = source.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.key !== 1) continue + if (change.type === `delete`) visible.delete(1) + else visible.set(1, { id: 1, version: change.value.version }) + } + }, + { includeInitialState: false }, + ) + const errors: Array = [] + direct.on(`loadSubset:error`, ({ error }) => errors.push(error)) + let replacement: ReturnType | undefined + let replacementDirect: typeof direct | undefined + try { + if (live) await live.preload() + else + direct.requestSnapshot({ + where: eq(sourceExpression(), 1), + optimizedOnly: false, + }) + await peer.preload() + const retainedChild = live?.get(1)?.children + if (live) expect(retainedChild).toBeDefined() + const read = () => + live ? live.get(1)?.version : visible.get(1)?.version + expect(read()).toBe(1) + phase = `failed` + sync.begin() + sync.truncate() + sync.commit() + // Observe the waiter before the queued acquisition can reject. + const waiter = live + ? live.utils.setWindow({ limit: 2 }) + : direct.pendingTruncateReplacement + expect(waiter).toBeInstanceOf(Promise) + const settled = Promise.allSettled([waiter]) + await flushPromises() + if (failureMode === `reject`) pending.reject(failure) + expect(await settled).toEqual([ + { status: `rejected`, reason: failure }, + ]) + await flushPromises() + expect(read()).toBe(1) + expect(peer.get(2)?.version).toBe(2) + if (!live) expect(errors).toEqual([failure]) + if (retainedChild) expect(retainedChild.get(1)?.version).toBe(1) + + sync.begin() + sync.write({ + type: source.has(1) ? `update` : `insert`, + value: { id: 1, version: 3 }, + }) + sync.write({ type: `update`, value: { id: 2, version: 3 } }) + sync.commit() + if (retainedChild) { + childSync.begin() + childSync.write({ type: `update`, value: { id: 1, version: 3 } }) + childSync.commit() + } + await flushPromises() + expect(read()).toBe(1) + expect(peer.get(2)?.version).toBe(3) + expect(source.status).toBe(`ready`) + if (retainedChild) expect(retainedChild.get(1)?.version).toBe(1) + + // Recreating only the failed consumer is a valid recovery action. + // Do not reset the shared source or force its healthy peer to restart. + phase = `recovered` + if (live) { + await live.cleanup() + replacement = makeLive() + await replacement.preload() + expect(replacement.get(1)?.version).toBe(4) + expect(replacement.get(1)?.children.get(1)?.version).toBe(3) + } else { + direct.unsubscribe() + replacementDirect = source.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.key === 1 && change.type !== `delete`) + visible.set(1, change.value) + } + }, + { includeInitialState: false }, + ) + replacementDirect.requestSnapshot({ + where: eq(sourceExpression(), 1), + optimizedOnly: false, + }) + expect(visible.get(1)?.version).toBe(4) + } + expect(peer.get(2)?.version).toBe(3) + } finally { + pending.resolve() + direct.unsubscribe() + replacementDirect?.unsubscribe() + await Promise.all([ + live?.cleanup(), + replacement?.cleanup(), + peer.cleanup(), + ]) + await Promise.all([source.cleanup(), children.cleanup()]) + } + }, + ) + }, +) + +function sourceExpression() { + return new PropRef([`id`]) +} diff --git a/packages/db/tests/query/scheduler.test.ts b/packages/db/tests/query/scheduler.test.ts index 1361231e7a..d7c8e546fc 100644 --- a/packages/db/tests/query/scheduler.test.ts +++ b/packages/db/tests/query/scheduler.test.ts @@ -1,15 +1,46 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' import { createLiveQueryCollection, eq, isNull } from '../../src/query/index.js' import { createTransaction } from '../../src/transactions.js' import { createOptimisticAction } from '../../src/optimistic-action.js' -import { transactionScopedScheduler } from '../../src/scheduler.js' +import { + Scheduler, + getActivePublicationContext, + recordPublicationError, + transactionScopedScheduler, + withPublicationContext, +} from '../../src/scheduler.js' import { CollectionConfigBuilder } from '../../src/query/live/collection-config-builder.js' -import { mockSyncCollectionOptions, stripVirtualProps } from '../utils.js' +import { getCollectionBuilder } from '../../src/query/live/collection-registry.js' +import { CollectionSubscriber } from '../../src/query/live/collection-subscriber.js' +import { Query, createEffect } from '../../src/index.js' +import { + flushPromises, + mockSyncCollectionOptions, + stripVirtualProps, +} from '../utils.js' +import type { SchedulerContextId } from '../../src/scheduler.js' import type { OutputWithVirtual } from '../utils.js' import type { FullSyncState } from '../../src/query/live/types.js' import type { SyncConfig } from '../../src/types.js' +type SchedulerInternals = { + contexts: Map }> +} +const flushAll = (scheduler: Scheduler) => { + const { contexts } = scheduler as unknown as SchedulerInternals + for (const contextId of Array.from(contexts.keys())) + scheduler.flush(contextId) +} +const hasPendingJobs = ( + scheduler: Scheduler, + contextId: SchedulerContextId, +) => { + const { contexts } = scheduler as unknown as SchedulerInternals + return (contexts.get(contextId)?.jobs.size ?? 0) > 0 +} + interface ChangeMessageLike { type: string value: any @@ -20,6 +51,17 @@ interface User { name: string } +const falsyListenerFailureCases = [ + { name: `undefined`, failure: undefined }, + { name: `null`, failure: null }, + { name: `false`, failure: false }, + { name: `zero`, failure: 0 }, + { name: `negative zero`, failure: -0 }, + { name: `bigint zero`, failure: 0n }, + { name: `empty string`, failure: `` }, + { name: `NaN`, failure: Number.NaN }, +] + type UserWithVirtual = OutputWithVirtual interface Task { @@ -84,10 +126,801 @@ function recordBatches(collection: any) { } afterEach(() => { - transactionScopedScheduler.flushAll() + flushAll(transactionScopedScheduler) +}) + +describe(`Scheduler dependency reentry`, () => { + it.each( + [false, true].flatMap((sourceFirst) => + [false, true].flatMap((pendingAware) => + [false, true].map((requeue) => ({ + sourceFirst, + pendingAware, + requeue, + })), + ), + ), + )( + `waits for current source work: sourceFirst=$sourceFirst pendingAware=$pendingAware requeue=$requeue`, + ({ sourceFirst, pendingAware, requeue }) => { + const scheduler = new Scheduler() + const contextId = Symbol(`source-reentry`) + let sourceRuns = 0 + let pending = true + const source = pendingAware + ? { hasPendingGraphRun: () => pending } + : Symbol(`source`) + const observedRuns: Array = [] + const runSource = () => { + sourceRuns++ + pending = false + if (requeue && sourceRuns === 1) { + pending = true + scheduler.schedule({ contextId, jobId: source, run: runSource }) + } + } + const jobs = [ + { contextId, jobId: source, run: runSource }, + { + contextId, + jobId: Symbol(`dependent`), + dependencies: [source], + run: () => observedRuns.push(sourceRuns), + }, + ] + for (const job of sourceFirst ? jobs : [...jobs].reverse()) { + scheduler.schedule(job) + } + scheduler.flush(contextId) + expect(sourceRuns).toBe(requeue ? 2 : 1) + expect(observedRuns).toEqual([sourceRuns]) + expect(hasPendingJobs(scheduler, contextId)).toBe(false) + }, + ) +}) + +describe(`Collection publication scheduler context`, () => { + it(`preserves the first listener error when a later graph job fails`, () => { + const listenerFailure = new Error(`listener failed first`) + const graphFailure = new Error(`graph failed later`) + const graphJob = vi.fn(() => { + throw graphFailure + }) + let contextId: ReturnType + expect(() => + withPublicationContext(() => { + contextId = getActivePublicationContext() + recordPublicationError(listenerFailure) + transactionScopedScheduler.schedule({ + contextId, + jobId: graphJob, + run: graphJob, + }) + }), + ).toThrow(listenerFailure) + expect(graphJob).toHaveBeenCalledOnce() + expect(hasPendingJobs(transactionScopedScheduler, contextId!)).toBe(false) + expect(getActivePublicationContext()).toBeUndefined() + }) + + it(`shares one context and flushes after the outer publication`, () => { + const calls: Array = [] + let contextId: ReturnType + + withPublicationContext(() => { + contextId = getActivePublicationContext() + expect(contextId).toBeDefined() + + transactionScopedScheduler.schedule({ + contextId, + jobId: `outer`, + run: () => calls.push(`outer`), + }) + withPublicationContext(() => { + expect(getActivePublicationContext()).toBe(contextId) + transactionScopedScheduler.schedule({ + contextId, + jobId: `inner`, + run: () => calls.push(`inner`), + }) + }) + + expect(calls).toEqual([]) + }) + + expect(calls).toEqual([`outer`, `inner`]) + expect(getActivePublicationContext()).toBeUndefined() + }) + + it(`clears queued work when publication throws`, () => { + const run = vi.fn() + let contextId: ReturnType + + expect(() => + withPublicationContext(() => { + contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: `discarded`, + run, + }) + throw new Error(`publication failed`) + }), + ).toThrow(`publication failed`) + + expect(run).not.toHaveBeenCalled() + expect(getActivePublicationContext()).toBeUndefined() + expect(hasPendingJobs(transactionScopedScheduler, contextId!)).toBe(false) + }) + + it(`preserves a falsy graph failure through a publication boundary`, () => { + let didThrow = false + let thrown: unknown + + try { + withPublicationContext(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: `failing`, + run: () => { + throw undefined + }, + }) + }) + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(thrown).toBeUndefined() + }) + + it(`attempts every clear listener and preserves its first failure`, () => { + const scheduler = new Scheduler() + const firstFailure = new Error(`first clear listener failed`) + const laterFailure = new Error(`later clear listener failed`) + const calls: Array = [] + let firstClear = true + let removeAdded: (() => void) | undefined + scheduler.onClear(() => { + calls.push(`first`) + if (!firstClear) return + removeSecond() + removeAdded ??= scheduler.onClear(() => calls.push(`added`)) + throw firstFailure + }) + const removeSecond = scheduler.onClear(() => { + calls.push(`second`) + if (firstClear) throw laterFailure + }) + + let thrown: unknown + try { + scheduler.clear(`context`) + } catch (error) { + thrown = error + } + + expect(thrown).toBe(firstFailure) + expect(calls).toEqual([`first`, `second`]) + + firstClear = false + expect(() => scheduler.clear(`next context`)).not.toThrow() + expect(calls).toEqual([`first`, `second`, `first`, `added`]) + removeAdded?.() + }) + + it.each([ + { source: `publication`, failureKind: `Error` }, + { source: `publication`, failureKind: `undefined` }, + { source: `graph`, failureKind: `Error` }, + { source: `graph`, failureKind: `undefined` }, + ] as const)( + `does not replace a $failureKind $source failure with a clear-listener failure`, + ({ source, failureKind }) => { + const primaryFailure = + failureKind === `Error` ? new Error(`${source} failed`) : undefined + const clearFailure = new Error(`clear listener failed`) + const laterClear = vi.fn() + const removeThrowingClear = transactionScopedScheduler.onClear(() => { + throw clearFailure + }) + const removeLaterClear = transactionScopedScheduler.onClear(laterClear) + + try { + let didThrow = false + let thrown: unknown + try { + withPublicationContext(() => { + if (source === `publication`) throw primaryFailure + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: `failing graph`, + run: () => { + throw primaryFailure + }, + }) + }) + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(Object.is(thrown, primaryFailure)).toBe(true) + expect(laterClear).toHaveBeenCalledOnce() + } finally { + removeThrowingClear() + removeLaterClear() + } + }, + ) }) describe(`live query scheduler`, () => { + it(`does not deliver a source batch after a snapshotted listener unsubscribes`, async () => { + let begin!: () => void + let write!: (message: { type: `insert`; value: User }) => void + let commit!: () => void + const calls: Array = [] + const source = createCollection({ + id: `ordinary-listener-membership-source`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: (actions) => { + begin = actions.begin + write = actions.write + commit = () => { + actions.commit() + } + actions.markReady() + }, + }, + }) + let added: { unsubscribe: () => void } | undefined + const first = source.subscribeChanges(() => { + calls.push(`first`) + second.unsubscribe() + added ??= source.subscribeChanges(() => calls.push(`added`), { + includeInitialState: false, + }) + }) + const second = source.subscribeChanges(() => calls.push(`second`)) + + try { + begin() + write({ type: `insert`, value: { id: 1, name: `Ada` } }) + commit() + expect(calls).toEqual([`first`]) + + begin() + write({ type: `insert`, value: { id: 2, name: `Grace` } }) + commit() + expect(calls).toEqual([`first`, `first`, `added`]) + } finally { + first.unsubscribe() + second.unsubscribe() + added?.unsubscribe() + await source.cleanup() + } + }) + + it(`delivers a layout-only batch to its frozen listener snapshot`, async () => { + type RankedUser = User & { rank: number } + const calls: Array = [] + const firstFailure = new Error(`first layout listener failed`) + const laterFailure = new Error(`later public listener failed`) + const graphJob = vi.fn(() => calls.push(`graph`)) + const source = createCollection( + mockSyncCollectionOptions({ + id: `layout-listener-membership-source`, + getKey: (user) => user.id, + initialData: [ + { id: 1, name: `Ada`, rank: 1 }, + { id: 2, name: `Grace`, rank: 2 }, + ], + }), + ) + const ordered = createLiveQueryCollection({ + id: `layout-listener-membership-ordered`, + startSync: true, + query: (q) => + q + .from({ user: source }) + .orderBy(({ user }) => user.rank, `asc`) + .select(({ user }) => ({ id: user.id, name: user.name })), + }) + await ordered.preload() + expect(ordered.toArray.map(({ id }) => id)).toEqual([1, 2]) + let firstPublication = true + let addedLayout: (() => void) | undefined + let addedPublic: { unsubscribe: () => void } | undefined + const unsubscribeFirstLayout = ordered._subscribeLayoutChanges(() => { + calls.push(`layout:first`) + if (!firstPublication) return + unsubscribeSecondLayout() + secondPublic.unsubscribe() + addedLayout ??= ordered._subscribeLayoutChanges(() => + calls.push(`layout:added`), + ) + addedPublic ??= ordered.subscribeChanges( + () => calls.push(`public:added`), + { includeInitialState: false }, + ) + throw firstFailure + }) + const unsubscribeSecondLayout = ordered._subscribeLayoutChanges(() => { + calls.push(`layout:second`) + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: graphJob, + run: graphJob, + }) + }) + const firstPublic = ordered.subscribeChanges( + () => { + calls.push(`public:first`) + if (firstPublication) throw laterFailure + }, + { includeInitialState: false }, + ) + const secondPublic = ordered.subscribeChanges( + () => calls.push(`public:second`), + { + includeInitialState: false, + }, + ) + + try { + let thrown: unknown + try { + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: 1, name: `Ada`, rank: 3 }, + }) + source.utils.commit() + } catch (error) { + thrown = error + } + + expect(thrown).toBe(firstFailure) + expect(calls).toEqual([ + `layout:first`, + `layout:second`, + `public:first`, + `graph`, + ]) + expect(graphJob).toHaveBeenCalledOnce() + expect(ordered.toArray.map(({ id }) => id)).toEqual([2, 1]) + + firstPublication = false + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: 1, name: `Ada`, rank: 0 }, + }) + expect(() => source.utils.commit()).not.toThrow() + expect(calls).toEqual([ + `layout:first`, + `layout:second`, + `public:first`, + `graph`, + `layout:first`, + `layout:added`, + `public:first`, + `public:added`, + ]) + } finally { + unsubscribeFirstLayout() + unsubscribeSecondLayout() + addedLayout?.() + firstPublic.unsubscribe() + secondPublic.unsubscribe() + addedPublic?.unsubscribe() + await ordered.cleanup() + await source.cleanup() + } + }) + + it(`settles a dependent live query when an earlier source listener throws`, async () => { + let begin!: () => void + let write!: (message: { type: `insert`; value: User }) => void + let commit!: () => void + const listenerFailure = new Error(`source listener failed`) + const source = createCollection({ + id: `throwing-listener-live-source`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: (actions) => { + begin = actions.begin + write = actions.write + commit = () => { + actions.commit() + } + actions.markReady() + }, + }, + }) + const throwingSubscription = source.subscribeChanges( + () => { + throw listenerFailure + }, + { includeInitialState: false }, + ) + const live = createLiveQueryCollection({ + id: `throwing-listener-live-dependent`, + startSync: true, + query: (q) => + q + .from({ user: source }) + .select(({ user }) => ({ id: user.id, name: user.name })), + }) + + try { + await live.preload() + begin() + write({ type: `insert`, value: { id: 1, name: `Ada` } }) + expect(() => commit()).toThrow(listenerFailure) + expect(live.get(1)).toEqual(expect.objectContaining({ name: `Ada` })) + } finally { + throwingSubscription.unsubscribe() + await live.cleanup() + await source.cleanup() + } + }) + + it.each(falsyListenerFailureCases)( + `preserves an exact $name row-listener failure after later delivery`, + async ({ name, failure }) => { + let begin!: () => void + let write!: (message: { type: `insert`; value: User }) => void + let commit!: () => void + type UserObservation = { + changes: Array<{ + type: string + key: string | number + value: UserWithVirtual + previousValue: UserWithVirtual | undefined + }> + rows: Array + } + const sourceObservations: Array = [] + const dependentObservations: Array = [] + const snapshotUser = ({ + id, + name: userName, + $collectionId, + $key, + $origin, + $synced, + }: UserWithVirtual): UserWithVirtual => ({ + id, + name: userName, + $collectionId, + $key, + $origin, + $synced, + }) + const source = createCollection({ + id: `falsy-row-listener-${name.replaceAll(` `, `-`)}`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: (actions) => { + begin = actions.begin + write = actions.write + commit = () => { + actions.commit() + } + actions.markReady() + }, + }, + }) + const throwingSubscription = source.subscribeChanges( + () => { + throw failure + }, + { includeInitialState: false }, + ) + const laterSubscription = source.subscribeChanges( + (changes) => { + sourceObservations.push({ + changes: changes.map(({ type, key, value, previousValue }) => ({ + type, + key, + value: snapshotUser(value), + previousValue: + previousValue === undefined + ? undefined + : snapshotUser(previousValue), + })), + rows: [...source.state.values()].map(snapshotUser), + }) + }, + { includeInitialState: false }, + ) + const live = createLiveQueryCollection({ + id: `falsy-row-listener-dependent-${name.replaceAll(` `, `-`)}`, + startSync: true, + query: (q) => + q + .from({ user: source }) + .select(({ user }) => ({ id: user.id, name: user.name })), + }) + let dependentSubscription: + | ReturnType + | undefined + + try { + await live.preload() + dependentSubscription = live.subscribeChanges( + (changes) => { + dependentObservations.push({ + changes: changes.map(({ type, key, value, previousValue }) => ({ + type, + key, + value: snapshotUser(value), + previousValue: + previousValue === undefined + ? undefined + : snapshotUser(previousValue), + })), + rows: [...live.state.values()].map(snapshotUser), + }) + }, + { includeInitialState: false }, + ) + begin() + write({ type: `insert`, value: { id: 1, name: `Ada` } }) + let didThrow = false + let thrown: unknown + try { + commit() + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(Object.is(thrown, failure)).toBe(true) + const expectedObservation = (collectionId: string): UserObservation => { + const row: UserWithVirtual = { + id: 1, + name: `Ada`, + $collectionId: collectionId, + $key: 1, + $origin: `remote`, + $synced: true, + } + return { + changes: [ + { + type: `insert`, + key: 1, + value: row, + previousValue: undefined, + }, + ], + rows: [row], + } + } + const expectedDependent = expectedObservation(live.id) + expect(sourceObservations).toEqual([expectedObservation(source.id)]) + expect(dependentObservations).toEqual([expectedDependent]) + expect([...live.state.values()].map(snapshotUser)).toEqual( + expectedDependent.rows, + ) + } finally { + throwingSubscription.unsubscribe() + laterSubscription.unsubscribe() + dependentSubscription?.unsubscribe() + await live.cleanup() + await source.cleanup() + } + }, + ) + + it.each([ + { + name: `Error`, + failure: new Error(`filtered source listener failed`), + }, + ...falsyListenerFailureCases, + ])( + `preserves an exact $name filtered row-listener failure`, + async ({ name, failure }) => { + let begin!: () => void + let write!: (message: { type: `insert`; value: User }) => void + let commit!: () => void + const filteredCalls = vi.fn() + const laterListener = vi.fn() + const source = createCollection({ + id: `filtered-throwing-listener-source-${name.replaceAll(` `, `-`)}`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: (actions) => { + begin = actions.begin + write = actions.write + commit = () => { + actions.commit() + } + actions.markReady() + }, + }, + }) + const throwingSubscription = source.subscribeChanges( + (changes) => { + filteredCalls(changes) + throw failure + }, + { + includeInitialState: false, + where: (user) => eq(user.name, `Ada`), + }, + ) + const laterSubscription = source.subscribeChanges(laterListener, { + includeInitialState: false, + }) + const live = createLiveQueryCollection({ + id: `filtered-throwing-listener-dependent-${name.replaceAll(` `, `-`)}`, + startSync: true, + query: (q) => + q + .from({ user: source }) + .select(({ user }) => ({ id: user.id, name: user.name })), + }) + + try { + await live.preload() + begin() + write({ type: `insert`, value: { id: 1, name: `Ada` } }) + let didThrow = false + let thrown: unknown + try { + commit() + } catch (error) { + didThrow = true + thrown = error + } + expect(didThrow).toBe(true) + expect(Object.is(thrown, failure)).toBe(true) + expect(filteredCalls).toHaveBeenCalledOnce() + expect(filteredCalls.mock.calls[0]?.[0]).toEqual([ + expect.objectContaining({ type: `insert`, key: 1 }), + ]) + expect(laterListener).toHaveBeenCalledOnce() + expect(live.get(1)).toEqual(expect.objectContaining({ name: `Ada` })) + + begin() + write({ type: `insert`, value: { id: 2, name: `Grace` } }) + expect(() => commit()).not.toThrow() + expect(filteredCalls).toHaveBeenCalledOnce() + expect(laterListener).toHaveBeenCalledTimes(2) + expect(live.get(2)).toEqual(expect.objectContaining({ name: `Grace` })) + } finally { + throwingSubscription.unsubscribe() + laterSubscription.unsubscribe() + await live.cleanup() + await source.cleanup() + } + }, + ) + + it(`keeps a nested ready failure when a later outer listener throws`, async () => { + let markInnerReady!: () => void + const readyFailure = new Error(`nested ready listener failed`) + const laterFailure = new Error(`later outer listener failed`) + const scheduledJob = vi.fn() + const inner = createCollection({ + id: `nested-ready-collision-inner`, + getKey: (user) => user.id, + sync: { + sync: ({ markReady }) => { + markInnerReady = markReady + }, + }, + }) + const innerFirst = inner.subscribeChanges(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + const innerSecond = inner.subscribeChanges(() => { + throw readyFailure + }) + + let beginOuter!: () => void + let writeOuter!: (message: { type: `insert`; value: User }) => void + let commitOuter!: () => void + const outer = createCollection({ + id: `nested-ready-collision-outer`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: (actions) => { + beginOuter = actions.begin + writeOuter = actions.write + commitOuter = () => { + actions.commit() + } + actions.markReady() + }, + }, + }) + const outerFirst = outer.subscribeChanges(() => markInnerReady()) + const outerSecond = outer.subscribeChanges(() => { + throw laterFailure + }) + + try { + beginOuter() + writeOuter({ type: `insert`, value: { id: 1, name: `Ada` } }) + expect(() => commitOuter()).toThrow(readyFailure) + expect(scheduledJob).toHaveBeenCalledOnce() + } finally { + outerFirst.unsubscribe() + outerSecond.unsubscribe() + innerFirst.unsubscribe() + innerSecond.unsubscribe() + await outer.cleanup() + await inner.cleanup() + } + }) + + it(`settles a dependent live query before a nested ready failure escapes`, async () => { + let markSourceReady: (() => void) | undefined + const listenerFailure = new Error(`source ready listener failed`) + const source = createCollection({ + id: `nested-ready-live-source`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: ({ begin, commit, markReady }) => { + begin() + commit() + markSourceReady = markReady + }, + }, + }) + const live = createLiveQueryCollection({ + id: `nested-ready-live-dependent`, + startSync: true, + query: (q) => + q + .from({ user: source }) + .select(({ user }) => ({ id: user.id, name: user.name })), + }) + const preload = live.preload() + const throwingSubscription = source.subscribeChanges(() => { + throw listenerFailure + }) + + try { + expect(live.status).toBe(`loading`) + expect(() => withPublicationContext(() => markSourceReady!())).toThrow( + listenerFailure, + ) + await expect(preload).resolves.toBeUndefined() + expect(source.status).toBe(`ready`) + expect(live.status).toBe(`ready`) + } finally { + throwingSubscription.unsubscribe() + await live.cleanup() + await source.cleanup() + } + }) + it(`runs the live query graph once per transaction that touches multiple collections`, async () => { const { users, tasks, assignments } = setupLiveQueryCollections(`single-batch`) @@ -183,7 +1016,7 @@ describe(`live query scheduler`, () => { const latestBatch = recorder.batches.at(-1)! expect(latestBatch[0]?.type).toBe(`delete`) } - expect(transactionScopedScheduler.hasPendingJobs(tx.id)).toBe(false) + expect(hasPendingJobs(transactionScopedScheduler, tx.id)).toBe(false) // We emit the optimistic insert and, after the explicit rollback, possibly a // compensating delete – but no duplicate inserts. expect(recorder.batches[0]![0]).toMatchObject({ type: `insert` }) @@ -223,6 +1056,127 @@ describe(`live query scheduler`, () => { tx.rollback() }) + it.each( + [`collection`, `effect`].flatMap((consumer) => + [false, true].flatMap((sharedSource) => + [false, true].flatMap((derivedRight) => + [false, true].map((reverseWrites) => ({ + consumer, + sharedSource, + derivedRight, + reverseWrites, + })), + ), + ), + ), + )( + `publishes settled dependencies once: $consumer shared=$sharedSource derivedRight=$derivedRight reverse=$reverseWrites`, + async ({ consumer, sharedSource, derivedRight, reverseWrites }) => { + type Row = { id: number; left: string; right: string } + const makeSource = (id: string) => + createCollection( + mockSyncCollectionOptions({ + id, + getKey: (row) => row.id, + initialData: [{ id: 1, left: `old-left`, right: `old-right` }], + }), + ) + const leftSource = makeSource(`dependency-left`) + const rightSource = sharedSource + ? leftSource + : makeSource(`dependency-right`) + const leftQuery = createLiveQueryCollection({ + query: (q) => + q + .from({ row: leftSource }) + .select(({ row }) => ({ id: row.id, value: row.left })), + }) + const rightQuery = derivedRight + ? createLiveQueryCollection({ + query: (q) => + q + .from({ row: rightSource }) + .select(({ row }) => ({ id: row.id, right: row.right })), + }) + : undefined + await Promise.all([ + leftQuery.preload(), + (rightQuery ?? rightSource).preload(), + ]) + const query = new Query() + .from({ left: leftQuery }) + .join( + { right: rightQuery ?? rightSource }, + ({ left, right }) => eq(left.id, right.id), + `inner`, + ) + .select(({ left, right }) => ({ + id: left.id, + left: left.value, + right: right.right, + })) + const publications: Array> = [] + let cleanupConsumer: () => Promise + if (consumer === `collection`) { + const joined = createLiveQueryCollection({ query }) + await joined.preload() + const subscription = joined.subscribeChanges(() => { + publications.push( + joined.toArray.map(({ left, right }) => ({ left, right })), + ) + }) + cleanupConsumer = async () => { + subscription.unsubscribe() + await joined.cleanup() + } + } else { + const effect = createEffect<{ + id: number + left: string + right: string + }>({ + query, + onBatch: (events) => { + publications.push( + events.map(({ value: { left, right } }) => ({ left, right })), + ) + }, + }) + cleanupConsumer = () => effect.dispose() + } + const tx = createTransaction({ + mutationFn: async () => {}, + autoCommit: false, + }) + try { + publications.length = 0 + const writes = [ + () => + leftSource.update(1, (row) => { + row.left = `next-left` + }), + () => + rightSource.update(1, (row) => { + row.right = `next-right` + }), + ] + tx.mutate(() => { + for (const write of reverseWrites ? [...writes].reverse() : writes) + write() + }) + expect([...publications]).toEqual([ + [{ left: `next-left`, right: `next-right` }], + ]) + } finally { + tx.rollback() + await cleanupConsumer() + await Promise.all([leftQuery.cleanup(), rightQuery?.cleanup()]) + await leftSource.cleanup() + if (!sharedSource) await rightSource.cleanup() + } + }, + ) + it(`runs join live queries once after their parent queries settle`, async () => { const collectionA = createCollection<{ id: number; value: string }>({ id: `diamond-A`, @@ -290,7 +1244,7 @@ describe(`live query scheduler`, () => { liveQueryB.preload(), liveQueryJoin.preload(), ]) - const baseRunCount = liveQueryJoin.utils.getRunCount() + const runs = vi.spyOn(getCollectionBuilder(liveQueryJoin)!, `maybeRunGraph`) const tx = createTransaction({ mutationFn: async () => {}, @@ -305,7 +1259,7 @@ describe(`live query scheduler`, () => { expect(liveQueryJoin.toArray.map((row) => stripVirtualProps(row))).toEqual([ { left: `A1`, right: `B1` }, ]) - expect(liveQueryJoin.utils.getRunCount()).toBe(baseRunCount + 1) + expect(runs).toHaveBeenCalledTimes(1) tx.mutate(() => { collectionA.update(1, (draft) => { @@ -319,8 +1273,9 @@ describe(`live query scheduler`, () => { expect(liveQueryJoin.toArray.map((row) => stripVirtualProps(row))).toEqual([ { left: `A1b`, right: `B1b` }, ]) - expect(liveQueryJoin.utils.getRunCount()).toBe(baseRunCount + 2) + expect(runs).toHaveBeenCalledTimes(2) tx.rollback() + runs.mockRestore() }) it(`runs hybrid joins once when they observe both a live query and a collection`, async () => { @@ -377,7 +1332,7 @@ describe(`live query scheduler`, () => { }) await Promise.all([liveQueryA.preload(), hybridJoin.preload()]) - const baseRunCount = hybridJoin.utils.getRunCount() + const runs = vi.spyOn(getCollectionBuilder(hybridJoin)!, `maybeRunGraph`) const tx = createTransaction({ mutationFn: async () => {}, @@ -392,7 +1347,7 @@ describe(`live query scheduler`, () => { expect(hybridJoin.toArray.map((row) => stripVirtualProps(row))).toEqual([ { left: `A7`, right: `B7` }, ]) - expect(hybridJoin.utils.getRunCount()).toBe(baseRunCount + 1) + expect(runs).toHaveBeenCalledTimes(1) tx.mutate(() => { collectionA.update(7, (draft) => { @@ -406,8 +1361,9 @@ describe(`live query scheduler`, () => { expect(hybridJoin.toArray.map((row) => stripVirtualProps(row))).toEqual([ { left: `A7b`, right: `B7b` }, ]) - expect(hybridJoin.utils.getRunCount()).toBe(baseRunCount + 2) + expect(runs).toHaveBeenCalledTimes(2) tx.rollback() + runs.mockRestore() }) it(`currently single batch when the join sees right-side data before the left`, async () => { @@ -464,7 +1420,7 @@ describe(`live query scheduler`, () => { }) await Promise.all([liveQueryA.preload(), join.preload()]) - const baseRunCount = join.utils.getRunCount() + const runs = vi.spyOn(getCollectionBuilder(join)!, `maybeRunGraph`) const tx = createTransaction({ mutationFn: async () => {}, @@ -479,10 +1435,91 @@ describe(`live query scheduler`, () => { expect(join.toArray.map((row) => stripVirtualProps(row))).toEqual([ { left: `left-later`, right: `right-first` }, ]) - expect(join.utils.getRunCount()).toBe(baseRunCount + 1) + expect(runs).toHaveBeenCalledTimes(1) tx.rollback() + runs.mockRestore() }) + it.each( + [`resolve`, `reject`].flatMap((outcome) => + [false, true].map((replacementSettled) => ({ + outcome, + replacementSettled, + })), + ), + )( + `isolates ordered publication participants across restart: $outcome replacementSettled=$replacementSettled`, + async ({ outcome, replacementSettled }) => { + let sync!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: ({ id }) => id, + sync: { + sync: (operations) => { + sync = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, name: `old` } }) + operations.commit() + operations.markReady() + }, + }, + }) + const builder = new CollectionConfigBuilder({ + query: (q) => q.from({ user: source }), + }) + const config = builder.getConfig() + const live = createCollection({ ...config, singleResult: undefined }) + const obsolete = createDeferred() + const replacement = createDeferred() + try { + await live.preload() + // Inject participants at the builder boundary: the ordered loader has + // its own stale-result guards, which must not mask this owner's law. + builder.trackOrderedLoadPromise(obsolete.promise, true) + await live.cleanup() + await live.preload() + builder.trackOrderedLoadPromise(replacement.promise, true) + const publications: Array> = [] + live.subscribeChanges(() => { + publications.push(live.toArray.map(({ name }) => name)) + }) + const update = (name: string) => { + sync.begin() + sync.write({ type: `update`, value: { id: 1, name } }) + sync.commit() + } + update(`replacement`) + expect(live.toArray.map(({ name }) => name)).toEqual([`old`]) + expect(publications).toEqual([]) + if (replacementSettled) { + replacement.resolve() + await flushPromises() + } + const beforeObsolete = [...publications] + if (outcome === `resolve`) obsolete.resolve() + else obsolete.reject(new Error(`discarded session failed`)) + await flushPromises() + expect(publications).toEqual(beforeObsolete) + if (!replacementSettled) { + expect(live.toArray.map(({ name }) => name)).toEqual([`old`]) + replacement.resolve() + await flushPromises() + } + expect(live.toArray.map(({ name }) => name)).toEqual([`replacement`]) + expect(publications).toEqual([[`replacement`]]) + update(`later`) + expect(live.toArray.map(({ name }) => name)).toEqual([`later`]) + expect(publications).toEqual([[`replacement`], [`later`]]) + expect(live.status).toBe(`ready`) + expect(config.utils.lastSubsetError).toBeUndefined() + } finally { + obsolete.resolve() + replacement.resolve() + await live.cleanup() + await source.cleanup() + } + }, + ) + it(`coalesces load-more callbacks scheduled within the same context`, () => { const baseCollection = createCollection({ id: `loader-users`, @@ -540,6 +1577,288 @@ describe(`live query scheduler`, () => { maybeRunGraphSpy.mockRestore() }) + it.each( + [false, true].flatMap((initialWork) => + [false, true].map((loaderResult) => ({ initialWork, loaderResult })), + ), + )( + `drains loader writes before publication: initial=$initialWork return=$loaderResult`, + ({ initialWork, loaderResult }) => { + const source = createCollection({ + getKey: (user) => user.id, + sync: { sync: () => () => {} }, + }) + const builder = new CollectionConfigBuilder({ + query: (q) => q.from({ user: source }), + }) + const events: Array = [] + let pendingWork = initialWork + let wrote = false + builder.currentSyncConfig = { + markReady: vi.fn(), + } as unknown as Parameters[`sync`]>[0] + builder.currentSyncState = { + messagesCount: 1, + subscribedToAllCollections: true, + graph: { + pendingWork: () => pendingWork, + run: () => { + events.push(`graph`) + pendingWork = false + }, + }, + flushPendingChanges: () => events.push(`publish`), + } as unknown as FullSyncState + const contextId = Symbol(`loader-write-context`) + builder.scheduleGraphRun( + () => { + events.push(`first`) + if (!wrote) { + wrote = true + pendingWork = true + } + return loaderResult + }, + { contextId }, + ) + builder.scheduleGraphRun( + () => { + events.push(`second`) + return true + }, + { contextId }, + ) + transactionScopedScheduler.flush(contextId) + expect(events).toEqual([ + ...(initialWork ? [`graph`] : []), + `first`, + `second`, + `graph`, + `first`, + `second`, + `publish`, + ]) + expect(builder.hasPendingGraphRun(contextId)).toBe(false) + }, + ) + + it.each( + [ + { name: `undefined`, failure: undefined }, + { name: `null`, failure: null }, + { name: `false`, failure: false }, + { name: `zero`, failure: 0 }, + { name: `empty string`, failure: `` }, + { name: `NaN`, failure: Number.NaN }, + ].flatMap((entry) => + [false, true].map((laterFails) => ({ ...entry, laterFails })), + ), + )( + `preserves the first falsy graph-loader failure: $name laterFails=$laterFails`, + ({ failure, laterFails }) => { + const baseCollection = createCollection({ + id: `falsy-loader-users-${String(failure)}`, + getKey: (user) => user.id, + sync: { + sync: () => () => {}, + }, + }) + const builder = new CollectionConfigBuilder({ + id: `falsy-loader-builder-${String(failure)}`, + query: (q) => q.from({ user: baseCollection }), + }) + const contextId = Symbol(`falsy-loader-context`) + const laterLoader = vi.fn(() => { + if (laterFails) throw new Error(`later loader failed`) + return false + }) + const config = { + begin: vi.fn(), + write: vi.fn(), + commit: vi.fn(), + markReady: vi.fn(), + truncate: vi.fn(), + } as unknown as Parameters[`sync`]>[0] + const syncState = { + messagesCount: 0, + subscribedToAllCollections: true, + unsubscribeCallbacks: new Set<() => void>(), + graph: { + pendingWork: () => false, + run: vi.fn(), + }, + inputs: {}, + pipeline: {}, + } as unknown as FullSyncState + const maybeRunGraphSpy = vi + .spyOn(builder, `maybeRunGraph`) + .mockImplementation((combinedLoader) => { + combinedLoader?.() + }) + + builder.currentSyncConfig = config + builder.currentSyncState = syncState + builder.scheduleGraphRun( + () => { + throw failure + }, + { contextId }, + ) + builder.scheduleGraphRun(laterLoader, { contextId }) + + let didThrow = false + let thrown: unknown + try { + transactionScopedScheduler.flush(contextId) + } catch (error) { + didThrow = true + thrown = error + } finally { + maybeRunGraphSpy.mockRestore() + } + + expect(didThrow).toBe(true) + expect(Object.is(thrown, failure)).toBe(true) + expect(laterLoader).toHaveBeenCalledOnce() + }, + ) + + it(`attempts every repeated-alias source loader and preserves the first failure`, async () => { + const createSource = (name: string) => + createCollection({ + id: `source-loader-${name}`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: ({ markReady }) => { + markReady() + return () => {} + }, + }, + }) + const firstSource = createSource(`first`) + const secondSource = createSource(`second`) + const thirdSource = createSource(`third`) + const builder = new CollectionConfigBuilder({ + id: `source-loader-builder`, + query: (q) => + q.from({ root: firstSource }).select(({ root }) => ({ + id: root.id, + second: q + .from({ item: secondSource }) + .where(({ item }) => eq(item.id, root.id)), + third: q + .from({ item: thirdSource }) + .where(({ item }) => eq(item.id, root.id)), + })), + }) + type BuilderSyncConfig = Parameters< + ReturnType[`sync`][`sync`] + >[0] + const config = { + begin: vi.fn(), + write: vi.fn(), + commit: vi.fn(), + markReady: vi.fn(), + truncate: vi.fn(), + } as unknown as BuilderSyncConfig + const builderInternals = builder as unknown as { + graphCache: FullSyncState[`graph`] + inputsCache: FullSyncState[`inputs`] + pipelineCache: FullSyncState[`pipeline`] + collectionSources: Array<{ + sourceId: string + alias: string + collection: object + }> + subscribeToAllCollections: ( + syncConfig: typeof config, + state: FullSyncState, + ) => () => void + } + const syncState = { + messagesCount: 0, + unsubscribeCallbacks: new Set<() => void>(), + subscribedToAllCollections: false, + graph: builderInternals.graphCache, + inputs: builderInternals.inputsCache, + pipeline: builderInternals.pipelineCache, + } as unknown as FullSyncState + const sourceIdFor = (collection: object): string => { + const source = builderInternals.collectionSources.find( + (candidate) => candidate.collection === collection, + ) + if (!source) throw new Error(`Expected a lexical source`) + return source.sourceId + } + const firstSourceId = sourceIdFor(firstSource) + const secondSourceId = sourceIdFor(secondSource) + const thirdSourceId = sourceIdFor(thirdSource) + expect( + builderInternals.collectionSources.map(({ alias }) => alias), + ).toEqual([`root`, `item`, `item`]) + expect(new Set([firstSourceId, secondSourceId, thirdSourceId]).size).toBe(3) + const laterFailure = new Error(`later source failed`) + const loaderCalls: Array = [] + const loaderCallCounts = new Map() + const loadMoreSpy = vi + .spyOn(CollectionSubscriber.prototype, `loadMoreIfNeeded`) + .mockImplementation(function (this: unknown) { + const { sourceId } = this as { sourceId: string } + loaderCalls.push(sourceId) + loaderCallCounts.set( + sourceId, + (loaderCallCounts.get(sourceId) ?? 0) + 1, + ) + if (sourceId === firstSourceId) throw undefined + if (sourceId === secondSourceId) throw laterFailure + if (sourceId === thirdSourceId) return true + throw new Error(`Unexpected source: ${sourceId}`) + }) + + try { + builder.currentSyncConfig = config + builder.currentSyncState = syncState + const loadAllSources = builderInternals.subscribeToAllCollections( + config, + syncState, + ) + + let didThrow = false + let thrown: unknown + try { + loadAllSources() + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(Object.is(thrown, undefined)).toBe(true) + expect(loaderCalls).toEqual([ + firstSourceId, + secondSourceId, + thirdSourceId, + ]) + expect(loaderCallCounts).toEqual( + new Map([ + [firstSourceId, 1], + [secondSourceId, 1], + [thirdSourceId, 1], + ]), + ) + expect(loadMoreSpy).toHaveBeenCalledTimes(3) + } finally { + for (const unsubscribe of syncState.unsubscribeCallbacks) unsubscribe() + loadMoreSpy.mockRestore() + await Promise.all([ + firstSource.cleanup(), + secondSource.cleanup(), + thirdSource.cleanup(), + ]) + } + }) + it(`should handle optimistic mutations with nested left joins without scheduler errors`, async () => { // This test verifies that optimistic mutations on collections with nested live query // collections using left joins complete successfully without scheduler errors. @@ -674,7 +1993,7 @@ describe(`live query scheduler`, () => { await new Promise((resolve) => setTimeout(resolve, 10)) // The scheduler should flush successfully without detecting unresolved dependencies - transactionScopedScheduler.flushAll() + flushAll(transactionScopedScheduler) } catch (e) { error = e as Error } @@ -761,7 +2080,7 @@ describe(`live query scheduler`, () => { try { action(`1`) await new Promise((resolve) => setTimeout(resolve, 10)) - transactionScopedScheduler.flushAll() + flushAll(transactionScopedScheduler) } catch (e) { error = e as Error } diff --git a/packages/db/tests/query/select.test-d.ts b/packages/db/tests/query/select.test-d.ts index 225decdfb3..f4ec842922 100644 --- a/packages/db/tests/query/select.test-d.ts +++ b/packages/db/tests/query/select.test-d.ts @@ -1,6 +1,6 @@ import { describe, expectTypeOf, test } from 'vitest' import { createCollection } from '../../src/collection/index.js' -import { createLiveQueryCollection } from '../../src/query/index.js' +import { createLiveQueryCollection, eq } from '../../src/query/index.js' import { mockSyncCollectionOptions } from '../utils.js' import { upper } from '../../src/query/builder/functions.js' import type { OutputWithVirtual } from '../utils.js' @@ -109,6 +109,126 @@ describe(`select types`, () => { expectTypeOf(results).toMatchTypeOf>() }) + test(`select preserves union types and where works on common keys`, () => { + type ItemDocument = + | { type: 'pdf'; url: string; pages: number } + | { type: 'image'; url: string; width: number; height: number } + | { type: 'legacy'; path: string } + + type Item = { id: number; name: string; document: ItemDocument } + + const items = createCollection( + mockSyncCollectionOptions({ + id: `union-field-items`, + getKey: (i) => i.id, + initialData: [], + }), + ) + + // Filtering by a common key of the union should compile, + // and the result should preserve the full discriminated union + const col = createLiveQueryCollection((q) => + q + .from({ i: items }) + .where(({ i }) => eq(i.document.type, `pdf`)) + .select(({ i }) => ({ + id: i.id, + document: i.document, + })), + ) + + const result = col.toArray[0]! + expectTypeOf(result.document).toEqualTypeOf() + }) + + test(`select preserves union when nested under another field`, () => { + type Payload = + | { kind: 'text'; body: string } + | { kind: 'binary'; bytes: number; mime: string } + + type Envelope = { id: number; payload: { inner: Payload } } + + const envelopes = createCollection( + mockSyncCollectionOptions({ + id: `nested-union-envelopes`, + getKey: (e) => e.id, + initialData: [], + }), + ) + + // Selecting a nested object whose field is a discriminated union + // must preserve the union (not collapse to the intersection of keys). + const col = createLiveQueryCollection((q) => + q.from({ e: envelopes }).select(({ e }) => ({ + id: e.id, + payload: e.payload, + })), + ) + const r = col.toArray[0]! + expectTypeOf(r.payload).toEqualTypeOf<{ inner: Payload }>() + expectTypeOf(r.payload.inner).toEqualTypeOf() + }) + + test(`spread with a same-key narrower override projects the override type`, () => { + type SpreadUser = { + id: number + code: string | number + slug: string + nickname?: string + } + + const spreadUsers = createCollection( + mockSyncCollectionOptions({ + id: `spread-override-users`, + getKey: (u) => u.id, + initialData: [], + }), + ) + + const col = createLiveQueryCollection((q) => + q.from({ u: spreadUsers }).select(({ u }) => ({ + narrowed: { ...u, code: u.slug }, + })), + ) + + const result = col.toArray[0]! + // `code` was overridden with `u.slug` (string), so the projected + // field must be `string`, not the original `string | number`. + expectTypeOf(result.narrowed.code).toEqualTypeOf() + }) + + test(`spread that omits an optional property drops the key`, () => { + type SpreadUser = { + id: number + code: string | number + slug: string + nickname?: string + } + + const spreadUsers = createCollection( + mockSyncCollectionOptions({ + id: `spread-omit-users`, + getKey: (u) => u.id, + initialData: [], + }), + ) + + const col = createLiveQueryCollection((q) => + q.from({ u: spreadUsers }).select(({ u }) => { + const { nickname: _nickname, ...withoutNickname } = u + return { trimmed: withoutNickname } + }), + ) + + const _result = col.toArray[0]! + // `nickname` was destructured out, so the projected object must + // not reintroduce the key. + type HasNickname = `nickname` extends keyof typeof _result.trimmed + ? true + : false + expectTypeOf().toEqualTypeOf() + }) + test(`nested spread preserves object structure types`, () => { const users = createUsers() const col = createLiveQueryCollection((q) => { diff --git a/packages/db/tests/query/select.test.ts b/packages/db/tests/query/select.test.ts index 79dd481bb8..448936885f 100644 --- a/packages/db/tests/query/select.test.ts +++ b/packages/db/tests/query/select.test.ts @@ -1,6 +1,7 @@ import { beforeEach, describe, expect, it } from 'vitest' import { createCollection } from '../../src/collection/index.js' -import { createLiveQueryCollection } from '../../src/query/index.js' +import { createLiveQueryCollection, queryOnce } from '../../src/query/index.js' +import { UnsafeAliasPathError } from '../../src/errors.js' import { mockSyncCollectionOptions } from '../utils.js' import { upper } from '../../src/query/builder/functions.js' @@ -129,3 +130,45 @@ describe(`nested select projections`, () => { }) }) }) + +function prototypeHasOwn(prop: string): boolean { + return Object.prototype.hasOwnProperty.call(Object.prototype, prop) +} + +describe(`select() alias prototype pollution`, () => { + let users: ReturnType + + beforeEach(() => { + users = createUsers() + }) + + it(`should reject __proto__ in alias path and not pollute Object.prototype`, async () => { + const hadBefore = prototypeHasOwn(`polluted`) + + await expect( + queryOnce((q) => + q.from({ user: users }).select(({ user }) => ({ + [`__proto__.polluted`]: user.name, + })), + ), + ).rejects.toThrow(UnsafeAliasPathError) + + expect(prototypeHasOwn(`polluted`)).toBe(hadBefore) + expect(prototypeHasOwn(`polluted`)).toBe(false) + }) + + it(`should reject constructor in alias path and not pollute Object.prototype`, async () => { + const hadBefore = prototypeHasOwn(`polluted`) + + await expect( + queryOnce((q) => + q.from({ user: users }).select(({ user }) => ({ + [`constructor.prototype.polluted`]: user.name, + })), + ), + ).rejects.toThrow(UnsafeAliasPathError) + + expect(prototypeHasOwn(`polluted`)).toBe(hadBefore) + expect(prototypeHasOwn(`polluted`)).toBe(false) + }) +}) diff --git a/packages/db/tests/query/subset-dedupe.test.ts b/packages/db/tests/query/subset-dedupe.test.ts index 02af82eb77..bff40c2c17 100644 --- a/packages/db/tests/query/subset-dedupe.test.ts +++ b/packages/db/tests/query/subset-dedupe.test.ts @@ -1,1249 +1,483 @@ +import { runInNewContext } from 'node:vm' import { describe, expect, it, vi } from 'vitest' -import { - DeduplicatedLoadSubset, - cloneOptions, -} from '../../src/query/subset-dedupe' +import { DeduplicatedLoadSubset } from '../../src/query/subset-dedupe' +import { eq, gt } from '../../src/query/builder/functions' import { Func, PropRef, Value } from '../../src/query/ir' -import type { BasicExpression, OrderBy } from '../../src/query/ir' -import type { LoadSubsetOptions } from '../../src/types' +import { compileSingleRowExpression } from '../../src/query/compiler/evaluators' +import type { LoadSubsetFn, LoadSubsetOptions } from '../../src/types' -// Helper functions to build expressions more easily -function ref(path: string | Array): PropRef { - return new PropRef(typeof path === `string` ? [path] : path) -} - -function val(value: T): Value { - return new Value(value) -} - -function gt(left: BasicExpression, right: BasicExpression): Func { - return new Func(`gt`, [left, right]) -} - -function lt(left: BasicExpression, right: BasicExpression): Func { - return new Func(`lt`, [left, right]) -} - -function eq(left: BasicExpression, right: BasicExpression): Func { - return new Func(`eq`, [left, right]) -} - -function and(...expressions: Array>): Func { - return new Func(`and`, expressions) -} - -function inOp(left: BasicExpression, values: Array): Func { - return new Func(`in`, [left, new Value(values)]) -} - -function lte(left: BasicExpression, right: BasicExpression): Func { - return new Func(`lte`, [left, right]) -} - -function not(expression: BasicExpression): Func { - return new Func(`not`, [expression]) -} - -describe(`createDeduplicatedLoadSubset`, () => { - it(`should call underlying loadSubset on first call`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - - expect(callCount).toBe(1) - }) - - it(`should return true immediately for subset unlimited calls`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: age > 10 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(1) - - // Second call: age > 20 (subset of age > 10) - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(20)), - }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not call underlying function - }) - - it(`should call underlying loadSubset for non-subset unlimited calls`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: age > 20 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) - - // Second call: age > 10 (NOT a subset of age > 20) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(2) // Should call underlying function - }) - - it(`should combine unlimited calls with union`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } +const ref = (name: string) => new PropRef([name]) +const val = (value: T) => new Value(value) +describe(`DeduplicatedLoadSubset`, () => { + it(`deduplicates only completed exact demands`, async () => { + const loadSubset = vi.fn().mockResolvedValue(undefined) + const onDeduplicate = vi.fn() const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, + loadSubset, + onDeduplicate, }) - // First call: age > 20 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) - - // Second call: age < 10 (different range) - await deduplicated.loadSubset({ where: lt(ref(`age`), val(10)) }) - expect(callCount).toBe(2) - - // Third call: age > 25 (subset of age > 20) - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(25)), - }) - expect(result).toBe(true) - expect(callCount).toBe(2) // Should not call - covered by first call - }) - - it(`should track limited calls separately`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - const whereClause = gt(ref(`age`), val(10)) - - // First call: age > 10, orderBy age asc, limit 10 - await deduplicated.loadSubset({ - where: whereClause, - orderBy: orderBy1, - limit: 10, - }) - expect(callCount).toBe(1) - - // Second call: SAME where clause, same orderBy, smaller limit (subset) - // For limited queries, where clauses must be EQUAL for subset relationship - const result = await deduplicated.loadSubset({ - where: whereClause, // Same where clause - orderBy: orderBy1, - limit: 5, - }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not call - subset of first - }) - - it(`should NOT dedupe limited calls with different where clauses`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - // First call: age > 10, orderBy age asc, limit 10 await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - limit: 10, + limit: 2, }) - expect(callCount).toBe(1) + expect( + deduplicated.loadSubset({ + where: gt(ref(`age`), val(10)), + limit: 2, + }), + ).toBe(true) + expect(loadSubset).toHaveBeenCalledTimes(1) + expect(onDeduplicate).toHaveBeenCalledTimes(1) - // Second call: DIFFERENT where clause (age > 20) - should NOT be deduped - // even though age > 20 is "more restrictive" than age > 10, - // the top 5 of age > 20 might not be in the top 10 of age > 10 await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)), - orderBy: orderBy1, - limit: 5, - }) - expect(callCount).toBe(2) // Should call - different where clause - }) - - it(`should call underlying for non-subset limited calls`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - // First call: age > 10, orderBy age asc, limit 10 - await deduplicated.loadSubset({ - where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - limit: 10, + limit: 2, }) - expect(callCount).toBe(1) - - // Second call: age > 10, orderBy age asc, limit 20 (NOT a subset) await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - limit: 20, + limit: 3, }) - expect(callCount).toBe(2) // Should call - limit is larger + expect(loadSubset).toHaveBeenCalledTimes(3) }) - it(`should check limited calls against unlimited combined predicate`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] + it(`does not infer coverage from a broader predicate or window`, async () => { + const loadSubset = vi.fn().mockResolvedValue(undefined) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - // First call: unlimited age > 10 await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(1) + await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) + await deduplicated.loadSubset({ limit: 10, offset: 0 }) + await deduplicated.loadSubset({ limit: 5, offset: 2 }) - // Second call: limited age > 20 with orderBy + limit - // Even though it has a limit, it's covered by the unlimited call - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(20)), - orderBy: orderBy1, - limit: 10, - }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not call - covered by unlimited + expect(loadSubset).toHaveBeenCalledTimes(4) }) - it(`should ignore orderBy for unlimited calls`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - + it(`shares exact in-flight work when it has no cancellation owner`, async () => { + let resolve!: () => void + const loadSubset = vi.fn( + () => new Promise((done) => (resolve = done)), + ) + const onDeduplicate = vi.fn() const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - // First call: unlimited with orderBy - await deduplicated.loadSubset({ - where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - }) - expect(callCount).toBe(1) - - // Second call: subset where, different orderBy, no limit - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(20)), + loadSubset, + onDeduplicate, }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not call - orderBy ignored for unlimited - }) - - it(`should handle undefined where clauses`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) + const first = deduplicated.loadSubset({ limit: 2 }) + const second = deduplicated.loadSubset({ limit: 2 }) - // First call: no where clause (all data) - await deduplicated.loadSubset({}) - expect(callCount).toBe(1) + expect(second).toBe(first) + expect(loadSubset).toHaveBeenCalledTimes(1) + expect(onDeduplicate).not.toHaveBeenCalled() - // Second call: with where clause (should be covered) - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(10)), - }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not call - all data already loaded + resolve() + await Promise.all([first, second]) + expect(onDeduplicate).toHaveBeenCalledTimes(1) + expect(deduplicated.loadSubset({ limit: 2 })).toBe(true) }) - it(`should handle complex real-world scenario`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(options) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderBy1: OrderBy = [ - { - expression: ref(`createdAt`), - compareOptions: { - direction: `desc`, - nulls: `last`, - stringSort: `lexical`, + describe.each([`resolve`, `reject`] as const)( + `shared transport %s with deduplication observers`, + (outcome) => { + it.each([ + { waiters: 2, throws: false }, + { waiters: 2, throws: true }, + { waiters: 3, throws: false }, + { waiters: 3, throws: true }, + ])( + `preserves settlement without unhandled rejections ($waiters waiters, throws=$throws)`, + async ({ waiters, throws }) => { + const transportError = new Error(`transport failed`) + const observerError = new Error(`deduplication observer failed`) + let resolve!: () => void + let reject!: (reason: unknown) => void + const loadSubset = vi.fn( + () => + new Promise((done, fail) => { + resolve = done + reject = fail + }), + ) + const onDeduplicate = vi.fn(() => { + if (throws) throw observerError + }) + const deduplicated = new DeduplicatedLoadSubset({ + loadSubset, + onDeduplicate, + }) + const unhandled: Array = [] + const recordUnhandled = (reason: unknown) => unhandled.push(reason) + process.on(`unhandledRejection`, recordUnhandled) + try { + const requests = Array.from({ length: waiters }, () => + deduplicated.loadSubset({ limit: 2 }), + ) + const settled = Promise.allSettled(requests) + expect(requests.every((request) => request === requests[0])).toBe( + true, + ) + expect(loadSubset).toHaveBeenCalledTimes(1) + expect(onDeduplicate).not.toHaveBeenCalled() + + if (outcome === `resolve`) resolve() + else reject(transportError) + + expect(await settled).toEqual( + Array.from({ length: waiters }, () => + outcome === `resolve` + ? { status: `fulfilled`, value: undefined } + : { status: `rejected`, reason: transportError }, + ), + ) + // Let the host report rejected detached observer promises too. + await new Promise((done) => setTimeout(done, 0)) + expect(onDeduplicate).toHaveBeenCalledTimes( + outcome === `resolve` ? waiters - 1 : 0, + ) + expect(unhandled).toEqual([]) + } finally { + process.off(`unhandledRejection`, recordUnhandled) + } }, - }, - ] - - // Load all active users - await deduplicated.loadSubset({ where: eq(ref(`status`), val(`active`)) }) - expect(callCount).toBe(1) - - // Load top 10 active users by createdAt - const result1 = await deduplicated.loadSubset({ - where: eq(ref(`status`), val(`active`)), - orderBy: orderBy1, - limit: 10, - }) - expect(result1).toBe(true) // Covered by unlimited call - expect(callCount).toBe(1) - - // Load all inactive users - await deduplicated.loadSubset({ where: eq(ref(`status`), val(`inactive`)) }) - expect(callCount).toBe(2) - - // Load top 5 inactive users - const result2 = await deduplicated.loadSubset({ - where: eq(ref(`status`), val(`inactive`)), - orderBy: orderBy1, - limit: 5, - }) - expect(result2).toBe(true) // Covered by unlimited inactive call - expect(callCount).toBe(2) - - // Verify only 2 actual calls were made - expect(calls).toHaveLength(2) - expect(calls[0]).toEqual({ where: eq(ref(`status`), val(`active`)) }) - expect(calls[1]).toEqual({ where: eq(ref(`status`), val(`inactive`)) }) - }) - - describe(`subset deduplication with minusWherePredicates`, () => { - it(`should request only the difference for range predicates`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: age > 20 (loads data for age > 20) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) - expect(calls[0]).toEqual({ where: gt(ref(`age`), val(20)) }) - - // Second call: age > 10 (should request only age > 10 AND age <= 20) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: and(gt(ref(`age`), val(10)), lte(ref(`age`), val(20))), - }) - }) - - it(`should request only the difference for set predicates`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: status IN ['B', 'C'] (loads data for B and C) - await deduplicated.loadSubset({ - where: inOp(ref(`status`), [`B`, `C`]), - }) - expect(callCount).toBe(1) - expect(calls[0]).toEqual({ where: inOp(ref(`status`), [`B`, `C`]) }) - - // Second call: status IN ['A', 'B', 'C', 'D'] (should request only A and D) - await deduplicated.loadSubset({ - where: inOp(ref(`status`), [`A`, `B`, `C`, `D`]), - }) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: inOp(ref(`status`), [`A`, `D`]), - }) - }) - - it(`should return true immediately for complete overlap`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: age > 10 (loads data for age > 10) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(1) - - // Second call: age > 20 (completely covered by first call) - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(20)), - }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not make additional call - }) - - it(`should handle complex predicate differences`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: age > 20 AND status = 'active' - const firstPredicate = and( - gt(ref(`age`), val(20)), - eq(ref(`status`), val(`active`)), ) - await deduplicated.loadSubset({ where: firstPredicate }) - expect(callCount).toBe(1) - expect(calls[0]).toEqual({ where: firstPredicate }) + }, + ) + + it(`gives independently abortable demands independent transports`, async () => { + const pending: Array<() => void> = [] + const signals: Array = [] + const loadSubset = vi.fn( + (options) => + new Promise((resolve) => { + signals.push(options.signal) + pending.push(resolve) + }), + ) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) + const firstOwner = new AbortController() + const secondOwner = new AbortController() + + const first = deduplicated.loadSubset({ + limit: 2, + signal: firstOwner.signal, + }) + const second = deduplicated.loadSubset({ + limit: 2, + signal: secondOwner.signal, + }) + + expect(first).not.toBe(second) + expect(loadSubset).toHaveBeenCalledTimes(2) + expect(signals).toEqual([firstOwner.signal, secondOwner.signal]) + + pending.forEach((resolve) => resolve()) + await Promise.all([first, second]) + }) - // Second call: age > 10 AND status = 'active' (should request only age > 10 AND age <= 20 AND status = 'active') - const secondPredicate = and( - gt(ref(`age`), val(10)), - eq(ref(`status`), val(`active`)), + it(`does not cache work that settles after its owner aborts`, async () => { + let resolve!: () => void + const loadSubset = vi + .fn() + .mockImplementationOnce( + () => new Promise((done) => (resolve = done)), ) + .mockResolvedValue(undefined) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) + const owner = new AbortController() - await deduplicated.loadSubset({ where: secondPredicate }) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: and( - eq(ref(`status`), val(`active`)), - gt(ref(`age`), val(10)), - lte(ref(`age`), val(20)), - ), - }) - }) - - it(`should not apply subset logic to limited calls`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - // First call: unlimited age > 20 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) - - // Second call: limited age > 10 with orderBy + limit - // Should request the full predicate, not the difference, because it's limited - await deduplicated.loadSubset({ - where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - limit: 10, - }) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - limit: 10, - }) - }) - - it(`should handle undefined where clauses in subset logic`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: age > 20 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) - - // Second call: no where clause (all data) - // Should request all data except what we already loaded - // i.e. should request NOT (age > 20) - await deduplicated.loadSubset({}) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ where: not(gt(ref(`age`), val(20))) }) - - // After loading all data, subsequent calls should be deduplicated - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(5)), - }) - expect(result).toBe(true) - expect(callCount).toBe(2) - }) - - describe(`hasLoadedAllData after loading filtered + unfiltered data`, () => { - it(`should set hasLoadedAllData after a filtered load followed by an unfiltered load`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: inOp(ref(`task_id`), [`id1`, `id2`, `id3`]), - }) - expect(callCount).toBe(1) - - await deduplicated.loadSubset({}) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: not(inOp(ref(`task_id`), [`id1`, `id2`, `id3`])), - }) - - const result = await deduplicated.loadSubset({}) - expect(result).toBe(true) - expect(callCount).toBe(2) - }) - - it(`should set hasLoadedAllData after a filtered load followed by an unfiltered load (with eq)`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`single-id`)), - }) - expect(callCount).toBe(1) - - await deduplicated.loadSubset({}) - expect(callCount).toBe(2) - - const result1 = await deduplicated.loadSubset({}) - expect(result1).toBe(true) - expect(callCount).toBe(2) - - const result2 = await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`other-id`)), - }) - expect(result2).toBe(true) - expect(callCount).toBe(2) - }) - - it(`should not produce exponentially growing predicates on repeated unfiltered loads`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: inOp(ref(`task_id`), [`id1`, `id2`, `id3`]), - }) - expect(callCount).toBe(1) - - await deduplicated.loadSubset({}) - expect(callCount).toBe(2) - - const rounds: Array<{ round: number; whereSize: number }> = [] - for (let i = 0; i < 10; i++) { - const result = await deduplicated.loadSubset({}) - if (result !== true) { - const whereJson = JSON.stringify(calls[calls.length - 1]?.where) - rounds.push({ round: i + 1, whereSize: whereJson.length }) - } - } - - expect(callCount).toBe(2) - expect(rounds).toEqual([]) - }) - }) - - it(`should mark all data as loaded after a narrowed all-data request`, async () => { - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-1`)), - }) - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-2`)), - }) - - await deduplicated.loadSubset({}) - - expect(calls[2]).toEqual({ - where: not(inOp(ref(`task_id`), [`uuid-1`, `uuid-2`])), - }) - - expect((deduplicated as any).hasLoadedAllData).toBe(true) - expect((deduplicated as any).unlimitedWhere).toBeUndefined() - }) - - it(`should not keep issuing increasingly nested all-data predicates`, async () => { - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-1`)), - }) - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-2`)), - }) - - await deduplicated.loadSubset({}) - await deduplicated.loadSubset({}) - - expect(calls[3]).toBeUndefined() - }) - - it(`should deduplicate identical all-data requests while a narrowed all-data request is in flight`, async () => { - let resolveAllDataLoad: (() => void) | undefined - let callCount = 0 - const calls: Array = [] - const allDataLoadPromise = new Promise((resolve) => { - resolveAllDataLoad = resolve - }) - - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - - if (callCount === 2) { - return allDataLoadPromise - } - - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-1`)), - }) - - const firstAllDataLoad = deduplicated.loadSubset({}) - const secondAllDataLoad = deduplicated.loadSubset({}) - - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: not(eq(ref(`task_id`), val(`uuid-1`))), - }) - expect(secondAllDataLoad).toBe(firstAllDataLoad) - - resolveAllDataLoad?.() - await firstAllDataLoad - await secondAllDataLoad - }) - - it(`should not produce unbounded WHERE expressions when loading all data after eq accumulation`, async () => { - // This test reproduces the production bug where accumulating many eq predicates - // and then loading all data (no WHERE clause) caused unboundedly growing - // expressions instead of correctly setting hasLoadedAllData=true. - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // Simulate visiting multiple tasks, each adding an eq predicate - for (let i = 0; i < 10; i++) { - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-${i}`)), - }) - } - // After 10 eq calls, unlimitedWhere should be IN(task_id, [uuid-0, ..., uuid-9]) - expect(callCount).toBe(10) - - // Now load all data (no WHERE clause) - // This should send NOT(IN(...)) to the backend but track as "all data loaded" - await deduplicated.loadSubset({}) - expect(callCount).toBe(11) - - // The load request should be NOT(IN(task_id, [all accumulated uuids])) - const loadWhere = calls[10]!.where as any - expect(loadWhere.name).toBe(`not`) - expect(loadWhere.args[0].name).toBe(`in`) - expect(loadWhere.args[0].args[0].path).toEqual([`task_id`]) - const loadedUuids = ( - loadWhere.args[0].args[1].value as Array - ).sort() - const expectedUuids = Array.from( - { length: 10 }, - (_, i) => `uuid-${i}`, - ).sort() - expect(loadedUuids).toEqual(expectedUuids) - - // Critical: after loading all data, subsequent requests should be deduplicated - const result1 = await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-999`)), - }) - expect(result1).toBe(true) // Covered by "all data" load - expect(callCount).toBe(11) // No additional call - - // Loading all data again should also be deduplicated - const result2 = await deduplicated.loadSubset({}) - expect(result2).toBe(true) - expect(callCount).toBe(11) // Still no additional call - }) - - it(`should not produce unbounded WHERE expressions with synchronous loadSubset`, () => { - // Same scenario as the async accumulation test, but with a sync mock - // to exercise the sync return path (line 150 of subset-dedupe.ts) - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return true as const - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // Accumulate eq predicates via sync returns - for (let i = 0; i < 10; i++) { - deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-${i}`)), - }) - } - expect(callCount).toBe(10) - - // Load all data (no WHERE clause) — should track as "all data loaded" - deduplicated.loadSubset({}) - expect(callCount).toBe(11) - - // Subsequent requests should be deduplicated - const result1 = deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-999`)), - }) - expect(result1).toBe(true) - expect(callCount).toBe(11) - - const result2 = deduplicated.loadSubset({}) - expect(result2).toBe(true) - expect(callCount).toBe(11) - }) - - it(`should handle multiple all-data loads without expression growth`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First: load some specific data - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-1`)), - }) - expect(callCount).toBe(1) - - // Load all data (first time) - await deduplicated.loadSubset({}) - expect(callCount).toBe(2) - - // Load all data (second time) - should be deduplicated since we already have everything - const result = await deduplicated.loadSubset({}) - expect(result).toBe(true) - expect(callCount).toBe(2) // No additional call - all data already loaded - }) - - it(`should handle multiple overlapping unlimited calls`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: age > 20 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) + const first = deduplicated.loadSubset({ limit: 2, signal: owner.signal }) + owner.abort() + resolve() + await first + await deduplicated.loadSubset({ limit: 2 }) - // Second call: age < 10 (different range) - await deduplicated.loadSubset({ where: lt(ref(`age`), val(10)) }) - expect(callCount).toBe(2) + expect(loadSubset).toHaveBeenCalledTimes(2) + }) - // Third call: age > 5 (should request only age >= 10 AND age <= 20, since age < 10 is already covered) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(5)) }) - expect(callCount).toBe(3) + it(`retries an exact demand after rejection`, async () => { + const loadSubset = vi + .fn() + .mockRejectedValueOnce(new Error(`offline`)) + .mockResolvedValueOnce(undefined) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - // Ideally it would be smart enough to optimize it to request only age >= 10 AND age <= 20, since age < 10 is already covered - // However, it doesn't do that currently, so it will not optimize and execute the original query - expect(calls[2]).toEqual({ - where: gt(ref(`age`), val(5)), - }) + await expect(deduplicated.loadSubset({ limit: 2 })).rejects.toThrow( + `offline`, + ) + await deduplicated.loadSubset({ limit: 2 }) - /* - expect(calls[2]).toEqual({ - where: and(gte(ref(`age`), val(10)), lte(ref(`age`), val(20))), - }) - */ - }) + expect(loadSubset).toHaveBeenCalledTimes(2) }) - describe(`onDeduplicate callback`, () => { - it(`should call onDeduplicate when all data already loaded`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } + it(`erases completed and in-flight evidence on reset`, async () => { + const pending: Array<() => void> = [] + const loadSubset = vi.fn( + () => new Promise((resolve) => pending.push(resolve)), + ) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) + + const stale = deduplicated.loadSubset({ limit: 2 }) + deduplicated.reset() + const fresh = deduplicated.loadSubset({ limit: 2 }) + expect(loadSubset).toHaveBeenCalledTimes(2) + + pending[0]!() + await stale + expect(deduplicated.loadSubset({ limit: 2 })).toBe(fresh) + + pending[1]!() + await fresh + expect(deduplicated.loadSubset({ limit: 2 })).toBe(true) + }) - const onDeduplicate = vi.fn() - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - onDeduplicate, + it(`does not retain synchronous work from before a reentrant reset`, () => { + const loadSubset = vi + .fn() + .mockImplementationOnce(() => { + deduplicated.reset() + return true }) + .mockReturnValue(true) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - // Load all data - await deduplicated.loadSubset({}) - expect(callCount).toBe(1) - - // Any subsequent request should be deduplicated - const subsetOptions = { where: gt(ref(`age`), val(10)) } - const result = await deduplicated.loadSubset(subsetOptions) - expect(result).toBe(true) - expect(callCount).toBe(1) - expect(onDeduplicate).toHaveBeenCalledTimes(1) - expect(onDeduplicate).toHaveBeenCalledWith(subsetOptions) - }) - - it(`should call onDeduplicate when unlimited superset already loaded`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } + expect(deduplicated.loadSubset({ limit: 2 })).toBe(true) + expect(deduplicated.loadSubset({ limit: 2 })).toBe(true) + expect(loadSubset).toHaveBeenCalledTimes(2) + }) - const onDeduplicate = vi.fn() - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - onDeduplicate: onDeduplicate, + it(`does not retain asynchronous work from before a reentrant reset`, async () => { + let resolveStale!: () => void + const loadSubset = vi + .fn() + .mockImplementationOnce(() => { + deduplicated.reset() + return new Promise((resolve) => (resolveStale = resolve)) }) + .mockResolvedValue(undefined) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - // First call loads a broader set - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(1) - - // Second call is a subset of the first; should dedupe and call callback - const subsetOptions = { where: gt(ref(`age`), val(20)) } - const result = await deduplicated.loadSubset(subsetOptions) - expect(result).toBe(true) - expect(callCount).toBe(1) - expect(onDeduplicate).toHaveBeenCalledTimes(1) - expect(onDeduplicate).toHaveBeenCalledWith(subsetOptions) - }) - - it(`should call onDeduplicate for limited subset requests`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } + const stale = deduplicated.loadSubset({ limit: 2 }) + const fresh = deduplicated.loadSubset({ limit: 2 }) + expect(loadSubset).toHaveBeenCalledTimes(2) - const onDeduplicate = vi.fn() - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - onDeduplicate, - }) + resolveStale() + await Promise.all([stale, fresh]) + }) - const orderBy1: OrderBy = [ + it.each([ + { + name: `Date`, + value: new Date(7), + equal: new Date(7), + different: new Date(8), + }, + { + name: `binary`, + value: new Uint8Array([1]), + equal: new Uint8Array([1]), + different: new Uint8Array([2]), + }, + { + name: `Buffer`, + value: Buffer.from([1]), + equal: new Uint8Array([1]), + different: Buffer.from([2]), + }, + ])( + `passes immutable $name values through and deduplicates by equality`, + ({ value, equal, different }) => { + const loadSubset = vi.fn().mockReturnValue(true) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) + const options = { where: eq(ref(`key`), val(value)) } + deduplicated.loadSubset(options) + expect(loadSubset.mock.calls[0]![0]).toBe(options) + const matches = compileSingleRowExpression( + loadSubset.mock.calls[0]![0].where!, + ) + expect([value, equal, different].map((key) => matches({ key }))).toEqual([ + true, + true, + false, + ]) + expect( + deduplicated.loadSubset({ where: eq(ref(`key`), val(equal)) }), + ).toBe(true) + expect(loadSubset).toHaveBeenCalledTimes(1) + deduplicated.loadSubset({ where: eq(ref(`key`), val(different)) }) + expect(loadSubset).toHaveBeenCalledTimes(2) + }, + ) + + it(`keeps immutable order and cursor data with its opaque identity`, () => { + const opaque = Object.freeze({ id: 1 }) + const options: LoadSubsetOptions = { + orderBy: [ { - expression: ref(`age`), + expression: ref(`rank`), compareOptions: { direction: `asc`, - nulls: `last`, - stringSort: `lexical`, + nulls: `first`, + stringSort: `locale`, + localeOptions: Object.freeze({ numeric: true }), }, }, - ] - - const whereClause = gt(ref(`age`), val(10)) - - // First limited call - await deduplicated.loadSubset({ - where: whereClause, - orderBy: orderBy1, - limit: 10, - }) - expect(callCount).toBe(1) + ], + cursor: { + whereFrom: gt(ref(`rank`), val(opaque)), + whereCurrent: eq(ref(`rank`), val(opaque)), + }, + } + const request = captureRequest(options) + expect(request).toBe(options) + expect(((request.cursor!.whereFrom as Func).args[1] as Value).value).toBe( + opaque, + ) + expect( + compileSingleRowExpression(request.cursor!.whereCurrent)({ + rank: opaque, + }), + ).toBe(true) + expect( + compileSingleRowExpression(request.cursor!.whereCurrent)({ + rank: { id: 1 }, + }), + ).toBe(false) + }) - // Second limited call is a subset (SAME where clause and smaller limit) - // For limited queries, where clauses must be EQUAL for subset relationship - const subsetOptions = { - where: whereClause, // Same where clause - orderBy: orderBy1, - limit: 5, - } - const result = await deduplicated.loadSubset(subsetOptions) - expect(result).toBe(true) - expect(callCount).toBe(1) - expect(onDeduplicate).toHaveBeenCalledTimes(1) - expect(onDeduplicate).toHaveBeenCalledWith(subsetOptions) + it(`keeps completed cursor requests distinct from replacement Date constants`, async () => { + const loadSubset = vi.fn().mockResolvedValue(undefined) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) + const request = (year: number): LoadSubsetOptions => ({ + cursor: { + whereFrom: gt(ref(`createdAt`), val(new Date(year, 0))), + whereCurrent: eq(ref(`createdAt`), val(new Date(year, 0))), + }, + limit: 10, }) + await deduplicated.loadSubset(request(2025)) + await deduplicated.loadSubset(request(2026)) + expect(deduplicated.loadSubset(request(2025))).toBe(true) + expect(loadSubset).toHaveBeenCalledTimes(2) + }) - it(`should delay onDeduplicate until covering in-flight request completes`, async () => { - let resolveFirst: (() => void) | undefined - let callCount = 0 - const firstPromise = new Promise((resolve) => { - resolveFirst = () => resolve() - }) + it(`does not substitute comparison payloads with custom instance methods`, () => { + const date = new Date(2) + const bytes = new Uint8Array([1, 2, 3]) + Object.defineProperty(date, `getTime`, { value: () => 1 }) + Object.defineProperty(bytes, `slice`, { value: () => bytes }) + const where = new Func(`and`, [ + eq(ref(`date`), val(date)), + eq(ref(`bytes`), val(bytes)), + ]) + const request = captureRequest({ where }) + const rows = [ + { date, bytes }, + { date: new Date(2), bytes: new Uint8Array([1, 2, 3]) }, + ] + expect(request.where).toBe(where) + expect(rows.map(compileSingleRowExpression(request.where!))).toEqual( + rows.map(compileSingleRowExpression(where)), + ) + }) - // First call will remain in-flight until we resolve it - let first = true - const mockLoadSubset = (_options: LoadSubsetOptions) => { - callCount++ - if (first) { - first = false - return firstPromise + describe.each([`Date`, `Uint8Array`] as const)( + `request transport preserves %s predicate matches`, + (type) => { + it.each([`local`, `foreign`] as const)(`in the %s realm`, (realm) => { + const local = type === `Date` ? new Date(2) : new Uint8Array([1, 2]) + const foreign: unknown = runInNewContext( + type === `Date` ? `new Date(2)` : `new Uint8Array([1, 2])`, + ) + const value = realm === `local` ? local : foreign + for (const where of [ + eq(ref(`value`), val(value)), + new Func(`in`, [ref(`value`), val([value])]), + ]) { + const request = captureRequest({ where }) + const matches = compileSingleRowExpression(request.where!) + expect( + [foreign, local].map((item) => matches({ value: item })), + ).toEqual(realm === `foreign` ? [true, false] : [false, true]) } - return Promise.resolve() - } - - const onDeduplicate = vi.fn() - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - onDeduplicate: onDeduplicate, }) - - // Start a broad in-flight request - const inflightOptions = { where: gt(ref(`age`), val(10)) } - const inflight = deduplicated.loadSubset(inflightOptions) - expect(inflight).toBeInstanceOf(Promise) - expect(callCount).toBe(1) - - // Issue a subset request while first is still in-flight - const subsetOptions = { where: gt(ref(`age`), val(20)) } - const subsetPromise = deduplicated.loadSubset(subsetOptions) - expect(subsetPromise).toBeInstanceOf(Promise) - - // onDeduplicate should NOT have fired yet - expect(onDeduplicate).not.toHaveBeenCalled() - - // Complete the first request - resolveFirst?.() - - // Wait for the subset promise to settle (which chains the first) - await subsetPromise - - // Now the callback should have been called exactly once, with the subset options - expect(onDeduplicate).toHaveBeenCalledTimes(1) - expect(onDeduplicate).toHaveBeenCalledWith(subsetOptions) - }) + }, + ) + + it.each([`coalesce`, `caseWhen`] as const)( + `preserves membership results through %s`, + (wrapper) => { + const candidates = Object.freeze([new Uint8Array([1])]) + const expression = + wrapper === `coalesce` + ? new Func(`coalesce`, [val(candidates)]) + : new Func(`caseWhen`, [val(true), val(candidates), val([])]) + const request = captureRequest({ + where: new Func(`in`, [ref(`token`), expression]), + }) + const matches = compileSingleRowExpression(request.where!) + expect( + [1, 2, 3].map((n) => matches({ token: new Uint8Array([n]) })), + ).toEqual([true, false, false]) + expect(candidates).toEqual([new Uint8Array([1])]) + }, + ) + + it(`preserves immutable array ordering operands`, () => { + const boundary = Object.freeze([1, Object.freeze([2])]) + const request = captureRequest({ where: gt(ref(`tuple`), val(boundary)) }) + const matches = compileSingleRowExpression(request.where!) + expect( + [ + [1, [1]], + [1, [2]], + [1, [3]], + ].map((tuple) => matches({ tuple })), + ).toEqual([false, false, true]) }) - describe(`limited queries with different where clauses`, () => { - // When a query has a limit, only the top N rows (by orderBy) are loaded. - // A subsequent query with a different where clause cannot reuse that data, - // even if the new where clause is "more restrictive", because the filtered - // top N might include rows outside the original unfiltered top N. - - it(`should NOT dedupe when where clause differs on limited queries`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(options) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderByCreatedAt: OrderBy = [ - { - expression: ref(`created_at`), - compareOptions: { - direction: `desc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - // First query: top 10 items with no filter - await deduplicated.loadSubset({ - where: undefined, - orderBy: orderByCreatedAt, - limit: 10, - }) - expect(callCount).toBe(1) - - // Second query: top 10 items WITH a filter - // This requires a separate request because the filtered top 10 - // might include items outside the unfiltered top 10 - const searchWhere = and(eq(ref(`title`), val(`test`))) - await deduplicated.loadSubset({ - where: searchWhere, - orderBy: orderByCreatedAt, - limit: 10, - }) - - expect(callCount).toBe(2) - expect(calls[1]?.where).toEqual(searchWhere) - }) - - it(`should dedupe when where clause is identical on limited queries`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderByCreatedAt: OrderBy = [ - { - expression: ref(`created_at`), - compareOptions: { - direction: `desc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - // First query: top 10 items with no filter - await deduplicated.loadSubset({ - where: undefined, - orderBy: orderByCreatedAt, - limit: 10, - }) - expect(callCount).toBe(1) - - // Second query: same where clause (undefined), smaller limit - // The top 5 are contained within the already-loaded top 10 - const result = await deduplicated.loadSubset({ - where: undefined, - orderBy: orderByCreatedAt, - limit: 5, - }) - expect(result).toBe(true) - expect(callCount).toBe(1) - }) - - it(`should not let caller mutations change stored limited call orderBy`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const mutableOrderBy: OrderBy = [ - { - expression: ref(`created_at`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - await deduplicated.loadSubset({ - where: eq(ref(`status`), val(`active`)), - orderBy: mutableOrderBy, - limit: 10, - }) - expect(callCount).toBe(1) - - mutableOrderBy[0]!.compareOptions.direction = `desc` - - const originalOrderBy: OrderBy = [ - { - expression: ref(`created_at`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] + it.each([`in`, `gt`])(`preserves immutable sparse %s array data`, (name) => { + const values = new Array(3) + values[1] = new Date(7) + Object.freeze(values) + const request = captureRequest({ + where: new Func(name, [ref(`value`), val(values)]), + }) + const payload = ((request.where as Func).args[1] as Value>) + .value + expect(payload).toBe(values) + expect(payload.length).toBe(3) + expect(Object.hasOwn(payload, 0)).toBe(false) + expect(Object.hasOwn(payload, 2)).toBe(false) + expect(payload[1]!.getTime()).toBe(7) + }) - const result = await deduplicated.loadSubset({ - where: eq(ref(`status`), val(`active`)), - orderBy: originalOrderBy, - limit: 5, - }) + it.each([`in`, `gt`])( + `preserves nested-array comparison semantics for %s`, + (name) => { + const nested = [2] + const values = Object.freeze([nested]) + const request = captureRequest({ + where: new Func(name, [ref(`value`), val(values)]), + }) + const matches = compileSingleRowExpression(request.where!) + const rows = name === `in` ? [nested, [2]] : [[[1]], [[2]], [[3]]] + expect(rows.map((value) => matches({ value }))).toEqual( + name === `in` ? [true, false] : [false, false, true], + ) + }, + ) +}) - expect(result).toBe(true) - expect(callCount).toBe(1) - }) +function captureRequest(options: LoadSubsetOptions): LoadSubsetOptions { + let request!: LoadSubsetOptions + const deduplicated = new DeduplicatedLoadSubset({ + loadSubset: (value) => { + request = value + return true + }, }) -}) + deduplicated.loadSubset(options) + return request +} diff --git a/packages/db/tests/query/subset-error-matrix.test.ts b/packages/db/tests/query/subset-error-matrix.test.ts new file mode 100644 index 0000000000..6ee34cf2fc --- /dev/null +++ b/packages/db/tests/query/subset-error-matrix.test.ts @@ -0,0 +1,580 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createEffect, createLiveQueryCollection, eq } from '../../src/index.js' +import { getLoadSubsetDemandKey } from '../../src/query/ir-stable-identity.js' +import { mockSyncCollectionOptions } from '../utils.js' + +type Delivery = `throw` | `reject` +type Consumer = `effect` | `live` +type StartupPath = `direct` | `ordered` | `lazy` +type IncrementalPath = Exclude +type FailureValue = `error` | `nan` | `undefined` + +type Row = { + id: number + rank: number + parentId: number +} + +type FailureCase = { + name: string + consumer: Consumer + path: TPath + delivery: Delivery +} + +type IncrementalFailureCase = FailureCase & { + failureValue: FailureValue +} + +type CleanupFailureCase = { + name: string + consumer: Consumer + failure: unknown +} + +const row: Row = { id: 1, rank: 1, parentId: 1 } + +// Every query form can fail while it acquires initial coverage. +const startupCases: ReadonlyArray> = ( + [`effect`, `live`] as const +).flatMap((consumer) => + ([`direct`, `ordered`, `lazy`] as const).flatMap((path) => + ([`throw`, `reject`] as const).map((delivery) => ({ + name: `${consumer} ${path} ${delivery}`, + consumer, + path, + delivery, + })), + ), +) + +// Direct queries have no automatic later demand. Ordered refills and lazy +// relationship routes do, so only those paths have incremental cells. +const incrementalCases: ReadonlyArray = ( + [`effect`, `live`] as const +).flatMap((consumer) => + ([`ordered`, `lazy`] as const).flatMap((path) => + ([`throw`, `reject`] as const).flatMap((delivery) => + ([`error`, `nan`, `undefined`] as const).map((failureValue) => ({ + name: `${consumer} ${path} ${delivery} ${failureValue}`, + consumer, + path, + delivery, + failureValue, + })), + ), + ), +) + +const cleanupFailureObject = { kind: `cleanup-failure` } +const cleanupFailureCases: ReadonlyArray = ( + [`effect`, `live`] as const +).flatMap((consumer) => [ + { name: `${consumer} undefined`, consumer, failure: undefined }, + { name: `${consumer} NaN`, consumer, failure: Number.NaN }, + { name: `${consumer} object`, consumer, failure: cleanupFailureObject }, +]) + +function fail(delivery: Delivery, error: unknown): Promise { + if (delivery === `throw`) throw error + return Promise.reject(error) +} + +function createFailingSource( + id: string, + delivery: Delivery, + error: unknown, + onLoad = () => {}, +) { + return createCollection({ + id, + getKey: (item) => item.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + onLoad() + return fail(delivery, error) + }, + } + }, + }, + }) +} + +function createStaticSource(id: string, initialData: ReadonlyArray) { + return createCollection( + mockSyncCollectionOptions({ + id, + getKey: (item) => item.id, + initialData: [...initialData], + }), + ) +} + +type RowCollection = ReturnType + +function startEffect( + path: StartupPath, + primary: RowCollection, + child: RowCollection, + sourceErrors: Array, +) { + const callbacks = { + onBatch: () => {}, + onSourceError: (error: Error) => sourceErrors.push(error), + } + if (path === `ordered`) { + return createEffect({ + query: (q) => + q + .from({ item: primary }) + .orderBy(({ item }) => item.rank, `asc`) + .limit(1), + ...callbacks, + }) + } + if (path === `lazy`) { + return createEffect({ + query: (q) => + q + .from({ item: primary }) + .leftJoin({ child }, ({ item, child: childRow }) => + eq(item.id, childRow.parentId), + ), + ...callbacks, + }) + } + return createEffect({ + query: (q) => q.from({ item: primary }), + ...callbacks, + }) +} + +function startLive( + path: StartupPath, + primary: RowCollection, + child: RowCollection, +) { + if (path === `ordered`) { + return createLiveQueryCollection((q) => + q + .from({ item: primary }) + .orderBy(({ item }) => item.rank, `asc`) + .limit(1), + ) + } + if (path === `lazy`) { + return createLiveQueryCollection((q) => + q + .from({ item: primary }) + .leftJoin({ child }, ({ item, child: childRow }) => + eq(item.id, childRow.parentId), + ), + ) + } + return createLiveQueryCollection((q) => q.from({ item: primary })) +} + +async function flushFailures() { + await new Promise((resolve) => setTimeout(resolve, 0)) + await new Promise((resolve) => setTimeout(resolve, 0)) +} + +describe(`loadSubset failure matrix`, () => { + it.each(startupCases)( + `releases startup ownership and reports the source error: $name`, + async ({ consumer, path, delivery }) => { + const error = new Error(`${consumer} ${path} startup failed`) + const suffix = `${consumer}-${path}-${delivery}` + const directOrOrderedSource = createFailingSource( + `failure-matrix-startup-primary-${suffix}`, + delivery, + error, + ) + const lazyParent = createStaticSource( + `failure-matrix-startup-parent-${suffix}`, + [row], + ) + const lazyChild = createFailingSource( + `failure-matrix-startup-child-${suffix}`, + delivery, + error, + ) + const primary = path === `lazy` ? lazyParent : directOrOrderedSource + const child = path === `lazy` ? lazyChild : directOrOrderedSource + + try { + if (consumer === `effect`) { + const sourceErrors: Array = [] + if (delivery === `throw`) { + expect(() => + startEffect(path, primary, child, sourceErrors), + ).toThrow(error) + } else { + const effect = startEffect(path, primary, child, sourceErrors) + await flushFailures() + expect(effect.disposed).toBe(true) + await effect.dispose() + } + expect(sourceErrors).toEqual([error]) + } else { + const live = startLive(path, primary, child) + try { + await expect( + Promise.resolve().then(() => live.preload()), + ).rejects.toBe(error) + expect(live.status).toBe(`error`) + } finally { + await live.cleanup() + } + } + + expect(primary.subscriberCount).toBe(0) + if (path === `lazy`) expect(child.subscriberCount).toBe(0) + } finally { + await Promise.all([ + directOrOrderedSource.cleanup(), + lazyParent.cleanup(), + lazyChild.cleanup(), + ]) + } + }, + ) + + it.each(incrementalCases)( + `reports an incremental failure without escaping its source commit: $name`, + async ({ consumer, path, delivery, failureValue }) => { + const error: unknown = + failureValue === `nan` + ? Number.NaN + : failureValue === `undefined` + ? undefined + : new Error(`${consumer} ${path} incremental failed`) + const suffix = `${consumer}-${path}-${delivery}-${failureValue}` + let triggerFailure: () => void + let primary: RowCollection + let child: RowCollection + let loadCount = 0 + const orderedLoadKeys: Array = [] + let loadsBeforeFailure = 0 + let failureArmed = false + + if (path === `ordered`) { + let begin!: () => void + let write!: (message: { type: `insert` | `delete`; value: Row }) => void + let commit!: () => void + primary = createCollection({ + id: `failure-matrix-incremental-ordered-${suffix}`, + getKey: (item) => item.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options) => { + loadCount++ + orderedLoadKeys.push(getLoadSubsetDemandKey(options)) + if (failureArmed) return fail(delivery, error) + // Initial coverage includes tie-boundary refinement, not + // just the first page. Inject failure only after it settles. + if (loadCount > 1) return true + begin() + write({ type: `insert`, value: row }) + commit() + return true + }, + } + }, + }, + }) + child = primary + triggerFailure = () => { + loadsBeforeFailure = orderedLoadKeys.length + failureArmed = true + begin() + write({ type: `delete`, value: row }) + commit() + } + } else { + primary = createStaticSource( + `failure-matrix-incremental-parent-${suffix}`, + [], + ) + child = createFailingSource( + `failure-matrix-incremental-child-${suffix}`, + delivery, + error, + () => loadCount++, + ) + triggerFailure = () => { + primary.utils.begin() + primary.utils.write({ type: `insert`, value: row }) + primary.utils.commit() + } + } + + try { + if (consumer === `effect`) { + const sourceErrors: Array = [] + const effect = startEffect(path, primary, child, sourceErrors) + try { + await flushFailures() + expect(sourceErrors).toEqual([]) + expect(effect.disposed).toBe(false) + expect(() => triggerFailure()).not.toThrow() + await flushFailures() + + expect(sourceErrors).toHaveLength(1) + if (failureValue === `error`) { + expect(sourceErrors[0]).toBe(error) + } else { + expect(sourceErrors[0]).toBeInstanceOf(Error) + } + expect(effect.disposed).toBe(true) + } finally { + await effect.dispose() + } + } else { + const live = startLive(path, primary, child) + try { + await live.preload() + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBeUndefined() + expect(() => triggerFailure()).not.toThrow() + await flushFailures() + + expect(live.status).toBe(path === `lazy` ? `error` : `ready`) + if (failureValue === `error`) { + expect(live.utils.lastSubsetError).toBe(error) + } else { + expect(live.utils.lastSubsetError).toBeInstanceOf(Error) + } + } finally { + await live.cleanup() + } + } + + if (path === `ordered`) { + const incrementalKeys = orderedLoadKeys.slice(loadsBeforeFailure) + expect(loadsBeforeFailure).toBeGreaterThan(1) + expect(incrementalKeys.length).toBeGreaterThan(0) + expect(new Set(incrementalKeys).size).toBe(incrementalKeys.length) + } else { + expect(loadCount).toBe(1) + } + + expect(primary.subscriberCount).toBe(0) + if (path === `lazy`) expect(child.subscriberCount).toBe(0) + } finally { + await Promise.all( + primary === child + ? [primary.cleanup()] + : [primary.cleanup(), child.cleanup()], + ) + } + }, + ) + + it.each(cleanupFailureCases)( + `reports obsolete-demand cleanup failure without failing the source commit: $name`, + async ({ consumer, failure }) => { + const suffix = `${consumer}-${ + failure === undefined + ? `undefined` + : typeof failure === `number` + ? `nan` + : `object` + }` + const parent = createStaticSource(`cleanup-failure-parent-${suffix}`, [ + row, + ]) + let unloadCount = 0 + const child = createCollection({ + id: `cleanup-failure-child-${suffix}`, + getKey: (item) => item.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloadCount++ + if (unloadCount === 1) throw failure + }, + } + }, + }, + }) + const sourceErrors: Array = [] + const effect = + consumer === `effect` + ? createEffect({ + query: (q) => + q + .from({ item: parent }) + .leftJoin({ child }, ({ item, child: childRow }) => + eq(item.id, childRow.parentId), + ), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }) + : undefined + const live = + consumer === `live` + ? createLiveQueryCollection((q) => + q + .from({ item: parent }) + .leftJoin({ child }, ({ item, child: childRow }) => + eq(item.id, childRow.parentId), + ), + ) + : undefined + + try { + if (live) await live.preload() + await flushFailures() + + expect(() => { + parent.utils.begin() + parent.utils.write({ type: `delete`, value: row }) + parent.utils.commit() + }).not.toThrow() + + await flushFailures() + + if (effect) { + expect(sourceErrors).toHaveLength(1) + expect(sourceErrors[0]?.message).toBe(String(failure)) + expect(effect.disposed).toBe(true) + } + if (live) { + expect(live.utils.lastSubsetError).toBeInstanceOf(Error) + expect(live.status).toBe(`ready`) + } + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + expect(unloadCount).toBe(1) + await Promise.all([parent.cleanup(), child.cleanup()]) + } + }, + ) + + it.each([undefined, NaN, new Error(`release failed`)])( + `does not repeat failed release after %s survives demand retirement`, + async (failure) => { + const parent = createStaticSource(`undefined-cleanup-retry-parent`, [row]) + let unloadCount = 0 + const child = createCollection({ + id: `undefined-cleanup-retry-child`, + getKey: (item) => item.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloadCount++ + if (unloadCount <= 2) throw failure + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ item: parent }) + .leftJoin({ child }, ({ item, child: childRow }) => + eq(item.id, childRow.parentId), + ), + ) + const originalQueueMicrotask = globalThis.queueMicrotask + const queuedMicrotasks: Array<() => void> = [] + + try { + await live.preload() + + parent.utils.begin() + parent.utils.write({ type: `delete`, value: row }) + parent.utils.commit() + await flushFailures() + + expect(unloadCount).toBe(1) + expect(live.utils.lastSubsetError).toBeInstanceOf(Error) + + globalThis.queueMicrotask = (callback) => { + queuedMicrotasks.push(callback) + } + await live.cleanup() + expect(unloadCount).toBe(1) + expect(queuedMicrotasks).toHaveLength(0) + await live.cleanup() + expect(unloadCount).toBe(1) + expect(parent.subscriberCount).toBe(0) + expect(child.subscriberCount).toBe(0) + await live.cleanup() + expect(unloadCount).toBe(1) + } finally { + globalThis.queueMicrotask = originalQueueMicrotask + await Promise.all([live.cleanup(), parent.cleanup(), child.cleanup()]) + } + }, + ) + + it(`preserves a synchronous ordered error after reentrant cleanup`, async () => { + const error = new Error(`ordered load failed after cleanup`) + let cleanupLive: () => Promise = () => Promise.resolve() + const source = createCollection({ + id: `ordered-reentrant-cleanup-error`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + void cleanupLive() + throw error + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ item: source }) + .orderBy(({ item }) => item.rank) + .limit(0), + ) + cleanupLive = () => live.cleanup() + + try { + await live.preload() + expect(() => live.utils.setWindow({ offset: 0, limit: 1 })).toThrow(error) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) +}) diff --git a/packages/db/tests/query/union-all.test.ts b/packages/db/tests/query/union-all.test.ts index a94308bb27..ce5f0faf51 100644 --- a/packages/db/tests/query/union-all.test.ts +++ b/packages/db/tests/query/union-all.test.ts @@ -18,6 +18,7 @@ import { } from '../utils.js' import { OnlyOneSourceAllowedError } from '../../src/errors.js' import type { LoadSubsetOptions } from '../../src/types.js' +import type { BasicExpression } from '../../src/query/ir.js' type Message = { id: number @@ -34,6 +35,18 @@ type ToolCall = { userId: number } +function referencesField( + expression: BasicExpression | undefined, + field: string, +): boolean { + if (!expression) return false + if (expression.type === `ref`) return expression.path.includes(field) + if (expression.type !== `func`) return false + return expression.args.some((argument) => + referencesField(argument as BasicExpression, field), + ) +} + type Chunk = { id: number messageId: number @@ -1234,7 +1247,9 @@ describe(`unionAll`, () => { expect(messageLoadSubsetCalls.length).toBeGreaterThan(0) expect(toolLoadSubsetCalls.length).toBeGreaterThan(0) expect( - messageLoadSubsetCalls.every((call) => call.where === undefined), + messageLoadSubsetCalls.every( + (call) => !referencesField(call.where, `userId`), + ), ).toBe(true) expect(toolLoadSubsetCalls.every((call) => call.where === undefined)).toBe( true, @@ -1291,7 +1306,9 @@ describe(`unionAll`, () => { expect(messageLoadSubsetCalls.length).toBeGreaterThan(0) expect(toolLoadSubsetCalls.length).toBeGreaterThan(0) expect( - messageLoadSubsetCalls.every((call) => call.where === undefined), + messageLoadSubsetCalls.every( + (call) => !referencesField(call.where, `userId`), + ), ).toBe(true) expect(toolLoadSubsetCalls.some((call) => call.where)).toBe(true) }) diff --git a/packages/db/tests/query/validate-aliases.test.ts b/packages/db/tests/query/validate-aliases.test.ts index e09635de9b..bdfa2955fd 100644 --- a/packages/db/tests/query/validate-aliases.test.ts +++ b/packages/db/tests/query/validate-aliases.test.ts @@ -81,6 +81,23 @@ describe(`Alias validation in subqueries`, () => { }).toThrow(/Subquery uses alias "vote"/) }) + test(`should throw DuplicateAliasInSubqueryError when an include reuses a parent alias`, () => { + expect(() => { + createLiveQueryCollection({ + startSync: true, + query: (q) => + q.from({ lock: locksCollection }).select(({ lock: parentLock }) => ({ + _id: parentLock._id, + votes: q + .from({ lock: votesCollection }) + .where(({ lock: childLock }) => + eq(childLock.lockId, parentLock._id), + ), + })), + }) + }).toThrow(/Subquery uses alias "lock"/) + }) + test(`should allow subqueries when all collection aliases are unique`, () => { const query = createLiveQueryCollection({ startSync: true, diff --git a/packages/db/tests/reference-expression.ts b/packages/db/tests/reference-expression.ts new file mode 100644 index 0000000000..839a7de33b --- /dev/null +++ b/packages/db/tests/reference-expression.ts @@ -0,0 +1,64 @@ +import type { BasicExpression } from '../src/query/ir.js' + +function compareReferenceValues(left: unknown, right: unknown): number { + if (left === right) return 0 + if (typeof left === `number` && typeof right === `number`) { + return left < right ? -1 : 1 + } + if (typeof left === `string` && typeof right === `string`) { + return left < right ? -1 : 1 + } + throw new Error(`reference comparison requires like-typed numbers or strings`) +} + +/** Evaluate the BasicExpression subset used by test-only reference models. */ +export function evaluateReferenceExpression( + expression: BasicExpression, + row: object, +): unknown { + if (expression.type === `val`) return expression.value + if (expression.type === `ref`) { + let value: unknown = row + for (const segment of expression.path) { + if (typeof value !== `object` || value === null) return undefined + value = (value as Record)[segment] + } + return value + } + + const args = expression.args.map((argument) => + evaluateReferenceExpression(argument, row), + ) + switch (expression.name) { + case `and`: + return args.every(Boolean) + case `or`: + return args.some(Boolean) + case `not`: + return !args[0] + case `isNull`: + return args[0] === null + case `isUndefined`: + return args[0] === undefined + case `eq`: + if (args[0] == null || args[1] == null) return null + return args[0] === args[1] + case `gt`: + if (args[0] == null || args[1] == null) return null + return compareReferenceValues(args[0], args[1]) > 0 + case `gte`: + if (args[0] == null || args[1] == null) return null + return compareReferenceValues(args[0], args[1]) >= 0 + case `lt`: + if (args[0] == null || args[1] == null) return null + return compareReferenceValues(args[0], args[1]) < 0 + case `lte`: + if (args[0] == null || args[1] == null) return null + return compareReferenceValues(args[0], args[1]) <= 0 + case `in`: + if (!Array.isArray(args[1])) throw new Error(`IN requires an array`) + return args[1].includes(args[0]) + default: + throw new Error(`unsupported reference expression: ${expression.name}`) + } +} diff --git a/packages/db/tests/replay-adapter-ownership.test.ts b/packages/db/tests/replay-adapter-ownership.test.ts new file mode 100644 index 0000000000..a2913cfd80 --- /dev/null +++ b/packages/db/tests/replay-adapter-ownership.test.ts @@ -0,0 +1,143 @@ +import { expect, it } from 'vitest' +import { createCollection } from '../src/collection' +import { createDeferred } from '../src/deferred' +import { flushPromises } from './utils' +import type { LoadSubsetOptions, SyncConfig } from '../src/types' + +type Row = { id: number; version: number } + +it.each( + [1, 2].flatMap((owners) => + ([`resolve`, `reject`, `throw`] as const).map((outcome) => ({ + owners, + outcome, + })), + ), +)( + `keeps replay adapter ownership balanced: %j`, + async ({ owners, outcome }) => { + const liveLeases = new Set() + const loads: Array = [] + const releases: Array = [] + const pending: Array>> = [] + const failure = new Error(`adapter startup failed`) + let generation = 1 + let starts = 0 + let stops = 0 + let sync!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + if (liveLeases.size === 0) starts++ + liveLeases.add(options) + operations.begin() + operations.write({ + type: source.has(1) ? `update` : `insert`, + value: { id: 1, version: generation }, + }) + operations.commit() + if (generation === 2 && outcome === `throw`) { + // The adapter, not unloadSubset, owns rollback of a throw. + liveLeases.delete(options) + if (liveLeases.size === 0) stops++ + throw failure + } + loads.push(options) + if (generation !== 2) return true + const result = createDeferred() + pending.push(result) + return result.promise + }, + unloadSubset: (options) => { + expect(liveLeases.delete(options)).toBe(true) + releases.push(options) + if (liveLeases.size === 0) stops++ + }, + } + }, + }, + }) + const views = Array.from( + { length: owners }, + () => new Map(), + ) + const subscriptions = views.map((view) => + source.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) view.delete(change.key) + else view.set(change.key, change.value.version) + } + }, + { includeInitialState: false }, + ), + ) + try { + for (const subscription of subscriptions) subscription.requestSnapshot({}) + expect(liveLeases.size).toBe(owners) + expect(starts).toBe(1) + expect(stops).toBe(0) + generation = 2 + sync.begin() + sync.truncate() + sync.commit() + const waiters = Promise.allSettled( + subscriptions.map( + (subscription) => subscription.pendingTruncateReplacement, + ), + ) + await flushPromises() + for (const view of views) expect([...view.values()]).toEqual([1]) + // Distinct logical owners may share one adapter resource. Retiring one + // must never stop it while another successful owner still holds a lease. + if (owners === 2 && outcome !== `throw`) expect(stops).toBe(0) + if (owners === 1) { + expect(starts).toBe(2) + expect(stops).toBe(outcome === `throw` ? 2 : 1) + } + for (const result of pending) { + if (outcome === `reject`) result.reject(failure) + else result.resolve() + } + const settled = await waiters + expect(settled).toEqual( + Array.from({ length: owners }, () => + outcome === `resolve` + ? { status: `fulfilled`, value: undefined } + : { status: `rejected`, reason: failure }, + ), + ) + await flushPromises() + for (const view of views) + expect([...view.values()]).toEqual([outcome === `resolve` ? 2 : 1]) + + generation = 3 + sync.begin() + sync.truncate() + sync.commit() + await flushPromises() + for (const view of views) expect([...view.values()]).toEqual([3]) + expect(liveLeases.size).toBe(owners) + subscriptions[0]!.unsubscribe() + expect(liveLeases.size).toBe(owners - 1) + for (const subscription of subscriptions) subscription.unsubscribe() + expect(liveLeases.size).toBe(0) + expect(starts).toBe(stops) + expect(releases).toHaveLength(loads.length) + for (const options of loads) + expect( + releases.filter((released) => released === options), + ).toHaveLength(1) + } finally { + for (const result of pending) result.resolve() + for (const subscription of subscriptions) subscription.unsubscribe() + await source.cleanup() + } + }, +) diff --git a/packages/db/tests/replay-publication-storage.test.ts b/packages/db/tests/replay-publication-storage.test.ts new file mode 100644 index 0000000000..3be47ce29a --- /dev/null +++ b/packages/db/tests/replay-publication-storage.test.ts @@ -0,0 +1,335 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection' +import { createDeferred } from '../src/deferred' +import { BasicIndex } from '../src/indexes/basic-index' +import { createLiveQueryCollection, eq } from '../src/query' +import { PropRef } from '../src/query/ir' +import { evaluateReferenceExpression } from './reference-expression' +import { flushPromises } from './utils' +import type { Deferred } from '../src/deferred' +import type { ChangeMessage, LoadSubsetOptions, SyncConfig } from '../src/types' + +type Row = { id: number; version: number } +type Ops = Parameters[`sync`]>[0] +type Batch = Array<[string, string | number, number]> + +const idRef = () => new PropRef([`id`]) +const shape = (changes: Array>): Batch => + changes.map((c) => [c.type, c.key, c.value.version]) + +/** Retention witness: private replacement rows held per subscription. */ +function replaySessions(collection: unknown) { + const internals = collection as { + _changes: { + changeSubscriptions: Iterable<{ + options: { truncateReplayPublication?: unknown } + truncateReplaySession?: { privateRows?: ReadonlyMap } + }> + } + } + return [...internals._changes.changeSubscriptions].flatMap((s) => + s.truncateReplaySession + ? [ + { + delegated: Boolean(s.options.truncateReplayPublication), + privateRows: s.truncateReplaySession.privateRows?.size ?? null, + }, + ] + : [], + ) +} + +function makeSource(id: string) { + let version = 1 + let hold: Deferred | undefined + let sync!: Ops + const loads: Array = [] + const source = createCollection({ + id, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + const ids = [1, 2, 3].filter( + (rowId) => + !options.where || + evaluateReferenceExpression(options.where, { + id: rowId, + version, + }), + ) + operations.begin() + for (const rowId of ids) { + operations.write({ + type: source.has(rowId) ? `update` : `insert`, + value: { id: rowId, version }, + }) + } + operations.commit() + return hold ? hold.promise : true + }, + unloadSubset: () => {}, + } + }, + }, + }) + return { + source, + loads, + get sync() { + return sync + }, + setVersion: (next: number) => { + version = next + }, + setHold: (next: Deferred | undefined) => { + hold = next + }, + truncate: () => { + sync.begin() + sync.truncate() + sync.commit() + }, + } +} + +describe(`Replay publication storage`, () => { + it(`direct: one replacement batch, healthy peers, late demand joins the barrier`, async () => { + const s = makeSource(`probe-direct`) + const batches: Array = [] + const sub = s.source.subscribeChanges( + (changes) => changes.length && batches.push(shape(changes)), + { includeInitialState: false }, + ) + sub.requestSnapshot({ where: eq(idRef(), 1), optimizedOnly: false }) + sub.requestSnapshot({ where: eq(idRef(), 2), optimizedOnly: false }) + // A demand-free peer sees every source delta immediately. + const peerBatches: Array = [] + const peer = s.source.subscribeChanges( + (changes) => changes.length && peerBatches.push(shape(changes)), + { includeInitialState: false }, + ) + // A query peer over the same source uses the delegated publication path. + const peerLive = createLiveQueryCollection((q) => + q.from({ row: s.source }).where(({ row }) => eq(row.id, 2)), + ) + await peerLive.preload() + expect(batches.flat()).toEqual([ + [`insert`, 1, 1], + [`insert`, 2, 1], + ]) + expect(peerLive.get(2)?.version).toBe(1) + batches.length = 0 + peerBatches.length = 0 + + s.setVersion(2) + const hold = createDeferred() + s.setHold(hold) + s.truncate() + const completion = sub.pendingTruncateReplacement + expect(completion).toBeInstanceOf(Promise) + await flushPromises() + expect(sub.status).toBe(`loadingSubset`) + // No flash of missing content for the direct subscriber. + expect(batches).toEqual([]) + // The query peer keeps its last complete result behind its own barrier. + expect(peerLive.get(2)?.version).toBe(1) + // The demand-free peer saw the truncate deletes and the reloads. + expect(peerBatches.flat().sort()).toEqual( + [ + [`delete`, 1, 1], + [`delete`, 2, 1], + [`insert`, 1, 2], + [`insert`, 2, 2], + ].sort(), + ) + + // Reentrant acquisition while the replay is open joins the barrier. + sub.requestSnapshot({ where: eq(idRef(), 3), optimizedOnly: false }) + expect(batches).toEqual([]) + const retention = replaySessions(s.source) + expect(retention).toContainEqual({ delegated: false, privateRows: 3 }) + expect(retention.filter((r) => r.delegated)).toHaveLength(1) + + hold.resolve() + await flushPromises() + await completion + expect(sub.status).toBe(`ready`) + expect(batches).toHaveLength(1) + expect(batches[0]!.sort()).toEqual([ + [`insert`, 3, 2], + [`update`, 1, 2], + [`update`, 2, 2], + ]) + expect(peerLive.get(2)?.version).toBe(2) + expect(replaySessions(s.source)).toEqual([]) + + // A later plain delta publishes normally. + s.sync.begin() + s.sync.write({ type: `update`, value: { id: 3, version: 5 } }) + s.sync.commit() + expect(batches.at(-1)).toEqual([[`update`, 3, 5]]) + + sub.unsubscribe() + peer.unsubscribe() + await peerLive.cleanup() + await s.source.cleanup() + }) + + it(`direct: releasing the last demand during replay retires it; re-acquisition reconciles`, async () => { + const s = makeSource(`probe-release`) + const visible = new Map() + const batches: Array = [] + const sub = s.source.subscribeChanges( + (changes) => { + changes.length && batches.push(shape(changes)) + for (const c of changes) { + if (c.type === `delete`) visible.delete(c.key) + else visible.set(c.key, c.value.version) + } + }, + { includeInitialState: false }, + ) + const where = eq(idRef(), 1) + sub.requestSnapshot({ where, optimizedOnly: false }) + expect(visible.get(1)).toBe(1) + + s.setVersion(2) + const hold = createDeferred() + s.setHold(hold) + s.truncate() + const settled = Promise.allSettled([sub.pendingTruncateReplacement]) + await flushPromises() + expect(sub.status).toBe(`loadingSubset`) + expect(s.source.get(1)?.version).toBe(2) + expect(visible.get(1)).toBe(1) + + sub.releaseSnapshot(where) + const [outcome] = await settled + expect(outcome.status).toBe(`rejected`) + expect((outcome as PromiseRejectedResult).reason.name).toBe(`AbortError`) + expect(sub.status).toBe(`ready`) + expect(sub.hasPendingTruncateReplacement).toBe(false) + expect(visible.get(1)).toBe(1) + expect(replaySessions(s.source)).toEqual([]) + + // Late settlement of the released transport changes nothing. + hold.resolve() + await flushPromises() + expect(visible.get(1)).toBe(1) + expect(sub.status).toBe(`ready`) + + // Re-acquiring reconciles the retained row against the source. + s.setHold(undefined) + s.setVersion(3) + sub.requestSnapshot({ where, optimizedOnly: false }) + await flushPromises() + expect(visible.get(1)).toBe(3) + expect(batches.at(-1)).toEqual([[`update`, 1, 3]]) + expect(sub.status).toBe(`ready`) + + sub.unsubscribe() + await s.source.cleanup() + }) + + it(`direct: on-demand restart reacquires demand behind one private batch`, async () => { + let loadCount = 0 + let ops!: Ops + const batches: Array = [] + const source = createCollection({ + id: `probe-restart-on-demand`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + ops = operations + operations.markReady() + return { + loadSubset: () => { + loadCount++ + ops.begin() + ops.write({ + type: `insert`, + value: { id: 1, version: loadCount }, + }) + ops.commit() + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const sub = source.subscribeChanges( + (changes) => changes.length && batches.push(shape(changes)), + { includeInitialState: false }, + ) + sub.requestSnapshot() + expect(batches).toEqual([[[`insert`, 1, 1]]]) + + await source.cleanup() + source.startSyncImmediate() + expect(sub.status).toBe(`loadingSubset`) + await flushPromises() + expect(loadCount).toBe(2) + expect(sub.status).toBe(`ready`) + expect(batches).toEqual([[[`insert`, 1, 1]], [[`update`, 1, 2]]]) + + sub.unsubscribe() + await source.cleanup() + }) + + it(`direct: eager restart reconciles retained rows on the next ready batch`, async () => { + let session = 0 + const batches: Array = [] + const source = createCollection({ + id: `probe-restart-eager`, + getKey: (row) => row.id, + sync: { + sync: (operations) => { + session++ + operations.begin() + const rows = + session === 1 + ? [ + { id: 1, version: 1 }, + { id: 2, version: 1 }, + ] + : [{ id: 1, version: 2 }] + for (const value of rows) operations.write({ type: `insert`, value }) + operations.commit() + operations.markReady() + }, + }, + }) + const sub = source.subscribeChanges( + (changes) => changes.length && batches.push(shape(changes)), + { includeInitialState: true }, + ) + expect(batches.flat().sort()).toEqual([ + [`insert`, 1, 1], + [`insert`, 2, 1], + ]) + batches.length = 0 + + await source.cleanup() + source.startSyncImmediate() + await flushPromises() + expect(batches).toHaveLength(1) + expect(batches[0]!.sort()).toEqual([ + [`delete`, 2, 1], + [`update`, 1, 2], + ]) + expect(sub.status).toBe(`ready`) + + sub.unsubscribe() + await source.cleanup() + }) +}) diff --git a/packages/db/tests/trace-runner.test-d.ts b/packages/db/tests/trace-runner.test-d.ts new file mode 100644 index 0000000000..a1e6a07045 --- /dev/null +++ b/packages/db/tests/trace-runner.test-d.ts @@ -0,0 +1,12 @@ +import type { TraceProjection } from './trace-runner.js' + +const asyncAssertionProjection: TraceProjection = { + observe: () => 0, + recompute: () => 0, + // @ts-expect-error trace assertions must finish synchronously + assertEqual: async () => { + await Promise.resolve() + }, +} + +void asyncAssertionProjection diff --git a/packages/db/tests/trace-runner.test.ts b/packages/db/tests/trace-runner.test.ts new file mode 100644 index 0000000000..ea90cffff4 --- /dev/null +++ b/packages/db/tests/trace-runner.test.ts @@ -0,0 +1,193 @@ +import { describe, expect, it, vi } from 'vitest' +import { runTrace } from './trace-runner.js' +import type { TraceDriver } from './trace-runner.js' + +type Context = { + observed: number + expected: number +} + +describe(`runTrace`, () => { + it(`checks after startup, explicit checkpoints, and every step`, async () => { + const checkpoints: Array = [] + const cleanup = vi.fn() + const driver: TraceDriver = { + setup: () => ({ observed: 0, expected: 0 }), + start: (context) => { + context.observed = 1 + context.expected = 1 + }, + apply: (step, context, checkpoint) => { + context.observed += step + context.expected += step + if (step === 2) checkpoint() + }, + cleanup, + } + + await runTrace({ + steps: [2, 3], + driver, + projection: { + observe: (context) => context.observed, + recompute: (context) => context.expected, + assertEqual: (observed, expected) => { + expect(observed).toBe(expected) + checkpoints.push(observed) + }, + }, + }) + + expect(checkpoints).toEqual([1, 3, 3, 6]) + expect(cleanup).toHaveBeenCalledOnce() + }) + + it(`cleans up when a checkpoint fails`, async () => { + const cleanup = vi.fn() + + await expect( + runTrace({ + steps: [1], + driver: { + setup: () => ({ observed: 0, expected: 1 }), + apply: () => undefined, + cleanup, + }, + projection: { + observe: (context) => context.observed, + recompute: (context) => context.expected, + assertEqual: (observed, expected) => { + expect(observed).toBe(expected) + }, + }, + }), + ).rejects.toThrow() + + expect(cleanup).toHaveBeenCalledOnce() + }) + + it(`checks synchronous steps before queued microtasks run`, async () => { + await expect( + runTrace({ + steps: [1], + driver: { + setup: () => ({ observed: 0, expected: 0 }), + apply: (step, context) => { + context.expected = step + queueMicrotask(() => { + context.observed = step + }) + }, + cleanup: () => undefined, + }, + projection: { + observe: (context) => context.observed, + recompute: (context) => context.expected, + assertEqual: (observed, expected) => { + expect(observed).toBe(expected) + }, + }, + }), + ).rejects.toThrow() + }) + + it(`preserves the trace failure when cleanup also fails`, async () => { + const cleanupError = new Error(`cleanup failed`) + + const run = runTrace({ + steps: [], + driver: { + setup: () => ({ observed: 0, expected: 1 }), + apply: () => undefined, + cleanup: () => { + throw cleanupError + }, + }, + projection: { + observe: (context) => context.observed, + recompute: (context) => context.expected, + assertEqual: (observed, expected) => { + expect(observed).toBe(expected) + }, + }, + }) + + const traceFailure = await run.catch((error: unknown) => error) + expect(traceFailure).toMatchObject({ + name: `TraceAssertionError`, + checkpoint: 0, + cause: { name: `AssertionError` }, + }) + expect( + (traceFailure as Error & { suppressed?: Array }).suppressed, + ).toEqual([cleanupError]) + }) + + it(`does not wrap observation errors as assertion failures`, async () => { + const observationError = new Error(`observation failed`) + + await expect( + runTrace({ + steps: [], + driver: { + setup: () => undefined, + apply: () => undefined, + cleanup: () => undefined, + }, + projection: { + observe: () => { + throw observationError + }, + recompute: () => undefined, + assertEqual: () => undefined, + }, + }), + ).rejects.toBe(observationError) + }) + + it(`does not wrap runtime errors from assertion callbacks`, async () => { + const runtimeError = new TypeError(`assertion callback failed`) + + await expect( + runTrace({ + steps: [], + driver: { + setup: () => undefined, + apply: () => undefined, + cleanup: () => undefined, + }, + projection: { + observe: () => undefined, + recompute: () => undefined, + assertEqual: () => { + throw runtimeError + }, + }, + }), + ).rejects.toBe(runtimeError) + }) + + it(`throws cleanup failures when the trace succeeds`, async () => { + const cleanupError = new Error(`cleanup failed`) + + await expect( + runTrace({ + steps: [], + driver: { + setup: () => ({ observed: 0, expected: 0 }), + apply: () => undefined, + cleanup: () => { + throw cleanupError + }, + }, + projection: { + observe: (context) => context.observed, + recompute: (context) => context.expected, + assertEqual: (observed, expected) => { + expect(observed).toBe(expected) + }, + }, + }), + ).rejects.toBe(cleanupError) + }) +}) diff --git a/packages/db/tests/trace-runner.ts b/packages/db/tests/trace-runner.ts new file mode 100644 index 0000000000..3c8868d525 --- /dev/null +++ b/packages/db/tests/trace-runner.ts @@ -0,0 +1,120 @@ +type MaybePromise = T | PromiseLike + +type ErrorWithSuppressed = Error & { + suppressed?: Array +} + +export class TraceAssertionError extends Error { + readonly checkpoint: number + + constructor(checkpoint: number, cause: unknown) { + super(`Trace assertion failed at checkpoint ${checkpoint}`, { cause }) + this.name = `TraceAssertionError` + this.checkpoint = checkpoint + } +} + +export type TraceCheckpoint = () => undefined + +export type TraceDriver = { + setup: () => MaybePromise + start?: (context: TContext) => MaybePromise + apply: ( + step: TStep, + context: TContext, + checkpoint: TraceCheckpoint, + ) => MaybePromise + cleanup: (context: TContext) => MaybePromise +} + +export type TraceProjection = { + observe: (context: TContext) => TObserved + recompute: (context: TContext) => TExpected + assertEqual: (observed: TObserved, expected: TExpected) => undefined +} + +type RunTraceOptions = { + steps: ReadonlyArray + driver: TraceDriver + projection: TraceProjection +} + +function isPromiseLike(value: MaybePromise): value is PromiseLike { + return ( + value !== null && + (typeof value === `object` || typeof value === `function`) && + `then` in value && + typeof value.then === `function` + ) +} + +function attachSuppressedError(error: unknown, suppressed: unknown): void { + if (!(error instanceof Error)) return + + try { + const errorWithSuppressed = error as ErrorWithSuppressed + errorWithSuppressed.suppressed = [ + ...(errorWithSuppressed.suppressed ?? []), + suppressed, + ] + } catch { + // A frozen or otherwise immutable error must still remain the primary one. + } +} + +/** + * Drives a trace against a system and checks its observable state against an + * independent projection after startup, after each step, and at any explicit + * checkpoint requested by the driver. + */ +export async function runTrace({ + steps, + driver, + projection, +}: RunTraceOptions): Promise { + const setupResult = driver.setup() + const context = isPromiseLike(setupResult) ? await setupResult : setupResult + let checkpointIndex = 0 + const checkpoint: TraceCheckpoint = () => { + const currentCheckpoint = checkpointIndex + checkpointIndex += 1 + const observed = projection.observe(context) + const expected = projection.recompute(context) + try { + projection.assertEqual(observed, expected) + } catch (error) { + if (!(error instanceof Error) || error.name !== `AssertionError`) { + throw error + } + throw new TraceAssertionError(currentCheckpoint, error) + } + return undefined + } + + let traceFailed = false + let traceError: unknown + try { + const startResult = driver.start?.(context) + if (isPromiseLike(startResult)) await startResult + checkpoint() + + for (const step of steps) { + const applyResult = driver.apply(step, context, checkpoint) + if (isPromiseLike(applyResult)) await applyResult + checkpoint() + } + } catch (error) { + traceFailed = true + traceError = error + } + + try { + const cleanupResult = driver.cleanup(context) + if (isPromiseLike(cleanupResult)) await cleanupResult + } catch (cleanupError) { + if (!traceFailed) throw cleanupError + attachSuppressedError(traceError, cleanupError) + } + + if (traceFailed) throw traceError +} diff --git a/packages/db/tests/transactions.test.ts b/packages/db/tests/transactions.test.ts index d9f27667d7..33373d4ade 100644 --- a/packages/db/tests/transactions.test.ts +++ b/packages/db/tests/transactions.test.ts @@ -1,14 +1,171 @@ import { describe, expect, it } from 'vitest' +import { DbClient, collectionOptions } from '../src/client.js' import { createTransaction } from '../src/transactions' import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' import { MissingMutationFunctionError, TransactionAlreadyCompletedRollbackError, TransactionNotPendingCommitError, TransactionNotPendingMutateError, } from '../src/errors' +import { flushPromises } from './utils.js' +import type { SyncConfig } from '../src/types.js' describe(`Transactions`, () => { + it(`settles persistence and reports a listener error while draining its parked echo`, async () => { + type Row = { id: number; value: string } + let sync!: Parameters[`sync`]>[0] + const gate = createDeferred() + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + sync: (operations) => { + sync = operations + sync.markReady() + }, + }, + }) + const tx = createTransaction({ + autoCommit: false, + mutationFn: () => gate.promise, + }) + const failure = new Error(`echo listener failed`) + const subscription = collection.subscribeChanges((batch) => { + if (batch.some((change) => change.value.value === `server`)) throw failure + }) + let persisted = false + const receipt = tx.isPersisted.promise.then( + () => { + persisted = true + }, + () => {}, + ) + try { + tx.mutate(() => collection.insert({ id: 1, value: `client` })) + const outcome = tx.commit().then( + () => ({ ok: true as const }), + (error: unknown) => ({ ok: false as const, error }), + ) + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: `server` } }) + const echo = sync.commit() + if (echo !== true) void echo.catch(() => {}) + gate.resolve() + const result = await outcome + await flushPromises() + expect.soft(result).toEqual({ ok: false, error: failure }) + expect.soft(tx.state).toBe(`completed`) + expect(persisted).toBe(true) + await receipt + } finally { + subscription.unsubscribe() + gate.resolve() + await collection.cleanup() + } + }) + it.each([ + { + name: `Error`, + reason: new Error(`mutation failed`), + message: `mutation failed`, + }, + { name: `string`, reason: `mutation failed`, message: `mutation failed` }, + { + name: `unprintable object`, + reason: { + toString() { + throw new Error(`cannot stringify`) + }, + }, + message: `Unknown error`, + }, + ])( + `rolls back a mutation rejected with an $name`, + async ({ reason, message }) => { + const collection = createCollection<{ id: number }>({ + getKey: (row) => row.id, + sync: { sync: () => {} }, + }) + const transaction = createTransaction({ + autoCommit: false, + mutationFn: () => Promise.reject(reason), + }) + const persisted = transaction.isPersisted.promise.catch( + (error: unknown) => error, + ) + try { + transaction.mutate(() => collection.insert({ id: 1 })) + await expect(transaction.commit()).rejects.toThrow(message) + expect(transaction.state).toBe(`failed`) + expect(collection.has(1)).toBe(false) + expect(await persisted).toBe(transaction.error?.error) + if (reason instanceof Error) + expect(transaction.error?.error).toBe(reason) + } finally { + if (transaction.state !== `failed`) transaction.rollback() + await collection.cleanup() + } + }, + ) + + it(`keeps a claimed default transaction ambient for later plain collection mutations`, () => { + const client = new DbClient() + const clientCollection = client.collection( + collectionOptions(`claimed-client-collection`, () => ({ + id: `claimed-client-collection`, + getKey: (row: { id: number }) => row.id, + sync: { sync: () => {} }, + })), + ) + const plainCollection = createCollection<{ id: number }>({ + id: `claimed-plain-collection`, + getKey: (row) => row.id, + sync: { sync: () => {} }, + }) + const transaction = createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + + transaction.mutate(() => clientCollection.insert({ id: 1 })) + transaction.mutate(() => plainCollection.insert({ id: 2 })) + + expect(transaction.mutations).toHaveLength(2) + }) + + it(`does not cascade rollbacks across isolated client and default scopes`, () => { + const options = { + id: `isolated-rollback-scope`, + getKey: (row: { id: number }) => row.id, + sync: { sync: () => {} }, + } + const plainCollection = createCollection(options) + const client = new DbClient() + const scopedCollection = client.collection( + collectionOptions(`isolated-rollback-scope`, () => ({ + ...options, + id: `isolated-rollback-scope`, + })), + ) + const clientTransaction = client.createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + const defaultTransaction = createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + + clientTransaction.mutate(() => scopedCollection.insert({ id: 1 })) + defaultTransaction.mutate(() => plainCollection.insert({ id: 1 })) + clientTransaction.rollback() + + expect(defaultTransaction.state).toBe(`pending`) + defaultTransaction.rollback() + }) + it(`calling createTransaction creates a transaction`, () => { const transaction = createTransaction({ mutationFn: async () => Promise.resolve(), @@ -159,6 +316,173 @@ describe(`Transactions`, () => { transaction.isPersisted.promise.catch(() => {}) expect(transaction.state).toBe(`failed`) }) + it(`keeps a persisting transaction failed when rollback wins`, async () => { + let releasePersistence!: () => void + const persistence = new Promise((resolve) => { + releasePersistence = resolve + }) + const collection = createCollection<{ id: number }>({ + id: `persisting-rollback-wins`, + getKey: (item) => item.id, + sync: { sync: () => {} }, + }) + const transaction = createTransaction({ + autoCommit: false, + mutationFn: () => persistence, + }) + + try { + transaction.mutate(() => collection.insert({ id: 1 })) + const persisted = transaction.isPersisted.promise.then( + (value) => ({ status: `fulfilled` as const, value }), + (reason: unknown) => ({ status: `rejected` as const, reason }), + ) + const commit = transaction.commit() + expect(transaction.state).toBe(`persisting`) + + transaction.rollback() + expect(transaction.state).toBe(`failed`) + + releasePersistence() + await expect(commit).resolves.toBe(transaction) + expect(await persisted).toEqual({ + status: `rejected`, + reason: undefined, + }) + expect(transaction.state).toBe(`failed`) + expect(transaction.error).toBeUndefined() + } finally { + releasePersistence() + await collection.cleanup() + } + }) + it.each([ + [`Error`, (): unknown => new Error(`late persistence rejection`)], + [`undefined`, (): unknown => undefined], + [`false`, (): unknown => false], + [`zero`, (): unknown => 0], + [`NaN`, (): unknown => Number.NaN], + [`string`, (): unknown => `late persistence rejection`], + [`object`, (): unknown => ({ late: true })], + ] as const)( + `ignores a late %s persistence rejection after rollback wins`, + async (reasonName, createReason) => { + type Row = { id: number; owner: string } + let rejectPersistence!: (reason: unknown) => void + const persistence = new Promise((_resolve, reject) => { + rejectPersistence = reject + }) + const collection = createCollection({ + id: `late-persistence-rejection-${reasonName}`, + getKey: (item) => item.id, + sync: { sync: () => {} }, + }) + const batches: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => { + batches.push( + changes.map(({ type, key }) => ({ + type, + key, + })), + ) + }, + { includeInitialState: false }, + ) + const first = createTransaction({ + autoCommit: false, + mutationFn: () => persistence, + }) + const second = createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + + try { + const persisted = first.isPersisted.promise.then( + (value) => ({ status: `fulfilled` as const, value }), + (reason: unknown) => ({ status: `rejected` as const, reason }), + ) + first.mutate(() => collection.insert({ id: 1, owner: `first` })) + const commit = first.commit().then( + (value) => ({ status: `fulfilled` as const, value }), + (reason: unknown) => ({ status: `rejected` as const, reason }), + ) + + first.rollback() + second.mutate(() => collection.insert({ id: 1, owner: `second` })) + rejectPersistence(createReason()) + + const commitOutcome = await commit + expect(commitOutcome.status).toBe(`fulfilled`) + if (commitOutcome.status === `fulfilled`) { + expect(commitOutcome.value).toBe(first) + } + expect(await persisted).toEqual({ + status: `rejected`, + reason: undefined, + }) + expect(first.state).toBe(`failed`) + expect(first.error).toBeUndefined() + expect(second.state).toBe(`pending`) + expect(collection.get(1)).toEqual({ + id: 1, + owner: `second`, + $collectionId: collection.id, + $key: 1, + $origin: `local`, + $synced: false, + }) + expect(batches).toEqual([ + [{ type: `insert`, key: 1 }], + [{ type: `delete`, key: 1 }], + [{ type: `insert`, key: 1 }], + ]) + } finally { + rejectPersistence(new Error(`test cleanup`)) + if (second.state === `pending`) { + second.rollback({ isSecondaryRollback: true }) + } + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + it(`keeps repeated rollback from affecting newer transactions`, async () => { + type Row = { id: number; owner: string } + const collection = createCollection({ + id: `repeated-rollback-is-terminal`, + getKey: (item) => item.id, + sync: { sync: () => {} }, + }) + const first = createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + const second = createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + + try { + void first.isPersisted.promise.catch(() => undefined) + first.mutate(() => collection.insert({ id: 1, owner: `first` })) + first.rollback() + + second.mutate(() => collection.insert({ id: 1, owner: `second` })) + expect(second.state).toBe(`pending`) + + expect(first.rollback()).toBe(first) + expect(first.state).toBe(`failed`) + expect(second.state).toBe(`pending`) + expect(collection.get(1)).toMatchObject({ id: 1, owner: `second` }) + } finally { + if (second.state === `pending`) { + second.rollback({ isSecondaryRollback: true }) + } + await collection.cleanup() + } + }) it(`should rollback if the mutationFn throws an error`, async () => { const transaction = createTransaction({ mutationFn: async () => { @@ -506,6 +830,10 @@ describe(`Transactions`, () => { mutationFn: async () => Promise.resolve(), autoCommit: false, }) + const transaction4 = createTransaction({ + mutationFn: async () => Promise.resolve(), + autoCommit: false, + }) const collection = createCollection<{ id: number value: string @@ -545,13 +873,23 @@ describe(`Transactions`, () => { }) }) + transaction4.mutate(() => { + collection.state.forEach((object) => { + collection.update(object.id, (draft) => { + draft.value = `foo-me-4` + }) + }) + }) + transaction1.rollback() transaction1.isPersisted.promise.catch(() => {}) transaction3.isPersisted.promise.catch(() => {}) + transaction4.isPersisted.promise.catch(() => {}) expect(transaction1.state).toBe(`failed`) expect(transaction2.state).toBe(`completed`) expect(transaction3.state).toBe(`failed`) + expect(transaction4.state).toBe(`failed`) }) describe(`duplicate instance detection`, () => { diff --git a/packages/db/tests/unsubscribed-sync-gc.test.ts b/packages/db/tests/unsubscribed-sync-gc.test.ts new file mode 100644 index 0000000000..fd93db412f --- /dev/null +++ b/packages/db/tests/unsubscribed-sync-gc.test.ts @@ -0,0 +1,107 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createLiveQueryCollection } from '../src/query/live-query-collection.js' +import { mockSyncCollectionOptions, resetCleanupQueue } from './utils.js' + +type Person = { id: string; name: string } + +const collections: Array<{ cleanup: () => Promise }> = [] + +const makeLiveQuery = (gcTime = 1, startSync = true) => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `unsubscribed-gc-source`, + getKey: (person) => person.id, + initialData: [{ id: `1`, name: `Alice` }], + }), + ) + const live = createLiveQueryCollection({ + startSync, + gcTime, + query: (q) => q.from({ person: source }), + }) + collections.push(source, live) + return { source, live } +} + +describe(`collections that start syncing without a subscriber`, () => { + beforeEach(() => { + vi.useFakeTimers() + resetCleanupQueue() + }) + + afterEach(async () => { + for (const collection of collections.reverse()) await collection.cleanup() + collections.length = 0 + await Promise.resolve() + resetCleanupQueue() + vi.useRealTimers() + }) + + it(`keeps the initial grace period but uses gcTime after the last subscriber leaves`, async () => { + const { source, live } = makeLiveQuery() + + await vi.advanceTimersByTimeAsync(49) + expect(live.status).toBe(`ready`) + expect(source.subscriberCount).toBe(1) + + const subscription = live.subscribeChanges(() => {}) + await vi.advanceTimersByTimeAsync(100) + expect(live.status).toBe(`ready`) + expect(live.size).toBe(1) + + subscription.unsubscribe() + await vi.advanceTimersByTimeAsync(2) + expect(live.status).toBe(`cleaned-up`) + expect(source.subscriberCount).toBe(0) + }) + + it(`honors a gcTime longer than the initial grace period`, async () => { + const { source, live } = makeLiveQuery(100) + + await vi.advanceTimersByTimeAsync(99) + expect(live.status).toBe(`ready`) + await vi.advanceTimersByTimeAsync(2) + expect(live.status).toBe(`cleaned-up`) + expect(source.subscriberCount).toBe(0) + }) + + it(`restarts sync when a subscriber attaches after reclamation`, async () => { + const { source, live } = makeLiveQuery() + + await vi.advanceTimersByTimeAsync(51) + expect(live.status).toBe(`cleaned-up`) + expect(source.subscriberCount).toBe(0) + + const subscription = live.subscribeChanges(() => {}) + expect(live.status).toBe(`ready`) + expect(live.size).toBe(1) + expect(source.subscriberCount).toBe(1) + subscription.unsubscribe() + }) + + it.each([0, -1, Infinity, -Infinity, NaN])( + `disables automatic GC for gcTime %s`, + async (gcTime) => { + const { source, live } = makeLiveQuery(gcTime) + + await vi.advanceTimersByTimeAsync(300001) + expect(live.status).toBe(`ready`) + expect(source.subscriberCount).toBe(1) + }, + ) + + it.each([`preload`, `startSyncImmediate`] as const)( + `reclaims unused collections started by %s`, + async (method) => { + const { source, live } = makeLiveQuery(1, false) + expect(source.subscriberCount).toBe(0) + + await live[method]() + expect(source.subscriberCount).toBe(1) + await vi.advanceTimersByTimeAsync(51) + expect(live.status).toBe(`cleaned-up`) + expect(source.subscriberCount).toBe(0) + }, + ) +}) diff --git a/packages/db/tests/utils.test.ts b/packages/db/tests/utils.test.ts index d6bb5e368b..15520a0ea2 100644 --- a/packages/db/tests/utils.test.ts +++ b/packages/db/tests/utils.test.ts @@ -1,9 +1,191 @@ import { describe, expect, it } from 'vitest' import { Temporal } from 'temporal-polyfill' import { deepEquals } from '../src/utils' +import { normalizeError } from '../src/utils/error' import { isPromiseLike } from '../src/utils/type-guards' +import { + oracleRandomParameters, + readOracleRunConfig, + validateOraclePropertyRegistry, +} from './oracle-config' + +describe(`normalizeError`, () => { + it(`normalizes unstringifiable thrown values`, () => { + const revoked = Proxy.revocable({}, {}) + revoked.revoke() + const thrownValues = [ + Object.create(null), + { + [Symbol.toPrimitive]: () => { + throw new Error(`conversion failed`) + }, + }, + new Proxy( + {}, + { + getPrototypeOf: () => { + throw new Error(`prototype lookup failed`) + }, + }, + ), + revoked.proxy, + ] + + for (const thrownValue of thrownValues) { + expect(() => normalizeError(thrownValue)).not.toThrow() + expect(normalizeError(thrownValue)).toEqual(new Error(`Unknown error`)) + } + }) +}) + +describe(`oracle run configuration`, () => { + it(`reads the multiplier and replay coordinates from an explicit environment`, () => { + expect( + readOracleRunConfig({ + TANSTACK_DB_ORACLE_RUNS_MULTIPLIER: `100`, + TANSTACK_DB_ORACLE_SEED: `-42`, + TANSTACK_DB_ORACLE_PATH: `1:0:2`, + TANSTACK_DB_ORACLE_PROPERTY: `includes.incremental-history`, + }), + ).toEqual({ + multiplier: 100, + replaySeed: -42, + replayPath: `1:0:2`, + replayProperty: `includes.incremental-history`, + }) + }) + + it(`uses one run multiplier and no replay coordinates by default`, () => { + expect(readOracleRunConfig({})).toEqual({ + multiplier: 1, + replaySeed: undefined, + replayPath: undefined, + replayProperty: undefined, + }) + }) + + it.each([ + [{ TANSTACK_DB_ORACLE_RUNS_MULTIPLIER: `0` }, `positive integer`], + [{ TANSTACK_DB_ORACLE_RUNS_MULTIPLIER: `1.5` }, `positive integer`], + [{ TANSTACK_DB_ORACLE_RUNS_MULTIPLIER: ` ` }, `positive integer`], + [{ TANSTACK_DB_ORACLE_SEED: `1.5` }, `must be an integer`], + [{ TANSTACK_DB_ORACLE_SEED: ` ` }, `must be an integer`], + [{ TANSTACK_DB_ORACLE_PATH: `1:0` }, `requires TANSTACK_DB_ORACLE_SEED`], + [ + { TANSTACK_DB_ORACLE_PROPERTY: `includes.incremental-history` }, + `requires TANSTACK_DB_ORACLE_PATH`, + ], + [ + { + TANSTACK_DB_ORACLE_SEED: `42`, + TANSTACK_DB_ORACLE_PATH: ` `, + TANSTACK_DB_ORACLE_PROPERTY: `includes.incremental-history`, + }, + `must be non-empty`, + ], + [ + { + TANSTACK_DB_ORACLE_SEED: `42`, + TANSTACK_DB_ORACLE_PATH: `1:-1`, + TANSTACK_DB_ORACLE_PROPERTY: `includes.incremental-history`, + }, + `colon-separated nonnegative integers`, + ], + [ + { + TANSTACK_DB_ORACLE_SEED: `42`, + TANSTACK_DB_ORACLE_PATH: `1:0`, + }, + `requires TANSTACK_DB_ORACLE_PROPERTY`, + ], + [ + { + TANSTACK_DB_ORACLE_SEED: `42`, + TANSTACK_DB_ORACLE_PATH: `1:0`, + TANSTACK_DB_ORACLE_PROPERTY: `includes.typo`, + }, + `unknown oracle property`, + ], + [ + { + TANSTACK_DB_ORACLE_SEED: `42`, + TANSTACK_DB_ORACLE_PROPERTY: `includes.incremental-history`, + }, + `requires TANSTACK_DB_ORACLE_PATH`, + ], + ] satisfies ReadonlyArray, string]>)( + `rejects invalid environment values`, + (environment, message) => { + expect(() => readOracleRunConfig(environment)).toThrow(message) + }, + ) + + it(`rejects duplicate registered property names`, () => { + expect(() => + validateOraclePropertyRegistry([`one.property`, `one.property`]), + ).toThrow(`duplicate oracle property`) + }) + + it(`adds a shrink path only to its named property`, () => { + const ordinaryRun = { + replaySeed: undefined, + replayPath: undefined, + replayProperty: undefined, + } + const replayRun = { + replaySeed: -42, + replayPath: `1:0:2`, + replayProperty: `includes.incremental-history`, + } + + expect( + oracleRandomParameters(40, ordinaryRun, `includes.incremental-history`), + ).toEqual({ numRuns: 40 }) + expect( + oracleRandomParameters(40, replayRun, `includes.alpha-renaming`), + ).toEqual({ + numRuns: 40, + seed: -42, + }) + expect( + oracleRandomParameters(40, replayRun, `includes.incremental-history`), + ).toEqual({ numRuns: 40, seed: -42, path: `1:0:2` }) + }) +}) describe(`deepEquals`, () => { + it.each([`later`, Symbol(`later`)])( + `checks own-key visibility after a getter runs: %s`, + (key) => { + const right = { first: 1, [key]: undefined } + const left = { + get first() { + Object.defineProperty(right, key, { enumerable: false }) + return 1 + }, + [key]: undefined, + } + expect(deepEquals(left, right)).toBe(false) + }, + ) + + it.each( + [`field`, Symbol(`field`)].flatMap((key) => + [false, true].map((inherited) => ({ key, inherited })), + ), + )( + `requires matching enumerable own keys: $key / inherited=$inherited`, + ({ key, inherited }) => { + const own = { [key]: 1 } + const other = { other: 1 } + if (inherited) Object.setPrototypeOf(other, { [key]: 1 }) + else Object.defineProperty(other, key, { value: 1, enumerable: false }) + expect(deepEquals(own, other)).toBe(false) + expect(deepEquals(other, own)).toBe(false) + expect(deepEquals(own, { [key]: 1 })).toBe(true) + }, + ) + describe(`primitives`, () => { it(`should handle identical primitives`, () => { expect(deepEquals(1, 1)).toBe(true) @@ -64,6 +246,16 @@ describe(`deepEquals`, () => { expect(deepEquals({ a: { b: 1 } }, { a: { b: 2 } })).toBe(false) }) + it(`should compare enumerable symbol properties`, () => { + const key = Symbol(`key`) + + expect(deepEquals({ [key]: 1 }, { [key]: 1 })).toBe(true) + expect(deepEquals({ [key]: 1 }, { [key]: 2 })).toBe(false) + expect(deepEquals({ [Symbol(`key`)]: 1 }, { [Symbol(`key`)]: 1 })).toBe( + false, + ) + }) + it(`should handle circular references in objects`, () => { const a: any = { x: 1 } a.self = a diff --git a/packages/db/tests/utils.ts b/packages/db/tests/utils.ts index 41fc65a9d8..759173a638 100644 --- a/packages/db/tests/utils.ts +++ b/packages/db/tests/utils.ts @@ -1,5 +1,9 @@ import { expect } from 'vitest' +import { createCollection } from '../src/collection/index.js' import { BTreeIndex } from '../src/indexes/btree-index' +import { withCollectionConfigFactory } from '../src/client' +import { denormalizeUndefined } from '../src/utils/comparison.js' +import { CleanupQueue } from '../src/collection/cleanup-queue.js' import type { CollectionConfig, MutationFnParams, @@ -14,6 +18,17 @@ export type OutputWithVirtual< TKey extends string | number = string | number, > = WithVirtualProps +// Keep sync startup, writes, readiness, and load outcomes in the test itself. +export function createOnDemandCollection( + config: Omit, `getKey` | `syncMode`>, +) { + return createCollection({ + ...config, + getKey: ({ id }) => id, + syncMode: `on-demand`, + }) +} + export const stripVirtualProps = | undefined>( value: T, ) => { @@ -219,9 +234,21 @@ type MockSyncCollectionConfig> = { defaultIndexType?: IndexConstructor } +type MockSyncCollectionUtils = { + begin: () => void + write: Parameters[`sync`]>[0][`write`] + commit: () => void + resolveSync: () => void + rejectSync: (error: Error) => void +} + export function mockSyncCollectionOptions< T extends object = Record, ->(config: MockSyncCollectionConfig) { +>( + config: MockSyncCollectionConfig, +): CollectionConfig & { + utils: MockSyncCollectionUtils +} { let begin: () => void let write: Parameters[`sync`]>[0][`write`] let commit: () => void @@ -238,14 +265,16 @@ export function mockSyncCollectionOptions< syncPendingResolve = resolve syncPendingReject = reject }) - syncPendingPromise.then(() => { - syncPendingPromise = undefined - syncPendingResolve = undefined - syncPendingReject = undefined - }) + void syncPendingPromise.finally(clearPendingSync) return syncPendingPromise } + const clearPendingSync = () => { + syncPendingPromise = undefined + syncPendingResolve = undefined + syncPendingReject = undefined + } + const utils = { begin: () => begin!(), write: ((value) => write!(value)) as typeof write, @@ -304,7 +333,9 @@ export function mockSyncCollectionOptions< (config.autoIndex === `eager` ? BTreeIndex : undefined), } - return options + return withCollectionConfigFactory(options, () => + mockSyncCollectionOptions(config), + ) } type MockSyncCollectionConfigNoInitialState = { @@ -336,14 +367,16 @@ export function mockSyncCollectionOptionsNoInitialState< syncPendingResolve = resolve syncPendingReject = reject }) - syncPendingPromise.then(() => { - syncPendingPromise = undefined - syncPendingResolve = undefined - syncPendingReject = undefined - }) + void syncPendingPromise.finally(clearPendingSync) return syncPendingPromise } + const clearPendingSync = () => { + syncPendingPromise = undefined + syncPendingResolve = undefined + syncPendingReject = undefined + } + const utils = { begin: () => begin!(), write: ((value) => write!(value)) as typeof write, @@ -479,3 +512,86 @@ export function withExpectedRejection( }) }) } + +type IndexInternals = { indexedKeys: Set } & ( + | { sortedValues: Array; valueMap: Map> } + | { + valueMap: Map }> + orderedEntries: { + size: number + minKey: () => unknown + maxKey: () => unknown + forRange: ( + low: unknown, + high: unknown, + includeHigh: boolean, + onFound: (key: unknown, bucket: { keys: Set }) => void, + ) => void + } + } +) + +function indexInternals(index: object): [IndexInternals, boolean] { + let reversed = false + let current = index as { originalIndex?: object } + while (current.originalIndex) { + reversed = !reversed + current = current.originalIndex as { originalIndex?: object } + } + return [current as IndexInternals, reversed] +} + +/** Test inspection of an index's tracked keys. */ +export function indexedKeysSet(index: object): Set { + return indexInternals(index)[0].indexedKeys +} + +/** Test inspection of an index's value buckets keyed by indexed value. */ +export function valueMapData(index: object): Map> { + const [internals] = indexInternals(index) + if (`sortedValues` in internals) return internals.valueMap + const result = new Map>() + for (const [key, bucket] of internals.valueMap) { + result.set(denormalizeUndefined(key), bucket.keys) + } + return result +} + +/** Test inspection of an index's ordered [value, keys] entries. */ +export function orderedEntriesArray( + index: object, +): Array<[unknown, Set]> { + const [internals, reversed] = indexInternals(index) + let entries: Array<[unknown, Set]> + if (`sortedValues` in internals) { + entries = internals.sortedValues.map((value) => [ + value, + internals.valueMap.get(value) ?? new Set(), + ]) + } else { + const tree = internals.orderedEntries + entries = [] + if (tree.size > 0) { + tree.forRange(tree.minKey(), tree.maxKey(), true, (key, bucket) => { + entries.push([denormalizeUndefined(key), bucket.keys]) + }) + } + } + return reversed ? entries.reverse() : entries +} + +export function orderedEntriesArrayReversed( + index: object, +): Array<[unknown, Set]> { + return orderedEntriesArray(index).reverse() +} + +/** Reset the CleanupQueue singleton between tests. */ +export function resetCleanupQueue(): void { + const holder = CleanupQueue as unknown as { + instance: { timeoutId: ReturnType | null } | null + } + if (holder.instance?.timeoutId != null) + clearTimeout(holder.instance.timeoutId) + holder.instance = null +} diff --git a/packages/db/tests/utils/collection-helpers.test.ts b/packages/db/tests/utils/collection-helpers.test.ts new file mode 100644 index 0000000000..192b934b8b --- /dev/null +++ b/packages/db/tests/utils/collection-helpers.test.ts @@ -0,0 +1,49 @@ +import { describe, expect, it, vi } from 'vitest' +import { getOrCreate } from '../../src/utils/get-or-create.js' +import { isPlainObject } from '../../src/utils/type-guards.js' + +describe(`getOrCreate`, () => { + it.each([`map`, `weak map`] as const)( + `initializes each %s owner once`, + (kind) => { + const createStore = () => + kind === `map` + ? new Map() + : new WeakMap() + const first = createStore() + const second = createStore() + const key = {} + const create = vi.fn(() => ({})) + const value = getOrCreate(first, key, create) + expect(getOrCreate(first, key, create)).toBe(value) + expect(create).toHaveBeenCalledTimes(1) + expect(getOrCreate(second, key, create)).not.toBe(value) + first.delete(key) + expect(getOrCreate(first, key, create)).not.toBe(value) + expect(create).toHaveBeenCalledTimes(3) + }, + ) + + it.each([false, 0, ``, null])(`retains a defined value %j`, (value) => { + const entries = new Map([[`key`, value]]) + const create = vi.fn(() => value) + expect(getOrCreate(entries, `key`, create)).toBe(value) + expect(create).not.toHaveBeenCalled() + }) +}) + +describe(`isPlainObject`, () => { + it.each([ + { name: `ordinary object`, value: {}, expected: true }, + { name: `null prototype`, value: Object.create(null), expected: true }, + { name: `custom prototype`, value: Object.create({}), expected: false }, + { name: `array`, value: [], expected: false }, + { name: `date`, value: new Date(0), expected: false }, + { name: `null`, value: null, expected: false }, + { name: `undefined`, value: undefined, expected: false }, + { name: `function`, value: () => {}, expected: false }, + { name: `string`, value: `value`, expected: false }, + ])(`classifies $name`, ({ value, expected }) => { + expect(isPlainObject(value)).toBe(expected) + }) +}) diff --git a/packages/db/tests/uuid.test.ts b/packages/db/tests/uuid.test.ts new file mode 100644 index 0000000000..f6cf577e70 --- /dev/null +++ b/packages/db/tests/uuid.test.ts @@ -0,0 +1,76 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { safeRandomUUID } from '../src/utils/uuid' + +const UUID_V4_REGEX = + /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/ + +describe(`safeRandomUUID helper`, () => { + afterEach(() => { + vi.restoreAllMocks() + vi.unstubAllGlobals() + }) + + it(`delegates to crypto.randomUUID when available`, () => { + const spy = vi + .spyOn(globalThis.crypto, `randomUUID`) + .mockReturnValue(`11111111-2222-4333-8444-555555555555`) + const id = safeRandomUUID() + expect(spy).toHaveBeenCalledTimes(1) + expect(id).toBe(`11111111-2222-4333-8444-555555555555`) + }) + + it(`falls back to getRandomValues when crypto.randomUUID is undefined (non-secure context)`, () => { + // Simulate a non-secure browser context where crypto.randomUUID is unavailable + // but getRandomValues remains. + vi.stubGlobal(`crypto`, { + randomUUID: undefined, + getRandomValues: (arr: Uint8Array) => { + // Deterministic-ish fill so we can verify version/variant bits land + // exactly where they should. + for (let i = 0; i < arr.length; i++) arr[i] = 0xff + return arr + }, + }) + + const id = safeRandomUUID() + expect(id).toMatch(UUID_V4_REGEX) + + // Verify version nibble == 4 and variant nibble in [8,9,a,b] + const versionChar = id[14] + const variantChar = id[19] + expect(versionChar).toBe(`4`) + expect([`8`, `9`, `a`, `b`]).toContain(variantChar) + + // With all bytes 0xff, expect ffffffff-ffff-4fff-bfff-ffffffffffff + expect(id).toBe(`ffffffff-ffff-4fff-bfff-ffffffffffff`) + }) + + it(`produces unique, well-formed UUIDs via the fallback path across many calls`, () => { + vi.stubGlobal(`crypto`, { + randomUUID: undefined, + getRandomValues: (arr: Uint8Array) => { + for (let i = 0; i < arr.length; i++) + arr[i] = Math.floor(Math.random() * 256) + return arr + }, + }) + + const seen = new Set() + for (let i = 0; i < 200; i++) { + const id = safeRandomUUID() + expect(id).toMatch(UUID_V4_REGEX) + seen.add(id) + } + expect(seen.size).toBe(200) + }) + + it(`throws when neither crypto.randomUUID nor crypto.getRandomValues is available`, () => { + vi.stubGlobal(`crypto`, {}) + expect(() => safeRandomUUID()).toThrow(/No secure random number generator/) + }) + + it(`throws when globalThis.crypto is undefined`, () => { + vi.stubGlobal(`crypto`, undefined) + expect(() => safeRandomUUID()).toThrow(/No secure random number generator/) + }) +}) diff --git a/packages/electric-db-collection/CHANGELOG.md b/packages/electric-db-collection/CHANGELOG.md index d8e0ee03db..9c2b6c33c8 100644 --- a/packages/electric-db-collection/CHANGELOG.md +++ b/packages/electric-db-collection/CHANGELOG.md @@ -1,5 +1,220 @@ # @tanstack/electric-db-collection +## 0.4.8 + +### Patch Changes + +- Fix on-demand load settlement, ordered pagination, and replay to preserve coherent results across cancellation, failure, cleanup, and restart. Preserve subset results and ownership across Electric, PowerSync, Query, and SQLite persistence adapters. Correct live-query grouping, include projections, value identity, and indexed comparisons. D2 hashing now rejects structural cycles and excessive traversal depth or work with an explicit error; Collection handles retain object-reference identity without traversing their mutable contents. ([#1797](https://github.com/TanStack/db/pull/1797)) + + Remove the unused public subset-algebra helpers: `isWhereSubset`, `unionWherePredicates`, `minusWherePredicates`, `isOrderBySubset`, `isLimitSubset`, `isOffsetLimitSubset`, `isPredicateSubset`, and `isLoadSubsetRequestSubsumedBy`. Apps that import these helpers must remove those imports; normal queries and adapters are unaffected. `DeduplicatedLoadSubset` remains available and shares only exact demand identities. + + Reject compiled Collection-valued includes as `fn.select()` inputs, including nested descendants, before invoking the callback. Use `toArray()` or `materialize()` in the upstream `.select()` for child-value calculations. To keep live child Collections, use expression `.select()` or perform parent-only functional work before adding the includes. Ordinary Collection-valued includes remain supported. + + Remove proxy `DEBUG` logging and automatic index timing statistics to avoid diagnostic work on reads and writes. Remove `getStats()` and `IndexStats`; use `index.keyCount` for the current entry count, and instrument index methods externally when profiling. Custom index subclasses must remove calls to the retired `trackLookup()` and `updateTimestamp()` helpers. + + Fix mutation drafts so Map/Set `forEach` calls each callback once and read-only iteration reports no changes. Track nested Map `for...of` edits through `collection.update()`. Keep Set entries in place during nested edits, preserving live iteration and usable `has`, `delete`, and `add` handles without duplicate entries or repeated visits. Reverting one entry no longer discards another entry's pending changes. + + Preserve shared values within a row's draft. Newly supplied objects use normal shared references during the update callback, including Map values and Set members; editing through either handle changes the same new object. Copy completed changes at callback return so later caller edits cannot mutate stored data. Existing collection values remain isolated. Copy a new caller-owned value before insertion if it must remain untouched during the callback. + + Keep rejected local-storage mutations out of later successful saves. Stage insert, update, delete, and manual transaction writes before persisting, then promote the shared cache only after storage succeeds. + + Deliver live-query observer publications to peer listeners even when one listener throws. Preserve queued publication order and stop delivery on disposal; report the first listener failure after delivery. + + Wait for PowerSync demands admitted during tracking finalization before reporting them loaded. Preserve untouched object and array key identity when they contain `NaN`. Retire every failed ordered acquisition on explicit retry, while retaining a full-source demand repaired by replay. + + After authoritative ordered recovery succeeds, retire settled page and tie demands so later truncate replay fetches only the full-source replacement. Keep unfinished requests observed until settlement and preserve independent query ownership. + + Remove the live-query `utils.getRunCount()` diagnostic and its runtime counter. Apps that call this diagnostic must remove those calls. Query scheduling and results are unchanged. + + Remove test-only index inspection getters (`indexedKeysSet`, `valueMapData`, `orderedEntriesArray`, and `orderedEntriesArrayReversed`) and unused scheduler diagnostics. `ReverseIndex` now exposes only the `IndexReader` lookup, range, and forward traversal surface returned by `findIndexForField`; mutate the original index instead. Export `IndexReader` for callers that name this return type. Remove the unused subscription `releaseLoadSubset()` method; request owners use the release callback supplied by `onLoadSubsetResult`. Ordinary query and adapter APIs remain unchanged by these removals. + + Remove unused internal helpers and the unused public error classes `WhereClauseConversionError`, `SubscriptionNotFoundError`, and `AggregateNotSupportedError`. These classes have no remaining runtime throw sites; remove any imports of them. Keep the existing query and index behavior and exercise identity/evaluation tests through the production entry points. + + Ensure failed mutations roll back even when their rejection value cannot be converted to a string. Preserve ordinary Error instances; report unprintable rejection values as `Unknown error`. + + Make reentrant effect disposal share the active cleanup result, including calls from abort listeners or source release callbacks. A release failure reaches every waiting disposer while each source still receives one release attempt. + + Reject starting or preloading a collection from inside its active cleanup callbacks with a clear `CollectionStateError`. Nested cleanup cannot admit replacement work that the old teardown would discard. Restart after cleanup completes, or from its final `cleaned-up` status event, remains supported. + + Prevent older page or tie-boundary completions from clearing a newer full-source failure or starting redundant loading. Failed window moves retain their settled public snapshot; an explicit retry releases the failed acquisition once and publishes the completed replacement. + + Restrict direct subscription `requestLimitedSnapshot()` cursor inputs to one order term and one `minValues` entry. Composite and partial-composite cursor inputs now throw before local delivery or adapter work. Use normal live-query ordering and window APIs for multi-column pagination; those remain supported through prefix-and-tie loading. Existing adapters need no changes. + + Treat `LoadSubsetOptions` and their nested request data as immutable from submission onward. Core no longer copies expression trees or mutable constant payloads at the sync and deduplication boundaries. Create a new Date, byte array, membership array, or options object when changing a demand instead of mutating submitted data. Adapters must also leave request data unchanged. Use stable data properties rather than stateful getters. `AbortSignal` cancellation and subscription release remain live. + + Replace replay acquisitions sequentially: release the prior physical lease before starting its replacement. The logical demand and last complete public result remain retained. A failed release prevents replacement startup; failed startup leaves demand available for a later authoritative replay. Custom adapters must support a release/load gap and preserve resources still held by other owners; a sole underlying resource may stop and restart. + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 0.4.7 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 0.4.6 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 0.4.5 + +### Patch Changes + +- Settle subset loads only after their committed rows and events are visible. A ([#1769](https://github.com/TanStack/db/pull/1769)) + commit receipt now rejects with `AbortError` when cancellation wins before + application and ignores later aborts. Preserve causal publication, + cancellation, persistence, and error handling across the affected sync + adapters. +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 0.4.4 + +### Patch Changes + +- Report incremental subset-load failures through subscriptions, live-query utilities, and effects while keeping cached source rows available. Recover cleanly from failed or overlapping must-refetch replays, collection cleanup, effect teardown errors, and cooperative adapter cancellation. Electric's shared-stream snapshot path still depends on upstream request identity or cancellation support to prevent rows from an aborted request from arriving before the request Promise settles. ([#1756](https://github.com/TanStack/db/pull/1756)) + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 0.4.3 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 0.4.2 + +### Patch Changes + +- Propagate initial query sync failures through dependent live queries and readiness promises, including recovery and late subscribers, while preserving a ready cached snapshot on later refetch failures. Let sync adapters pass the original failure to `markError(error)` so readiness promises reject with that cause. Isolate adapter callbacks by sync session, preserve synchronous startup errors, and prevent rejected deduplicated subset requests from creating detached promise rejections. ([#1751](https://github.com/TanStack/db/pull/1751)) + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 0.4.1 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 0.4.0 + +### Minor Changes + +- Add SSR through request-scoped `DbClient` instances, collection descriptors, ([#1564](https://github.com/TanStack/db/pull/1564)) + explicit collection-row hydration, live-query result snapshots, adapter sync + metadata, and React and Svelte descriptor resolution. + + React live queries now derive identity from structured query IR. Opaque queries + can provide `queryKey`; legacy dependency arrays and unkeyed opaque queries keep + working with development warnings until 1.0. + + Add TanStack Router integration that streams live queries discovered during a + Suspense render as pending promises which resolve to ordered result snapshots. + The browser starts normal source sync and atomically replaces the snapshot when + its live result is ready. + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 0.3.18 + +### Patch Changes + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 0.3.17 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 0.3.16 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 0.3.15 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 0.3.14 + +### Patch Changes + +- Prevent progressive Electric collections from truncating persisted rows when resuming from saved Electric shape metadata. ([#1493](https://github.com/TanStack/db/pull/1493)) + +- Bound the wait for Electric stream refreshes before loading on-demand subsets so native fetch implementations that do not promptly abort long polls do not keep live queries loading until the poll times out. ([#1575](https://github.com/TanStack/db/pull/1575)) + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 0.3.13 + +### Patch Changes + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 0.3.12 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 0.3.11 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 0.3.10 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 0.3.9 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 0.3.8 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 0.3.7 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 0.3.6 ### Patch Changes diff --git a/packages/electric-db-collection/e2e/electric.e2e.test.ts b/packages/electric-db-collection/e2e/electric.e2e.test.ts index 3e36ece785..4fde39ab2c 100644 --- a/packages/electric-db-collection/e2e/electric.e2e.test.ts +++ b/packages/electric-db-collection/e2e/electric.e2e.test.ts @@ -5,7 +5,7 @@ */ import { afterAll, afterEach, beforeAll, describe, inject } from 'vitest' -import { createCollection, BasicIndex } from '@tanstack/db' +import { BasicIndex, createCollection } from '@tanstack/db' import { ELECTRIC_TEST_HOOKS, electricCollectionOptions } from '../src/electric' import { makePgClient } from '../../db-collection-e2e/support/global-setup' import { diff --git a/packages/electric-db-collection/package.json b/packages/electric-db-collection/package.json index a265310672..299b97095d 100644 --- a/packages/electric-db-collection/package.json +++ b/packages/electric-db-collection/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/electric-db-collection", - "version": "0.3.6", + "version": "0.4.8", "description": "ElectricSQL collection for TanStack DB", "author": "Kyle Mathews", "license": "MIT", @@ -21,6 +21,7 @@ "dev": "vite build --watch", "lint": "eslint . --fix", "test": "vitest run", + "test:oracles": "vitest run tests/electric-oracle.property.test.ts tests/electric-recovery-oracle.test.ts tests/electric-descriptor-isolation.test.ts tests/electric-sdk-framing.test.ts", "test:e2e": "vitest run --config vitest.e2e.config.ts" }, "type": "module", @@ -49,10 +50,10 @@ "@electric-sql/client": "^1.5.15", "@standard-schema/spec": "^1.1.0", "@tanstack/db": "workspace:*", - "@tanstack/store": "^0.9.2", "debug": "^4.4.3" }, "devDependencies": { + "@tanstack/query-core": "^5.90.20", "@types/debug": "^4.1.12", "@types/pg": "^8.16.0", "@vitest/coverage-istanbul": "^3.2.4", diff --git a/packages/electric-db-collection/src/electric.ts b/packages/electric-db-collection/src/electric.ts index 72a9a072e0..8152fd0a54 100644 --- a/packages/electric-db-collection/src/electric.ts +++ b/packages/electric-db-collection/src/electric.ts @@ -4,9 +4,15 @@ import { isControlMessage, isVisibleInSnapshot, } from '@electric-sql/client' -import { Store } from '@tanstack/store' import DebugModule from 'debug' -import { DeduplicatedLoadSubset, and } from '@tanstack/db' +import { + DeduplicatedLoadSubset, + LoadSubsetOperationAbortedError, + and, + withCollectionConfigFactory, + withCollectionSyncConfigCleanup, + withCollectionSyncConfigFactory, +} from '@tanstack/db' import { ExpectedNumberInAwaitTxIdError, StreamAbortedError, @@ -43,7 +49,9 @@ import type { DeleteMutationFnParams, InsertMutationFnParams, LoadSubsetOptions, + SyncAppliedReceipt, SyncConfig, + SyncMetadataApi, SyncMode, UpdateMutationFnParams, UtilsRecord, @@ -59,11 +67,21 @@ import type { ShapeStreamOptions, } from '@electric-sql/client' +type ElectricSyncMetadataWithHydration = SyncMetadataApi & { + row: SyncMetadataApi[`row`] & { + whenHydrated?: () => Promise + // Capability marker for wrappers predating the hydration barrier. + scanPersisted?: unknown + } +} + // Re-export for user convenience in custom match functions export { isChangeMessage, isControlMessage } from '@electric-sql/client' const debug = DebugModule.debug(`ts/db:electric`) +const FORCE_DISCONNECT_AND_REFRESH_TIMEOUT_MS = 250 + /** * Symbol for internal test hooks (hidden from public API) */ @@ -85,6 +103,191 @@ export interface ElectricTestHooks { */ export type Txid = number +type ElectricResumeState = + | { + kind: `resume` + offset: string + handle: string + shapeId: string + updatedAt: number + requiresTagState?: boolean + } + | { + kind: `reset` + updatedAt: number + } + +type ElectricSyncMeta = { + version: 1 + resume?: ElectricResumeState + seenTxids: Array +} + +type ElectricLifecycleEvidence = { + seenTxids: Set + seenSnapshots: Array + hydratedResumeState?: ElectricResumeState +} + +type ElectricPendingMatch> = { + matchFn: (message: Message) => boolean + resolve: (value: boolean) => void + reject: (error: Error) => void + timeoutId: ReturnType + matched: boolean +} + +type ElectricMatchBuffer> = { + committedMessages: Array> + pendingMessages: Array> +} + +function exportElectricSyncMeta( + evidence: ElectricLifecycleEvidence, +): ElectricSyncMeta { + const resume = evidence.hydratedResumeState + return { + version: 1, + ...(resume ? { resume } : {}), + seenTxids: Array.from(evidence.seenTxids).sort((a, b) => a - b), + } +} + +function importElectricSyncMeta( + evidence: ElectricLifecycleEvidence, + meta: unknown, +): void { + const parsed = parseElectricSyncMeta(meta) + if (!parsed) return + + evidence.hydratedResumeState = parsed.resume + evidence.seenTxids = new Set(parsed.seenTxids) +} + +function parseElectricResumeState( + value: unknown, +): ElectricResumeState | undefined { + if (!value || typeof value !== `object`) { + return undefined + } + + const record = value as Record + if ( + record.kind === `resume` && + typeof record.offset === `string` && + typeof record.handle === `string` && + typeof record.shapeId === `string` && + typeof record.updatedAt === `number` && + Number.isFinite(record.updatedAt) + ) { + return { + kind: `resume`, + offset: record.offset, + handle: record.handle, + shapeId: record.shapeId, + updatedAt: record.updatedAt, + ...(typeof record.requiresTagState === `boolean` && { + requiresTagState: record.requiresTagState, + }), + } + } + + if ( + record.kind === `reset` && + typeof record.updatedAt === `number` && + Number.isFinite(record.updatedAt) + ) { + return { + kind: `reset`, + updatedAt: record.updatedAt, + } + } + + return undefined +} + +function parseElectricSyncMeta(value: unknown): ElectricSyncMeta | undefined { + if (!value || typeof value !== `object`) { + return undefined + } + + const record = value as Record + if ( + record.version !== 1 || + !Array.isArray(record.seenTxids) || + !record.seenTxids.every( + (txid) => typeof txid === `number` && Number.isFinite(txid), + ) + ) { + return undefined + } + + const resume = + record.resume === undefined + ? undefined + : parseElectricResumeState(record.resume) + if (record.resume !== undefined && resume === undefined) { + return undefined + } + + return { + version: 1, + ...(resume ? { resume } : {}), + seenTxids: Array.from(new Set(record.seenTxids)).sort((a, b) => a - b), + } +} + +function mergeElectricSyncMeta( + current: unknown, + incoming: unknown, +): ElectricSyncMeta | unknown { + const currentMeta = parseElectricSyncMeta(current) + const incomingMeta = parseElectricSyncMeta(incoming) + + if (!incomingMeta) { + return current + } + if (!currentMeta) { + return incomingMeta + } + + const resume = getNewestElectricResumeState( + currentMeta.resume, + incomingMeta.resume, + ) + + return { + version: 1, + ...(resume ? { resume } : {}), + seenTxids: Array.from( + new Set([...currentMeta.seenTxids, ...incomingMeta.seenTxids]), + ).sort((a, b) => a - b), + } +} + +function getNewestElectricResumeState( + current: ElectricResumeState | undefined, + incoming: ElectricResumeState | undefined, +): ElectricResumeState | undefined { + if (!current) return incoming + if (!incoming) return current + if (incoming.updatedAt > current.updatedAt) return incoming + if (incoming.updatedAt < current.updatedAt) return current + + const isSameState = + current.kind === incoming.kind && + (current.kind === `reset` || + (incoming.kind === `resume` && + current.offset === incoming.offset && + current.handle === incoming.handle && + current.shapeId === incoming.shapeId)) + + // Equal timestamps have no causal ordering. Preserve an identical state, + // but collapse every conflict to reset so merge order cannot resurrect an + // offset that another source has already declared unsafe. + return isSameState ? current : { kind: `reset`, updatedAt: current.updatedAt } +} + /** * Custom match function type - receives stream messages and returns boolean * indicating if the mutation has been synchronized @@ -394,6 +597,8 @@ function createLoadSubsetDedupe>({ begin, write, commit, + getCommitCursor, + waitForCommitsAfter, collectionId, encodeColumnName, signal, @@ -407,7 +612,9 @@ function createLoadSubsetDedupe>({ value: T metadata: Record }) => void - commit: () => void + commit: (signal?: AbortSignal) => SyncAppliedReceipt + getCommitCursor: () => number + waitForCommitsAfter: (cursor: number) => Promise collectionId?: string /** * Optional function to encode column names (e.g., camelCase to snake_case). @@ -427,6 +634,9 @@ function createLoadSubsetDedupe>({ const compileOptions = encodeColumnName ? { encodeColumnName } : undefined const logPrefix = collectionId ? `[${collectionId}] ` : `` + const abortReason = (abortedSignal: AbortSignal): unknown => + abortedSignal.reason ?? new LoadSubsetOperationAbortedError() + /** * Handles errors from snapshot operations. Returns true if the error was * handled (signal aborted during cleanup), false if it should be re-thrown. @@ -441,12 +651,18 @@ function createLoadSubsetDedupe>({ } const loadSubset = async (opts: LoadSubsetOptions) => { + const commitCursor = getCommitCursor() + const throwIfAborted = () => { + if (signal.aborted) throw abortReason(signal) + if (opts.signal?.aborted) throw abortReason(opts.signal) + } + throwIfAborted() + if (isBufferingInitialSync()) { const snapshotParams = compileSQL(opts, compileOptions) try { const { data: rows } = await stream.fetchSnapshot(snapshotParams) - - if (!isBufferingInitialSync()) { + if (opts.signal?.aborted || !isBufferingInitialSync()) { debug(`${logPrefix}Ignoring snapshot - sync completed while fetching`) return } @@ -460,10 +676,11 @@ function createLoadSubsetDedupe>({ metadata: { ...row.headers }, }) } - commit() + await commit(opts.signal) debug(`${logPrefix}Applied snapshot with ${rows.length} rows`) } } catch (error) { + if (opts.signal?.aborted) return if (handleSnapshotError(error, `fetchSnapshot`)) { return } @@ -481,12 +698,38 @@ function createLoadSubsetDedupe>({ // When the stream is already up-to-date, it may be in a long-poll wait. // Forcing a disconnect-and-refresh ensures requestSnapshot gets a response // from a fresh server round-trip rather than waiting for the current poll to end. - // If the refresh fails (e.g., PauseLock held during subscriber processing in - // join pipelines), we fall through to requestSnapshot which still works. + // Some native fetch implementations (notably React Native/Expo) may not abort + // long-poll requests promptly. Bound the wait so on-demand live queries don't + // remain loading until the long-poll naturally times out. + // If the refresh fails or times out, we fall through to requestSnapshot which + // still works. if (stream.isUpToDate) { + let timeoutId: ReturnType | undefined + const abortSignals = [signal, opts.signal].filter( + (candidate): candidate is AbortSignal => candidate !== undefined, + ) + let rejectAbort: (reason: unknown) => void = () => {} + const aborted = new Promise((_resolve, reject) => { + rejectAbort = reject + }) + const abort = (event: Event) => + rejectAbort(abortReason(event.currentTarget as AbortSignal)) + for (const abortSignal of abortSignals) { + abortSignal.addEventListener(`abort`, abort, { once: true }) + } try { - await stream.forceDisconnectAndRefresh() + await Promise.race([ + stream.forceDisconnectAndRefresh(), + new Promise((resolve) => { + timeoutId = setTimeout( + resolve, + FORCE_DISCONNECT_AND_REFRESH_TIMEOUT_MS, + ) + }), + aborted, + ]) } catch (error) { + if (signal.aborted || opts.signal?.aborted) throw error if (handleSnapshotError(error, `forceDisconnectAndRefresh`)) { return } @@ -494,9 +737,22 @@ function createLoadSubsetDedupe>({ `${logPrefix}forceDisconnectAndRefresh failed, proceeding to requestSnapshot: %o`, error, ) + } finally { + clearTimeout(timeoutId) + for (const abortSignal of abortSignals) { + abortSignal.removeEventListener(`abort`, abort) + } } } + throwIfAborted() + + // Upstream limitation: ShapeStream.requestSnapshot() publishes its rows + // through the stream callback before its Promise resolves. It accepts no + // request signal and exposes no request identity on those messages, so an + // aborted request can already have installed rows before the check below. + // Full request-scoped cancellation requires support in the Electric client; + // matching snapshots by parameters is unsafe for overlapping equal requests. try { if (cursor) { const whereCurrentOpts: LoadSubsetOptions = { @@ -529,11 +785,13 @@ function createLoadSubsetDedupe>({ await stream.requestSnapshot(snapshotParams) } } catch (error) { + if (opts.signal?.aborted) return if (handleSnapshotError(error, `requestSnapshot`)) { return } throw error } + await waitForCommitsAfter(commitCursor) } return new DeduplicatedLoadSubset({ loadSubset }) @@ -562,6 +820,244 @@ export interface ElectricCollectionUtils< awaitMatch: AwaitMatchFn } +/** Owns evidence and pending work for one materialized Collection. */ +class ElectricLifecycle> { + private readonly evidence: ElectricLifecycleEvidence = { + seenTxids: new Set(), + seenSnapshots: [], + } + + private readonly pendingMatches = new Map>() + private readonly pendingTxidWaits = new Map< + number, + { + txId: Txid + resolve: (value: boolean) => void + reject: (error: Error) => void + timeoutId: ReturnType + } + >() + private matchBuffer: ElectricMatchBuffer = { + committedMessages: [], + pendingMessages: [], + } + private nextWaiterId = 0 + private epoch = 0 + private active = false + + constructor(private readonly collectionId?: string) {} + + readonly utils: ElectricCollectionUtils = { + awaitTxId: (txId, timeout) => this.awaitTxId(txId, timeout), + awaitMatch: (matchFn, timeout) => this.awaitMatch(matchFn, timeout), + } + + start(): number { + if (this.active) this.retire() + this.active = true + this.epoch++ + this.matchBuffer = { committedMessages: [], pendingMessages: [] } + return this.epoch + } + + isActive(epoch: number): boolean { + return this.active && this.epoch === epoch + } + + retire(epoch?: number): void { + if (epoch !== undefined && !this.isActive(epoch)) return + this.active = false + this.epoch++ + + for (const match of this.pendingMatches.values()) { + clearTimeout(match.timeoutId) + match.reject(new StreamAbortedError(this.collectionId)) + } + this.pendingMatches.clear() + + for (const waiter of this.pendingTxidWaits.values()) { + clearTimeout(waiter.timeoutId) + waiter.reject(new StreamAbortedError(this.collectionId)) + } + this.pendingTxidWaits.clear() + this.matchBuffer = { committedMessages: [], pendingMessages: [] } + this.evidence.hydratedResumeState = undefined + } + + exportMeta(): ElectricSyncMeta { + return exportElectricSyncMeta(this.evidence) + } + + importMeta(meta: unknown): void { + importElectricSyncMeta(this.evidence, meta) + this.resolveTxidWaiters() + } + + get resumeState(): ElectricResumeState | undefined { + return this.evidence.hydratedResumeState + } + + set resumeState(value: ElectricResumeState | undefined) { + this.evidence.hydratedResumeState = value + } + + publishEvidence( + txids: ReadonlySet, + snapshots: ReadonlyArray, + ): void { + txids.forEach((txid) => this.evidence.seenTxids.add(txid)) + this.evidence.seenSnapshots.push(...snapshots) + this.resolveTxidWaiters() + } + + private hasTxid(txId: Txid): boolean { + return ( + this.evidence.seenTxids.has(txId) || + this.evidence.seenSnapshots.some((snapshot) => + isVisibleInSnapshot(txId, snapshot), + ) + ) + } + + private resolveTxidWaiters(): void { + for (const [waitId, waiter] of this.pendingTxidWaits) { + if (!this.hasTxid(waiter.txId)) continue + clearTimeout(waiter.timeoutId) + this.pendingTxidWaits.delete(waitId) + waiter.resolve(true) + } + } + + private async awaitTxId( + txId: Txid, + timeout: number = 5000, + ): Promise { + debug( + `${this.collectionId ? `[${this.collectionId}] ` : ``}awaitTxId called with txid %d`, + txId, + ) + if (typeof txId !== `number`) { + throw new ExpectedNumberInAwaitTxIdError(typeof txId, this.collectionId) + } + if (this.hasTxid(txId)) return true + + return new Promise((resolve, reject) => { + const waitId = this.nextWaiterId++ + const timeoutId = setTimeout(() => { + this.pendingTxidWaits.delete(waitId) + reject(new TimeoutWaitingForTxIdError(txId, this.collectionId)) + }, timeout) + this.pendingTxidWaits.set(waitId, { + txId, + resolve, + reject, + timeoutId, + }) + }) + } + + private async awaitMatch( + matchFn: MatchFunction, + timeout: number = 3000, + ): Promise { + debug( + `${this.collectionId ? `[${this.collectionId}] ` : ``}awaitMatch called with custom function`, + ) + + for (const message of this.matchBuffer.committedMessages) { + if (!matchFn(message)) continue + return true + } + for (const message of this.matchBuffer.pendingMessages) { + if (matchFn(message)) return this.registerMatch(matchFn, timeout, true) + } + return this.registerMatch(matchFn, timeout, false) + } + + private registerMatch( + matchFn: MatchFunction, + timeout: number, + matched: boolean, + ): Promise { + return new Promise((resolve, reject) => { + const matchId = this.nextWaiterId++ + const timeoutId = setTimeout(() => { + this.pendingMatches.delete(matchId) + reject(new TimeoutWaitingForMatchError(this.collectionId)) + }, timeout) + this.pendingMatches.set(matchId, { + matchFn, + resolve, + reject, + timeoutId, + matched, + }) + }) + } + + beginMatchGeneration(messages: ReadonlyArray>): void { + if (!messages.some(isMustRefetchMessage)) return + this.matchBuffer = { committedMessages: [], pendingMessages: [] } + for (const match of this.pendingMatches.values()) match.matched = false + } + + observeMatchMessage(message: Message): void { + if ( + isChangeMessage(message) || + isMoveOutMessage(message) || + isMoveInMessage(message) + ) { + this.matchBuffer.pendingMessages.push(message) + let overflow = + this.matchBuffer.committedMessages.length + + this.matchBuffer.pendingMessages.length - + 1000 + if (overflow > 0) { + const committedOverflow = Math.min( + overflow, + this.matchBuffer.committedMessages.length, + ) + this.matchBuffer.committedMessages.splice(0, committedOverflow) + overflow -= committedOverflow + if (overflow > 0) { + this.matchBuffer.pendingMessages.splice(0, overflow) + } + } + } + + const epoch = this.epoch + for (const [matchId, match] of this.pendingMatches) { + if (match.matched) continue + try { + match.matched = match.matchFn(message) + } catch (error) { + clearTimeout(match.timeoutId) + this.pendingMatches.delete(matchId) + match.reject(error instanceof Error ? error : new Error(String(error))) + } + // Reentry can register replacement-session waiters in this same Map. + if (!this.isActive(epoch)) return + } + } + + commitMatches(): void { + this.matchBuffer.committedMessages.push(...this.matchBuffer.pendingMessages) + this.matchBuffer.pendingMessages = [] + if (this.matchBuffer.committedMessages.length > 1000) { + this.matchBuffer.committedMessages.splice( + 0, + this.matchBuffer.committedMessages.length - 1000, + ) + } + for (const [matchId, match] of this.pendingMatches) { + if (!match.matched) continue + clearTimeout(match.timeoutId) + this.pendingMatches.delete(matchId) + match.resolve(true) + } + } +} + /** * Creates Electric collection options for use with a standard Collection * @@ -615,263 +1111,55 @@ export function electricCollectionOptions>( utils: ElectricCollectionUtils schema?: any } { - const seenTxids = new Store>(new Set([])) - const seenSnapshots = new Store>([]) + let descriptorLifecycle = new ElectricLifecycle(config.id) + let utilityLifecycle = descriptorLifecycle const internalSyncMode = config.syncMode ?? `eager` const finalSyncMode = internalSyncMode === `progressive` ? `on-demand` : internalSyncMode - const pendingMatches = new Store< - Map< - string, - { - matchFn: (message: Message) => boolean - resolve: (value: boolean) => void - reject: (error: Error) => void - timeoutId: ReturnType - matched: boolean - } - > - >(new Map()) - - // Buffer messages since last up-to-date to handle race conditions - const currentBatchMessages = new Store>>([]) - - // Track whether the current batch has been committed (up-to-date received) - // This allows awaitMatch to resolve immediately for messages from committed batches - const batchCommitted = new Store(false) - - /** - * Helper function to remove multiple matches from the pendingMatches store - */ - const removePendingMatches = (matchIds: Array) => { - if (matchIds.length > 0) { - pendingMatches.setState((current) => { - const newMatches = new Map(current) - matchIds.forEach((id) => newMatches.delete(id)) - return newMatches - }) - } - } - - /** - * Helper function to resolve and cleanup matched pending matches - */ - const resolveMatchedPendingMatches = () => { - const matchesToResolve: Array = [] - pendingMatches.state.forEach((match, matchId) => { - if (match.matched) { - clearTimeout(match.timeoutId) - match.resolve(true) - matchesToResolve.push(matchId) - debug( - `${config.id ? `[${config.id}] ` : ``}awaitMatch resolved on up-to-date for match %s`, - matchId, - ) - } - }) - removePendingMatches(matchesToResolve) - } - const sync = createElectricSync(config.shapeOptions, { - seenTxids, - seenSnapshots, - syncMode: internalSyncMode, - pendingMatches, - currentBatchMessages, - batchCommitted, - removePendingMatches, - resolveMatchedPendingMatches, - collectionId: config.id, - testHooks: config[ELECTRIC_TEST_HOOKS], - }) - - /** - * Wait for a specific transaction ID to be synced - * @param txId The transaction ID to wait for as a number - * @param timeout Optional timeout in milliseconds (defaults to 5000ms) - * @returns Promise that resolves when the txId is synced - */ - const awaitTxId: AwaitTxIdFn = async ( - txId: Txid, - timeout: number = 5000, - ): Promise => { - debug( - `${config.id ? `[${config.id}] ` : ``}awaitTxId called with txid %d`, - txId, - ) - if (typeof txId !== `number`) { - throw new ExpectedNumberInAwaitTxIdError(typeof txId, config.id) - } - - // First check if the txid is in the seenTxids store - const hasTxid = seenTxids.state.has(txId) - if (hasTxid) return true - - // Then check if the txid is in any of the seen snapshots - const hasSnapshot = seenSnapshots.state.some((snapshot) => - isVisibleInSnapshot(txId, snapshot), - ) - if (hasSnapshot) return true - - return new Promise((resolve, reject) => { - const cleanup = () => { - clearTimeout(timeoutId) - subSeenTxids.unsubscribe() - subSeenSnapshots.unsubscribe() - } - - const timeoutId = setTimeout(() => { - cleanup() - reject(new TimeoutWaitingForTxIdError(txId, config.id)) - }, timeout) - - const subSeenTxids = seenTxids.subscribe(() => { - if (seenTxids.state.has(txId)) { - debug( - `${config.id ? `[${config.id}] ` : ``}awaitTxId found match for txid %o`, - txId, - ) - cleanup() - resolve(true) - } - }) - - const subSeenSnapshots = seenSnapshots.subscribe(() => { - const visibleSnapshot = seenSnapshots.state.find((snapshot) => - isVisibleInSnapshot(txId, snapshot), - ) - if (visibleSnapshot) { - debug( - `${config.id ? `[${config.id}] ` : ``}awaitTxId found match for txid %o in snapshot %o`, - txId, - visibleSnapshot, - ) - cleanup() - resolve(true) - } - }) - }) - } - - /** - * Wait for a custom match function to find a matching message - * @param matchFn Function that returns true when a message matches - * @param timeout Optional timeout in milliseconds (defaults to 5000ms) - * @returns Promise that resolves when a matching message is found - */ - const awaitMatch: AwaitMatchFn = async ( - matchFn: MatchFunction, - timeout: number = 3000, - ): Promise => { - debug( - `${config.id ? `[${config.id}] ` : ``}awaitMatch called with custom function`, - ) - - return new Promise((resolve, reject) => { - const matchId = Math.random().toString(36) - - const cleanupMatch = () => { - pendingMatches.setState((current) => { - const newMatches = new Map(current) - newMatches.delete(matchId) - return newMatches - }) - } - - const onTimeout = () => { - cleanupMatch() - reject(new TimeoutWaitingForMatchError(config.id)) - } - - const timeoutId = setTimeout(onTimeout, timeout) - - // We need access to the stream messages to check against the match function - // This will be handled by the sync configuration - const checkMatch = (message: Message) => { - if (matchFn(message)) { - debug( - `${config.id ? `[${config.id}] ` : ``}awaitMatch found matching message, waiting for up-to-date`, - ) - // Mark as matched but don't resolve yet - wait for up-to-date - pendingMatches.setState((current) => { - const newMatches = new Map(current) - const existing = newMatches.get(matchId) - if (existing) { - newMatches.set(matchId, { ...existing, matched: true }) - } - return newMatches - }) - return true - } - return false - } - - // Check against current batch messages first to handle race conditions - for (const message of currentBatchMessages.state) { - if (matchFn(message)) { - // If batch is committed (up-to-date already received), resolve immediately - // just like awaitTxId does when it finds a txid in seenTxids - if (batchCommitted.state) { - debug( - `${config.id ? `[${config.id}] ` : ``}awaitMatch found immediate match in committed batch, resolving immediately`, - ) - clearTimeout(timeoutId) - resolve(true) - return - } - - // If batch is not yet committed, register match and wait for up-to-date - debug( - `${config.id ? `[${config.id}] ` : ``}awaitMatch found immediate match in current batch, waiting for up-to-date`, - ) - pendingMatches.setState((current) => { - const newMatches = new Map(current) - newMatches.set(matchId, { - matchFn: checkMatch, - resolve, - reject, - timeoutId, - matched: true, // Already matched, will resolve on up-to-date - }) - return newMatches - }) - return - } - } - - // Store the match function for the sync process to use - // We'll add this to a pending matches store - pendingMatches.setState((current) => { - const newMatches = new Map(current) - newMatches.set(matchId, { - matchFn: checkMatch, - resolve, - reject, - timeoutId, - matched: false, - }) - return newMatches - }) + const boundLifecycles = new WeakMap>() + const awaitTxId: AwaitTxIdFn = (txId, timeout) => + utilityLifecycle.utils.awaitTxId(txId, timeout) + + const awaitMatch: AwaitMatchFn = (matchFn, timeout) => + utilityLifecycle.utils.awaitMatch(matchFn, timeout) + + const createSync = () => + createElectricSync(config.shapeOptions, { + getLifecycle: (collection) => + boundLifecycles.get(collection) ?? descriptorLifecycle, + syncMode: internalSyncMode, + collectionId: config.id, + testHooks: config[ELECTRIC_TEST_HOOKS], }) - } + const sync = createSync() /** * Process matching strategy and wait for synchronization */ const processMatchingStrategy = async ( result: MatchingStrategy, + waitForTxId: AwaitTxIdFn, ): Promise => { // Only wait if result contains txid if (result && `txid` in result) { const timeout = result.timeout // Handle both single txid and array of txids if (Array.isArray(result.txid)) { - await Promise.all(result.txid.map((txid) => awaitTxId(txid, timeout))) + await Promise.all(result.txid.map((txid) => waitForTxId(txid, timeout))) } else { - await awaitTxId(result.txid, timeout) + await waitForTxId(result.txid, timeout) } } // If result is void/undefined, don't wait - mutation completes immediately } + const getMutationAwaitTxId = (params: unknown): AwaitTxIdFn => { + const collection = ( + params as { + collection?: { utils?: { awaitTxId?: AwaitTxIdFn } } + } + ).collection + return collection?.utils?.awaitTxId ?? awaitTxId + } // Create wrapper handlers for direct persistence operations that handle different matching strategies const wrappedOnInsert = config.onInsert @@ -883,7 +1171,10 @@ export function electricCollectionOptions>( >, ) => { const handlerResult = await config.onInsert!(params) - await processMatchingStrategy(handlerResult) + await processMatchingStrategy( + handlerResult, + getMutationAwaitTxId(params), + ) return handlerResult } : undefined @@ -897,7 +1188,10 @@ export function electricCollectionOptions>( >, ) => { const handlerResult = await config.onUpdate!(params) - await processMatchingStrategy(handlerResult) + await processMatchingStrategy( + handlerResult, + getMutationAwaitTxId(params), + ) return handlerResult } : undefined @@ -911,7 +1205,10 @@ export function electricCollectionOptions>( >, ) => { const handlerResult = await config.onDelete!(params) - await processMatchingStrategy(handlerResult) + await processMatchingStrategy( + handlerResult, + getMutationAwaitTxId(params), + ) return handlerResult } : undefined @@ -925,18 +1222,75 @@ export function electricCollectionOptions>( ...restConfig } = config - return { + const utilityTemplate: ElectricCollectionUtils = { + awaitTxId, + awaitMatch, + } + const consumeDescriptorLifecycle = (): ElectricLifecycle => { + const lifecycle = descriptorLifecycle + descriptorLifecycle = new ElectricLifecycle(config.id) + utilityLifecycle = lifecycle + return lifecycle + } + const createBoundSync = ( + source: SyncConfig, + utilities: object, + ): SyncConfig => { + const lifecycle = consumeDescriptorLifecycle() + let collectionKey: object | undefined + Object.assign(utilities, lifecycle.utils) + + const boundSync: SyncConfig = { + ...source, + sync: (params) => { + // A copied materialized config may delegate through an older binding. + // The outermost binding owns this collection; inner wrappers only run + // the source sync and must not retarget either owner's lifecycle. + if (!boundLifecycles.has(params.collection)) { + collectionKey = params.collection + boundLifecycles.set(params.collection, lifecycle) + } + return source.sync(params) + }, + exportSyncMeta: () => lifecycle.exportMeta(), + importSyncMeta: (meta) => lifecycle.importMeta(meta), + mergeSyncMeta: mergeElectricSyncMeta, + } + return withCollectionSyncConfigCleanup(boundSync, () => { + lifecycle.retire() + if (collectionKey) boundLifecycles.delete(collectionKey) + }) + } + const syncTemplate = withCollectionSyncConfigFactory( + { + ...sync, + exportSyncMeta: () => descriptorLifecycle.exportMeta(), + importSyncMeta: (meta) => descriptorLifecycle.importMeta(meta), + mergeSyncMeta: mergeElectricSyncMeta, + }, + createBoundSync, + ) + const options = { ...restConfig, syncMode: finalSyncMode, - sync, + sync: syncTemplate, onInsert: wrappedOnInsert, onUpdate: wrappedOnUpdate, onDelete: wrappedOnDelete, - utils: { - awaitTxId, - awaitMatch, - }, + utils: utilityTemplate, } + Object.defineProperty(options, `utils`, { + enumerable: true, + get: () => ({ ...utilityTemplate }), + }) + + return withCollectionConfigFactory(options, () => + ( + electricCollectionOptions as ( + nextConfig: ElectricCollectionConfig, + ) => typeof options + )(config), + ) } /** @@ -946,397 +1300,395 @@ function createElectricSync>( shapeOptions: ShapeStreamOptions>, options: { syncMode: ElectricSyncMode - seenTxids: Store> - seenSnapshots: Store> - pendingMatches: Store< - Map< - string, - { - matchFn: (message: Message) => boolean - resolve: (value: boolean) => void - reject: (error: Error) => void - timeoutId: ReturnType - matched: boolean - } - > - > - currentBatchMessages: Store>> - batchCommitted: Store - removePendingMatches: (matchIds: Array) => void - resolveMatchedPendingMatches: () => void + getLifecycle: (collection: object) => ElectricLifecycle collectionId?: string testHooks?: ElectricTestHooks }, ): SyncConfig { - const { - seenTxids, - seenSnapshots, - syncMode, - pendingMatches, - currentBatchMessages, - batchCommitted, - removePendingMatches, - resolveMatchedPendingMatches, - collectionId, - testHooks, - } = options - const MAX_BATCH_MESSAGES = 1000 // Safety limit for message buffer - - // Store for the relation schema information - const relationSchema = new Store(undefined) - - const tagCache = new Map() - - // Parses a tag string into a ParsedMoveTag. - // It memoizes the result parsed tag such that future calls - // for the same tag string return the same ParsedMoveTag array. - const parseTag = (tag: MoveTag): ParsedMoveTag => { - const cachedTag = tagCache.get(tag) - if (cachedTag) { - return cachedTag + const { getLifecycle, syncMode, collectionId, testHooks } = options + + let relationSchema: string | undefined + let warnedUnverifiableResume = false + + const createTagState = () => { + const tagCache = new Map() + + // Parses a tag string into a ParsedMoveTag. + // It memoizes the result parsed tag such that future calls + // for the same tag string return the same ParsedMoveTag array. + const parseTag = (tag: MoveTag): ParsedMoveTag => { + const cachedTag = tagCache.get(tag) + if (cachedTag) { + return cachedTag + } + + const parsedTag = parseTagString(tag) + tagCache.set(tag, parsedTag) + return parsedTag } - const parsedTag = parseTagString(tag) - tagCache.set(tag, parsedTag) - return parsedTag - } + // Tag tracking state + const rowTagSets = new Map>() + const tagIndex: TagIndex = [] + let tagLength: number | undefined = undefined + + // DNF state: active_conditions are per-row, disjunct_positions are global + // (fixed by the shape's WHERE clause, derived once from the first tagged message). + const rowActiveConditions = new Map() + let disjunctPositions: DisjunctPositions | undefined = undefined + + /** + * Initialize the tag index with the correct length + */ + const initializeTagIndex = (length: number): void => { + if (tagIndex.length < length) { + // Extend the index array to the required length + for (let i = tagIndex.length; i < length; i++) { + tagIndex[i] = new Map() + } + } + } - // Tag tracking state - const rowTagSets = new Map>() - const tagIndex: TagIndex = [] - let tagLength: number | undefined = undefined + /** + * Add tags to a row and update the tag index + */ + const addTagsToRow = ( + tags: Array, + rowId: RowId, + rowTagSet: Set, + ): void => { + for (const tag of tags) { + const parsedTag = parseTag(tag) + + // Infer tag length from first tag + if (tagLength === undefined) { + tagLength = getTagLength(parsedTag) + initializeTagIndex(tagLength) + } - // DNF state: active_conditions are per-row, disjunct_positions are global - // (fixed by the shape's WHERE clause, derived once from the first tagged message). - const rowActiveConditions = new Map() - let disjunctPositions: DisjunctPositions | undefined = undefined + // Validate tag length matches + const currentTagLength = getTagLength(parsedTag) + if (currentTagLength !== tagLength) { + debug( + `${collectionId ? `[${collectionId}] ` : ``}Tag length mismatch: expected ${tagLength}, got ${currentTagLength}`, + ) + continue + } - /** - * Initialize the tag index with the correct length - */ - const initializeTagIndex = (length: number): void => { - if (tagIndex.length < length) { - // Extend the index array to the required length - for (let i = tagIndex.length; i < length; i++) { - tagIndex[i] = new Map() + rowTagSet.add(tag) + addTagToIndex(parsedTag, rowId, tagIndex, tagLength) } } - } - /** - * Add tags to a row and update the tag index - */ - const addTagsToRow = ( - tags: Array, - rowId: RowId, - rowTagSet: Set, - ): void => { - for (const tag of tags) { - const parsedTag = parseTag(tag) - - // Infer tag length from first tag + /** + * Remove tags from a row and update the tag index + */ + const removeTagsFromRow = ( + removedTags: Array, + rowId: RowId, + rowTagSet: Set, + ): void => { if (tagLength === undefined) { - tagLength = getTagLength(parsedTag) - initializeTagIndex(tagLength) + return } - // Validate tag length matches - const currentTagLength = getTagLength(parsedTag) - if (currentTagLength !== tagLength) { - debug( - `${collectionId ? `[${collectionId}] ` : ``}Tag length mismatch: expected ${tagLength}, got ${currentTagLength}`, - ) - continue + for (const tag of removedTags) { + const parsedTag = parseTag(tag) + rowTagSet.delete(tag) + removeTagFromIndex(parsedTag, rowId, tagIndex, tagLength) + // We aggresively evict the tag from the cache + // if this tag is shared with another row + // and is not removed from that other row + // then next time we encounter the tag it will be parsed again + tagCache.delete(tag) } - - rowTagSet.add(tag) - addTagToIndex(parsedTag, rowId, tagIndex, tagLength) } - } - /** - * Remove tags from a row and update the tag index - */ - const removeTagsFromRow = ( - removedTags: Array, - rowId: RowId, - rowTagSet: Set, - ): void => { - if (tagLength === undefined) { - return - } + /** + * Process tags for a change message (add and remove tags) + */ + const processTagsForChangeMessage = ( + tags: Array | undefined, + removedTags: Array | undefined, + rowId: RowId, + activeConditions?: ActiveConditions, + ): Set => { + // Initialize tag set for this row if it doesn't exist (needed for checking deletion) + if (!rowTagSets.has(rowId)) { + rowTagSets.set(rowId, new Set()) + } + const rowTagSet = rowTagSets.get(rowId)! - for (const tag of removedTags) { - const parsedTag = parseTag(tag) - rowTagSet.delete(tag) - removeTagFromIndex(parsedTag, rowId, tagIndex, tagLength) - // We aggresively evict the tag from the cache - // if this tag is shared with another row - // and is not removed from that other row - // then next time we encounter the tag it will be parsed again - tagCache.delete(tag) - } - } + // Add new tags + if (tags) { + addTagsToRow(tags, rowId, rowTagSet) - /** - * Process tags for a change message (add and remove tags) - */ - const processTagsForChangeMessage = ( - tags: Array | undefined, - removedTags: Array | undefined, - rowId: RowId, - activeConditions?: ActiveConditions, - ): Set => { - // Initialize tag set for this row if it doesn't exist (needed for checking deletion) - if (!rowTagSets.has(rowId)) { - rowTagSets.set(rowId, new Set()) - } - const rowTagSet = rowTagSets.get(rowId)! + // Derive disjunct positions once — they are fixed by the shape's WHERE clause. + if (disjunctPositions === undefined) { + const parsedTags = tags.map(parseTag) + disjunctPositions = deriveDisjunctPositions(parsedTags) + } + } - // Add new tags - if (tags) { - addTagsToRow(tags, rowId, rowTagSet) + // Remove tags + if (removedTags) { + removeTagsFromRow(removedTags, rowId, rowTagSet) + } - // Derive disjunct positions once — they are fixed by the shape's WHERE clause. - if (disjunctPositions === undefined) { - const parsedTags = tags.map(parseTag) - disjunctPositions = deriveDisjunctPositions(parsedTags) + // Store active conditions if provided (overwrite on re-send) + if (activeConditions && activeConditions.length > 0) { + rowActiveConditions.set(rowId, [...activeConditions]) } - } - // Remove tags - if (removedTags) { - removeTagsFromRow(removedTags, rowId, rowTagSet) + return rowTagSet } - // Store active conditions if provided (overwrite on re-send) - if (activeConditions && activeConditions.length > 0) { - rowActiveConditions.set(rowId, [...activeConditions]) + /** + * Clear all tag tracking state (used when truncating) + */ + const clearTagTrackingState = (): void => { + rowTagSets.clear() + tagIndex.length = 0 + tagLength = undefined + rowActiveConditions.clear() + disjunctPositions = undefined } - return rowTagSet - } + /** + * Remove all tags for a row from both the tag set and the index + * Used when a row is deleted + */ + const clearTagsForRow = (rowId: RowId): void => { + if (tagLength === undefined) { + return + } - /** - * Clear all tag tracking state (used when truncating) - */ - const clearTagTrackingState = (): void => { - rowTagSets.clear() - tagIndex.length = 0 - tagLength = undefined - rowActiveConditions.clear() - disjunctPositions = undefined - } + const rowTagSet = rowTagSets.get(rowId) + if (!rowTagSet) { + return + } - /** - * Remove all tags for a row from both the tag set and the index - * Used when a row is deleted - */ - const clearTagsForRow = (rowId: RowId): void => { - if (tagLength === undefined) { - return - } + // Remove each tag from the index + for (const tag of rowTagSet) { + const parsedTag = parseTag(tag) + const currentTagLength = getTagLength(parsedTag) + if (currentTagLength === tagLength) { + removeTagFromIndex(parsedTag, rowId, tagIndex, tagLength) + } + tagCache.delete(tag) + } - const rowTagSet = rowTagSets.get(rowId) - if (!rowTagSet) { - return + // Remove the row from the tag sets map + rowTagSets.delete(rowId) + rowActiveConditions.delete(rowId) } - // Remove each tag from the index - for (const tag of rowTagSet) { - const parsedTag = parseTag(tag) - const currentTagLength = getTagLength(parsedTag) - if (currentTagLength === tagLength) { - removeTagFromIndex(parsedTag, rowId, tagIndex, tagLength) + /** + * Remove matching tags from a row based on a pattern + * Returns true if the row should be deleted (no longer visible) + */ + const removeMatchingTagsFromRow = ( + rowId: RowId, + pattern: MovePattern, + ): boolean => { + const rowTagSet = rowTagSets.get(rowId) + if (!rowTagSet) { + return false } - tagCache.delete(tag) - } - - // Remove the row from the tag sets map - rowTagSets.delete(rowId) - rowActiveConditions.delete(rowId) - } - /** - * Remove matching tags from a row based on a pattern - * Returns true if the row should be deleted (no longer visible) - */ - const removeMatchingTagsFromRow = ( - rowId: RowId, - pattern: MovePattern, - ): boolean => { - const rowTagSet = rowTagSets.get(rowId) - if (!rowTagSet) { - return false - } + // DNF mode: check visibility using active conditions. + // Tag index entries are preserved so that move-in can re-activate positions. + const activeConditions = rowActiveConditions.get(rowId) + if (activeConditions && disjunctPositions) { + // Set the condition at this pattern's position to false + activeConditions[pattern.pos] = false + + if (!rowVisible(activeConditions, disjunctPositions)) { + // Row is no longer visible — clean up all state including tag index + for (const tag of rowTagSet) { + const parsedTag = parseTag(tag) + removeTagFromIndex(parsedTag, rowId, tagIndex, tagLength!) + tagCache.delete(tag) + } + rowTagSets.delete(rowId) + rowActiveConditions.delete(rowId) + return true + } + return false + } - // DNF mode: check visibility using active conditions. - // Tag index entries are preserved so that move-in can re-activate positions. - const activeConditions = rowActiveConditions.get(rowId) - if (activeConditions && disjunctPositions) { - // Set the condition at this pattern's position to false - activeConditions[pattern.pos] = false - - if (!rowVisible(activeConditions, disjunctPositions)) { - // Row is no longer visible — clean up all state including tag index - for (const tag of rowTagSet) { - const parsedTag = parseTag(tag) + // Simple shape (no subquery dependencies — server sends no active_conditions): + // Remove matching tags and delete if tag set is empty + for (const tag of rowTagSet) { + const parsedTag = parseTag(tag) + if (tagMatchesPattern(parsedTag, pattern)) { + rowTagSet.delete(tag) removeTagFromIndex(parsedTag, rowId, tagIndex, tagLength!) - tagCache.delete(tag) } + } + + if (rowTagSet.size === 0) { rowTagSets.delete(rowId) - rowActiveConditions.delete(rowId) return true } + return false } - // Simple shape (no subquery dependencies — server sends no active_conditions): - // Remove matching tags and delete if tag set is empty - for (const tag of rowTagSet) { - const parsedTag = parseTag(tag) - if (tagMatchesPattern(parsedTag, pattern)) { - rowTagSet.delete(tag) - removeTagFromIndex(parsedTag, rowId, tagIndex, tagLength!) + /** + * Process move-out event: remove matching tags from rows and delete rows with empty tag sets + */ + const processMoveOutEvent = ( + patterns: Array, + begin: () => void, + write: (message: ChangeMessageOrDeleteKeyMessage) => void, + transactionStarted: boolean, + onDelete: (rowId: RowId) => void, + ): boolean => { + if (tagLength === undefined) { + debug( + `${collectionId ? `[${collectionId}] ` : ``}Received move-out message but no tag length set yet, ignoring`, + ) + return transactionStarted } - } - - if (rowTagSet.size === 0) { - rowTagSets.delete(rowId) - return true - } - return false - } - - /** - * Process move-out event: remove matching tags from rows and delete rows with empty tag sets - */ - const processMoveOutEvent = ( - patterns: Array, - begin: () => void, - write: (message: ChangeMessageOrDeleteKeyMessage) => void, - transactionStarted: boolean, - ): boolean => { - if (tagLength === undefined) { - debug( - `${collectionId ? `[${collectionId}] ` : ``}Received move-out message but no tag length set yet, ignoring`, - ) - return transactionStarted - } + let txStarted = transactionStarted - let txStarted = transactionStarted + // Process all patterns and collect rows to delete + for (const pattern of patterns) { + // Find all rows that match this pattern + const affectedRowIds = findRowsMatchingPattern(pattern, tagIndex) - // Process all patterns and collect rows to delete - for (const pattern of patterns) { - // Find all rows that match this pattern - const affectedRowIds = findRowsMatchingPattern(pattern, tagIndex) + for (const rowId of affectedRowIds) { + if (removeMatchingTagsFromRow(rowId, pattern)) { + // Delete rows with empty tag sets + if (!txStarted) { + begin() + txStarted = true + } - for (const rowId of affectedRowIds) { - if (removeMatchingTagsFromRow(rowId, pattern)) { - // Delete rows with empty tag sets - if (!txStarted) { - begin() - txStarted = true + write({ + type: `delete`, + key: rowId, + }) + onDelete(rowId) } - - write({ - type: `delete`, - key: rowId, - }) } } - } - - return txStarted - } - /** - * Process move-in event: re-activate conditions for rows matching the patterns. - * This is a silent operation — no messages are emitted to the collection. - */ - const processMoveInEvent = (patterns: Array): void => { - if (tagLength === undefined) { - debug( - `${collectionId ? `[${collectionId}] ` : ``}Received move-in message but no tag length set yet, ignoring`, - ) - return + return txStarted } - for (const pattern of patterns) { - const affectedRowIds = findRowsMatchingPattern(pattern, tagIndex) + /** + * Process move-in event: re-activate conditions for rows matching the patterns. + * This is a silent operation — no messages are emitted to the collection. + */ + const processMoveInEvent = (patterns: Array): void => { + if (tagLength === undefined) { + debug( + `${collectionId ? `[${collectionId}] ` : ``}Received move-in message but no tag length set yet, ignoring`, + ) + return + } + + for (const pattern of patterns) { + const affectedRowIds = findRowsMatchingPattern(pattern, tagIndex) - for (const rowId of affectedRowIds) { - const activeConditions = rowActiveConditions.get(rowId) - if (activeConditions) { - activeConditions[pattern.pos] = true + for (const rowId of affectedRowIds) { + const activeConditions = rowActiveConditions.get(rowId) + if (activeConditions) { + activeConditions[pattern.pos] = true + } } } } - } - - /** - * Get the sync metadata for insert operations - * @returns Record containing relation information - */ - const getSyncMetadata = (): Record => { - // Use the stored schema if available, otherwise default to 'public' - const schema = relationSchema.state || `public` return { - relation: shapeOptions.params?.table - ? [schema, shapeOptions.params.table] - : undefined, + hasTags: () => tagLength !== undefined, + processTagsForChangeMessage, + clearTagTrackingState, + clearTagsForRow, + processMoveOutEvent, + processMoveInEvent, } } - - let unsubscribeStream: () => void + // Tags belong to a collection, and survive a compatible resume of that + // collection. Reusing an options descriptor must not share visibility. + const collectionTags = new WeakMap< + object, + ReturnType + >() return { + getSyncMetadata: () => ({ + relation: shapeOptions.params?.table + ? [relationSchema || `public`, shapeOptions.params.table] + : undefined, + }), sync: (params: Parameters[`sync`]>[0]) => { + const retainsTagState = collectionTags.has(params.collection) + let tagState = collectionTags.get(params.collection) + if (!tagState) { + tagState = createTagState() + collectionTags.set(params.collection, tagState) + } + const { + processTagsForChangeMessage, + clearTagTrackingState, + clearTagsForRow, + processMoveOutEvent, + processMoveInEvent, + } = tagState + const lifecycle = getLifecycle(params.collection) + const lifecycleEpoch = lifecycle.start() + const isActiveLifecycle = () => lifecycle.isActive(lifecycleEpoch) + Object.assign(params.collection.utils, lifecycle.utils) + const { begin, write, - commit, + commit: commitSyncTransaction, markReady, + markError, truncate, collection, metadata, } = params - const readPersistedResumeState = () => { - const persistedResumeState = metadata?.collection.get(`electric:resume`) - if (!persistedResumeState || typeof persistedResumeState !== `object`) { - return undefined - } - - const record = persistedResumeState as Record - if ( - record.kind === `resume` && - typeof record.offset === `string` && - typeof record.handle === `string` && - typeof record.shapeId === `string` && - typeof record.updatedAt === `number` - ) { - return { - kind: `resume` as const, - offset: record.offset, - handle: record.handle, - shapeId: record.shapeId, - updatedAt: record.updatedAt, - } - } - - if (record.kind === `reset` && typeof record.updatedAt === `number`) { - return { - kind: `reset` as const, - updatedAt: record.updatedAt, - } + let commitSequence = 0 + const pendingAppliedReceipts = new Map>() + const commit = (signal?: AbortSignal): SyncAppliedReceipt => { + const sequence = ++commitSequence + const applied = commitSyncTransaction(signal) + if (applied === true) { + return true } - - return undefined + pendingAppliedReceipts.set(sequence, applied) + const removeReceipt = () => pendingAppliedReceipts.delete(sequence) + void applied.then(removeReceipt, removeReceipt) + return applied + } + const waitForCommitsAfter = async (cursor: number): Promise => { + await Promise.all( + Array.from(pendingAppliedReceipts, ([sequence, applied]) => + sequence > cursor ? applied : undefined, + ), + ) + } + const readPersistedResumeState = (): ElectricResumeState | undefined => { + const persistedResumeState = metadata?.collection.get(`electric:resume`) + return parseElectricResumeState(persistedResumeState) } - const persistedResumeState = readPersistedResumeState() + const persistedMetadata = metadata as + | ElectricSyncMetadataWithHydration + | undefined + const scanPersisted = persistedMetadata?.row.scanPersisted + const whenHydrated = persistedMetadata?.row.whenHydrated + + const persistedResumeState = getNewestElectricResumeState( + readPersistedResumeState(), + lifecycle.resumeState, + ) const shapeIdentity = getStableShapeIdentity({ url: shapeOptions.url, params: shapeOptions.params as Record | undefined, @@ -1344,15 +1696,62 @@ function createElectricSync>( const hasIncompatiblePersistedResume = persistedResumeState?.kind === `resume` && persistedResumeState.shapeId !== shapeIdentity + const hasUnverifiablePersistedResume = + shapeOptions.offset === undefined && + shapeOptions.handle === undefined && + persistedResumeState?.kind === `resume` && + scanPersisted !== undefined && + whenHydrated === undefined + if (hasUnverifiablePersistedResume && !warnedUnverifiableResume) { + warnedUnverifiableResume = true + console.warn( + `Electric persistence cannot verify hydration for saved resume state. Update the persistence adapter alongside Electric to enable safe resume.`, + ) + } + const needsFullSnapshot = + shapeOptions.offset === undefined && + shapeOptions.handle === undefined && + persistedResumeState !== undefined && + (persistedResumeState.kind === `reset` || + (!retainsTagState && persistedResumeState.requiresTagState !== false)) const canUsePersistedResume = shapeOptions.offset === undefined && shapeOptions.handle === undefined && persistedResumeState?.kind === `resume` && - !hasIncompatiblePersistedResume + !hasIncompatiblePersistedResume && + !hasUnverifiablePersistedResume && + // Cached rows do not contain authoritative tag/active-condition state. + // Unknown (older) metadata is conservative; untagged shapes still resume. + !needsFullSnapshot + const hasExplicitResumeOffset = + shapeOptions.offset !== undefined && shapeOptions.offset !== `-1` + if (!canUsePersistedResume && !hasExplicitResumeOffset) { + clearTagTrackingState() + } + const receivesCompleteRows = shapeOptions.params?.replica === `full` + // Eager and progressive streams that start after the initial offset can + // only apply partial updates when the local materialization is complete. + const requiresCompleteResume = + syncMode !== `on-demand` && + (canUsePersistedResume || + (hasExplicitResumeOffset && !receivesCompleteRows)) + // A fresh snapshot replaces its hydrated cache; omitting the old offset + // alone would merge rows that no longer exist on the server. + let freshSnapshotPending = + (syncMode === `eager` || needsFullSnapshot) && + !canUsePersistedResume && + !hasExplicitResumeOffset && + whenHydrated !== undefined // Wrap markReady to wait for test hook in progressive mode let progressiveReadyGate: Promise | null = null - const wrappedMarkReady = (isBuffering: boolean) => { + let streamErrorVersion = 0 + const wrappedMarkReady = ( + isBuffering: boolean, + expectedErrorVersion = streamErrorVersion, + ) => { + if (streamErrorVersion !== expectedErrorVersion) return + // Only create gate if we're in buffering phase (first up-to-date) if ( isBuffering && @@ -1362,7 +1761,9 @@ function createElectricSync>( // Create a new gate promise for this sync cycle progressiveReadyGate = testHooks.beforeMarkingReady() progressiveReadyGate.then(() => { - markReady() + if (streamErrorVersion === expectedErrorVersion) { + markReady() + } }) } else { // No hook, not buffering, or already past first up-to-date @@ -1372,44 +1773,36 @@ function createElectricSync>( // Abort controller for the stream - wraps the signal if provided const abortController = new AbortController() + const forwardExternalAbort = () => abortController.abort() if (shapeOptions.signal) { - shapeOptions.signal.addEventListener( - `abort`, - () => { - abortController.abort() - }, - { - once: true, - }, - ) + shapeOptions.signal.addEventListener(`abort`, forwardExternalAbort, { + once: true, + }) if (shapeOptions.signal.aborted) { abortController.abort() } } - // Cleanup pending matches on abort abortController.signal.addEventListener(`abort`, () => { - pendingMatches.setState((current) => { - current.forEach((match) => { - clearTimeout(match.timeoutId) - match.reject(new StreamAbortedError()) - }) - return new Map() // Clear all pending matches - }) + lifecycle.retire(lifecycleEpoch) }) const stream = new ShapeStream({ ...shapeOptions, - // In on-demand mode, we only want to sync changes, so we set the log to `changes_only` - log: syncMode === `on-demand` ? `changes_only` : undefined, + // Recovery needs a complete snapshot even for on-demand shapes. + // Normal on-demand startup still subscribes to changes only. + log: + syncMode === `on-demand` && !needsFullSnapshot + ? `changes_only` + : undefined, // In on-demand mode, we only need the changes from the point of time the collection was created // so we default to `now` when there is no saved offset. offset: shapeOptions.offset ?? (canUsePersistedResume ? (persistedResumeState.offset as Offset) - : syncMode === `on-demand` + : syncMode === `on-demand` && !needsFullSnapshot ? `now` : undefined), handle: @@ -1417,19 +1810,23 @@ function createElectricSync>( (canUsePersistedResume ? persistedResumeState.handle : undefined), signal: abortController.signal, onError: (errorParams) => { - // Just immediately mark ready if there's an error to avoid blocking - // apps waiting for `.preload()` to finish. + streamErrorVersion++ // Note that Electric sends a 409 error on a `must-refetch` message, but the // ShapeStream handled this and it will not reach this handler, therefor - // this markReady will not be triggers by a `must-refetch`. - markReady() + // this handler will not run for a `must-refetch`. + const initialSyncFailed = collection.status === `loading` + if (initialSyncFailed) { + markError(errorParams) + } if (shapeOptions.onError) { return shapeOptions.onError(errorParams) } else { console.error( `An error occurred while syncing collection: ${collection.id}, \n` + - `it has been marked as ready to avoid blocking apps waiting for '.preload()' to finish. \n` + + (initialSyncFailed + ? `the initial sync has been marked as failed. \n` + : `the last ready snapshot has been preserved. \n`) + `You can provide an 'onError' handler on the shapeOptions to handle this error, and this message will not be logged.`, errorParams, ) @@ -1441,12 +1838,23 @@ function createElectricSync>( let transactionStarted = false const newTxids = new Set() const newSnapshots: Array = [] - let hasReceivedUpToDate = false // Track if we've completed initial sync in progressive mode + // Track if we've completed initial sync in progressive mode. A persisted + // resume starts from an already-committed stream offset, so the next + // up-to-date message must not run the initial atomic swap again. + let hasReceivedUpToDate = + syncMode === `progressive` && requiresCompleteResume + // A must-refetch starts a new snapshot generation. Until its up-to-date + // commit is applied, old Collection keys cannot make an update valid and + // the durable resume marker must remain reset. + let isResettingSnapshot = false + let resetGeneration = 0 // Progressive mode state // Helper to determine if we're buffering the initial sync const isBufferingInitialSync = () => - syncMode === `progressive` && !hasReceivedUpToDate + syncMode === `progressive` && + !hasReceivedUpToDate && + !isResettingSnapshot const bufferedMessages: Array> = [] // Buffer change messages during initial sync // Track keys that have been synced to handle overlapping subset queries. @@ -1454,9 +1862,10 @@ function createElectricSync>( // for each response. We convert subsequent inserts to updates to avoid // duplicate key errors when the row's data has changed between requests. const syncedKeys = new Set() + let resumeInvalid = false const stageResumeMetadata = () => { - if (!metadata) { + if (!isActiveLifecycle() || resumeInvalid) { return } const shapeHandle = stream.shapeHandle @@ -1465,29 +1874,37 @@ function createElectricSync>( return } - metadata.collection.set(`electric:resume`, { + const resumeState: ElectricResumeState = { kind: `resume`, offset: lastOffset, handle: shapeHandle, shapeId: shapeIdentity, updatedAt: Date.now(), - }) + requiresTagState: tagState.hasTags(), + } + lifecycle.resumeState = resumeState + metadata?.collection.set(`electric:resume`, resumeState) } const commitResetResumeMetadataImmediately = () => { - if (!metadata) { - return - } - - begin({ immediate: true }) - metadata.collection.set(`electric:resume`, { + const resetState: ElectricResumeState = { kind: `reset`, updatedAt: Date.now(), - }) - commit() + } + lifecycle.resumeState = resetState + + if (metadata) { + begin({ immediate: true }) + metadata.collection.set(`electric:resume`, resetState) + commit() + } } - if (hasIncompatiblePersistedResume) { + if ( + hasIncompatiblePersistedResume || + hasUnverifiablePersistedResume || + (needsFullSnapshot && persistedResumeState.kind === `resume`) + ) { commitResetResumeMetadataImmediately() } @@ -1557,6 +1974,8 @@ function createElectricSync>( begin, write, commit, + getCommitCursor: () => commitSequence, + waitForCommitsAfter, collectionId, // Pass the columnMapper's encode function to transform column names // (e.g., camelCase to snake_case) when compiling SQL for subset queries @@ -1565,32 +1984,85 @@ function createElectricSync>( signal: abortController.signal, }) - unsubscribeStream = stream.subscribe((messages: Array>) => { + const resumeKeysPromise = + requiresCompleteResume || freshSnapshotPending + ? whenHydrated?.() + : undefined + let areResumeKeysReady = !resumeKeysPromise + const pendingResumeBatches: Array>> = [] + let unsubscribeStream: () => void = () => {} + + const invalidateResume = () => { + resumeInvalid = true + if (transactionStarted) { + const cancellation = new AbortController() + cancellation.abort() + commit(cancellation.signal) + transactionStarted = false + } + syncedKeys.clear() + newTxids.clear() + newSnapshots.length = 0 + commitResetResumeMetadataImmediately() + streamErrorVersion++ + unsubscribeStream() + abortController.abort() + markError( + new Error( + `Electric resume state referenced an unseen row; a full snapshot is required`, + ), + ) + } + + const processMessages = (messages: Array>): void => { + if (!isActiveLifecycle() || resumeInvalid) { + return + } + + if (freshSnapshotPending) { + freshSnapshotPending = false + begin() + transactionStarted = true + truncate() + syncedKeys.clear() + clearTagTrackingState() + isResettingSnapshot = true + resetGeneration++ + } + + // Applied rows can also arrive through persistence invalidations. + // Overlay only unapplied writes, once per callback rather than once + // per message. A queued truncate fences off the previous snapshot. + const pendingPresence = new Map() + let usesBaseline = true + for (const pending of collection._state.pendingSyncedTransactions) { + if (pending.truncate) { + pendingPresence.clear() + usesBaseline = false + } + for (const operation of pending.operations) { + pendingPresence.set(operation.key, operation.type !== `delete`) + } + } + for (const message of bufferedMessages) { + if (isChangeMessage(message)) { + pendingPresence.set( + collection.getKeyFromItem(message.value), + message.headers.operation !== `delete`, + ) + } + } + // Track commit point type - up-to-date takes precedence as it also triggers progressive mode atomic swap let commitPoint: `up-to-date` | `subset-end` | null = null - // Don't clear the buffer between batches - this preserves messages for awaitMatch - // to find even if multiple batches arrive before awaitMatch is called. - // The buffer is naturally limited by MAX_BATCH_MESSAGES (oldest messages are dropped). - // Reset batchCommitted since we're starting a new batch - batchCommitted.setState(() => false) + lifecycle.beginMatchGeneration(messages) for (const message of messages) { - // Add message to current batch buffer (for race condition handling) - if ( - isChangeMessage(message) || - isMoveOutMessage(message) || - isMoveInMessage(message) - ) { - currentBatchMessages.setState((currentBuffer) => { - const newBuffer = [...currentBuffer, message] - // Limit buffer size for safety - if (newBuffer.length > MAX_BATCH_MESSAGES) { - newBuffer.splice(0, newBuffer.length - MAX_BATCH_MESSAGES) - } - return newBuffer - }) - } + lifecycle.observeMatchMessage(message) + // A match predicate can synchronously clean up and restart sync. + // Nothing after that boundary belongs to the replacement session. + if (!isActiveLifecycle()) return // Check for txids in the message and add them to our store // Skip during buffered initial sync in progressive mode (txids will be extracted during atomic swap) @@ -1603,34 +2075,31 @@ function createElectricSync>( message.headers.txids?.forEach((txid) => newTxids.add(txid)) } - // Check pending matches against this message - // Note: matchFn will mark matches internally, we don't resolve here - const matchesToRemove: Array = [] - pendingMatches.state.forEach((match, matchId) => { - if (!match.matched) { - try { - match.matchFn(message) - } catch (err) { - // If matchFn throws, clean up and reject the promise - clearTimeout(match.timeoutId) - match.reject( - err instanceof Error ? err : new Error(String(err)), - ) - matchesToRemove.push(matchId) - debug(`matchFn error: %o`, err) + if (isChangeMessage(message)) { + const rowId = collection.getKeyFromItem(message.value) + const operation = message.headers.operation + const hasKnownRow = + pendingPresence.get(rowId) ?? + (usesBaseline && collection._state.syncedData.has(rowId)) + if (operation === `update` && !hasKnownRow) { + // Validate after all earlier events, including tag move-outs. + // Cancel staged writes before publishing any part of an invalid + // resumed callback; the next lifecycle must take a full snapshot. + if (requiresCompleteResume && !isResettingSnapshot) { + invalidateResume() + return } + if (!receivesCompleteRows) continue } - }) - - // Remove matches that errored - removePendingMatches(matchesToRemove) + pendingPresence.set(rowId, operation !== `delete`) + } if (isChangeMessage(message)) { // Check if the message contains schema information const schema = message.headers.schema if (schema && typeof schema === `string`) { // Store the schema for future use if it's a valid string - relationSchema.setState(() => schema) + relationSchema = schema } // In buffered initial sync of progressive mode, buffer messages instead of writing @@ -1676,6 +2145,10 @@ function createElectricSync>( begin, write, transactionStarted, + (rowId) => { + pendingPresence.set(rowId, false) + syncedKeys.delete(rowId) + }, ) } } else if (isMoveInMessage(message)) { @@ -1706,6 +2179,10 @@ function createElectricSync>( // Clear synced keys tracking since we're starting fresh syncedKeys.clear() + pendingPresence.clear() + usesBaseline = false + isResettingSnapshot = true + resetGeneration++ // Reset the loadSubset deduplication state since we're starting fresh // This ensures that previously loaded predicates don't prevent refetching after truncate @@ -1718,7 +2195,20 @@ function createElectricSync>( } } + // A subset completion cannot publish a partial cold-recovery snapshot. + if ( + needsFullSnapshot && + isResettingSnapshot && + commitPoint === `subset-end` + ) + return + if (commitPoint !== null) { + let applied: SyncAppliedReceipt = true + const wasBufferingInitialSync = isBufferingInitialSync() + const finishesReset = + isResettingSnapshot && commitPoint === `up-to-date` + const finishingResetGeneration = resetGeneration // PROGRESSIVE MODE: Atomic swap on first up-to-date (not subset-end) // EXCEPTION: Skip atomic swap if a transaction is already started (e.g., from must-refetch). // In that case, do a normal commit to properly close the existing transaction. @@ -1763,7 +2253,13 @@ function createElectricSync>( bufferedMsg.headers.patterns, begin, write, - transactionStarted, + // The swap already opened a transaction, even though the + // normal-stream transactionStarted flag is still false. + true, + (rowId) => { + pendingPresence.set(rowId, false) + syncedKeys.delete(rowId) + }, ) } else if (isMoveInMessage(bufferedMsg)) { // Process buffered move-in messages during atomic swap @@ -1773,7 +2269,7 @@ function createElectricSync>( // Commit the atomic swap stageResumeMetadata() - commit() + applied = commit() // Exit buffering phase by marking that we've received up-to-date // isBufferingInitialSync() will now return false @@ -1786,73 +2282,119 @@ function createElectricSync>( // Normal mode or on-demand: commit transaction if one was started // Both up-to-date and subset-end trigger a commit if (transactionStarted) { - stageResumeMetadata() - commit() + if (!isResettingSnapshot || finishesReset) { + stageResumeMetadata() + } + applied = commit() transactionStarted = false } else if (commitPoint === `up-to-date` && metadata) { begin() stageResumeMetadata() - commit() + applied = commit() + } + } + const readyErrorVersion = streamErrorVersion + if (applied === true) { + wrappedMarkReady(wasBufferingInitialSync, readyErrorVersion) + } else { + void applied.then( + () => + wrappedMarkReady(wasBufferingInitialSync, readyErrorVersion), + () => undefined, + ) + } + + if (finishesReset) { + const finishReset = () => { + if (resetGeneration === finishingResetGeneration) { + isResettingSnapshot = false + } + } + if (applied === true) { + finishReset() + } else { + void applied.then(finishReset, () => undefined) } } - wrappedMarkReady(isBufferingInitialSync()) // Track that we've received the first up-to-date for progressive mode if (commitPoint === `up-to-date`) { hasReceivedUpToDate = true } - // Always commit txids when we receive up-to-date, regardless of transaction state - seenTxids.setState((currentTxids) => { - const clonedSeen = new Set(currentTxids) - if (newTxids.size > 0) { - debug( - `${collectionId ? `[${collectionId}] ` : ``}new txids synced from pg %O`, - Array.from(newTxids), - ) - } - newTxids.forEach((txid) => clonedSeen.add(txid)) - newTxids.clear() - return clonedSeen - }) - - // Always commit snapshots when we receive up-to-date, regardless of transaction state - seenSnapshots.setState((currentSnapshots) => { - const seen = [...currentSnapshots, ...newSnapshots] - newSnapshots.forEach((snapshot) => - debug( - `${collectionId ? `[${collectionId}] ` : ``}new snapshot synced from pg %o`, - snapshot, - ), + // Stream evidence is the acknowledgement boundary used by mutation + // handlers. It must publish before a parked applied receipt or the + // optimistic transaction and its acknowledgement can deadlock. + if (newTxids.size > 0) { + debug( + `${collectionId ? `[${collectionId}] ` : ``}new txids synced from pg %O`, + Array.from(newTxids), ) - newSnapshots.length = 0 - return seen - }) - - // Resolve all matched pending matches on up-to-date or subset-end - // Set batchCommitted BEFORE resolving to avoid timing window where late awaitMatch - // calls could register as "matched" after resolver pass already ran - batchCommitted.setState(() => true) + } + newSnapshots.forEach((snapshot) => + debug( + `${collectionId ? `[${collectionId}] ` : ``}new snapshot synced from pg %o`, + snapshot, + ), + ) + lifecycle.publishEvidence(newTxids, newSnapshots) + newTxids.clear() + newSnapshots.length = 0 + lifecycle.commitMatches() + } + } - resolveMatchedPendingMatches() + unsubscribeStream = stream.subscribe((messages: Array>) => { + if (!areResumeKeysReady) { + pendingResumeBatches.push([...messages]) + return } + processMessages(messages) }) + if (!areResumeKeysReady && resumeKeysPromise) { + void resumeKeysPromise.then( + () => { + if (abortController.signal.aborted) return + + areResumeKeysReady = true + + const queuedBatches = pendingResumeBatches.splice(0) + queuedBatches.forEach(processMessages) + }, + (error: unknown) => { + if (abortController.signal.aborted) return + + pendingResumeBatches.length = 0 + resumeInvalid = true + commitResetResumeMetadataImmediately() + streamErrorVersion++ + unsubscribeStream() + abortController.abort() + markError(error) + }, + ) + } + // Return the deduplicated loadSubset if available (on-demand or progressive mode) // The loadSubset method is auto-bound, so it can be safely returned directly return { loadSubset: loadSubsetDedupe?.loadSubset, cleanup: () => { + shapeOptions.signal?.removeEventListener( + `abort`, + forwardExternalAbort, + ) // Unsubscribe from the stream unsubscribeStream() // Abort the abort controller to stop the stream abortController.abort() + pendingResumeBatches.length = 0 // Reset deduplication tracking so collection can load fresh data if restarted loadSubsetDedupe?.reset() + lifecycle.retire(lifecycleEpoch) }, } }, - // Expose the getSyncMetadata function - getSyncMetadata, } } diff --git a/packages/electric-db-collection/tests/ORACLE_MUTATIONS.md b/packages/electric-db-collection/tests/ORACLE_MUTATIONS.md new file mode 100644 index 0000000000..ddb1b16e7a --- /dev/null +++ b/packages/electric-db-collection/tests/ORACLE_MUTATIONS.md @@ -0,0 +1,217 @@ +# Electric oracle mutation ledger + +Run each mutation alone from `packages/electric-db-collection`, confirm the +named test fails, then restore the source before trying the next mutation. +The baseline command is: + +```sh +pnpm test:oracles +``` + +## 1. Collection-local evidence + +In `src/electric.ts`, replace `consumeDescriptorLifecycle()` with +`descriptorLifecycle` when a bound sync is created. + +Killed by: `binds sync metadata import and export to the receiving collection`. The generated process grammar creates fresh descriptors; it +does not prove shared-descriptor isolation. The descriptor-isolation suite +also crosses raw/once-spread reuse, eager/lazy startup, equal keys, and peer +reset/cleanup. + +## 2. Stale callback isolation + +In `src/electric.ts`, remove the active-lifecycle term from either +`processMessages` lifecycle guard. + +Killed by: `settles every startup, hydration, snapshot availability, commit, and cleanup permutation` and `keeps stream cleanup and stale callbacks scoped to their lifecycle`. + +## 3. Acknowledgement liveness + +In `src/electric.ts`, delay `seenTxids`, `seenSnapshots`, and matched-message +publication until a pending applied receipt resolves. + +Killed by: `txid tracking > should simulate the complete flow` and the +direct-persistence-handler flow tests. Those handlers must receive stream +acknowledgement before the parked optimistic transaction can finish. + +## 4. Durable convergence + +In `runPersistedTrace` in `electric-oracle.property.test.ts`, make the +wrapped `applyCommittedTx` resolve without calling the saved adapter method. + +Killed by: `denotational reference, Electric, persisted Electric, and query adapters converge across controls and publication epochs`. + +## 5. Late match evidence + +Clear committed match messages when the next change-bearing batch starts. + +Killed by: `keeps committed match evidence across newer writer batches`. +The paired `clears committed match evidence when the stream must refetch` +test prevents the opposite error of retaining evidence across a reset. + +## 6. Applied baseline and pending presence + +In `processMessages`, remove the `syncedData.has(rowId)` fallback from +`hasKnownRow` (replace the fallback with `false`). + +Killed by: `independent persistence publications and stream deltas agree with complete-row state` +and `applies on-demand catch-up updates to hydrated persisted rows`. The new +history generator publishes complete rows through the actual persistence +coordinator, independently of Electric events or subset acquisition. It crosses +targeted/full reload, insertion/removal, and later partial updates. A six-cell +mode × publication-path matrix also checks public and durable rows. + +Conversely, replace `hasKnownRow` with `collection._state.syncedData.has(rowId)`, +ignoring pending-presence overrides. + +Killed by: `keeps $removal removal authoritative across an optimistic write and a new acquisition` +for delete and move-out. The old public row is still visible while its removal +is parked; a subsequent partial update must not resurrect it. The reset control +still passes because truncation drains immediately. The generated acquisition +histories and nine-cell reset/delete/move-out × acquisition-timing matrix remain. +These execute real acquisitions; a subset-end marker is not an acquisition. + +No retained full-key index is needed. `subset acquisition avoids scanning the applied baseline with $n rows` +counts key iteration for both new and deduplicated acquisitions. + +## 7. Resume capability fencing + +Accept a persisted offset when `scanPersisted` exists but `whenHydrated` +does not. + +Killed by: `warns once and restarts a persisted resume when hydration completion is unavailable`. +The restart control also verifies that the compatibility warning is not repeated. + +## 8. Complete-row discrimination + +Treat every resumed `update` as a partial row, including updates from a +`replica: 'full'` stream. + +Killed by: `accepts complete replica updates from an explicit eager resume`. +The paired `rejects an unseen partial update from an explicit eager resume` +test proves that the exception does not admit partial rows. + +## 9. Authoritative fresh recovery + +Remove the truncate from `freshSnapshotPending` startup. + +Killed by the eager cells in `electric-recovery-oracle.test.ts`: persisted +rows omitted from a fresh empty/nonempty snapshot must disappear from both +public and durable state, regardless of hydration/callback order. Normal +resume controls still retain unchanged cached rows. + +## 10. Resumed tag-removal validation + +Ignore unknown resumed updates instead of calling `invalidateResume()`. + +Killed by `generated invalid resume transitions fail under every batch partition`, which crosses delete/move-out with eager/progressive streams and +checks retained rows, error state, and reset metadata together. + +## 11. Tag ownership across collection reuse and restart + +Share one tag tracker between collections, or recreate it on every sync +session instead of retaining it for a compatible same-collection resume. + +Killed by `electric-descriptor-isolation.test.ts`: equal-key peer streams +cannot remove each other's rows, and compatible persisted restart must +still apply move-outs to retained tagged rows. The fresh-restart control +proves old tags do not leak into a new snapshot. + +Sharing the original factory-bound utilities instead of copying them is killed +by `keeps insert acknowledgements on the owner of a reused persisted descriptor`: +an actual insert must settle from its own stream even after a peer starts. + +## 12. SDK reset framing + +In `isSdkResetFramedPartition`, allow resets to share a callback with data, or +validate only the first reset. Killed by +`keeps SDK reset callbacks separate from every neighboring message kind`: +seven message kinds cross both sides of the reset, with split controls. +The previous predicate rejected only commit-before-reset; it incorrectly +accepted data-before-reset and reset-before-data in the protocol model. + +`electric-sdk-framing.test.ts` independently uses the real SDK with controlled +HTTP responses. Both normal and stale-row-bearing 409 bodies produce singleton +reset callbacks, as the generated protocol histories now require. This pins the +installed SDK's HTTP reset path, not an exhaustive specification of every possible +server response or SSE path. + +The synthetic partition property remains as extra adapter robustness coverage; +it deliberately tests more callback shapes than the SDK-framed differential +properties. Those shapes are not evidence of a reachable protocol regression. + +## 13. Progressive snapshot transaction ownership + +Pass `transactionStarted` instead of `true` when applying a buffered move-out +during the initial progressive atomic swap. That normal-stream flag is false, +but the swap has already called `begin()`. Opening another transaction strands +the first truncate. Later presence checks then reject valid live updates. + +Killed by `preserves live updates after ... initial tagged move-outs` and +`generated initial tagged move-outs preserve later live updates`. The matrix +crosses all three modes with zero, one and repeated matching move-outs, then +delivers a new insert and a partial update after the initial up-to-date. Every +contiguous callback partition is tested. Expected rows follow tag membership +and ordinary row updates, without consulting adapter transaction state. + +The bug was RED in two progressive fixed cases and the random property (seed +`-1632249566`, path `0:1`); seven fixed controls passed. Previous tag tests began +after readiness, while the initial-sync generators lacked tag move-outs followed +by later live work. This was also reproduced by the full service-backed Electric +E2E suite, not only a synthetic callback driver. + +Use this focused form while iterating: + +```sh +pnpm exec vitest run tests/electric-oracle.property.test.ts -t '' +``` + +## 14. Owner, durable membership, and callback boundaries + +The descriptor-isolation histories now derive descriptors from original options, +once-spread options, and an existing collection's `config`. They vary startup, +peer edits, and which peer is retired. Each owner's public rows and cleanup +counts must remain independent. Nesting an already-bound sync fails this law: +the original owner stays ready but stops receiving updates. The binding guard +keeps ownership with the outermost wrapper while preserving source delegation; +removing that guard makes the generated law fail again. + +The persisted-tag histories compare row/tag sets against real adapter execution +across warm restart and cold recreation, each with resume and fresh-snapshot +controls. Generated updates either preserve tags or replace membership; generated +move-outs remove membership until rows disappear. The durable fixture copies +values and applies row and collection metadata mutations, rather than preserving +in-memory object references. Cold resume failed because cached values and an +offset survived, but membership did not. Later untagged updates matter because +their last-message headers cannot reconstruct earlier tag membership. + +The chosen recovery contract refetches a full snapshot when a cold start lacks +required membership, including older unknown metadata. The generator crosses +tagged/untagged histories, legacy/current metadata, all three sync modes, and +interruption during replacement. It checks cached rows before the final commit, +omitted rows afterward, subsequent move-outs, and public/durable agreement. +Lazy modes make an actual subset acquisition to hydrate cached rows. Existing +untagged resume fixtures explicitly declare that they do not need tag state; +the legacy cells retain coverage for missing metadata. A subset-end during cold +recovery must not publish the incomplete replacement. + +The callback-reentry histories retire a session either before a stale callback +or inside an `awaitMatch` predicate at a generated row position. Only the +replacement stream may acknowledge new-session waiters. Generated message tails +must not cross that boundary. The law also registers a replacement waiter inside +the restart callback: the old match iteration must not visit it. Epoch guards +after user callbacks fence both the waiter loop and the remaining message batch. + +All three extensions have fixed-seed and random properties, with the shared +oracle multiplier and seed/path replay controls. These are ordinary assertions, +not expected-failure classifiers. At `049cc9a5d`, fixed seeds `42711`, `42712`, +and `42713` reproduce the three failures respectively. Binding and reentry use +epoch/ownership guards; persisted membership uses the approved refetch contract. + +The separate mixed `[insert, must-refetch, up-to-date]` report exposed the +protocol-model gap corrected in section 12. It remains outside the verified +generated domain. A production fix would still need evidence of a conforming +server/SDK path that delivers that mixed callback. + +These mutants test the named laws. They do not claim exhaustive mutation +coverage of the package. diff --git a/packages/electric-db-collection/tests/electric-descriptor-isolation.test.ts b/packages/electric-db-collection/tests/electric-descriptor-isolation.test.ts new file mode 100644 index 0000000000..ca179f92f4 --- /dev/null +++ b/packages/electric-db-collection/tests/electric-descriptor-isolation.test.ts @@ -0,0 +1,672 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { beforeEach, describe, expect, it, vi } from 'vitest' +import { createCollection } from '@tanstack/db' +import { ShapeStream } from '@electric-sql/client' +import { persistedCollectionOptions } from '../../db-sqlite-persistence-core/src' +import { electricCollectionOptions } from '../src/electric' +import { oraclePropertyOptions, oracleRuns } from '../../db/tests/oracle-config' +import type { Message } from '@electric-sql/client' +import type { PersistenceAdapter } from '../../db-sqlite-persistence-core/src' +import type { ElectricCollectionUtils } from '../src/electric' + +type TestRow = { id: number; name: string; stable: string } +type StreamHarness = { + send: (messages: Array>) => void + unsubscribe: ReturnType +} + +const streams: Array = [] + +vi.mock(`@electric-sql/client`, async () => { + const actual = await vi.importActual(`@electric-sql/client`) + return { + ...actual, + ShapeStream: vi.fn(() => { + const unsubscribe = vi.fn() + return { + subscribe: (send: StreamHarness[`send`]) => { + streams.push({ send, unsubscribe }) + return unsubscribe + }, + requestSnapshot: vi.fn().mockResolvedValue(undefined), + fetchSnapshot: vi.fn().mockResolvedValue({ metadata: {}, data: [] }), + isUpToDate: false, + shapeHandle: `shape-current`, + lastOffset: `20_0`, + } + }), + } +}) + +const upToDate: Message = { headers: { control: `up-to-date` } } +const mustRefetch: Message = { headers: { control: `must-refetch` } } + +function insert(id: number, tag: string): Message { + return { + key: String(id), + value: { id, name: tag, stable: `stable-${id}` }, + headers: { + operation: `insert`, + tags: [tag], + }, + } +} + +function moveOut(tag: string): Message { + return { headers: { event: `move-out`, patterns: [{ pos: 0, value: tag }] } } +} + +function descriptor( + form: `original` | `once-spread`, + startSync = true, + syncMode: `eager` | `on-demand` | `progressive` = `eager`, +) { + const options = electricCollectionOptions({ + shapeOptions: { url: `http://test-url`, params: { table: `test_table` } }, + getKey: (row) => row.id, + startSync, + syncMode, + }) + // Spreading once consumes the options creator's utils getter. Reusing this + // plain descriptor must be as safe as reading that getter for each instance. + return form === `once-spread` ? { ...options } : options +} + +function tagPersistence() { + const rows = new Map< + string | number, + { value: TestRow; metadata?: unknown } + >() + const metadata = new Map() + const adapter: PersistenceAdapter = { + loadSubset: () => + Promise.resolve( + Array.from(rows, ([key, row]) => ({ key, ...structuredClone(row) })), + ), + loadCollectionMetadata: () => + Promise.resolve( + Array.from(metadata, ([key, value]) => ({ + key, + value: structuredClone(value), + })), + ), + applyCommittedTx: (_id, transaction) => { + if (transaction.truncate) rows.clear() + for (const mutation of transaction.mutations) { + if (mutation.type === `delete`) rows.delete(mutation.key) + else + rows.set(mutation.key, { + value: { + ...rows.get(mutation.key)?.value, + ...structuredClone(mutation.value), + } as TestRow, + metadata: structuredClone( + mutation.metadata ?? rows.get(mutation.key)?.metadata, + ), + }) + } + for (const mutation of transaction.rowMetadataMutations ?? []) { + const row = rows.get(mutation.key) + if (row) + row.metadata = + mutation.type === `delete` + ? undefined + : structuredClone(mutation.value) + } + for (const mutation of transaction.collectionMetadataMutations ?? []) { + if (mutation.type === `delete`) metadata.delete(mutation.key) + else metadata.set(mutation.key, structuredClone(mutation.value)) + } + return Promise.resolve() + }, + ensureIndex: () => Promise.resolve(), + } + return { rows, metadata, adapter } +} + +const tagHistory = fc.record({ + syncMode: fc.constantFrom( + `eager` as const, + `on-demand` as const, + `progressive` as const, + ), + tagged: fc.boolean(), + legacyResume: fc.boolean(), + interruptRecovery: fc.boolean(), + edits: fc.array( + fc.record({ + id: fc.integer({ min: 1, max: 3 }), + renameOnly: fc.boolean(), + tag: fc.constantFrom(`left`, `right`, `other`), + }), + { maxLength: 6 }, + ), + removals: fc.shuffledSubarray([`left`, `right`, `other`], { + minLength: 3, + maxLength: 3, + }), +}) + +async function runTagHistory(history: { + syncMode: `eager` | `on-demand` | `progressive` + tagged: boolean + legacyResume: boolean + interruptRecovery: boolean + edits: Array<{ id: number; renameOnly: boolean; tag: string }> + removals: Array +}) { + for (const cold of [false, true]) { + for (const fresh of [true, false]) { + const start = streams.length + const { rows, metadata, adapter } = tagPersistence() + const model = new Map( + [1, 2, 3].map((id) => [ + id, + { + row: { id, name: `row-${id}`, stable: `stable-${id}` }, + tags: new Set( + id === 1 ? [`left`] : id === 2 ? [`right`] : [`left`, `right`], + ), + }, + ]), + ) + const create = () => + createCollection( + persistedCollectionOptions< + TestRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...descriptor(`original`, false, history.syncMode), + id: `persisted-tag-history`, + persistence: { adapter }, + }), + ) + const first = create() + let current = first + const expectedRows = () => [...model.values()].map((entry) => entry.row) + const publicRows = () => + [...current.values()].map(({ id, name, stable }) => ({ + id, + name, + stable, + })) + const durableRows = () => [...rows.values()].map((entry) => entry.value) + const check = async () => { + expect(publicRows(), `cold=${cold}, fresh=${fresh}`).toEqual( + expectedRows(), + ) + await vi.waitFor(() => expect(durableRows()).toEqual(expectedRows()), { + interval: 1, + }) + } + const snapshot = (): Array> => + [...model.values()].map(({ row, tags }) => ({ + key: String(row.id), + value: { ...row }, + headers: { + operation: `insert`, + ...(history.tagged && { tags: [...tags] }), + }, + })) + try { + first.startSyncImmediate() + await vi.waitFor(() => expect(streams).toHaveLength(start + 1), { + interval: 1, + }) + streams[start]!.send([...snapshot(), upToDate]) + await check() + for (const [step, edit] of history.edits.entries()) { + const entry = model.get(edit.id)! + const previousTags = [...entry.tags] + entry.row = { ...entry.row, name: `edit-${step}` } + if (!edit.renameOnly) entry.tags = new Set([edit.tag]) + streams[start]!.send([ + { + key: String(edit.id), + value: { ...entry.row }, + headers: { + operation: `update`, + ...(history.tagged && + !edit.renameOnly && { + tags: [edit.tag], + removed_tags: previousTags.filter( + (tag) => tag !== edit.tag, + ), + }), + }, + }, + upToDate, + ]) + await check() + } + await vi.waitFor( + () => + expect(metadata.get(`electric:resume`)).toMatchObject({ + kind: `resume`, + }), + { interval: 1 }, + ) + await first.cleanup() + if (history.legacyResume) { + const resume = { + ...(metadata.get(`electric:resume`) as Record), + } + delete resume.requiresTagState + metadata.set(`electric:resume`, resume) + } + if (fresh) + metadata.set(`electric:resume`, { + kind: `reset`, + updatedAt: Date.now() + 1, + }) + if (cold) current = create() + current.startSyncImmediate() + await vi.waitFor(() => expect(streams).toHaveLength(start + 2), { + interval: 1, + }) + // Lazy persistence hydrates cached rows only when a consumer acquires + // a subset. The stream alone does not materialize that cache. + if (history.syncMode !== `eager`) await current._sync.loadSubset({}) + await vi.waitFor(() => expect(publicRows()).toEqual(expectedRows()), { + interval: 1, + }) + const rebuild = + fresh || (cold && (history.tagged || history.legacyResume)) + let resumedStream = streams[start + 1]! + expect(vi.mocked(ShapeStream).mock.calls[start + 1]?.[0]).toMatchObject( + { offset: rebuild ? undefined : `20_0` }, + ) + if (rebuild) { + const cachedRows = expectedRows() + // Replacement omits a cached row. Its partial delivery must not + // expose a torn snapshot or erase the still-visible cached rows. + model.delete(2) + for (const entry of model.values()) entry.tags = new Set([`fresh`]) + resumedStream.send(snapshot()) + expect(publicRows()).toEqual(cachedRows) + // A concurrent subset request finishing is not completion of the + // full replacement snapshot used to recover cold membership. + if (!fresh && cold && (history.tagged || history.legacyResume)) { + resumedStream.send([{ headers: { control: `subset-end` } }]) + expect(publicRows()).toEqual(cachedRows) + } + if (history.interruptRecovery) { + await current.cleanup() + current.startSyncImmediate() + await vi.waitFor(() => expect(streams).toHaveLength(start + 3), { + interval: 1, + }) + if (history.syncMode !== `eager`) await current._sync.loadSubset({}) + await vi.waitFor(() => expect(publicRows()).toEqual(cachedRows), { + interval: 1, + }) + expect( + vi.mocked(ShapeStream).mock.calls[start + 2]?.[0], + ).toMatchObject({ offset: undefined }) + resumedStream = streams[start + 2]! + resumedStream.send(snapshot()) + } + resumedStream.send([upToDate]) + await check() + } + for (const tag of history.tagged + ? [...history.removals, `fresh`] + : []) { + // Membership is a set law, independent of Electric's tag index. + for (const [id, entry] of model) { + entry.tags.delete(tag) + if (entry.tags.size === 0) model.delete(id) + } + resumedStream.send([moveOut(tag), upToDate]) + await check() + } + } finally { + await current.cleanup() + if (current !== first) await first.cleanup() + } + } + } +} + +fcTest.prop([tagHistory], { seed: 42712, numRuns: oracleRuns(6) })( + `persisted tag histories preserve membership across warm and cold restart (fixed)`, + runTagHistory, +) +fcTest.prop( + [tagHistory], + oraclePropertyOptions(10, `electric.persisted-tag-history`), +)( + `persisted tag histories preserve membership across warm and cold restart (random)`, + runTagHistory, +) + +beforeEach(() => { + streams.length = 0 + vi.clearAllMocks() +}) + +const ownerHistory = fc.record({ + startSync: fc.boolean(), + retire: fc.integer({ min: 0, max: 1 }), + edits: fc.array( + fc.record({ + owner: fc.integer({ min: 0, max: 1 }), + name: fc.string({ maxLength: 8 }), + }), + { maxLength: 8 }, + ), +}) + +async function runOwnerHistory(history: { + startSync: boolean + retire: number + edits: Array<{ owner: number; name: string }> +}) { + for (const form of [`original`, `once-spread`, `materialized`] as const) { + const start = streams.length + const options = descriptor( + form === `materialized` ? `once-spread` : form, + history.startSync, + ) + const first = createCollection({ ...options, id: `owner-first` }) + first.startSyncImmediate() + streams[start]!.send([insert(1, `first`), upToDate]) + const second = + form === `materialized` + ? createCollection({ ...first.config, id: `owner-second` }) + : createCollection({ ...options, id: `owner-second` }) + const collections = [first, second] + const expected = [`first`, `second`] + const edit = (owner: number, name: string) => { + expected[owner] = name + streams[start + owner]!.send([ + { + key: `1`, + value: { id: 1, name, stable: `stable-1` }, + headers: { operation: `update` }, + }, + upToDate, + ]) + } + const check = () => { + for (const [owner, collection] of collections.entries()) { + expect(collection.status, `${form}: owner ${owner}`).toBe(`ready`) + expect(collection.get(1)?.name, `${form}: owner ${owner}`).toBe( + expected[owner], + ) + } + } + try { + second.startSyncImmediate() + streams[start + 1]!.send([insert(1, `second`), upToDate]) + check() + // Both owners must still receive data after binding the peer, even when + // the generated edit history shrinks to empty. + edit(0, `first-updated`) + edit(1, `second-updated`) + check() + for (const { owner, name } of history.edits) { + edit(owner, name) + check() + } + await collections[history.retire]!.cleanup() + const survivor = 1 - history.retire + edit(survivor, `after-peer-cleanup`) + expect(collections[survivor]!.get(1)?.name).toBe(`after-peer-cleanup`) + expect( + streams[start + history.retire]!.unsubscribe, + ).toHaveBeenCalledOnce() + expect(streams[start + survivor]!.unsubscribe).not.toHaveBeenCalled() + } finally { + await first.cleanup() + await second.cleanup() + } + } +} + +fcTest.prop([ownerHistory], { seed: 42711, numRuns: oracleRuns(12) })( + `config derivation preserves independent owner histories (fixed)`, + runOwnerHistory, +) +fcTest.prop( + [ownerHistory], + oraclePropertyOptions(20, `electric.bound-descriptor-history`), +)( + `config derivation preserves independent owner histories (random)`, + runOwnerHistory, +) + +it(`keeps insert acknowledgements on the owner of a reused persisted descriptor`, async () => { + const adapter: PersistenceAdapter = { + loadSubset: () => Promise.resolve([]), + loadCollectionMetadata: () => Promise.resolve([]), + applyCommittedTx: () => Promise.resolve(), + ensureIndex: () => Promise.resolve(), + } + const options = persistedCollectionOptions< + TestRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricCollectionOptions({ + id: `shared-persisted-options`, + shapeOptions: { url: `http://test-url`, params: { table: `test_table` } }, + getKey: (row) => row.id, + startSync: false, + onInsert: () => Promise.resolve({ txid: 200, timeout: 100 }), + }), + persistence: { adapter }, + }) + const first = createCollection(options) + const second = createCollection(options) + try { + first.startSyncImmediate() + await vi.waitFor(() => expect(streams).toHaveLength(1)) + streams[0]!.send([upToDate]) + await vi.waitFor(() => expect(first.status).toBe(`ready`)) + second.startSyncImmediate() + await vi.waitFor(() => expect(streams).toHaveLength(2)) + streams[1]!.send([upToDate]) + await vi.waitFor(() => expect(second.status).toBe(`ready`)) + + const transaction = first.insert({ id: 1, name: `own`, stable: `stable-1` }) + void transaction.isPersisted.promise.catch(() => undefined) + streams[0]!.send([ + insert(1, `own`), + { headers: { control: `up-to-date`, txids: [200] } }, + ]) + await expect(transaction.isPersisted.promise).resolves.toBeDefined() + expect(first.get(1)?.name).toBe(`own`) + expect(second.has(1)).toBe(false) + } finally { + await first.cleanup() + await second.cleanup() + } +}) + +it.each([`resume`, `fresh`] as const)( + `restores compatible tags and discards obsolete tags on persisted $0 restart`, + async (restart) => { + const { rows, metadata, adapter } = tagPersistence() + const collection = createCollection( + persistedCollectionOptions< + TestRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...descriptor(`original`, false), + id: `tag-restart-${restart}`, + persistence: { adapter }, + }), + ) + try { + collection.startSyncImmediate() + await vi.waitFor(() => expect(streams).toHaveLength(1)) + streams[0]!.send([insert(1, `old`), upToDate]) + await vi.waitFor(() => expect(collection.status).toBe(`ready`)) + await vi.waitFor(() => + expect(metadata.get(`electric:resume`)).toMatchObject({ + kind: `resume`, + offset: `20_0`, + }), + ) + await collection.cleanup() + if (restart === `fresh`) { + metadata.set(`electric:resume`, { + kind: `reset`, + updatedAt: Date.now() + 1, + }) + } + collection.startSyncImmediate() + await vi.waitFor(() => expect(streams).toHaveLength(2)) + await vi.waitFor(() => expect(collection.get(1)?.stable).toBe(`stable-1`)) + expect(vi.mocked(ShapeStream).mock.calls[1]?.[0]).toMatchObject({ + offset: restart === `resume` ? `20_0` : undefined, + }) + const currentTag = restart === `resume` ? `old` : `current` + if (restart === `fresh`) + streams[1]!.send([insert(1, currentTag), upToDate]) + streams[1]!.send([moveOut(currentTag), upToDate]) + await vi.waitFor(() => expect(collection.status).toBe(`ready`)) + expect(collection.has(1)).toBe(false) + await vi.waitFor(() => expect(rows.has(1)).toBe(false)) + } finally { + await collection.cleanup() + } + }, +) + +describe.each([`original`, `once-spread`] as const)( + `%s Electric descriptor`, + (form) => { + it.each([false, true])( + `keeps acknowledgement helpers on their owning collection, eager=%s`, + async (startSync) => { + const options = descriptor(form, startSync) + const first = createCollection({ ...options, id: `first` }) + const second = createCollection({ ...options, id: `second` }) + const firstWait = first.utils.awaitTxId(11, 100) + const secondWait = second.utils.awaitTxId(22, 100) + // Observe rejections even if an earlier assertion fails and cleanup + // aborts a still-pending waiter. + void firstWait.catch(() => undefined) + void secondWait.catch(() => undefined) + try { + first.config.sync.importSyncMeta?.({ version: 1, seenTxids: [11] }) + second.config.sync.importSyncMeta?.({ version: 1, seenTxids: [22] }) + await expect(firstWait).resolves.toBe(true) + await expect(secondWait).resolves.toBe(true) + if (!startSync) expect(streams).toHaveLength(0) + + first.startSyncImmediate() + second.startSyncImmediate() + expect(streams).toHaveLength(2) + const peerMatch = second.utils.awaitMatch( + (message) => `value` in message && message.value.name === `first`, + 100, + ) + const ownMatch = first.utils.awaitMatch( + (message) => `value` in message && message.value.name === `first`, + 100, + ) + const ownTxid = first.utils.awaitTxId(33, 100) + const peerTxid = second.utils.awaitTxId(33, 100) + let peerMatched = false + let peerAcknowledged = false + void peerMatch.then( + () => { + peerMatched = true + }, + () => undefined, + ) + void peerTxid.then( + () => { + peerAcknowledged = true + }, + () => undefined, + ) + void ownMatch.catch(() => undefined) + void ownTxid.catch(() => undefined) + streams[0]!.send([ + insert(1, `first`), + { headers: { control: `up-to-date`, txids: [33] } }, + ]) + await expect(ownMatch).resolves.toBe(true) + await expect(ownTxid).resolves.toBe(true) + expect(peerMatched).toBe(false) + expect(peerAcknowledged).toBe(false) + await second.cleanup() + await expect(peerMatch).rejects.toThrow(/aborted/i) + await expect(peerTxid).rejects.toThrow(/aborted/i) + await expect(first.utils.awaitTxId(11, 20)).resolves.toBe(true) + expect(streams[0]!.unsubscribe).not.toHaveBeenCalled() + expect(streams[1]!.unsubscribe).toHaveBeenCalledOnce() + } finally { + await first.cleanup() + await second.cleanup() + } + }, + ) + + it.each( + [false, true].flatMap((equalKeys) => + ([`none`, `reset`, `cleanup`] as const).map((peerAction) => ({ + equalKeys, + peerAction, + })), + ), + )( + `keeps tag visibility independent, equal keys=$equalKeys, peer=$peerAction`, + async ({ equalKeys, peerAction }) => { + const options = descriptor(form) + const first = createCollection({ ...options, id: `tag-first` }) + const second = createCollection({ ...options, id: `tag-second` }) + const secondKey = equalKeys ? 1 : 2 + try { + streams[0]!.send([insert(1, `left`), upToDate]) + streams[1]!.send([insert(secondKey, `right`), upToDate]) + if (peerAction === `reset`) { + streams[1]!.send([ + mustRefetch, + insert(secondKey, `right`), + upToDate, + ]) + } else if (peerAction === `cleanup`) { + await second.cleanup() + } + + expect(first.get(1)?.stable).toBe(`stable-1`) + streams[0]!.send([moveOut(`left`), upToDate]) + expect(first.has(1)).toBe(false) + if (peerAction !== `cleanup`) { + expect(second.get(secondKey)?.name).toBe(`right`) + streams[1]!.send([moveOut(`right`), upToDate]) + expect(second.has(secondKey)).toBe(false) + } + } finally { + await first.cleanup() + await second.cleanup() + } + }, + ) + + it(`discards tag state when the same collection starts a fresh session`, async () => { + const collection = createCollection(descriptor(form)) + try { + streams[0]!.send([insert(1, `old`), upToDate]) + await collection.cleanup() + collection.startSyncImmediate() + expect(streams).toHaveLength(2) + streams[1]!.send([insert(1, `current`), upToDate]) + streams[1]!.send([moveOut(`current`), upToDate]) + expect(collection.has(1)).toBe(false) + } finally { + await collection.cleanup() + } + }) + }, +) diff --git a/packages/electric-db-collection/tests/electric-live-query.test.ts b/packages/electric-db-collection/tests/electric-live-query.test.ts index 8bd5ac7b8a..19468bfc78 100644 --- a/packages/electric-db-collection/tests/electric-live-query.test.ts +++ b/packages/electric-db-collection/tests/electric-live-query.test.ts @@ -1,11 +1,11 @@ import { beforeEach, describe, expect, it, vi } from 'vitest' import { + BasicIndex, createCollection, createLiveQueryCollection, eq, gt, lt, - BasicIndex, } from '@tanstack/db' import { electricCollectionOptions } from '../src/electric' import type { ElectricCollectionUtils } from '../src/electric' @@ -58,6 +58,13 @@ const sampleUsers: Array = [ const mockSubscribe = vi.fn() const mockRequestSnapshot = vi.fn() const mockFetchSnapshot = vi.fn() + +function expectNoRepeatedSnapshotRequests() { + const requests = mockRequestSnapshot.mock.calls.map(([request]) => request) + const keys = requests.map((request) => JSON.stringify(request)) + expect(keys).toHaveLength(new Set(keys).size) +} + const mockStream = { subscribe: mockSubscribe, fetchSnapshot: mockFetchSnapshot, @@ -553,7 +560,20 @@ describe.each([ expect(limitedLiveQuery.status).toBe(`ready`) expect(limitedLiveQuery.size).toBe(2) // Only first 2 active users - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + expect( + mockRequestSnapshot.mock.calls.map(([request]) => request), + ).toEqual([ + { + params: { '1': `true` }, + where: `"active" = $1`, + orderBy: `"age" NULLS FIRST`, + limit: 2, + }, + { + params: { '1': `true`, '2': `22` }, + where: `"active" = $1 AND "age" = $2`, + }, + ]) const callArgs = (index: number) => mockRequestSnapshot.mock.calls[index]?.[0] @@ -566,32 +586,34 @@ describe.each([ // Next call will return a snapshot containing 2 rows // Calls after that will return the default empty snapshot - mockRequestSnapshot.mockResolvedValueOnce({ - data: [ - { - headers: { operation: `insert` }, - key: 5, - value: { - id: 5, - name: `Eve`, - age: 30, - email: `eve@example.com`, - active: true, - }, - }, - { - headers: { operation: `insert` }, - key: 6, - value: { - id: 6, - name: `Frank`, - age: 35, - email: `frank@example.com`, - active: true, - }, - }, - ], - }) + mockRequestSnapshot.mockImplementation(async ({ where }) => ({ + data: where.includes(` > `) + ? [ + { + headers: { operation: `insert` }, + key: 5, + value: { + id: 5, + name: `Eve`, + age: 30, + email: `eve@example.com`, + active: true, + }, + }, + { + headers: { operation: `insert` }, + key: 6, + value: { + id: 6, + name: `Frank`, + age: 35, + email: `frank@example.com`, + active: true, + }, + }, + ] + : [], + })) // Create second live query with higher limit of 6 const expandedLiveQuery = createLiveQueryCollection({ @@ -613,11 +635,7 @@ describe.each([ // Wait for the live query to process await new Promise((resolve) => setTimeout(resolve, 0)) - // Limited queries are only deduplicated when their where clauses are equal. - // Both queries have the same where clause (active = true), but the second query - // with limit 6 needs more data than the first query with limit 2 provided. - // With cursor-based pagination, initial loads (without cursor) make 1 requestSnapshot call each. - expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + expectNoRepeatedSnapshotRequests() // Check that first it requested a limit of 2 users (from first query) expect(callArgs(0)).toMatchObject({ @@ -627,13 +645,18 @@ describe.each([ limit: 2, }) - // Check that second it requested a limit of 6 users (from second query) - expect(callArgs(1)).toMatchObject({ + expect(mockRequestSnapshot).toHaveBeenCalledWith({ params: { '1': `true` }, where: `"active" = $1`, orderBy: `"age" NULLS FIRST`, limit: 6, }) + expect(mockRequestSnapshot).toHaveBeenCalledWith( + expect.objectContaining({ + where: `"active" = $1 AND "age" > $2`, + orderBy: `"age" NULLS FIRST`, + }), + ) // The expanded live query should have the locally available data expect(expandedLiveQuery.status).toBe(`ready`) @@ -883,9 +906,9 @@ describe(`Electric Collection with Live Query - syncMode integration`, () => { }), ) - // For limited queries, only requests with identical where clauses can be deduplicated. - // With cursor-based pagination, initial loads (without cursor) make 1 requestSnapshot call. - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + // Electric maps one cursor demand to a bounded page plus its exact tie. + expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + expectNoRepeatedSnapshotRequests() }) it(`should pass correct WHERE clause to requestSnapshot when live query has filters`, async () => { @@ -1005,15 +1028,14 @@ describe(`Electric Collection - loadSubset deduplication`, () => { subscriber(messages) } - it(`should deduplicate identical concurrent loadSubset requests`, async () => { + it(`keeps independently abortable live-query requests independent`, async () => { const electricCollection = createElectricCollectionWithSyncMode(`on-demand`) simulateInitialSync([]) expect(electricCollection.status).toBe(`ready`) - // Create three identical live queries concurrently - // Without deduplication, this would trigger 3 requestSnapshot calls - // With deduplication, only 1 should be made + // Each live query owns its own abort signal, so canceling one cannot cancel + // transport work still needed by a peer. createLiveQueryCollection({ startSync: true, query: (q) => @@ -1046,19 +1068,18 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // With deduplication, only 1 requestSnapshot call should be made - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) - expect(mockRequestSnapshot).toHaveBeenCalledWith( - expect.objectContaining({ + expect(mockRequestSnapshot).toHaveBeenCalledTimes(3) + for (const [request] of mockRequestSnapshot.mock.calls) { + expect(request).toMatchObject({ where: `"active" = $1`, params: { '1': `true` }, orderBy: `"age" NULLS FIRST`, limit: 10, - }), - ) + }) + } }) - it(`should deduplicate subset loadSubset requests with same where clause`, async () => { + it(`keeps different exact windows independent despite a shared predicate`, async () => { const electricCollection = createElectricCollectionWithSyncMode(`on-demand`) simulateInitialSync([]) @@ -1079,8 +1100,8 @@ describe(`Electric Collection - loadSubset deduplication`, () => { expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) - // Create a live query with SAME where clause but smaller limit - // This SHOULD be deduped because where clauses are equal and limit is smaller + // A smaller limit is a distinct exact demand. A requested wider window does + // not prove that its rows were applied or that the source was exhausted. createLiveQueryCollection({ startSync: true, query: (q) => @@ -1093,8 +1114,10 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // Still only 1 call - the second was deduped (same where, smaller limit) - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + expect( + mockRequestSnapshot.mock.calls.map(([request]) => request.limit), + ).toEqual([20, 10]) }) it(`should NOT deduplicate limited queries with different where clauses`, async () => { @@ -1195,9 +1218,10 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // For limited queries, only requests with identical where clauses can be deduplicated. - // With cursor-based pagination, initial loads (without cursor) make 1 requestSnapshot call. - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + const requestsBeforeReset = mockRequestSnapshot.mock.calls.map( + ([request]) => JSON.stringify(request), + ) + expect(requestsBeforeReset.length).toBeGreaterThan(0) // Simulate a must-refetch (which triggers truncate and reset) subscriber([{ headers: { control: `must-refetch` } }]) @@ -1206,9 +1230,15 @@ describe(`Electric Collection - loadSubset deduplication`, () => { // Wait for the existing live query to re-request data after truncate await new Promise((resolve) => setTimeout(resolve, 0)) - // The existing live query re-requests its data after truncate - // After must-refetch, the query requests data again (1 initial + 1 after truncate) - expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + const requestsAfterReset = mockRequestSnapshot.mock.calls + .slice(requestsBeforeReset.length) + .map(([request]) => JSON.stringify(request)) + expect(requestsAfterReset.length).toBeGreaterThan(0) + expect( + requestsAfterReset.some((request) => + requestsBeforeReset.includes(request), + ), + ).toBe(true) // Create the same live query again after reset // This should NOT be deduped because the reset cleared the deduplication state, @@ -1226,12 +1256,12 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // Should have more calls - the different query triggered a new request - // 1 initial + 1 after must-refetch + 1 for new query = 3 - expect(mockRequestSnapshot).toHaveBeenCalledTimes(3) + expect(mockRequestSnapshot).toHaveBeenCalledWith( + expect.objectContaining({ params: { '1': `false` } }), + ) }) - it(`should deduplicate unlimited queries regardless of orderBy`, async () => { + it(`keeps different exact unlimited orderings independent`, async () => { const electricCollection = createElectricCollectionWithSyncMode(`on-demand`) simulateInitialSync([]) @@ -1251,8 +1281,7 @@ describe(`Electric Collection - loadSubset deduplication`, () => { expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) - // Create another unlimited query with same where but different orderBy - // This should be deduped - orderBy is ignored for unlimited queries + // Order remains part of exact demand identity even without a limit. createLiveQueryCollection({ startSync: true, query: (q) => @@ -1264,11 +1293,13 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // Still only 1 call - different orderBy doesn't matter for unlimited queries - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + expect( + mockRequestSnapshot.mock.calls.map(([request]) => request.orderBy), + ).toEqual([`"age" NULLS FIRST`, `"name" DESC NULLS FIRST`]) }) - it(`should combine multiple unlimited queries with union`, async () => { + it(`does not infer union coverage across different predicates`, async () => { const electricCollection = createElectricCollectionWithSyncMode(`on-demand`) simulateInitialSync([]) @@ -1301,8 +1332,8 @@ describe(`Electric Collection - loadSubset deduplication`, () => { expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) - // Create third query (age > 35) - this is a subset of (age > 30) - // This should be deduped + // A broader requested predicate does not prove applied coverage for this + // distinct exact predicate. createLiveQueryCollection({ startSync: true, query: (q) => @@ -1313,7 +1344,55 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // Still 2 calls - third was covered by the union of first two - expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + expect(mockRequestSnapshot).toHaveBeenCalledTimes(3) + expect( + mockRequestSnapshot.mock.calls.map(([request]) => request.params), + ).toEqual([{ '1': `30` }, { '1': `20` }, { '1': `35` }]) + }) + + it(`reuses retained Electric rows after the final live-query owner leaves`, async () => { + const electricCollection = createElectricCollectionWithSyncMode(`on-demand`) + const row = sampleUsers[0]! + simulateInitialSync([]) + mockRequestSnapshot.mockResolvedValue({ + data: [ + { + headers: { operation: `insert` }, + key: row.id, + value: row, + }, + ], + }) + const createLive = (id: string) => + createLiveQueryCollection({ + id, + startSync: true, + query: (q) => + q + .from({ user: electricCollection }) + .where(({ user }) => eq(user.active, true)), + }) + const first = createLive(`electric-remount-first`) + let second: ReturnType | undefined + + try { + await first.preload() + expect(first.toArray.map(({ id }) => id)).toEqual([row.id]) + + await first.cleanup() + expect(electricCollection.size).toBe(1) + + second = createLive(`electric-remount-second`) + await second.preload() + + expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + expect(second.toArray.map(({ id }) => id)).toEqual([row.id]) + } finally { + await Promise.all([ + first.cleanup(), + second?.cleanup(), + electricCollection.cleanup(), + ]) + } }) }) diff --git a/packages/electric-db-collection/tests/electric-oracle.property.test.ts b/packages/electric-db-collection/tests/electric-oracle.property.test.ts new file mode 100644 index 0000000000..d0f23d08da --- /dev/null +++ b/packages/electric-db-collection/tests/electric-oracle.property.test.ts @@ -0,0 +1,3789 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { beforeEach, describe, expect, it, vi } from 'vitest' +import { IR, createCollection, createTransaction } from '@tanstack/db' +import { ShapeStream } from '@electric-sql/client' +import { QueryClient } from '@tanstack/query-core' +import { persistedCollectionOptions } from '../../db-sqlite-persistence-core/src' +import { queryCollectionOptions } from '../../query-db-collection/src/query' +import { electricCollectionOptions } from '../src/electric' +import { oraclePropertyOptions, oracleRuns } from '../../db/tests/oracle-config' +import type { Collection, SyncMetadataApi } from '@tanstack/db' +import type { ChangeMessage, Message, Offset, Row } from '@electric-sql/client' +import type { + PersistedTx, + PersistenceAdapter, +} from '../../db-sqlite-persistence-core/src' +import type { ElectricCollectionUtils, ElectricSyncMode } from '../src/electric' + +type OracleRow = Row & { + id: number + name: string + stable: string +} + +const mockSubscribe = vi.fn() +const shapeId = `{"params":{"table":"test_table"},"url":"http://test-url"}` +const mockStream = { + subscribe: mockSubscribe, + requestSnapshot: vi.fn().mockResolvedValue(undefined), + fetchSnapshot: vi.fn().mockResolvedValue({ metadata: {}, data: [] }), + forceDisconnectAndRefresh: vi.fn().mockResolvedValue(undefined), + isUpToDate: false, + shapeHandle: undefined as string | undefined, + lastOffset: `-1` as string, +} + +function createDeferred(): { + promise: Promise + resolve: (value: T | PromiseLike) => void +} { + let resolve!: (value: T | PromiseLike) => void + const promise = new Promise((resolvePromise) => { + resolve = resolvePromise + }) + return { promise, resolve } +} + +vi.mock(`@electric-sql/client`, async () => { + const actual = await vi.importActual(`@electric-sql/client`) + return { + ...actual, + ShapeStream: vi.fn(() => mockStream), + } +}) + +function everyContiguousPartition(values: Array): Array>> { + if (values.length === 0) return [[]] + const partitions: Array>> = [] + const boundaryCount = values.length - 1 + for (let mask = 0; mask < 1 << boundaryCount; mask++) { + const batches: Array> = [[values[0]!]] + for (let index = 1; index < values.length; index++) { + if ((mask & (1 << (index - 1))) !== 0) { + batches.push([values[index]!]) + } else { + batches.at(-1)!.push(values[index]!) + } + } + partitions.push(batches) + } + return partitions +} + +function isSdkResetFramedPartition( + batches: Array>>, +): boolean { + // Model the installed SDK's HTTP 409 reset path: it publishes a synthetic + // singleton reset, not the response body. See electric-sdk-framing.test.ts. + // Arbitrary data/reset coalescing is outside this verified protocol domain; + // this is not a claim that the SDK validates every other server response. + return batches.every( + (batch) => + batch.length <= 1 || + batch.every( + (message) => + (message.headers as Record).control !== + `must-refetch`, + ), + ) +} + +function createMetadata(seed: ReadonlyMap): { + api: SyncMetadataApi + state: Map +} { + const state = new Map(seed) + return { + state, + api: { + row: { + get: () => undefined, + set: () => {}, + delete: () => {}, + }, + collection: { + get: (key) => state.get(key), + set: (key, value) => { + state.set(key, value) + }, + delete: (key) => { + state.delete(key) + }, + list: (prefix) => + Array.from(state, ([key, value]) => ({ key, value })).filter( + ({ key }) => !prefix || key.startsWith(prefix), + ), + }, + }, + } +} + +function resumeState(): ReadonlyMap { + // This fixture represents an untagged source. Legacy/unknown membership is + // exercised separately by the cold-recovery history generator. + return new Map([ + [ + `electric:resume`, + { + kind: `resume`, + requiresTagState: false, + offset: `10_0`, + handle: `shape-1`, + shapeId, + updatedAt: 1, + }, + ], + ]) +} + +function createPersistedAdapter( + collectionMetadata: Map, + rows: Map, + loadGate: Promise = Promise.resolve(), +): PersistenceAdapter { + return { + loadSubset: () => + loadGate.then(() => Array.from(rows, ([key, value]) => ({ key, value }))), + loadCollectionMetadata: () => + Promise.resolve( + Array.from(collectionMetadata, ([key, value]) => ({ key, value })), + ), + applyCommittedTx: (_collectionId: string, tx: PersistedTx) => { + for (const mutation of tx.collectionMetadataMutations ?? []) { + if (mutation.type === `delete`) { + collectionMetadata.delete(mutation.key) + } else { + collectionMetadata.set(mutation.key, mutation.value) + } + } + if (tx.truncate) rows.clear() + for (const mutation of tx.mutations) { + if (mutation.type === `delete`) { + rows.delete(mutation.key) + } else if (mutation.type === `update`) { + rows.set(mutation.key, { + ...rows.get(mutation.key), + ...mutation.value, + } as OracleRow) + } else { + rows.set(mutation.key, mutation.value as OracleRow) + } + } + return Promise.resolve() + }, + ensureIndex: () => Promise.resolve(), + } +} + +function createOracleCollection( + id: string, + syncMode: ElectricSyncMode, + metadata: SyncMetadataApi, + shapeResumeOptions: { offset?: Offset; handle?: string } = {}, +) { + let subscriber!: (messages: Array>) => void + const unsubscribe = vi.fn() + mockSubscribe.mockImplementationOnce((callback) => { + subscriber = callback + return unsubscribe + }) + const options = electricCollectionOptions({ + id, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + ...shapeResumeOptions, + }, + syncMode, + getKey: (row) => row.id, + startSync: true, + }) + const originalSync = options.sync + return { + collection: createCollection({ + ...options, + sync: { + sync: (params: Parameters[0]) => + originalSync.sync({ ...params, metadata }), + }, + }), + subscriber, + unsubscribe, + } +} + +type TraceResult = { + rows: Array<[string | number, string, string]> + snapshots: Array> + status: string + resume: unknown +} + +type PersistedTraceResult = TraceResult & { + durableRows: Array<[string | number, string, string]> + durableResume: unknown + persistenceCommits: number +} + +type ReferenceState = { + committed: Map + pending: Map +} + +function rowsFromCollection( + collection: Collection, +): Array<[string | number, string, string]> { + return Array.from( + collection, + ([key, row]): [string | number, string, string] => [ + key, + row.name, + row.stable, + ], + ).sort(([left], [right]) => String(left).localeCompare(String(right))) +} + +function rowsFromMap( + rows: ReadonlyMap, +): Array<[string | number, string, string]> { + return Array.from(rows, ([key, row]): [string | number, string, string] => [ + key, + row.name, + row.stable, + ]).sort(([left], [right]) => String(left).localeCompare(String(right))) +} + +function applyReferenceBatch( + state: ReferenceState, + batch: ReadonlyArray>, +): boolean { + let commits = false + for (const message of batch) { + const headers = message.headers as Record + const control = headers.control + if (typeof control === `string`) { + if (control === `must-refetch`) { + state.pending.clear() + } + if (control === `up-to-date` || control === `subset-end`) commits = true + continue + } + if (!(`value` in message)) continue + const value = message.value + const id = value.id + if (headers.operation === `delete`) { + state.pending.delete(id) + } else if (headers.operation === `insert`) { + state.pending.set(id, value) + } else { + const current = state.pending.get(id) + if (current) { + state.pending.set(id, { ...current, ...value } as OracleRow) + } + } + } + if (commits) state.committed = new Map(state.pending) + return commits +} + +function expectedSnapshots( + prefix: Array>>, + batches: Array>>, +): Array> { + const state: ReferenceState = { + committed: new Map(), + pending: new Map(), + } + const observed: Array> = [] + for (const batch of prefix) applyReferenceBatch(state, batch) + for (const batch of batches) { + applyReferenceBatch(state, batch) + observed.push(rowsFromMap(state.committed)) + } + return observed +} + +function recomputeCommittedRows( + batches: ReadonlyArray>>, + unknownUpdate: `ignore` | `promote-complete` = `ignore`, +): Array<[string | number, string, string]> { + let commitBatchIndex = -1 + for (let index = 0; index < batches.length; index++) { + const commits = batches[index]!.some((message) => { + const control = (message.headers as Record).control + return control === `up-to-date` || control === `subset-end` + }) + if (commits) commitBatchIndex = index + } + if (commitBatchIndex < 0) return [] + + // A control message commits its entire callback, including messages that + // happen to follow the control inside that atomic delivery. + const committedPrefix = batches.slice(0, commitBatchIndex + 1).flat() + let resetIndex = -1 + for (let index = 0; index < committedPrefix.length; index++) { + if ( + (committedPrefix[index]!.headers as Record).control === + `must-refetch` + ) { + resetIndex = index + } + } + + const rows = new Map() + for (const message of committedPrefix.slice(resetIndex + 1)) { + if (!(`value` in message)) continue + const headers = message.headers as Record + const value = message.value + if (headers.operation === `delete`) { + rows.delete(value.id) + } else if (headers.operation === `insert`) { + rows.set(value.id, value) + } else { + const current = rows.get(value.id) + if (current) { + rows.set(value.id, { ...current, ...value } as OracleRow) + } else if ( + unknownUpdate === `promote-complete` && + typeof value.name === `string` && + typeof value.stable === `string` + ) { + rows.set(value.id, value) + } + } + } + return rowsFromMap(rows) +} + +function recomputedSnapshots( + prefix: Array>>, + batches: Array>>, +): Array> { + const history = [...prefix] + return batches.map((batch) => { + history.push(batch) + return recomputeCommittedRows(history) + }) +} + +function observableResume(value: unknown): unknown { + if (value === null || typeof value !== `object`) return value + const state = value as Record + return { + kind: state.kind, + offset: state.offset, + handle: state.handle, + shapeId: state.shapeId, + } +} + +async function runTrace( + id: string, + syncMode: ElectricSyncMode, + prefix: Array>>, + batches: Array>>, + seed: ReadonlyMap = new Map(), +): Promise { + const metadata = createMetadata(seed) + const { collection, subscriber } = createOracleCollection( + id, + syncMode, + metadata.api, + ) + for (const batch of prefix) subscriber(batch) + const snapshots: Array> = [] + for (const batch of batches) { + subscriber(batch) + snapshots.push(rowsFromCollection(collection)) + } + const rows = rowsFromCollection(collection) + const result = { + rows, + snapshots, + status: collection.status, + resume: observableResume(metadata.state.get(`electric:resume`)), + } + await collection.cleanup() + return result +} + +async function runPersistedTrace( + id: string, + syncMode: ElectricSyncMode, + batches: Array>>, +): Promise { + let subscriber!: (messages: Array>) => void + mockSubscribe.mockImplementationOnce((callback) => { + subscriber = callback + return vi.fn() + }) + const persistedRows = new Map() + const persistedMetadata = new Map() + const adapter = createPersistedAdapter(persistedMetadata, persistedRows) + const applyCommittedTx = adapter.applyCommittedTx.bind(adapter) + let persistenceCommits = 0 + adapter.applyCommittedTx = (...args) => { + persistenceCommits++ + return applyCommittedTx(...args) + } + const collection = createCollection( + persistedCollectionOptions< + OracleRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricCollectionOptions({ + id, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode, + getKey: (row) => row.id, + startSync: true, + }), + persistence: { + adapter, + }, + }), + ) + collection.startSyncImmediate() + await vi.waitFor(() => expect(subscriber).toBeTypeOf(`function`), { + interval: 1, + timeout: 250, + }) + + const snapshots: Array> = [] + const history: Array>> = [] + for (const batch of batches) { + subscriber(batch) + history.push(batch) + const expected = recomputeCommittedRows(history) + await vi.waitFor( + () => + expect( + rowsFromCollection(collection), + `${syncMode}: ${JSON.stringify(history)}`, + ).toEqual(expected), + { interval: 1, timeout: 250 }, + ) + snapshots.push(rowsFromCollection(collection)) + } + await vi.waitFor(() => expect(collection.status).toBe(`ready`), { + interval: 1, + timeout: 250, + }) + const exported = collection.config.sync.exportSyncMeta?.() as + | { resume?: unknown } + | undefined + await vi.waitFor( + () => { + expect(persistenceCommits).toBeGreaterThan(0) + expect(rowsFromMap(persistedRows)).toEqual(rowsFromCollection(collection)) + }, + { interval: 1, timeout: 250 }, + ) + const result = { + rows: rowsFromCollection(collection), + snapshots, + status: collection.status, + resume: observableResume(exported?.resume), + durableRows: rowsFromMap(persistedRows), + durableResume: observableResume(persistedMetadata.get(`electric:resume`)), + persistenceCommits, + } + await collection.cleanup() + return result +} + +async function runQueryTrace( + id: string, + batches: Array>>, +): Promise { + const queryClient = new QueryClient({ + defaultOptions: { + queries: { + gcTime: Number.POSITIVE_INFINITY, + retry: false, + staleTime: Number.POSITIVE_INFINITY, + }, + }, + }) + const queryKey = [id] as const + let queryRows: Array = [] + const collection = createCollection( + queryCollectionOptions({ + id, + queryClient, + queryKey, + queryFn: () => Promise.resolve(queryRows), + getKey: (row) => row.id, + startSync: true, + }), + ) + await collection.preload() + + const snapshots: TraceResult[`snapshots`] = [] + const history: Array>> = [] + for (const batch of batches) { + history.push(batch) + const commits = batch.some((message) => { + const control = (message.headers as Record).control + return control === `up-to-date` || control === `subset-end` + }) + if (commits) { + queryRows = recomputeCommittedRows(history).map( + ([rowId, name, stable]) => ({ + id: Number(rowId), + name, + stable, + }), + ) + queryClient.setQueryData(queryKey, queryRows) + await vi.waitFor( + () => { + expect(rowsFromCollection(collection)).toEqual( + queryRows.map((row): [number, string, string] => [ + row.id, + row.name, + row.stable, + ]), + ) + }, + { interval: 1, timeout: 250 }, + ) + } + snapshots.push(rowsFromCollection(collection)) + } + + const result: TraceResult = { + rows: rowsFromCollection(collection), + snapshots, + status: collection.status, + resume: undefined, + } + await collection.cleanup() + queryClient.clear() + return result +} + +function change( + operation: `insert` | `update` | `delete`, + id: number, + name: string, +): ChangeMessage { + const value = + operation === `insert` + ? { id, name, stable: `stable-${id}` } + : operation === `update` + ? { id, name } + : { id } + return { + key: String(id), + value: value as OracleRow, + headers: { operation }, + } +} + +const upToDate: Message = { + headers: { control: `up-to-date` }, +} +const subsetEnd: Message = { + headers: { control: `subset-end` }, +} +const mustRefetch: Message = { + headers: { control: `must-refetch` }, +} + +async function checkInitialMoveOut( + syncMode: ElectricSyncMode, + removals: number, + id: number, + updated: string, +) { + const inserted = change(`insert`, id, `snapshot`) + const initial: Array> = [ + { ...inserted, headers: { ...inserted.headers, tags: [`left`] } }, + ...Array.from( + { length: removals }, + () => + ({ + headers: { + event: `move-out`, + patterns: [{ pos: 0, value: `left` }], + }, + }) as Message, + ), + upToDate, + ] + // The contract is a tagged relation: removing its only tag removes the row. + // Completing that snapshot must not affect later, unrelated live updates. + const initialRows: TraceResult[`rows`] = + removals === 0 ? [[id, `snapshot`, `stable-${id}`]] : [] + for (const partition of everyContiguousPartition(initial)) { + const result = await runTrace( + `initial-move-out-${syncMode}`, + syncMode, + partition, + [ + [change(`insert`, id + 1, `live`), upToDate], + [change(`update`, id + 1, updated), upToDate], + ], + ) + expect(result.snapshots).toEqual([ + [...initialRows, [id + 1, `live`, `stable-${id + 1}`]].sort( + ([left], [right]) => String(left).localeCompare(String(right)), + ), + [...initialRows, [id + 1, updated, `stable-${id + 1}`]].sort( + ([left], [right]) => String(left).localeCompare(String(right)), + ), + ]) + expect(result.status).toBe(`ready`) + } +} + +type PartitionScenario = { + name: string + prefix: Array>> + messages: Array> + expectedRows: Array<[number, string, string]> + resume: boolean +} + +type HistoryToken = { + operation: `insert` | `update` | `delete` + id: number + name: string +} + +type DesignToken = + | HistoryToken + | { operation: `reset` | `commit` | `subset` | `neutral` } + +type ProcessSlot = `a` | `b` + +type ProcessCommand = + | { kind: `create`; slot: ProcessSlot } + | { kind: `import`; slot: ProcessSlot; txid: number; resume: boolean } + | { kind: `preload`; slot: ProcessSlot } + | { + kind: `batch` + slot: ProcessSlot + operation: HistoryToken[`operation`] + id: number + name: string + txid: number + } + | { kind: `reset`; slot: ProcessSlot } + | { kind: `snapshot`; slot: ProcessSlot; id: number; name: string } + | { kind: `cleanup`; slot: ProcessSlot } + | { kind: `restart`; slot: ProcessSlot } + +type ProcessRuntime = { + collection: Collection + subscriber?: (messages: Array>) => void + reference: ReferenceState + seenTxids: Set + resumeAvailable: boolean + requiresCompleteResume: boolean + resettingSnapshot: boolean + terminalError: boolean + active: boolean + retired: boolean + preloadPromises: Array> +} + +const processCommandArb: fc.Arbitrary = fc.oneof( + fc.record({ + kind: fc.constant(`create` as const), + slot: fc.constantFrom(`a`, `b`), + }), + fc.record({ + kind: fc.constant(`import` as const), + slot: fc.constantFrom(`a`, `b`), + txid: fc.integer({ min: 1, max: 50 }), + resume: fc.boolean(), + }), + fc.record({ + kind: fc.constant(`preload` as const), + slot: fc.constantFrom(`a`, `b`), + }), + fc.record({ + kind: fc.constant(`batch` as const), + slot: fc.constantFrom(`a`, `b`), + operation: fc.constantFrom( + `insert`, + `update`, + `delete`, + ), + id: fc.integer({ min: 1, max: 3 }), + name: fc.string({ maxLength: 8 }), + txid: fc.integer({ min: 51, max: 100 }), + }), + fc.record({ + kind: fc.constant(`reset` as const), + slot: fc.constantFrom(`a`, `b`), + }), + fc.record({ + kind: fc.constant(`snapshot` as const), + slot: fc.constantFrom(`a`, `b`), + id: fc.integer({ min: 1, max: 3 }), + name: fc.string({ maxLength: 8 }), + }), + fc.record({ + kind: fc.constant(`cleanup` as const), + slot: fc.constantFrom(`a`, `b`), + }), + fc.record({ + kind: fc.constant(`restart` as const), + slot: fc.constantFrom(`a`, `b`), + }), +) + +const designTokenArb: fc.Arbitrary = fc.oneof( + fc.record({ + operation: fc.constantFrom( + `insert`, + `update`, + `delete`, + ), + id: fc.integer({ min: 1, max: 3 }), + name: fc.string({ maxLength: 8 }), + }), + fc.record({ + operation: fc.constantFrom<`reset` | `commit` | `subset` | `neutral`>( + `reset`, + `commit`, + `subset`, + `neutral`, + ), + }), +) + +type SchedulerEvent = + | `startup-promise` + | `hydration` + | `snapshot-available` + | `commit` + | `cleanup` + +function permutations(values: ReadonlyArray): Array> { + if (values.length <= 1) return [[...values]] + const result: Array> = [] + values.forEach((value, index) => { + const rest = [...values.slice(0, index), ...values.slice(index + 1)] + for (const suffix of permutations(rest)) result.push([value, ...suffix]) + }) + return result +} + +async function drainScheduler(): Promise { + for (let turn = 0; turn < 12; turn++) await Promise.resolve() +} + +function buildValidHistory(tokens: Array): { + messages: Array> + expectedRows: Array<[number, string, string]> +} { + const state = new Map() + const messages: Array> = [] + + for (const token of tokens) { + if (token.operation === `delete`) { + if (!state.has(token.id)) continue + messages.push(change(`delete`, token.id, token.name)) + state.delete(token.id) + continue + } + + const operation = + token.operation === `update` && !state.has(token.id) + ? `insert` + : token.operation + messages.push(change(operation, token.id, token.name)) + state.set(token.id, { + name: token.name, + stable: `stable-${token.id}`, + }) + } + + return { + messages: [...messages, upToDate], + expectedRows: Array.from(state, ([id, row]): [number, string, string] => [ + id, + row.name, + row.stable, + ]).sort(([left], [right]) => left - right), + } +} + +function designMessage(token: DesignToken): Message { + if (token.operation === `reset`) return mustRefetch + if (token.operation === `commit`) return upToDate + if (token.operation === `subset`) return subsetEnd + if (token.operation === `neutral`) { + return { + headers: { + control: `snapshot-end`, + xmin: `100`, + xmax: `150`, + xip_list: [], + }, + } + } + if (!(`id` in token)) throw new Error(`Unknown design token`) + return change(token.operation, token.id, token.name) +} + +function buildDifferentialHistory( + tokens: Array, +): Array> { + const known = new Set([99]) + const messages: Array> = [ + change(`insert`, 99, `baseline`), + upToDate, + ] + + for (const token of tokens) { + if (token.operation === `reset`) { + known.clear() + messages.push(mustRefetch) + continue + } + if ( + token.operation === `commit` || + token.operation === `subset` || + token.operation === `neutral` + ) { + messages.push(designMessage(token)) + continue + } + if (!(`id` in token)) throw new Error(`Unknown differential token`) + if (token.operation === `delete`) { + if (!known.delete(token.id)) continue + messages.push(change(`delete`, token.id, token.name)) + continue + } + const operation = + token.operation === `update` && !known.has(token.id) + ? `insert` + : token.operation + known.add(token.id) + messages.push(change(operation, token.id, token.name)) + } + + messages.push(subsetEnd) + return messages +} + +async function runProcessGrammar( + idPrefix: string, + generated: Array, +): Promise { + const subscribers: Array<(messages: Array>) => void> = [] + const runtimes = new Map() + const allPreloads: Array> = [] + let generation = 0 + mockSubscribe.mockReset() + mockSubscribe.mockImplementation((callback) => { + subscribers.push(callback) + return vi.fn() + }) + + const createRuntime = async (slot: ProcessSlot) => { + const previous = runtimes.get(slot) + if (previous) await previous.collection.cleanup() + generation++ + const collection = createCollection( + electricCollectionOptions({ + id: `${idPrefix}-${slot}-${generation}`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: false, + }), + ) + runtimes.set(slot, { + collection, + reference: { committed: new Map(), pending: new Map() }, + seenTxids: new Set(), + resumeAvailable: false, + requiresCompleteResume: false, + resettingSnapshot: false, + terminalError: false, + active: false, + retired: false, + preloadPromises: [], + }) + } + + const startRuntime = (runtime: ProcessRuntime, preload: boolean) => { + if (runtime.active) return + const subscriberIndex = subscribers.length + if (preload) { + const promise = runtime.collection.preload() + runtime.preloadPromises.push(promise) + allPreloads.push(promise) + } else { + runtime.collection.startSyncImmediate() + } + const subscriber = subscribers[subscriberIndex] + if (!subscriber) throw new Error(`Electric stream did not subscribe`) + runtime.subscriber = subscriber + runtime.requiresCompleteResume = runtime.resumeAvailable + runtime.resettingSnapshot = false + runtime.terminalError = false + runtime.active = true + runtime.retired = false + } + + const assertAllSlots = () => { + runtimes.forEach((runtime) => { + const expected = runtime.active + ? rowsFromMap(runtime.reference.committed) + : [] + expect(rowsFromCollection(runtime.collection)).toEqual(expected) + if (runtime.terminalError) { + expect(runtime.collection.status).toBe(`error`) + } + if (!runtime.active && runtime.retired) { + expect(runtime.collection.status).toBe(`cleaned-up`) + } + }) + } + + const execute = async (command: ProcessCommand) => { + if (command.kind === `create`) { + await createRuntime(command.slot) + return + } + const runtime = runtimes.get(command.slot) + if (!runtime) return + + if (command.kind === `import`) { + if (runtime.active) return + runtime.collection.config.sync.importSyncMeta?.({ + version: 1, + seenTxids: [command.txid], + ...(command.resume + ? { + resume: { + kind: `resume`, + requiresTagState: false, + offset: `10_0`, + handle: `shape-1`, + shapeId, + updatedAt: command.txid, + }, + } + : {}), + }) + await expect( + runtime.collection.utils.awaitTxId(command.txid, 20), + ).resolves.toBe(true) + runtime.resumeAvailable ||= command.resume + runtime.seenTxids.add(command.txid) + for (const [otherSlot, other] of runtimes) { + if (otherSlot === command.slot || other.seenTxids.has(command.txid)) { + continue + } + await expect( + other.collection.utils.awaitTxId(command.txid, 2), + ).rejects.toThrow() + } + return + } + + if (command.kind === `preload`) { + startRuntime(runtime, true) + return + } + if (command.kind === `restart`) { + startRuntime(runtime, false) + return + } + if (command.kind === `cleanup`) { + await runtime.collection.cleanup() + runtime.active = false + runtime.retired = true + runtime.subscriber = undefined + runtime.reference = { + committed: new Map(), + pending: new Map(), + } + runtime.resumeAvailable = false + runtime.requiresCompleteResume = false + runtime.resettingSnapshot = false + runtime.terminalError = false + return + } + if (!runtime.active || !runtime.subscriber || runtime.terminalError) return + + if (command.kind === `reset`) { + runtime.subscriber([mustRefetch]) + applyReferenceBatch(runtime.reference, [mustRefetch]) + runtime.resettingSnapshot = true + return + } + + if (command.kind === `snapshot`) { + const messages = [change(`insert`, command.id, command.name), upToDate] + runtime.subscriber(messages) + applyReferenceBatch(runtime.reference, messages) + runtime.resettingSnapshot = false + runtime.resumeAvailable = true + return + } + + const evidenceChange = change(command.operation, command.id, command.name) + const messages: Array> = [ + { + ...evidenceChange, + headers: { + operation: command.operation, + txids: [command.txid], + }, + }, + upToDate, + ] + const observesTxid = + command.operation !== `update` || + runtime.reference.pending.has(command.id) + const invalidResume = + runtime.requiresCompleteResume && + !runtime.resettingSnapshot && + command.operation === `update` && + !runtime.reference.pending.has(command.id) + const txidOutcome = observesTxid + ? runtime.collection.utils.awaitTxId(command.txid, 100) + : undefined + runtime.subscriber(messages) + if (invalidResume) { + runtime.terminalError = true + runtime.resumeAvailable = false + return + } + applyReferenceBatch(runtime.reference, messages) + runtime.resettingSnapshot = false + runtime.resumeAvailable = true + if (txidOutcome) { + await expect(txidOutcome).resolves.toBe(true) + runtime.seenTxids.add(command.txid) + } + } + + const requiredPrefix: Array = [ + { kind: `create`, slot: `a` }, + { kind: `create`, slot: `b` }, + { kind: `import`, slot: `a`, txid: 1, resume: true }, + { kind: `import`, slot: `b`, txid: 2, resume: false }, + { kind: `preload`, slot: `a` }, + { kind: `preload`, slot: `b` }, + { + kind: `batch`, + slot: `a`, + operation: `insert`, + id: 1, + name: `a-initial`, + txid: 51, + }, + { + kind: `batch`, + slot: `b`, + operation: `insert`, + id: 1, + name: `b-initial`, + txid: 52, + }, + { kind: `reset`, slot: `a` }, + { kind: `snapshot`, slot: `a`, id: 2, name: `a-snapshot` }, + ] + const requiredSuffix: Array = [ + { kind: `cleanup`, slot: `a` }, + { kind: `restart`, slot: `a` }, + { + kind: `batch`, + slot: `a`, + operation: `insert`, + id: 3, + name: `a-restarted`, + txid: 53, + }, + { kind: `cleanup`, slot: `b` }, + { kind: `restart`, slot: `b` }, + { + kind: `batch`, + slot: `b`, + operation: `insert`, + id: 3, + name: `b-restarted`, + txid: 54, + }, + ] + + try { + for (const command of [ + ...requiredPrefix, + ...generated, + ...requiredSuffix, + ]) { + await execute(command) + assertAllSlots() + } + } finally { + await Promise.all( + Array.from(runtimes.values(), ({ collection }) => collection.cleanup()), + ) + await Promise.allSettled(allPreloads) + mockSubscribe.mockReset() + } +} + +async function runSchedulerPermutation( + id: string, + order: Array, +): Promise<{ + outcome: `resolved` | `aborted` + acknowledgedBeforeDurable: boolean +}> { + const startup = createDeferred() + const hydration = createDeferred() + const commit = createDeferred() + const persistedMetadata = new Map(resumeState()) + const persistedRows = new Map([ + [1, { id: 1, name: `persisted`, stable: `stable-1` }], + ]) + const adapter = createPersistedAdapter( + persistedMetadata, + persistedRows, + hydration.promise, + ) + const loadMetadata = adapter.loadCollectionMetadata!.bind(adapter) + adapter.loadCollectionMetadata = async (...args) => { + await startup.promise + return loadMetadata(...args) + } + const applyCommittedTx = adapter.applyCommittedTx.bind(adapter) + adapter.applyCommittedTx = async (...args) => { + await commit.promise + return applyCommittedTx(...args) + } + + let subscriber: ((messages: Array>) => void) | undefined + let snapshotRequested = false + let snapshotDelivered = false + const deliveryPhases = new Set<`before-cleanup` | `after-cleanup`>() + let cleanupCompleted = false + let acknowledgedBeforeDurable = false + const schedulerStream = { + ...mockStream, + subscribe: (callback: (messages: Array>) => void) => { + subscriber = callback + return vi.fn() + }, + } + vi.mocked(ShapeStream).mockReset() + vi.mocked(ShapeStream).mockImplementation(() => schedulerStream as never) + + const collection = createCollection( + persistedCollectionOptions< + OracleRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricCollectionOptions({ + id, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `progressive`, + getKey: (row) => row.id, + startSync: false, + }), + persistence: { adapter }, + }), + ) + let matchSettlement: `resolved` | `aborted` | `timed-out` | undefined + let txidSettlement: `resolved` | `aborted` | `timed-out` | undefined + const matchOutcome = collection.utils + .awaitMatch((message) => `value` in message && message.value.id === 1, 250) + .then( + () => `resolved` as const, + (error: unknown) => + /aborted/i.test(String(error)) + ? (`aborted` as const) + : (`timed-out` as const), + ) + .then((outcome) => (matchSettlement = outcome)) + const txidOutcome = collection.utils + .awaitTxId(91, 250) + .then( + () => `resolved` as const, + (error: unknown) => + /aborted/i.test(String(error)) + ? (`aborted` as const) + : (`timed-out` as const), + ) + .then((outcome) => (txidSettlement = outcome)) + collection.startSyncImmediate() + + const deliverSnapshotIfPossible = () => { + if (!snapshotRequested || snapshotDelivered || !subscriber) return + snapshotDelivered = true + deliveryPhases.add(cleanupCompleted ? `after-cleanup` : `before-cleanup`) + const update = change(`update`, 1, `scheduled`) + subscriber([ + { ...update, headers: { operation: `update`, txids: [91] } }, + upToDate, + ]) + } + + const assertSchedulerCheckpoint = () => { + expect(matchSettlement === undefined).toBe(txidSettlement === undefined) + if (matchSettlement === `resolved`) { + expect(txidSettlement).toBe(`resolved`) + expect(snapshotDelivered).toBe(true) + acknowledgedBeforeDurable ||= persistedRows.get(1)?.name === `persisted` + } + if (matchSettlement === `aborted`) { + expect(txidSettlement).toBe(`aborted`) + expect(collection.status).toBe(`cleaned-up`) + } + expect(matchSettlement).not.toBe(`timed-out`) + expect(txidSettlement).not.toBe(`timed-out`) + } + + try { + for (const event of order) { + if (event === `startup-promise`) startup.resolve() + if (event === `hydration`) hydration.resolve() + if (event === `snapshot-available`) snapshotRequested = true + if (event === `commit`) commit.resolve() + if (event === `cleanup`) { + await collection.cleanup() + cleanupCompleted = true + } + await drainScheduler() + deliverSnapshotIfPossible() + await drainScheduler() + assertSchedulerCheckpoint() + } + + startup.resolve() + hydration.resolve() + commit.resolve() + snapshotRequested = true + await drainScheduler() + deliverSnapshotIfPossible() + await drainScheduler() + assertSchedulerCheckpoint() + + const [match, txid] = await Promise.all([matchOutcome, txidOutcome]) + if (match === `timed-out` || txid === `timed-out`) { + throw new Error(`scheduler waiter timed out`) + } + expect(match).toBe(txid) + expect(collection.status).toBe(`cleaned-up`) + expect(rowsFromCollection(collection)).toEqual([]) + if (deliveryPhases.has(`after-cleanup`)) { + expect(persistedRows.get(1)?.name).toBe(`persisted`) + } + + if (order[0] === `cleanup`) { + expect(match).toBe(`aborted`) + expect(subscriber).toBeUndefined() + expect(persistedRows.get(1)?.name).toBe(`persisted`) + } + if (match === `resolved`) { + expect(persistedRows.get(1)).toEqual({ + id: 1, + name: `scheduled`, + stable: `stable-1`, + }) + } + return { outcome: match, acknowledgedBeforeDurable } + } finally { + await collection.cleanup() + vi.mocked(ShapeStream).mockReset() + vi.mocked(ShapeStream).mockImplementation(() => mockStream as never) + } +} + +describe(`Electric adapter laws`, () => { + let processGrammarRun = 0 + + beforeEach(() => { + vi.clearAllMocks() + mockStream.isUpToDate = false + mockStream.shapeHandle = `shape-current` + mockStream.lastOffset = `20_0` + }) + + it.each( + ([`eager`, `on-demand`, `progressive`] as const).flatMap((syncMode) => + [0, 1, 2].map((removals) => ({ syncMode, removals })), + ), + )( + `preserves live updates after $removals initial tagged move-outs in $syncMode mode`, + ({ syncMode, removals }) => + checkInitialMoveOut(syncMode, removals, 1, `updated`), + ) + + fcTest.prop( + [ + fc.integer({ min: 1, max: 20 }), + fc.string({ maxLength: 8 }), + fc.integer({ min: 0, max: 2 }), + ], + { numRuns: 20 }, + )( + `generated initial tagged move-outs preserve later live updates`, + async (id, updated, removals) => { + for (const syncMode of [`eager`, `on-demand`, `progressive`] as const) { + await checkInitialMoveOut(syncMode, removals, id, updated) + } + }, + ) + + fcTest.prop([fc.array(processCommandArb, { maxLength: 20 })], { + numRuns: 20, + })( + `generated process grammar preserves lifecycle and concurrent-collection isolation`, + async (commands) => { + processGrammarRun++ + await runProcessGrammar(`process-grammar-${processGrammarRun}`, commands) + }, + ) + + it(`stops lifecycle replay after an invalid resumed update`, async () => { + await runProcessGrammar(`process-grammar-terminal-error`, [ + { + kind: `batch`, + slot: `a`, + operation: `update`, + id: 1, + name: `unseen`, + txid: 55, + }, + { kind: `snapshot`, slot: `a`, id: 1, name: `stale callback` }, + ]) + }) + + it.each([`reset`, `delete`, `move-out`] as const)( + `keeps $0 removal authoritative across an optimistic write and a new acquisition`, + async (removal) => { + const trace = createOracleCollection( + `parked-presence`, + `on-demand`, + createMetadata(new Map()).api, + ) + const persistence = createDeferred() + let acquired: Promise | undefined + let transaction: ReturnType | undefined + try { + const inserted = change(`insert`, 1, `complete`) + trace.subscriber([ + { ...inserted, headers: { ...inserted.headers, tags: [`left`] } }, + upToDate, + ]) + transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => + trace.collection.insert({ + id: 99, + name: `optimistic`, + stable: `stable-99`, + }), + ) + const removed: Message = + removal === `reset` + ? mustRefetch + : removal === `delete` + ? change(`delete`, 1, `complete`) + : { + headers: { + event: `move-out`, + patterns: [{ pos: 0, value: `left` }], + }, + } + trace.subscriber([removed, subsetEnd]) + // Truncation drains immediately; ordinary removals remain parked. + expect(trace.collection.has(1)).toBe(removal !== `reset`) + acquired = Promise.resolve( + trace.collection._sync.loadSubset({ + where: new IR.Func(`eq`, [new IR.PropRef([`id`]), new IR.Value(2)]), + }), + ) + void acquired.catch(() => {}) + trace.subscriber([change(`update`, 1, `partial`), upToDate]) + persistence.resolve() + await transaction.isPersisted.promise + await acquired + expect(trace.collection.has(1)).toBe(false) + } finally { + persistence.resolve() + await transaction?.isPersisted.promise + await trace.collection.cleanup() + await acquired + } + }, + ) + + fcTest.prop( + [fc.array(designTokenArb, { minLength: 1, maxLength: 7 }), fc.nat()], + { numRuns: 24 }, + )( + `operational and denotational reference designs agree before checking production`, + async (tokens, partitionSeed) => { + const prefix = [[upToDate]] + const messages = [...tokens.map(designMessage), upToDate] + const partitions = everyContiguousPartition(messages) + const selected = [ + [messages], + messages.map((message) => [message]), + partitions[partitionSeed % partitions.length]!, + ].filter(isSdkResetFramedPartition) + + for (const [partitionId, partition] of selected.entries()) { + const operational = expectedSnapshots(prefix, partition) + const denotational = recomputedSnapshots(prefix, partition) + expect(operational).toEqual(denotational) + + for (const syncMode of [`eager`, `on-demand`, `progressive`] as const) { + const actual = await runTrace( + `design-grammar-${syncMode}-${partitionId}`, + syncMode, + prefix, + partition, + ) + expect(actual.snapshots).toEqual(denotational) + } + } + }, + ) + + it(`distinguishes plausible unseen-update and reset-visibility semantics`, async () => { + const completeUnseenUpdate: ChangeMessage = { + key: `1`, + value: { id: 1, name: `complete`, stable: `stable-1` }, + headers: { operation: `update` }, + } + const updateHistory = [[completeUnseenUpdate, upToDate]] + expect(recomputeCommittedRows(updateHistory, `ignore`)).toEqual([]) + expect(recomputeCommittedRows(updateHistory, `promote-complete`)).toEqual([ + [1, `complete`, `stable-1`], + ]) + const updateActual = await runTrace( + `design-unseen-update`, + `eager`, + [], + updateHistory, + ) + expect(updateActual.rows).toEqual( + recomputeCommittedRows(updateHistory, `ignore`), + ) + + const resetHistory = [ + [change(`insert`, 1, `committed`), upToDate], + [mustRefetch], + ] + const atomicReset = recomputedSnapshots([], resetHistory) + const immediateReset: typeof atomicReset = [atomicReset[0]!, []] + expect(atomicReset).not.toEqual(immediateReset) + const resetActual = await runTrace( + `design-reset-visibility`, + `eager`, + [], + resetHistory, + ) + expect(resetActual.snapshots).toEqual(atomicReset) + }) + + it(`keeps SDK reset callbacks separate from every neighboring message kind`, () => { + const neighbors: Array> = [ + change(`insert`, 1, `row`), + change(`update`, 1, `row`), + change(`delete`, 1, `row`), + { headers: { event: `move-out`, patterns: [{ pos: 0, value: `left` }] } }, + upToDate, + subsetEnd, + mustRefetch, + ] + expect(isSdkResetFramedPartition([[mustRefetch]])).toBe(true) + for (const neighbor of neighbors) { + for (const messages of [ + [neighbor, mustRefetch], + [mustRefetch, neighbor], + ]) { + expect( + isSdkResetFramedPartition([messages]), + JSON.stringify(messages), + ).toBe(false) + expect( + isSdkResetFramedPartition(messages.map((message) => [message])), + ).toBe(true) + } + } + }) + + it(`distinguishes callback-atomic and subset publication semantics`, async () => { + const callbackHistory = [ + [ + change(`insert`, 1, `before-control`), + upToDate, + change(`update`, 1, `after-control`), + ], + ] + const callbackAtomic = recomputeCommittedRows(callbackHistory) + const freezeAtControl: typeof callbackAtomic = [ + [1, `before-control`, `stable-1`], + ] + expect(callbackAtomic).toEqual([[1, `after-control`, `stable-1`]]) + expect(callbackAtomic).not.toEqual(freezeAtControl) + expect(isSdkResetFramedPartition([[upToDate, mustRefetch]])).toBe(false) + expect( + isSdkResetFramedPartition([[mustRefetch, subsetEnd, mustRefetch]]), + ).toBe(false) + + const readyPrefix = [[upToDate]] + const subsetHistory = [ + [change(`insert`, 1, `subset-publication`)], + [subsetEnd], + ] + const subsetPublishes = recomputedSnapshots(readyPrefix, subsetHistory) + const upToDateOnly: typeof subsetPublishes = [[], []] + expect(subsetPublishes).not.toEqual(upToDateOnly) + + for (const syncMode of [`eager`, `on-demand`, `progressive`] as const) { + const callbackActual = await runTrace( + `design-callback-atomic-${syncMode}`, + syncMode, + [], + callbackHistory, + ) + expect(callbackActual.rows).toEqual(callbackAtomic) + + const subsetActual = await runTrace( + `design-subset-publication-${syncMode}`, + syncMode, + readyPrefix, + subsetHistory, + ) + expect(subsetActual.snapshots).toEqual(subsetPublishes) + } + }) + + it(`settles every startup, hydration, snapshot availability, commit, and cleanup permutation`, async () => { + const events: Array = [ + `startup-promise`, + `hydration`, + `snapshot-available`, + `commit`, + `cleanup`, + ] + let permutationIndex = 0 + const outcomes = new Set<`resolved` | `aborted`>() + let sawAcknowledgementBeforeDurability = false + for (const order of permutations(events)) { + const currentIndex = permutationIndex++ + try { + const result = await runSchedulerPermutation( + `scheduler-permutation-${currentIndex}`, + order, + ) + outcomes.add(result.outcome) + sawAcknowledgementBeforeDurability ||= result.acknowledgedBeforeDurable + } catch (error) { + throw new Error( + `scheduler permutation ${currentIndex} failed: ${order.join(` → `)}`, + { cause: error }, + ) + } + } + expect(outcomes).toEqual(new Set([`resolved`, `aborted`])) + expect(sawAcknowledgementBeforeDurability).toBe(true) + }, 30_000) + + fcTest.prop( + [ + fc.tuple( + fc.string({ maxLength: 8 }), + fc.string({ maxLength: 8 }), + fc.string({ maxLength: 8 }), + fc.string({ maxLength: 8 }), + ), + ], + { numRuns: 10 }, + )( + `synthetic batch partition is invariant across Electric modes and phases`, + async (names) => { + const [first, second, updated, reinserted] = names + const readyPrefix = [ + [change(`insert`, 1, first), change(`insert`, 2, second), upToDate], + ] + const scenarios: Array = [ + { + name: `bootstrap`, + prefix: [], + messages: [ + change(`insert`, 1, first), + change(`insert`, 2, second), + change(`update`, 1, updated), + change(`delete`, 2, second), + change(`insert`, 2, reinserted), + upToDate, + ], + expectedRows: [ + [1, updated, `stable-1`], + [2, reinserted, `stable-2`], + ], + resume: false, + }, + { + name: `steady`, + prefix: readyPrefix, + messages: [ + change(`update`, 1, updated), + change(`delete`, 2, second), + change(`insert`, 2, reinserted), + upToDate, + ], + expectedRows: [ + [1, updated, `stable-1`], + [2, reinserted, `stable-2`], + ], + resume: false, + }, + { + name: `split-delete-update`, + prefix: [[change(`insert`, 1, first), upToDate]], + messages: [ + change(`delete`, 1, first), + change(`update`, 1, updated), + upToDate, + ], + expectedRows: [], + resume: false, + }, + { + name: `subset`, + prefix: readyPrefix, + messages: [ + change(`update`, 1, updated), + change(`delete`, 2, second), + subsetEnd, + ], + expectedRows: [[1, updated, `stable-1`]], + resume: false, + }, + { + name: `must-refetch`, + prefix: readyPrefix, + messages: [ + mustRefetch, + change(`insert`, 1, first), + change(`update`, 1, updated), + change(`insert`, 2, reinserted), + upToDate, + ], + expectedRows: [ + [1, updated, `stable-1`], + [2, reinserted, `stable-2`], + ], + resume: false, + }, + { + name: `must-refetch-unseen-update`, + prefix: readyPrefix, + messages: [mustRefetch, change(`update`, 1, updated), upToDate], + expectedRows: [], + resume: false, + }, + { + name: `must-refetch-subset`, + prefix: readyPrefix, + messages: [ + mustRefetch, + change(`insert`, 1, first), + subsetEnd, + change(`update`, 1, updated), + upToDate, + ], + expectedRows: [[1, updated, `stable-1`]], + resume: false, + }, + { + name: `resume`, + prefix: [], + messages: [ + change(`insert`, 1, first), + change(`update`, 1, updated), + upToDate, + ], + expectedRows: [[1, updated, `stable-1`]], + resume: true, + }, + ] + + // Keep these stronger adapter robustness checks, including mixed-reset + // callbacks, separate from the SDK-framed differential oracle above. + for (const syncMode of [`eager`, `on-demand`, `progressive`] as const) { + for (const scenario of scenarios) { + const seed = scenario.resume ? resumeState() : new Map() + const atomic = await runTrace( + `${syncMode}-${scenario.name}-atomic`, + syncMode, + scenario.prefix, + [scenario.messages], + seed, + ) + expect(atomic.rows).toEqual(scenario.expectedRows) + expect(atomic.status).toBe(`ready`) + expect(atomic.resume).toEqual( + expect.objectContaining({ kind: `resume`, offset: `20_0` }), + ) + + let partitionId = 0 + for (const partition of everyContiguousPartition(scenario.messages)) { + const currentPartition = partitionId++ + const split = await runTrace( + `${syncMode}-${scenario.name}-${currentPartition}`, + syncMode, + scenario.prefix, + partition, + seed, + ) + expect( + { + rows: split.rows, + status: split.status, + resume: split.resume, + }, + `${syncMode}/${scenario.name}/partition-${currentPartition}: ${JSON.stringify({ partition, atomic, split })}`, + ).toEqual({ + rows: atomic.rows, + status: atomic.status, + resume: atomic.resume, + }) + expect(split.snapshots).toEqual( + expectedSnapshots(scenario.prefix, partition), + ) + } + } + } + }, + 30_000, + ) + + fcTest.prop( + [ + fc.array( + fc.record({ + operation: fc.constantFrom( + `insert`, + `update`, + `delete`, + ), + id: fc.integer({ min: 1, max: 3 }), + name: fc.string({ maxLength: 8 }), + }), + { minLength: 1, maxLength: 5 }, + ), + ], + { numRuns: 20 }, + )( + `generated valid histories are invariant under every batch partition`, + async (tokens) => { + const history = buildValidHistory(tokens) + for (const syncMode of [`eager`, `on-demand`, `progressive`] as const) { + let partitionId = 0 + for (const partition of everyContiguousPartition(history.messages)) { + const result = await runTrace( + `generated-${syncMode}-${partitionId++}`, + syncMode, + [], + partition, + ) + expect(result.rows).toEqual(history.expectedRows) + expect(result.snapshots).toEqual(expectedSnapshots([], partition)) + expect(result.status).toBe(`ready`) + } + } + }, + ) + + fcTest.prop( + [fc.array(designTokenArb, { minLength: 1, maxLength: 7 }), fc.nat()], + { + numRuns: 20, + examples: [ + [ + [ + { operation: `reset` }, + { operation: `subset` }, + { operation: `reset` }, + ], + 30, + ], + ], + }, + )( + `denotational reference, Electric, persisted Electric, and query adapters converge across controls and publication epochs`, + async (tokens, partitionSeed) => { + const messages = buildDifferentialHistory(tokens) + const partitions = everyContiguousPartition(messages).filter( + isSdkResetFramedPartition, + ) + const partition = partitions[partitionSeed % partitions.length]! + const referenceSnapshots = recomputedSnapshots([], partition) + + for (const syncMode of [`eager`, `on-demand`, `progressive`] as const) { + const direct = await runTrace( + `direct-differential-${syncMode}`, + syncMode, + [], + partition, + ) + const persisted = await runPersistedTrace( + `persisted-differential-${syncMode}`, + syncMode, + partition, + ) + const query = await runQueryTrace( + `query-differential-${syncMode}`, + partition, + ) + + expect(direct.snapshots).toEqual(referenceSnapshots) + expect(persisted.snapshots).toEqual(referenceSnapshots) + expect(query.snapshots).toEqual(referenceSnapshots) + expect(persisted.rows).toEqual(direct.rows) + expect(query.rows).toEqual(direct.rows) + expect(persisted.status).toBe(direct.status) + expect(query.status).toBe(`ready`) + expect(persisted.resume).toEqual(direct.resume) + expect(persisted.durableRows).toEqual(direct.rows) + expect(persisted.durableResume).toEqual(direct.resume) + expect(persisted.persistenceCommits).toBeGreaterThan(0) + } + }, + 30_000, + ) + + fcTest.prop( + [ + fc.integer({ min: 1, max: 20 }), + fc.string({ maxLength: 8 }), + fc.string({ maxLength: 8 }), + ], + { numRuns: 20 }, + )( + `generated invalid resume transitions fail under every batch partition`, + async (id, completeName, partialName) => { + const messages = [ + change(`delete`, id, completeName), + change(`update`, id, partialName), + upToDate, + ] + + for (const syncMode of [`eager`, `progressive`] as const) { + for (const removal of [`delete`, `move-out`] as const) { + const removed = + removal === `delete` + ? messages[0]! + : ({ + headers: { + event: `move-out`, + patterns: [{ pos: 0, value: `left` }], + }, + } as Message) + for (const [partitionId, partition] of everyContiguousPartition([ + removed, + ...messages.slice(1), + ]).entries()) { + const metadata = createMetadata(resumeState()) + const trace = createOracleCollection( + `generated-invalid-${syncMode}-${id}-${partitionId}`, + syncMode, + metadata.api, + ) + const inserted = change(`insert`, id, completeName) + trace.subscriber([ + { ...inserted, headers: { ...inserted.headers, tags: [`left`] } }, + upToDate, + ]) + for (const batch of partition) trace.subscriber(batch) + + expect(trace.collection.status).toBe(`error`) + expect(trace.collection.get(id)).toEqual( + expect.objectContaining({ stable: `stable-${id}` }), + ) + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `reset` }), + ) + await trace.collection.cleanup() + } + } + } + }, + ) + + for (const syncMode of [`eager`, `progressive`] as const) { + it(`rejects unseen resumed updates and recovers on the next ${syncMode} lifecycle`, async () => { + const metadata = createMetadata(resumeState()) + const old = createOracleCollection( + `invalid-${syncMode}-resume`, + syncMode, + metadata.api, + ) + + old.subscriber([change(`update`, 1, `partial`)]) + expect(old.collection.status).toBe(`error`) + expect(old.collection.has(1)).toBe(false) + expect(old.unsubscribe).toHaveBeenCalledOnce() + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `reset` }), + ) + + old.subscriber([change(`insert`, 1, `old generation`), upToDate]) + expect(old.collection.has(1)).toBe(false) + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `reset` }), + ) + await old.collection.cleanup() + + const fresh = createOracleCollection( + `fresh-${syncMode}-snapshot`, + syncMode, + metadata.api, + ) + fresh.subscriber([change(`insert`, 1, `complete snapshot`), upToDate]) + + expect(fresh.collection.status).toBe(`ready`) + expect(fresh.collection.get(1)?.name).toBe(`complete snapshot`) + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `resume`, offset: `20_0` }), + ) + await fresh.collection.cleanup() + }) + + it(`treats delete then update as an invalid ${syncMode} resume`, async () => { + const metadata = createMetadata(resumeState()) + const trace = createOracleCollection( + `delete-update-${syncMode}-resume`, + syncMode, + metadata.api, + ) + + trace.subscriber([ + change(`insert`, 1, `seen`), + change(`delete`, 1, `seen`), + change(`update`, 1, `partial`), + ]) + + expect(trace.collection.status).toBe(`error`) + expect(trace.collection.has(1)).toBe(false) + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `reset` }), + ) + await trace.collection.cleanup() + }) + + it(`rejects delete then update across every ${syncMode} resume partition`, async () => { + const messages = [ + change(`delete`, 1, `complete`), + change(`update`, 1, `partial`), + upToDate, + ] + + for (const [partitionId, partition] of everyContiguousPartition( + messages, + ).entries()) { + const metadata = createMetadata(resumeState()) + const trace = createOracleCollection( + `invalid-partition-${syncMode}-${partitionId}`, + syncMode, + metadata.api, + ) + trace.subscriber([change(`insert`, 1, `complete`), upToDate]) + for (const batch of partition) trace.subscriber(batch) + + expect(trace.collection.status).toBe(`error`) + expect(trace.collection.get(1)).toEqual( + expect.objectContaining({ stable: `stable-1` }), + ) + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `reset` }), + ) + await trace.collection.cleanup() + } + }) + } + + it(`keeps stream cleanup and stale callbacks scoped to their lifecycle`, async () => { + const subscribers: Array<(messages: Array>) => void> = [] + const unsubscribes = [vi.fn(), vi.fn()] + mockSubscribe.mockImplementation((callback) => { + const index = subscribers.length + subscribers.push(callback) + return unsubscribes[index]! + }) + const options = electricCollectionOptions({ + id: `reused-sync-config`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: true, + }) + const createLifecycle = (id: string) => { + const metadata = createMetadata(resumeState()) + const lifecycleSync = options.sync + return createCollection({ + ...options, + id, + sync: { + ...lifecycleSync, + sync: (params: Parameters[0]) => + lifecycleSync.sync({ ...params, metadata: metadata.api }), + }, + }) + } + + const oldCollection = createLifecycle(`old-sync-lifecycle`) + const currentCollection = createLifecycle(`current-sync-lifecycle`) + + subscribers[0]!([change(`insert`, 1, `first row`), upToDate]) + subscribers[1]!([change(`insert`, 2, `second row`), upToDate]) + expect(oldCollection.get(1)?.name).toBe(`first row`) + expect(currentCollection.get(2)?.name).toBe(`second row`) + subscribers[0]!([change(`update`, 1, `first row updated`), upToDate]) + const currentMatch = currentCollection.utils.awaitMatch( + (message) => `value` in message && message.value.id === 3, + 100, + ) + const currentTxid = currentCollection.utils.awaitTxId(42, 100) + const currentSnapshotTxid = currentCollection.utils.awaitTxId(50, 100) + let currentMatchResolved = false + let currentTxidResolved = false + let currentSnapshotTxidResolved = false + void currentMatch.then(() => { + currentMatchResolved = true + }) + void currentTxid.then(() => { + currentTxidResolved = true + }) + void currentSnapshotTxid.then(() => { + currentSnapshotTxidResolved = true + }) + subscribers[0]!([ + change(`insert`, 3, `wrong lifecycle`), + { + headers: { + control: `snapshot-end`, + xmin: `100`, + xmax: `150`, + xip_list: [], + }, + }, + { headers: { control: `up-to-date`, txids: [42] } }, + ]) + await Promise.resolve() + expect(currentMatchResolved).toBe(false) + expect(currentTxidResolved).toBe(false) + expect(currentSnapshotTxidResolved).toBe(false) + + await oldCollection.cleanup() + + expect(unsubscribes[0]).toHaveBeenCalledOnce() + expect(unsubscribes[1]).not.toHaveBeenCalled() + + subscribers[0]!([change(`update`, 1, `stale`)]) + expect(unsubscribes[1]).not.toHaveBeenCalled() + subscribers[1]!([ + change(`insert`, 3, `still live`), + { + headers: { + control: `snapshot-end`, + xmin: `100`, + xmax: `150`, + xip_list: [], + }, + }, + { headers: { control: `up-to-date`, txids: [42] } }, + ]) + await expect(currentMatch).resolves.toBe(true) + await expect(currentTxid).resolves.toBe(true) + await expect(currentSnapshotTxid).resolves.toBe(true) + expect(currentCollection.get(3)?.name).toBe(`still live`) + await currentCollection.cleanup() + expect(unsubscribes[1]).toHaveBeenCalledOnce() + }) + + it(`binds sync metadata import and export to the receiving collection`, async () => { + const subscribers: Array<(messages: Array>) => void> = [] + mockSubscribe.mockImplementation((callback) => { + subscribers.push(callback) + return vi.fn() + }) + const options = electricCollectionOptions({ + id: `shared-metadata-config`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: true, + }) + const first = createCollection({ ...options, id: `metadata-first` }) + const second = createCollection({ ...options, id: `metadata-second` }) + + subscribers[0]!([{ headers: { control: `up-to-date`, txids: [11] } }]) + subscribers[1]!([{ headers: { control: `up-to-date`, txids: [22] } }]) + + expect(first.config.sync.exportSyncMeta?.()).toMatchObject({ + version: 1, + seenTxids: [11], + }) + expect(second.config.sync.exportSyncMeta?.()).toMatchObject({ + version: 1, + seenTxids: [22], + }) + + const importedResume = { + kind: `resume` as const, + requiresTagState: false, + offset: `7_0` as const, + handle: `shape-7`, + shapeId, + updatedAt: 7, + } + first.config.sync.importSyncMeta?.({ + version: 1, + resume: importedResume, + seenTxids: [99], + }) + + await expect(first.utils.awaitTxId(99, 50)).resolves.toBe(true) + await expect(second.utils.awaitTxId(99, 10)).rejects.toThrow() + expect(first.config.sync.exportSyncMeta?.()).toEqual({ + version: 1, + resume: importedResume, + seenTxids: [99], + }) + expect(second.config.sync.exportSyncMeta?.()).toMatchObject({ + version: 1, + seenTxids: [22], + }) + expect(second.config.sync.exportSyncMeta?.()).not.toMatchObject({ + resume: importedResume, + }) + + const third = createCollection({ ...options, id: `metadata-third` }) + await expect(third.utils.awaitTxId(99, 10)).rejects.toThrow() + expect(third.config.sync.exportSyncMeta?.()).not.toMatchObject({ + resume: importedResume, + }) + + await first.cleanup() + await second.cleanup() + await third.cleanup() + }) + + it(`consumes raw sync metadata when the next collection is materialized`, async () => { + mockSubscribe.mockImplementation(() => vi.fn()) + const options = electricCollectionOptions({ + id: `seeded-metadata-config`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: true, + }) + const importedResume = { + kind: `resume` as const, + requiresTagState: false, + offset: `8_0` as const, + handle: `shape-8`, + shapeId, + updatedAt: 8, + } + options.sync.importSyncMeta?.({ + version: 1, + resume: importedResume, + seenTxids: [55], + }) + + const seeded = createCollection({ ...options, id: `metadata-seeded` }) + await expect(seeded.utils.awaitTxId(55, 50)).resolves.toBe(true) + expect(seeded.config.sync.exportSyncMeta?.()).toMatchObject({ + resume: importedResume, + seenTxids: [55], + }) + + const unseeded = createCollection({ ...options, id: `metadata-unseeded` }) + await expect(unseeded.utils.awaitTxId(55, 10)).rejects.toThrow() + expect(unseeded.config.sync.exportSyncMeta?.()).toEqual({ + version: 1, + seenTxids: [], + }) + + await seeded.cleanup() + await unseeded.cleanup() + }) + + it(`binds imported evidence and pending matches before lazy sync starts`, async () => { + let subscriber!: (messages: Array>) => void + mockSubscribe.mockImplementation((callback) => { + subscriber = callback + return vi.fn() + }) + const options = electricCollectionOptions({ + id: `lazy-bound-evidence`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: false, + }) + const collection = createCollection(options) + const importedResume = { + kind: `resume` as const, + requiresTagState: false, + offset: `9_0` as const, + handle: `shape-9`, + shapeId, + updatedAt: 9, + } + collection.config.sync.importSyncMeta?.({ + version: 1, + resume: importedResume, + seenTxids: [33], + }) + + await expect(collection.utils.awaitTxId(33, 20)).resolves.toBe(true) + const pendingMatch = collection.utils.awaitMatch( + (message) => `value` in message && message.value.id === 5, + 100, + ) + const pendingTxid = collection.utils.awaitTxId(34, 100) + const preload = collection.preload() + expect(vi.mocked(ShapeStream).mock.calls.at(-1)?.[0]).toMatchObject({ + offset: `9_0`, + handle: `shape-9`, + }) + const evidenceChange = change(`insert`, 5, `matched after start`) + subscriber([ + { + ...evidenceChange, + headers: { operation: `insert`, txids: [34] }, + }, + upToDate, + ]) + + await expect(pendingMatch).resolves.toBe(true) + await expect(pendingTxid).resolves.toBe(true) + await preload + expect(collection.get(5)?.name).toBe(`matched after start`) + await collection.cleanup() + }) + + it(`keeps descriptor utilities captured before collection startup live`, async () => { + let subscriber!: (messages: Array>) => void + mockSubscribe.mockImplementation((callback) => { + subscriber = callback + return vi.fn() + }) + const options = electricCollectionOptions({ + id: `captured-descriptor-utilities`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: false, + }) + const { awaitMatch, awaitTxId } = options.utils + const collection = createCollection(options) + const pendingMatch = awaitMatch( + (message) => `value` in message && message.value.id === 77, + 100, + ) + const pendingTxid = awaitTxId(77, 100) + + const preload = collection.preload() + const evidenceChange = change(`insert`, 77, `captured utilities`) + subscriber([ + { + ...evidenceChange, + headers: { operation: `insert`, txids: [77] }, + }, + upToDate, + ]) + + await expect(pendingMatch).resolves.toBe(true) + await expect(pendingTxid).resolves.toBe(true) + await preload + await collection.cleanup() + }) + + it(`retires a pending pre-start match when a lazy collection is cleaned up`, async () => { + mockSubscribe.mockImplementation(() => vi.fn()) + const collection = createCollection( + electricCollectionOptions({ + id: `lazy-pre-start-cleanup`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: false, + }), + ) + const pendingMatch = collection.utils.awaitMatch(() => false, 100) + + await collection.cleanup() + + await expect(pendingMatch).rejects.toThrow(/aborted/i) + expect(mockSubscribe).not.toHaveBeenCalled() + }) + + it(`retires pre-start waiters when persisted metadata startup is interrupted`, async () => { + const metadataStarted = createDeferred() + const metadataGate = createDeferred() + const adapter = createPersistedAdapter(new Map(), new Map()) + adapter.loadCollectionMetadata = async () => { + metadataStarted.resolve() + await metadataGate.promise + return [] + } + const collection = createCollection( + persistedCollectionOptions< + OracleRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricCollectionOptions({ + id: `persisted-startup-waiter-cleanup`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: false, + }), + persistence: { adapter }, + }), + ) + const pendingMatch = collection.utils.awaitMatch(() => false, 100) + const pendingTxid = collection.utils.awaitTxId(702, 100) + const preload = expect(collection.preload()).rejects.toMatchObject({ + name: `AbortError`, + }) + await metadataStarted.promise + const matchOutcome = expect(pendingMatch).rejects.toThrow(/aborted/i) + const txidOutcome = expect(pendingTxid).rejects.toThrow(/aborted/i) + + await collection.cleanup() + + await matchOutcome + await txidOutcome + expect(mockSubscribe).not.toHaveBeenCalled() + metadataGate.resolve() + await preload + }) + + it(`retires pre-start waiters through automatic collection GC`, async () => { + const metadataGate = createDeferred() + const adapter = createPersistedAdapter(new Map(), new Map()) + adapter.loadCollectionMetadata = vi.fn(async () => { + await metadataGate.promise + return [] + }) + const collection = createCollection( + persistedCollectionOptions< + OracleRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricCollectionOptions({ + id: `persisted-gc-waiter-cleanup`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: false, + gcTime: 10, + }), + persistence: { adapter }, + }), + ) + const pendingMatch = collection.utils.awaitMatch(() => false, 5000) + const pendingTxid = collection.utils.awaitTxId(703, 5000) + // A pending preload owns retention; exercise unowned sync for automatic GC. + collection.startSyncImmediate() + await vi.waitFor( + () => expect(adapter.loadCollectionMetadata).toHaveBeenCalledOnce(), + { interval: 1, timeout: 250 }, + ) + const subscription = collection.subscribeChanges(() => {}) + const matchOutcome = pendingMatch.catch((error: unknown) => error) + const txidOutcome = pendingTxid.catch((error: unknown) => error) + + subscription.unsubscribe() + await vi.waitFor(() => expect(collection.status).toBe(`cleaned-up`), { + interval: 10, + timeout: 1500, + }) + const pendingSentinel = Symbol(`pending`) + const matchResult = await Promise.race([ + matchOutcome, + new Promise((resolve) => setTimeout(() => resolve(pendingSentinel), 50)), + ]) + const txidResult = await Promise.race([ + txidOutcome, + new Promise((resolve) => setTimeout(() => resolve(pendingSentinel), 50)), + ]) + + expect(matchResult).toEqual( + expect.objectContaining({ + message: expect.stringMatching(/aborted/i), + }), + ) + expect(txidResult).toEqual( + expect.objectContaining({ + message: expect.stringMatching(/aborted/i), + }), + ) + expect(mockSubscribe).not.toHaveBeenCalled() + metadataGate.resolve() + }) + + it(`retires every pending waiter when its collection lifecycle is cleaned up`, async () => { + mockSubscribe.mockImplementation(() => vi.fn()) + const lazy = createCollection( + electricCollectionOptions({ + id: `lazy-waiter-cleanup`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: false, + }), + ) + const lazyTxid = lazy.utils.awaitTxId(700, 100) + await lazy.cleanup() + await expect(lazyTxid).rejects.toThrow(/aborted/i) + + const active = createOracleCollection( + `active-waiter-cleanup`, + `eager`, + createMetadata(new Map()).api, + ) + const activeMatch = active.collection.utils.awaitMatch(() => false, 100) + const activeTxid = active.collection.utils.awaitTxId(701, 100) + await active.collection.cleanup() + + await expect(activeMatch).rejects.toThrow(/aborted/i) + await expect(activeTxid).rejects.toThrow(/aborted/i) + }) + + it(`settles waiters according to whether evidence or cleanup wins`, async () => { + for (const cleanupWins of [true, false]) { + const trace = createOracleCollection( + `waiter-race-${cleanupWins}`, + `eager`, + createMetadata(new Map()).api, + ) + const pendingMatch = trace.collection.utils.awaitMatch( + (message) => `value` in message && message.value.id === 91, + 100, + ) + const pendingTxid = trace.collection.utils.awaitTxId(91, 100) + const evidenceChange = change(`insert`, 91, `race winner`) + const evidence: Array> = [ + { + ...evidenceChange, + headers: { operation: `insert`, txids: [91] }, + }, + upToDate, + ] + + if (cleanupWins) { + await trace.collection.cleanup() + trace.subscriber(evidence) + await expect(pendingMatch).rejects.toThrow(/aborted/i) + await expect(pendingTxid).rejects.toThrow(/aborted/i) + expect(trace.collection.get(91)).toBeUndefined() + } else { + trace.subscriber(evidence) + await trace.collection.cleanup() + await expect(pendingMatch).resolves.toBe(true) + await expect(pendingTxid).resolves.toBe(true) + } + } + }) + + it(`keeps committed match evidence across newer writer batches`, async () => { + const metadata = createMetadata(new Map()) + const trace = createOracleCollection( + `await-match-batch-generation`, + `eager`, + metadata.api, + ) + + trace.subscriber([change(`insert`, 1, `old batch`), upToDate]) + trace.subscriber([change(`insert`, 2, `current batch`), upToDate]) + + await expect( + trace.collection.utils.awaitMatch( + (message) => `value` in message && message.value.name === `old batch`, + 20, + ), + ).resolves.toBe(true) + await trace.collection.cleanup() + }) + + it(`keeps committed match evidence across callbacks with no new data`, async () => { + const trace = createOracleCollection( + `await-match-neutral-callback`, + `eager`, + createMetadata(new Map()).api, + ) + + trace.subscriber([change(`insert`, 1, `committed`), upToDate]) + trace.subscriber([ + { + headers: { + control: `snapshot-end`, + xmin: `100`, + xmax: `150`, + xip_list: [], + }, + }, + ]) + trace.subscriber([]) + + await expect( + trace.collection.utils.awaitMatch( + (message) => `value` in message && message.value.id === 1, + 20, + ), + ).resolves.toBe(true) + await trace.collection.cleanup() + }) + + it(`clears committed match evidence when the stream must refetch`, async () => { + const trace = createOracleCollection( + `await-match-reset-generation`, + `eager`, + createMetadata(new Map()).api, + ) + + trace.subscriber([change(`insert`, 1, `old generation`), upToDate]) + trace.subscriber([mustRefetch, upToDate]) + + await expect( + trace.collection.utils.awaitMatch( + (message) => + `value` in message && message.value.name === `old generation`, + 20, + ), + ).rejects.toThrow(/Timeout waiting for custom match function/) + await trace.collection.cleanup() + }) + + const reentryHistory = fc.record({ + txid: fc.integer({ min: 1, max: 10000 }), + names: fc.array(fc.string({ maxLength: 8 }), { + minLength: 1, + maxLength: 5, + }), + trigger: fc.nat({ max: 4 }), + }) + + async function runReentryHistory(history: { + txid: number + names: Array + trigger: number + }) { + // Same ownership law, with retirement either outside or inside a callback. + // Previously the grammar only retired a session between complete callbacks. + for (const insideCallback of [false, true]) { + const subscribers: Array<(messages: Array>) => void> = + [] + mockSubscribe.mockImplementation((callback) => { + subscribers.push(callback) + return vi.fn() + }) + const collection = createCollection( + electricCollectionOptions({ + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: true, + }), + ) + const triggerId = (history.trigger % history.names.length) + 1 + let retired = false + let cleanup: Promise | undefined + let replacementVisits = 0 + let replacementOutcome: Promise | undefined + const restart = () => { + retired = true + cleanup = collection.cleanup() + collection.startSyncImmediate() + replacementOutcome = collection.utils + .awaitMatch(() => { + replacementVisits++ + return false + }) + .catch(() => undefined) + } + const waiting = collection.utils.awaitMatch((message) => { + if ( + insideCallback && + !retired && + `value` in message && + message.value.id === triggerId + ) + restart() + return false + }, 20) + // Observe the old waiter before any callback can retire its session. + const oldOutcome = waiting.then( + () => `resolved`, + () => `rejected`, + ) + try { + if (!insideCallback) restart() + const messages = history.names.map((name, index) => + change(`insert`, index + 1, name), + ) + const acknowledgement: Message = { + headers: { control: `up-to-date`, txids: [history.txid] }, + } + subscribers[0]!([...messages, acknowledgement]) + await cleanup + expect(subscribers).toHaveLength(2) + expect(await oldOutcome).toBe(`rejected`) + expect(replacementVisits).toBe(0) + // The old callback's tail is not evidence from the replacement stream. + // This is the utility's own deadline, not a sleep used to infer readiness. + await expect( + collection.utils.awaitTxId(history.txid, 10), + ).rejects.toThrow(/Timeout/) + expect(collection.size).toBe(0) + subscribers[1]!([...messages, acknowledgement]) + await expect( + collection.utils.awaitTxId(history.txid, 10), + ).resolves.toBe(true) + expect( + [...collection.values()].map((row) => ({ + id: row.id, + name: row.name, + })), + ).toEqual(history.names.map((name, index) => ({ id: index + 1, name }))) + } finally { + await collection.cleanup() + await oldOutcome + await replacementOutcome + } + } + } + + fcTest.prop([reentryHistory], { seed: 42713, numRuns: oracleRuns(6) })( + `replacement sessions reject evidence from callback reentry histories (fixed)`, + runReentryHistory, + ) + fcTest.prop( + [reentryHistory], + oraclePropertyOptions(10, `electric.match-reentry`), + )( + `replacement sessions reject evidence from callback reentry histories (random)`, + runReentryHistory, + ) + + it(`does not let a committed message satisfy awaitMatch after restart`, async () => { + const subscribers: Array<(messages: Array>) => void> = [] + mockSubscribe.mockImplementation((callback) => { + subscribers.push(callback) + return vi.fn() + }) + const collection = createCollection( + electricCollectionOptions({ + id: `await-match-lifecycle-generation`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: true, + }), + ) + + subscribers[0]!([change(`insert`, 1, `old lifecycle`), upToDate]) + await collection.cleanup() + collection.startSyncImmediate() + await vi.waitFor(() => expect(subscribers).toHaveLength(2)) + + await expect( + collection.utils.awaitMatch( + (message) => + `value` in message && message.value.name === `old lifecycle`, + 20, + ), + ).rejects.toThrow(/Timeout waiting for custom match function/) + await collection.cleanup() + }) + + for (const syncMode of [`eager`, `on-demand`, `progressive`] as const) { + it(`does not apply an unseen ${syncMode} update after must-refetch`, async () => { + const metadata = createMetadata(new Map()) + const trace = createOracleCollection( + `must-refetch-unseen-${syncMode}`, + syncMode, + metadata.api, + ) + trace.subscriber([change(`insert`, 1, `complete`), upToDate]) + + trace.subscriber([mustRefetch, change(`update`, 1, `partial`), upToDate]) + + expect(trace.collection.status).toBe(`ready`) + expect(trace.collection.has(1)).toBe(false) + await trace.collection.cleanup() + }) + + it(`keeps the ${syncMode} reset marker until the replacement snapshot is complete`, async () => { + const metadata = createMetadata(resumeState()) + const trace = createOracleCollection( + `durable-reset-${syncMode}`, + syncMode, + metadata.api, + ) + + trace.subscriber([mustRefetch]) + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `reset` }), + ) + + trace.subscriber([change(`insert`, 1, `partial snapshot`), subsetEnd]) + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `reset` }), + ) + + trace.subscriber([ + change(`update`, 2, `unseen`), + change(`update`, 1, `complete snapshot`), + upToDate, + ]) + expect(trace.collection.status).toBe(`ready`) + expect(trace.collection.get(1)).toEqual( + expect.objectContaining({ + id: 1, + name: `complete snapshot`, + stable: `stable-1`, + }), + ) + expect(trace.collection.has(2)).toBe(false) + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `resume`, offset: `20_0` }), + ) + await trace.collection.cleanup() + }) + } + + it(`does not let an older applied receipt finish a newer reset`, async () => { + const metadata = createMetadata(resumeState()) + const trace = createOracleCollection( + `overlapping-reset-generations`, + `eager`, + metadata.api, + ) + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + + trace.subscriber([ + mustRefetch, + change(`insert`, 1, `first replacement`), + subsetEnd, + ]) + transaction.mutate(() => + trace.collection.insert({ + id: 99, + name: `optimistic`, + stable: `stable-99`, + }), + ) + trace.subscriber([upToDate]) + trace.subscriber([mustRefetch]) + await Promise.resolve() + + trace.subscriber([change(`update`, 2, `unseen`), upToDate]) + + expect(trace.collection.status).not.toBe(`error`) + expect(trace.collection.has(2)).toBe(false) + + persistence.resolve() + await transaction.isPersisted.promise + await trace.collection.cleanup() + }) + + it(`waits for a deferred applied receipt before publishing readiness`, async () => { + const metadata = createMetadata(new Map()) + const trace = createOracleCollection( + `deferred-applied-receipt`, + `eager`, + metadata.api, + ) + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => + trace.collection.insert({ + id: 99, + name: `optimistic`, + stable: `stable-99`, + }), + ) + + trace.subscriber([change(`insert`, 1, `synced`), upToDate]) + await Promise.resolve() + + expect(trace.collection.status).toBe(`loading`) + expect(trace.collection.has(1)).toBe(false) + + persistence.resolve() + await transaction.isPersisted.promise + await trace.collection.stateWhenReady() + + expect(trace.collection.status).toBe(`ready`) + expect(trace.collection.get(1)).toEqual( + expect.objectContaining({ stable: `stable-1` }), + ) + await trace.collection.cleanup() + }) + + it(`does not publish readiness after a parked receipt is rejected by cleanup`, async () => { + const metadata = createMetadata(new Map()) + const trace = createOracleCollection( + `rejected-applied-receipt`, + `eager`, + metadata.api, + ) + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => + trace.collection.insert({ + id: 99, + name: `optimistic`, + stable: `stable-99`, + }), + ) + trace.subscriber([change(`insert`, 1, `synced`), upToDate]) + await Promise.resolve() + expect(trace.collection.status).toBe(`loading`) + + await trace.collection.cleanup() + persistence.resolve() + await transaction.isPersisted.promise + await Promise.resolve() + + expect(trace.collection.status).toBe(`cleaned-up`) + expect(trace.collection.has(1)).toBe(false) + }) + + it(`accepts partial resumed updates for rows restored by persistence`, async () => { + let subscriber!: (messages: Array>) => void + mockSubscribe.mockImplementationOnce((callback) => { + subscriber = callback + return vi.fn() + }) + const collectionMetadata = new Map(resumeState()) + const persistedRows = new Map([ + [ + 1, + { + id: 1, + name: `persisted`, + stable: `stable-1`, + }, + ], + ]) + const electricOptions = electricCollectionOptions({ + id: `persisted-resume-oracle`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `progressive`, + getKey: (row) => row.id, + startSync: true, + }) + const collection = createCollection( + persistedCollectionOptions< + OracleRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricOptions, + persistence: { + adapter: createPersistedAdapter(collectionMetadata, persistedRows), + }, + }), + ) + + collection.startSyncImmediate() + await collection._sync.loadSubset({ limit: 10 }) + expect(collection.get(1)).toEqual( + expect.objectContaining({ name: `persisted`, stable: `stable-1` }), + ) + expect(vi.mocked(ShapeStream).mock.calls.at(-1)?.[0]).toMatchObject({ + offset: `10_0`, + handle: `shape-1`, + }) + + subscriber([change(`update`, 1, `resumed update`), upToDate]) + await vi.waitFor(() => expect(collection.status).toBe(`ready`)) + + expect(collection.get(1)).toEqual( + expect.objectContaining({ + name: `resumed update`, + stable: `stable-1`, + }), + ) + await vi.waitFor(() => { + expect(persistedRows.get(1)).toEqual( + expect.objectContaining({ + name: `resumed update`, + stable: `stable-1`, + }), + ) + }) + await collection.cleanup() + }) + + it(`buffers resumed updates that arrive while persisted rows are hydrating`, async () => { + let subscriber: ((messages: Array>) => void) | undefined + mockSubscribe.mockImplementationOnce((callback) => { + subscriber = callback + return vi.fn() + }) + const hydration = createDeferred() + const collectionMetadata = new Map(resumeState()) + const persistedRows = new Map([ + [ + 1, + { + id: 1, + name: `persisted`, + stable: `stable-1`, + }, + ], + ]) + const electricOptions = electricCollectionOptions({ + id: `concurrent-persisted-resume-oracle`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `progressive`, + getKey: (row) => row.id, + startSync: true, + }) + const collection = createCollection( + persistedCollectionOptions< + OracleRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricOptions, + persistence: { + adapter: createPersistedAdapter( + collectionMetadata, + persistedRows, + hydration.promise, + ), + }, + }), + ) + + collection.startSyncImmediate() + await vi.waitFor(() => expect(subscriber).toBeTypeOf(`function`)) + expect(collection.status).not.toBe(`error`) + subscriber!([change(`update`, 1, `concurrent update`), upToDate]) + + expect(collection.status).not.toBe(`error`) + hydration.resolve() + await vi.waitFor(() => expect(collection.status).toBe(`ready`)) + expect(collection.get(1)).toEqual( + expect.objectContaining({ + name: `concurrent update`, + stable: `stable-1`, + }), + ) + await collection.cleanup() + }) + + it(`rehydrates persisted rows and resume metadata after cleanup and restart`, async () => { + const subscribers: Array<(messages: Array>) => void> = [] + mockSubscribe.mockImplementation((callback) => { + subscribers.push(callback) + return vi.fn() + }) + const collectionMetadata = new Map(resumeState()) + const persistedRows = new Map([ + [1, { id: 1, name: `persisted`, stable: `stable-1` }], + ]) + const collection = createCollection( + persistedCollectionOptions< + OracleRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricCollectionOptions({ + id: `persisted-restart-oracle`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `progressive`, + getKey: (row) => row.id, + startSync: true, + }), + persistence: { + adapter: createPersistedAdapter(collectionMetadata, persistedRows), + }, + }), + ) + + collection.startSyncImmediate() + await vi.waitFor(() => expect(collection.get(1)?.name).toBe(`persisted`)) + await vi.waitFor(() => expect(subscribers).toHaveLength(1)) + subscribers[0]!([upToDate]) + await collection.stateWhenReady() + + await collection.cleanup() + collection.startSyncImmediate() + await vi.waitFor(() => expect(subscribers).toHaveLength(2)) + await vi.waitFor(() => expect(collection.get(1)?.name).toBe(`persisted`)) + expect(collectionMetadata.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `resume` }), + ) + await collection.cleanup() + }) + + it(`does not let hydration from a cleaned-up lifecycle poison restart`, async () => { + const subscribers: Array<(messages: Array>) => void> = [] + mockSubscribe.mockImplementation((callback) => { + subscribers.push(callback) + return vi.fn() + }) + const firstHydration = createDeferred() + const collectionMetadata = new Map(resumeState()) + const persistedRows = new Map([ + [1, { id: 1, name: `current`, stable: `stable-1` }], + ]) + const adapter = createPersistedAdapter(collectionMetadata, persistedRows) + let hydrationCall = 0 + adapter.loadSubset = vi.fn(async () => { + hydrationCall++ + if (hydrationCall === 1) { + await firstHydration.promise + return [ + { + key: 1, + value: { id: 1, name: `stale`, stable: `stable-1` }, + }, + ] + } + return Array.from(persistedRows, ([key, value]) => ({ key, value })) + }) + const collection = createCollection( + persistedCollectionOptions< + OracleRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricCollectionOptions({ + id: `persisted-stale-hydration-oracle`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `progressive`, + getKey: (row) => row.id, + startSync: true, + }), + persistence: { adapter }, + }), + ) + + collection.startSyncImmediate() + await vi.waitFor(() => expect(adapter.loadSubset).toHaveBeenCalledTimes(1)) + await collection.cleanup() + collection.startSyncImmediate() + await vi.waitFor(() => expect(subscribers).toHaveLength(2)) + + firstHydration.resolve() + await vi.waitFor(() => expect(adapter.loadSubset).toHaveBeenCalledTimes(2)) + await vi.waitFor(() => expect(collection.get(1)?.name).toBe(`current`)) + subscribers[1]!([upToDate]) + await collection.stateWhenReady() + await collection.cleanup() + }) + + it(`does not hydrate persisted on-demand rows before subset demand`, async () => { + let subscriber: ((messages: Array>) => void) | undefined + mockSubscribe.mockImplementationOnce((callback) => { + subscriber = callback + return vi.fn() + }) + const collectionMetadata = new Map() + const persistedRows = new Map([ + [1, { id: 1, name: `persisted`, stable: `stable-1` }], + ]) + const adapter = createPersistedAdapter(collectionMetadata, persistedRows) + const loadSubset = vi.fn(adapter.loadSubset) + adapter.loadSubset = loadSubset + const electricOptions = electricCollectionOptions({ + id: `persisted-on-demand-oracle`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `on-demand`, + getKey: (row) => row.id, + startSync: true, + }) + const collection = createCollection( + persistedCollectionOptions< + OracleRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricOptions, + persistence: { adapter }, + }), + ) + + collection.startSyncImmediate() + await vi.waitFor(() => expect(subscriber).toBeTypeOf(`function`)) + subscriber!([upToDate]) + await vi.waitFor(() => expect(collection.status).toBe(`ready`)) + + expect(loadSubset).not.toHaveBeenCalled() + expect(collection.has(1)).toBe(false) + await collection.cleanup() + }) + + it.each( + [`reset`, `delete`, `move-out`].flatMap((removal) => + [`none`, `before-commit`, `after-commit`].map((acquire) => ({ + removal, + acquire, + })), + ), + )( + `keeps removed rows unknown across $removal and subset acquisition $acquire`, + async ({ removal, acquire }) => { + const trace = createOracleCollection( + `presence-${removal}-${acquire}`, + `on-demand`, + createMetadata(new Map()).api, + ) + const snapshot = createDeferred() + mockStream.requestSnapshot.mockReturnValueOnce(snapshot.promise) + let acquired: Promise | undefined + const request = () => { + acquired = Promise.resolve( + trace.collection._sync.loadSubset({ + where: new IR.Func(`eq`, [new IR.PropRef([`id`]), new IR.Value(2)]), + }), + ) + expect(mockStream.requestSnapshot).toHaveBeenCalledOnce() + } + try { + const inserted = change(`insert`, 1, `complete`) + trace.subscriber([ + { ...inserted, headers: { ...inserted.headers, tags: [`left`] } }, + upToDate, + ]) + expect(trace.collection.get(1)?.stable).toBe(`stable-1`) + const removed: Message = + removal === `reset` + ? mustRefetch + : removal === `delete` + ? change(`delete`, 1, `complete`) + : ({ + headers: { + event: `move-out`, + patterns: [{ pos: 0, value: `left` }], + }, + } as Message) + trace.subscriber([removed]) + if (acquire === `before-commit`) request() + trace.subscriber([subsetEnd]) + expect(trace.collection.has(1)).toBe(false) + if (acquire === `after-commit`) request() + snapshot.resolve() + await acquired + trace.subscriber([change(`update`, 1, `partial`), upToDate]) + expect(trace.collection.has(1)).toBe(false) + } finally { + snapshot.resolve() + await trace.collection.cleanup() + await acquired + } + }, + ) + + fcTest.prop( + [ + fc.array( + fc.record({ + operation: fc.constantFrom< + HistoryToken[`operation`] | `reset` | `acquire` | `commit` + >(`insert`, `update`, `delete`, `reset`, `acquire`, `commit`), + id: fc.integer({ min: 1, max: 3 }), + name: fc.string({ maxLength: 8 }), + }), + { minLength: 1, maxLength: 40 }, + ), + ], + { + numRuns: 40, + examples: [ + [ + ([`insert`, `commit`, `reset`, `acquire`, `update`] as const).map( + (operation) => ({ operation, id: 1, name: `` }), + ), + ], + ], + }, + )( + `subset acquisitions preserve row validity through generated stream histories`, + async (commands) => { + const trace = createOracleCollection( + `acquisition-history`, + `on-demand`, + createMetadata(new Map()).api, + ) + const reference: ReferenceState = { + committed: new Map(), + pending: new Map(), + } + let acquisition = 100 + try { + for (const command of commands) { + if (command.operation === `acquire`) { + // A real acquisition, not just a synthetic subset-end marker. Use a + // fresh predicate so exact request deduplication cannot bypass it. + await trace.collection._sync.loadSubset({ + where: new IR.Func(`eq`, [ + new IR.PropRef([`id`]), + new IR.Value(++acquisition), + ]), + }) + } else { + const message = + command.operation === `reset` + ? mustRefetch + : command.operation === `commit` + ? subsetEnd + : change( + command.operation === `insert` && + reference.pending.has(command.id) + ? `update` + : command.operation, + command.id, + command.name, + ) + trace.subscriber([message]) + applyReferenceBatch(reference, [message]) + } + expect(rowsFromCollection(trace.collection)).toEqual( + rowsFromMap(reference.committed), + ) + } + trace.subscriber([upToDate]) + applyReferenceBatch(reference, [upToDate]) + expect(rowsFromCollection(trace.collection)).toEqual( + rowsFromMap(reference.committed), + ) + } finally { + await trace.collection.cleanup() + } + }, + ) + + it(`applies on-demand catch-up updates to hydrated persisted rows`, async () => { + let subscriber!: (messages: Array>) => void + mockSubscribe.mockImplementationOnce((callback) => { + subscriber = callback + return vi.fn() + }) + const collectionMetadata = new Map(resumeState()) + const persistedRows = new Map([ + [1, { id: 1, name: `persisted`, stable: `stable-1` }], + ]) + const collection = createCollection( + persistedCollectionOptions< + OracleRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricCollectionOptions({ + id: `on-demand-persisted-catch-up`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `on-demand`, + getKey: (row) => row.id, + startSync: true, + }), + persistence: { + adapter: createPersistedAdapter(collectionMetadata, persistedRows), + }, + }), + ) + + collection.startSyncImmediate() + await vi.waitFor(() => expect(subscriber).toBeTypeOf(`function`)) + await collection._sync.loadSubset({}) + subscriber([change(`update`, 1, `caught up`), upToDate]) + await vi.waitFor(() => expect(collection.status).toBe(`ready`)) + + expect(collection.get(1)).toEqual( + expect.objectContaining({ name: `caught up`, stable: `stable-1` }), + ) + expect(collectionMetadata.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `resume`, offset: `20_0` }), + ) + await collection.cleanup() + }) + + it.each([ + { + name: `malformed`, + seed: new Map([ + [ + `electric:resume`, + { + kind: `resume`, + requiresTagState: false, + offset: 10, + handle: `shape-1`, + shapeId, + updatedAt: 1, + }, + ], + ]), + }, + { + name: `reset`, + seed: new Map([ + [`electric:resume`, { kind: `reset`, updatedAt: 1 }], + ]), + }, + { + name: `incompatible`, + seed: new Map([ + [ + `electric:resume`, + { + kind: `resume`, + requiresTagState: false, + offset: `10_0`, + handle: `shape-1`, + shapeId: `different-shape`, + updatedAt: 1, + }, + ], + ]), + }, + ])(`starts a full snapshot for $name resume metadata`, async ({ seed }) => { + const metadata = createMetadata(seed) + const trace = createOracleCollection( + `non-resumable-metadata`, + `eager`, + metadata.api, + ) + + expect(vi.mocked(ShapeStream).mock.calls.at(-1)?.[0]).toMatchObject({ + offset: undefined, + handle: undefined, + }) + trace.subscriber([change(`insert`, 1, `full snapshot`), upToDate]) + + expect(trace.collection.status).toBe(`ready`) + expect(trace.collection.get(1)).toEqual( + expect.objectContaining({ stable: `stable-1` }), + ) + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `resume`, offset: `20_0` }), + ) + await trace.collection.cleanup() + }) + + it(`does not mix an explicit resume option with persisted metadata`, async () => { + const metadata = createMetadata(resumeState()) + const trace = createOracleCollection( + `explicit-resume`, + `on-demand`, + metadata.api, + { handle: `explicit-handle` }, + ) + + expect(vi.mocked(ShapeStream).mock.calls.at(-1)?.[0]).toMatchObject({ + offset: `now`, + handle: `explicit-handle`, + }) + trace.subscriber([upToDate]) + expect(trace.collection.status).toBe(`ready`) + await trace.collection.cleanup() + }) + + it(`lets an equal-timestamp reset dominate a stale hydrated resume`, async () => { + let subscriber!: (messages: Array>) => void + mockSubscribe.mockImplementationOnce((callback) => { + subscriber = callback + return vi.fn() + }) + const options = electricCollectionOptions({ + id: `equal-timestamp-reset`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + startSync: true, + }) + options.sync.importSyncMeta?.({ + version: 1, + resume: { + kind: `resume`, + requiresTagState: false, + offset: `10_0`, + handle: `stale-handle`, + shapeId, + updatedAt: 1, + }, + seenTxids: [], + }) + const originalSync = options.sync + const metadata = createMetadata( + new Map([[`electric:resume`, { kind: `reset`, updatedAt: 1 }]]), + ) + const collection = createCollection({ + ...options, + sync: { + ...originalSync, + sync: (params: Parameters[0]) => + originalSync.sync({ ...params, metadata: metadata.api }), + }, + }) + + expect(vi.mocked(ShapeStream).mock.calls.at(-1)?.[0]).toMatchObject({ + offset: undefined, + handle: undefined, + }) + subscriber([change(`insert`, 1, `full snapshot`), upToDate]) + expect(collection.status).toBe(`ready`) + await collection.cleanup() + }) + + fcTest.prop( + [ + fc.integer({ min: 0, max: 3 }), + fc.integer({ min: 0, max: 3 }), + fc.integer({ min: 0, max: 3 }), + fc.string({ minLength: 1, maxLength: 8 }), + fc.string({ minLength: 1, maxLength: 8 }), + fc.string({ minLength: 1, maxLength: 8 }), + fc.boolean(), + ], + { numRuns: 50 }, + )( + `resume metadata merge is commutative and reset-safe on timestamp ties`, + ( + leftTimestamp, + rightTimestamp, + thirdTimestamp, + leftHandle, + rightHandle, + thirdHandle, + thirdIsReset, + ) => { + const options = electricCollectionOptions({ + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (row) => row.id, + }) + const merge = options.sync.mergeSyncMeta! + const left = { + version: 1, + resume: { + kind: `resume`, + requiresTagState: false, + offset: `10_0`, + handle: leftHandle, + shapeId, + updatedAt: leftTimestamp, + }, + seenTxids: [], + } + const right = { + version: 1, + resume: + leftTimestamp === rightTimestamp && leftHandle !== rightHandle + ? { kind: `reset`, updatedAt: rightTimestamp } + : { + kind: `resume`, + requiresTagState: false, + offset: `20_0`, + handle: rightHandle, + shapeId, + updatedAt: rightTimestamp, + }, + seenTxids: [], + } + const third = { + version: 1, + resume: thirdIsReset + ? { kind: `reset`, updatedAt: thirdTimestamp } + : { + kind: `resume`, + requiresTagState: false, + offset: `30_0`, + handle: thirdHandle, + shapeId, + updatedAt: thirdTimestamp, + }, + seenTxids: [], + } + + const leftThenRight = merge(left, right) + const rightThenLeft = merge(right, left) + expect(leftThenRight).toEqual(rightThenLeft) + expect(merge(left, left)).toEqual(left) + expect(merge(merge(left, right), third)).toEqual( + merge(left, merge(right, third)), + ) + if (leftTimestamp === rightTimestamp) { + expect(leftThenRight).toEqual( + expect.objectContaining({ + resume: expect.objectContaining({ kind: `reset` }), + }), + ) + } + }, + ) + + it(`rejects an unseen partial update from an explicit eager resume`, async () => { + const metadata = createMetadata(resumeState()) + const trace = createOracleCollection( + `explicit-eager-resume`, + `eager`, + metadata.api, + { offset: `10_0` as Offset, handle: `explicit-handle` }, + ) + + trace.subscriber([change(`update`, 1, `partial`), upToDate]) + + expect(trace.collection.status).toBe(`error`) + expect(trace.collection.has(1)).toBe(false) + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `reset` }), + ) + await trace.collection.cleanup() + }) + + it(`accepts complete replica updates from an explicit eager resume`, async () => { + let subscriber!: (messages: Array>) => void + mockSubscribe.mockImplementationOnce((callback) => { + subscriber = callback + return vi.fn() + }) + const collection = createCollection( + electricCollectionOptions({ + id: `explicit-full-replica-resume`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table`, replica: `full` }, + offset: `10_0` as Offset, + handle: `explicit-handle`, + }, + syncMode: `eager`, + getKey: (row) => row.id, + startSync: true, + }), + ) + + subscriber([ + { + key: `1`, + value: { id: 1, name: `complete update`, stable: `stable-1` }, + headers: { operation: `update` }, + }, + upToDate, + ]) + + expect(collection.status).toBe(`ready`) + expect(collection.get(1)).toEqual( + expect.objectContaining({ + name: `complete update`, + stable: `stable-1`, + }), + ) + await collection.cleanup() + }) + + it.each([10, 100])( + `subset acquisition avoids scanning the applied baseline with %s rows`, + async (size) => { + const trace = createOracleCollection( + `acquisition-work`, + `on-demand`, + createMetadata(new Map()).api, + ) + try { + trace.subscriber([ + ...Array.from({ length: size }, (_, id) => + change(`insert`, id, `row`), + ), + upToDate, + ]) + await trace.collection._sync.loadSubset({}) + const keys = vi.spyOn(trace.collection._state.syncedData, `keys`) + try { + for (let i = 0; i < 3; i++) + await trace.collection._sync.loadSubset({}) + expect(keys).not.toHaveBeenCalled() + for (let i = 0; i < 3; i++) + await trace.collection._sync.loadSubset({ + where: new IR.Func(`eq`, [ + new IR.PropRef([`id`]), + new IR.Value(size + i), + ]), + }) + expect(keys).not.toHaveBeenCalled() + expect(mockStream.requestSnapshot).toHaveBeenCalledTimes(4) + } finally { + keys.mockRestore() + } + } finally { + await trace.collection.cleanup() + } + }, + ) + + it(`warns once and restarts a persisted resume when hydration completion is unavailable`, async () => { + const warn = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const metadata = createMetadata(resumeState()) + Object.assign(metadata.api.row, { + scanPersisted: () => Promise.resolve([{ key: 1 }]), + }) + const trace = createOracleCollection( + `unverifiable-persisted-resume`, + `eager`, + metadata.api, + ) + + try { + expect(vi.mocked(ShapeStream).mock.calls.at(-1)?.[0]).toMatchObject({ + offset: undefined, + handle: undefined, + }) + trace.subscriber([change(`insert`, 1, `full snapshot`), upToDate]) + + expect(trace.collection.status).toBe(`ready`) + expect(trace.collection.get(1)).toEqual( + expect.objectContaining({ stable: `stable-1` }), + ) + await trace.collection.cleanup() + trace.collection.startSyncImmediate() + mockSubscribe.mock.calls.at(-1)![0]([upToDate]) + expect(warn).toHaveBeenCalledTimes(1) + expect(warn.mock.calls[0]?.[0]).toMatch( + /persistence.*cannot verify hydration.*[Uu]pdate/, + ) + } finally { + await trace.collection.cleanup() + warn.mockRestore() + } + }) + + it(`ignores an unseen on-demand update without blocking readiness`, async () => { + const metadata = createMetadata(resumeState()) + const trace = createOracleCollection( + `on-demand-unseen-update`, + `on-demand`, + metadata.api, + ) + + trace.subscriber([change(`update`, 1, `partial`), upToDate]) + + expect(trace.collection.status).toBe(`ready`) + expect(trace.collection.has(1)).toBe(false) + expect(metadata.state.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `resume`, offset: `20_0` }), + ) + await trace.collection.cleanup() + }) + + it(`records match and txid evidence for an ignored on-demand update`, async () => { + const trace = createOracleCollection( + `on-demand-ignored-update-evidence`, + `on-demand`, + createMetadata(resumeState()).api, + ) + trace.subscriber([upToDate]) + + const pendingMatch = trace.collection.utils.awaitMatch( + (message) => `value` in message && message.value.id === 808, + 100, + ) + const pendingTxid = trace.collection.utils.awaitTxId(808, 100) + const update = change(`update`, 808, `ignored`) + update.headers.txids = [808] + + trace.subscriber([update, upToDate]) + + await Promise.all([ + expect(pendingMatch).resolves.toBe(true), + expect(pendingTxid).resolves.toBe(true), + ]) + expect(trace.collection.has(808)).toBe(false) + await trace.collection.cleanup() + }) +}) diff --git a/packages/electric-db-collection/tests/electric-recovery-oracle.test.ts b/packages/electric-db-collection/tests/electric-recovery-oracle.test.ts new file mode 100644 index 0000000000..01f3aaba6c --- /dev/null +++ b/packages/electric-db-collection/tests/electric-recovery-oracle.test.ts @@ -0,0 +1,429 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { beforeEach, describe, expect, it, vi } from 'vitest' +import { createCollection } from '@tanstack/db' +import { ShapeStream } from '@electric-sql/client' +import { persistedCollectionOptions } from '../../db-sqlite-persistence-core/src' +import { electricCollectionOptions } from '../src/electric' +import type { Message, Row } from '@electric-sql/client' +import type { + PersistedCollectionCoordinator, + PersistedTx, + PersistenceAdapter, + ProtocolEnvelope, +} from '../../db-sqlite-persistence-core/src' +import type { ElectricCollectionUtils, ElectricSyncMode } from '../src/electric' + +type Item = Row & { id: number; name: string; stable: string } +type Subscriber = (messages: Array>) => void +const subscribers: Array = [] +const mockSubscribe = vi.fn((callback: Subscriber) => { + subscribers.push(callback) + return vi.fn() +}) + +vi.mock(`@electric-sql/client`, async () => ({ + ...(await vi.importActual(`@electric-sql/client`)), + ShapeStream: vi.fn(() => ({ + subscribe: mockSubscribe, + requestSnapshot: vi.fn().mockResolvedValue(undefined), + fetchSnapshot: vi.fn().mockResolvedValue({ metadata: {}, data: [] }), + forceDisconnectAndRefresh: vi.fn().mockResolvedValue(undefined), + isUpToDate: false, + shapeHandle: `shape-current`, + lastOffset: `20_0`, + })), +})) + +function deferred() { + let resolve!: () => void + const promise = new Promise((complete) => { + resolve = complete + }) + return { promise, resolve } +} + +const oldRow: Item = { id: 1, name: `old`, stable: `stable-1` } +const freshRow: Item = { id: 2, name: `fresh`, stable: `stable-2` } +const upToDate: Message = { headers: { control: `up-to-date` } } + +function change( + operation: `insert` | `update` | `delete`, + value: Partial, +): Message { + return { key: String(value.id), value: value as Item, headers: { operation } } +} + +function fixture( + syncMode: ElectricSyncMode, + coordinator?: PersistedCollectionCoordinator, +) { + const rows = new Map([[oldRow.id, { ...oldRow }]]) + const metadata = new Map([ + [ + `electric:resume`, + { + kind: `resume`, + requiresTagState: false, + offset: `10_0`, + handle: `shape-old`, + shapeId: `{"params":{"table":"test_table"},"url":"http://test-url"}`, + updatedAt: 1, + }, + ], + ]) + let hydrationGate = Promise.resolve() + const commits: Array = [] + const adapter: PersistenceAdapter = { + loadSubset: () => { + const snapshot = Array.from(rows, ([key, value]) => ({ + key, + value: { ...value }, + })) + return hydrationGate.then(() => snapshot) + }, + loadCollectionMetadata: () => + Promise.resolve(Array.from(metadata, ([key, value]) => ({ key, value }))), + applyCommittedTx: (_collectionId, tx) => { + for (const mutation of tx.collectionMetadataMutations ?? []) { + if (mutation.type === `delete`) metadata.delete(mutation.key) + else metadata.set(mutation.key, mutation.value) + } + if (tx.truncate) rows.clear() + for (const mutation of tx.mutations) { + if (mutation.type === `delete`) rows.delete(mutation.key) + else { + rows.set(mutation.key, { + ...rows.get(mutation.key), + ...mutation.value, + } as Item) + } + } + commits.push(tx) + return Promise.resolve() + }, + ensureIndex: () => Promise.resolve(), + } + const collection = createCollection( + persistedCollectionOptions< + Item, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricCollectionOptions({ + id: `persisted-recovery-${syncMode}`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode, + getKey: (row) => row.id, + startSync: false, + }), + persistence: { adapter, coordinator }, + }), + ) + const publicRows = () => + Array.from(collection.values(), ({ id, name, stable }) => ({ + id, + name, + stable, + })).sort((a, b) => a.id - b.id) + const durableRows = () => [...rows.values()].sort((a, b) => a.id - b.id) + return { + collection, + rows, + metadata, + commits, + publicRows, + durableRows, + pauseHydration: (gate: Promise) => { + hydrationGate = gate + }, + } +} + +const scenarios = ([`eager`, `progressive`] as const).flatMap((syncMode) => + [false, true].flatMap((empty) => + ([`before`, `after`] as const).map((hydration) => ({ + syncMode, + empty, + hydration, + })), + ), +) + +describe(`persisted Electric recovery laws`, () => { + beforeEach(() => { + subscribers.length = 0 + vi.clearAllMocks() + }) + + function externalPublisher() { + let receive: ((message: ProtocolEnvelope) => void) | undefined + let id = `` + let term = 100 + const coordinator: PersistedCollectionCoordinator = { + getNodeId: () => `local`, + subscribe: (collectionId, callback) => { + id = collectionId + receive = callback + return () => { + receive = undefined + } + }, + publish: () => {}, + isLeader: () => true, + ensureLeadership: () => Promise.resolve(), + requestEnsurePersistedIndex: () => Promise.resolve(), + requestEnsureRemoteSubset: () => Promise.resolve(), + } + return { + coordinator, + publish: ( + row: Item, + deleted: boolean, + fullReload: boolean, + metadata: Map, + ) => { + const revision = term++ + metadata.set(`oracle:publication`, revision) + receive?.({ + v: 1, + dbName: id, + collectionId: id, + senderId: `peer`, + ts: Date.now(), + payload: { + type: `tx:committed`, + term: revision, + seq: 1, + txId: `peer-${term}`, + latestRowVersion: term, + requiresFullReload: fullReload, + changedRows: deleted ? [] : [{ key: row.id, value: row }], + deletedKeys: deleted ? [row.id] : [], + collectionMetadataMutations: [ + { type: `set`, key: `oracle:publication`, value: revision }, + ], + }, + }) + return revision + }, + } + } + + it.each( + ([`eager`, `progressive`, `on-demand`] as const).flatMap((syncMode) => + [false, true].map((fullReload) => ({ syncMode, fullReload })), + ), + )( + `$syncMode merges stream deltas into independently published rows, fullReload=$fullReload`, + async ({ syncMode, fullReload }) => { + const peer = externalPublisher() + const f = fixture(syncMode, peer.coordinator) + try { + f.collection.startSyncImmediate() + await vi.waitFor(() => expect(subscribers).toHaveLength(1)) + if (syncMode === `on-demand`) await f.collection._sync.loadSubset({}) + await vi.waitFor(() => expect(f.publicRows()).toEqual([oldRow])) + subscribers[0]!([upToDate]) + f.rows.set(freshRow.id, freshRow) + peer.publish(freshRow, false, fullReload, f.metadata) + await vi.waitFor(() => + expect(f.publicRows()).toEqual([oldRow, freshRow]), + ) + subscribers[0]!([ + change(`update`, { id: freshRow.id, name: `changed` }), + upToDate, + ]) + const expected = [oldRow, { ...freshRow, name: `changed` }] + expect(f.publicRows()).toEqual(expected) + await vi.waitFor(() => expect(f.durableRows()).toEqual(expected)) + } finally { + await f.collection.cleanup() + } + }, + ) + + fcTest.prop( + [ + fc.array( + fc.record({ + id: fc.integer({ min: 2, max: 4 }), + name: fc.string({ maxLength: 8 }), + deleted: fc.boolean(), + fullReload: fc.boolean(), + }), + { minLength: 1, maxLength: 8 }, + ), + ], + { + numRuns: 20, + examples: [ + [ + [ + { id: 2, name: `external`, deleted: false, fullReload: false }, + { id: 2, name: `removed`, deleted: true, fullReload: true }, + ], + ], + ], + }, + )( + `independent persistence publications and stream deltas agree with complete-row state`, + async (commands) => { + subscribers.length = 0 + const peer = externalPublisher() + const f = fixture(`on-demand`, peer.coordinator) + const expected = new Map([[oldRow.id, oldRow]]) + const expectedRows = () => + [...expected.values()].sort((a, b) => a.id - b.id) + try { + f.collection.startSyncImmediate() + await vi.waitFor(() => expect(subscribers).toHaveLength(1), { + interval: 1, + }) + await f.collection._sync.loadSubset({}) + subscribers[0]!([upToDate]) + for (const command of commands) { + const row = { + id: command.id, + name: command.name, + stable: `peer-${command.id}`, + } + if (command.deleted) { + f.rows.delete(row.id) + expected.delete(row.id) + } else { + f.rows.set(row.id, row) + expected.set(row.id, row) + } + const revision = peer.publish( + row, + command.deleted, + command.fullReload, + f.metadata, + ) + // An unchanged row set is not proof that the peer publication ran. + // Its metadata marker commits with the rows, including empty deletes. + await vi.waitFor( + () => + expect( + f.collection._state.syncedCollectionMetadata.get( + `oracle:publication`, + ), + ).toBe(revision), + { interval: 1 }, + ) + expect(f.publicRows()).toEqual(expectedRows()) + subscribers[0]!([ + change(`update`, { id: row.id, name: `stream` }), + upToDate, + ]) + if (!command.deleted) expected.set(row.id, { ...row, name: `stream` }) + expect(f.publicRows()).toEqual(expectedRows()) + await vi.waitFor( + () => expect(f.durableRows()).toEqual(expectedRows()), + { interval: 1 }, + ) + } + } finally { + await f.collection.cleanup() + } + }, + ) + + it.each(scenarios)( + `$syncMode invalid resume replaces omitted cached rows: empty=$empty, hydration=$hydration callback`, + async ({ syncMode, empty, hydration }) => { + const f = fixture(syncMode) + const gate = deferred() + try { + f.collection.startSyncImmediate() + await vi.waitFor(() => expect(f.publicRows()).toEqual([oldRow])) + await vi.waitFor(() => expect(subscribers).toHaveLength(1)) + expect(vi.mocked(ShapeStream).mock.calls[0]?.[0]).toMatchObject({ + offset: `10_0`, + handle: `shape-old`, + }) + subscribers[0]!([ + change(`delete`, { id: 1 }), + change(`update`, { id: 2, name: `partial` }), + upToDate, + ]) + await vi.waitFor(() => expect(f.collection.status).toBe(`error`)) + await vi.waitFor(() => + expect(f.metadata.get(`electric:resume`)).toMatchObject({ + kind: `reset`, + }), + ) + expect(f.publicRows()).toEqual([oldRow]) + expect(f.durableRows()).toEqual([oldRow]) + + await f.collection.cleanup() + f.pauseHydration(gate.promise) + f.collection.startSyncImmediate() + await vi.waitFor(() => expect(subscribers).toHaveLength(2)) + expect(vi.mocked(ShapeStream).mock.calls[1]?.[0]).toMatchObject({ + offset: undefined, + handle: undefined, + }) + // Fresh progressive mode hydrates persisted rows only on demand. + const hydrationDone = + syncMode === `progressive` + ? Promise.resolve(f.collection._sync.loadSubset({ limit: 10 })) + : undefined + if (hydration === `before`) { + gate.resolve() + await hydrationDone + await vi.waitFor(() => expect(f.publicRows()).toEqual([oldRow])) + } + const expected = empty ? [] : [freshRow] + subscribers[1]!([ + ...expected.map((row) => change(`insert`, row)), + upToDate, + ]) + gate.resolve() + await hydrationDone + await vi.waitFor(() => expect(f.collection.status).toBe(`ready`)) + await vi.waitFor(() => + expect(f.metadata.get(`electric:resume`)).toMatchObject({ + kind: `resume`, + requiresTagState: false, + offset: `20_0`, + }), + ) + // The source's complete snapshot defines both results. A reset marker + // plus a fresh offset is not proof that the old materialization left. + expect.soft(f.publicRows()).toEqual(expected) + expect.soft(f.durableRows()).toEqual(expected) + } finally { + gate.resolve() + await f.collection.cleanup() + } + }, + ) + + it.each([`eager`, `progressive`] as const)( + `%s valid resume retains cached rows and their unchanged fields`, + async (syncMode) => { + const f = fixture(syncMode) + try { + f.collection.startSyncImmediate() + await vi.waitFor(() => expect(f.publicRows()).toEqual([oldRow])) + await vi.waitFor(() => expect(subscribers).toHaveLength(1)) + subscribers[0]!([ + change(`update`, { id: 1, name: `changed` }), + upToDate, + ]) + await vi.waitFor(() => expect(f.collection.status).toBe(`ready`)) + const expected = [{ ...oldRow, name: `changed` }] + expect(f.publicRows()).toEqual(expected) + await vi.waitFor(() => expect(f.durableRows()).toEqual(expected)) + expect(f.commits.every((tx) => !tx.truncate)).toBe(true) + } finally { + await f.collection.cleanup() + } + }, + ) +}) diff --git a/packages/electric-db-collection/tests/electric-sdk-framing.test.ts b/packages/electric-db-collection/tests/electric-sdk-framing.test.ts new file mode 100644 index 0000000000..88b59e15f5 --- /dev/null +++ b/packages/electric-db-collection/tests/electric-sdk-framing.test.ts @@ -0,0 +1,79 @@ +import { expect, it, vi } from 'vitest' +import { ShapeStream } from '@electric-sql/client' +import type { Message } from '@electric-sql/client' + +// Use the real SDK, not the ShapeStream mock in the adapter oracle. Its reset +// framing is the evidence for the oracle's singleton-reset partition rule. +it.each([false, true])( + `isolates HTTP 409 reset callbacks with stale response rows=%s`, + async (staleRows) => { + const controller = new AbortController() + const old = { + key: `1`, + value: { id: `1`, name: `discarded` }, + headers: { operation: `insert` }, + } + const reset = { headers: { control: `must-refetch` } } + const ready = { + headers: { control: `up-to-date`, global_last_seen_lsn: `2` }, + } + const headers = { + 'electric-handle': `old-shape`, + 'electric-offset': `1_0`, + 'electric-schema': JSON.stringify({ + id: { type: `int4` }, + name: { type: `text` }, + }), + } + const responses = [ + new Response(JSON.stringify([old]), { headers }), + new Response(JSON.stringify(staleRows ? [old, reset, ready] : [reset]), { + status: 409, + headers: { 'electric-handle': `new-shape` }, + }), + new Response(JSON.stringify([ready]), { + headers: { + ...headers, + 'electric-handle': `new-shape`, + 'electric-offset': `2_0`, + 'electric-cursor': `1`, + }, + }), + ] + const batches: Array>> = [] + const stream = new ShapeStream<{ id: number; name: string }>({ + // The SDK caches expired handles/up-to-date offsets by shape URL. + url: `http://test-url/v1/shape-${staleRows}`, + params: { table: `rows` }, + signal: controller.signal, + fetchClient: async () => { + const response = responses.shift() + if (response) return response + return new Promise((_resolve, reject) => { + if (controller.signal.aborted) reject(controller.signal.reason) + else + controller.signal.addEventListener( + `abort`, + () => reject(controller.signal.reason), + { once: true }, + ) + }) + }, + }) + const unsubscribe = stream.subscribe((messages) => { + batches.push(messages) + }) + try { + await vi.waitFor(() => expect(batches).toHaveLength(3)) + expect(batches[0]).toHaveLength(1) + expect(batches[0]![0]).toMatchObject({ + value: { id: 1, name: `discarded` }, + }) + expect(batches[1]).toEqual([reset]) + expect(batches[2]).toEqual([ready]) + } finally { + unsubscribe() + controller.abort() + } + }, +) diff --git a/packages/electric-db-collection/tests/electric.test.ts b/packages/electric-db-collection/tests/electric.test.ts index 868c085213..bce2926fd0 100644 --- a/packages/electric-db-collection/tests/electric.test.ts +++ b/packages/electric-db-collection/tests/electric.test.ts @@ -1,6 +1,8 @@ -import { beforeEach, describe, expect, it, vi } from 'vitest' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { ShapeStream } from '@electric-sql/client' import { CollectionImpl, + IR, createCollection, createTransaction, } from '@tanstack/db' @@ -20,6 +22,19 @@ import type { import type { Message, Row } from '@electric-sql/client' import type { StandardSchemaV1 } from '@standard-schema/spec' +const NativeAbortController = globalThis.AbortController + +function createDeferred(): { + promise: Promise + resolve: (value: T | PromiseLike) => void +} { + let resolve!: (value: T | PromiseLike) => void + const promise = new Promise((resolvePromise) => { + resolve = resolvePromise + }) + return { promise, resolve } +} + // Mock the ShapeStream module const mockSubscribe = vi.fn() const mockRequestSnapshot = vi.fn() @@ -95,16 +110,22 @@ describe(`Electric Integration`, () => { const createPersistedAdapter = ( collectionMetadata?: Map, + rows: Map = new Map(), ) => ({ - loadSubset: async () => [], - loadCollectionMetadata: async () => - Array.from((collectionMetadata ?? new Map()).entries()).map( - ([key, value]) => ({ - key, - value, - }), + loadSubset: () => + Promise.resolve( + Array.from(rows.entries()).map(([key, value]) => ({ key, value })), + ), + loadCollectionMetadata: () => + Promise.resolve( + Array.from((collectionMetadata ?? new Map()).entries()).map( + ([key, value]) => ({ + key, + value, + }), + ), ), - applyCommittedTx: async (_collectionId: string, tx: any) => { + applyCommittedTx: (_collectionId: string, tx: any) => { for (const mutation of tx.collectionMetadataMutations ?? []) { if (mutation.type === `delete`) { collectionMetadata?.delete(mutation.key) @@ -112,8 +133,19 @@ describe(`Electric Integration`, () => { collectionMetadata?.set(mutation.key, mutation.value) } } + if (tx.truncate) { + rows.clear() + } + for (const mutation of tx.mutations ?? []) { + if (mutation.type === `delete`) { + rows.delete(mutation.key) + } else { + rows.set(mutation.key, mutation.value) + } + } + return Promise.resolve() }, - ensureIndex: async () => {}, + ensureIndex: () => Promise.resolve(), }) beforeEach(() => { @@ -172,6 +204,55 @@ describe(`Electric Integration`, () => { expect(collection.status).toEqual(`ready`) }) + it(`reports an initial stream error instead of publishing an empty ready snapshot`, async () => { + const loggedError = vi.spyOn(console, `error`).mockImplementation(() => {}) + const preload = collection.preload() + const streamOptions = vi.mocked(ShapeStream).mock.calls.at(-1)?.[0] as + | { onError?: (error: unknown) => void } + | undefined + const initialError = new Error(`initial stream failed`) + + try { + streamOptions?.onError?.(initialError) + + expect(collection.status).toBe(`error`) + await expect(preload).rejects.toBe(initialError) + } finally { + loggedError.mockRestore() + } + }) + + it(`does not let a parked ready receipt overwrite a later stream error`, async () => { + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + const streamError = new Error(`stream failed`) + const loggedError = vi.spyOn(console, `error`).mockImplementation(() => {}) + + try { + transaction.mutate(() => collection.insert({ id: 99, name: `Local row` })) + subscriber([{ headers: { control: `up-to-date` } }]) + expect(collection.status).toBe(`loading`) + + const streamOptions = vi.mocked(ShapeStream).mock.calls.at(-1)?.[0] as + | { onError?: (error: unknown) => void } + | undefined + streamOptions?.onError?.(streamError) + expect(collection.status).toBe(`error`) + + persistence.resolve() + await transaction.isPersisted.promise + await Promise.resolve() + + expect(collection.status).toBe(`error`) + } finally { + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + loggedError.mockRestore() + } + }) + it(`should handle incoming insert messages and commit on up-to-date`, () => { // Simulate incoming insert message subscriber([ @@ -195,6 +276,38 @@ describe(`Electric Integration`, () => { ) }) + it(`marks the source ready only after its initial rows are applied`, async () => { + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => + collection.insert({ id: 99, name: `Optimistic user` }), + ) + + subscriber([ + { + key: `1`, + value: { id: 1, name: `Synced user` }, + headers: { operation: `insert` }, + }, + { headers: { control: `up-to-date` } }, + ]) + await Promise.resolve() + + expect(collection.status).toBe(`loading`) + expect(collection.get(1)).toBeUndefined() + + persistence.resolve() + await transaction.isPersisted.promise + await collection.stateWhenReady() + + expect(collection.status).toBe(`ready`) + expect(collection.get(1)).toEqual( + expect.objectContaining({ id: 1, name: `Synced user` }), + ) + }) + it(`should handle multiple changes before committing`, () => { // First batch of changes subscriber([ @@ -264,6 +377,76 @@ describe(`Electric Integration`, () => { ) }) + it(`ignores an update for a key that has never been materialized`, () => { + subscriber([ + { + key: `2`, + value: { id: 2, name: `Only changed columns` }, + headers: { operation: `update` }, + }, + { headers: { control: `up-to-date` } }, + ]) + + expect(collection.has(2)).toBe(false) + }) + + it(`accepts an update after an insert for the same key in one batch`, () => { + subscriber([ + { + key: `2`, + value: { id: 2, name: `Initial value` }, + headers: { operation: `insert` }, + }, + { + key: `2`, + value: { id: 2, name: `Updated value` }, + headers: { operation: `update` }, + }, + { headers: { control: `up-to-date` } }, + ]) + + expect(collection.get(2)?.name).toBe(`Updated value`) + }) + + it(`accepts a progressive update after its insert in an earlier callback`, () => { + let testSubscriber!: (messages: Array>) => void + mockSubscribe.mockImplementation((callback) => { + testSubscriber = callback + return () => {} + }) + + const testCollection = createCollection( + electricCollectionOptions({ + id: `progressive-split-insert-update-test`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `progressive`, + getKey: (item: Row) => item.id as number, + startSync: true, + }), + ) + + testSubscriber([ + { + key: `2`, + value: { id: 2, name: `Initial value` }, + headers: { operation: `insert` }, + }, + ]) + testSubscriber([ + { + key: `2`, + value: { id: 2, name: `Updated value` }, + headers: { operation: `update` }, + }, + ]) + testSubscriber([{ headers: { control: `up-to-date` } }]) + + expect(testCollection.get(2)?.name).toBe(`Updated value`) + }) + it(`should handle delete operations`, () => { // Insert and commit subscriber([ @@ -425,6 +608,101 @@ describe(`Electric Integration`, () => { await expect(collection.utils.awaitTxId(txid2)).resolves.not.toThrow() }) + it(`exports and imports versioned hydration sync metadata`, async () => { + mockStream.shapeHandle = `shape-handle` + mockStream.lastOffset = `42_0` + + subscriber([ + { + key: `1`, + value: { id: 1, name: `Test User` }, + headers: { + operation: `insert`, + txids: [100, 200], + }, + }, + { + headers: { control: `up-to-date` }, + }, + ]) + + const exported = collection.config.sync.exportSyncMeta?.() + expect(exported).toMatchObject({ + version: 1, + resume: { + kind: `resume`, + requiresTagState: false, + offset: `42_0`, + handle: `shape-handle`, + }, + seenTxids: [100, 200], + }) + + const resumedOptions = electricCollectionOptions({ + id: `resumed`, + shapeOptions: { + url: `http://test-url`, + params: { + table: `test_table`, + }, + }, + startSync: false, + getKey: (item) => item.id as number, + }) + + const merged = resumedOptions.sync.mergeSyncMeta?.( + { + version: 1, + seenTxids: [50], + }, + exported, + ) + resumedOptions.sync.importSyncMeta?.(merged) + + await expect(resumedOptions.utils.awaitTxId(50)).resolves.toBe(true) + await expect(resumedOptions.utils.awaitTxId(200)).resolves.toBe(true) + + const resumedCollection = createCollection({ + ...resumedOptions, + startSync: true, + }) + + expect(vi.mocked(ShapeStream).mock.calls.at(-1)?.[0]).toMatchObject({ + offset: `42_0`, + handle: `shape-handle`, + }) + + await resumedCollection.cleanup() + }) + + it(`ignores non-finite hydration sync metadata`, () => { + const options = electricCollectionOptions({ + id: `invalid-hydration-sync-meta`, + shapeOptions: { + url: `http://test-url`, + params: { + table: `test_table`, + }, + }, + startSync: false, + getKey: (item) => item.id as number, + }) + + options.sync.importSyncMeta?.({ + version: 1, + resume: { + kind: `reset`, + updatedAt: Number.POSITIVE_INFINITY, + }, + seenTxids: [Number.NaN], + }) + + expect(options.sync.exportSyncMeta?.()).toEqual({ + version: 1, + seenTxids: [], + }) + }) + it(`should reject with timeout when waiting for unknown txid`, async () => { // Set a short timeout for the test const unknownTxid = 0 @@ -1868,6 +2146,10 @@ describe(`Electric Integration`, () => { .mockImplementation(() => mockAbortController) }) + afterEach(() => { + globalThis.AbortController = NativeAbortController + }) + it(`should call unsubscribe and abort when collection is cleaned up`, async () => { const config = { id: `cleanup-test`, @@ -1977,6 +2259,9 @@ describe(`Electric Integration`, () => { // Initial stream setup expect(mockSubscribe).toHaveBeenCalledTimes(1) + mockStream.shapeHandle = `discarded-handle` + mockStream.lastOffset = `42_0` + subscriber([{ headers: { control: `up-to-date` } }]) // Cleanup await testCollection.cleanup() @@ -1988,6 +2273,10 @@ describe(`Electric Integration`, () => { // Should have started a new stream expect(mockSubscribe).toHaveBeenCalledTimes(2) expect(testCollection.status).toBe(`loading`) + expect(vi.mocked(ShapeStream).mock.calls.at(-1)?.[0]).toMatchObject({ + offset: undefined, + handle: undefined, + }) subscription.unsubscribe() }) @@ -2442,6 +2731,53 @@ describe(`Electric Integration`, () => { // Tests for syncMode configuration describe(`syncMode configuration`, () => { + const createOnDemandCollection = (id: string) => + createCollection( + electricCollectionOptions({ + id, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `on-demand`, + getKey: (item: Row) => item.id as number, + startSync: true, + }), + ) + + it(`removes the external shape abort listener across cleanup and restart`, async () => { + const externalAbort = new NativeAbortController() + const addSpy = vi.spyOn(externalAbort.signal, `addEventListener`) + const removeSpy = vi.spyOn(externalAbort.signal, `removeEventListener`) + const testCollection = createCollection( + electricCollectionOptions({ + id: `shape-signal-listener-cleanup-test`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + signal: externalAbort.signal, + }, + syncMode: `progressive`, + getKey: (item: Row) => item.id as number, + startSync: true, + }), + ) + + await testCollection.cleanup() + const subscription = testCollection.subscribeChanges(() => {}) + await testCollection.cleanup() + subscription.unsubscribe() + + const addedListeners = addSpy.mock.calls + .filter(([type]) => type === `abort`) + .map(([, listener]) => listener) + const removedListeners = removeSpy.mock.calls + .filter(([type]) => type === `abort`) + .map(([, listener]) => listener) + expect(addedListeners).toHaveLength(2) + expect(removedListeners).toEqual(addedListeners) + }) + it(`should not request snapshots during subscription in eager mode`, () => { vi.clearAllMocks() @@ -2506,6 +2842,263 @@ describe(`Electric Integration`, () => { ) }) + it(`retains Electric coverage when the adapter cannot unload it`, async () => { + const testCollection = createCollection( + electricCollectionOptions({ + id: `on-demand-unload-coverage-test`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `on-demand`, + getKey: (item: Row) => item.id as number, + startSync: true, + }), + ) + const options = { limit: 10 } + + try { + await testCollection._sync.loadSubset(options) + testCollection._sync.unloadSubset(options) + await testCollection._sync.loadSubset(options) + + expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + } finally { + await testCollection.cleanup() + } + }) + + it(`waits for an on-demand commit to become public`, async () => { + const request = createDeferred() + mockRequestSnapshot.mockReturnValueOnce(request.promise) + const testCollection = createOnDemandCollection( + `on-demand-successful-parked-commit-test`, + ) + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + + try { + transaction.mutate(() => + testCollection.insert({ id: 3, name: `Local row` }), + ) + const load = Promise.resolve( + testCollection._sync.loadSubset({ limit: 10 }), + ) + await vi.waitFor(() => + expect(mockRequestSnapshot).toHaveBeenCalledOnce(), + ) + subscriber([ + { + key: `2`, + value: { id: 2, name: `Applied parked row` }, + headers: { operation: `insert` }, + }, + { headers: { control: `subset-end` } }, + ]) + request.resolve() + + const nextTurn = new Promise<`next-turn`>((resolve) => + setTimeout(() => resolve(`next-turn`), 0), + ) + await expect( + Promise.race([load.then(() => `load-settled` as const), nextTurn]), + ).resolves.toBe(`next-turn`) + expect(testCollection.has(2)).toBe(false) + + persistence.resolve() + await transaction.isPersisted.promise + await load + expect(stripVirtualProps(testCollection.get(2))).toEqual({ + id: 2, + name: `Applied parked row`, + }) + } finally { + request.resolve() + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await testCollection.cleanup() + } + }) + + it(`waits for both physical requests of one cursor demand`, async () => { + const whereCurrent = createDeferred() + const whereFrom = createDeferred() + mockRequestSnapshot + .mockReturnValueOnce(whereCurrent.promise) + .mockReturnValueOnce(whereFrom.promise) + const testCollection = createOnDemandCollection( + `on-demand-cursor-all-requests-test`, + ) + const id = new IR.PropRef([`id`]) + + try { + const load = Promise.resolve( + testCollection._sync.loadSubset({ + limit: 10, + orderBy: [ + { + expression: id, + compareOptions: { + direction: `asc`, + nulls: `last`, + stringSort: `lexical`, + }, + }, + ], + cursor: { + whereCurrent: new IR.Func(`eq`, [id, new IR.Value(1)]), + whereFrom: new IR.Func(`gt`, [id, new IR.Value(1)]), + lastKey: 1, + }, + }), + ) + await vi.waitFor(() => + expect(mockRequestSnapshot).toHaveBeenCalledTimes(2), + ) + + whereCurrent.resolve() + const nextTurn = new Promise<`next-turn`>((resolve) => + setTimeout(() => resolve(`next-turn`), 0), + ) + await expect( + Promise.race([load.then(() => `load-settled` as const), nextTurn]), + ).resolves.toBe(`next-turn`) + + whereFrom.resolve() + await load + } finally { + whereCurrent.resolve() + whereFrom.resolve() + await testCollection.cleanup() + } + }) + + it.each([ + { syncMode: `on-demand`, signalSource: `collection` }, + { syncMode: `on-demand`, signalSource: `request` }, + { syncMode: `progressive`, signalSource: `collection` }, + { syncMode: `progressive`, signalSource: `request` }, + ] as const)( + `starts no $syncMode work for an already-aborted $signalSource signal`, + async ({ syncMode, signalSource }) => { + const abortController = new AbortController() + abortController.abort() + const testCollection = createCollection( + electricCollectionOptions({ + id: `${syncMode}-${signalSource}-already-aborted`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + signal: + signalSource === `collection` + ? abortController.signal + : undefined, + }, + syncMode, + getKey: (item: Row) => item.id as number, + startSync: true, + }), + ) + + await expect( + testCollection._sync.loadSubset({ + limit: 10, + signal: + signalSource === `request` ? abortController.signal : undefined, + }), + ).rejects.toMatchObject({ name: `AbortError` }) + expect(mockForceDisconnectAndRefresh).not.toHaveBeenCalled() + expect(mockRequestSnapshot).not.toHaveBeenCalled() + expect(mockFetchSnapshot).not.toHaveBeenCalled() + await testCollection.cleanup() + }, + ) + + it.each([true, false])( + `settles reasonless cancellation after refresh with DOMException available %s`, + async (hasDOMException) => { + const originalDOMException = globalThis.DOMException + const controller = new NativeAbortController() + const refresh = createDeferred() + mockStream.isUpToDate = true + mockForceDisconnectAndRefresh.mockReturnValueOnce(refresh.promise) + const testCollection = createOnDemandCollection( + `reasonless-refresh-abort`, + ) + try { + const load = testCollection._sync.loadSubset({ + limit: 10, + signal: controller.signal, + }) + const outcome = Promise.resolve(load).then( + () => undefined, + (error: unknown) => error, + ) + await Promise.resolve() + // Model a platform signal without reason; no event is required for + // the post-refresh cancellation check to observe its terminal state. + Object.defineProperty(controller.signal, `aborted`, { value: true }) + Object.defineProperty(controller.signal, `reason`, { + value: undefined, + }) + if (!hasDOMException) vi.stubGlobal(`DOMException`, undefined) + refresh.resolve() + await expect(outcome).resolves.toMatchObject({ name: `AbortError` }) + expect(mockRequestSnapshot).not.toHaveBeenCalled() + } finally { + vi.stubGlobal(`DOMException`, originalDOMException) + refresh.resolve() + await testCollection.cleanup() + } + }, + ) + + it(`cancels a pending refresh wait when the collection is cleaned up`, async () => { + vi.useFakeTimers() + const schedule = vi.spyOn(globalThis, `setTimeout`) + const cancel = vi.spyOn(globalThis, `clearTimeout`) + const refresh = createDeferred() + mockStream.isUpToDate = true + mockForceDisconnectAndRefresh.mockReturnValueOnce(refresh.promise) + const testCollection = createOnDemandCollection( + `on-demand-refresh-cleanup-test`, + ) + try { + const load = Promise.resolve( + testCollection._sync.loadSubset({ limit: 10 }), + ) + const loadError = load.then( + () => undefined, + (error: unknown) => error, + ) + const refreshTimerIndex = schedule.mock.calls.findIndex( + ([, delay]) => delay === 250, + ) + expect(refreshTimerIndex).toBeGreaterThanOrEqual(0) + const refreshTimer = schedule.mock.results[refreshTimerIndex]!.value + + await Promise.resolve() + await testCollection.cleanup() + await expect(loadError).resolves.toMatchObject({ name: `AbortError` }) + expect(mockRequestSnapshot).not.toHaveBeenCalled() + // The shared collection GC timer may still exist; this wait must not. + expect(cancel).toHaveBeenCalledWith(refreshTimer) + + refresh.resolve() + await refresh.promise + await load.catch(() => undefined) + expect(mockRequestSnapshot).not.toHaveBeenCalled() + } finally { + refresh.resolve() + await testCollection.cleanup() + schedule.mockRestore() + cancel.mockRestore() + vi.useRealTimers() + } + }) + it(`should refresh the stream before requesting on-demand snapshots when already up-to-date`, async () => { vi.clearAllMocks() @@ -2565,6 +3158,98 @@ describe(`Electric Integration`, () => { expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) }) + it(`should request the snapshot after the refresh timeout and ignore late fulfillment`, async () => { + vi.useFakeTimers() + const refresh = createDeferred() + mockStream.isUpToDate = true + mockForceDisconnectAndRefresh.mockReturnValueOnce(refresh.promise) + const testCollection = createOnDemandCollection( + `on-demand-refresh-timeout-fulfillment-test`, + ) + try { + let loadSettled = false + const load = Promise.resolve( + testCollection._sync.loadSubset({ limit: 10 }), + ).then(() => { + loadSettled = true + }) + + await vi.advanceTimersByTimeAsync(249) + expect(mockRequestSnapshot).not.toHaveBeenCalled() + expect(loadSettled).toBe(false) + + await vi.advanceTimersByTimeAsync(1) + await load + expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + expect(loadSettled).toBe(true) + + refresh.resolve() + await refresh.promise + await Promise.resolve() + expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + } finally { + refresh.resolve() + await testCollection.cleanup() + vi.useRealTimers() + } + }) + + it(`should handle late refresh rejection after requesting the snapshot`, async () => { + vi.useFakeTimers() + let rejectRefresh: (error: Error) => void = () => {} + let resolveRefresh: () => void = () => {} + const refresh = new Promise((resolve, reject) => { + resolveRefresh = resolve + rejectRefresh = reject + }) + mockStream.isUpToDate = true + mockForceDisconnectAndRefresh.mockReturnValueOnce(refresh) + const testCollection = createOnDemandCollection( + `on-demand-refresh-timeout-rejection-test`, + ) + try { + const load = testCollection._sync.loadSubset({ limit: 10 }) + await vi.advanceTimersByTimeAsync(250) + await load + + rejectRefresh(new Error(`late refresh failure`)) + await expect(refresh).rejects.toThrow(`late refresh failure`) + await Promise.resolve() + expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + } finally { + resolveRefresh() + await testCollection.cleanup() + vi.useRealTimers() + } + }) + + it(`should clear the refresh timeout when refresh settles early`, async () => { + vi.useFakeTimers() + const schedule = vi.spyOn(globalThis, `setTimeout`) + const cancel = vi.spyOn(globalThis, `clearTimeout`) + mockStream.isUpToDate = true + mockForceDisconnectAndRefresh.mockResolvedValueOnce(undefined) + const testCollection = createOnDemandCollection( + `on-demand-refresh-clears-timeout-test`, + ) + try { + await testCollection._sync.loadSubset({ limit: 10 }) + const refreshTimerIndex = schedule.mock.calls.findIndex( + ([, delay]) => delay === 250, + ) + expect(refreshTimerIndex).toBeGreaterThanOrEqual(0) + const refreshTimer = schedule.mock.results[refreshTimerIndex]!.value + + expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + expect(cancel).toHaveBeenCalledWith(refreshTimer) + } finally { + await testCollection.cleanup() + schedule.mockRestore() + cancel.mockRestore() + vi.useRealTimers() + } + }) + it(`should fetch snapshots in progressive mode when loadSubset is called before sync completes`, async () => { vi.clearAllMocks() @@ -2620,6 +3305,121 @@ describe(`Electric Integration`, () => { }) }) + it(`ignores a progressive snapshot after its subset request is aborted`, async () => { + mockFetchSnapshot.mockReset() + let resolveSnapshot!: (value: { + metadata: Record + data: Array<{ + key: string + value: Row + headers: { operation: `insert` } + }> + }) => void + mockFetchSnapshot.mockReturnValue( + new Promise((resolve) => { + resolveSnapshot = resolve + }), + ) + mockSubscribe.mockImplementation(() => () => {}) + const testCollection = createCollection( + electricCollectionOptions({ + id: `progressive-aborted-snapshot-test`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `progressive`, + getKey: (item: Row) => item.id as number, + startSync: true, + }), + ) + const abortController = new AbortController() + + try { + expect(mockFetchSnapshot).not.toHaveBeenCalled() + const load = testCollection._sync.loadSubset({ + limit: 1, + signal: abortController.signal, + }) + expect(mockFetchSnapshot).toHaveBeenCalledOnce() + expect(testCollection.has(2)).toBe(false) + abortController.abort() + resolveSnapshot({ + metadata: {}, + data: [ + { + key: `2`, + value: { id: 2, name: `Obsolete snapshot` }, + headers: { operation: `insert` }, + }, + ], + }) + if (load instanceof Promise) await load + + expect(testCollection.has(2)).toBe(false) + } finally { + resolveSnapshot({ metadata: {}, data: [] }) + await testCollection.cleanup() + } + }) + + it(`does not publish a progressive snapshot aborted while its commit is parked`, async () => { + mockFetchSnapshot.mockResolvedValue({ + metadata: {}, + data: [ + { + key: `2`, + value: { id: 2, name: `Obsolete snapshot` }, + headers: { operation: `insert` }, + }, + ], + }) + mockSubscribe.mockImplementation(() => () => {}) + const testCollection = createCollection( + electricCollectionOptions({ + id: `progressive-parked-abort-test`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `progressive`, + getKey: (item: Row) => item.id as number, + startSync: true, + }), + ) + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + const abortController = new AbortController() + + try { + transaction.mutate(() => + testCollection.insert({ id: 3, name: `Local row` }), + ) + const load = testCollection._sync.loadSubset({ + limit: 1, + signal: abortController.signal, + }) + await vi.waitFor(() => expect(mockFetchSnapshot).toHaveBeenCalledOnce()) + await Promise.resolve() + await Promise.resolve() + + expect(testCollection.has(2)).toBe(false) + abortController.abort() + persistence.resolve() + await transaction.isPersisted.promise + if (load instanceof Promise) await load + + expect(testCollection.has(2)).toBe(false) + } finally { + abortController.abort() + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await testCollection.cleanup() + } + }) + it(`should not request snapshots when loadSubset is called in eager mode`, async () => { vi.clearAllMocks() @@ -3043,6 +3843,7 @@ describe(`Electric Integration`, () => { `electric:resume`, { kind: `resume`, + requiresTagState: false, offset: `10_0`, handle: `handle-1`, shapeId: `{"params":{"table":"test_table"},"url":"http://test-url"}`, @@ -3085,7 +3886,63 @@ describe(`Electric Integration`, () => { ) }) - it(`should ignore reset resume metadata and fall back to default startup`, async () => { + it(`prefers newer persisted resume metadata over hydrated metadata`, () => { + vi.clearAllMocks() + const metadataHarness = createInMemorySyncMetadataApi( + new Map([ + [ + `electric:resume`, + { + kind: `resume`, + requiresTagState: false, + offset: `20_0`, + handle: `persisted-newer`, + shapeId: `{"params":{"table":"test_table"},"url":"http://test-url"}`, + updatedAt: 20, + }, + ], + ]), + ) + const options = electricCollectionOptions({ + id: `resume-recency-test`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + startSync: false, + getKey: (item) => item.id as number, + }) + options.sync.importSyncMeta?.({ + version: 1, + resume: { + kind: `resume`, + requiresTagState: false, + offset: `10_0`, + handle: `hydrated-older`, + shapeId: `{"params":{"table":"test_table"},"url":"http://test-url"}`, + updatedAt: 10, + }, + seenTxids: [], + }) + const originalSync = options.sync + + createCollection({ + ...options, + startSync: true, + sync: { + ...originalSync, + sync: (params: Parameters[0]) => + originalSync.sync({ ...params, metadata: metadataHarness.api }), + }, + }) + + expect(vi.mocked(ShapeStream).mock.calls.at(-1)?.[0]).toMatchObject({ + offset: `20_0`, + handle: `persisted-newer`, + }) + }) + + it(`should replace reset resume state with a full snapshot`, async () => { vi.clearAllMocks() const { ShapeStream } = await import(`@electric-sql/client`) @@ -3128,7 +3985,8 @@ describe(`Electric Integration`, () => { expect(ShapeStream).toHaveBeenCalledWith( expect.objectContaining({ - offset: `now`, + offset: undefined, + log: undefined, handle: undefined, }), ) @@ -3173,12 +4031,104 @@ describe(`Electric Integration`, () => { expect(ShapeStream).toHaveBeenCalledWith( expect.objectContaining({ - offset: `now`, + offset: undefined, + log: undefined, handle: undefined, }), ) }) + it(`should preserve hydrated rows and resumed changes received before up-to-date`, async () => { + vi.clearAllMocks() + + const { ShapeStream } = await import(`@electric-sql/client`) + mockFetchSnapshot.mockResolvedValue({ + metadata: {}, + data: [], + }) + + const collectionMetadata = new Map([ + [ + `electric:resume`, + { + kind: `resume`, + requiresTagState: false, + offset: `10_0`, + handle: `handle-1`, + shapeId: `{"params":{"table":"test_table"},"url":"http://test-url"}`, + updatedAt: 1, + }, + ], + ]) + const persistedRows = new Map([ + [1, { id: 1, name: `Persisted User` }], + ]) + + const persistedCollection = createCollection( + persistedCollectionOptions({ + ...(electricCollectionOptions({ + id: `persisted-progressive-resume-test`, + shapeOptions: { + url: `http://test-url`, + params: { + table: `test_table`, + }, + }, + syncMode: `progressive` as const, + getKey: (item: Row) => item.id as number, + startSync: true, + }) as any), + persistence: { + adapter: createPersistedAdapter(collectionMetadata, persistedRows), + }, + }) as any, + ) + + persistedCollection.startSyncImmediate() + await persistedCollection._sync.loadSubset({ limit: 10 }) + + expect(ShapeStream).toHaveBeenCalledWith( + expect.objectContaining({ + offset: `10_0`, + handle: `handle-1`, + }), + ) + expect(stripVirtualProps(persistedCollection.get(1))).toEqual({ + id: 1, + name: `Persisted User`, + }) + + subscriber([ + { + key: `2`, + value: { id: 2, name: `Resumed User` }, + headers: { operation: `insert` }, + }, + ]) + subscriber([ + { + headers: { control: `up-to-date` }, + }, + ]) + + expect( + [persistedCollection.get(1), persistedCollection.get(2)].map( + stripVirtualProps, + ), + ).toEqual([ + { id: 1, name: `Persisted User` }, + { id: 2, name: `Resumed User` }, + ]) + await vi.waitFor(() => { + expect( + [persistedRows.get(1), persistedRows.get(2)].map(stripVirtualProps), + ).toEqual([ + { id: 1, name: `Persisted User` }, + { id: 2, name: `Resumed User` }, + ]) + }) + }) + it(`should not mix explicit handle with persisted offset`, async () => { vi.clearAllMocks() @@ -3189,6 +4139,7 @@ describe(`Electric Integration`, () => { `electric:resume`, { kind: `resume`, + requiresTagState: false, offset: `10_0`, handle: `persisted-handle`, shapeId: JSON.stringify({ @@ -3245,6 +4196,7 @@ describe(`Electric Integration`, () => { `electric:resume`, { kind: `resume`, + requiresTagState: false, offset: `10_0`, handle: `persisted-handle`, shapeId: JSON.stringify({ @@ -3301,6 +4253,7 @@ describe(`Electric Integration`, () => { `electric:resume`, { kind: `resume`, + requiresTagState: false, offset: 10, updatedAt: 1, }, @@ -3351,6 +4304,7 @@ describe(`Electric Integration`, () => { `electric:resume`, { kind: `resume`, + requiresTagState: false, offset: `10_0`, handle: `handle-1`, shapeId: `{"url":"http://other-url","params":{"table":"test_table"}}`, @@ -3408,6 +4362,7 @@ describe(`Electric Integration`, () => { `electric:resume`, { kind: `resume`, + requiresTagState: false, offset: `10_0`, handle: `handle-1`, shapeId: `{"params":{"table":"test_table","where":"room=1"},"url":"http://test-url"}`, @@ -3540,11 +4495,128 @@ describe(`Electric Integration`, () => { expect(metadataHarness.collectionMetadata.get(`electric:resume`)).toEqual( expect.objectContaining({ kind: `resume`, + requiresTagState: false, offset: `10_0`, handle: `shape-1`, }), ) }) + + it(`refuses an update for an unseen key and invalidates persisted resume state`, () => { + const metadataHarness = createInMemorySyncMetadataApi( + new Map([ + [ + `electric:resume`, + { + kind: `resume`, + requiresTagState: false, + offset: `10_0`, + handle: `shape-1`, + shapeId: `{"params":{"table":"test_table"},"url":"http://test-url"}`, + updatedAt: 1, + }, + ], + ]), + ) + mockStream.shapeHandle = `shape-1` + mockStream.lastOffset = `11_0` + + const baseOptions = electricCollectionOptions({ + id: `unseen-update-resume-test`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (item) => item.id as number, + startSync: true, + }) + const originalSync = baseOptions.sync + const testCollection = createCollection({ + ...baseOptions, + sync: { + sync: (params: Parameters[0]) => + originalSync.sync({ ...params, metadata: metadataHarness.api }), + }, + }) + + subscriber([ + { + key: `2`, + value: { id: 2, name: `Changed without immutable fields` }, + headers: { operation: `update` }, + }, + { headers: { control: `up-to-date` } }, + ]) + + expect(testCollection.has(2)).toBe(false) + expect(metadataHarness.collectionMetadata.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `reset` }), + ) + expect(testCollection.status).toBe(`error`) + }) + + it(`rejects a resumed batch that updates a key after deleting it`, () => { + const metadataHarness = createInMemorySyncMetadataApi( + new Map([ + [ + `electric:resume`, + { + kind: `resume`, + requiresTagState: false, + offset: `10_0`, + handle: `shape-1`, + shapeId: `{"params":{"table":"test_table"},"url":"http://test-url"}`, + updatedAt: 1, + }, + ], + ]), + ) + const baseOptions = electricCollectionOptions({ + id: `delete-then-update-resume-test`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + getKey: (item) => item.id as number, + startSync: true, + }) + const originalSync = baseOptions.sync + const testCollection = createCollection({ + ...baseOptions, + sync: { + sync: (params: Parameters[0]) => + originalSync.sync({ ...params, metadata: metadataHarness.api }), + }, + }) + + subscriber([ + { + key: `2`, + value: { id: 2, name: `Complete row` }, + headers: { operation: `insert` }, + }, + { headers: { control: `up-to-date` } }, + ]) + expect(testCollection.has(2)).toBe(true) + + subscriber([ + { + key: `2`, + value: { id: 2 }, + headers: { operation: `delete` }, + }, + { + key: `2`, + value: { id: 2, name: `Partial replacement` }, + headers: { operation: `update` }, + }, + ]) + + expect(testCollection.get(2)?.name).toBe(`Complete row`) + expect(metadataHarness.collectionMetadata.get(`electric:resume`)).toEqual( + expect.objectContaining({ kind: `reset` }), + ) + }) }) // Tests for overlapping subset queries with duplicate keys diff --git a/packages/electric-db-collection/tests/tags.test.ts b/packages/electric-db-collection/tests/tags.test.ts index 2aa765ef2d..48af9c6b15 100644 --- a/packages/electric-db-collection/tests/tags.test.ts +++ b/packages/electric-db-collection/tests/tags.test.ts @@ -688,6 +688,42 @@ describe(`Electric Tag Tracking and GC`, () => { expect(collection.state.get(3)).toEqual({ id: 3, name: `User 3` }) }) + it(`does not accept a partial update after move-out deletes a row`, () => { + subscriber([ + { + key: `1`, + value: { id: 1, name: `complete`, stable: `preserved` }, + headers: { + operation: `insert`, + tags: [`hash1/hash2/hash3`], + }, + }, + { headers: { control: `up-to-date` } }, + ]) + + subscriber([ + { + headers: { + event: `move-out`, + patterns: [{ pos: 0, value: `hash1` }], + }, + }, + { headers: { control: `up-to-date` } }, + ]) + expect(collection.has(1)).toBe(false) + + subscriber([ + { + key: `1`, + value: { id: 1, name: `partial` }, + headers: { operation: `update` }, + }, + { headers: { control: `up-to-date` } }, + ]) + + expect(collection.has(1)).toBe(false) + }) + it(`should remove shared tags from all rows when move-out pattern matches`, () => { // Create tags where some are shared between rows const sharedTag1 = `hash1/hash2/hash3` // Shared by rows 1 and 2 diff --git a/packages/electron-db-sqlite-persistence/CHANGELOG.md b/packages/electron-db-sqlite-persistence/CHANGELOG.md index 9a74f0ce27..fed79ac9d3 100644 --- a/packages/electron-db-sqlite-persistence/CHANGELOG.md +++ b/packages/electron-db-sqlite-persistence/CHANGELOG.md @@ -1,5 +1,154 @@ # @tanstack/electron-db-sqlite-persistence +## 0.1.33 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db-sqlite-persistence-core@0.2.21 + +## 0.1.32 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.20 + +## 0.1.31 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.19 + +## 0.1.30 + +### Patch Changes + +- Updated dependencies [[`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db-sqlite-persistence-core@0.2.18 + +## 0.1.29 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.17 + +## 0.1.28 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.16 + +## 0.1.27 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.15 + +## 0.1.26 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.14 + +## 0.1.25 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db-sqlite-persistence-core@0.2.13 + +## 0.1.24 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.12 + +## 0.1.23 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.11 + +## 0.1.22 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.10 + +## 0.1.21 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.9 + +## 0.1.20 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.8 + +## 0.1.19 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.7 + +## 0.1.18 + +### Patch Changes + +- Updated dependencies [[`f7da776`](https://github.com/TanStack/db/commit/f7da77660b16cbfe30817fb5c938267d696c8d1c)]: + - @tanstack/db-sqlite-persistence-core@0.2.6 + +## 0.1.17 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.5 + +## 0.1.16 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.4 + +## 0.1.15 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.3 + +## 0.1.14 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.2 + +## 0.1.13 + +### Patch Changes + +- Use a safe `randomUUID` helper that falls back to `crypto.getRandomValues` when `crypto.randomUUID` is unavailable (non-secure browser contexts such as dev servers reached via a LAN IP over HTTP). Fixes #1541. ([#1593](https://github.com/TanStack/db/pull/1593)) + +- Updated dependencies [[`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db-sqlite-persistence-core@0.2.1 + ## 0.1.12 ### Patch Changes diff --git a/packages/electron-db-sqlite-persistence/package.json b/packages/electron-db-sqlite-persistence/package.json index 18f38372f7..d50b29706b 100644 --- a/packages/electron-db-sqlite-persistence/package.json +++ b/packages/electron-db-sqlite-persistence/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/electron-db-sqlite-persistence", - "version": "0.1.12", + "version": "0.1.33", "description": "Electron SQLite persisted collection bridge for TanStack DB", "author": "TanStack Team", "license": "MIT", diff --git a/packages/electron-db-sqlite-persistence/src/electron-coordinator.ts b/packages/electron-db-sqlite-persistence/src/electron-coordinator.ts index ea9735e709..a4c6bb7fe8 100644 --- a/packages/electron-db-sqlite-persistence/src/electron-coordinator.ts +++ b/packages/electron-db-sqlite-persistence/src/electron-coordinator.ts @@ -1,3 +1,4 @@ +import { safeRandomUUID } from '@tanstack/db-sqlite-persistence-core' import type { ApplyLocalMutationsResponse, PersistedCollectionCoordinator, @@ -118,7 +119,7 @@ export type ElectronCollectionCoordinatorOptions = { // --------------------------------------------------------------------------- export class ElectronCollectionCoordinator implements PersistedCollectionCoordinator { - private readonly nodeId = crypto.randomUUID() + private readonly nodeId = safeRandomUUID() private readonly dbName: string private adapter: AdapterWithPullSince | null private readonly channel: BroadcastChannel @@ -205,7 +206,7 @@ export class ElectronCollectionCoordinator implements PersistedCollectionCoordin error?: string }>(collectionId, { type: `rpc:ensureRemoteSubset:req`, - rpcId: crypto.randomUUID(), + rpcId: safeRandomUUID(), options, }) @@ -233,7 +234,7 @@ export class ElectronCollectionCoordinator implements PersistedCollectionCoordin error?: string }>(collectionId, { type: `rpc:ensurePersistedIndex:req`, - rpcId: crypto.randomUUID(), + rpcId: safeRandomUUID(), signature, spec, }) @@ -252,16 +253,16 @@ export class ElectronCollectionCoordinator implements PersistedCollectionCoordin if (this.isLeader(collectionId)) { return this.handleApplyLocalMutations(collectionId, { type: `rpc:applyLocalMutations:req`, - rpcId: crypto.randomUUID(), - envelopeId: crypto.randomUUID(), + rpcId: safeRandomUUID(), + envelopeId: safeRandomUUID(), mutations, }) } return this.sendRPC(collectionId, { type: `rpc:applyLocalMutations:req`, - rpcId: crypto.randomUUID(), - envelopeId: crypto.randomUUID(), + rpcId: safeRandomUUID(), + envelopeId: safeRandomUUID(), mutations, }) } @@ -273,14 +274,14 @@ export class ElectronCollectionCoordinator implements PersistedCollectionCoordin if (this.isLeader(collectionId)) { return this.handlePullSince(collectionId, { type: `rpc:pullSince:req`, - rpcId: crypto.randomUUID(), + rpcId: safeRandomUUID(), fromRowVersion, }) } return this.sendRPC(collectionId, { type: `rpc:pullSince:req`, - rpcId: crypto.randomUUID(), + rpcId: safeRandomUUID(), fromRowVersion, }) } @@ -663,7 +664,7 @@ export class ElectronCollectionCoordinator implements PersistedCollectionCoordin // Build and apply the persisted transaction const tx = { - txId: crypto.randomUUID(), + txId: safeRandomUUID(), term, seq, rowVersion, diff --git a/packages/expo-db-sqlite-persistence/CHANGELOG.md b/packages/expo-db-sqlite-persistence/CHANGELOG.md index b1cded9a2c..363fe9c2ef 100644 --- a/packages/expo-db-sqlite-persistence/CHANGELOG.md +++ b/packages/expo-db-sqlite-persistence/CHANGELOG.md @@ -1,5 +1,152 @@ # @tanstack/expo-db-sqlite-persistence +## 0.2.21 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db-sqlite-persistence-core@0.2.21 + +## 0.2.20 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.20 + +## 0.2.19 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.19 + +## 0.2.18 + +### Patch Changes + +- Updated dependencies [[`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db-sqlite-persistence-core@0.2.18 + +## 0.2.17 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.17 + +## 0.2.16 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.16 + +## 0.2.15 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.15 + +## 0.2.14 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.14 + +## 0.2.13 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db-sqlite-persistence-core@0.2.13 + +## 0.2.12 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.12 + +## 0.2.11 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.11 + +## 0.2.10 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.10 + +## 0.2.9 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.9 + +## 0.2.8 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.8 + +## 0.2.7 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.7 + +## 0.2.6 + +### Patch Changes + +- Updated dependencies [[`f7da776`](https://github.com/TanStack/db/commit/f7da77660b16cbfe30817fb5c938267d696c8d1c)]: + - @tanstack/db-sqlite-persistence-core@0.2.6 + +## 0.2.5 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.5 + +## 0.2.4 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.4 + +## 0.2.3 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.3 + +## 0.2.2 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.2 + +## 0.2.1 + +### Patch Changes + +- Updated dependencies [[`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db-sqlite-persistence-core@0.2.1 + ## 0.2.0 ### Minor Changes diff --git a/packages/expo-db-sqlite-persistence/e2e/expo-runtime-app/CHANGELOG.md b/packages/expo-db-sqlite-persistence/e2e/expo-runtime-app/CHANGELOG.md index 935e723540..3a5a9c4f21 100644 --- a/packages/expo-db-sqlite-persistence/e2e/expo-runtime-app/CHANGELOG.md +++ b/packages/expo-db-sqlite-persistence/e2e/expo-runtime-app/CHANGELOG.md @@ -1,5 +1,173 @@ # @tanstack/expo-db-sqlite-persistence-e2e-app +## 0.0.33 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + - @tanstack/expo-db-sqlite-persistence@0.2.21 + +## 0.0.32 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + - @tanstack/expo-db-sqlite-persistence@0.2.20 + +## 0.0.31 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + - @tanstack/expo-db-sqlite-persistence@0.2.19 + +## 0.0.30 + +### Patch Changes + +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + - @tanstack/expo-db-sqlite-persistence@0.2.18 + +## 0.0.29 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + - @tanstack/expo-db-sqlite-persistence@0.2.17 + +## 0.0.28 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + - @tanstack/expo-db-sqlite-persistence@0.2.16 + +## 0.0.27 + +### Patch Changes + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + - @tanstack/expo-db-sqlite-persistence@0.2.15 + +## 0.0.26 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + - @tanstack/expo-db-sqlite-persistence@0.2.14 + +## 0.0.25 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + - @tanstack/expo-db-sqlite-persistence@0.2.13 + +## 0.0.24 + +### Patch Changes + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + - @tanstack/expo-db-sqlite-persistence@0.2.12 + +## 0.0.23 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + - @tanstack/expo-db-sqlite-persistence@0.2.11 + +## 0.0.22 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + - @tanstack/expo-db-sqlite-persistence@0.2.10 + +## 0.0.21 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + - @tanstack/expo-db-sqlite-persistence@0.2.9 + +## 0.0.20 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + - @tanstack/expo-db-sqlite-persistence@0.2.8 + +## 0.0.19 + +### Patch Changes + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + - @tanstack/expo-db-sqlite-persistence@0.2.7 + +## 0.0.18 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + - @tanstack/expo-db-sqlite-persistence@0.2.6 + +## 0.0.17 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + - @tanstack/expo-db-sqlite-persistence@0.2.5 + +## 0.0.16 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + - @tanstack/expo-db-sqlite-persistence@0.2.4 + +## 0.0.15 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + - @tanstack/expo-db-sqlite-persistence@0.2.3 + +## 0.0.14 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + - @tanstack/expo-db-sqlite-persistence@0.2.2 + +## 0.0.13 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + - @tanstack/expo-db-sqlite-persistence@0.2.1 + ## 0.0.12 ### Patch Changes diff --git a/packages/expo-db-sqlite-persistence/e2e/expo-runtime-app/package.json b/packages/expo-db-sqlite-persistence/e2e/expo-runtime-app/package.json index 8922e1a5c6..08956dddb5 100644 --- a/packages/expo-db-sqlite-persistence/e2e/expo-runtime-app/package.json +++ b/packages/expo-db-sqlite-persistence/e2e/expo-runtime-app/package.json @@ -1,7 +1,7 @@ { "name": "@tanstack/expo-db-sqlite-persistence-e2e-app", "private": true, - "version": "0.0.12", + "version": "0.0.33", "main": "index.js", "scripts": { "start": "expo start", diff --git a/packages/expo-db-sqlite-persistence/package.json b/packages/expo-db-sqlite-persistence/package.json index 7ff8f3554c..178039a158 100644 --- a/packages/expo-db-sqlite-persistence/package.json +++ b/packages/expo-db-sqlite-persistence/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/expo-db-sqlite-persistence", - "version": "0.2.0", + "version": "0.2.21", "description": "Expo SQLite persisted collection adapter for TanStack DB", "author": "TanStack Team", "license": "MIT", diff --git a/packages/node-db-sqlite-persistence/CHANGELOG.md b/packages/node-db-sqlite-persistence/CHANGELOG.md index b427f31a5e..5ae10c2271 100644 --- a/packages/node-db-sqlite-persistence/CHANGELOG.md +++ b/packages/node-db-sqlite-persistence/CHANGELOG.md @@ -1,5 +1,152 @@ # @tanstack/node-db-sqlite-persistence +## 0.2.21 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db-sqlite-persistence-core@0.2.21 + +## 0.2.20 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.20 + +## 0.2.19 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.19 + +## 0.2.18 + +### Patch Changes + +- Updated dependencies [[`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db-sqlite-persistence-core@0.2.18 + +## 0.2.17 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.17 + +## 0.2.16 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.16 + +## 0.2.15 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.15 + +## 0.2.14 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.14 + +## 0.2.13 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db-sqlite-persistence-core@0.2.13 + +## 0.2.12 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.12 + +## 0.2.11 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.11 + +## 0.2.10 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.10 + +## 0.2.9 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.9 + +## 0.2.8 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.8 + +## 0.2.7 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.7 + +## 0.2.6 + +### Patch Changes + +- Updated dependencies [[`f7da776`](https://github.com/TanStack/db/commit/f7da77660b16cbfe30817fb5c938267d696c8d1c)]: + - @tanstack/db-sqlite-persistence-core@0.2.6 + +## 0.2.5 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.5 + +## 0.2.4 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.4 + +## 0.2.3 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.3 + +## 0.2.2 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.2 + +## 0.2.1 + +### Patch Changes + +- Updated dependencies [[`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db-sqlite-persistence-core@0.2.1 + ## 0.2.0 ### Minor Changes diff --git a/packages/node-db-sqlite-persistence/package.json b/packages/node-db-sqlite-persistence/package.json index 4391f7b071..0e54a70f4d 100644 --- a/packages/node-db-sqlite-persistence/package.json +++ b/packages/node-db-sqlite-persistence/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/node-db-sqlite-persistence", - "version": "0.2.0", + "version": "0.2.21", "description": "Node SQLite persisted collection adapter for TanStack DB", "author": "TanStack Team", "license": "MIT", diff --git a/packages/offline-transactions/CHANGELOG.md b/packages/offline-transactions/CHANGELOG.md index 2057952ea4..8d16f8ffc1 100644 --- a/packages/offline-transactions/CHANGELOG.md +++ b/packages/offline-transactions/CHANGELOG.md @@ -1,5 +1,156 @@ # @tanstack/offline-transactions +## 1.0.54 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 1.0.53 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 1.0.52 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 1.0.51 + +### Patch Changes + +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 1.0.50 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 1.0.49 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 1.0.48 + +### Patch Changes + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 1.0.47 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 1.0.46 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 1.0.45 + +### Patch Changes + +- Update agent skills to match current APIs and behavior. ([#1696](https://github.com/TanStack/db/pull/1696)) + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 1.0.44 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 1.0.43 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 1.0.42 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 1.0.41 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 1.0.40 + +### Patch Changes + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 1.0.39 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 1.0.38 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 1.0.37 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 1.0.36 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 1.0.35 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 1.0.34 + +### Patch Changes + +- Use a safe `randomUUID` helper that falls back to `crypto.getRandomValues` when `crypto.randomUUID` is unavailable (non-secure browser contexts such as dev servers reached via a LAN IP over HTTP). Fixes #1541. ([#1593](https://github.com/TanStack/db/pull/1593)) + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 1.0.33 ### Patch Changes diff --git a/packages/offline-transactions/package.json b/packages/offline-transactions/package.json index 872a47b924..1b3f886344 100644 --- a/packages/offline-transactions/package.json +++ b/packages/offline-transactions/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/offline-transactions", - "version": "1.0.33", + "version": "1.0.54", "description": "Offline-first transaction capabilities for TanStack DB", "author": "TanStack", "license": "MIT", diff --git a/packages/offline-transactions/skills/offline/SKILL.md b/packages/offline-transactions/skills/offline/SKILL.md index 1295640dc6..6bf6dfc326 100644 --- a/packages/offline-transactions/skills/offline/SKILL.md +++ b/packages/offline-transactions/skills/offline/SKILL.md @@ -10,7 +10,7 @@ description: > React Native support via separate entry point. type: composition library: db -library_version: '0.6.0' +library_version: '0.6.17' requires: - db-core - db-core/mutations-optimistic @@ -31,6 +31,7 @@ import { startOfflineExecutor, IndexedDBAdapter, } from '@tanstack/offline-transactions' +import { safeRandomUUID } from '@tanstack/db' import { todoCollection } from './collections' const executor = startOfflineExecutor({ @@ -68,7 +69,7 @@ const tx = executor.createOfflineTransaction({ // Mutations run inside tx.mutate() — uses ambient transaction context tx.mutate(() => { - todoCollection.insert({ id: crypto.randomUUID(), text: 'New todo' }) + todoCollection.insert({ id: safeRandomUUID(), text: 'New todo' }) }) tx.commit() ``` @@ -82,7 +83,7 @@ const addTodo = executor.createOfflineAction({ mutationFnName: 'createTodo', onMutate: (variables) => { todoCollection.insert({ - id: crypto.randomUUID(), + id: safeRandomUUID(), text: variables.text, }) }, diff --git a/packages/offline-transactions/src/OfflineExecutor.ts b/packages/offline-transactions/src/OfflineExecutor.ts index 8f443277c2..a6140cfebc 100644 --- a/packages/offline-transactions/src/OfflineExecutor.ts +++ b/packages/offline-transactions/src/OfflineExecutor.ts @@ -1,5 +1,9 @@ // Storage adapters -import { createOptimisticAction, createTransaction } from '@tanstack/db' +import { + createOptimisticAction, + createTransaction, + safeRandomUUID, +} from '@tanstack/db' import { IndexedDBAdapter } from './storage/IndexedDBAdapter' import { LocalStorageAdapter } from './storage/LocalStorageAdapter' @@ -367,7 +371,7 @@ export class OfflineExecutor { mutationFn: (params) => mutationFn({ ...params, - idempotencyKey: options.idempotencyKey || crypto.randomUUID(), + idempotencyKey: options.idempotencyKey || safeRandomUUID(), }), metadata: options.metadata, }) @@ -399,7 +403,7 @@ export class OfflineExecutor { mutationFn({ ...vars, ...params, - idempotencyKey: crypto.randomUUID(), + idempotencyKey: safeRandomUUID(), }), onMutate: options.onMutate, }) diff --git a/packages/offline-transactions/src/api/OfflineTransaction.ts b/packages/offline-transactions/src/api/OfflineTransaction.ts index 32c96fd26a..af13484a75 100644 --- a/packages/offline-transactions/src/api/OfflineTransaction.ts +++ b/packages/offline-transactions/src/api/OfflineTransaction.ts @@ -1,4 +1,4 @@ -import { createTransaction } from '@tanstack/db' +import { createTransaction, safeRandomUUID } from '@tanstack/db' import { NonRetriableError } from '../types' import type { PendingMutation, Transaction } from '@tanstack/db' import type { @@ -23,10 +23,10 @@ export class OfflineTransaction { persistTransaction: (tx: OfflineTransactionType) => Promise, executor: any, ) { - this.offlineId = crypto.randomUUID() + this.offlineId = safeRandomUUID() this.mutationFnName = options.mutationFnName this.autoCommit = options.autoCommit ?? true - this.idempotencyKey = options.idempotencyKey ?? crypto.randomUUID() + this.idempotencyKey = options.idempotencyKey ?? safeRandomUUID() this.metadata = options.metadata ?? {} this.persistTransaction = persistTransaction this.executor = executor diff --git a/packages/offline-transactions/src/coordination/BroadcastChannelLeader.ts b/packages/offline-transactions/src/coordination/BroadcastChannelLeader.ts index ce11a8abc3..7f50d06459 100644 --- a/packages/offline-transactions/src/coordination/BroadcastChannelLeader.ts +++ b/packages/offline-transactions/src/coordination/BroadcastChannelLeader.ts @@ -1,3 +1,4 @@ +import { safeRandomUUID } from '@tanstack/db' import { BaseLeaderElection } from './LeaderElection' interface LeaderMessage { @@ -19,7 +20,7 @@ export class BroadcastChannelLeader extends BaseLeaderElection { constructor(channelName = `offline-executor-leader`) { super() this.channelName = channelName - this.tabId = crypto.randomUUID() + this.tabId = safeRandomUUID() this.setupChannel() } diff --git a/packages/powersync-db-collection/CHANGELOG.md b/packages/powersync-db-collection/CHANGELOG.md index b5447bdea9..ccf75f65dc 100644 --- a/packages/powersync-db-collection/CHANGELOG.md +++ b/packages/powersync-db-collection/CHANGELOG.md @@ -1,5 +1,218 @@ # @tanstack/powersync-db-collection +## 0.1.67 + +### Patch Changes + +- Fix on-demand load settlement, ordered pagination, and replay to preserve coherent results across cancellation, failure, cleanup, and restart. Preserve subset results and ownership across Electric, PowerSync, Query, and SQLite persistence adapters. Correct live-query grouping, include projections, value identity, and indexed comparisons. D2 hashing now rejects structural cycles and excessive traversal depth or work with an explicit error; Collection handles retain object-reference identity without traversing their mutable contents. ([#1797](https://github.com/TanStack/db/pull/1797)) + + Remove the unused public subset-algebra helpers: `isWhereSubset`, `unionWherePredicates`, `minusWherePredicates`, `isOrderBySubset`, `isLimitSubset`, `isOffsetLimitSubset`, `isPredicateSubset`, and `isLoadSubsetRequestSubsumedBy`. Apps that import these helpers must remove those imports; normal queries and adapters are unaffected. `DeduplicatedLoadSubset` remains available and shares only exact demand identities. + + Reject compiled Collection-valued includes as `fn.select()` inputs, including nested descendants, before invoking the callback. Use `toArray()` or `materialize()` in the upstream `.select()` for child-value calculations. To keep live child Collections, use expression `.select()` or perform parent-only functional work before adding the includes. Ordinary Collection-valued includes remain supported. + + Remove proxy `DEBUG` logging and automatic index timing statistics to avoid diagnostic work on reads and writes. Remove `getStats()` and `IndexStats`; use `index.keyCount` for the current entry count, and instrument index methods externally when profiling. Custom index subclasses must remove calls to the retired `trackLookup()` and `updateTimestamp()` helpers. + + Fix mutation drafts so Map/Set `forEach` calls each callback once and read-only iteration reports no changes. Track nested Map `for...of` edits through `collection.update()`. Keep Set entries in place during nested edits, preserving live iteration and usable `has`, `delete`, and `add` handles without duplicate entries or repeated visits. Reverting one entry no longer discards another entry's pending changes. + + Preserve shared values within a row's draft. Newly supplied objects use normal shared references during the update callback, including Map values and Set members; editing through either handle changes the same new object. Copy completed changes at callback return so later caller edits cannot mutate stored data. Existing collection values remain isolated. Copy a new caller-owned value before insertion if it must remain untouched during the callback. + + Keep rejected local-storage mutations out of later successful saves. Stage insert, update, delete, and manual transaction writes before persisting, then promote the shared cache only after storage succeeds. + + Deliver live-query observer publications to peer listeners even when one listener throws. Preserve queued publication order and stop delivery on disposal; report the first listener failure after delivery. + + Wait for PowerSync demands admitted during tracking finalization before reporting them loaded. Preserve untouched object and array key identity when they contain `NaN`. Retire every failed ordered acquisition on explicit retry, while retaining a full-source demand repaired by replay. + + After authoritative ordered recovery succeeds, retire settled page and tie demands so later truncate replay fetches only the full-source replacement. Keep unfinished requests observed until settlement and preserve independent query ownership. + + Remove the live-query `utils.getRunCount()` diagnostic and its runtime counter. Apps that call this diagnostic must remove those calls. Query scheduling and results are unchanged. + + Remove test-only index inspection getters (`indexedKeysSet`, `valueMapData`, `orderedEntriesArray`, and `orderedEntriesArrayReversed`) and unused scheduler diagnostics. `ReverseIndex` now exposes only the `IndexReader` lookup, range, and forward traversal surface returned by `findIndexForField`; mutate the original index instead. Export `IndexReader` for callers that name this return type. Remove the unused subscription `releaseLoadSubset()` method; request owners use the release callback supplied by `onLoadSubsetResult`. Ordinary query and adapter APIs remain unchanged by these removals. + + Remove unused internal helpers and the unused public error classes `WhereClauseConversionError`, `SubscriptionNotFoundError`, and `AggregateNotSupportedError`. These classes have no remaining runtime throw sites; remove any imports of them. Keep the existing query and index behavior and exercise identity/evaluation tests through the production entry points. + + Ensure failed mutations roll back even when their rejection value cannot be converted to a string. Preserve ordinary Error instances; report unprintable rejection values as `Unknown error`. + + Make reentrant effect disposal share the active cleanup result, including calls from abort listeners or source release callbacks. A release failure reaches every waiting disposer while each source still receives one release attempt. + + Reject starting or preloading a collection from inside its active cleanup callbacks with a clear `CollectionStateError`. Nested cleanup cannot admit replacement work that the old teardown would discard. Restart after cleanup completes, or from its final `cleaned-up` status event, remains supported. + + Prevent older page or tie-boundary completions from clearing a newer full-source failure or starting redundant loading. Failed window moves retain their settled public snapshot; an explicit retry releases the failed acquisition once and publishes the completed replacement. + + Restrict direct subscription `requestLimitedSnapshot()` cursor inputs to one order term and one `minValues` entry. Composite and partial-composite cursor inputs now throw before local delivery or adapter work. Use normal live-query ordering and window APIs for multi-column pagination; those remain supported through prefix-and-tie loading. Existing adapters need no changes. + + Treat `LoadSubsetOptions` and their nested request data as immutable from submission onward. Core no longer copies expression trees or mutable constant payloads at the sync and deduplication boundaries. Create a new Date, byte array, membership array, or options object when changing a demand instead of mutating submitted data. Adapters must also leave request data unchanged. Use stable data properties rather than stateful getters. `AbortSignal` cancellation and subscription release remain live. + + Replace replay acquisitions sequentially: release the prior physical lease before starting its replacement. The logical demand and last complete public result remain retained. A failed release prevents replacement startup; failed startup leaves demand available for a later authoritative replay. Custom adapters must support a release/load gap and preserve resources still held by other owners; a sole underlying resource may stop and restart. + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 0.1.66 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 0.1.65 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 0.1.64 + +### Patch Changes + +- Settle subset loads only after their committed rows and events are visible. A ([#1769](https://github.com/TanStack/db/pull/1769)) + commit receipt now rejects with `AbortError` when cancellation wins before + application and ignores later aborts. Preserve causal publication, + cancellation, persistence, and error handling across the affected sync + adapters. +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 0.1.63 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 0.1.62 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 0.1.61 + +### Patch Changes + +- Propagate initial query sync failures through dependent live queries and readiness promises, including recovery and late subscribers, while preserving a ready cached snapshot on later refetch failures. Let sync adapters pass the original failure to `markError(error)` so readiness promises reject with that cause. Isolate adapter callbacks by sync session, preserve synchronous startup errors, and prevent rejected deduplicated subset requests from creating detached promise rejections. ([#1751](https://github.com/TanStack/db/pull/1751)) + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 0.1.60 + +### Patch Changes + +- Rebuild correlated include materialization as one D2 graph, fixing stale or missing nested results across route changes, batching, lazy loading, optimistic updates, and layered queries. Add canonical structural relation keys, abortable subset demand, and coherent publication for Collection-valued includes. Dispose delayed PowerSync subset hooks after cleanup, and prevent released Query Collection cache results from reaching the collection. ([#1740](https://github.com/TanStack/db/pull/1740)) + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 0.1.59 + +### Patch Changes + +- Add SSR through request-scoped `DbClient` instances, collection descriptors, ([#1564](https://github.com/TanStack/db/pull/1564)) + explicit collection-row hydration, live-query result snapshots, adapter sync + metadata, and React and Svelte descriptor resolution. + + React live queries now derive identity from structured query IR. Opaque queries + can provide `queryKey`; legacy dependency arrays and unkeyed opaque queries keep + working with development warnings until 1.0. + + Add TanStack Router integration that streams live queries discovered during a + Suspense render as pending promises which resolve to ordered result snapshots. + The browser starts normal source sync and atomically replaces the snapshot when + its live result is ready. + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 0.1.58 + +### Patch Changes + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 0.1.57 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 0.1.56 + +### Patch Changes + +- Fixed `no such table` errors logged by the `on-demand` sync handler. Records are no longer flushed before the `diffTrigger` has been set up, and the tracking state is now cleared as part of disposal so unloading a subset or cleaning up the collection no longer flushes the dropped tracking table. ([#1585](https://github.com/TanStack/db/pull/1585)) + +- Fix: applyTransaction hangs forever when a transaction mixes delete+insert on one collection for a same-millisecond tie. ([#1649](https://github.com/TanStack/db/pull/1649)) + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 0.1.55 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 0.1.54 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 0.1.53 + +### Patch Changes + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 0.1.52 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 0.1.51 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 0.1.50 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 0.1.49 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 0.1.48 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 0.1.47 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 0.1.46 ### Patch Changes diff --git a/packages/powersync-db-collection/package.json b/packages/powersync-db-collection/package.json index 93444c4f26..9750917239 100644 --- a/packages/powersync-db-collection/package.json +++ b/packages/powersync-db-collection/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/powersync-db-collection", - "version": "0.1.46", + "version": "0.1.67", "description": "PowerSync collection for TanStack DB", "author": "POWERSYNC", "license": "MIT", @@ -27,7 +27,8 @@ "build": "vite build", "dev": "vite build --watch", "lint": "eslint . --fix", - "test": "vitest --run" + "test": "vitest --run", + "test:upstream-repros": "vitest run --config tests/upstream.config.ts --maxWorkers=2" }, "type": "module", "main": "dist/cjs/index.cjs", @@ -59,11 +60,11 @@ "p-defer": "^4.0.1" }, "peerDependencies": { - "@powersync/common": "^1.41.0" + "@powersync/common": "^1.57.0" }, "devDependencies": { - "@powersync/common": "1.49.0", - "@powersync/node": "0.18.1", + "@powersync/common": "1.57.0", + "@powersync/node": "0.19.2", "@types/debug": "^4.1.12", "@vitest/coverage-istanbul": "^3.2.4", "better-sqlite3": "^12.6.2" diff --git a/packages/powersync-db-collection/src/PowerSyncTransactor.ts b/packages/powersync-db-collection/src/PowerSyncTransactor.ts index 2542b6d545..b5d14a5287 100644 --- a/packages/powersync-db-collection/src/PowerSyncTransactor.ts +++ b/packages/powersync-db-collection/src/PowerSyncTransactor.ts @@ -1,4 +1,5 @@ import { sanitizeSQL } from '@powersync/common' +import { LoadSubsetOperationAbortedError } from '@tanstack/db' import DebugModule from 'debug' import { PendingOperationStore } from './PendingOperationStore' import { asPowerSyncRecord, mapOperationToPowerSync } from './helpers' @@ -94,7 +95,29 @@ export class PowerSyncTransactor { if (collection.isReady()) { return } - await new Promise((resolve) => collection.onFirstReady(resolve)) + // Observe this session without starting new demand from mutationFn. + // Cleanup and startup failure must settle the wait before taking a lock. + await new Promise((resolve, reject) => { + const check = () => { + if (collection.isReady()) { + unsubscribe() + resolve() + } else if ( + collection.status === `error` || + collection.status === `cleaned-up` + ) { + unsubscribe() + reject( + collection.status === `error` + ? (collection._lifecycle.getSyncError() ?? + new Error(`Collection failed before readiness`)) + : new LoadSubsetOperationAbortedError(), + ) + } + } + const unsubscribe = collection.on(`status:change`, check) + check() + }) }), ) @@ -284,7 +307,7 @@ export class PowerSyncTransactor { // Need to get the operation in order to wait for it const diffOperation = await context.get<{ id: string; timestamp: string }>( - sanitizeSQL`SELECT id, timestamp FROM ${trackedTableName} ORDER BY timestamp DESC LIMIT 1`, + sanitizeSQL`SELECT id, timestamp FROM ${trackedTableName} ORDER BY operation_id DESC LIMIT 1`, ) return { tableName, diff --git a/packages/powersync-db-collection/src/attachments.ts b/packages/powersync-db-collection/src/attachments.ts new file mode 100644 index 0000000000..465d45d13a --- /dev/null +++ b/packages/powersync-db-collection/src/attachments.ts @@ -0,0 +1,263 @@ +import { + AttachmentQueue, + AttachmentState, + sanitizeSQL, +} from '@powersync/common' +import { createLiveQueryCollection, createTransaction, eq } from '@tanstack/db' +import { PowerSyncTransactor } from './PowerSyncTransactor' + +import type { + AbstractPowerSyncDatabase, + AttachmentData, + AttachmentQueueOptions, + AttachmentTable, +} from '@powersync/common' +import type { Collection, Transaction } from '@tanstack/db' +import type { OptionalExtractedTable } from './helpers' + +// SDK context locks belong to individual queues. Reserve saves across queues +// sharing this database so a rejected insert cannot remove another save's file. +const savingIds = new WeakMap>() + +export type TanStackDBAttachmentQueueOptions = AttachmentQueueOptions & { + /** + * For TanStack, we want access to the synced TanStackDB collection. + * In order to have the same relational data be set in a single transaction. + * This also allows for joining both TanStackDB collections. + */ + attachmentsCollection: Collection +} + +export interface SaveOptions { + data: AttachmentData + fileExtension: string + mediaType?: string + metaData?: string + /** + * Optional custom ID. If not provided, a UUID will be generated. + * + * Rejected if this ID is already in the queue or is being saved by another + * call sharing the same PowerSync database object. + */ + id?: string + /** + * Called synchronously within the same TanStackDB transaction as the attachment write, + * so any mutations made to other collections are committed atomically with it. + * + * Must not be async. `Transaction.mutate` unregisters the ambient transaction as soon as + * this callback returns, so any mutation made after an `await` inside the hook escapes the + * transaction and is not committed atomically with the attachment. Do asynchronous work + * before calling `save` or `delete`. + */ + updateHook?: (attachment: AttachmentQueueRow) => void +} + +export interface DeleteOptions { + id: string + /** + * Called synchronously within the same TanStackDB transaction as the attachment write, + * so any mutations made to other collections are committed atomically with it. + * + * Must not be async. `Transaction.mutate` unregisters the ambient transaction as soon as + * this callback returns, so any mutation made after an `await` inside the hook escapes the + * transaction and is not committed atomically with the attachment. Do asynchronous work + * before calling `save` or `delete`. + */ + updateHook?: (attachment: AttachmentQueueRow) => void +} + +export type AttachmentQueueRow = OptionalExtractedTable + +/** + * A custom extension of the PowerSyncAttachmentQueue for TanStackDB. + */ +export class TanStackDBAttachmentQueue extends AttachmentQueue { + readonly powersync: AbstractPowerSyncDatabase + readonly collection: Collection + + constructor(params: TanStackDBAttachmentQueueOptions) { + super(params) + this.powersync = params.db + this.collection = params.attachmentsCollection + } + + /** + * Saves a file to local storage and queues it for upload to remote storage. + * + * Exposes an `updateHook` option which is called inside a TanStackDB transaction, + * relational associations with the provided attachment ID should be made in this hook. + */ + async save({ + data, + fileExtension, + mediaType, + metaData, + id, + updateHook, + }: SaveOptions): Promise { + const resolvedId = id ?? (await this.generateAttachmentId()) + const filename = `${resolvedId}.${fileExtension}` + const localUri = this.localStorage.getLocalUri(filename) + let pending = savingIds.get(this.powersync) + if (!pending) savingIds.set(this.powersync, (pending = new Set())) + if (pending.has(resolvedId)) { + throw new Error(`Attachment with id ${resolvedId} is already being saved`) + } + pending.add(resolvedId) + + try { + return await this.withLoadedAttachment(resolvedId, () => + this.withAttachmentContext(async (ctx) => { + // A missing in-memory row is not proof that SQLite has no attachment. + if ( + this.collection.get(resolvedId) || + (await ctx.db.getOptional( + sanitizeSQL`SELECT id FROM ${ctx.tableName} WHERE id = ?`, + [resolvedId], + )) + ) { + throw new Error(`Attachment with id ${resolvedId} already exists`) + } + + try { + const size = await this.localStorage.saveFile(localUri, data) + const attachment: AttachmentQueueRow = { + id: resolvedId, + filename, + media_type: mediaType ?? null, + local_uri: localUri, + state: AttachmentState.QUEUED_UPLOAD, + has_synced: 0, + size, + timestamp: new Date().getTime(), + meta_data: metaData ?? null, + } + + const tanStackDBTransaction = createTransaction({ + autoCommit: false, + mutationFn: async ({ transaction }) => { + await new PowerSyncTransactor({ + database: ctx.db, + }).applyTransaction(transaction) + }, + }) + + await this.runInTransaction(tanStackDBTransaction, () => { + this.collection.insert(attachment) + // allow the user to associate values in this transaction + updateHook?.(attachment) + }) + return attachment + } catch (error) { + /** + * The file is written before the transaction opens, so a failed transaction would + * otherwise leave an orphaned file behind that no attachment record points to. + */ + await this.deleteLocalFile(localUri) + throw error + } + }), + ) + } finally { + pending.delete(resolvedId) + } + } + + /** + * Queues a file for deletion from local and remote storage. + * + * Exposes an `updateHook` option which is called inside a TanStackDB transaction, + * relational associations with the provided attachment ID should be cleaned up in this hook. + */ + async delete({ id, updateHook }: DeleteOptions): Promise { + await this.withLoadedAttachment(id, () => + this.withAttachmentContext(async (ctx) => { + const tanStackDBTransaction = createTransaction({ + autoCommit: false, + mutationFn: async ({ transaction }) => { + await new PowerSyncTransactor({ + database: ctx.db, + }).applyTransaction(transaction) + }, + }) + + await this.runInTransaction(tanStackDBTransaction, () => { + const attachment = this.collection.get(id) + if (!attachment) { + throw new Error(`Attachment with id ${id} not found`) + } + + this.collection.update(id, (draft) => { + draft.state = AttachmentState.QUEUED_DELETE + draft.has_synced = 0 + }) + + // allow the user to associate values in this transaction + updateHook?.(attachment) + }) + }), + ) + } + + private async withLoadedAttachment( + id: string, + operation: () => Promise, + ): Promise { + const query = createLiveQueryCollection({ + query: (q) => + q + .from({ attachment: this.collection }) + .where(({ attachment }) => eq(attachment.id, id)), + }) + try { + // Acquire just this ID before opening a mutation and retain its demand + // until PowerSync has confirmed the transaction back to the collection. + await query.preload() + return await operation() + } finally { + await query.cleanup() + } + } + + /** + * Applies `mutations` to `transaction` and commits it, rolling back on any failure. + * + * `Transaction.mutate` does not roll back when its callback throws, so a throwing + * `updateHook` would otherwise leave the transaction pending with its optimistic + * mutations still applied to the collections. + */ + protected async runInTransaction( + transaction: Transaction, + mutations: () => void, + ): Promise { + /** + * `rollback` rejects this promise. The error is already surfaced to the caller by the + * throw below, so this catch only stops it from becoming an unhandled rejection. + */ + void transaction.isPersisted.promise.catch(() => {}) + + try { + transaction.mutate(mutations) + } catch (error) { + transaction.rollback() + throw error + } + + await transaction.commit() + } + + /** + * Best-effort removal of a local file. A cleanup failure is logged rather than thrown, + * so that it can never mask the error which triggered the cleanup. + */ + protected async deleteLocalFile(localUri: string): Promise { + try { + await this.localStorage.deleteFile(localUri) + } catch (error) { + this.logger.error( + `Could not clean up local attachment file ${localUri}`, + error, + ) + } + } +} diff --git a/packages/powersync-db-collection/src/index.ts b/packages/powersync-db-collection/src/index.ts index f8d0928056..f96a7a0ee4 100644 --- a/packages/powersync-db-collection/src/index.ts +++ b/packages/powersync-db-collection/src/index.ts @@ -1,3 +1,4 @@ +export * from './attachments' export * from './definitions' export * from './powersync' export * from './PowerSyncTransactor' diff --git a/packages/powersync-db-collection/src/powersync.ts b/packages/powersync-db-collection/src/powersync.ts index c11c1d2698..786bb817c8 100644 --- a/packages/powersync-db-collection/src/powersync.ts +++ b/packages/powersync-db-collection/src/powersync.ts @@ -1,5 +1,5 @@ import { DiffTriggerOperation, sanitizeSQL } from '@powersync/common' -import { or } from '@tanstack/db' +import { or, withCollectionConfigFactory } from '@tanstack/db' import { compileSQLite } from './sqlite-compiler' import { PendingOperationStore } from './PendingOperationStore' import { PowerSyncTransactor } from './PowerSyncTransactor' @@ -11,6 +11,7 @@ import type { CleanupFn, LoadSubsetOptions, OperationType, + SyncAppliedReceipt, SyncConfig, } from '@tanstack/db' import type { @@ -226,6 +227,18 @@ export function powerSyncCollectionOptions< export function powerSyncCollectionOptions< TTable extends Table, TSchema extends StandardSchemaV1 = never, +>( + config: PowerSyncCollectionConfig, +): ReturnType> { + const outputConfig = createPowerSyncCollectionConfig(config) + return withCollectionConfigFactory(outputConfig, () => + createPowerSyncCollectionConfig(config), + ) +} + +function createPowerSyncCollectionConfig< + TTable extends Table, + TSchema extends StandardSchemaV1 = never, >(config: PowerSyncCollectionConfig) { const { database, @@ -304,12 +317,13 @@ export function powerSyncCollectionOptions< */ const sync: SyncConfig = { sync: (params) => { - const { begin, write, collection, commit, markReady } = params + const { begin, write, collection, commit, markReady, markError } = params const abortController = new AbortController() let disposeTracking: | ((options?: { context?: LockContext }) => Promise) | null = null + let trackingSetup: Promise | null = null if (syncMode === `eager`) { return runEagerSync() @@ -317,18 +331,63 @@ export function powerSyncCollectionOptions< return runOnDemandSync() } - async function createDiffTrigger(options: { - setupContext?: LockContext - when: Record - writeType: (rowId: string) => OperationType - batchQuery: ( - lockContext: LockContext, - batchSize: number, - cursor: number, - ) => Promise> - onReady: () => void - }) { - const { setupContext, when, writeType, batchQuery, onReady } = options + /** + * Disposes the current diff trigger, if one is active, and clears the + * tracking state. + */ + async function safelyDisposeTracking( + context?: LockContext, + ): Promise { + // Cleanup can race trigger creation. Wait until the disposer has been + // published so an abort cannot strand a freshly-created trigger. + const setup = trackingSetup + if (setup) { + await setup.catch(() => undefined) + } + + const dispose = disposeTracking + if (!dispose) { + return + } + + disposeTracking = null + await dispose(context ? { context } : undefined) + } + + async function establishTracking( + options: Parameters[0], + appliedReceipts: Array, + ): Promise { + const setup = (async () => { + const dispose = await createDiffTrigger(options, appliedReceipts) + disposeTracking = dispose + })() + trackingSetup = setup + + try { + await setup + } finally { + if (trackingSetup === setup) { + trackingSetup = null + } + } + } + + async function createDiffTrigger( + options: { + setupContext?: LockContext + immediate?: boolean + when: Record + writeType: (rowId: string) => OperationType + batchQuery: ( + lockContext: LockContext, + batchSize: number, + cursor: number, + ) => Promise> + }, + appliedReceipts: Array, + ) { + const { setupContext, immediate, when, writeType, batchQuery } = options return await database.triggers.createDiffTrigger({ source: viewName, @@ -340,7 +399,7 @@ export function powerSyncCollectionOptions< let currentBatchCount = syncBatchSize let cursor = 0 while (currentBatchCount == syncBatchSize) { - begin() + begin(immediate ? { immediate: true } : undefined) const batchItems = await batchQuery( context, @@ -355,9 +414,8 @@ export function powerSyncCollectionOptions< value: deserializeSyncRow(row), }) } - commit() + appliedReceipts.push(commit()) } - onReady() database.logger.info( `Sync is ready for ${viewName} into ${trackedTableName}`, ) @@ -367,9 +425,21 @@ export function powerSyncCollectionOptions< } async function flushDiffRecords(): Promise { + // PowerSync can notify after creating the tracking table but before its + // create call returns. Preserve that notification until the disposer, + // which proves the trigger is usable, has been published. + const setup = trackingSetup + if (setup) { + await setup.catch(() => undefined) + } + if (!disposeTracking) { + return + } + + const ignoredReceipts: Array = [] await database .writeTransaction(async (context) => { - await flushDiffRecordsWithContext(context) + await flushDiffRecordsWithContext(context, ignoredReceipts) }) .catch((error) => { database.logger.error( @@ -382,7 +452,13 @@ export function powerSyncCollectionOptions< // We can use this directly if we want to pair a flush with dispose+recreate diff trigger. async function flushDiffRecordsWithContext( context: LockContext, + appliedReceipts: Array, ): Promise { + // There is nothing to flush if no tracking table is currently active. + if (!disposeTracking) { + return + } + try { begin() const operations = await context.getAll( @@ -419,7 +495,12 @@ export function powerSyncCollectionOptions< // clear the current operations await context.execute(`DELETE FROM ${trackedTableName}`) - commit() + const applied = commit() + appliedReceipts.push(applied) + // Mutation persistence is what releases the Collection's FIFO gate. + // Confirm these local operations after the sync transaction is + // staged; waiting for its applied receipt would deadlock the user + // transaction that currently parks it. pendingOperationStore.resolvePendingFor(pendingOperations) } catch (error) { database.logger.error( @@ -451,12 +532,12 @@ export function powerSyncCollectionOptions< // If the abort controller was aborted while processing the request above if (abortController.signal.aborted) { - await disposeTracking?.() + await safelyDisposeTracking() } else { abortController.signal.addEventListener( `abort`, async () => { - await disposeTracking?.() + await safelyDisposeTracking() }, { once: true }, ) @@ -469,32 +550,48 @@ export function powerSyncCollectionOptions< let onUnload: CleanupFn | void | null = null start(async () => { - onUnload = await restConfig.onLoad?.() - - disposeTracking = await createDiffTrigger({ - when: { - [DiffTriggerOperation.INSERT]: `TRUE`, - [DiffTriggerOperation.UPDATE]: `TRUE`, - [DiffTriggerOperation.DELETE]: `TRUE`, + const cleanup = await restConfig.onLoad?.() + if (abortController.signal.aborted) { + cleanup?.() + return + } + onUnload = cleanup + + const appliedReceipts: Array = [] + await establishTracking( + { + // Initial eager hydration must make the source usable before + // PowerSync can persist a mutation queued during startup. + immediate: true, + when: { + [DiffTriggerOperation.INSERT]: `TRUE`, + [DiffTriggerOperation.UPDATE]: `TRUE`, + [DiffTriggerOperation.DELETE]: `TRUE`, + }, + writeType: (_rowId: string) => `insert`, + batchQuery: ( + lockContext: LockContext, + batchSize: number, + cursor: number, + ) => + lockContext.getAll( + sanitizeSQL`SELECT * FROM ${viewName} LIMIT ? OFFSET ?`, + [batchSize, cursor], + ), }, - writeType: (_rowId: string) => `insert`, - batchQuery: ( - lockContext: LockContext, - batchSize: number, - cursor: number, - ) => - lockContext.getAll( - sanitizeSQL`SELECT * FROM ${viewName} LIMIT ? OFFSET ?`, - [batchSize, cursor], - ), - onReady: () => markReady(), - }) - }).catch((error) => + appliedReceipts, + ) + await Promise.all(appliedReceipts) + markReady() + }).catch((error) => { database.logger.error( `Could not start syncing process for ${viewName} into ${trackedTableName}`, error, - ), - ) + ) + if (collection.status === `loading`) { + markError(error) + } + }) return () => { database.logger.info( @@ -508,85 +605,181 @@ export function powerSyncCollectionOptions< // On-demand mode. // Registers a diff trigger for the active WHERE expressions. function runOnDemandSync() { - let onUnloadSubset: CleanupFn | void | null = null + type DemandRecord = { + options: LoadSubsetOptions + active: boolean + cleanup?: CleanupFn + } + type PendingRelease = { + options: LoadSubsetOptions + failures: number + } - start().catch((error) => + const demands = new Map() + const releasedSubsets = new WeakSet() + const pendingReleases: Array = [] + let stopped = false + let trackingRevision = 0 + let reconciledTrackingRevision = 0 + let rebuildPromise: Promise | null = null + let drainingReleases = false + let releaseRetryTimer: ReturnType | undefined + const startup = start() + void startup.catch((error) => database.logger.error( `Could not start syncing process for ${viewName} into ${trackedTableName}`, error, ), ) - // Tracks all active WHERE expressions for on-demand sync filtering. - // Each loadSubset call pushes its predicate; unloadSubset removes it. - const activeWhereExpressions: Array = [] + const activeWhereExpressions = () => + Array.from(demands.values()) + .filter((demand) => demand.active) + .map((demand) => demand.options.where) - const loadSubset = async ( - options?: LoadSubsetOptions, - ): Promise => { - if (options) { - activeWhereExpressions.push(options.where) - onUnloadSubset = await restConfig.onLoadSubset?.(options) - } + // One reconciliation owns every queued revision so callers cannot + // settle against a stale trigger configuration. + const reconcileTracking = async (): Promise => { + while (!stopped && reconciledTrackingRevision !== trackingRevision) { + const revision = trackingRevision + const isCurrent = () => !stopped && trackingRevision === revision + const appliedReceipts: Array = [] - if (activeWhereExpressions.length === 0) { await database.writeLock(async (ctx) => { - await flushDiffRecordsWithContext(ctx) - await disposeTracking?.({ context: ctx }) + if (!isCurrent()) return + await flushDiffRecordsWithContext(ctx, appliedReceipts) + if (!isCurrent()) return + await safelyDisposeTracking(ctx) + if (!isCurrent()) return + + const active = activeWhereExpressions() + // Tracking was absent during an error. Positive baseline rows + // alone cannot reveal deletes or predicate exits from that gap. + const missing = + collection.status === `error` + ? new Set(collection.keys()) + : undefined + if (active.length > 0) { + const combinedWhere = + active.length === 1 + ? active[0] + : or(active[0], active[1], ...active.slice(2)) + const compiledNewData = compileSQLite( + { where: combinedWhere }, + { jsonColumn: 'NEW.data' }, + ) + const compiledOldData = compileSQLite( + { where: combinedWhere }, + { jsonColumn: 'OLD.data' }, + ) + const compiledView = compileSQLite({ where: combinedWhere }) + const newDataWhenClause = toInlinedWhereClause(compiledNewData) + const oldDataWhenClause = toInlinedWhereClause(compiledOldData) + const viewWhereClause = toInlinedWhereClause(compiledView) + await establishTracking( + { + setupContext: ctx, + when: { + [DiffTriggerOperation.INSERT]: newDataWhenClause, + [DiffTriggerOperation.UPDATE]: `(${newDataWhenClause}) OR (${oldDataWhenClause})`, + [DiffTriggerOperation.DELETE]: oldDataWhenClause, + }, + writeType: (rowId: string) => + collection.has(rowId) ? `update` : `insert`, + batchQuery: ( + lockContext: LockContext, + batchSize: number, + cursor: number, + ) => + lockContext + .getAll( + `SELECT * FROM ${viewName} WHERE ${viewWhereClause} LIMIT ? OFFSET ?`, + [batchSize, cursor], + ) + .then((rows) => { + for (const row of rows) missing?.delete(row.id) + return rows + }), + }, + appliedReceipts, + ) + } + if (!isCurrent()) await safelyDisposeTracking(ctx) + else if (missing?.size) { + begin() + for (const key of missing) write({ type: `delete`, key }) + appliedReceipts.push(commit()) + } }) - return + await Promise.all(appliedReceipts) + if (isCurrent()) { + reconciledTrackingRevision = revision + // Replacing the trigger alone is not recovery: its baseline + // writes must also be applied before the source is ready again. + if (collection.status === `error`) markReady() + } } + } - const combinedWhere = - activeWhereExpressions.length === 1 - ? activeWhereExpressions[0] - : or( - activeWhereExpressions[0], - activeWhereExpressions[1], - ...activeWhereExpressions.slice(2), - ) - - const compiledNewData = compileSQLite( - { where: combinedWhere }, - { jsonColumn: 'NEW.data' }, - ) - - const compiledOldData = compileSQLite( - { where: combinedWhere }, - { jsonColumn: 'OLD.data' }, - ) + const rebuildTracking = async (): Promise => { + // New demand can join after reconciliation exits but before its + // shared promise clears. Each waiter must check its revision again. + while (!stopped && reconciledTrackingRevision !== trackingRevision) { + await (rebuildPromise ??= reconcileTracking() + .catch((error) => { + // A rebuild may already have removed every active diff trigger. + // Do not leave healthy consumers ready against a stale source. + if (!stopped) markError(error) + throw error + }) + .finally(() => { + rebuildPromise = null + })) + } + } - const compiledView = compileSQLite({ where: combinedWhere }) + const loadSubset = async ( + options: LoadSubsetOptions, + ): Promise => { + if (stopped) return + // Never create a trigger that has no observer to drain its diff table. + await startup + if ( + // Cleanup can run while startup is pending. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + stopped || + releasedSubsets.has(options) || + options.signal?.aborted + ) { + return + } - const newDataWhenClause = toInlinedWhereClause(compiledNewData) - const oldDataWhenClause = toInlinedWhereClause(compiledOldData) - const viewWhereClause = toInlinedWhereClause(compiledView) + const demand: DemandRecord = { options, active: false } + demands.set(options, demand) + try { + const cleanup = await restConfig.onLoadSubset?.(options) + if (cleanup) demand.cleanup = cleanup + } catch (error) { + demands.delete(options) + throw error + } - await database.writeLock(async (ctx) => { - await flushDiffRecordsWithContext(ctx) - await disposeTracking?.({ context: ctx }) + if ( + // The user hook can reenter cleanup. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + stopped || + releasedSubsets.has(options) || + options.signal?.aborted || + demands.get(options) !== demand + ) { + demands.delete(options) + demand.cleanup?.() + return + } - disposeTracking = await createDiffTrigger({ - setupContext: ctx, - when: { - [DiffTriggerOperation.INSERT]: newDataWhenClause, - [DiffTriggerOperation.UPDATE]: `(${newDataWhenClause}) OR (${oldDataWhenClause})`, - [DiffTriggerOperation.DELETE]: oldDataWhenClause, - }, - writeType: (rowId: string) => - collection.has(rowId) ? `update` : `insert`, - batchQuery: ( - lockContext: LockContext, - batchSize: number, - cursor: number, - ) => - lockContext.getAll( - `SELECT * FROM ${viewName} WHERE ${viewWhereClause} LIMIT ? OFFSET ?`, - [batchSize, cursor], - ), - onReady: () => {}, - }) - }) + demand.active = true + trackingRevision++ + await rebuildTracking() } const toInlinedWhereClause = (compiled: { @@ -601,63 +794,138 @@ export function powerSyncCollectionOptions< ) } - const unloadSubset = async (options: LoadSubsetOptions) => { - onUnloadSubset?.() - - const idx = activeWhereExpressions.indexOf(options.where) - if (idx !== -1) { - activeWhereExpressions.splice(idx, 1) + const cleanupDemand = (demand: DemandRecord): void => { + demands.delete(demand.options) + try { + demand.cleanup?.() + } catch (error) { + database.logger.error( + `Could not clean up subset hook for ${viewName}`, + error, + ) } + } - // Evict rows that were exclusively loaded by the departing predicate. - // These are rows matching the departing WHERE that are no longer covered - // by any remaining active predicate. + const performPhysicalRelease = async ( + options: LoadSubsetOptions, + ): Promise => { const compiledDeparting = compileSQLite({ where: options.where }) const departingWhereSQL = toInlinedWhereClause(compiledDeparting) + let rowsToEvict: Array<{ id: string }> + for (;;) { + if (stopped) return + const revision = trackingRevision + const active = activeWhereExpressions() + let evictionSQL: string + if (active.length === 0) { + evictionSQL = `SELECT id FROM ${viewName} WHERE ${departingWhereSQL}` + } else { + const combinedRemaining = + active.length === 1 + ? active[0]! + : or(active[0], active[1], ...active.slice(2)) + const compiledRemaining = compileSQLite({ + where: combinedRemaining, + }) + const remainingWhereSQL = toInlinedWhereClause(compiledRemaining) + evictionSQL = `SELECT id FROM ${viewName} WHERE (${departingWhereSQL}) AND NOT (${remainingWhereSQL})` + } - let evictionSQL: string - if (activeWhereExpressions.length === 0) { - evictionSQL = `SELECT id FROM ${viewName} WHERE ${departingWhereSQL}` - } else { - const combinedRemaining = - activeWhereExpressions.length === 1 - ? activeWhereExpressions[0]! - : or( - activeWhereExpressions[0], - activeWhereExpressions[1], - ...activeWhereExpressions.slice(2), - ) - const compiledRemaining = compileSQLite({ - where: combinedRemaining, - }) - const remainingWhereSQL = toInlinedWhereClause(compiledRemaining) - evictionSQL = `SELECT id FROM ${viewName} WHERE (${departingWhereSQL}) AND NOT (${remainingWhereSQL})` + rowsToEvict = await database.getAll<{ id: string }>(evictionSQL) + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition -- cleanup can run during the query + if (stopped) return + if (trackingRevision === revision) break } - - const rowsToEvict = await database.getAll<{ id: string }>(evictionSQL) if (rowsToEvict.length > 0) { begin() for (const { id } of rowsToEvict) { write({ type: `delete`, key: id }) } - commit() + void commit() } + await rebuildTracking() + } - // Recreate the diff trigger for the remaining active WHERE expressions. - await loadSubset() + function scheduleReleaseDrain(delay = 0): void { + if (stopped || drainingReleases || releaseRetryTimer) return + if (delay > 0) { + releaseRetryTimer = setTimeout(() => { + releaseRetryTimer = undefined + void drainReleases() + }, delay) + return + } + void drainReleases() + } + + async function drainReleases(): Promise { + if (stopped || drainingReleases) return + drainingReleases = true + let retryDelay = 0 + try { + const attempts = pendingReleases.length + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition -- each release can reenter cleanup + for (let index = 0; !stopped && index < attempts; index++) { + const pending = pendingReleases.shift()! + try { + await performPhysicalRelease(pending.options) + } catch (error) { + pending.failures++ + pendingReleases.push(pending) + const delay = Math.min( + 1000 * 2 ** (pending.failures - 1), + 30000, + ) + retryDelay = + retryDelay === 0 ? delay : Math.min(retryDelay, delay) + database.logger.error( + `Could not release subset tracking for ${viewName}; retrying`, + error, + ) + } + } + } finally { + drainingReleases = false + } + if (pendingReleases.length > 0) scheduleReleaseDrain(retryDelay) + } + + const unloadSubset = (options: LoadSubsetOptions): void => { + releasedSubsets.add(options) + const demand = demands.get(options) + if (!demand) return + + const wasActive = demand.active + if (wasActive) trackingRevision++ + cleanupDemand(demand) + + if (wasActive) { + pendingReleases.push({ options, failures: 0 }) + // New work must not wait for another release's backoff. + clearTimeout(releaseRetryTimer) + releaseRetryTimer = undefined + scheduleReleaseDrain() + } } markReady() return { cleanup: () => { + stopped = true + clearTimeout(releaseRetryTimer) + releaseRetryTimer = undefined database.logger.info( `Sync has been stopped for ${viewName} into ${trackedTableName}`, ) abortController.abort() + for (const demand of demands.values()) { + cleanupDemand(demand) + } + pendingReleases.length = 0 }, loadSubset: (options: LoadSubsetOptions) => loadSubset(options), - unloadSubset: (options: LoadSubsetOptions) => unloadSubset(options), + unloadSubset, } } }, diff --git a/packages/powersync-db-collection/tests/ATTACHMENT-ORACLE.md b/packages/powersync-db-collection/tests/ATTACHMENT-ORACLE.md new file mode 100644 index 0000000000..bf5a2e9c43 --- /dev/null +++ b/packages/powersync-db-collection/tests/ATTACHMENT-ORACLE.md @@ -0,0 +1,37 @@ +# Attachment lifecycle oracle + +The model records user intent (absent, present, deleted) and accepted bytes. It +does not mirror the SDK's attachment-state machine. Real SQLite transactions, +TanStack collection delivery, local files, and SDK completion writes run under +the tests. Only remote upload completion and failure are controlled. + +Each command checks owner references, SQL/collection convergence, accepted file +bytes, and cleanup of every file destination touched by save. Draining checks +remote cleanup and an idle extra tick. Rejected hooks and duplicate saves must +not change the accepted reference or bytes. + +The ordinary suite includes a committed corpus, a fixed fast-check campaign, +and a fresh random campaign. To replay a generated failure, run the matching +test with `POWERSYNC_ATTACHMENT_ORACLE_SEED` and, if supplied by fast-check, +`POWERSYNC_ATTACHMENT_ORACLE_PATH`. + +## Upstream completion limitation + +The SDK currently writes the captured upload record after remote I/O without +preserving a newer `QUEUED_DELETE`. Successful upload can lose the remote delete; +failed upload can restore an obsolete upload for retry. The ordinary generated +suite excludes only deletion while upload is in flight. This is a known gap in +supported behavior, not proof that every lifecycle is correct. + +The same model and fixture retain both desired-contract histories and a full +generated schedule in `attachments-sdk-completion.repro.ts`. Run them explicitly: + +```sh +pnpm --filter @tanstack/powersync-db-collection test:upstream-repros +``` + +These are real failing assertions, not expected failures or skipped assertions. +They are separate from the normal gate because the SDK fix is upstream. Once +completion preserves newer intent, move these histories into the normal corpus +and enable in-flight deletion in its generator. Do not weaken the oracle to +accept a detached owner with leaked remote bytes or a resumed obsolete upload. diff --git a/packages/powersync-db-collection/tests/attachments-lifecycle-fixture.ts b/packages/powersync-db-collection/tests/attachments-lifecycle-fixture.ts new file mode 100644 index 0000000000..8653e2d70c --- /dev/null +++ b/packages/powersync-db-collection/tests/attachments-lifecycle-fixture.ts @@ -0,0 +1,467 @@ +import { randomUUID } from 'node:crypto' +import { mkdtemp, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fc } from '@fast-check/vitest' +import { AttachmentTable, Schema, Table, column } from '@powersync/common' +import { NodeFileSystemAdapter, PowerSyncDatabase } from '@powersync/node' +import { createCollection } from '@tanstack/db' +import pDefer from 'p-defer' +import { expect, vi } from 'vitest' +import { powerSyncCollectionOptions } from '../src' +import { TanStackDBAttachmentQueue } from '../src/attachments' +import { TEST_DATABASE_IMPLEMENTATION } from './test-db-implementation' +import type { RemoteStorageAdapter } from '@powersync/common' + +const schema = new Schema({ + owners: new Table({ photo_id: column.text }), + attachments: new AttachmentTable(), +}) +const attachmentId = `photo` +const ownerId = `owner` + +type Command = + | `save` + | `reject-save` + | `duplicate-warm` + | `duplicate-cold` + | `delete` + | `reject-delete` + | `start-upload` + | `finish-upload` + | `fail-upload` + | `drain` + +export interface History { + name: string + commands: Array + bytes: Array +} + +// This is a user-intent model, not an AttachmentState transition table. An ID +// owns bytes only after save commits. Rejected operations leave intent intact. +// Transport completion cannot alter the most recent committed reference. +interface Model { + intent: `absent` | `present` | `deleted` + bytes: Array +} + +interface AttachmentRow { + id: string + local_uri: string | null + size: number | null + state: number + has_synced: number +} + +async function setupOracle() { + const directory = await mkdtemp(join(tmpdir(), `ps-attachment-oracle-`)) + const db = new PowerSyncDatabase({ + database: { + dbFilename: `${randomUUID()}.sqlite`, + dbLocation: directory, + implementation: TEST_DATABASE_IMPLEMENTATION, + }, + schema, + }) + await db.disconnectAndClear() + const local = new NodeFileSystemAdapter(join(directory, `files`)) + await local.initialize() + const attachments = createCollection( + powerSyncCollectionOptions({ + database: db, + table: schema.props.attachments, + }), + ) + const owners = createCollection( + powerSyncCollectionOptions({ database: db, table: schema.props.owners }), + ) + await Promise.all([attachments.stateWhenReady(), owners.stateWhenReady()]) + + // Only the remote transport is controlled. Local bytes, transactions, SQL, + // collection delivery, and SDK completion writes all use the real adapters. + const remote = new Map>() + const io = { uploads: 0, deletes: 0, downloads: 0 } + let uploadGate: + | { + entered: ReturnType> + outcome: ReturnType> + } + | undefined + let syncing: Promise | undefined + const transport: RemoteStorageAdapter = { + async uploadFile(data, attachment) { + io.uploads++ + const captured = Array.from(new Uint8Array(data)) + const gate = uploadGate + if (gate) { + gate.entered.resolve() + if (!(await gate.outcome.promise)) + throw new Error(`injected upload failure`) + } + remote.set(attachment.id, captured) + }, + deleteFile(attachment) { + io.deletes++ + remote.delete(attachment.id) + return Promise.resolve() + }, + downloadFile() { + io.downloads++ + throw new Error(`download is outside this save/delete history`) + }, + } + const queue = new TanStackDBAttachmentQueue({ + db, + attachmentsCollection: attachments, + localStorage: local, + remoteStorage: transport, + // These histories exercise explicit save/delete intent. Watcher lifecycle + // has its own boundary tests; no periodic timer can race our schedule. + watchAttachments: () => {}, + archivedCacheLimit: 0, + }) + // Track actual destinations, including writes whose transaction later fails. + // A fixed ID-derived filename would miss orphaned call-owned files. + const writtenFiles = new Set() + const saveFile = local.saveFile.bind(local) + vi.spyOn(local, `saveFile`).mockImplementation((path, data) => { + writtenFiles.add(path) + return saveFile(path, data) + }) + let uri: string | undefined + + async function finishUpload(succeeds: boolean) { + if (!uploadGate || !syncing) throw new Error(`no upload in flight`) + uploadGate.outcome.resolve(succeeds) + await syncing + syncing = undefined + uploadGate = undefined + } + + async function assertObserved(model: Model, label: string, drained: boolean) { + const rows = await db.getAll( + `SELECT id, local_uri, size, state, has_synced FROM attachments ORDER BY id`, + ) + const references = await db.getAll<{ id: string; photo_id: string | null }>( + `SELECT id, photo_id FROM owners ORDER BY id`, + ) + // SQL is authoritative; collection convergence is an additional assertion, + // never the source of the oracle's expected reference or expected bytes. + await vi.waitFor( + () => { + expect( + attachments.toArray.map( + ({ id, local_uri, size, state, has_synced }) => ({ + id, + local_uri, + size, + state, + has_synced, + }), + ), + label, + ).toEqual(rows) + expect( + owners.toArray.map(({ id, photo_id }) => ({ id, photo_id })), + label, + ).toEqual(references) + }, + { timeout: 3000, interval: 10 }, + ) + const exists = uri !== undefined && (await local.fileExists(uri)) + for (const path of writtenFiles) { + if ( + path !== uri || + model.intent === `absent` || + (drained && model.intent === `deleted`) + ) { + expect( + await local.fileExists(path), + `${label}: unexpected file ${path}`, + ).toBe(false) + } + } + const observed = { + references, + localBytes: exists + ? Array.from(new Uint8Array(await local.readFile(uri!))) + : null, + remoteBytes: remote.get(attachmentId) ?? null, + rows, + } + const evidence = `${label}; I/O=${JSON.stringify(io)}` + expect(references, label).toEqual( + model.intent === `absent` + ? [] + : [ + { + id: ownerId, + photo_id: model.intent === `present` ? attachmentId : null, + }, + ], + ) + // This is a logical FK: PowerSync's synced table views do not enforce a + // SQLite FOREIGN KEY. An attached owner must still resolve to its own row. + if (model.intent === `present`) { + expect(observed, evidence).toMatchObject({ + localBytes: model.bytes, + rows: [{ id: attachmentId, local_uri: uri, size: model.bytes.length }], + }) + if (drained) expect(observed.remoteBytes, label).toEqual(model.bytes) + } else if (model.intent === `absent` || drained) { + expect(observed, evidence).toMatchObject({ + localBytes: null, + remoteBytes: null, + rows: [], + }) + } + // If remote bytes exist at an intermediate boundary, they must come from + // the accepted save, never the rejected duplicate (same size, other bytes). + if (observed.remoteBytes) + expect(observed.remoteBytes, label).toEqual(model.bytes) + } + + async function run(command: Command, model: Model) { + switch (command) { + case `save`: + case `reject-save`: { + const saving = queue.save({ + id: attachmentId, + fileExtension: `bin`, + data: new Uint8Array(model.bytes).buffer, + updateHook: (attachment) => { + owners.insert({ id: ownerId, photo_id: attachment.id }) + if (command === `reject-save`) + throw new Error(`injected hook failure`) + }, + }) + if (command === `reject-save`) { + await expect(saving).rejects.toThrow(`injected hook failure`) + } else { + const saved = await saving + uri = saved.local_uri ?? undefined + model.intent = `present` + } + break + } + case `duplicate-warm`: + case `duplicate-cold`: { + const collection = + command === `duplicate-cold` + ? createCollection( + powerSyncCollectionOptions({ + database: db, + table: schema.props.attachments, + }), + ) + : attachments + const duplicateQueue = new TanStackDBAttachmentQueue({ + db, + attachmentsCollection: collection, + localStorage: local, + remoteStorage: transport, + watchAttachments: () => {}, + }) + try { + if (command === `duplicate-cold`) + expect(collection.get(attachmentId)).toBeUndefined() + await expect( + duplicateQueue.save({ + id: attachmentId, + fileExtension: `bin`, + data: new Uint8Array(model.bytes.map((byte) => byte ^ 255)) + .buffer, + updateHook: () => + owners.update(ownerId, (owner) => { + owner.photo_id = null + }), + }), + ).rejects.toThrow() + } finally { + await duplicateQueue.stopSync() + if (collection !== attachments) await collection.cleanup() + } + break + } + case `delete`: + case `reject-delete`: { + const deleting = queue.delete({ + id: attachmentId, + updateHook: () => { + owners.update(ownerId, (owner) => { + owner.photo_id = null + }) + if (command === `reject-delete`) + throw new Error(`injected hook failure`) + }, + }) + if (command === `reject-delete`) { + await expect(deleting).rejects.toThrow(`injected hook failure`) + } else { + await deleting + model.intent = `deleted` + } + break + } + case `start-upload`: + if (syncing) throw new Error(`upload already in flight`) + uploadGate = { entered: pDefer(), outcome: pDefer() } + syncing = queue.syncStorage() + await vi.waitFor(() => expect(io.uploads).toBeGreaterThan(0), { + timeout: 3000, + interval: 10, + }) + await uploadGate.entered.promise + break + case `finish-upload`: + await finishUpload(true) + break + case `fail-upload`: + await finishUpload(false) + break + case `drain`: { + if (syncing) throw new Error(`finish the held upload before draining`) + // No more injected failures: two passes suffice for these single-ID + // histories. A third tick must be idle, not merely leave SQL detached. + await queue.syncStorage() + await queue.syncStorage() + const settledIO = { ...io } + await queue.syncStorage() + expect(io).toEqual(settledIO) + break + } + } + } + + return { + run, + assertObserved, + async dispose() { + if (uploadGate) uploadGate.outcome.resolve(true) + await syncing + await queue.stopSync() + await Promise.all([attachments.cleanup(), owners.cleanup()]) + await db.disconnectAndClear() + await db.close() + await local.clear() + await rm(directory, { recursive: true }) + }, + } +} + +export async function runHistory(history: History) { + const fixture = await setupOracle() + const model: Model = { intent: `absent`, bytes: [...history.bytes] } + try { + for (const [index, command] of history.commands.entries()) { + const label = `${history.name}: step ${index + 1}/${history.commands.length} ${command}; ${JSON.stringify(history)}` + await fixture.run(command, model) + await fixture.assertObserved(model, label, command === `drain`) + } + } finally { + await fixture.dispose() + } +} + +export const corpus: Array> = [ + { + name: `warm duplicate preserves bytes and reference`, + commands: [`save`, `duplicate-warm`, `drain`], + }, + { + name: `cold duplicate preserves bytes and reference`, + commands: [`save`, `duplicate-cold`, `drain`], + }, + { + name: `delete before upload removes local data`, + commands: [`save`, `delete`, `drain`], + }, + { + name: `delete during successful upload removes remote data`, + commands: [`save`, `start-upload`, `delete`, `finish-upload`, `drain`], + }, + { + name: `delete after upload removes remote data`, + commands: [`save`, `start-upload`, `finish-upload`, `delete`, `drain`], + }, + { + name: `delete during failed upload does not retry obsolete intent`, + commands: [`save`, `start-upload`, `delete`, `fail-upload`, `drain`], + }, + { + name: `upload failure retries accepted bytes`, + commands: [`save`, `start-upload`, `fail-upload`, `drain`], + }, + { + name: `delete after upload failure removes local data`, + commands: [`save`, `start-upload`, `fail-upload`, `delete`, `drain`], + }, + { + name: `save hook failure rolls back both rows and local data`, + commands: [`reject-save`, `save`, `drain`], + }, + { + name: `delete hook failure preserves bytes and reference`, + commands: [`save`, `reject-delete`, `drain`], + }, + { + name: `repeated delete stays deleted`, + commands: [`save`, `delete`, `delete`, `drain`], + }, +] + +// The ordinary suite excludes only the named SDK completion race; the opt-in +// repro suite runs that same law and harness without an expected-failure waiver. +export const sdkCompletionCorpus = corpus.filter( + ({ commands }) => + commands.indexOf(`delete`) > commands.indexOf(`start-upload`) && + commands.indexOf(`delete`) < + Math.max( + commands.indexOf(`finish-upload`), + commands.indexOf(`fail-upload`), + ), +) +export const supportedCorpus = corpus.filter( + (history) => !sdkCompletionCorpus.includes(history), +) + +export function historyArbitrary(includeInFlightDelete: boolean) { + return fc + .record({ + bytes: fc.array(fc.integer({ min: 0, max: 255 }), { + minLength: 1, + maxLength: 16, + }), + duplicate: fc.constantFrom(`none`, `warm`, `cold`), + deletion: includeInFlightDelete + ? fc.constantFrom(`never`, `before`, `during`, `after`) + : fc.constantFrom(`never`, `before`, `after`), + uploadSucceeds: fc.boolean(), + rejectDelete: fc.boolean(), + }) + .map( + ({ + bytes, + duplicate, + deletion, + uploadSucceeds, + rejectDelete, + }): History => { + const commands: Array = [`save`] + if (duplicate === `warm`) commands.push(`duplicate-warm`) + if (duplicate === `cold`) commands.push(`duplicate-cold`) + if (rejectDelete) commands.push(`reject-delete`) + if (deletion === `before`) { + commands.push(`delete`) + } else { + commands.push(`start-upload`) + if (deletion === `during`) commands.push(`delete`) + commands.push(uploadSucceeds ? `finish-upload` : `fail-upload`) + if (deletion === `after`) commands.push(`delete`) + } + commands.push(`drain`) + return { name: `generated`, commands, bytes } + }, + ) +} diff --git a/packages/powersync-db-collection/tests/attachments-lifecycle-oracle.test.ts b/packages/powersync-db-collection/tests/attachments-lifecycle-oracle.test.ts new file mode 100644 index 0000000000..2b5e50d448 --- /dev/null +++ b/packages/powersync-db-collection/tests/attachments-lifecycle-oracle.test.ts @@ -0,0 +1,37 @@ +import { fc } from '@fast-check/vitest' +import { describe, it } from 'vitest' +import { + historyArbitrary, + runHistory, + supportedCorpus, +} from './attachments-lifecycle-fixture' +import { TEST_DATABASE_IMPLEMENTATION } from './test-db-implementation' + +const describePowerSync = TEST_DATABASE_IMPLEMENTATION + ? describe + : describe.skip + +describePowerSync(`attachment lifecycle intent oracle`, () => { + it.each(supportedCorpus)(`$name`, async (history) => { + await runHistory({ ...history, bytes: [0, 7, 128, 255] }) + }) + + it.each([`fixed`, `random`] as const)( + `preserves intent across %s command histories`, + async (campaign) => { + const replaySeed = process.env.POWERSYNC_ATTACHMENT_ORACLE_SEED + await fc.assert(fc.asyncProperty(historyArbitrary(false), runHistory), { + ...(replaySeed + ? { seed: Number(replaySeed) } + : campaign === `fixed` + ? { seed: 1616 } + : {}), + numRuns: 12, + ...(process.env.POWERSYNC_ATTACHMENT_ORACLE_PATH + ? { path: process.env.POWERSYNC_ATTACHMENT_ORACLE_PATH } + : {}), + }) + }, + 30000, + ) +}) diff --git a/packages/powersync-db-collection/tests/attachments-native-sdk.repro.ts b/packages/powersync-db-collection/tests/attachments-native-sdk.repro.ts new file mode 100644 index 0000000000..81920c1b84 --- /dev/null +++ b/packages/powersync-db-collection/tests/attachments-native-sdk.repro.ts @@ -0,0 +1,85 @@ +import { mkdtemp, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { + AttachmentQueue, + AttachmentState, + AttachmentTable, + Schema, +} from '@powersync/common' +import { NodeFileSystemAdapter, PowerSyncDatabase } from '@powersync/node' +import pDefer from 'p-defer' +import { describe, expect, it } from 'vitest' +import { TEST_DATABASE_IMPLEMENTATION } from './test-db-implementation' + +const describePowerSync = TEST_DATABASE_IMPLEMENTATION + ? describe + : describe.skip + +// No TanStack collection, subclass, transaction, or watcher participates in this +// control. The native SDK alone loses deletion intent in its completion write. +describePowerSync(`native SDK completion contract`, () => { + it.each([true, false])( + `keeps deletion queued after upload success=%s`, + async (succeeds) => { + const directory = await mkdtemp(join(tmpdir(), `ps-native-completion-`)) + const db = new PowerSyncDatabase({ + database: { + dbFilename: `attachments.sqlite`, + dbLocation: directory, + implementation: TEST_DATABASE_IMPLEMENTATION, + }, + schema: new Schema({ attachments: new AttachmentTable() }), + }) + await db.disconnectAndClear() + const localStorage = new NodeFileSystemAdapter(join(directory, `files`)) + await localStorage.initialize() + const entered = pDefer() + const release = pDefer() + const queue = new AttachmentQueue({ + db, + localStorage, + remoteStorage: { + async uploadFile() { + entered.resolve() + await release.promise + if (!succeeds) throw new Error(`injected upload failure`) + }, + async deleteFile() {}, + downloadFile() { + throw new Error(`unexpected download`) + }, + }, + watchAttachments: () => {}, + }) + let syncing: Promise | undefined + try { + const record = await queue.saveFile({ + data: new Uint8Array([0, 7, 128, 255]).buffer, + fileExtension: `bin`, + }) + syncing = queue.syncStorage() + await entered.promise + await queue.deleteFile({ id: record.id }) + const readState = () => + db.get(`SELECT state FROM attachments WHERE id = ?`, [record.id]) + expect(await readState()).toEqual({ + state: AttachmentState.QUEUED_DELETE, + }) + release.resolve() + await syncing + expect(await readState()).toEqual({ + state: AttachmentState.QUEUED_DELETE, + }) + } finally { + release.resolve() + await syncing + await queue.stopSync() + await db.disconnectAndClear() + await db.close() + await localStorage.clear() + await rm(directory, { recursive: true }) + } + }, + ) +}) diff --git a/packages/powersync-db-collection/tests/attachments-sdk-completion.repro.ts b/packages/powersync-db-collection/tests/attachments-sdk-completion.repro.ts new file mode 100644 index 0000000000..0026ab9437 --- /dev/null +++ b/packages/powersync-db-collection/tests/attachments-sdk-completion.repro.ts @@ -0,0 +1,30 @@ +import { fc } from '@fast-check/vitest' +import { describe, it } from 'vitest' +import { + historyArbitrary, + runHistory, + sdkCompletionCorpus, +} from './attachments-lifecycle-fixture' +import { TEST_DATABASE_IMPLEMENTATION } from './test-db-implementation' + +const describePowerSync = TEST_DATABASE_IMPLEMENTATION + ? describe + : describe.skip + +// Desired contract, deliberately not test.fails: run separately until the SDK +// preserves newer deletion intent in both successful and failed upload writes. +describePowerSync(`upstream attachment completion contract`, () => { + it.each(sdkCompletionCorpus)(`$name`, async (history) => { + await runHistory({ ...history, bytes: [0, 7, 128, 255] }) + }) + + it(`preserves deletion across all generated completion schedules`, async () => { + await fc.assert(fc.asyncProperty(historyArbitrary(true), runHistory), { + seed: Number(process.env.POWERSYNC_ATTACHMENT_ORACLE_SEED ?? 1616), + numRuns: 12, + ...(process.env.POWERSYNC_ATTACHMENT_ORACLE_PATH + ? { path: process.env.POWERSYNC_ATTACHMENT_ORACLE_PATH } + : {}), + }) + }, 30000) +}) diff --git a/packages/powersync-db-collection/tests/attachments.test.ts b/packages/powersync-db-collection/tests/attachments.test.ts new file mode 100644 index 0000000000..1cafe169fd --- /dev/null +++ b/packages/powersync-db-collection/tests/attachments.test.ts @@ -0,0 +1,952 @@ +import { randomUUID } from 'node:crypto' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import pDefer from 'p-defer' +import { + AttachmentState, + AttachmentTable, + Schema, + Table, + column, +} from '@powersync/common' +import { NodeFileSystemAdapter, PowerSyncDatabase } from '@powersync/node' +import { + createCollection, + isNull, + liveQueryCollectionOptions, + not, +} from '@tanstack/db' +import { describe, expect, it, onTestFinished, vi } from 'vitest' +import { powerSyncCollectionOptions } from '../src' +import { TanStackDBAttachmentQueue } from '../src/attachments' +import { TEST_DATABASE_IMPLEMENTATION } from './test-db-implementation' +import type { + AttachmentErrorHandler, + RemoteStorageAdapter, + WatchedAttachmentItem, +} from '@powersync/common' +import type { AttachmentQueueRow } from '../src/attachments' + +// A minimal valid 1x1 pixel JPEG used as the remote payload for downloads. +const MOCK_JPEG_U8A = [ + 0xff, 0xd8, 0xff, 0xe0, 0x00, 0x10, 0x4a, 0x46, 0x49, 0x46, 0x00, 0x01, 0x01, + 0x01, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0xff, 0xd9, +] +const createMockJpegBuffer = (): ArrayBuffer => + new Uint8Array(MOCK_JPEG_U8A).buffer + +const SYNC_INTERVAL_MS = 300 +const WAIT_TIMEOUT = 8000 + +const APP_SCHEMA = new Schema({ + users: new Table({ + name: column.text, + email: column.text, + photo_id: column.text, + }), + attachments: new AttachmentTable(), +}) + +type WatchAttachments = ( + onUpdate: (attachments: Array) => Promise, + signal: AbortSignal, +) => void + +const describePowerSync = TEST_DATABASE_IMPLEMENTATION + ? describe + : describe.skip + +describePowerSync(`PowerSync AttachmentQueue (TanStackDB)`, () => { + async function setup(syncMode: `eager` | `on-demand` = `eager`) { + const db = new PowerSyncDatabase({ + database: { + dbFilename: `attachments-test-${randomUUID()}.sqlite`, + dbLocation: tmpdir(), + implementation: TEST_DATABASE_IMPLEMENTATION, + }, + schema: APP_SCHEMA, + }) + await db.disconnectAndClear() + + const localStorage = new NodeFileSystemAdapter( + join(tmpdir(), `ps-attachments-${randomUUID()}`), + ) + await localStorage.initialize() + + const uploadFile = vi.fn(() => + Promise.resolve(), + ) + const downloadFile = vi.fn(() => + Promise.resolve(createMockJpegBuffer()), + ) + const deleteFile = vi.fn(() => + Promise.resolve(), + ) + const remoteStorage: RemoteStorageAdapter = { + uploadFile, + downloadFile, + deleteFile, + } + + const attachmentsCollection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.attachments, + syncMode, + }), + ) + const usersCollection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.users, + }), + ) + await Promise.all([ + attachmentsCollection.stateWhenReady(), + usersCollection.stateWhenReady(), + ]) + + onTestFinished(async () => { + attachmentsCollection.cleanup() + usersCollection.cleanup() + await db.disconnectAndClear() + await db.close() + await localStorage.clear().catch(() => {}) + }) + + function createQueue( + overrides: { + watchAttachments?: WatchAttachments + archivedCacheLimit?: number + errorHandler?: AttachmentErrorHandler + remoteStorage?: RemoteStorageAdapter + } = {}, + ) { + const queue = new TanStackDBAttachmentQueue({ + db, + attachmentsCollection, + remoteStorage: overrides.remoteStorage ?? remoteStorage, + localStorage, + watchAttachments: overrides.watchAttachments ?? watchPhotoIds, + syncIntervalMs: SYNC_INTERVAL_MS, + archivedCacheLimit: overrides.archivedCacheLimit ?? 0, + errorHandler: overrides.errorHandler, + }) + onTestFinished(() => queue.stopSync()) + return queue + } + + // Reports every photo_id referenced by the users collection as a watched + // attachment. This mirrors how an application links its domain model to the + // attachment queue using a TanStack DB live query rather than a raw SQL + // watch: the `photo_id IS NOT NULL` filter lives in the query, and each + // change re-emits the full set of referenced ids. + const watchPhotoIdsWith = ( + toItem: (photoId: string) => WatchedAttachmentItem, + ): WatchAttachments => { + return async (onUpdate, signal) => { + const livePhotoIds = createCollection( + liveQueryCollectionOptions({ + query: (q) => + q + .from({ user: usersCollection }) + .where(({ user }) => not(isNull(user.photo_id))) + .select(({ user }) => ({ photo_id: user.photo_id })), + }), + ) + + const emit = () => + void onUpdate( + livePhotoIds.toArray + .map((row) => row.photo_id) + .filter((photoId): photoId is string => photoId != null) + .map(toItem), + ) + + // Emit the current snapshot once ready, then on every change. + await livePhotoIds.stateWhenReady() + emit() + const subscription = livePhotoIds.subscribeChanges(() => emit()) + + signal.addEventListener(`abort`, () => { + subscription.unsubscribe() + livePhotoIds.cleanup() + }) + } + } + + const watchPhotoIds = watchPhotoIdsWith((id) => ({ + id, + fileExtension: `jpg`, + })) + + return { + db, + localStorage, + remoteStorage, + uploadFile, + downloadFile, + deleteFile, + attachmentsCollection, + usersCollection, + createQueue, + watchPhotoIds, + watchPhotoIdsWith, + } + } + + /** Waits until the attachment with `id` reaches the expected state. */ + function waitForState( + collection: { get: (id: string) => TRow | undefined }, + id: string, + state: AttachmentState, + ): Promise { + return vi.waitFor( + () => { + const attachment = collection.get(id) + expect( + (attachment as { state?: AttachmentState } | undefined)?.state, + ).toBe(state) + return attachment! + }, + { timeout: WAIT_TIMEOUT, interval: 50 }, + ) + } + + describe(`save`, () => { + it(`serializes successive watched snapshots through the SDK context`, async () => { + const fixture = await setup() + let update!: Parameters[0] + const queue = fixture.createQueue({ + archivedCacheLimit: 100, + watchAttachments: (callback) => { + update = callback + }, + }) + const record = await queue.save({ + data: createMockJpegBuffer(), + fileExtension: `jpg`, + }) + await queue.syncStorage() + await queue.startSync() + const entered = pDefer() + const release = pDefer() + const held = queue.withAttachmentContext(async () => { + entered.resolve() + await release.promise + }) + await entered.promise + const completed: Array = [] + const first = update([]).then(() => { + completed.push(1) + }) + const second = update([{ id: record.id, fileExtension: `jpg` }]).then( + () => { + completed.push(2) + }, + ) + try { + await Promise.resolve() + expect(completed).toEqual([]) + release.resolve() + await Promise.all([held, first, second]) + expect(completed).toEqual([1, 2]) + expect( + await fixture.db.get(`SELECT state FROM attachments WHERE id = ?`, [ + record.id, + ]), + ).toEqual({ state: AttachmentState.SYNCED }) + } finally { + release.resolve() + await Promise.all([held, first, second]) + } + }) + + it(`preserves the winner file across two queue instances sharing storage`, async () => { + const fixture = await setup() + const first = fixture.createQueue() + const second = fixture.createQueue() + const entered = pDefer() + const release = pDefer() + const saveFile = fixture.localStorage.saveFile.bind(fixture.localStorage) + const write = vi + .spyOn(fixture.localStorage, `saveFile`) + .mockImplementation(async (...args) => { + const size = await saveFile(...args) + if (write.mock.calls.length === 1) { + entered.resolve() + await release.promise + } + return size + }) + const saving = first.save({ + id: `shared-id`, + data: createMockJpegBuffer(), + fileExtension: `jpg`, + }) + try { + await entered.promise + await expect( + second.save({ + id: `shared-id`, + data: new Uint8Array([7, 8, 9]).buffer, + fileExtension: `jpg`, + }), + ).rejects.toThrow(/already/) + expect(write).toHaveBeenCalledTimes(1) + } finally { + release.resolve() + await saving + } + const winner = await saving + expect( + new Uint8Array(await fixture.localStorage.readFile(winner.local_uri!)), + ).toEqual(new Uint8Array(createMockJpegBuffer())) + }) + + it.each([`ready`, `cold`, `on-demand`] as const)( + `preserves an existing file when reused through a %s collection`, + async (phase) => { + const fixture = await setup() + const original = await fixture.createQueue().save({ + id: `reused-after-reopen`, + data: createMockJpegBuffer(), + fileExtension: `jpg`, + }) + const collection = + phase === `ready` + ? fixture.attachmentsCollection + : createCollection( + powerSyncCollectionOptions({ + database: fixture.db, + table: APP_SCHEMA.props.attachments, + syncMode: phase === `on-demand` ? `on-demand` : `eager`, + }), + ) + onTestFinished(() => collection.cleanup()) + const queue = new TanStackDBAttachmentQueue({ + db: fixture.db, + attachmentsCollection: collection, + localStorage: fixture.localStorage, + remoteStorage: fixture.remoteStorage, + watchAttachments: () => {}, + }) + onTestFinished(() => queue.stopSync()) + if (phase !== `ready`) + expect(collection.get(original.id)).toBeUndefined() + await expect( + queue.save({ + id: original.id, + data: new Uint8Array([7, 8, 9]).buffer, + fileExtension: `jpg`, + }), + ).rejects.toThrow() + expect( + await fixture.db.getOptional( + `SELECT id FROM attachments WHERE id = ?`, + [original.id], + ), + ).toEqual({ id: original.id }) + expect(await fixture.localStorage.fileExists(original.local_uri!)).toBe( + true, + ) + expect( + new Uint8Array( + await fixture.localStorage.readFile(original.local_uri!), + ), + ).toEqual(new Uint8Array(createMockJpegBuffer())) + }, + ) + + it(`cleans a partial write and releases its ID for retry`, async () => { + const fixture = await setup() + const write = fixture.localStorage.saveFile.bind(fixture.localStorage) + const savedPaths: Array = [] + vi.spyOn(fixture.localStorage, `saveFile`).mockImplementationOnce( + async (uri, data) => { + savedPaths.push(uri) + await write(uri, data) + throw new Error(`partial write failure`) + }, + ) + const options = { + id: `partial`, + data: createMockJpegBuffer(), + fileExtension: `jpg`, + } + await expect(fixture.createQueue().save(options)).rejects.toThrow( + `partial write failure`, + ) + expect(savedPaths).toHaveLength(1) + expect(await fixture.localStorage.fileExists(savedPaths[0]!)).toBe(false) + expect( + await fixture.db.getOptional( + `SELECT id FROM attachments WHERE id = ?`, + [options.id], + ), + ).toBeNull() + const saved = await fixture.createQueue().save(options) + expect(await fixture.localStorage.fileExists(saved.local_uri!)).toBe(true) + }) + + it(`allows another queue to save a distinct ID while a write is held`, async () => { + const fixture = await setup() + const entered = pDefer() + const release = pDefer() + const write = fixture.localStorage.saveFile.bind(fixture.localStorage) + vi.spyOn(fixture.localStorage, `saveFile`).mockImplementationOnce( + async (...args) => { + const size = await write(...args) + entered.resolve() + await release.promise + return size + }, + ) + const saving = fixture + .createQueue() + .save({ + id: `held`, + data: createMockJpegBuffer(), + fileExtension: `jpg`, + }) + try { + await entered.promise + const other = await fixture + .createQueue() + .save({ + id: `other`, + data: createMockJpegBuffer(), + fileExtension: `jpg`, + }) + expect(await fixture.localStorage.fileExists(other.local_uri!)).toBe( + true, + ) + } finally { + release.resolve() + await saving + } + }) + + it.each([`before`, `after`] as const)( + `preserves delete intent %s an SDK upload`, + async (timing) => { + const fixture = await setup() + const queue = fixture.createQueue() + const uploaded = pDefer() + const release = pDefer() + const remoteFiles = new Set() + fixture.uploadFile.mockImplementation(async (_, attachment) => { + uploaded.resolve() + await release.promise + remoteFiles.add(attachment.id) + }) + fixture.deleteFile.mockImplementation((attachment) => { + remoteFiles.delete(attachment.id) + return Promise.resolve() + }) + const userId = randomUUID() + const record = await queue.save({ + data: createMockJpegBuffer(), + fileExtension: `jpg`, + updateHook: (attachment) => { + fixture.usersCollection.insert({ + id: userId, + name: `owner`, + email: null, + photo_id: attachment.id, + }) + }, + }) + let sync: Promise | undefined + try { + if (timing !== `before`) { + sync = queue.syncStorage() + await uploaded.promise + release.resolve() + await sync + } + await queue.delete({ + id: record.id, + updateHook: () => { + fixture.usersCollection.update(userId, (row) => { + row.photo_id = null + }) + }, + }) + expect( + await fixture.db.get(`SELECT state FROM attachments WHERE id = ?`, [ + record.id, + ]), + ).toEqual({ state: AttachmentState.QUEUED_DELETE }) + expect( + await fixture.db.get(`SELECT photo_id FROM users WHERE id = ?`, [ + userId, + ]), + ).toEqual({ photo_id: null }) + release.resolve() + await sync + // Complete two real SDK passes, not a timeout waiting for a mock state. + await queue.syncStorage() + await queue.syncStorage() + expect(fixture.deleteFile).toHaveBeenCalledTimes(1) + expect(remoteFiles.has(record.id)).toBe(false) + expect(await fixture.localStorage.fileExists(record.local_uri!)).toBe( + false, + ) + } finally { + release.resolve() + await sync + } + }, + ) + + it(`writes the local file and inserts a QUEUED_UPLOAD row into the collection`, async () => { + const { createQueue, attachmentsCollection, localStorage } = await setup() + const queue = createQueue() + + const data = new Uint8Array(123).fill(42).buffer + const record = await queue.save({ + data, + fileExtension: `jpg`, + mediaType: `image/jpeg`, + }) + + expect(record.size).toBe(123) + expect(record.state).toBe(AttachmentState.QUEUED_UPLOAD) + expect(record.media_type).toBe(`image/jpeg`) + expect(record.filename).toBe(`${record.id}.jpg`) + expect(record.has_synced).toBe(0) + + // The file should exist on disk at the returned local_uri. + expect(await localStorage.fileExists(record.local_uri!)).toBe(true) + + // The row should be reflected in the collection once it syncs back. + await waitForState( + attachmentsCollection, + record.id, + AttachmentState.QUEUED_UPLOAD, + ) + }) + + it(`commits the updateHook mutation atomically with the attachment row`, async () => { + const { createQueue, attachmentsCollection, usersCollection } = + await setup() + const queue = createQueue() + + const userId = randomUUID() + const record = await queue.save({ + data: createMockJpegBuffer(), + fileExtension: `jpg`, + updateHook: (attachment) => { + usersCollection.insert({ + id: userId, + name: `steven`, + email: `steven@journeyapps.com`, + photo_id: attachment.id, + }) + }, + }) + + // Both the attachment and the linked user row should appear together. + await waitForState( + attachmentsCollection, + record.id, + AttachmentState.QUEUED_UPLOAD, + ) + await vi.waitFor( + () => { + const user = usersCollection.get(userId) + expect(user?.photo_id).toBe(record.id) + }, + { timeout: WAIT_TIMEOUT, interval: 50 }, + ) + }) + + it(`uploads the saved file and transitions it to SYNCED`, async () => { + const { + createQueue, + attachmentsCollection, + usersCollection, + uploadFile, + } = await setup() + const queue = createQueue() + await queue.startSync() + + const userId = randomUUID() + const record = await queue.save({ + data: createMockJpegBuffer(), + fileExtension: `jpg`, + updateHook: (attachment) => { + usersCollection.insert({ + id: userId, + name: `steven`, + email: `steven@journeyapps.com`, + photo_id: attachment.id, + }) + }, + }) + + await waitForState( + attachmentsCollection, + record.id, + AttachmentState.SYNCED, + ) + + expect(uploadFile).toHaveBeenCalled() + const [, uploadedAttachment] = uploadFile.mock.calls[0]! + expect(uploadedAttachment.id).toBe(record.id) + }) + + it(`honours a caller-supplied id`, async () => { + const { createQueue } = await setup() + const queue = createQueue() + + const id = `my-custom-id` + const record = await queue.save({ + id, + data: createMockJpegBuffer(), + fileExtension: `png`, + }) + + expect(record.id).toBe(id) + expect(record.filename).toBe(`${id}.png`) + }) + + it(`rejects a reused id without disturbing the existing attachment`, async () => { + const { createQueue, attachmentsCollection, localStorage } = await setup() + const queue = createQueue() + + const original = await queue.save({ + data: createMockJpegBuffer(), + fileExtension: `jpg`, + }) + + await expect( + queue.save({ + id: original.id, + // A different payload, so an overwrite would be detectable by size alone. + data: new Uint8Array(999).fill(7).buffer, + fileExtension: `jpg`, + }), + ).rejects.toThrow(/already exists/) + + // Without the up-front check the reused id overwrites this file and cleanup + // then deletes it, leaving the original record pointing at nothing. + expect(await localStorage.fileExists(original.local_uri!)).toBe(true) + expect(attachmentsCollection.get(original.id)?.size).toBe(original.size) + }) + + it(`keeps the winner's file intact when two saves race on the same id`, async () => { + const { createQueue, attachmentsCollection, localStorage } = await setup() + const queue = createQueue() + + const id = `contended-id` + // Distinct payload sizes, so an overwrite is detectable from the file length + const smallPayload = createMockJpegBuffer() + const largePayload = new Uint8Array(999).fill(7).buffer + expect(smallPayload.byteLength).not.toBe(largePayload.byteLength) + + // Both calls run their duplicate check before either has inserted + const results = await Promise.allSettled([ + queue.save({ id, data: smallPayload, fileExtension: `jpg` }), + queue.save({ id, data: largePayload, fileExtension: `jpg` }), + ]) + + const fulfilled = results.filter( + (result): result is PromiseFulfilledResult => + result.status === `fulfilled`, + ) + const rejected = results.filter( + (result): result is PromiseRejectedResult => + result.status === `rejected`, + ) + + expect(fulfilled).toHaveLength(1) + expect(rejected).toHaveLength(1) + expect(rejected[0]!.reason).toEqual( + expect.objectContaining({ + message: expect.stringMatching(/exists|being saved/), + }), + ) + + const winner = fulfilled[0]!.value + expect(attachmentsCollection.size).toBe(1) + expect(attachmentsCollection.get(id)?.size).toBe(winner.size) + + // The loser's cleanup must not delete the file the winner's record points at + expect(await localStorage.fileExists(winner.local_uri!)).toBe(true) + const onDisk = await localStorage.readFile(winner.local_uri!) + expect(onDisk.byteLength).toBe(winner.size) + }) + + it(`removes the local file and rolls back when the updateHook throws`, async () => { + const { + createQueue, + attachmentsCollection, + usersCollection, + localStorage, + } = await setup() + const queue = createQueue() + + const id = randomUUID() + let localUri: string | undefined + + await expect( + queue.save({ + id, + data: createMockJpegBuffer(), + fileExtension: `jpg`, + updateHook: (attachment) => { + localUri = attachment.local_uri! + usersCollection.insert({ + id: randomUUID(), + name: `steven`, + email: `steven@journeyapps.com`, + photo_id: attachment.id, + }) + throw new Error(`updateHook failed`) + }, + }), + ).rejects.toThrow(/updateHook failed/) + + // The file is written before the transaction opens, so it must be cleaned up. + expect(localUri).toBeDefined() + expect(await localStorage.fileExists(localUri!)).toBe(false) + + // Neither the attachment nor the hook's own mutation may survive the failure. + expect(attachmentsCollection.get(id)).toBeUndefined() + expect(usersCollection.size).toBe(0) + }) + }) + + describe(`delete file`, () => { + it.each([`eager`, `on-demand`] as const)( + `can retry a failed save and later delete in %s mode`, + async (syncMode) => { + const fixture = await setup(syncMode) + const first = fixture.createQueue() + const options = { + id: `retry`, + data: createMockJpegBuffer(), + fileExtension: `jpg`, + } + await expect( + first.save({ + ...options, + updateHook: () => { + throw new Error(`hook failure`) + }, + }), + ).rejects.toThrow(`hook failure`) + const second = fixture.createQueue() + const saved = await second.save(options) + expect( + new Uint8Array(await fixture.localStorage.readFile(saved.local_uri!)), + ).toEqual(new Uint8Array(createMockJpegBuffer())) + await first.delete({ id: saved.id }) + expect( + await fixture.db.get(`SELECT state FROM attachments WHERE id = ?`, [ + saved.id, + ]), + ).toEqual({ state: AttachmentState.QUEUED_DELETE }) + }, + ) + + it(`finds a queued file after its storage root moves`, async () => { + const fixture = await setup() + const original = await fixture.createQueue().save({ + id: `moving`, + data: createMockJpegBuffer(), + fileExtension: `jpg`, + }) + const moved = new NodeFileSystemAdapter( + join(tmpdir(), `ps-moved-${randomUUID()}`), + ) + await moved.initialize() + onTestFinished(() => moved.clear()) + // Move the bytes without changing SQLite, as a changed app directory does. + const movedUri = moved.getLocalUri(original.local_uri!.split(`/`).at(-1)!) + await moved.saveFile( + movedUri, + await fixture.localStorage.readFile(original.local_uri!), + ) + await fixture.localStorage.deleteFile(original.local_uri!) + const queue = new TanStackDBAttachmentQueue({ + db: fixture.db, + attachmentsCollection: fixture.attachmentsCollection, + localStorage: moved, + remoteStorage: fixture.remoteStorage, + watchAttachments: () => {}, + }) + onTestFinished(() => queue.stopSync()) + await queue.startSync() + await vi.waitFor(() => + expect(fixture.uploadFile).toHaveBeenCalledTimes(1), + ) + expect(fixture.uploadFile.mock.calls[0]![1].localUri).toBe(movedUri) + expect(new Uint8Array(await moved.readFile(movedUri))).toEqual( + new Uint8Array(createMockJpegBuffer()), + ) + }) + + it.each([`eager`, `on-demand`] as const)( + `loads an uncached attachment before deleting in %s mode`, + async (syncMode) => { + const fixture = await setup() + const original = await fixture.createQueue().save({ + id: `uncached`, + data: createMockJpegBuffer(), + fileExtension: `jpg`, + }) + const collection = createCollection( + powerSyncCollectionOptions({ + database: fixture.db, + table: APP_SCHEMA.props.attachments, + syncMode, + }), + ) + onTestFinished(() => collection.cleanup()) + const queue = new TanStackDBAttachmentQueue({ + db: fixture.db, + attachmentsCollection: collection, + localStorage: fixture.localStorage, + remoteStorage: fixture.remoteStorage, + watchAttachments: () => {}, + }) + onTestFinished(() => queue.stopSync()) + expect(collection.get(original.id)).toBeUndefined() + await queue.delete({ id: original.id }) + expect( + await fixture.db.get(`SELECT state FROM attachments WHERE id = ?`, [ + original.id, + ]), + ).toEqual({ state: AttachmentState.QUEUED_DELETE }) + }, + ) + + it(`queues an existing attachment for deletion and removes the local file`, async () => { + const { + createQueue, + attachmentsCollection, + usersCollection, + localStorage, + } = await setup() + const queue = createQueue() + await queue.startSync() + + const userId = randomUUID() + const record = await queue.save({ + data: createMockJpegBuffer(), + fileExtension: `jpg`, + updateHook: (attachment) => { + usersCollection.insert({ + id: userId, + name: `steven`, + email: `steven@journeyapps.com`, + photo_id: attachment.id, + }) + }, + }) + + await waitForState( + attachmentsCollection, + record.id, + AttachmentState.SYNCED, + ) + + await queue.delete({ + id: record.id, + updateHook: (attachment) => { + usersCollection.update(userId, (draft) => { + if (draft.photo_id === attachment.id) { + draft.photo_id = null + } + }) + }, + }) + + // It should immediately be marked for deletion (and no longer synced). + const queued = attachmentsCollection.get(record.id) + expect(queued?.state).toBe(AttachmentState.QUEUED_DELETE) + expect(queued?.has_synced).toBe(0) + + // The user reference should have been cleared in the same transaction. + expect(usersCollection.get(userId)?.photo_id).toBeNull() + + // Eventually the row and the local file are removed. + await vi.waitFor( + () => expect(attachmentsCollection.get(record.id)).toBeUndefined(), + { timeout: WAIT_TIMEOUT, interval: 50 }, + ) + expect(await localStorage.fileExists(record.local_uri!)).toBe(false) + }) + + it(`throws for an unknown id and commits nothing`, async () => { + const { createQueue, attachmentsCollection, usersCollection } = + await setup() + const queue = createQueue() + + const hook = vi.fn() + await expect( + queue.delete({ id: `does-not-exist`, updateHook: hook }), + ).rejects.toThrow(/not found/i) + + // The failing transaction must not have run the hook or touched state. + expect(hook).not.toHaveBeenCalled() + expect(attachmentsCollection.get(`does-not-exist`)).toBeUndefined() + expect(usersCollection.size).toBe(0) + }) + + it(`rolls back the queued deletion when the updateHook throws`, async () => { + const { + createQueue, + attachmentsCollection, + usersCollection, + localStorage, + } = await setup() + const queue = createQueue() + + const userId = randomUUID() + const record = await queue.save({ + data: createMockJpegBuffer(), + fileExtension: `jpg`, + updateHook: (attachment) => { + usersCollection.insert({ + id: userId, + name: `steven`, + email: `steven@journeyapps.com`, + photo_id: attachment.id, + }) + }, + }) + + // Sync is deliberately left stopped: the rollback happens entirely in the + // foreground transaction, and a running sync loop would only race teardown. + await waitForState( + attachmentsCollection, + record.id, + AttachmentState.QUEUED_UPLOAD, + ) + + await expect( + queue.delete({ + id: record.id, + updateHook: () => { + usersCollection.update(userId, (draft) => { + draft.photo_id = null + }) + throw new Error(`updateHook failed`) + }, + }), + ).rejects.toThrow(/updateHook failed/) + + // Both the QUEUED_DELETE transition and the hook's mutation must be rolled back, + // leaving the attachment intact rather than half-deleted. + expect(attachmentsCollection.get(record.id)?.state).toBe( + AttachmentState.QUEUED_UPLOAD, + ) + expect(usersCollection.get(userId)?.photo_id).toBe(record.id) + expect(await localStorage.fileExists(record.local_uri!)).toBe(true) + }) + }) +}) diff --git a/packages/powersync-db-collection/tests/load-hooks.test.ts b/packages/powersync-db-collection/tests/load-hooks.test.ts index 6a3f5cb1e0..b5094f6156 100644 --- a/packages/powersync-db-collection/tests/load-hooks.test.ts +++ b/packages/powersync-db-collection/tests/load-hooks.test.ts @@ -2,6 +2,7 @@ import { randomUUID } from 'node:crypto' import { tmpdir } from 'node:os' import { PowerSyncDatabase, Schema, Table, column } from '@powersync/node' import { createCollection, createLiveQueryCollection, eq } from '@tanstack/db' +import pDefer from 'p-defer' import { describe, expect, it, onTestFinished, vi } from 'vitest' import { powerSyncCollectionOptions } from '../src' @@ -75,12 +76,58 @@ describe(`Sync Streams`, () => { expect(onUnloadMock).toHaveBeenCalledOnce() }) + it(`eager mode: reports an initial load failure`, async () => { + const db = await createDatabase() + const initialError = new Error(`initial PowerSync load failed`) + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + onLoad: () => Promise.reject(initialError), + }), + ) + onTestFinished(() => collection.cleanup()) + + await expect(collection.preload()).rejects.toBe(initialError) + expect(collection.status).toBe(`error`) + }) + + it(`eager mode: releases a load hook that resolves after cleanup`, async () => { + const db = await createDatabase() + const releaseLoad = pDefer() + const loadStarted = pDefer() + const cleanupLoad = vi.fn() + const createDiffTrigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockResolvedValue(async () => {}) + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + onLoad: async () => { + loadStarted.resolve() + await releaseLoad.promise + return cleanupLoad + }, + }), + ) + + await loadStarted.promise + collection.cleanup() + releaseLoad.resolve() + + await vi.waitFor(() => expect(cleanupLoad).toHaveBeenCalledOnce()) + expect(createDiffTrigger).not.toHaveBeenCalled() + }) + it(`on-demand mode: should call onLoadSubset/onUnloadSubset for each live query`, async () => { const db = await createDatabase() await createTestProducts(db) const onLoadSubsetMock = vi.fn() const onUnloadSubsetMock = vi.fn() + const unloadedRequests: Array = [] + let nextRequest = 0 const collection = createCollection( powerSyncCollectionOptions({ @@ -89,9 +136,12 @@ describe(`Sync Streams`, () => { syncMode: `on-demand`, onLoadSubset: () => { onLoadSubsetMock() + nextRequest += 1 + const request = nextRequest return () => { onUnloadSubsetMock() + unloadedRequests.push(request) } }, }), @@ -158,6 +208,7 @@ describe(`Sync Streams`, () => { await vi.waitFor( () => { expect(onUnloadSubsetMock).toHaveBeenCalledTimes(1) + expect(unloadedRequests).toEqual([1]) }, { timeout: 2000 }, ) @@ -168,8 +219,51 @@ describe(`Sync Streams`, () => { await vi.waitFor( () => { expect(onUnloadSubsetMock).toHaveBeenCalledTimes(2) + expect(unloadedRequests).toEqual([1, 2]) }, { timeout: 2000 }, ) }) + + it(`disposes a subset hook that resolves after collection cleanup`, async () => { + const db = await createDatabase() + await createTestProducts(db) + let resolveHook!: (cleanup: () => void) => void + const hook = new Promise<() => void>((resolve) => { + resolveHook = resolve + }) + const cleanupHook = vi.fn() + + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + syncMode: `on-demand`, + onLoadSubset: () => hook, + }), + ) + await collection.stateWhenReady() + const query = createLiveQueryCollection({ + query: (q) => + q + .from({ product: collection }) + .where(({ product }) => eq(product.category, `electronics`)) + .select(({ product }) => ({ + id: product.id, + name: product.name, + price: product.price, + category: product.category, + })), + }) + const preload = query.preload() + void preload.catch(() => {}) + + await vi.waitFor(() => expect(resolveHook).toBeTypeOf(`function`)) + const collectionCleanup = collection.cleanup() + resolveHook(cleanupHook) + await collectionCleanup + + await vi.waitFor(() => expect(cleanupHook).toHaveBeenCalledOnce()) + await query.cleanup() + }) }) diff --git a/packages/powersync-db-collection/tests/on-demand-sync.test.ts b/packages/powersync-db-collection/tests/on-demand-sync.test.ts index 4460fd0216..f6676d5e3f 100644 --- a/packages/powersync-db-collection/tests/on-demand-sync.test.ts +++ b/packages/powersync-db-collection/tests/on-demand-sync.test.ts @@ -2,17 +2,21 @@ import { randomUUID } from 'node:crypto' import { tmpdir } from 'node:os' import { PowerSyncDatabase, Schema, Table, column } from '@powersync/node' import { + IR, and, createCollection, createLiveQueryCollection, + createTransaction, eq, gt, gte, lt, or, } from '@tanstack/db' +import pDefer from 'p-defer' import { describe, expect, it, onTestFinished, vi } from 'vitest' import { powerSyncCollectionOptions } from '../src' +import type { LoadSubsetOptions } from '@tanstack/db' const APP_SCHEMA = new Schema({ products: new Table({ @@ -140,6 +144,82 @@ describe(`On-Demand Sync Mode`, () => { expect(prices).toEqual([150, 200]) }) + it(`resolves subset readiness only after its rows are applied`, async () => { + const db = await createDatabase() + await createTestProducts(db) + + let resolvePersistence!: () => void + const persistence = new Promise((resolve) => { + resolvePersistence = resolve + }) + const transaction = createTransaction({ + mutationFn: () => persistence, + }) + const options = powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + syncMode: `on-demand`, + onLoadSubset: () => { + transaction.mutate(() => + collection.insert({ + id: `local`, + name: `Local product`, + price: 1, + category: `local`, + }), + ) + }, + }) + const collection = createCollection(options) + onTestFinished(() => collection.cleanup()) + await collection.stateWhenReady() + + const electronics = createLiveQueryCollection({ + query: (q) => + q + .from({ product: collection }) + .where(({ product }) => eq(product.category, `electronics`)), + }) + onTestFinished(() => electronics.cleanup()) + const preload = electronics.preload() + let settled = false + void preload.then(() => { + settled = true + }) + + try { + const { trackedTableName } = options.utils.getMeta() + await vi.waitFor( + async () => { + const table = await db.writeLock((context) => + context.get<{ count: number }>( + `SELECT COUNT(*) as count FROM sqlite_temp_master WHERE type = 'table' AND name = ?`, + [trackedTableName], + ), + ) + expect(table.count).toBe(1) + }, + { timeout: 2_000 }, + ) + + expect(transaction.state).toBe(`persisting`) + expect(settled).toBe(false) + expect(electronics.size).toBe(0) + + resolvePersistence() + await transaction.isPersisted.promise + await preload + + expect(electronics.toArray.map((product) => product.name).sort()).toEqual( + [`Product A`, `Product B`, `Product D`], + ) + } finally { + resolvePersistence() + await transaction.isPersisted.promise.catch(() => undefined) + await Promise.allSettled([preload]) + } + }) + it(`should reactively update live query when new matching data is inserted into SQLite`, async () => { const db = await createDatabase() await createTestProducts(db) @@ -2054,26 +2134,120 @@ describe(`On-Demand Sync Mode`, () => { ) }) - it(`should resolve isPersisted when all live queries are cleaned up during a pending mutation`, async () => { - const db = await createDatabase() - await createTestProducts(db) + it.each([`insert`, `update`, `delete`] as const)( + `persists a pending %s when its last live query is cleaned up`, + async (operation) => { + const db = await createDatabase() + await createTestProducts(db) + + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + syncMode: `on-demand`, + }), + ) + onTestFinished(() => collection.cleanup()) + await collection.stateWhenReady() + + // Start with 1 live query (electronics) + const electronicsQuery = createLiveQueryCollection({ + query: (q) => + q + .from({ product: collection }) + .where(({ product }) => eq(product.category, `electronics`)) + .select(({ product }) => ({ + id: product.id, + name: product.name, + price: product.price, + category: product.category, + })), + }) - const collection = createCollection( + await electronicsQuery.preload() + + await vi.waitFor( + () => { + expect(electronicsQuery.size).toBe(3) + }, + { timeout: 2000 }, + ) + + const existing = Array.from(electronicsQuery.values())[0]! + const id = operation === `insert` ? randomUUID() : existing.id + const mutation = + operation === `insert` + ? collection.insert({ + id, + name: `New Gadget`, + price: 99, + category: `electronics`, + }) + : operation === `update` + ? collection.update(id, (draft) => { + draft.name = `New Gadget` + }) + : collection.delete(id) + let settled = false + const observed = mutation.isPersisted.promise.then( + () => { + settled = true + return { status: `fulfilled` as const } + }, + (error: unknown) => { + settled = true + return { status: `rejected` as const, reason: error } + }, + ) + + // Dropping the last demand must still drain the mutation's diff record + // before removing the trigger that acknowledges its persistence. + electronicsQuery.cleanup() + await vi.waitFor(() => expect(settled).toBe(true), { timeout: 2000 }) + expect(await observed).toEqual({ status: `fulfilled` }) + expect( + await db.getAll(`SELECT id, name FROM products WHERE id = ?`, [id]), + ).toEqual(operation === `delete` ? [] : [{ id, name: `New Gadget` }]) + }, + ) + }) + + describe(`Tracking lifecycle`, () => { + const categoryEquals = (category: string) => + new IR.Func(`eq`, [ + new IR.PropRef([`category`]), + new IR.Value(category), + ]) + + // The sync handler catches its own errors and surfaces them only through the + // logger, so captured errors are how these tests assert it stayed healthy. + function captureSyncErrors(db: PowerSyncDatabase) { + const errors: Array = [] + vi.spyOn(db.logger, `error`).mockImplementation((...args: Array) => { + errors.push(args.map(String).join(` `)) + }) + return () => errors + } + + function makeCollection(db: PowerSyncDatabase) { + return createCollection( powerSyncCollectionOptions({ database: db, table: APP_SCHEMA.props.products, syncMode: `on-demand`, }), ) - onTestFinished(() => collection.cleanup()) - await collection.stateWhenReady() + } - // Start with 1 live query (electronics) - const electronicsQuery = createLiveQueryCollection({ + function categoryQuery( + collection: ReturnType, + category: string, + ) { + return createLiveQueryCollection({ query: (q) => q .from({ product: collection }) - .where(({ product }) => eq(product.category, `electronics`)) + .where(({ product }) => eq(product.category, category)) .select(({ product }) => ({ id: product.id, name: product.name, @@ -2081,9 +2255,733 @@ describe(`On-Demand Sync Mode`, () => { category: product.category, })), }) + } + + function startOnDemandSync( + db: PowerSyncDatabase, + settings: { + onLoadSubset?: ( + options: LoadSubsetOptions, + ) => void | (() => void) | Promise void)> + syncBatchSize?: number + } = {}, + overrides: Partial<{ + begin: ReturnType + write: ReturnType + commit: ReturnType + }> = {}, + ) { + const begin = overrides.begin ?? vi.fn() + const write = overrides.write ?? vi.fn() + const commit = overrides.commit ?? vi.fn(() => true) + const config = powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + syncMode: `on-demand`, + onLoadSubset: settings.onLoadSubset, + syncBatchSize: settings.syncBatchSize, + }) + const sync = config.sync.sync({ + collection: { status: `ready`, has: () => false }, + begin, + write, + commit, + markReady: vi.fn(), + markError: vi.fn(), + truncate: vi.fn(), + } as never) + const loadSubset = + sync && typeof sync !== `function` ? sync.loadSubset : undefined + const unloadSubset = + sync && typeof sync !== `function` ? sync.unloadSubset : undefined + if (!sync || typeof sync === `function` || !loadSubset || !unloadSubset) { + throw new Error(`Expected on-demand sync controls`) + } + return { sync, loadSubset, unloadSubset, begin, write, commit } + } - await electronicsQuery.preload() + it(`does not publish a provisional or rejected subset`, async () => { + const db = await createDatabase() + const firstHook = pDefer() + const hookFailure = new Error(`subset hook failed`) + const onLoadSubset = vi + .fn() + .mockReturnValueOnce(firstHook.promise) + .mockRejectedValueOnce(hookFailure) + .mockResolvedValueOnce(undefined) + const createDiffTrigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockResolvedValue(vi.fn()) + const { sync, loadSubset } = startOnDemandSync(db, { onLoadSubset }) + + try { + const provisional = loadSubset({ + where: categoryEquals(`electronics`), + }) + await vi.waitFor(() => expect(onLoadSubset).toHaveBeenCalledOnce()) + await expect( + loadSubset({ where: categoryEquals(`outdoors`) }), + ).rejects.toBe(hookFailure) + await loadSubset({ where: categoryEquals(`clothing`) }) + + const when = createDiffTrigger.mock.calls.at(-1)?.[0].when + expect(when?.INSERT).toContain(`clothing`) + expect(when?.INSERT).not.toContain(`electronics`) + expect(when?.INSERT).not.toContain(`outdoors`) + + firstHook.resolve() + await provisional + } finally { + firstHook.resolve() + sync.cleanup?.() + } + }) + + it(`does not acquire a subset released during startup`, async () => { + const db = await createDatabase() + const onLoadSubset = vi.fn() + const createDiffTrigger = vi.spyOn(db.triggers, `createDiffTrigger`) + const { sync, loadSubset, unloadSubset } = startOnDemandSync(db, { + onLoadSubset, + }) + const controller = new AbortController() + const request = { + where: categoryEquals(`electronics`), + signal: controller.signal, + } + + const load = loadSubset(request) + controller.abort() + unloadSubset(request) + + try { + await load + expect(onLoadSubset).not.toHaveBeenCalled() + expect(createDiffTrigger).not.toHaveBeenCalled() + } finally { + sync.cleanup?.() + } + }) + + it(`settles concurrent loads only after the latest trigger is live`, async () => { + const db = await createDatabase() + const locks: Array<() => Promise> = [] + vi.spyOn(db, `writeLock`).mockImplementation( + (callback) => + new Promise((resolve, reject) => { + locks.push(async () => { + try { + await callback({} as never) + resolve(undefined as never) + } catch (error) { + reject(error) + } + }) + }) as never, + ) + vi.spyOn(db, `getAll`).mockResolvedValue([]) + const createDiffTrigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockResolvedValue(vi.fn()) + const { sync, loadSubset } = startOnDemandSync(db) + let firstSettled = false + let secondSettled = false + + const first = Promise.resolve( + loadSubset({ where: categoryEquals(`electronics`) }), + ).then(() => { + firstSettled = true + }) + await vi.waitFor(() => expect(locks).toHaveLength(1)) + const second = Promise.resolve( + loadSubset({ where: categoryEquals(`clothing`) }), + ).then(() => { + secondSettled = true + }) + + try { + await locks[0]!() + expect(firstSettled).toBe(false) + expect(secondSettled).toBe(false) + expect(createDiffTrigger).not.toHaveBeenCalled() + + await vi.waitFor(() => expect(locks).toHaveLength(2)) + await locks[1]!() + await Promise.all([first, second]) + + expect(createDiffTrigger).toHaveBeenCalledOnce() + const when = createDiffTrigger.mock.calls[0]?.[0].when + expect(when?.INSERT).toContain(`electronics`) + expect(when?.INSERT).toContain(`clothing`) + } finally { + sync.cleanup?.() + await Promise.all(locks.map((run) => run())) + await Promise.allSettled([first, second]) + } + }) + + it.each(Array.from({ length: 12 }, (_, turn) => turn))( + `covers demand admitted %s microtasks after the final applied receipt`, + async (turn) => { + const db = await createDatabase() + vi.spyOn(db, `writeLock`).mockImplementation(async (callback) => + callback({ + getAll: () => Promise.resolve([]), + execute: () => Promise.resolve({}), + } as never), + ) + const applied = pDefer() + const entered = pDefer() + const createDiffTrigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockImplementation(async (options) => { + await options.hooks?.beforeCreate?.({ + getAll: () => Promise.resolve([]), + } as never) + return vi.fn() + }) + const commit = vi.fn(() => { + entered.resolve() + return applied.promise + }) + const { sync, loadSubset } = startOnDemandSync(db, {}, { commit }) + const first = Promise.resolve( + loadSubset({ where: categoryEquals(`electronics`) }), + ) + let second: Promise | undefined + try { + await entered.promise + // Allow setup to reach the applied-receipt barrier, then vary only + // admission around its promise finalization, not wall-clock timing. + for (let i = 0; i < 20; i++) await Promise.resolve() + applied.resolve() + for (let i = 0; i < turn; i++) await Promise.resolve() + second = Promise.resolve( + loadSubset({ where: categoryEquals(`clothing`) }), + ) + await second + const when = createDiffTrigger.mock.calls.at(-1)?.[0].when + expect(when?.INSERT).toContain(`electronics`) + expect(when?.INSERT).toContain(`clothing`) + // Literal names in SQL are not proof of a working column filter. + // Execute the exact trigger clause against matching and excluded rows. + for (const category of [`electronics`, `clothing`, `outdoors`]) { + const row = await db.get<{ matches: number }>( + `SELECT CASE WHEN (${when!.INSERT}) THEN 1 ELSE 0 END AS matches FROM (SELECT ? AS data) AS NEW`, + [JSON.stringify({ category })], + ) + expect(row.matches).toBe(category === `outdoors` ? 0 : 1) + } + await first + } finally { + applied.resolve() + await Promise.allSettled([first, second]) + sync.cleanup?.() + } + }, + ) + + it(`disposes a trigger superseded while it is being created`, async () => { + const db = await createDatabase() + const triggerStarted = pDefer() + const finishTrigger = pDefer() + const staleDispose = vi.fn(async () => {}) + const currentDispose = vi.fn(async () => {}) + const createDiffTrigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockImplementationOnce(async () => { + triggerStarted.resolve() + await finishTrigger.promise + return staleDispose + }) + .mockResolvedValueOnce(currentDispose) + const { sync, loadSubset } = startOnDemandSync(db) + const first = Promise.resolve( + loadSubset({ where: categoryEquals(`electronics`) }), + ) + + try { + await triggerStarted.promise + const second = Promise.resolve( + loadSubset({ where: categoryEquals(`clothing`) }), + ) + finishTrigger.resolve() + await Promise.all([first, second]) + + expect(createDiffTrigger).toHaveBeenCalledTimes(2) + expect(staleDispose).toHaveBeenCalledOnce() + expect(currentDispose).not.toHaveBeenCalled() + } finally { + finishTrigger.resolve() + sync.cleanup?.() + await first + } + }) + + it(`waits for every applied batch before settling a subset`, async () => { + const db = await createDatabase() + const receipts: Array>> = [] + const rows = [ + { id: `a`, name: `A`, price: 1, category: `electronics` }, + { id: `b`, name: `B`, price: 2, category: `electronics` }, + ] + vi.spyOn(db.triggers, `createDiffTrigger`).mockImplementation( + async (options) => { + let cursor = 0 + await options.hooks?.beforeCreate?.({ + getAll: async () => rows.slice(cursor, ++cursor), + } as never) + return vi.fn() + }, + ) + const commit = vi.fn(() => { + const receipt = pDefer() + receipts.push(receipt) + return receipt.promise + }) + const { sync, loadSubset } = startOnDemandSync( + db, + { syncBatchSize: 1 }, + { commit }, + ) + let settled = false + const load = Promise.resolve( + loadSubset({ where: categoryEquals(`electronics`) }), + ).then(() => { + settled = true + }) + + try { + await vi.waitFor(() => expect(receipts).toHaveLength(3)) + receipts[0]!.resolve() + receipts[1]!.resolve() + await Promise.resolve() + expect(settled).toBe(false) + + receipts[2]!.resolve() + await load + expect(settled).toBe(true) + } finally { + receipts.forEach((receipt) => receipt.resolve()) + sync.cleanup?.() + await load + } + }) + + it(`does not start queued tracking after cleanup`, async () => { + const db = await createDatabase() + const lockQueued = pDefer() + let runLock!: () => Promise + vi.spyOn(db, `writeLock`).mockImplementation( + (callback) => + new Promise((resolve, reject) => { + runLock = async () => { + try { + await callback({} as never) + resolve(undefined as never) + } catch (error) { + reject(error) + } + } + lockQueued.resolve() + }) as never, + ) + const createDiffTrigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockResolvedValue(vi.fn()) + const { sync, loadSubset } = startOnDemandSync(db) + + const load = loadSubset({ where: categoryEquals(`electronics`) }) + await lockQueued.promise + sync.cleanup?.() + await runLock() + await load + + expect(createDiffTrigger).not.toHaveBeenCalled() + }) + + it(`cleans each acquired subset at most once during reentrant cleanup`, async () => { + const db = await createDatabase() + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const first = { where: categoryEquals(`electronics`) } + const second = { where: categoryEquals(`clothing`) } + const firstCleanup = vi.fn() + const secondCleanup = vi.fn(() => started.unloadSubset(first)) + const onLoadSubset = vi.fn((options: LoadSubsetOptions) => + options === first ? firstCleanup : secondCleanup, + ) + const started = startOnDemandSync(db, { onLoadSubset }) + + await Promise.all([started.loadSubset(first), started.loadSubset(second)]) + started.sync.cleanup?.() + expect(firstCleanup).toHaveBeenCalledOnce() + expect(secondCleanup).toHaveBeenCalledOnce() + }) + + it(`does not repeat release work started by a reentrant cleanup`, async () => { + const db = await createDatabase() + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const getAll = vi.spyOn(db, `getAll`).mockResolvedValue([]) + const first = { where: categoryEquals(`electronics`) } + const second = { where: categoryEquals(`clothing`) } + const onLoadSubset = vi.fn((options: LoadSubsetOptions) => + options === first ? () => started.unloadSubset(second) : undefined, + ) + const started = startOnDemandSync(db, { onLoadSubset }) + + try { + await Promise.all([ + started.loadSubset(first), + started.loadSubset(second), + ]) + started.unloadSubset(first) + await vi.waitFor(() => + expect( + getAll.mock.calls.some(([sql]) => + String(sql).includes(`electronics`), + ), + ).toBe(true), + ) + + expect( + getAll.mock.calls.filter(([sql]) => String(sql).includes(`clothing`)), + ).toHaveLength(1) + } finally { + started.sync.cleanup?.() + } + }) + + it(`does not create tracking when change observation cannot start`, async () => { + const db = await createDatabase() + const startupError = new Error(`change observation failed`) + vi.spyOn(db.logger, `error`).mockImplementation(() => {}) + vi.spyOn(db, `onChangeWithCallback`).mockImplementation(() => { + throw startupError + }) + const createDiffTrigger = vi.spyOn(db.triggers, `createDiffTrigger`) + const collection = makeCollection(db) + onTestFinished(() => collection.cleanup()) + await collection.stateWhenReady() + const query = categoryQuery(collection, `electronics`) + onTestFinished(() => query.cleanup()) + + await expect(query.preload()).rejects.toBe(startupError) + expect(createDiffTrigger).not.toHaveBeenCalled() + }) + + it(`flushes a change observed while eager tracking starts`, async () => { + const db = await createDatabase() + await createTestProducts(db) + let flush: + | ((event: { changedTables: Array }) => Promise | void) + | undefined + vi.spyOn(db, `onChangeWithCallback`).mockImplementation((handler) => { + flush = handler?.onChange + return () => {} + }) + const triggerCreated = pDefer() + const publishTrigger = pDefer() + const createDiffTrigger = db.triggers.createDiffTrigger.bind(db.triggers) + vi.spyOn(db.triggers, `createDiffTrigger`).mockImplementation( + async (options) => { + const dispose = await createDiffTrigger(options) + triggerCreated.resolve() + await publishTrigger.promise + return dispose + }, + ) + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + }), + ) + onTestFinished(() => collection.cleanup()) + + await triggerCreated.promise + await db.execute(` + INSERT INTO products (id, name, price, category) + VALUES ('during-startup', 'During startup', 300, 'electronics') + `) + const observed = Promise.resolve( + flush?.({ + changedTables: [collection.utils.getMeta().trackedTableName], + }), + ) + publishTrigger.resolve() + await Promise.all([observed, collection.stateWhenReady()]) + + expect(collection.get(`during-startup`)?.name).toBe(`During startup`) + }) + + it(`disposes eager tracking that finishes after cleanup`, async () => { + const db = await createDatabase() + vi.spyOn(db, `onChangeWithCallback`).mockImplementation(() => () => {}) + const triggerStarted = pDefer() + const finishTrigger = pDefer() + const dispose = vi.fn(async () => {}) + vi.spyOn(db.triggers, `createDiffTrigger`).mockImplementation( + async () => { + triggerStarted.resolve() + await finishTrigger.promise + return dispose + }, + ) + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + }), + ) + + await triggerStarted.promise + collection.cleanup() + finishTrigger.resolve() + + await vi.waitFor(() => expect(dispose).toHaveBeenCalledOnce()) + }) + + it(`reports a source error when a rebuild removes tracking and cannot replace it`, async () => { + const db = await createDatabase() + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + syncMode: `on-demand`, + }), + ) + const failure = new Error(`trigger installation failed`) + try { + await collection._sync.loadSubset({ + where: categoryEquals(`electronics`), + }) + expect(collection.status).toBe(`ready`) + vi.spyOn(db.triggers, `createDiffTrigger`).mockRejectedValueOnce( + failure, + ) + await expect( + Promise.resolve( + collection._sync.loadSubset({ where: categoryEquals(`clothing`) }), + ), + ).rejects.toBe(failure) + expect(collection.status).toBe(`error`) + } finally { + await collection.cleanup() + } + }) + + it.each([`unchanged`, `delete`, `predicate exit`, `release-last`] as const)( + `reconciles rows after a release rebuild outage with %s`, + async (change) => { + vi.useFakeTimers() + const db = await createDatabase() + await db.execute( + `INSERT INTO products (id, name, price, category) VALUES ('retained', 'Before', 10, 'clothing')`, + ) + vi.spyOn(db.logger, `error`).mockImplementation(() => {}) + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + syncMode: `on-demand`, + }), + ) + const first = { where: categoryEquals(`electronics`) } + const second = { where: categoryEquals(`clothing`) } + try { + await collection._sync.loadSubset(first) + await collection._sync.loadSubset(second) + const trigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockRejectedValueOnce(new Error(`release rebuild failed`)) + collection._sync.unloadSubset(first) + await vi.waitFor(() => expect(collection.status).toBe(`error`)) + if (change === `delete` || change === `release-last`) { + await db.execute(`DELETE FROM products WHERE id = 'retained'`) + } else if (change === `predicate exit`) { + await db.execute( + `UPDATE products SET category = 'outdoors' WHERE id = 'retained'`, + ) + } + if (change === `release-last`) collection._sync.unloadSubset(second) + await vi.advanceTimersByTimeAsync(1_000) + if (change !== `release-last`) + await vi.waitFor(() => + expect(trigger.mock.calls.length).toBeGreaterThan(1), + ) + await vi.waitFor(() => expect(collection.status).toBe(`ready`)) + expect([...collection.keys()]).toEqual( + change === `unchanged` ? [`retained`] : [], + ) + if (change === `release-last`) + await collection._sync.loadSubset({ ...second }) + if (change !== `unchanged`) { + await db.execute( + `INSERT OR REPLACE INTO products (id, name, price, category) VALUES ('retained', 'Before', 10, 'clothing')`, + ) + await vi.waitFor(() => + expect(collection.get(`retained`)?.name).toBe(`Before`), + ) + } + await db.execute( + `UPDATE products SET name = 'After' WHERE id = 'retained'`, + ) + await vi.waitFor(() => + expect(collection.get(`retained`)?.name).toBe(`After`), + ) + } finally { + await collection.cleanup() + await vi.runOnlyPendingTimersAsync() + vi.useRealTimers() + } + }, + ) + + it(`retries a failed physical release`, async () => { + vi.useFakeTimers() + const db = await createDatabase() + vi.spyOn(db.logger, `error`).mockImplementation(() => {}) + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const getAll = vi + .spyOn(db, `getAll`) + .mockRejectedValueOnce(new Error(`transient eviction failure`)) + .mockResolvedValueOnce([]) + const { sync, loadSubset, unloadSubset } = startOnDemandSync(db) + const request = { where: categoryEquals(`electronics`) } + + try { + await loadSubset(request) + expect(unloadSubset(request)).toBeUndefined() + await vi.waitFor(() => expect(getAll).toHaveBeenCalledOnce()) + await vi.advanceTimersByTimeAsync(1_000) + await vi.waitFor(() => expect(getAll).toHaveBeenCalledTimes(2)) + } finally { + sync.cleanup?.() + await vi.runOnlyPendingTimersAsync() + vi.useRealTimers() + } + }) + + it(`does not let one failed release block another`, async () => { + vi.useFakeTimers() + const db = await createDatabase() + vi.spyOn(db.logger, `error`).mockImplementation(() => {}) + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const getAll = vi + .spyOn(db, `getAll`) + .mockImplementation((sql) => + String(sql).includes(`electronics`) + ? Promise.reject(new Error(`persistent eviction failure`)) + : Promise.resolve([]), + ) + const { sync, loadSubset, unloadSubset } = startOnDemandSync(db) + const failing = { where: categoryEquals(`electronics`) } + const succeeding = { where: categoryEquals(`clothing`) } + + try { + await Promise.all([loadSubset(failing), loadSubset(succeeding)]) + unloadSubset(failing) + unloadSubset(succeeding) + await vi.waitFor(() => expect(getAll).toHaveBeenCalled()) + await vi.advanceTimersByTimeAsync(1_000) + + expect( + getAll.mock.calls.some(([sql]) => { + const query = String(sql) + return query.includes(`clothing`) && !query.includes(`electronics`) + }), + ).toBe(true) + } finally { + sync.cleanup?.() + await vi.runOnlyPendingTimersAsync() + vi.useRealTimers() + } + }) + + it(`evicts a newly released demand without waiting for another demand's retry timer`, async () => { + vi.useFakeTimers() + const db = await createDatabase() + vi.spyOn(db.logger, `error`).mockImplementation(() => {}) + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const getAll = vi + .spyOn(db, `getAll`) + .mockImplementation((sql) => + String(sql).includes(`electronics`) + ? Promise.reject(new Error(`eviction failed`)) + : Promise.resolve([]), + ) + const { sync, loadSubset, unloadSubset } = startOnDemandSync(db) + const first = { where: categoryEquals(`electronics`) } + const second = { where: categoryEquals(`clothing`) } + try { + await Promise.all([loadSubset(first), loadSubset(second)]) + unloadSubset(first) + for (let turn = 0; turn < 30; turn++) await Promise.resolve() + const callsAfterFailure = getAll.mock.calls.length + expect(callsAfterFailure).toBe(1) + unloadSubset(second) + for (let turn = 0; turn < 30; turn++) await Promise.resolve() + expect( + getAll.mock.calls + .slice(callsAfterFailure) + .some(([sql]) => String(sql).includes(`clothing`)), + ).toBe(true) + } finally { + sync.cleanup?.() + await vi.runOnlyPendingTimersAsync() + vi.useRealTimers() + } + }) + + it(`rechecks active demand before evicting released rows`, async () => { + const db = await createDatabase() + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const firstEviction = pDefer>() + const getAll = vi + .spyOn(db, `getAll`) + .mockReturnValueOnce(firstEviction.promise) + .mockResolvedValueOnce([]) + const write = vi.fn() + const { sync, loadSubset, unloadSubset } = startOnDemandSync( + db, + {}, + { write }, + ) + const departing = { where: categoryEquals(`electronics`) } + + try { + await loadSubset(departing) + unloadSubset(departing) + await vi.waitFor(() => expect(getAll).toHaveBeenCalledOnce()) + + await loadSubset({ where: categoryEquals(`clothing`) }) + firstEviction.resolve([{ id: `now-owned` }]) + + await vi.waitFor(() => expect(getAll).toHaveBeenCalledTimes(2)) + expect(write).not.toHaveBeenCalledWith({ + type: `delete`, + key: `now-owned`, + }) + } finally { + firstEviction.resolve([]) + sync.cleanup?.() + } + }) + + it(`should start tracking again when a subset is loaded after every subset was unloaded`, async () => { + const db = await createDatabase() + await createTestProducts(db) + const syncErrors = captureSyncErrors(db) + + const collection = makeCollection(db) + onTestFinished(() => collection.cleanup()) + await collection.stateWhenReady() + + // Load a subset, then unload it so no predicates remain. Tracking stops and + // the tracking table is dropped. + const electronicsQuery = categoryQuery(collection, `electronics`) + await electronicsQuery.preload() await vi.waitFor( () => { expect(electronicsQuery.size).toBe(3) @@ -2091,25 +2989,113 @@ describe(`On-Demand Sync Mode`, () => { { timeout: 2000 }, ) - // Insert a new electronics product — creates a pending mutation - const insertResult = collection.insert({ - id: randomUUID(), - name: `New Gadget`, - price: 99, - category: `electronics`, - }) + electronicsQuery.cleanup() + await vi.waitFor( + () => { + expect(collection.size).toBe(0) + }, + { timeout: 2000 }, + ) + + // A new subset gets a freshly created tracking table and syncs normally. + const clothingQuery = categoryQuery(collection, `clothing`) + onTestFinished(() => clothingQuery.cleanup()) + await clothingQuery.preload() + + await vi.waitFor( + () => { + expect(clothingQuery.size).toBe(2) + }, + { timeout: 2000 }, + ) + + expect(syncErrors()).toEqual([]) + }) + + it(`should stop tracking cleanly when every subset is unloaded and the collection is cleaned up`, async () => { + const db = await createDatabase() + await createTestProducts(db) + const syncErrors = captureSyncErrors(db) - // Immediately clean up the only live query — triggers unloadSubset → loadSubset - // with 0 predicates (early-return path), which must still call resolveAllPendingFor + const collection = makeCollection(db) + await collection.stateWhenReady() + + const electronicsQuery = categoryQuery(collection, `electronics`) + const clothingQuery = categoryQuery(collection, `clothing`) + await electronicsQuery.preload() + await clothingQuery.preload() + + await vi.waitFor( + () => { + expect(collection.size).toBe(5) + }, + { timeout: 2000 }, + ) + + // Unload every predicate, then tear the collection down. + clothingQuery.cleanup() electronicsQuery.cleanup() + await vi.waitFor( + () => { + expect(collection.size).toBe(0) + }, + { timeout: 2000 }, + ) - // isPersisted.promise should resolve — if the bug is present, this hangs forever + // Allow any flush queued by the tracking table's onChange watcher to run. + await new Promise((resolve) => setTimeout(resolve, 200)) + + collection.cleanup() + await new Promise((resolve) => setTimeout(resolve, 200)) + + expect(syncErrors()).toEqual([]) + }) + + it(`should dispose each diff trigger exactly once`, async () => { + const db = await createDatabase() + await createTestProducts(db) + + // Count dispose calls per created trigger. The collection should release its + // reference to a trigger once disposed, so no trigger is disposed twice. + const disposeCounts: Array = [] + const createDiffTrigger = db.triggers.createDiffTrigger.bind(db.triggers) + vi.spyOn(db.triggers, `createDiffTrigger`).mockImplementation( + async (options) => { + const dispose = await createDiffTrigger(options) + const index = disposeCounts.push(0) - 1 + return async (disposeOptions) => { + disposeCounts[index]! += 1 + return dispose(disposeOptions) + } + }, + ) + + const collection = makeCollection(db) + await collection.stateWhenReady() + + const electronicsQuery = categoryQuery(collection, `electronics`) + await electronicsQuery.preload() await vi.waitFor( - async () => { - await insertResult.isPersisted.promise + () => { + expect(electronicsQuery.size).toBe(3) }, - { timeout: 5000 }, + { timeout: 2000 }, ) + + electronicsQuery.cleanup() + await vi.waitFor( + () => { + expect(collection.size).toBe(0) + }, + { timeout: 2000 }, + ) + + collection.cleanup() + await new Promise((resolve) => setTimeout(resolve, 200)) + + // One trigger is created for the electronics subset and disposed when that + // subset unloads. Cleaning up the collection must not dispose it again. + expect(disposeCounts).toEqual([1]) }) }) }) diff --git a/packages/powersync-db-collection/tests/powersync.test.ts b/packages/powersync-db-collection/tests/powersync.test.ts index ebf93223ff..adedda7988 100644 --- a/packages/powersync-db-collection/tests/powersync.test.ts +++ b/packages/powersync-db-collection/tests/powersync.test.ts @@ -17,7 +17,8 @@ import { describe, expect, it, onTestFinished, vi } from 'vitest' import { powerSyncCollectionOptions } from '../src' import { PowerSyncTransactor } from '../src/PowerSyncTransactor' import { TEST_DATABASE_IMPLEMENTATION } from './test-db-implementation' -import type { AbstractPowerSyncDatabase } from '@powersync/node' +import type { AbstractPowerSyncDatabase, LockContext } from '@powersync/node' +import type { PendingMutation } from '@tanstack/db' const APP_SCHEMA = new Schema({ users: new Table({ @@ -275,6 +276,70 @@ describePowerSync(`PowerSync Integration`, () => { ).true }) + it(`should complete transactions that delete one key and insert another (same-millisecond tie)`, async () => { + const db = await createDatabase() + + const options = powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.documents, + }) + const collection = createCollection(options) + onTestFinished(() => collection.cleanup()) + await collection.stateWhenReady() + + // Seed the row which will be deleted in the transaction + await collection.insert({ id: `a`, name: `row a` }).isPersisted.promise + + const { trackedTableName } = options.utils.getMeta() + + class SameMillisecondTransactor extends PowerSyncTransactor { + protected override async handleDelete( + mutation: PendingMutation, + context: LockContext, + waitForCompletion?: boolean, + ) { + const result = await super.handleDelete( + mutation, + context, + waitForCompletion, + ) + // Simulate the same-millisecond tie: ensure the delete's diff row + // wins the `ORDER BY timestamp DESC` readback of the insert below. + await context.execute( + `UPDATE ${trackedTableName} SET timestamp = '9999-12-31T23:59:59.999Z'`, + ) + return result + } + } + + const transactor = new SameMillisecondTransactor({ database: db }) + + const tx = createTransaction({ + autoCommit: false, + mutationFn: async ({ transaction }) => { + await transactor.applyTransaction(transaction) + }, + }) + + tx.mutate(() => { + collection.delete(`a`) + collection.insert({ id: `b`, name: `row b` }) + }) + + const outcome = await Promise.race([ + tx.commit().then(() => `persisted` as const), + new Promise<`timed out`>((resolve) => + setTimeout(() => resolve(`timed out`), 2_000), + ), + ]) + expect(outcome).toBe(`persisted`) + + const documents = await db.getAll<{ id: string }>( + `SELECT id FROM documents`, + ) + expect(documents.map((doc) => doc.id)).toEqual([`b`]) + }) + it(`should handle transactions with multiple collections`, async () => { const db = await createDatabase() await createTestData(db) diff --git a/packages/powersync-db-collection/tests/transactor-readiness.test.ts b/packages/powersync-db-collection/tests/transactor-readiness.test.ts new file mode 100644 index 0000000000..dad011bda3 --- /dev/null +++ b/packages/powersync-db-collection/tests/transactor-readiness.test.ts @@ -0,0 +1,69 @@ +import { createCollection, createTransaction } from '@tanstack/db' +import { expect, it, vi } from 'vitest' +import { PowerSyncTransactor } from '../src/PowerSyncTransactor' +import type { AbstractPowerSyncDatabase } from '@powersync/common' + +it.each([`cleanup`, `error`, `ready`] as const)( + `settles a transaction waiting for source readiness on %s`, + async (outcome) => { + const writeTransaction = vi + .fn() + .mockResolvedValue({ whenComplete: Promise.resolve() }) + // This boundary must settle before taking a database lock; no SQL runs. + const transactor = new PowerSyncTransactor({ + database: { writeTransaction } as unknown as AbstractPowerSyncDatabase, + }) + let markSourceReady!: () => void + const collection = createCollection<{ id: string }>({ + getKey: (row) => row.id, + sync: { + sync: ({ markReady }) => { + markSourceReady = markReady + return {} + }, + }, + }) + collection.startSyncImmediate() + const transaction = createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + transaction.mutate(() => collection.insert({ id: `pending` })) + let result: { error: unknown } | { ready: true } | undefined + const waiting = transactor.applyTransaction(transaction).then( + () => { + result = { ready: true } + }, + (error: unknown) => { + result = { error } + }, + ) + const failure = new Error(`source failed before readiness`) + try { + expect(collection.status).toBe(`loading`) + if (outcome === `cleanup`) await collection.cleanup() + else if (outcome === `error`) collection._lifecycle.markError(failure) + else markSourceReady() + // Drain promise reactions without waiting on the possibly orphaned wait. + for (let turn = 0; turn < 10; turn++) await Promise.resolve() + expect(result).toBeDefined() + expect(result).toEqual( + outcome === `ready` + ? { ready: true } + : { + error: + outcome === `error` + ? failure + : expect.objectContaining({ name: `AbortError` }), + }, + ) + await waiting + expect(writeTransaction).toHaveBeenCalledTimes( + outcome === `ready` ? 1 : 0, + ) + } finally { + transaction.rollback() + await collection.cleanup() + } + }, +) diff --git a/packages/powersync-db-collection/tests/upstream.config.ts b/packages/powersync-db-collection/tests/upstream.config.ts new file mode 100644 index 0000000000..2dd49b2631 --- /dev/null +++ b/packages/powersync-db-collection/tests/upstream.config.ts @@ -0,0 +1,12 @@ +import { defineConfig } from 'vitest/config' + +export default defineConfig({ + test: { + name: `powersync-upstream-contract-repros`, + dir: `./tests`, + include: [`**/*.repro.ts`], + environment: `node`, + coverage: { enabled: false }, + typecheck: { enabled: false }, + }, +}) diff --git a/packages/query-db-collection/CHANGELOG.md b/packages/query-db-collection/CHANGELOG.md index c64cc6fe15..569ca17c11 100644 --- a/packages/query-db-collection/CHANGELOG.md +++ b/packages/query-db-collection/CHANGELOG.md @@ -1,5 +1,248 @@ # @tanstack/query-db-collection +## 1.2.13 + +### Patch Changes + +- Fix on-demand load settlement, ordered pagination, and replay to preserve coherent results across cancellation, failure, cleanup, and restart. Preserve subset results and ownership across Electric, PowerSync, Query, and SQLite persistence adapters. Correct live-query grouping, include projections, value identity, and indexed comparisons. D2 hashing now rejects structural cycles and excessive traversal depth or work with an explicit error; Collection handles retain object-reference identity without traversing their mutable contents. ([#1797](https://github.com/TanStack/db/pull/1797)) + + Remove the unused public subset-algebra helpers: `isWhereSubset`, `unionWherePredicates`, `minusWherePredicates`, `isOrderBySubset`, `isLimitSubset`, `isOffsetLimitSubset`, `isPredicateSubset`, and `isLoadSubsetRequestSubsumedBy`. Apps that import these helpers must remove those imports; normal queries and adapters are unaffected. `DeduplicatedLoadSubset` remains available and shares only exact demand identities. + + Reject compiled Collection-valued includes as `fn.select()` inputs, including nested descendants, before invoking the callback. Use `toArray()` or `materialize()` in the upstream `.select()` for child-value calculations. To keep live child Collections, use expression `.select()` or perform parent-only functional work before adding the includes. Ordinary Collection-valued includes remain supported. + + Remove proxy `DEBUG` logging and automatic index timing statistics to avoid diagnostic work on reads and writes. Remove `getStats()` and `IndexStats`; use `index.keyCount` for the current entry count, and instrument index methods externally when profiling. Custom index subclasses must remove calls to the retired `trackLookup()` and `updateTimestamp()` helpers. + + Fix mutation drafts so Map/Set `forEach` calls each callback once and read-only iteration reports no changes. Track nested Map `for...of` edits through `collection.update()`. Keep Set entries in place during nested edits, preserving live iteration and usable `has`, `delete`, and `add` handles without duplicate entries or repeated visits. Reverting one entry no longer discards another entry's pending changes. + + Preserve shared values within a row's draft. Newly supplied objects use normal shared references during the update callback, including Map values and Set members; editing through either handle changes the same new object. Copy completed changes at callback return so later caller edits cannot mutate stored data. Existing collection values remain isolated. Copy a new caller-owned value before insertion if it must remain untouched during the callback. + + Keep rejected local-storage mutations out of later successful saves. Stage insert, update, delete, and manual transaction writes before persisting, then promote the shared cache only after storage succeeds. + + Deliver live-query observer publications to peer listeners even when one listener throws. Preserve queued publication order and stop delivery on disposal; report the first listener failure after delivery. + + Wait for PowerSync demands admitted during tracking finalization before reporting them loaded. Preserve untouched object and array key identity when they contain `NaN`. Retire every failed ordered acquisition on explicit retry, while retaining a full-source demand repaired by replay. + + After authoritative ordered recovery succeeds, retire settled page and tie demands so later truncate replay fetches only the full-source replacement. Keep unfinished requests observed until settlement and preserve independent query ownership. + + Remove the live-query `utils.getRunCount()` diagnostic and its runtime counter. Apps that call this diagnostic must remove those calls. Query scheduling and results are unchanged. + + Remove test-only index inspection getters (`indexedKeysSet`, `valueMapData`, `orderedEntriesArray`, and `orderedEntriesArrayReversed`) and unused scheduler diagnostics. `ReverseIndex` now exposes only the `IndexReader` lookup, range, and forward traversal surface returned by `findIndexForField`; mutate the original index instead. Export `IndexReader` for callers that name this return type. Remove the unused subscription `releaseLoadSubset()` method; request owners use the release callback supplied by `onLoadSubsetResult`. Ordinary query and adapter APIs remain unchanged by these removals. + + Remove unused internal helpers and the unused public error classes `WhereClauseConversionError`, `SubscriptionNotFoundError`, and `AggregateNotSupportedError`. These classes have no remaining runtime throw sites; remove any imports of them. Keep the existing query and index behavior and exercise identity/evaluation tests through the production entry points. + + Ensure failed mutations roll back even when their rejection value cannot be converted to a string. Preserve ordinary Error instances; report unprintable rejection values as `Unknown error`. + + Make reentrant effect disposal share the active cleanup result, including calls from abort listeners or source release callbacks. A release failure reaches every waiting disposer while each source still receives one release attempt. + + Reject starting or preloading a collection from inside its active cleanup callbacks with a clear `CollectionStateError`. Nested cleanup cannot admit replacement work that the old teardown would discard. Restart after cleanup completes, or from its final `cleaned-up` status event, remains supported. + + Prevent older page or tie-boundary completions from clearing a newer full-source failure or starting redundant loading. Failed window moves retain their settled public snapshot; an explicit retry releases the failed acquisition once and publishes the completed replacement. + + Restrict direct subscription `requestLimitedSnapshot()` cursor inputs to one order term and one `minValues` entry. Composite and partial-composite cursor inputs now throw before local delivery or adapter work. Use normal live-query ordering and window APIs for multi-column pagination; those remain supported through prefix-and-tie loading. Existing adapters need no changes. + + Treat `LoadSubsetOptions` and their nested request data as immutable from submission onward. Core no longer copies expression trees or mutable constant payloads at the sync and deduplication boundaries. Create a new Date, byte array, membership array, or options object when changing a demand instead of mutating submitted data. Adapters must also leave request data unchanged. Use stable data properties rather than stateful getters. `AbortSignal` cancellation and subscription release remain live. + + Replace replay acquisitions sequentially: release the prior physical lease before starting its replacement. The logical demand and last complete public result remain retained. A failed release prevents replacement startup; failed startup leaves demand available for a later authoritative replay. Custom adapters must support a release/load gap and preserve resources still held by other owners; a sole underlying resource may stop and restart. + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 1.2.12 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 1.2.11 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 1.2.10 + +### Patch Changes + +- Canonicalize equivalent loadSubset queries to one demand identity while preserving observable output aliases, exact projected values, and distinct ordered windows. Query DB now reuses the same canonical identity for its on-demand cache keys. ([#1768](https://github.com/TanStack/db/pull/1768)) + +- Settle subset loads only after their committed rows and events are visible. A ([#1769](https://github.com/TanStack/db/pull/1769)) + commit receipt now rejects with `AbortError` when cancellation wins before + application and ignores later aborts. Preserve causal publication, + cancellation, persistence, and error handling across the affected sync + adapters. +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 1.2.9 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 1.2.8 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 1.2.7 + +### Patch Changes + +- Propagate initial query sync failures through dependent live queries and readiness promises, including recovery and late subscribers, while preserving a ready cached snapshot on later refetch failures. Let sync adapters pass the original failure to `markError(error)` so readiness promises reject with that cause. Isolate adapter callbacks by sync session, preserve synchronous startup errors, and prevent rejected deduplicated subset requests from creating detached promise rejections. ([#1751](https://github.com/TanStack/db/pull/1751)) + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 1.2.6 + +### Patch Changes + +- Rebuild correlated include materialization as one D2 graph, fixing stale or missing nested results across route changes, batching, lazy loading, optimistic updates, and layered queries. Add canonical structural relation keys, abortable subset demand, and coherent publication for Collection-valued includes. Dispose delayed PowerSync subset hooks after cleanup, and prevent released Query Collection cache results from reaching the collection. ([#1740](https://github.com/TanStack/db/pull/1740)) + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 1.2.5 + +### Patch Changes + +- Add SSR through request-scoped `DbClient` instances, collection descriptors, ([#1564](https://github.com/TanStack/db/pull/1564)) + explicit collection-row hydration, live-query result snapshots, adapter sync + metadata, and React and Svelte descriptor resolution. + + React live queries now derive identity from structured query IR. Opaque queries + can provide `queryKey`; legacy dependency arrays and unkeyed opaque queries keep + working with development warnings until 1.0. + + Add TanStack Router integration that streams live queries discovered during a + Suspense render as pending promises which resolve to ordered result snapshots. + The browser starts normal source sync and atomically replaces the snapshot when + its live result is ready. + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 1.2.4 + +### Patch Changes + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 1.2.3 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 1.2.2 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 1.2.1 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 1.2.0 + +### Minor Changes + +- Add eager collection support for TanStack Query `initialData` and `initialDataUpdatedAt`, including wrapped response projection and collection-local initialization on shared QueryClient instances. ([#1683](https://github.com/TanStack/db/pull/1683)) + + QueryClient-default `placeholderData` no longer materializes as collection rows, and QueryClient-default `initialData` no longer seeds on-demand subset observers. + +### Patch Changes + +- Clean up empty query ownership state while preserving authoritative empty results and retained-row lifecycle behavior. ([#1672](https://github.com/TanStack/db/pull/1672)) + +## 1.1.0 + +### Minor Changes + +- Add top-level Query Collection support for additional Query observer options while preserving QueryClient defaultOptions behavior. ([#1665](https://github.com/TanStack/db/pull/1665)) + +### Patch Changes + +- Fix temporary query readiness listeners so subset unload and collection cleanup release them correctly during in-flight requests. ([#1673](https://github.com/TanStack/db/pull/1673)) + +- Extract internal query row ownership helpers to make lifecycle cleanup paths easier to reason about while preserving existing behavior. ([#1664](https://github.com/TanStack/db/pull/1664)) + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 1.0.48 + +### Patch Changes + +- Clarify that `select` extracts rows for DB materialization while preserving the wrapped TanStack Query cache response. ([#1654](https://github.com/TanStack/db/pull/1654)) + +- Document the current TanStack Query option compatibility surface for Query Collections, including forwarded options, QueryClient defaults, adapter-owned fields, and common options that are not currently exposed. ([#1653](https://github.com/TanStack/db/pull/1653)) + +- Add coverage for query invalidation behavior across eager and on-demand query collections. ([#1655](https://github.com/TanStack/db/pull/1655)) + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 1.0.47 + +### Patch Changes + +- Keep on-demand load subset subscription state out of TanStack Query metadata so dehydrated query state remains safe to persist with structured-clone based persisters. ([#1644](https://github.com/TanStack/db/pull/1644)) + +## 1.0.46 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 1.0.45 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 1.0.44 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 1.0.43 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 1.0.42 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 1.0.41 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 1.0.40 ### Patch Changes diff --git a/packages/query-db-collection/e2e/offline-refresh.e2e.test.ts b/packages/query-db-collection/e2e/offline-refresh.e2e.test.ts index ae25dd85fa..79e1e44f4c 100644 --- a/packages/query-db-collection/e2e/offline-refresh.e2e.test.ts +++ b/packages/query-db-collection/e2e/offline-refresh.e2e.test.ts @@ -6,7 +6,7 @@ */ import { describe, expect, it, vi } from 'vitest' -import { createCollection, BTreeIndex } from '@tanstack/db' +import { BTreeIndex, createCollection } from '@tanstack/db' import { QueryClient } from '@tanstack/query-core' import { startOfflineExecutor } from '@tanstack/offline-transactions' import { queryCollectionOptions } from '../src/query' diff --git a/packages/query-db-collection/e2e/query-filter.ts b/packages/query-db-collection/e2e/query-filter.ts index aa3de76b15..cf9ac16508 100644 --- a/packages/query-db-collection/e2e/query-filter.ts +++ b/packages/query-db-collection/e2e/query-filter.ts @@ -3,7 +3,7 @@ * Uses expression helpers to implement proper predicate push-down */ -import { parseLoadSubsetOptions } from '@tanstack/db' +import { getLoadSubsetDemandKey, parseLoadSubsetOptions } from '@tanstack/db' import type { IR, LoadSubsetOptions, @@ -41,117 +41,11 @@ export function buildQueryKey( namespace: string, options: LoadSubsetOptions | undefined, ) { - return [`e2e`, namespace, serializeLoadSubsetOptions(options)] -} - -export function serializeLoadSubsetOptions( - options: LoadSubsetOptions | undefined, -): unknown { - if (!options) { - return null - } - - const result: Record = {} - - if (options.where) { - result.where = serializeExpression(options.where) - } - - if (options.orderBy?.length) { - result.orderBy = options.orderBy.map((clause) => ({ - expression: serializeExpression(clause.expression), - direction: clause.compareOptions.direction, - nulls: clause.compareOptions.nulls, - })) - } - - if (options.limit !== undefined) { - result.limit = options.limit - } - - // Include offset for pagination support - different offsets need different query keys - if (options.offset !== undefined) { - result.offset = options.offset - } - - return JSON.stringify(Object.keys(result).length === 0 ? null : result) -} - -function serializeExpression(expr: IR.BasicExpression | undefined): unknown { - if (!expr) { - return null - } - - switch (expr.type) { - case `val`: - return { - type: `val`, - value: serializeValue(expr.value), - } - case `ref`: - return { - type: `ref`, - path: [...expr.path], - } - case `func`: - return { - type: `func`, - name: expr.name, - args: expr.args.map((arg) => serializeExpression(arg)), - } - default: - return null - } -} - -function serializeValue(value: unknown): unknown { - if (value === undefined) { - return { __type: `undefined` } - } - - if (typeof value === `number`) { - if (Number.isNaN(value)) { - return { __type: `nan` } - } - if (value === Number.POSITIVE_INFINITY) { - return { __type: `infinity`, sign: 1 } - } - if (value === Number.NEGATIVE_INFINITY) { - return { __type: `infinity`, sign: -1 } - } - } - - if (typeof value === `bigint`) { - return { __type: `bigint`, value: value.toString() } - } - - if ( - value === null || - typeof value === `string` || - typeof value === `number` || - typeof value === `boolean` - ) { - return value - } - - if (value instanceof Date) { - return { __type: `date`, value: value.toJSON() } - } - - if (Array.isArray(value)) { - return value.map((item) => serializeValue(item)) - } - - if (typeof value === `object`) { - return Object.fromEntries( - Object.entries(value as Record).map(([key, val]) => [ - key, - serializeValue(val), - ]), - ) - } - - return value + return [ + `e2e`, + namespace, + options === undefined ? undefined : getLoadSubsetDemandKey(options), + ] } type Predicate = (item: T) => boolean diff --git a/packages/query-db-collection/e2e/query.e2e.test.ts b/packages/query-db-collection/e2e/query.e2e.test.ts index 00863707d1..c25b113289 100644 --- a/packages/query-db-collection/e2e/query.e2e.test.ts +++ b/packages/query-db-collection/e2e/query.e2e.test.ts @@ -5,7 +5,7 @@ */ import { afterAll, afterEach, beforeAll, describe } from 'vitest' -import { createCollection, BTreeIndex } from '@tanstack/db' +import { BTreeIndex, createCollection } from '@tanstack/db' import { QueryClient } from '@tanstack/query-core' import { queryCollectionOptions } from '../src/query' import { diff --git a/packages/query-db-collection/package.json b/packages/query-db-collection/package.json index f3599c7a5e..acfe7b52f2 100644 --- a/packages/query-db-collection/package.json +++ b/packages/query-db-collection/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/query-db-collection", - "version": "1.0.40", + "version": "1.2.13", "description": "TanStack Query collection for TanStack DB", "author": "Kyle Mathews", "license": "MIT", @@ -21,6 +21,7 @@ "dev": "vite build --watch", "lint": "eslint . --fix", "test": "vitest run", + "test:oracles": "vitest run tests/includes-work-counter-oracle.test.ts tests/load-subset-lifecycle-oracle.test.ts tests/ownership-lifecycle.oracle.test.ts", "test:e2e": "vitest run --config vitest.e2e.config.ts" }, "type": "module", diff --git a/packages/query-db-collection/src/errors.ts b/packages/query-db-collection/src/errors.ts index 7224745255..2a226181bc 100644 --- a/packages/query-db-collection/src/errors.ts +++ b/packages/query-db-collection/src/errors.ts @@ -36,6 +36,15 @@ export class GetKeyRequiredError extends QueryCollectionError { } } +export class InitialDataInOnDemandModeError extends QueryCollectionError { + constructor() { + super( + `[QueryCollection] initialData and initialDataUpdatedAt are only supported when syncMode is 'eager'. Seed or hydrate the exact Query cache key for on-demand subsets instead.`, + ) + this.name = `InitialDataInOnDemandModeError` + } +} + export class SyncNotInitializedError extends QueryCollectionError { constructor() { super( diff --git a/packages/query-db-collection/src/manual-sync.ts b/packages/query-db-collection/src/manual-sync.ts index addf6808b2..ab61dd5eb6 100644 --- a/packages/query-db-collection/src/manual-sync.ts +++ b/packages/query-db-collection/src/manual-sync.ts @@ -5,7 +5,11 @@ import { UpdateOperationItemNotFoundError, } from './errors' import type { QueryClient } from '@tanstack/query-core' -import type { ChangeMessage, Collection } from '@tanstack/db' +import type { + ChangeMessage, + Collection, + SyncAppliedReceipt, +} from '@tanstack/db' // Track active batch operations per context to prevent cross-collection contamination const activeBatchContexts = new WeakMap< @@ -42,7 +46,7 @@ export interface SyncContext< */ begin: (options?: { immediate?: boolean }) => void write: (message: Omit, `key`>) => void - commit: () => void + commit: () => SyncAppliedReceipt /** * Optional function to update the query cache with the latest synced data. * Handles both direct array caches and wrapped response formats (when `select` is used). diff --git a/packages/query-db-collection/src/query.ts b/packages/query-db-collection/src/query.ts index 5012036127..620a3c5bb3 100644 --- a/packages/query-db-collection/src/query.ts +++ b/packages/query-db-collection/src/query.ts @@ -1,13 +1,18 @@ import { QueryObserver, hashKey } from '@tanstack/query-core' -import { deepEquals } from '@tanstack/db' +import { + LoadSubsetOperationAbortedError, + deepEquals, + getLoadSubsetDemandKey, + withCollectionConfigFactory, +} from '@tanstack/db' import { GetKeyRequiredError, + InitialDataInOnDemandModeError, QueryClientRequiredError, QueryFnRequiredError, QueryKeyRequiredError, } from './errors' import { createWriteUtils } from './manual-sync' -import { serializeLoadSubsetOptions } from './serialization' import type { BaseCollectionConfig, ChangeMessage, @@ -15,6 +20,7 @@ import type { DeleteMutationFnParams, InsertMutationFnParams, LoadSubsetOptions, + SyncAppliedReceipt, SyncConfig, SyncMetadataApi, UpdateMutationFnParams, @@ -49,6 +55,40 @@ type InferSchemaInput = T extends StandardSchemaV1 type TQueryKeyBuilder = (opts: LoadSubsetOptions) => TQueryKey +const queryObserverOptionKeys = [ + `enabled`, + `refetchInterval`, + `retry`, + `retryDelay`, + `staleTime`, + `gcTime`, + `refetchOnWindowFocus`, + `refetchOnReconnect`, + `refetchOnMount`, + `networkMode`, +] as const + +type QueryObserverOptionKey = (typeof queryObserverOptionKeys)[number] + +type QueryObserverOptionValues = Pick< + QueryObserverOptions, any, Array, Array, any>, + QueryObserverOptionKey +> + +function pickDefinedQueryObserverOptions( + config: Partial, +): Partial { + const options: Partial = {} + + for (const key of queryObserverOptionKeys) { + if (config[key] !== undefined) { + ;(options as Record)[key] = config[key] + } + } + + return options +} + /** * Configuration options for creating a Query Collection * @template T - The explicit type of items stored in the collection @@ -77,7 +117,10 @@ export interface QueryCollectionConfig< ) => Promise> | Array ? (context: QueryFunctionContext) => Promise> | Array : TQueryFn - /* Function that extracts array items from wrapped API responses (e.g metadata, pagination) */ + /** + * Extracts the row array TanStack DB materializes from the Query response. + * The Query cache keeps the original response shape. + */ select?: (data: TQueryData) => Array /** The TanStack Query client instance */ queryClient: QueryClient @@ -126,6 +169,54 @@ export interface QueryCollectionConfig< TQueryData, TQueryKey >[`gcTime`] + refetchOnWindowFocus?: QueryObserverOptions< + TQueryData, + TError, + Array, + TQueryData, + TQueryKey + >[`refetchOnWindowFocus`] + refetchOnReconnect?: QueryObserverOptions< + TQueryData, + TError, + Array, + TQueryData, + TQueryKey + >[`refetchOnReconnect`] + refetchOnMount?: QueryObserverOptions< + TQueryData, + TError, + Array, + TQueryData, + TQueryKey + >[`refetchOnMount`] + networkMode?: QueryObserverOptions< + TQueryData, + TError, + Array, + TQueryData, + TQueryKey + >[`networkMode`] + /** + * Data used to initialize the TanStack Query cache for an eager collection. + * The value has the original Query response shape and is projected through + * the collection's select option before rows are materialized. + */ + initialData?: QueryObserverOptions< + TQueryData, + TError, + Array, + TQueryData, + TQueryKey + >[`initialData`] + /** The timestamp TanStack Query uses to determine initialData freshness. */ + initialDataUpdatedAt?: QueryObserverOptions< + TQueryData, + TError, + Array, + TQueryData, + TQueryKey + >[`initialDataUpdatedAt`] persistedGcTime?: number /** @@ -349,6 +440,13 @@ class QueryCollectionUtilsImpl { } } +function getLoadSubsetOptionsForMeta( + opts: LoadSubsetOptions, +): Omit { + const { subscription: _subscription, ...serializableOptions } = opts + return serializableOptions +} + /** * Creates query collection options for use with a standard Collection. * This integrates TanStack Query with TanStack DB for automatic synchronization. @@ -586,6 +684,12 @@ export function queryCollectionOptions( retryDelay, staleTime, gcTime, + refetchOnWindowFocus, + refetchOnReconnect, + refetchOnMount, + networkMode, + initialData, + initialDataUpdatedAt, persistedGcTime, getKey, onInsert, @@ -598,6 +702,35 @@ export function queryCollectionOptions( // Default to eager sync mode if not provided const syncMode = baseCollectionConfig.syncMode ?? `eager` + if ( + syncMode === `on-demand` && + (initialData !== undefined || initialDataUpdatedAt !== undefined) + ) { + throw new InitialDataInOnDemandModeError() + } + + const initialDataObserverOptions = + syncMode === `eager` + ? { + ...(initialData !== undefined && { initialData }), + ...(initialDataUpdatedAt !== undefined && { + initialDataUpdatedAt, + }), + // Placeholder data is observer-local UI state in TanStack Query. It + // must not be exposed as collection-wide normalized rows, including + // when supplied through QueryClient defaults. + placeholderData: undefined, + } + : { + // A collection-wide initializer cannot establish membership for + // arbitrary on-demand subsets. A defined initializer is needed to + // override a QueryClient default because Query Core can apply + // defaults again while constructing each Query. + initialData: () => undefined, + initialDataUpdatedAt: undefined, + placeholderData: undefined, + } + // Compute the base query key once for cache lookups. // All derived keys (from on-demand predicates or function-based queryKey) must // share this prefix so that queryCache.findAll({ queryKey: baseKey }) can find them. @@ -661,7 +794,8 @@ export function queryCollectionOptions( // hashedQueryKey → queryKey const hashToQueryKey = new Map() - // queryKey → Set + // queryKey → Set. Entry presence means ownership is resolved; + // an empty set represents a resolved query that currently owns no rows. const queryToRows = new Map>() // RowKey → Set @@ -669,6 +803,7 @@ export function queryCollectionOptions( // queryKey → QueryObserver's unsubscribe function const unsubscribes = new Map void>() + const pendingReadyUnsubscribes = new Map void>>() // queryKey → reference count (how many loadSubset calls are active) // Reference counting for QueryObserver lifecycle management @@ -687,32 +822,72 @@ export function queryCollectionOptions( // 3. Decrements refcount and GCs rows where count reaches 0 const queryRefCounts = new Map() - // Helper function to add a row to the internal state - const addRow = (rowKey: string | number, hashedQueryKey: string) => { - const rowToQueriesSet = rowToQueries.get(rowKey) || new Set() - rowToQueriesSet.add(hashedQueryKey) - rowToQueries.set(rowKey, rowToQueriesSet) + // Eager startup holds one reference until cleanup. Cache removal detaches + // observation, not that ownership or its rows. + let ensureEagerSubscription = () => {} + + const addRowOwner = (rowKey: string | number, hashedQueryKey: string) => { + const owners = rowToQueries.get(rowKey) || new Set() + owners.add(hashedQueryKey) + rowToQueries.set(rowKey, owners) - const queryToRowsSet = queryToRows.get(hashedQueryKey) || new Set() - queryToRowsSet.add(rowKey) - queryToRows.set(hashedQueryKey, queryToRowsSet) + const ownedRows = + queryToRows.get(hashedQueryKey) || new Set() + ownedRows.add(rowKey) + queryToRows.set(hashedQueryKey, ownedRows) } - // Helper function to remove a row from the internal state - const removeRow = (rowKey: string | number, hashedQuerKey: string) => { - const rowToQueriesSet = rowToQueries.get(rowKey) || new Set() - rowToQueriesSet.delete(hashedQuerKey) - rowToQueries.set(rowKey, rowToQueriesSet) + const addRowOwners = (rowKey: string | number, owners: Set) => { + if (owners.size === 0) { + rowToQueries.delete(rowKey) + return + } - const queryToRowsSet = queryToRows.get(hashedQuerKey) || new Set() - queryToRowsSet.delete(rowKey) - queryToRows.set(hashedQuerKey, queryToRowsSet) + rowToQueries.set(rowKey, new Set(owners)) + owners.forEach((owner) => { + const ownedRows = queryToRows.get(owner) || new Set() + ownedRows.add(rowKey) + queryToRows.set(owner, ownedRows) + }) + } - return rowToQueriesSet.size === 0 + const removeRowOwner = (rowKey: string | number, hashedQueryKey: string) => { + const owners = rowToQueries.get(rowKey) + owners?.delete(hashedQueryKey) + if (!owners?.size) { + rowToQueries.delete(rowKey) + } + + const ownedRows = queryToRows.get(hashedQueryKey) + ownedRows?.delete(rowKey) + + return !owners?.size + } + + const removeQueryOwnership = (hashedQueryKey: string) => { + const nextOwnersByRow = new Map>() + + const rowKeys = + queryToRows.get(hashedQueryKey) ?? new Set() + + rowKeys.forEach((rowKey) => { + const owners = rowToQueries.get(rowKey) + + if (!owners) { + return + } + + const nextOwners = new Set(owners) + nextOwners.delete(hashedQueryKey) + nextOwnersByRow.set(rowKey, nextOwners) + }) + + return nextOwnersByRow } const internalSync: SyncConfig[`sync`] = (params) => { - const { begin, write, commit, markReady, collection, metadata } = params + const { begin, write, commit, markReady, markError, collection, metadata } = + params const persistedMetadata = metadata as | QuerySyncMetadataWithPersistedScan | undefined @@ -720,7 +895,12 @@ export function queryCollectionOptions( // Track whether sync has been started let syncStarted = false let startupRetentionSettled = false + const pendingStartupLoads = new Set() const retainedQueriesPendingRevalidation = new Set() + const pendingResultApplications = new Map>() + const failedResultApplications = new Map() + const resultApplicationTokens = new Map() + const resultApplicationControllers = new Map>() const effectivePersistedGcTimes = new Map() const persistedRetentionTimers = new Map< string, @@ -728,6 +908,29 @@ export function queryCollectionOptions( >() let persistedRetentionMaintenance = Promise.resolve() + const invalidatePendingResultApplication = (hashedQueryKey: string) => { + pendingResultApplications.delete(hashedQueryKey) + failedResultApplications.delete(hashedQueryKey) + resultApplicationTokens.delete(hashedQueryKey) + resultApplicationControllers + .get(hashedQueryKey) + ?.forEach((controller) => controller.abort()) + resultApplicationControllers.delete(hashedQueryKey) + } + + const getResultApplicationSettlement = ( + hashedQueryKey: string, + ): true | Promise => { + const pending = pendingResultApplications.get(hashedQueryKey) + if (pending) return pending + + if (failedResultApplications.has(hashedQueryKey)) { + return Promise.reject(failedResultApplications.get(hashedQueryKey)) + } + + return true + } + const getRowMetadata = (rowKey: string | number) => { return (metadata?.row.get(rowKey) ?? collection._state.syncedMetadata.get(rowKey)) as @@ -843,12 +1046,7 @@ export function queryCollectionOptions( continue } - rowToQueries.set(rowKey, new Set(owners)) - owners.forEach((owner) => { - const queryToRowsSet = queryToRows.get(owner) || new Set() - queryToRowsSet.add(rowKey) - queryToRows.set(owner, queryToRowsSet) - }) + addRowOwners(rowKey, owners) if (owners.has(hashedQueryKey)) { ownedRows.add(rowKey) @@ -934,12 +1132,7 @@ export function queryCollectionOptions( return } - rowToQueries.set(row.key, new Set(ownerSet)) - ownerSet.forEach((owner) => { - const queryToRowsSet = queryToRows.get(owner) || new Set() - queryToRowsSet.add(row.key) - queryToRows.set(owner, queryToRowsSet) - }) + addRowOwners(row.key, ownerSet) if (ownerSet.has(hashedQueryKey)) { baseline.set(row.key, { @@ -965,7 +1158,7 @@ export function queryCollectionOptions( baseline.forEach(({ value: oldItem, owners }, rowKey) => { owners.delete(hashedQueryKey) setPersistedOwners(rowKey, owners) - const needToRemove = removeRow(rowKey, hashedQueryKey) + const needToRemove = removeRowOwner(rowKey, hashedQueryKey) if (needToRemove) { rowsToDelete.push(oldItem) } @@ -979,6 +1172,7 @@ export function queryCollectionOptions( `${QUERY_COLLECTION_GC_PREFIX}${hashedQueryKey}`, ) commit() + queryToRows.delete(hashedQueryKey) } const schedulePersistedRetentionExpiry = ( @@ -1051,10 +1245,10 @@ export function queryCollectionOptions( // Function-based queryKey: use it to build the key from opts return queryKey(opts) } else if (syncMode === `on-demand`) { - // Static queryKey in on-demand mode: automatically append serialized predicates - // to create separate cache entries for different predicate combinations - const serialized = serializeLoadSubsetOptions(opts) - return serialized !== undefined ? [...queryKey, serialized] : queryKey + // A static on-demand key is extended by exact semantic demand so + // equivalent predicates share one entry while distinct windows do not. + const demandKey = getLoadSubsetDemandKey(opts) + return demandKey !== undefined ? [...queryKey, demandKey] : queryKey } else { // Static queryKey in eager mode: use as-is return queryKey @@ -1078,12 +1272,53 @@ export function queryCollectionOptions( } }) + const waitForQueryReady = ( + observer: QueryObserver, any, Array, Array, any>, + hashedQueryKey: string, + ): Promise => + new Promise((resolve, reject) => { + const unsubscribe = observer.subscribe((result) => { + // Use a microtask in case `subscribe` is called synchronously, before `unsubscribe` is initialized + queueMicrotask(() => { + if ( + (result.isSuccess && !collection.deferDataRefresh) || + result.isError + ) { + unsubscribe() + const pending = pendingReadyUnsubscribes.get(hashedQueryKey) + pending?.delete(cancel) + if (pending?.size === 0) { + pendingReadyUnsubscribes.delete(hashedQueryKey) + } + + if (result.isSuccess) { + resolve() + } else { + reject(result.error) + } + } + }) + }) + const cancel = () => { + unsubscribe() + reject(new LoadSubsetOperationAbortedError()) + } + const pending = + pendingReadyUnsubscribes.get(hashedQueryKey) ?? new Set() + pending.add(cancel) + pendingReadyUnsubscribes.set(hashedQueryKey, pending) + }) + const createQueryFromOpts = ( opts: LoadSubsetOptions = {}, queryFunction: typeof queryFn = queryFn, ): true | Promise => { if (!startupRetentionSettled) { + pendingStartupLoads.add(opts) return startupRetentionMaintenancePromise.then(() => { + if (!pendingStartupLoads.delete(opts)) { + throw new LoadSubsetOperationAbortedError() + } const resumed = createQueryFromOpts(opts, queryFunction) return resumed === true ? undefined : resumed }) @@ -1092,7 +1327,10 @@ export function queryCollectionOptions( // Generate key using common function const key = generateQueryKeyFromOptions(opts) const hashedQueryKey = hashKey(key) - const extendedMeta = { ...meta, loadSubsetOptions: opts } + const extendedMeta = { + ...meta, + loadSubsetOptions: getLoadSubsetOptionsForMeta(opts), + } const retainedEntry = metadata?.collection.get( `${QUERY_COLLECTION_GC_PREFIX}${hashedQueryKey}`, ) @@ -1119,35 +1357,20 @@ export function queryCollectionOptions( const currentResult = observer.getCurrentResult() if (currentResult.isSuccess) { - // Data is already available, return true synchronously - return true + if (collection.deferDataRefresh) { + return waitForQueryReady(observer, hashedQueryKey).then(() => { + const settlement = getResultApplicationSettlement(hashedQueryKey) + return settlement === true ? undefined : settlement + }) + } + return getResultApplicationSettlement(hashedQueryKey) } else if (currentResult.isError) { // Error already occurred, reject immediately return Promise.reject(currentResult.error) } else { - // Check QueryClient cache directly - observer's getCurrentResult() may show - // a loading state even when data exists in cache. This happens because observer - // state can lag behind the QueryClient cache during unsubscribe/resubscribe - // cycles (e.g., when a live query is cleaned up and recreated). - const cachedData = queryClient.getQueryData(key) - if (cachedData !== undefined) { - return true - } - - // Query is still loading, wait for the first result - return new Promise((resolve, reject) => { - const unsubscribe = observer.subscribe((result) => { - // Use a microtask in case `subscribe` is called synchronously, before `unsubscribe` is initialized - queueMicrotask(() => { - if (result.isSuccess) { - unsubscribe() - resolve() - } else if (result.isError) { - unsubscribe() - reject(result.error) - } - }) - }) + return waitForQueryReady(observer, hashedQueryKey).then(() => { + const settlement = getResultApplicationSettlement(hashedQueryKey) + return settlement === true ? undefined : settlement }) } } @@ -1159,19 +1382,24 @@ export function queryCollectionOptions( Array, any > = { + ...pickDefinedQueryObserverOptions({ + enabled, + refetchInterval, + retry, + retryDelay, + staleTime, + gcTime, + refetchOnWindowFocus, + refetchOnReconnect, + refetchOnMount, + networkMode, + }), + ...initialDataObserverOptions, queryKey: key, queryFn: queryFunction, meta: extendedMeta, structuralSharing: true, notifyOnChangeProps: `all`, - - // Only include options that are explicitly defined to allow QueryClient defaultOptions to be used - ...(enabled !== undefined && { enabled }), - ...(refetchInterval !== undefined && { refetchInterval }), - ...(retry !== undefined && { retry }), - ...(retryDelay !== undefined && { retryDelay }), - ...(staleTime !== undefined && { staleTime }), - ...(gcTime !== undefined && { gcTime }), } const localObserver = new QueryObserver< @@ -1210,24 +1438,17 @@ export function queryCollectionOptions( if (syncStarted || collection.subscriberCount > 0) { subscribeToQuery(localObserver, hashedQueryKey) } - return true + if (collection.deferDataRefresh) { + return waitForQueryReady(localObserver, hashedQueryKey).then(() => { + const settlement = getResultApplicationSettlement(hashedQueryKey) + return settlement === true ? undefined : settlement + }) + } + return getResultApplicationSettlement(hashedQueryKey) } // Create a promise that resolves when the query result is first available - const readyPromise = new Promise((resolve, reject) => { - const unsubscribe = localObserver.subscribe((result) => { - // Use a microtask in case `subscribe` is called synchronously, before `unsubscribe` is initialized - queueMicrotask(() => { - if (result.isSuccess) { - unsubscribe() - resolve() - } else if (result.isError) { - unsubscribe() - reject(result.error) - } - }) - }) - }) + const readyPromise = waitForQueryReady(localObserver, hashedQueryKey) // If sync has started or there are subscribers to the collection, subscribe to the query straight away // This creates the main subscription that handles data updates @@ -1235,12 +1456,15 @@ export function queryCollectionOptions( subscribeToQuery(localObserver, hashedQueryKey) } - return readyPromise + return readyPromise.then(() => { + const settlement = getResultApplicationSettlement(hashedQueryKey) + return settlement === true ? undefined : settlement + }) } type UpdateHandler = Parameters[0] - const applySuccessfulResult = ( + const applySuccessfulResult = async ( queryKey: QueryKey, result: QueryObserverResult, persistedBaseline?: Map< @@ -1250,17 +1474,14 @@ export function queryCollectionOptions( owners: Set } >, - ) => { + signal?: AbortSignal, + ): Promise => { const hashedQueryKey = hashKey(queryKey) - if (collection.status === `cleaned-up`) { + if (collection.status === `cleaned-up` || signal?.aborted) { return } - // Clear error state - state.lastError = undefined - state.errorCount = 0 - const rawData = result.data const newItemsArray = select ? select(rawData) : rawData @@ -1283,71 +1504,206 @@ export function queryCollectionOptions( const previouslyOwnedRows = shouldUsePersistedBaseline ? new Set(persistedBaseline.keys()) : getHydratedOwnedRowsForQueryBaseline(hashedQueryKey) + const newItemsMap = new Map() newItemsArray.forEach((item) => { const key = getKey(item) newItemsMap.set(key, item) }) - begin() - if (metadata) { - metadata.collection.delete( - `${QUERY_COLLECTION_GC_PREFIX}${hashedQueryKey}`, - ) + const previousOwnedRows = queryToRows.has(hashedQueryKey) + ? new Set(queryToRows.get(hashedQueryKey)) + : undefined + const affectedRowKeys = new Set([ + ...previouslyOwnedRows, + ...newItemsMap.keys(), + ]) + const previousOwnersByRow = new Map< + string | number, + Set | undefined + >() + affectedRowKeys.forEach((key) => { + const owners = rowToQueries.get(key) + previousOwnersByRow.set(key, owners ? new Set(owners) : undefined) + }) + let transactionActive = false + + const restoreOwnershipTracking = () => { + if (!state.observers.has(hashedQueryKey)) return + + if (previousOwnedRows === undefined) { + queryToRows.delete(hashedQueryKey) + } else { + queryToRows.set(hashedQueryKey, previousOwnedRows) + } + previousOwnersByRow.forEach((owners, key) => { + if (owners === undefined) { + rowToQueries.delete(key) + } else { + rowToQueries.set(key, owners) + } + }) } - previouslyOwnedRows.forEach((key) => { - const oldItem = shouldUsePersistedBaseline - ? persistedBaseline.get(key)?.value - : currentSyncedItems.get(key) - if (!oldItem) { - return + try { + // From this point onward the result, including an empty result, is the + // authoritative ownership baseline until this query is cleaned up. + queryToRows.set( + hashedQueryKey, + queryToRows.get(hashedQueryKey) ?? new Set(), + ) + + begin() + transactionActive = true + if (metadata) { + metadata.collection.delete( + `${QUERY_COLLECTION_GC_PREFIX}${hashedQueryKey}`, + ) } - const newItem = newItemsMap.get(key) - if (!newItem) { + + previouslyOwnedRows.forEach((key) => { + const oldItem = shouldUsePersistedBaseline + ? persistedBaseline.get(key)?.value + : currentSyncedItems.get(key) + if (!oldItem) { + return + } + const newItem = newItemsMap.get(key) + if (!newItem) { + const owners = getPersistedOwners(key) + owners.delete(hashedQueryKey) + setPersistedOwners(key, owners) + const needToRemove = removeRowOwner(key, hashedQueryKey) + if (needToRemove) { + write({ type: `delete`, value: oldItem }) + } + } else if (!deepEquals(oldItem, newItem)) { + write({ type: `update`, value: newItem }) + } + }) + + newItemsMap.forEach((newItem, key) => { const owners = getPersistedOwners(key) - owners.delete(hashedQueryKey) - setPersistedOwners(key, owners) - const needToRemove = removeRow(key, hashedQueryKey) - if (needToRemove) { - write({ type: `delete`, value: oldItem }) + const addsOwner = !owners.has(hashedQueryKey) + const insertsRow = !currentSyncedItems.has(key) + if (addsOwner) { + owners.add(hashedQueryKey) } - } else if (!deepEquals(oldItem, newItem)) { - write({ type: `update`, value: newItem }) - } - }) + addRowOwner(key, hashedQueryKey) + if (insertsRow) { + write({ type: `insert`, value: newItem }) + } + if (addsOwner || insertsRow) { + // An insert clears stale metadata for its key. Stage ownership + // afterward so rows and ownership commit as one state change. + setPersistedOwners(key, owners) + } + }) - newItemsMap.forEach((newItem, key) => { - const owners = getPersistedOwners(key) - if (!owners.has(hashedQueryKey)) { - owners.add(hashedQueryKey) - setPersistedOwners(key, owners) + const applied = commit(signal) + transactionActive = false + retainedQueriesPendingRevalidation.delete(hashedQueryKey) + cancelPersistedRetentionExpiry(hashedQueryKey) + + // Readiness is publication: do not expose it until the establishing + // transaction's rows and events are visible. + if (applied !== true) { + await applied } - addRow(key, hashedQueryKey) - if (!currentSyncedItems.has(key)) { - write({ type: `insert`, value: newItem }) + if (signal?.aborted) { + restoreOwnershipTracking() + return } - }) - - commit() - retainedQueriesPendingRevalidation.delete(hashedQueryKey) - cancelPersistedRetentionExpiry(hashedQueryKey) - - // Mark collection as ready after first successful query result - markReady() + markReady() + } catch (error) { + restoreOwnershipTracking() + + if (transactionActive) { + const cancellation = new AbortController() + cancellation.abort() + try { + commit(cancellation.signal) + } catch { + // Preserve the application error that caused the rollback. + } + } + throw error + } } const reconcileSuccessfulResult = async ( queryKey: QueryKey, result: QueryObserverResult, + applicationToken: object, + signal: AbortSignal, ) => { const hashedQueryKey = hashKey(queryKey) const persistedBaseline = await loadPersistedBaselineForQuery(hashedQueryKey) - if (collection.status === `cleaned-up`) { + if ( + collection.status === `cleaned-up` || + resultApplicationTokens.get(hashedQueryKey) !== applicationToken + ) { return } - applySuccessfulResult(queryKey, result, persistedBaseline) + await applySuccessfulResult(queryKey, result, persistedBaseline, signal) + } + + const trackResultApplication = ( + hashedQueryKey: string, + application: Promise, + ): void => { + pendingResultApplications.set(hashedQueryKey, application) + const finish = () => { + if (pendingResultApplications.get(hashedQueryKey) === application) { + pendingResultApplications.delete(hashedQueryKey) + return true + } + return false + } + void application.then( + () => { + if (finish()) failedResultApplications.delete(hashedQueryKey) + }, + (error) => { + if (!finish()) return + failedResultApplications.set(hashedQueryKey, error) + state.lastError = error + state.errorCount++ + state.lastErrorUpdatedAt = Date.now() + console.error( + `[QueryCollection] Error applying query ${String(hashToQueryKey.get(hashedQueryKey))}:`, + error, + ) + if (collection.status === `loading`) { + markError(error) + } + }, + ) + } + + const enqueueResultApplication = ( + hashedQueryKey: string, + apply: (signal: AbortSignal) => Promise, + ): void => { + const controller = new AbortController() + const controllers = + resultApplicationControllers.get(hashedQueryKey) ?? new Set() + controllers.add(controller) + resultApplicationControllers.set(hashedQueryKey, controllers) + const previousApplication = pendingResultApplications.get(hashedQueryKey) + const run = () => apply(controller.signal) + const application = previousApplication + ? previousApplication.then(run, run) + : run() + const cleanupController = () => { + controllers.delete(controller) + if (controllers.size === 0) { + resultApplicationControllers.delete(hashedQueryKey) + } + } + void application.then(cleanupController, cleanupController) + trackResultApplication(hashedQueryKey, application) } // eslint-disable-next-line no-shadow @@ -1355,6 +1711,11 @@ export function queryCollectionOptions( const hashedQueryKey = hashKey(queryKey) const handleQueryResult: UpdateHandler = (result) => { if (result.isSuccess) { + // Error state follows observer notification order, not the later + // publication time of a queued successful result. + state.lastError = undefined + state.errorCount = 0 + // Skip processing this result while data refreshes are deferred. // Optimistic state covers the gap. Once the barrier resolves, // trigger a fresh refetch to get authoritative data. @@ -1371,14 +1732,38 @@ export function queryCollectionOptions( } if (retainedQueriesPendingRevalidation.has(hashedQueryKey)) { - void reconcileSuccessfulResult(queryKey, result).catch((error) => { - console.error( - `[QueryCollection] Error reconciling query ${String(queryKey)}:`, - error, - ) + const query = queryClient.getQueryCache().find({ + queryKey, + exact: true, }) + if (query?.state.dataUpdateCount === 0) { + // Initial data seeds Query state without a fetch. It must not + // satisfy persistence retention that lasts until revalidation. + if (!result.isFetching) { + state.observers + .get(hashedQueryKey) + ?.refetch() + .catch(() => { + // Errors handled by the next handleQueryResult invocation + }) + } + return + } + + const applicationToken = {} + resultApplicationTokens.set(hashedQueryKey, applicationToken) + enqueueResultApplication(hashedQueryKey, (signal) => + reconcileSuccessfulResult( + queryKey, + result, + applicationToken, + signal, + ), + ) } else { - applySuccessfulResult(queryKey, result) + enqueueResultApplication(hashedQueryKey, (signal) => + applySuccessfulResult(queryKey, result, undefined, signal), + ) } } else if (result.isError) { const isNewError = @@ -1395,8 +1780,12 @@ export function queryCollectionOptions( result.error, ) - // Mark collection as ready even on error to avoid blocking apps - markReady() + // A failure before the first successful snapshot leaves no usable + // collection state. Later refetch failures keep the last ready + // snapshot available while utils expose the error. + if (collection.status === `loading`) { + markError(result.error) + } } } return handleQueryResult @@ -1411,6 +1800,9 @@ export function queryCollectionOptions( hashedQueryKey: string, ) => { if (!isSubscribed(hashedQueryKey)) { + // Cache removal does not retire eager ownership. Reattach the observer + // to the current cache entry before subscribing to its updates. + if (syncMode === `eager`) observer.setOptions(observer.options) const cachedQueryKey = hashToQueryKey.get(hashedQueryKey)! const handleQueryResult = makeQueryResultHandler(cachedQueryKey) const unsubscribeFn = observer.subscribe(handleQueryResult) @@ -1436,6 +1828,16 @@ export function queryCollectionOptions( unsubscribes.clear() } + ensureEagerSubscription = () => { + if (syncMode !== `eager`) return + state.observers.forEach((observer, key) => { + const query = observer.getCurrentQuery() + if (queryClient.getQueryCache().get(query.queryHash) !== query) { + subscribeToQuery(observer, key) + } + }) + } + // Mark that sync has started syncStarted = true @@ -1453,11 +1855,10 @@ export function queryCollectionOptions( // If syncMode is eager, create the initial query without any predicates if (syncMode === `eager`) { - // Catch any errors to prevent unhandled rejections - const initialResult = createQueryFromOpts({}) - if (initialResult instanceof Promise) { - initialResult.catch(() => { - // Errors are already handled by the query result handler + const result = createQueryFromOpts({}) + if (result instanceof Promise) { + void result.catch(() => { + // Errors are handled by the query result handler. }) } } else { @@ -1467,7 +1868,9 @@ export function queryCollectionOptions( // In on-demand mode, there is no initial query, but retained-placeholder // maintenance still needs to finish before the collection is treated as ready. void startupRetentionMaintenancePromise.then(() => { - markReady() + if (collection.status === `loading`) { + markReady() + } }) } } @@ -1487,27 +1890,25 @@ export function queryCollectionOptions( * Perform row-level cleanup and remove all tracking for a query. * Callers are responsible for ensuring the query is safe to cleanup. */ + const unsubscribePendingReadyListeners = (hashedQueryKey: string) => { + pendingReadyUnsubscribes.get(hashedQueryKey)?.forEach((unsubscribe) => { + unsubscribe() + }) + pendingReadyUnsubscribes.delete(hashedQueryKey) + } + const cleanupQueryInternal = (hashedQueryKey: string) => { unsubscribes.get(hashedQueryKey)?.() unsubscribes.delete(hashedQueryKey) + unsubscribePendingReadyListeners(hashedQueryKey) cancelPersistedRetentionExpiry(hashedQueryKey) retainedQueriesPendingRevalidation.delete(hashedQueryKey) + invalidatePendingResultApplication(hashedQueryKey) - const rowKeys = queryToRows.get(hashedQueryKey) ?? new Set() - const nextOwnersByRow = new Map>() + const nextOwnersByRow = removeQueryOwnership(hashedQueryKey) const rowsToDelete: Array = [] - rowKeys.forEach((rowKey) => { - const queries = rowToQueries.get(rowKey) - - if (!queries) { - return - } - - const nextOwners = new Set(queries) - nextOwners.delete(hashedQueryKey) - nextOwnersByRow.set(rowKey, nextOwners) - + nextOwnersByRow.forEach((nextOwners, rowKey) => { if (nextOwners.size === 0 && collection.has(rowKey)) { rowsToDelete.push(collection.get(rowKey)) } @@ -1515,7 +1916,11 @@ export function queryCollectionOptions( const shouldWriteMetadata = metadata !== undefined && nextOwnersByRow.size > 0 - const needsTransaction = shouldWriteMetadata || rowsToDelete.length > 0 + const retentionKey = `${QUERY_COLLECTION_GC_PREFIX}${hashedQueryKey}` + const hasRetentionMarker = + metadata?.collection.get(retentionKey) !== undefined + const needsTransaction = + shouldWriteMetadata || rowsToDelete.length > 0 || hasRetentionMarker if (needsTransaction) { begin() } @@ -1538,6 +1943,10 @@ export function queryCollectionOptions( }) } + if (hasRetentionMarker) { + metadata.collection.delete(retentionKey) + } + if (needsTransaction) { commit() } @@ -1563,6 +1972,13 @@ export function queryCollectionOptions( // Drop our subscription so hasListeners reflects only active consumers unsubscribes.get(hashedQueryKey)?.() unsubscribes.delete(hashedQueryKey) + unsubscribePendingReadyListeners(hashedQueryKey) + } + + // Refcounts are explicit ownership tokens. A cache event can remove the + // observer while an active acquisition still owns this query. + if (refcount > 0) { + return } const hasListeners = observer?.hasListeners() ?? false @@ -1574,21 +1990,12 @@ export function queryCollectionOptions( return } - // No listeners means the query is truly idle. - // Even if refcount > 0, we treat hasListeners as authoritative to prevent leaks. - // This can happen if subscriptions are GC'd without calling unloadSubset. - if (refcount > 0) { - console.warn( - `[cleanupQueryIfIdle] Invariant violation: refcount=${refcount} but no listeners. Cleaning up to prevent leak.`, - { hashedQueryKey }, - ) - } - if ( effectivePersistedGcTime !== undefined && metadata && persistedMetadata?.row.scanPersisted ) { + invalidatePendingResultApplication(hashedQueryKey) begin() metadata.collection.set( `${QUERY_COLLECTION_GC_PREFIX}${hashedQueryKey}`, @@ -1634,10 +2041,19 @@ export function queryCollectionOptions( const unsubscribeQueryCache = queryClient .getQueryCache() .subscribe((event) => { - const hashedKey = event.query.queryHash + // Ownership uses our stable key, not the Query client's optional + // custom cache hash function. + const hashedKey = hashKey(event.query.queryKey) if (event.type === `removed`) { // Only cleanup if this is OUR query (we track it) if (hashToQueryKey.has(hashedKey)) { + if (syncMode === `eager`) { + unsubscribes.get(hashedKey)?.() + unsubscribes.delete(hashedKey) + unsubscribePendingReadyListeners(hashedKey) + if (collection.subscriberCount > 0) ensureEagerSubscription() + return + } // TanStack Query GC'd this query after gcTime expired. // Use the guarded cleanup path to avoid deleting rows for active queries. cleanupQueryIfIdle(hashedKey) @@ -1645,7 +2061,9 @@ export function queryCollectionOptions( } }) - const cleanup = async () => { + const cleanup = () => { + pendingStartupLoads.clear() + ensureEagerSubscription = () => {} unsubscribeFromCollectionEvents() unsubscribeFromQueries() persistedRetentionTimers.forEach((timer) => { @@ -1653,8 +2071,10 @@ export function queryCollectionOptions( }) persistedRetentionTimers.clear() - const allQueryKeys = [...hashToQueryKey.values()] - const allHashedKeys = [...state.observers.keys()] + const allHashedKeys = new Set([ + ...state.observers.keys(), + ...queryToRows.keys(), + ]) // Force cleanup all queries (explicit cleanup path) // This ignores hasListeners and always cleans up @@ -1665,13 +2085,11 @@ export function queryCollectionOptions( // Unsubscribe from cache events (cleanup already happened above) unsubscribeQueryCache() - // Remove queries from TanStack Query cache - await Promise.all( - allQueryKeys.map(async (qKey) => { - await queryClient.cancelQueries({ queryKey: qKey, exact: true }) - queryClient.removeQueries({ queryKey: qKey, exact: true }) - }), - ) + // Removing a Query destroys it and synchronously cancels its retryer. + // Finish this before a later collection sync can create a replacement. + queryClient.removeQueries({ + predicate: (query) => allHashedKeys.has(hashKey(query.queryKey)), + }) } /** @@ -1699,6 +2117,8 @@ export function queryCollectionOptions( * by TanStack Query, allowing quick remounts to restore data without refetching. */ const unloadSubset = (options: LoadSubsetOptions) => { + // No observer lease exists until startup maintenance has finished. + if (pendingStartupLoads.delete(options)) return // 1. Same predicates → 2. Same queryKey const key = generateQueryKeyFromOptions(options) const hashedQueryKey = hashKey(key) @@ -1749,6 +2169,8 @@ export function queryCollectionOptions( * @returns Promise that resolves when the refetch is complete, with QueryObserverResult */ const refetch: RefetchFn = async (opts) => { + // An idle eager observer still owns rows; refetch must deliver its result. + ensureEagerSubscription() const allQueryKeys = [...hashToQueryKey.values()] const refetchPromises = allQueryKeys.map((qKey) => { const queryObserver = state.observers.get(hashKey(qKey))! @@ -1856,13 +2278,30 @@ export function queryCollectionOptions( getKey: (item: any) => string | number begin: () => void write: (message: Omit, `key`>) => void - commit: () => void + commit: () => SyncAppliedReceipt updateCacheData?: (items: Array) => void } | null = null // Enhanced internalSync that captures write functions for manual use const enhancedInternalSync: SyncConfig[`sync`] = (params) => { const { begin, write, commit, collection } = params + let queryClientMounted = false + + const mountQueryClient = () => { + if (!queryClientMounted) { + queryClient.mount() + queryClientMounted = true + } + } + + const unmountQueryClient = () => { + if (queryClientMounted) { + queryClient.unmount() + queryClientMounted = false + } + } + + mountQueryClient() // Get the base query key for the context (handle both static and function-based keys) const contextQueryKey = @@ -1882,8 +2321,27 @@ export function queryCollectionOptions( updateCacheData, } - // Call the original internalSync logic - return internalSync(params) + // Call the original internalSync logic, pairing QueryClient mount with the + // collection sync lifecycle so focus/reconnect managers dispatch events for + // standalone QueryClient usage. + const syncResult = internalSync(params) + const sync = + typeof syncResult === `function` + ? { cleanup: syncResult } + : typeof syncResult === `object` + ? syncResult + : {} + + return { + ...sync, + cleanup: () => { + try { + sync.cleanup?.() + } finally { + unmountQueryClient() + } + }, + } } // Create write utils using the manual-sync module @@ -1937,7 +2395,7 @@ export function queryCollectionOptions( // Create utils instance with state and dependencies passed explicitly const utils: any = new QueryCollectionUtilsImpl(state, refetch, writeUtils) - return { + const options = { ...baseCollectionConfig, getKey, syncMode, @@ -1947,4 +2405,16 @@ export function queryCollectionOptions( onDelete: wrappedOnDelete, utils, } + + return withCollectionConfigFactory( + options, + (client) => + queryCollectionOptions({ + ...config, + queryClient: + client.getDependency(`queryClient`) ?? + config.queryClient, + id: options.id, + }) as typeof options, + ) } diff --git a/packages/query-db-collection/src/serialization.ts b/packages/query-db-collection/src/serialization.ts deleted file mode 100644 index 9849c4bd33..0000000000 --- a/packages/query-db-collection/src/serialization.ts +++ /dev/null @@ -1,135 +0,0 @@ -import type { IR, LoadSubsetOptions } from '@tanstack/db' - -/** - * Serializes LoadSubsetOptions into a stable, hashable format for query keys. - * Includes where, orderBy, limit, and offset for pagination support. - * Note: cursor expressions are not serialized as they are backend-specific. - * @internal - */ -export function serializeLoadSubsetOptions( - options: LoadSubsetOptions | undefined, -): string | undefined { - if (!options) { - return undefined - } - - const result: Record = {} - - if (options.where) { - result.where = serializeExpression(options.where) - } - - if (options.orderBy?.length) { - result.orderBy = options.orderBy.map((clause) => { - const baseOrderBy = { - expression: serializeExpression(clause.expression), - direction: clause.compareOptions.direction, - nulls: clause.compareOptions.nulls, - stringSort: clause.compareOptions.stringSort, - } - - // Handle locale-specific options when stringSort is 'locale' - if (clause.compareOptions.stringSort === `locale`) { - return { - ...baseOrderBy, - locale: clause.compareOptions.locale, - localeOptions: clause.compareOptions.localeOptions, - } - } - - return baseOrderBy - }) - } - - if (options.limit !== undefined) { - result.limit = options.limit - } - - // Include offset for pagination support - if (options.offset !== undefined) { - result.offset = options.offset - } - - return Object.keys(result).length === 0 ? undefined : JSON.stringify(result) -} - -/** - * Recursively serializes an IR expression for stable hashing - * @internal - */ -function serializeExpression(expr: IR.BasicExpression | undefined): unknown { - if (!expr) { - return null - } - - switch (expr.type) { - case `val`: - return { - type: `val`, - value: serializeValue(expr.value), - } - case `ref`: - return { - type: `ref`, - path: [...expr.path], - } - case `func`: - return { - type: `func`, - name: expr.name, - args: expr.args.map((arg) => serializeExpression(arg)), - } - default: - return null - } -} - -/** - * Serializes special JavaScript values (undefined, NaN, Infinity, Date) - * @internal - */ -function serializeValue(value: unknown): unknown { - if (value === undefined) { - return { __type: `undefined` } - } - - if (typeof value === `number`) { - if (Number.isNaN(value)) { - return { __type: `nan` } - } - if (value === Number.POSITIVE_INFINITY) { - return { __type: `infinity`, sign: 1 } - } - if (value === Number.NEGATIVE_INFINITY) { - return { __type: `infinity`, sign: -1 } - } - } - - if ( - value === null || - typeof value === `string` || - typeof value === `number` || - typeof value === `boolean` - ) { - return value - } - - if (value instanceof Date) { - return { __type: `date`, value: value.toJSON() } - } - - if (Array.isArray(value)) { - return value.map((item) => serializeValue(item)) - } - - if (typeof value === `object`) { - return Object.fromEntries( - Object.entries(value as Record).map(([key, val]) => [ - key, - serializeValue(val), - ]), - ) - } - - return value -} diff --git a/packages/query-db-collection/tests/includes-work-counter-oracle.test.ts b/packages/query-db-collection/tests/includes-work-counter-oracle.test.ts new file mode 100644 index 0000000000..460f98549c --- /dev/null +++ b/packages/query-db-collection/tests/includes-work-counter-oracle.test.ts @@ -0,0 +1,345 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { QueryClient } from '@tanstack/query-core' +import { + BasicIndex, + createCollection, + createLiveQueryCollection, + eq, +} from '@tanstack/db' +import { describe, expect, it } from 'vitest' +import { queryCollectionOptions } from '../src/query' +import type { Collection } from '@tanstack/db' + +let nextCollectionId = 0 + +type RootRow = { id: string; value: string } +type BranchRow = { id: string; parentId: string; value: string } +type TwigRow = { id: string; parentId: string; value: string } +type LeafRow = { id: string; parentId: string; value: string } + +type TreeCounts = { + roots: number + branches: number + twigs: number + leaves: number +} + +type ChildLevel = Exclude + +type NodeRow = { + id: string + value: string + children?: NodeCollection +} + +type NodeCollection = Pick< + Collection, + 'cleanup' | 'preload' | 'size' | 'toArray' +> + +type NestedTreeShape = { + reachableTreeRows: number + sourceRowsDeliveredAtPreload: TreeCounts + sourceRowsDeliveredAfterTraversal: TreeCounts + reachableChildCollections: Record + reachableChildRows: Record +} + +const branchesPerRoot = 2 +const twigsPerBranch = 5 +const leavesPerTwig = 10 + +function createNestedTreeRows(rootCount: number) { + const roots: Array = [] + const branches: Array = [] + const twigs: Array = [] + const leaves: Array = [] + + for (let rootIndex = 0; rootIndex < rootCount; rootIndex++) { + const rootId = `root-${rootIndex}` + roots.push({ id: rootId, value: rootId }) + + for (let branchIndex = 0; branchIndex < branchesPerRoot; branchIndex++) { + const branchId = `branch-${rootIndex}-${branchIndex}` + branches.push({ id: branchId, parentId: rootId, value: branchId }) + + for (let twigIndex = 0; twigIndex < twigsPerBranch; twigIndex++) { + const twigId = `twig-${rootIndex}-${branchIndex}-${twigIndex}` + twigs.push({ id: twigId, parentId: branchId, value: twigId }) + + for (let leafIndex = 0; leafIndex < leavesPerTwig; leafIndex++) { + const leafId = `leaf-${rootIndex}-${branchIndex}-${twigIndex}-${leafIndex}` + leaves.push({ id: leafId, parentId: twigId, value: leafId }) + } + } + } + } + + return { roots, branches, twigs, leaves } +} + +function createQuerySource( + name: string, + rows: Array, + queryClient: QueryClient, +) { + const id = `${name}-${nextCollectionId++}` + return createCollection( + queryCollectionOptions({ + id, + queryClient, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + queryKey: [id], + queryFn: () => Promise.resolve(rows), + getKey: (row) => row.id, + }), + ) +} + +function countDeliveredRows(collection: Collection) { + let deliveredRows = 0 + const originalSubscribeChanges = collection.subscribeChanges.bind(collection) + + collection.subscribeChanges = (callback, options) => { + return originalSubscribeChanges((changes) => { + deliveredRows += changes.length + callback(changes) + }, options) + } + + return () => deliveredRows +} + +function expectedTreeCounts(rootCount: number): TreeCounts { + const branches = rootCount * branchesPerRoot + const twigs = branches * twigsPerBranch + + return { + roots: rootCount, + branches, + twigs, + leaves: twigs * leavesPerTwig, + } +} + +function requireChildren(row: NodeRow, level: string): NodeCollection { + if (row.children === undefined) { + throw new Error(`Expected ${level} row ${row.id} to have children`) + } + return row.children +} + +function observeReachableTreeShape(roots: NodeCollection) { + const reachableChildCollections = { branches: 0, twigs: 0, leaves: 0 } + const reachableChildRows = { branches: 0, twigs: 0, leaves: 0 } + let reachableTreeRows = roots.size + + const countChildren = ( + collection: NodeCollection, + level: ChildLevel, + ): void => { + const children = collection.toArray + reachableChildCollections[level]++ + reachableChildRows[level] += children.length + reachableTreeRows += children.length + + const nextLevel = + level === `branches` ? `twigs` : level === `twigs` ? `leaves` : undefined + if (nextLevel === undefined) return + + for (const child of children) { + countChildren(requireChildren(child, level), nextLevel) + } + } + + for (const root of roots.toArray) { + countChildren(requireChildren(root, `root`), `branches`) + } + + return { reachableTreeRows, reachableChildCollections, reachableChildRows } +} + +function snapshotSourceRowsDelivered( + sourceCounters: Record number>, +): TreeCounts { + return { + roots: sourceCounters.roots(), + branches: sourceCounters.branches(), + twigs: sourceCounters.twigs(), + leaves: sourceCounters.leaves(), + } +} + +async function runCleanups( + cleanups: ReadonlyArray<() => void | Promise>, +): Promise { + const results = await Promise.allSettled( + cleanups.map(async (cleanup) => cleanup()), + ) + const firstRejection = results.find( + (result): result is PromiseRejectedResult => result.status === `rejected`, + ) + if (firstRejection !== undefined) throw firstRejection.reason +} + +function rethrowFirstCleanupError( + results: ReadonlyArray<{ rejected: boolean; error: unknown }>, +): void { + const firstRejection = results.find((result) => result.rejected) + if (firstRejection !== undefined) throw firstRejection.error +} + +async function observeNestedTreeShape( + rootCount: number, +): Promise { + const rows = createNestedTreeRows(rootCount) + const queryClient = new QueryClient({ + defaultOptions: { queries: { retry: false } }, + }) + const sources = { + roots: createQuerySource(`tree-roots`, rows.roots, queryClient), + branches: createQuerySource(`tree-branches`, rows.branches, queryClient), + twigs: createQuerySource(`tree-twigs`, rows.twigs, queryClient), + leaves: createQuerySource(`tree-leaves`, rows.leaves, queryClient), + } + let rootCollection: NodeCollection | undefined + + try { + const sourceCounters = { + roots: countDeliveredRows(sources.roots), + branches: countDeliveredRows(sources.branches), + twigs: countDeliveredRows(sources.twigs), + leaves: countDeliveredRows(sources.leaves), + } + + // This matches the nested result shape in #1634. Each children property + // remains a live Collection. The public result API exposes the reachable + // tree, not internal allocation counts, so this oracle constrains reachable + // cardinality and source delivery rather than claiming to count allocations. + const roots: NodeCollection = createLiveQueryCollection({ + startSync: true, + query: (q) => + q.from({ root: sources.roots }).select(({ root }) => ({ + id: root.id, + value: root.value, + children: q + .from({ branch: sources.branches }) + .where(({ branch }) => eq(branch.parentId, root.id)) + .select(({ branch }) => ({ + id: branch.id, + value: branch.value, + children: q + .from({ twig: sources.twigs }) + .where(({ twig }) => eq(twig.parentId, branch.id)) + .select(({ twig }) => ({ + id: twig.id, + value: twig.value, + children: q + .from({ leaf: sources.leaves }) + .where(({ leaf }) => eq(leaf.parentId, twig.id)) + .select(({ leaf }) => ({ + id: leaf.id, + value: leaf.value, + })), + })), + })), + })), + }) + rootCollection = roots + await roots.preload() + + const sourceRowsDeliveredAtPreload = + snapshotSourceRowsDelivered(sourceCounters) + const { reachableTreeRows, reachableChildCollections, reachableChildRows } = + observeReachableTreeShape(roots) + + return { + reachableTreeRows, + sourceRowsDeliveredAtPreload, + sourceRowsDeliveredAfterTraversal: + snapshotSourceRowsDelivered(sourceCounters), + reachableChildCollections, + reachableChildRows, + } + } finally { + let cleanupRejected = false + let cleanupError: unknown + try { + await runCleanups([ + async () => rootCollection?.cleanup(), + ...Object.values(sources).map((source) => async () => source.cleanup()), + ]) + } catch (error) { + cleanupRejected = true + cleanupError = error + } + + let clearRejected = false + let clearError: unknown + try { + queryClient.clear() + } catch (error) { + clearRejected = true + clearError = error + } + + rethrowFirstCleanupError([ + { rejected: cleanupRejected, error: cleanupError }, + { rejected: clearRejected, error: clearError }, + ]) + } +} + +function expectNestedTreeShape( + observation: NestedTreeShape, + rootCount: number, +): void { + const expected = expectedTreeCounts(rootCount) + expect(observation.reachableTreeRows).toBe( + Object.values(expected).reduce((sum, count) => sum + count, 0), + ) + expect(observation.sourceRowsDeliveredAtPreload).toEqual(expected) + expect(observation.sourceRowsDeliveredAfterTraversal).toEqual( + observation.sourceRowsDeliveredAtPreload, + ) + expect(observation.reachableChildCollections).toEqual({ + branches: expected.roots, + twigs: expected.branches, + leaves: expected.twigs, + }) + expect(observation.reachableChildRows).toEqual({ + branches: expected.branches, + twigs: expected.twigs, + leaves: expected.leaves, + }) +} + +describe(`nested includes reachable-shape oracle`, () => { + fcTest.prop([fc.integer({ min: 0, max: 20 })], { + numRuns: 6, + seed: 1634, + })( + `preserves the complete reachable nested tree shape (#1634)`, + async (rootCount) => { + expectNestedTreeShape(await observeNestedTreeShape(rootCount), rootCount) + }, + ) + + it(`exposes no nested collections for an empty root query`, async () => { + expectNestedTreeShape(await observeNestedTreeShape(0), 0) + }) + + it(`pins #1634's reported 20-by-2-by-5-by-10 tree`, async () => { + // These semantic counters do not claim to measure elapsed time or internal + // allocations. They pin source delivery at preload and reachable shape. + expectNestedTreeShape(await observeNestedTreeShape(20), 20) + }) + + it(`does not deliver more source rows while traversing the result`, async () => { + const observation = await observeNestedTreeShape(20) + expect(observation.sourceRowsDeliveredAfterTraversal).toEqual( + observation.sourceRowsDeliveredAtPreload, + ) + }) +}) diff --git a/packages/query-db-collection/tests/load-subset-lifecycle-oracle.test.ts b/packages/query-db-collection/tests/load-subset-lifecycle-oracle.test.ts new file mode 100644 index 0000000000..b7d4b395ad --- /dev/null +++ b/packages/query-db-collection/tests/load-subset-lifecycle-oracle.test.ts @@ -0,0 +1,602 @@ +import { QueryClient } from '@tanstack/query-core' +import { + BasicIndex, + IR, + createCollection, + createLiveQueryCollection, + eq, +} from '@tanstack/db' +import { describe, expect, it, vi } from 'vitest' +import { TraceAssertionError } from '../../db/tests/trace-runner.js' +import { queryCollectionOptions } from '../src/query.js' +import type { QueryFunctionContext } from '@tanstack/query-core' +import type { SyncMetadataApi } from '@tanstack/db' + +type Row = { + id: string + group?: string +} + +let collectionSequence = 0 + +function createQueryClient(): QueryClient { + return new QueryClient({ + defaultOptions: { + queries: { + gcTime: Number.POSITIVE_INFINITY, + retry: false, + staleTime: Number.POSITIVE_INFINITY, + }, + }, + }) +} + +async function expectInitialQueryFailureStatus(): Promise { + const error = new Error(`initial query failed`) + const queryClient = createQueryClient() + const id = `load-subset-error-status-${collectionSequence++}` + const loggedError = vi.spyOn(console, `error`).mockImplementation(() => {}) + const queryFn = vi + .fn() + .mockRejectedValueOnce(error) + .mockResolvedValueOnce([{ id: `recovered` }]) + const collection = createCollection( + queryCollectionOptions({ + id, + queryClient, + queryKey: [id], + queryFn, + getKey: (row) => row.id, + startSync: true, + retry: false, + }), + ) + const live = createLiveQueryCollection((query) => + query.from({ row: collection }).select(({ row }) => ({ id: row.id })), + ) + const preloadOutcomes = Promise.allSettled([ + collection.preload(), + live.preload(), + ]) + + try { + await vi.waitFor(() => { + expect(collection.utils.lastError).toBe(error) + expect(collection.utils.isError).toBe(true) + }) + expect(loggedError).toHaveBeenCalled() + try { + expect(collection.status).toBe(`error`) + expect(live.status).toBe(`error`) + expect((await preloadOutcomes).map(({ status }) => status)).toEqual([ + `rejected`, + `rejected`, + ]) + } catch (caught) { + throw new TraceAssertionError(0, caught) + } + + await collection.utils.clearError() + await vi.waitFor(() => { + expect(collection.status).toBe(`ready`) + expect(collection.get(`recovered`)).toBeDefined() + expect(live.status).toBe(`ready`) + expect(live.get(`recovered`)).toBeDefined() + }) + await expect(collection.preload()).resolves.toBeUndefined() + await expect(live.preload()).resolves.toBeUndefined() + } finally { + await live.cleanup() + await collection.cleanup() + queryClient.clear() + loggedError.mockRestore() + } +} + +async function expectLateDependentObservesInitialFailure(): Promise { + const error = new Error(`source failed before dependent construction`) + const queryClient = createQueryClient() + const id = `load-subset-late-dependent-error-${collectionSequence++}` + const loggedError = vi.spyOn(console, `error`).mockImplementation(() => {}) + const collection = createCollection( + queryCollectionOptions({ + id, + queryClient, + queryKey: [id], + queryFn: vi.fn().mockRejectedValue(error), + getKey: (row) => row.id, + startSync: true, + retry: false, + }), + ) + + await expect(collection.preload()).rejects.toBe(error) + expect(collection.status).toBe(`error`) + + const live = createLiveQueryCollection((query) => + query.from({ row: collection }).select(({ row }) => ({ id: row.id })), + ) + const livePreload = live.preload() + void livePreload.catch(() => undefined) + + try { + expect(live.status).toBe(`error`) + await expect(livePreload).rejects.toThrow() + } finally { + await live.cleanup() + await collection.cleanup() + await Promise.allSettled([livePreload]) + queryClient.clear() + loggedError.mockRestore() + } +} + +async function expectEveryFailedSourceToRecover(): Promise { + const createControlledSource = (id: string) => { + let fail: () => void = () => { + throw new Error(`Source '${id}' has not started`) + } + let recover: (row: Row) => void = (_row) => { + throw new Error(`Source '${id}' has not started`) + } + const collection = createCollection({ + id, + getKey: (row) => row.id, + startSync: false, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: ({ begin, write, commit, markReady, markError }) => { + fail = markError + recover = (row) => { + begin() + write({ type: `insert`, value: row }) + commit() + markReady() + } + }, + }, + }) + return { + collection, + fail: () => fail(), + recover: (row: Row) => recover(row), + } + } + + const left = createControlledSource( + `load-subset-multi-error-left-${collectionSequence++}`, + ) + const right = createControlledSource( + `load-subset-multi-error-right-${collectionSequence++}`, + ) + const loggedError = vi.spyOn(console, `error`).mockImplementation(() => {}) + const live = createLiveQueryCollection((query) => + query + .from({ left: left.collection }) + .join({ right: right.collection }, ({ left: leftRow, right: rightRow }) => + eq(leftRow.id, rightRow.id), + ) + .select(({ left: row }) => ({ id: row.id })), + ) + const preload = live.preload() + void preload.catch(() => undefined) + + try { + left.fail() + right.fail() + await expect(preload).rejects.toThrow() + expect(live.status).toBe(`error`) + + left.recover({ id: `shared` }) + expect(left.collection.status).toBe(`ready`) + expect(right.collection.status).toBe(`error`) + expect(live.status).toBe(`error`) + + right.recover({ id: `shared` }) + await expect(live.preload()).resolves.toBeUndefined() + expect(live.status).toBe(`ready`) + expect(live.toArray.map((row) => row.id)).toEqual([`shared`]) + } finally { + await live.cleanup() + await left.collection.cleanup() + await right.collection.cleanup() + await Promise.allSettled([preload]) + loggedError.mockRestore() + } +} + +async function expectRefetchFailureKeepsReadySnapshot(): Promise { + const error = new Error(`refetch failed`) + const queryClient = createQueryClient() + const id = `load-subset-refetch-status-${collectionSequence++}` + const loggedError = vi.spyOn(console, `error`).mockImplementation(() => {}) + const queryFn = vi + .fn() + .mockResolvedValueOnce([{ id: `cached` }]) + .mockRejectedValueOnce(error) + const collection = createCollection( + queryCollectionOptions({ + id, + queryClient, + queryKey: [id], + queryFn, + getKey: (row) => row.id, + startSync: true, + retry: false, + }), + ) + const live = createLiveQueryCollection((query) => + query.from({ row: collection }).select(({ row }) => ({ id: row.id })), + ) + + try { + await live.preload() + await collection.utils.refetch() + await vi.waitFor(() => { + expect(collection.utils.lastError).toBe(error) + }) + expect(collection.status).toBe(`ready`) + expect(live.status).toBe(`ready`) + expect(collection.get(`cached`)).toBeDefined() + expect(live.get(`cached`)).toBeDefined() + } finally { + await live.cleanup() + await collection.cleanup() + queryClient.clear() + loggedError.mockRestore() + } +} + +async function expectDeferredStartupReadyDoesNotOverrideError(): Promise { + const loggedError = vi.spyOn(console, `error`).mockImplementation(() => {}) + const queryClient = createQueryClient() + const id = `load-subset-deferred-ready-${collectionSequence++}` + const queryError = new Error(`cached observer failed`) + const queryFn = vi.fn().mockRejectedValue(queryError) + const baseOptions = queryCollectionOptions({ + id, + queryClient, + queryKey: [id], + queryFn, + getKey: (row) => row.id, + startSync: true, + syncMode: `on-demand`, + retry: false, + }) + const originalSync = baseOptions.sync + let syncParams!: Parameters[0] + const collection = createCollection({ + ...baseOptions, + sync: { + sync: (params) => { + syncParams = params + return originalSync.sync(params) + }, + }, + }) + + const firstLoad = collection._sync.loadSubset({}) + if (!(firstLoad instanceof Promise)) { + throw new Error(`The failing query must be asynchronous`) + } + await expect(firstLoad).rejects.toBe(queryError) + expect(collection.status).toBe(`ready`) + + let releaseScan!: () => void + const scanReleased = new Promise((resolve) => { + releaseScan = resolve + }) + let resolveMaintenanceDelete!: () => void + const maintenanceDeleted = new Promise((resolve) => { + resolveMaintenanceDelete = resolve + }) + type MetadataWithPersistedScan = SyncMetadataApi & { + row: SyncMetadataApi[`row`] & { + scanPersisted: () => Promise< + Array<{ key: string | number; value: Row; metadata?: unknown }> + > + } + } + const metadata: MetadataWithPersistedScan = { + row: { + get: () => undefined, + set: () => {}, + delete: () => {}, + scanPersisted: async () => { + await scanReleased + return [] + }, + }, + collection: { + get: () => undefined, + set: () => {}, + delete: () => { + resolveMaintenanceDelete() + }, + list: () => [ + { + key: `queryCollection:gc:expired`, + value: { queryHash: `expired`, mode: `ttl`, expiresAt: 0 }, + }, + ], + }, + } + + collection._lifecycle.setStatus(`cleaned-up`) + collection._lifecycle.setStatus(`loading`) + const secondSync = originalSync.sync({ ...syncParams, metadata }) + + try { + expect(collection.status).toBe(`error`) + releaseScan() + await maintenanceDeleted + for (let turn = 0; turn < 10; turn++) await Promise.resolve() + expect(collection.status).toBe(`error`) + expect(collection.utils.lastError).toBe(queryError) + } finally { + if (typeof secondSync === `function`) { + await secondSync() + } else { + await secondSync?.cleanup?.() + } + await collection.cleanup() + queryClient.clear() + loggedError.mockRestore() + } +} + +async function expectEquivalentPredicatesShareOneLoad( + form: `commutative-and` | `commutative-or` | `reversed-equality`, +): Promise { + const { queryClient, collection, queryFn } = createOnDemandCollection( + `load-subset-canonical-predicate`, + [{ id: `a`, group: `x` }], + ) + const firstComparison = new IR.Func(`eq`, [ + new IR.PropRef([`id`]), + new IR.Value(`a`), + ]) + const secondComparison = new IR.Func(`eq`, [ + new IR.PropRef([`group`]), + new IR.Value(`x`), + ]) + let first: IR.BasicExpression + let second: IR.BasicExpression + switch (form) { + case `commutative-and`: + first = new IR.Func(`and`, [firstComparison, secondComparison]) + second = new IR.Func(`and`, [secondComparison, firstComparison]) + break + case `commutative-or`: + first = new IR.Func(`or`, [firstComparison, secondComparison]) + second = new IR.Func(`or`, [secondComparison, firstComparison]) + break + case `reversed-equality`: + first = firstComparison + second = new IR.Func(`eq`, [new IR.Value(`a`), new IR.PropRef([`id`])]) + break + } + + try { + await collection._sync.loadSubset({ where: first }) + await collection._sync.loadSubset({ where: second }) + try { + expect(queryFn.mock.calls.length).toBe(1) + } catch (error) { + throw new TraceAssertionError(0, error) + } + } finally { + await collection.cleanup() + queryClient.clear() + } +} + +async function expectEquivalentComparisonValuesShareOneLoad( + firstValue: unknown, + secondValue: unknown, +): Promise { + const { queryClient, collection, queryFn } = createOnDemandCollection( + `load-subset-comparison-value`, + [{ id: `a` }], + ) + const value = new IR.PropRef([`value`]) + + try { + await collection._sync.loadSubset({ + where: new IR.Func(`eq`, [value, new IR.Value(firstValue)]), + }) + await collection._sync.loadSubset({ + where: new IR.Func(`eq`, [value, new IR.Value(secondValue)]), + }) + expect(queryFn).toHaveBeenCalledOnce() + } finally { + await collection.cleanup() + queryClient.clear() + } +} + +function createOnDemandCollection(idPrefix: string, rows: Array) { + const queryClient = createQueryClient() + const id = `${idPrefix}-${collectionSequence++}` + const queryFn = vi.fn().mockResolvedValue(rows) + const collection = createCollection( + queryCollectionOptions({ + id, + queryClient, + queryKey: [id], + queryFn, + getKey: (row) => row.id, + startSync: true, + syncMode: `on-demand`, + retry: false, + }), + ) + return { queryClient, collection, queryFn } +} + +async function expectFinalOwnerCleanupAbortsQuery(): Promise { + const queryClient = createQueryClient() + const id = `load-subset-cancel-final-owner-${collectionSequence++}` + let capturedSignal: AbortSignal | undefined + let resolveStarted!: () => void + const started = new Promise((resolve) => { + resolveStarted = resolve + }) + const queryFn = vi.fn((context: QueryFunctionContext) => { + capturedSignal = context.signal + resolveStarted() + return new Promise>((_resolve, reject) => { + context.signal.addEventListener(`abort`, () => { + const error = new Error(`query aborted`) + error.name = `AbortError` + reject(error) + }) + }) + }) + const source = createCollection( + queryCollectionOptions({ + id, + queryClient, + queryKey: [id], + queryFn, + getKey: (row) => row.id, + startSync: true, + syncMode: `on-demand`, + retry: false, + }), + ) + const live = createLiveQueryCollection((query) => + query.from({ row: source }).select(({ row }) => ({ id: row.id })), + ) + const preloadOutcome = live.preload().catch((error: unknown) => error) + + try { + await started + expect(queryFn).toHaveBeenCalledOnce() + expect(capturedSignal?.aborted).toBe(false) + + await live.cleanup() + expect(capturedSignal?.aborted).toBe(true) + } finally { + await live.cleanup() + await source.cleanup() + queryClient.clear() + await preloadOutcome + } +} + +async function expectRemountAfterAbortStartsFreshQuery(): Promise { + const queryClient = createQueryClient() + const id = `load-subset-remount-after-abort-${collectionSequence++}` + let resolveFirstStarted!: () => void + const firstStarted = new Promise((resolve) => { + resolveFirstStarted = resolve + }) + const queryFn = vi + .fn<(context: QueryFunctionContext) => Promise>>() + .mockImplementationOnce((context) => { + resolveFirstStarted() + return new Promise>((_resolve, reject) => { + context.signal.addEventListener(`abort`, () => { + const error = new Error(`first query aborted`) + error.name = `AbortError` + reject(error) + }) + }) + }) + .mockResolvedValueOnce([{ id: `fresh` }]) + const source = createCollection( + queryCollectionOptions({ + id, + queryClient, + queryKey: [id], + queryFn, + getKey: (row) => row.id, + startSync: true, + syncMode: `on-demand`, + retry: false, + }), + ) + const buildLive = () => + createLiveQueryCollection((query) => + query.from({ row: source }).select(({ row }) => ({ id: row.id })), + ) + const first = buildLive() + const firstOutcome = first.preload().catch((error: unknown) => error) + let second: ReturnType | undefined + + try { + await firstStarted + await first.cleanup() + await firstOutcome + + second = buildLive() + const rows = await second.toArrayWhenReady() + expect(queryFn).toHaveBeenCalledTimes(2) + expect(rows.map(({ id: rowId }) => rowId)).toEqual([`fresh`]) + } finally { + await first.cleanup() + await second?.cleanup() + await source.cleanup() + queryClient.clear() + } +} + +describe(`loadSubset lifecycle oracle`, () => { + it(`reports an initial query failure and recovers after a successful refetch`, async () => { + await expectInitialQueryFailureStatus() + }) + + it(`reports an initial failure to a dependent created after the source failed`, async () => { + await expectLateDependentObservesInitialFailure() + }) + + it(`recovers a dependent only after every failed source recovers`, async () => { + await expectEveryFailedSourceToRecover() + }) + + it(`keeps the last ready snapshot after a refetch failure`, async () => { + await expectRefetchFailureKeepsReadySnapshot() + }) + + it(`does not let deferred startup readiness override a replayed error`, async () => { + await expectDeferredStartupReadyDoesNotOverrideError() + }) + + it.each([`commutative-and`, `commutative-or`] as const)( + `%s predicate forms share one query-db transport load`, + async (form) => { + await expectEquivalentPredicatesShareOneLoad(form) + }, + ) + + it(`reversed equality operands share one query-db transport load`, async () => { + await expectEquivalentPredicatesShareOneLoad(`reversed-equality`) + }) + + it.each([ + [ + `valid Date`, + new Date(`2024-01-15T00:00:00Z`), + new Date(`2024-01-15T00:00:00Z`), + ], + [`invalid Date`, new Date(Number.NaN), new Date(Number.NaN)], + ])( + `shares one query-db transport load for equivalent %s values`, + async (_label, firstValue, secondValue) => { + await expectEquivalentComparisonValuesShareOneLoad( + firstValue, + secondValue, + ) + }, + ) + + it(`aborts an in-flight query when its final live-query owner cleans up`, async () => { + await expectFinalOwnerCleanupAbortsQuery() + }) + + it(`starts a fresh query after an aborted owner immediately remounts`, async () => { + await expectRemountAfterAbortStartsFreshQuery() + }) +}) diff --git a/packages/query-db-collection/tests/optimistic-writeback.test.ts b/packages/query-db-collection/tests/optimistic-writeback.test.ts new file mode 100644 index 0000000000..f9ace4cfb9 --- /dev/null +++ b/packages/query-db-collection/tests/optimistic-writeback.test.ts @@ -0,0 +1,114 @@ +import { expect, it } from 'vitest' +import { QueryClient } from '@tanstack/query-core' +import { + createCollection, + createLiveQueryCollection, + createOptimisticAction, + createTransaction, +} from '@tanstack/db' +import { queryCollectionOptions } from '../src/query' + +type Row = { id: string; text: string } + +it.each([`insert`, `upsert`] as const)( + `keeps repeated optimistic writes valid after direct %s acknowledgement`, + async (method) => { + const queryClient = new QueryClient({ + defaultOptions: { queries: { retry: false } }, + }) + const source = createCollection( + queryCollectionOptions({ + queryKey: [`optimistic-writeback`, method], + queryClient, + queryFn: async (): Promise> => [], + getKey: (row) => row.id, + }), + ) + const live = createLiveQueryCollection({ + query: (q) => + q.from({ row: source }).select(({ row }) => ({ + id: row.id, + text: row.text, + })), + }) + const batches: Array> = [] + const subscription = source.subscribeChanges((changes) => { + batches.push(changes.map((change) => change.key)) + }) + const insert = createOptimisticAction({ + onMutate: (row) => source.insert(row), + mutationFn: async (row) => { + await Promise.resolve() + if (method === `insert`) source.utils.writeInsert({ ...row }) + else source.utils.writeUpsert({ ...row }) + }, + }) + const rename = createOptimisticAction({ + onMutate: (row) => + source.update(row.id, (draft) => { + draft.text = row.text + }), + mutationFn: async (row) => { + await Promise.resolve() + source.utils.writeUpdate({ ...row }) + }, + }) + try { + await live.preload() + await insert({ id: `one`, text: `created` }).isPersisted.promise + for (const text of [`renamed`, `renamed again`]) { + await rename({ id: `one`, text }).isPersisted.promise + expect([...live.values()].map((row) => row.text)).toEqual([text]) + } + await insert({ id: `two`, text: `second` }).isPersisted.promise + expect([...live.values()].map((row) => row.text).sort()).toEqual([ + `renamed again`, + `second`, + ]) + for (const keys of batches) expect(new Set(keys).size).toBe(keys.length) + } finally { + subscription.unsubscribe() + await live.cleanup() + await source.cleanup() + queryClient.clear() + } + }, +) + +it(`keeps repeated optimistic updates valid after direct upsert acknowledgement`, async () => { + const queryClient = new QueryClient({ + defaultOptions: { queries: { retry: false } }, + }) + let position = 0 + const source = createCollection( + queryCollectionOptions({ + queryKey: [`optimistic-upsert-rounds`], + queryClient, + queryFn: async () => [{ id: `one`, position }], + getKey: (row) => row.id, + }), + ) + const live = createLiveQueryCollection((q) => q.from({ row: source })) + try { + await live.preload() + for (const next of [1, 2, 3]) { + const tx = createTransaction({ + mutationFn: async () => { + position = next + source.utils.writeUpsert({ id: `one`, position }) + }, + }) + tx.mutate(() => + source.update(`one`, (draft) => { + draft.position += 1 + }), + ) + await tx.isPersisted.promise + expect([...live.values()].map((row) => row.position)).toEqual([next]) + } + } finally { + await live.cleanup() + await source.cleanup() + queryClient.clear() + } +}) diff --git a/packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts b/packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts new file mode 100644 index 0000000000..0b6816a0f0 --- /dev/null +++ b/packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts @@ -0,0 +1,515 @@ +import { QueryClient, hashKey, isCancelledError } from '@tanstack/query-core' +import { createCollection, eq, getLoadSubsetDemandKey } from '@tanstack/db' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { createDeferred } from '../../db/src/deferred.js' +import { queryCollectionOptions } from '../src/query.js' +import type { Collection, SyncMetadataApi } from '@tanstack/db' +import type { NonSingleResult } from '../../db/src/types.js' +import type { QueryCollectionUtils } from '../src/query.js' + +type Item = { + id: string + category: string + name: string +} + +type MetadataRecorder = { + rows: Map + writes: Array<{ type: `set` | `delete`; key: string | number }> +} + +type OwnershipFixtureOptions = { + id: string + results: Array | Promise>> + syncMode?: `eager` | `on-demand` + customHash?: boolean + staleTime?: number + metadataRecorder?: MetadataRecorder + setupMetadata?: (metadata: SyncMetadataApi) => void +} + +type OwnershipFixture = { + collection: Collection< + Item, + string | number, + QueryCollectionUtils, + never, + Item + > & + NonSingleResult + queryClient: QueryClient + queryFn: ReturnType Promise>>> +} + +const shared = { id: `shared`, category: `shared`, name: `Shared` } +const detailOnly = { id: `detail`, category: `detail`, name: `Detail` } +const listOnly = { id: `list`, category: `list`, name: `List` } +const cleanups: Array<() => Promise> = [] + +function createQueryClient( + customHash = false, + staleTime = Number.POSITIVE_INFINITY, +): QueryClient { + return new QueryClient({ + defaultOptions: { + queries: { + gcTime: Number.POSITIVE_INFINITY, + retry: false, + staleTime, + queryKeyHashFn: customHash + ? (key) => `custom:${hashKey(key)}` + : undefined, + }, + }, + }) +} + +function recordMetadata( + metadata: SyncMetadataApi, + recorder: MetadataRecorder, +): SyncMetadataApi { + return { + row: { + get: (key) => metadata.row.get(key), + set: (key, value) => { + recorder.writes.push({ type: `set`, key }) + recorder.rows.set(key, value) + metadata.row.set(key, value) + }, + delete: (key) => { + recorder.writes.push({ type: `delete`, key }) + recorder.rows.delete(key) + metadata.row.delete(key) + }, + }, + collection: { + get: (key) => metadata.collection.get(key), + set: (key, value) => metadata.collection.set(key, value), + delete: (key) => metadata.collection.delete(key), + list: (prefix) => metadata.collection.list(prefix), + }, + } +} + +function createOwnershipFixture({ + id, + results, + syncMode = `on-demand`, + metadataRecorder, + setupMetadata, + customHash, + staleTime, +}: OwnershipFixtureOptions): OwnershipFixture { + const queryClient = createQueryClient(customHash, staleTime) + const queryFn = vi.fn<() => Promise>>() + results.forEach((result) => + queryFn.mockImplementationOnce(() => Promise.resolve(result)), + ) + queryFn.mockRejectedValue(new Error(`Unexpected ownership refetch`)) + const baseOptions = queryCollectionOptions({ + id, + queryClient, + queryKey: [id], + queryFn, + getKey: (item) => item.id, + syncMode, + startSync: true, + }) + const originalSync = baseOptions.sync + let pendingSetup = setupMetadata + const collection = createCollection( + metadataRecorder || setupMetadata + ? { + ...baseOptions, + sync: { + sync: (params: Parameters[0]) => { + if (!params.metadata) { + throw new Error(`Sync metadata API is unavailable`) + } + const observedMetadata = metadataRecorder + ? recordMetadata(params.metadata, metadataRecorder) + : params.metadata + if (pendingSetup) { + params.begin() + pendingSetup(observedMetadata) + params.commit() + pendingSetup = undefined + } + return originalSync.sync({ + ...params, + metadata: observedMetadata, + }) + }, + }, + } + : baseOptions, + ) + cleanups.push(async () => { + await collection.cleanup() + queryClient.clear() + }) + return { collection, queryClient, queryFn } +} + +function rows(collection: { + keys: () => Iterable +}): Array { + return Array.from(collection.keys()).map(String).sort() +} + +function persistedOwners( + metadata: ReadonlyMap, + rowId: string, +): Array { + const rowMetadata = metadata.get(rowId) + if (!rowMetadata || typeof rowMetadata !== `object`) return [] + const queryCollection = (rowMetadata as Record) + .queryCollection + if (!queryCollection || typeof queryCollection !== `object`) return [] + const owners = (queryCollection as Record).owners + return owners && typeof owners === `object` ? Object.keys(owners).sort() : [] +} + +describe(`query collection ownership lifecycle`, () => { + afterEach(async () => { + await Promise.all(cleanups.splice(0).map((cleanup) => cleanup())) + }) + + it(`keeps cached rows until the final exact acquisition is released`, async () => { + const { collection, queryFn } = createOwnershipFixture({ + id: `shared-acquisition`, + results: [[shared, detailOnly]], + }) + const subset = { where: eq(`category`, `detail`) } + + await collection._sync.loadSubset(subset) + await collection._sync.loadSubset(subset) + expect(queryFn).toHaveBeenCalledOnce() + expect(rows(collection)).toEqual([detailOnly.id, shared.id]) + + collection._sync.unloadSubset(subset) + expect(rows(collection)).toEqual([detailOnly.id, shared.id]) + collection._sync.unloadSubset(subset) + expect(rows(collection)).toEqual([]) + + await collection._sync.loadSubset(subset) + expect(queryFn).toHaveBeenCalledOnce() + expect(rows(collection)).toEqual([detailOnly.id, shared.id]) + }) + + it(`removes only rows whose final query owner is released`, async () => { + const { collection, queryFn } = createOwnershipFixture({ + id: `overlapping-acquisitions`, + results: [ + [shared, detailOnly], + [shared, listOnly], + ], + }) + const detail = { where: eq(`category`, `detail`) } + const list = { where: eq(`category`, `list`) } + + await collection._sync.loadSubset(detail) + await collection._sync.loadSubset(list) + expect(rows(collection)).toEqual([detailOnly.id, listOnly.id, shared.id]) + + collection._sync.unloadSubset(detail) + expect(rows(collection)).toEqual([listOnly.id, shared.id]) + await collection._sync.loadSubset(detail) + expect(queryFn).toHaveBeenCalledTimes(2) + expect(rows(collection)).toEqual([detailOnly.id, listOnly.id, shared.id]) + + collection._sync.unloadSubset(list) + expect(rows(collection)).toEqual([detailOnly.id, shared.id]) + }) + + it.each([`remount`, `refetch`] as const)( + `keeps eager rows idle after cache removal and recovers on %s`, + async (action) => { + const id = `eager-lifetime-owner` + const { collection, queryClient, queryFn } = createOwnershipFixture({ + id, + syncMode: `eager`, + results: [[shared], [{ ...shared, name: `Refetched` }]], + }) + await collection.stateWhenReady() + const subscription = collection.subscribeChanges(() => {}) + subscription.unsubscribe() + + queryClient.removeQueries({ queryKey: [id], exact: true }) + + expect(rows(collection)).toEqual([shared.id]) + await Promise.resolve() + expect(queryFn).toHaveBeenCalledOnce() + + const remounted = + action === `remount` ? collection.subscribeChanges(() => {}) : undefined + if (action === `refetch`) await collection.utils.refetch() + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(collection.get(shared.id)?.name).toBe(`Refetched`) + }) + remounted?.unsubscribe() + }, + ) + + it.each([false, true])( + `replaces an active eager cache entry with custom hash %s`, + async (customHash) => { + const id = `active-eager-custom-hash-${customHash}` + const { collection, queryClient, queryFn } = createOwnershipFixture({ + id, + customHash, + syncMode: `eager`, + results: [[shared], [{ ...shared, name: `Replaced` }]], + }) + await collection.stateWhenReady() + const subscription = collection.subscribeChanges(() => {}) + try { + queryClient.removeQueries({ queryKey: [id], exact: true }) + for (let turn = 0; turn < 20; turn++) await Promise.resolve() + expect(queryFn).toHaveBeenCalledTimes(2) + expect(collection.get(shared.id)?.name).toBe(`Replaced`) + } finally { + subscription.unsubscribe() + } + }, + ) + + it.each( + [false, true].flatMap((mounted) => + [false, true].flatMap((customHash) => + [false, true].map((rejectOld) => ({ mounted, customHash, rejectOld })), + ), + ), + )( + `replaces a removed pending eager refetch without reviving idle demand: %j`, + async ({ mounted, customHash, rejectOld }) => { + const old = createDeferred>() + const next = createDeferred>() + const id = `pending-eager-removal` + const { collection, queryClient, queryFn } = createOwnershipFixture({ + id, + customHash, + syncMode: `eager`, + results: [[shared], old.promise, next.promise], + }) + await collection.stateWhenReady() + let subscription = collection.subscribeChanges(() => {}) + if (!mounted) subscription.unsubscribe() + let settled = false + const refetch = collection.utils.refetch({ throwOnError: true }).then( + () => { + settled = true + }, + (error: unknown) => { + settled = true + return error + }, + ) + try { + await vi.waitFor(() => expect(queryFn).toHaveBeenCalledTimes(2)) + queryClient.removeQueries({ queryKey: [id], exact: true }) + for (let turn = 0; turn < 30; turn++) await Promise.resolve() + expect(queryFn).toHaveBeenCalledTimes(mounted ? 3 : 2) + expect(collection.get(shared.id)?.name).toBe(`Shared`) + if (!mounted) subscription = collection.subscribeChanges(() => {}) + await vi.waitFor(() => expect(queryFn).toHaveBeenCalledTimes(3)) + next.resolve([{ ...shared, name: `Current` }]) + await vi.waitFor(() => + expect(collection.get(shared.id)?.name).toBe(`Current`), + ) + if (rejectOld) old.reject(new Error(`retired request failed`)) + else old.resolve([{ ...shared, name: `Obsolete` }]) + await vi.waitFor(() => expect(settled).toBe(true)) + expect(isCancelledError(await refetch)).toBe(true) + expect(collection.get(shared.id)?.name).toBe(`Current`) + expect(queryFn).toHaveBeenCalledTimes(3) + expect(collection.status).toBe(`ready`) + } finally { + old.resolve([shared]) + next.resolve([shared]) + subscription.unsubscribe() + } + }, + ) + + it.each( + [0, Number.POSITIVE_INFINITY].flatMap((staleTime) => + [false, true].map((customHash) => ({ staleTime, customHash })), + ), + )( + `starts only the requested fetch for an idle eager observer: %j`, + async ({ staleTime, customHash }) => { + const { collection, queryFn } = createOwnershipFixture({ + id: `idle-explicit-refetch`, + syncMode: `eager`, + staleTime, + customHash, + results: [[shared]], + }) + await collection.stateWhenReady() + queryFn.mockResolvedValue([{ ...shared, name: `Refetched` }]) + const subscription = collection.subscribeChanges(() => {}) + subscription.unsubscribe() + for (let turn = 0; turn < 30; turn++) await Promise.resolve() + const before = queryFn.mock.calls.length + await collection.utils.refetch({ throwOnError: true }) + expect(queryFn).toHaveBeenCalledTimes(before + 1) + expect(collection.subscriberCount).toBe(0) + }, + ) + + it.each([`release`, `cleanup`, `retain`] as const)( + `honors %s during startup retention maintenance`, + async (action) => { + const id = `released-startup-retention` + const subset = { where: eq(`category`, `shared`) } + const key = `queryCollection:gc:${hashKey([id, getLoadSubsetDemandKey(subset)])}` + const { collection, queryFn } = createOwnershipFixture({ + id, + results: [[shared]], + setupMetadata: (metadata) => + metadata.collection.set(key, { + queryHash: hashKey([id, getLoadSubsetDemandKey(subset)]), + mode: `until-revalidated`, + }), + }) + collection.startSyncImmediate() + const result = Promise.resolve(collection._sync.loadSubset(subset)).then( + () => `ready`, + (error: unknown) => error, + ) + if (action === `release`) collection._sync.unloadSubset(subset) + if (action === `cleanup`) await collection.cleanup() + for (let turn = 0; turn < 30; turn++) await Promise.resolve() + if (action === `retain`) { + expect(queryFn).toHaveBeenCalledTimes(1) + await expect(result).resolves.toBe(`ready`) + expect(collection.size).toBe(1) + } else { + expect(queryFn).not.toHaveBeenCalled() + await expect(result).resolves.toMatchObject({ name: `AbortError` }) + expect(collection.size).toBe(0) + } + }, + ) + + it.each([false, true])( + `removes owned cache entries on cleanup with custom hash %s`, + async (customHash) => { + const id = `cleanup-custom-${customHash}` + const { collection, queryClient } = createOwnershipFixture({ + id, + customHash, + syncMode: `eager`, + results: [[shared]], + }) + await collection.stateWhenReady() + expect(queryClient.getQueryCache().getAll()).toHaveLength(1) + await collection.cleanup() + expect(queryClient.getQueryCache().getAll()).toHaveLength(0) + }, + ) + + it(`settles an unfinished load when its final owner leaves`, async () => { + const pending = createDeferred>() + const { collection } = createOwnershipFixture({ + id: `release-before-result`, + results: [pending.promise], + }) + const subset = { where: eq(`category`, `shared`) } + let outcome: unknown = `pending` + const load = Promise.resolve(collection._sync.loadSubset(subset)).then( + () => { + outcome = `ready` + }, + (error: unknown) => { + outcome = error + }, + ) + try { + expect(collection.isLoadingSubset).toBe(true) + collection._sync.unloadSubset(subset) + for (let turn = 0; turn < 20; turn++) await Promise.resolve() + expect(outcome).toMatchObject({ name: `AbortError` }) + expect(collection.isLoadingSubset).toBe(false) + await load + } finally { + pending.resolve([shared]) + } + }) + + it(`keeps active on-demand rows when the Query cache entry departs`, async () => { + const id = `active-cache-removal` + const { collection, queryClient } = createOwnershipFixture({ + id, + results: [[shared]], + }) + const subset = { where: eq(`category`, `detail`) } + await collection._sync.loadSubset(subset) + + queryClient.removeQueries({ queryKey: [id] }) + expect(rows(collection)).toEqual([shared.id]) + + collection._sync.unloadSubset(subset) + expect(rows(collection)).toEqual([]) + }) + + it(`persists every owner of rows shared by overlapping queries`, async () => { + const metadata: MetadataRecorder = { rows: new Map(), writes: [] } + const { collection } = createOwnershipFixture({ + id: `persisted-overlap`, + results: [[shared], [shared, listOnly]], + metadataRecorder: metadata, + }) + const detail = { where: eq(`category`, `detail`) } + const list = { where: eq(`category`, `list`) } + + await collection._sync.loadSubset(detail) + expect(persistedOwners(metadata.rows, shared.id)).toHaveLength(1) + + await collection._sync.loadSubset(list) + expect(persistedOwners(metadata.rows, shared.id)).toHaveLength(2) + expect(persistedOwners(metadata.rows, listOnly.id)).toHaveLength(1) + + collection._sync.unloadSubset(list) + expect(rows(collection)).toEqual([shared.id]) + expect(persistedOwners(metadata.rows, shared.id)).toHaveLength(1) + }) + + it(`restages a persisted owner when its absent row arrives`, async () => { + const id = `persisted-owner-before-row` + const queryHash = hashKey([id]) + const result = createDeferred>() + const metadata: MetadataRecorder = { rows: new Map(), writes: [] } + let setupCalls = 0 + const { collection, queryFn } = createOwnershipFixture({ + id, + syncMode: `eager`, + results: [result.promise, [{ ...shared, name: `Restarted` }]], + metadataRecorder: metadata, + setupMetadata: (api) => { + setupCalls++ + api.row.set(shared.id, { + queryCollection: { owners: { [queryHash]: true } }, + }) + }, + }) + + expect(rows(collection)).toEqual([]) + expect(persistedOwners(metadata.rows, shared.id)).toEqual([queryHash]) + result.resolve([shared]) + await collection.stateWhenReady() + expect(rows(collection)).toEqual([shared.id]) + expect(persistedOwners(metadata.rows, shared.id)).toEqual([queryHash]) + + await collection.cleanup() + await collection.preload() + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(collection.get(shared.id)?.name).toBe(`Restarted`) + }) + expect(setupCalls).toBe(1) + expect(persistedOwners(metadata.rows, shared.id)).toEqual([queryHash]) + }) +}) diff --git a/packages/query-db-collection/tests/query.test-d.ts b/packages/query-db-collection/tests/query.test-d.ts index c526398f8a..30e8f64c9c 100644 --- a/packages/query-db-collection/tests/query.test-d.ts +++ b/packages/query-db-collection/tests/query.test-d.ts @@ -32,6 +32,30 @@ describe(`Query collection type resolution tests`, () => { // Create a mock QueryClient for tests const queryClient = new QueryClient() + it(`should type supported top-level Query observer options and reject adapter-owned fields`, () => { + queryCollectionOptions({ + id: `query-options-types`, + queryClient, + queryKey: [`query-options-types`], + queryFn: () => Promise.resolve([]), + getKey: (item) => item.id, + refetchOnWindowFocus: true, + refetchOnReconnect: true, + refetchOnMount: `always`, + networkMode: `online`, + }) + + queryCollectionOptions({ + id: `query-options-subscribed-owned`, + queryClient, + queryKey: [`query-options-subscribed-owned`], + queryFn: () => Promise.resolve([]), + getKey: (item) => item.id, + // @ts-expect-error Query Collection owns observer subscription lifecycle. + subscribed: false, + }) + }) + it(`should prioritize explicit type in QueryCollectionConfig`, () => { const options = queryCollectionOptions({ id: `test`, diff --git a/packages/query-db-collection/tests/query.test.ts b/packages/query-db-collection/tests/query.test.ts index 0c501ab027..51de217b1b 100644 --- a/packages/query-db-collection/tests/query.test.ts +++ b/packages/query-db-collection/tests/query.test.ts @@ -1,15 +1,28 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' -import { QueryClient, hashKey } from '@tanstack/query-core' +import { + QueryClient, + dehydrate, + focusManager, + hashKey, + onlineManager, +} from '@tanstack/query-core' import { BTreeIndex, + DbClient, + collectionOptions, createCollection, createLiveQueryCollection, + createTransaction, eq, ilike, inArray, or, } from '@tanstack/db' -import { stripVirtualProps } from '../../db/tests/utils' +import { + mockSyncCollectionOptions, + stripVirtualProps, +} from '../../db/tests/utils' +import { evaluateReferenceExpression } from '../../db/tests/reference-expression' import { persistedCollectionOptions } from '../../db-sqlite-persistence-core/src' import { queryCollectionOptions } from '../src/query' import type { QueryFunctionContext } from '@tanstack/query-core' @@ -17,6 +30,7 @@ import type { Collection, DeleteMutationFnParams, InsertMutationFnParams, + LoadSubsetOptions, SyncMetadataApi, TransactionWithMutations, UpdateMutationFnParams, @@ -40,6 +54,16 @@ const getKey = (item: TestItem) => item.id // Helper to advance timers and allow microtasks to flush const flushPromises = () => new Promise((resolve) => setTimeout(resolve, 0)) +function createDeferred() { + let resolve!: (value: T) => void + let reject!: (reason?: unknown) => void + const promise = new Promise((resolvePromise, rejectPromise) => { + resolve = resolvePromise + reject = rejectPromise + }) + return { promise, resolve, reject } +} + function createInMemorySyncMetadataApi< TKey extends string | number = string | number, TItem extends object = Record, @@ -176,11 +200,768 @@ describe(`QueryCollection`, () => { }) }) + it(`materializes against each DbClient QueryClient dependency`, async () => { + const constructionClient = new QueryClient() + const queryClientA = new QueryClient() + const queryClientB = new QueryClient() + const queryKey = [`db-client-query-dependency`] as const + const descriptor = collectionOptions( + queryCollectionOptions({ + id: `db-client-query-dependency`, + queryClient: constructionClient, + queryKey, + queryFn: async () => [{ id: `1`, name: `Item` }], + getKey, + }), + ) + const dbClientA = new DbClient({ queryClient: queryClientA }) + const dbClientB = new DbClient({ queryClient: queryClientB }) + const collectionA = dbClientA.collection(descriptor) + const collectionB = dbClientB.collection(descriptor) + + await Promise.all([collectionA.preload(), collectionB.preload()]) + + expect(queryClientA.getQueryData(queryKey)).toEqual([ + { id: `1`, name: `Item` }, + ]) + expect(queryClientB.getQueryData(queryKey)).toEqual([ + { id: `1`, name: `Item` }, + ]) + expect(constructionClient.getQueryData(queryKey)).toBeUndefined() + + await Promise.all([dbClientA.cleanup(), dbClientB.cleanup()]) + constructionClient.clear() + queryClientA.clear() + queryClientB.clear() + }) + + it(`falls back to the configured QueryClient when DbClient has no dependency`, async () => { + const constructionClient = new QueryClient() + const queryKey = [`db-client-query-fallback`] as const + const descriptor = collectionOptions( + queryCollectionOptions({ + id: `db-client-query-fallback`, + queryClient: constructionClient, + queryKey, + queryFn: async () => [{ id: `1`, name: `Item` }], + getKey, + }), + ) + const dbClient = new DbClient() + + const collection = dbClient.collection(descriptor) + await collection.preload() + + expect(constructionClient.getQueryData(queryKey)).toEqual([ + { id: `1`, name: `Item` }, + ]) + + await dbClient.cleanup() + constructionClient.clear() + }) + afterEach(() => { // Ensure all queries are properly cleaned up after each test queryClient.clear() }) + it(`should pass through additional top-level Query observer options`, async () => { + const queryKey = [`query-options-pass-through`] + const queryFn = vi.fn().mockResolvedValue([{ id: `1`, name: `Item 1` }]) + + const collection = createCollection( + queryCollectionOptions({ + id: `query-options-pass-through`, + queryClient, + queryKey, + queryFn, + getKey, + startSync: true, + refetchOnWindowFocus: true, + refetchOnReconnect: true, + refetchOnMount: `always`, + networkMode: `online`, + }), + ) + + await vi.waitFor(() => { + expect(collection.size).toBe(1) + }) + + const query = queryClient.getQueryCache().find({ queryKey, exact: true }) + const options = query?.options as any + expect(options.refetchOnWindowFocus).toBe(true) + expect(options.refetchOnReconnect).toBe(true) + expect(options.refetchOnMount).toBe(`always`) + expect(options.networkMode).toBe(`online`) + }) + + describe(`initialData`, () => { + it(`materializes eager initial data without fetching while it is fresh`, async () => { + const queryKey = [`initial-data-eager`] + const initialData: Array = [{ id: `1`, name: `Initial item` }] + const queryFn = vi + .fn<() => Promise>>() + .mockResolvedValue([{ id: `1`, name: `Fetched item` }]) + + const collection = createCollection( + queryCollectionOptions({ + id: `initial-data-eager`, + queryClient, + queryKey, + queryFn, + getKey, + startSync: true, + initialData, + staleTime: Infinity, + }), + ) + + try { + await vi.waitFor(() => { + expect(collection.status).toBe(`ready`) + expect(stripVirtualProps(collection.get(`1`))).toEqual(initialData[0]) + }) + + expect(queryFn).not.toHaveBeenCalled() + expect(queryClient.getQueryData(queryKey)).toEqual(initialData) + } finally { + await collection.cleanup() + } + }) + + it(`evaluates a function initializer once for a missing Query cache entry`, async () => { + const initialData = vi.fn(() => [{ id: `1`, name: `Initial item` }]) + const collection = createCollection( + queryCollectionOptions({ + id: `initial-data-function`, + queryClient, + queryKey: [`initial-data-function`], + queryFn: vi.fn().mockResolvedValue([]), + getKey, + startSync: true, + initialData, + staleTime: Infinity, + }), + ) + + try { + await vi.waitFor(() => { + expect(collection.get(`1`)?.name).toBe(`Initial item`) + }) + expect(initialData).toHaveBeenCalledTimes(1) + } finally { + await collection.cleanup() + } + }) + + it(`keeps stale initial rows while fetching and reconciles the server result`, async () => { + let resolveQuery: ((items: Array) => void) | undefined + const queryFn = vi.fn( + () => + new Promise>((resolve) => { + resolveQuery = resolve + }), + ) + const collection = createCollection( + queryCollectionOptions({ + id: `initial-data-stale`, + queryClient, + queryKey: [`initial-data-stale`], + queryFn, + getKey, + startSync: true, + initialData: [{ id: `initial`, name: `Initial` }], + initialDataUpdatedAt: 1, + staleTime: 0, + }), + ) + + try { + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(collection.get(`initial`)?.name).toBe(`Initial`) + }) + + resolveQuery?.([{ id: `server`, name: `Server` }]) + await vi.waitFor(() => { + expect(collection.has(`initial`)).toBe(false) + expect(collection.get(`server`)?.name).toBe(`Server`) + }) + } finally { + await collection.cleanup() + } + }) + + it(`retains initial rows when a refetch fails`, async () => { + const initialRow = { id: `initial`, name: `Initial` } + const error = new Error(`Refetch failed`) + const queryFn = vi.fn().mockRejectedValue(error) + const consoleErrorSpy = vi + .spyOn(console, `error`) + .mockImplementation(() => {}) + const collection = createCollection( + queryCollectionOptions({ + id: `initial-data-refetch-error`, + queryClient, + queryKey: [`initial-data-refetch-error`], + queryFn, + getKey, + startSync: true, + initialData: [initialRow], + initialDataUpdatedAt: 1, + staleTime: 0, + retry: false, + }), + ) + + try { + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(collection.utils.lastError).toBe(error) + }) + expect(stripVirtualProps(collection.get(initialRow.id))).toEqual( + initialRow, + ) + } finally { + consoleErrorSpy.mockRestore() + await collection.cleanup() + } + }) + + it(`ignores a late refetch result after initial data is cleaned up`, async () => { + const serverResult = createDeferred>() + const collection = createCollection( + queryCollectionOptions({ + id: `initial-data-late-cleanup`, + queryClient, + queryKey: [`initial-data-late-cleanup`], + queryFn: () => serverResult.promise, + getKey, + startSync: true, + initialData: [{ id: `initial`, name: `Initial` }], + initialDataUpdatedAt: 1, + staleTime: 0, + }), + ) + + await vi.waitFor(() => { + expect(collection.get(`initial`)?.name).toBe(`Initial`) + expect(queryClient.isFetching()).toBe(1) + }) + await collection.cleanup() + + serverResult.resolve([{ id: `server`, name: `Server` }]) + await vi.waitFor(() => expect(queryClient.isFetching()).toBe(0)) + expect(collection.status).toBe(`cleaned-up`) + expect(collection.size).toBe(0) + }) + + it(`projects a wrapped initial response while preserving its Query cache shape`, async () => { + const queryKey = [`initial-data-wrapped`] + const initialResponse = { + items: [{ id: `1`, name: `Initial item` }], + nextCursor: `next`, + } + const queryFn = vi.fn().mockResolvedValue(initialResponse) + + const collection = createCollection( + queryCollectionOptions({ + id: `initial-data-wrapped`, + queryClient, + queryKey, + queryFn, + select: (response: typeof initialResponse) => response.items, + getKey, + startSync: true, + initialData: initialResponse, + staleTime: Infinity, + }), + ) + + try { + await vi.waitFor(() => { + expect(stripVirtualProps(collection.get(`1`))).toEqual( + initialResponse.items[0], + ) + }) + + expect(queryFn).not.toHaveBeenCalled() + expect(queryClient.getQueryData(queryKey)).toEqual(initialResponse) + } finally { + await collection.cleanup() + } + }) + + it(`preserves a wrapped initial response when writing rows directly`, async () => { + const queryKey = [`initial-data-wrapped-writes`] + const initialResponse = { + items: [{ id: `1`, name: `Initial item` }], + nextCursor: `next`, + } + const collection = createCollection( + queryCollectionOptions({ + id: `initial-data-wrapped-writes`, + queryClient, + queryKey, + queryFn: vi.fn().mockResolvedValue(initialResponse), + select: (response: typeof initialResponse) => response.items, + getKey, + startSync: true, + initialData: initialResponse, + staleTime: Infinity, + }), + ) + + try { + await vi.waitFor(() => { + expect(collection.get(`1`)?.name).toBe(`Initial item`) + }) + + collection.utils.writeInsert({ id: `2`, name: `Inserted item` }) + collection.utils.writeUpdate({ id: `1`, name: `Updated item` }) + + expect(queryClient.getQueryData(queryKey)).toEqual({ + items: [ + { id: `1`, name: `Updated item` }, + { id: `2`, name: `Inserted item` }, + ], + nextCursor: `next`, + }) + } finally { + await collection.cleanup() + } + }) + + it(`keeps initial data scoped to each collection on a shared QueryClient`, async () => { + const first = createCollection( + queryCollectionOptions({ + id: `initial-data-shared-client-first`, + queryClient, + queryKey: [`initial-data-shared-client`, `first`], + queryFn: vi.fn().mockResolvedValue([]), + getKey, + startSync: true, + initialData: [{ id: `1`, name: `First` }], + staleTime: Infinity, + }), + ) + const second = createCollection( + queryCollectionOptions({ + id: `initial-data-shared-client-second`, + queryClient, + queryKey: [`initial-data-shared-client`, `second`], + queryFn: vi.fn().mockResolvedValue([]), + getKey, + startSync: true, + initialData: [{ id: `2`, name: `Second` }], + staleTime: Infinity, + }), + ) + + try { + await vi.waitFor(() => { + expect(first.get(`1`)?.name).toBe(`First`) + expect(second.get(`2`)?.name).toBe(`Second`) + }) + expect(first.has(`2`)).toBe(false) + expect(second.has(`1`)).toBe(false) + } finally { + await Promise.all([first.cleanup(), second.cleanup()]) + } + }) + + it(`does not replace existing Query data with a later collection initializer`, async () => { + const queryKey = [`initial-data-shared-key`] + queryClient.setQueryData(queryKey, [{ id: `cached`, name: `Cached` }]) + + const collection = createCollection( + queryCollectionOptions({ + id: `initial-data-shared-key`, + queryClient, + queryKey, + queryFn: vi.fn().mockResolvedValue([]), + getKey, + startSync: true, + initialData: [{ id: `initial`, name: `Initial` }], + staleTime: Infinity, + }), + ) + + try { + await vi.waitFor(() => { + expect(collection.get(`cached`)?.name).toBe(`Cached`) + }) + expect(collection.has(`initial`)).toBe(false) + } finally { + await collection.cleanup() + } + }) + + it(`rejects collection-level initial data in on-demand mode`, () => { + expect(() => + queryCollectionOptions({ + id: `initial-data-on-demand`, + queryClient, + queryKey: [`initial-data-on-demand`], + queryFn: vi.fn().mockResolvedValue([]), + getKey, + syncMode: `on-demand`, + initialData: [{ id: `1`, name: `Initial` }], + }), + ).toThrow( + `initialData and initialDataUpdatedAt are only supported when syncMode is 'eager'`, + ) + }) + + it(`does not apply QueryClient initial data defaults to on-demand subsets`, async () => { + const defaultInitialQueryClient = new QueryClient({ + defaultOptions: { + queries: { + initialData: [{ id: `default`, name: `Default` }], + staleTime: Infinity, + retry: false, + }, + }, + }) + const queryFn = vi + .fn() + .mockResolvedValue([{ id: `server`, name: `Server` }]) + const collection = createCollection( + queryCollectionOptions({ + id: `initial-data-default-on-demand`, + queryClient: defaultInitialQueryClient, + queryKey: [`initial-data-default-on-demand`], + queryFn, + getKey, + syncMode: `on-demand`, + startSync: true, + }), + ) + + try { + await collection._sync.loadSubset({}) + await vi.waitFor(() => { + expect(collection.get(`server`)?.name).toBe(`Server`) + }) + expect(collection.has(`default`)).toBe(false) + expect(queryFn).toHaveBeenCalledTimes(1) + } finally { + await collection.cleanup() + defaultInitialQueryClient.clear() + } + }) + + it(`keeps an eager result loading until its rows are applied`, async () => { + const queryResult = createDeferred>() + const queryFn = vi.fn(() => queryResult.promise) + const collection = createCollection( + queryCollectionOptions({ + id: `eager-applied-settlement`, + queryClient, + queryKey: [`eager-applied-settlement`], + queryFn, + getKey, + syncMode: `eager`, + startSync: true, + }), + ) + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => + collection.insert({ id: `local`, name: `Local` }), + ) + + try { + const ready = collection.stateWhenReady() + await vi.waitFor(() => expect(queryFn).toHaveBeenCalledOnce()) + queryResult.resolve([{ id: `server`, name: `Server` }]) + await flushPromises() + + expect(collection.status).toBe(`loading`) + expect(collection.get(`server`)).toBeUndefined() + + persistence.resolve() + await transaction.isPersisted.promise + await ready + + expect(collection.status).toBe(`ready`) + expect(collection.get(`server`)).toEqual( + expect.objectContaining({ id: `server`, name: `Server` }), + ) + } finally { + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } + }) + + it(`does not publish queued query results after the subset is released`, async () => { + const queryKey = [`released-result-application`] + const queryResult = createDeferred>() + const collection = createCollection( + queryCollectionOptions({ + id: `released-result-application`, + queryClient, + queryKey, + queryFn: () => queryResult.promise, + getKey, + syncMode: `on-demand`, + startSync: true, + }), + ) + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + + try { + transaction.mutate(() => + collection.insert({ id: `local`, name: `Local` }), + ) + collection._sync.loadSubset({}) + queryResult.resolve([{ id: `first`, name: `First` }]) + await flushPromises() + + queryClient.setQueryData(queryKey, [{ id: `second`, name: `Second` }]) + await flushPromises() + collection._sync.unloadSubset({}) + + persistence.resolve() + await transaction.isPersisted.promise + await flushPromises() + + expect(collection.has(`first`)).toBe(false) + expect(collection.has(`second`)).toBe(false) + } finally { + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } + }) + + it(`keeps a deferred successful result pending until its refetch applies`, async () => { + const barrier = createDeferred() + const queryFn = vi + .fn() + .mockResolvedValue([{ id: `server`, name: `Server` }]) + const collection = createCollection( + queryCollectionOptions({ + id: `deferred-result-application`, + queryClient, + queryKey: [`deferred-result-application`], + queryFn, + getKey, + syncMode: `on-demand`, + startSync: true, + }), + ) + collection.deferDataRefresh = barrier.promise + + try { + const load = collection._sync.loadSubset({}) + let settled = false + void Promise.resolve(load).then(() => { + settled = true + }) + await vi.waitFor(() => expect(queryFn).toHaveBeenCalledOnce()) + await flushPromises() + + expect(settled).toBe(false) + expect(collection.has(`server`)).toBe(false) + + collection.deferDataRefresh = null + barrier.resolve() + if (load !== true) await load + + expect(queryFn).toHaveBeenCalledTimes(2) + expect(collection.has(`server`)).toBe(true) + } finally { + collection.deferDataRefresh = null + barrier.resolve() + await collection.cleanup() + } + }) + + it(`applies successive eager results in publication order`, async () => { + const queryKey = [`eager-result-publication-order`] + const collection = createCollection( + queryCollectionOptions({ + id: `eager-result-publication-order`, + queryClient, + queryKey, + queryFn: vi.fn().mockResolvedValue([]), + getKey, + syncMode: `eager`, + startSync: true, + }), + ) + + await collection.stateWhenReady() + + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + transaction.mutate(() => + collection.insert({ id: `local`, name: `Local` }), + ) + + try { + queryClient.setQueryData(queryKey, [{ id: `server`, name: `Server` }]) + await flushPromises() + queryClient.setQueryData(queryKey, []) + await flushPromises() + + persistence.resolve() + await transaction.isPersisted.promise + + await vi.waitFor(() => { + expect(collection.get(`server`)).toBeUndefined() + }) + } finally { + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } + }) + + it(`does not materialize QueryClient placeholder defaults`, async () => { + const placeholderQueryClient = new QueryClient({ + defaultOptions: { + queries: { + placeholderData: [{ id: `placeholder`, name: `Placeholder` }], + retry: false, + }, + }, + }) + let resolveQuery: ((items: Array) => void) | undefined + const queryFn = vi.fn( + () => + new Promise>((resolve) => { + resolveQuery = resolve + }), + ) + const collection = createCollection( + queryCollectionOptions({ + id: `placeholder-default`, + queryClient: placeholderQueryClient, + queryKey: [`placeholder-default`], + queryFn, + getKey, + startSync: true, + }), + ) + + try { + await vi.waitFor(() => expect(queryFn).toHaveBeenCalledTimes(1)) + expect(collection.has(`placeholder`)).toBe(false) + + resolveQuery?.([{ id: `server`, name: `Server` }]) + await vi.waitFor(() => { + expect(collection.get(`server`)?.name).toBe(`Server`) + }) + } finally { + await collection.cleanup() + placeholderQueryClient.clear() + } + }) + }) + + it(`should refetch on focus and reconnect with standalone QueryClient`, async () => { + const queryKey = [`query-options-event-refetch`] + const queryFn = vi + .fn() + .mockResolvedValueOnce([{ id: `1`, name: `Initial` }]) + .mockResolvedValueOnce([{ id: `1`, name: `Focused` }]) + .mockResolvedValueOnce([{ id: `1`, name: `Reconnected` }]) + + const collection = createCollection( + queryCollectionOptions({ + id: `query-options-event-refetch`, + queryClient, + queryKey, + queryFn, + getKey, + startSync: true, + staleTime: 0, + refetchOnWindowFocus: true, + refetchOnReconnect: true, + }), + ) + + try { + await vi.waitFor(() => { + expect(collection.size).toBe(1) + expect(queryFn).toHaveBeenCalledTimes(1) + }) + + focusManager.setFocused(false) + focusManager.setFocused(true) + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + }) + + onlineManager.setOnline(false) + onlineManager.setOnline(true) + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(3) + }) + } finally { + await collection.cleanup() + focusManager.setFocused(undefined) + onlineManager.setOnline(true) + } + }) + + it(`should omit undefined Query observer options to preserve defaults`, async () => { + const queryKey = [`query-options-default-preservation`] + const queryFn = vi.fn().mockResolvedValue([{ id: `1`, name: `Item 1` }]) + + const clientWithDefaults = new QueryClient({ + defaultOptions: { + queries: { + staleTime: 1234, + retry: 3, + refetchOnWindowFocus: false, + }, + }, + }) + + const collection = createCollection( + queryCollectionOptions({ + id: `query-options-default-preservation`, + queryClient: clientWithDefaults, + queryKey, + queryFn, + getKey, + startSync: true, + staleTime: 5678, + retry: undefined, + refetchOnWindowFocus: true, + }), + ) + + await vi.waitFor(() => { + expect(collection.size).toBe(1) + }) + + const query = clientWithDefaults.getQueryCache().find({ + queryKey, + exact: true, + }) + const options = query?.options as any + expect(options.staleTime).toBe(5678) + expect(options.retry).toBe(3) + expect(options.refetchOnWindowFocus).toBe(true) + + clientWithDefaults.clear() + }) + it(`should initialize and fetch initial data`, async () => { const queryKey = [`testItems`] const initialItems: Array = [ @@ -228,6 +1009,154 @@ describe(`QueryCollection`, () => { expect(collection._state.syncedData.get(`2`)).toEqual(initialItems[1]) }) + it(`should not duplicate insert into includes child collection after update refetch`, async () => { + type LineItem = { id: string; productId: string } + type Product = { id: string; categoryId: number; name: string } + + const lineItems = createCollection( + mockSyncCollectionOptions({ + id: `query-collection-line-items`, + getKey: (lineItem) => lineItem.id, + initialData: [{ id: `line-1`, productId: `product-1` }], + }), + ) + + let productsData: Array = [ + { id: `product-1`, categoryId: 1, name: `Widget` }, + ] + + const products = createCollection( + queryCollectionOptions({ + id: `query-collection-products`, + queryClient, + queryKey: [`products`], + queryFn: vi + .fn() + .mockImplementation(() => Promise.resolve(productsData)), + getKey: (product) => product.id, + startSync: true, + onUpdate: async ({ transaction }) => { + for (const mutation of transaction.mutations) { + productsData = productsData.map((product) => + product.id === mutation.key ? mutation.modified : product, + ) + } + }, + }), + ) + + const collection = createLiveQueryCollection((q) => + q.from({ lineItem: lineItems }).select(({ lineItem }) => ({ + id: lineItem.id, + product: q + .from({ product: products }) + .where(({ product }) => eq(product.id, lineItem.productId)) + .select(({ product }) => ({ + id: product.id, + categoryId: product.categoryId, + name: product.name, + })), + })), + ) + + await collection.preload() + + expect(() => { + products.update(`product-1`, (draft) => { + draft.categoryId = 2 + }) + }).not.toThrow() + + await vi.waitFor(() => { + expect( + stripVirtualProps((collection.get(`line-1`) as any).product.toArray[0]), + ).toEqual({ + id: `product-1`, + categoryId: 2, + name: `Widget`, + }) + }) + }) + + it(`reconciles a retained cached subset before an include becomes ready`, async () => { + type LineItem = { id: string; productId: string } + type Product = { id: string; name: string } + + const queryKey = [`cached-includes-product`] + const cachedProducts: Array = [ + { id: `product-1`, name: `Cached widget` }, + ] + queryClient.setQueryData(queryKey, cachedProducts) + const queryHash = hashKey(queryKey) + const metadataHarness = createInMemorySyncMetadataApi< + string | number, + Product + >({ + collectionMetadata: new Map([ + [ + `queryCollection:gc:${queryHash}`, + { queryHash, mode: `until-revalidated` }, + ], + ]), + }) + + const lineItems = createCollection( + mockSyncCollectionOptions({ + id: `cached-includes-line-items`, + getKey: (lineItem) => lineItem.id, + initialData: [{ id: `line-1`, productId: `product-1` }], + }), + ) + const productOptions = queryCollectionOptions({ + id: `cached-includes-products`, + queryClient, + queryKey: () => queryKey, + queryFn: vi.fn().mockResolvedValue(cachedProducts), + getKey: (product) => product.id, + syncMode: `on-demand`, + startSync: true, + staleTime: Infinity, + }) + const originalSync = productOptions.sync + const products = createCollection({ + ...productOptions, + sync: { + sync: (params: Parameters[0]) => + originalSync.sync({ ...params, metadata: metadataHarness.api }), + }, + }) + const live = createLiveQueryCollection((q) => + q.from({ lineItem: lineItems }).select(({ lineItem }) => ({ + id: lineItem.id, + product: q + .from({ product: products }) + .where(({ product }) => eq(product.id, lineItem.productId)) + .select(({ product }) => ({ + id: product.id, + name: product.name, + })), + })), + ) + + try { + await new Promise((resolve) => products.onFirstReady(resolve)) + await live.preload() + + expect(live.status).toBe(`ready`) + expect( + (live.get(`line-1`) as any).product.toArray.map((product: Product) => + stripVirtualProps(product), + ), + ).toEqual(cachedProducts) + } finally { + await Promise.all([ + live.cleanup(), + products.cleanup(), + lineItems.cleanup(), + ]) + } + }) + it(`should update collection when query data changes`, async () => { const queryKey = [`testItems`] const initialItems: Array = [ @@ -825,6 +1754,46 @@ describe(`QueryCollection`, () => { expect(initialCache).toEqual(initialMetaData) }) + it(`materializes selected rows while preserving wrapped Query cache response`, async () => { + const queryKey = [`select-row-extraction-test`] + const wrappedResponse = { + items: initialMetaData.data, + meta: { page: 1, total: initialMetaData.data.length }, + } + const expectedCacheResponse = { + items: initialMetaData.data.map((item) => ({ ...item })), + meta: { page: 1, total: initialMetaData.data.length }, + } + + const queryFn = vi.fn().mockResolvedValue(wrappedResponse) + const select = vi.fn((data: typeof wrappedResponse) => data.items) + + const options = queryCollectionOptions({ + id: `select-row-extraction-test`, + queryClient, + queryKey, + queryFn, + select, + getKey, + startSync: true, + }) + const collection = createCollection(options) + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(select).toHaveBeenCalledTimes(1) + expect(collection.size).toBe(wrappedResponse.items.length) + }) + + expect(stripVirtualProps(collection.get(`1`))).toEqual( + wrappedResponse.items[0], + ) + expect(stripVirtualProps(collection.get(`2`))).toEqual( + wrappedResponse.items[1], + ) + expect(queryClient.getQueryData(queryKey)).toEqual(expectedCacheResponse) + }) + it(`should not throw error when using writeInsert with select option`, async () => { const queryKey = [`select-writeInsert-test`] const consoleErrorSpy = vi @@ -1257,7 +2226,7 @@ describe(`QueryCollection`, () => { // We're mainly verifying the collection cleanup works without errors }) - it(`should call cancelQueries and removeQueries on sync cleanup`, async () => { + it(`should remove its Query cache entry on sync cleanup`, async () => { const queryKey = [`sync-cleanup-test`] const items = [{ id: `1`, name: `Item 1` }] const queryFn = vi.fn().mockResolvedValue(items) @@ -1271,12 +2240,6 @@ describe(`QueryCollection`, () => { startSync: true, } - // Spy on the queryClient methods that should be called during sync cleanup - const cancelQueriesSpy = vi - .spyOn(queryClient, `cancelQueries`) - .mockResolvedValue() - const removeQueriesSpy = vi.spyOn(queryClient, `removeQueries`) - const options = queryCollectionOptions(config) const collection = createCollection(options) @@ -1290,6 +2253,7 @@ describe(`QueryCollection`, () => { // be an active subscription to the query expect(collection.subscriberCount).toBe(0) expect(collection.status).toBe(`ready`) + expect(queryClient.getQueryCache().find({ queryKey })).toBeDefined() // Add explicit subscribers to test cleanup with active subscribers const subscription1 = collection.subscribeChanges(() => {}) @@ -1299,27 +2263,13 @@ describe(`QueryCollection`, () => { // Cleanup the collection which should trigger sync cleanup await collection.cleanup() - // Wait a bit to ensure all async operations complete - await flushPromises() - - // Verify collection status expect(collection.status).toBe(`cleaned-up`) - - // Verify that cleanup methods are called regardless of subscriber state - expect(cancelQueriesSpy).toHaveBeenCalledWith({ - queryKey, - exact: true, - }) - expect(removeQueriesSpy).toHaveBeenCalledWith({ queryKey, exact: true }) + expect(queryClient.getQueryCache().find({ queryKey })).toBeUndefined() // Verify subscribers can be safely cleaned up after collection cleanup subscription1.unsubscribe() subscription2.unsubscribe() expect(collection.subscriberCount).toBe(0) - - // Restore spies - cancelQueriesSpy.mockRestore() - removeQueriesSpy.mockRestore() }) it(`should handle multiple cleanup calls gracefully`, async () => { @@ -1432,12 +2382,6 @@ describe(`QueryCollection`, () => { startSync: true, } - // Spy on queryClient methods - const cancelQueriesSpy = vi - .spyOn(queryClient, `cancelQueries`) - .mockResolvedValue() - const removeQueriesSpy = vi.spyOn(queryClient, `removeQueries`) - const options = queryCollectionOptions(config) const collection = createCollection(options) @@ -1446,90 +2390,480 @@ describe(`QueryCollection`, () => { expect(collection.size).toBe(1) }) - // Cleanup which should call query cleanup methods await collection.cleanup() - await flushPromises() expect(collection.status).toBe(`cleaned-up`) - - // Verify cleanup methods were called - expect(cancelQueriesSpy).toHaveBeenCalledWith({ - queryKey, - exact: true, - }) - expect(removeQueriesSpy).toHaveBeenCalledWith({ queryKey, exact: true }) - - // Clear the spies to track new calls - cancelQueriesSpy.mockClear() - removeQueriesSpy.mockClear() + expect(queryClient.getQueryCache().find({ queryKey })).toBeUndefined() // Restart by accessing collection const subscription = collection.subscribeChanges(() => {}) // Should restart sync expect([`loading`, `ready`]).toContain(collection.status) + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(queryClient.getQueryCache().find({ queryKey })).toBeDefined() + }) // Cleanup again to verify the new sync cleanup works subscription.unsubscribe() await collection.cleanup() - await flushPromises() + expect(queryClient.getQueryCache().find({ queryKey })).toBeUndefined() + }) + + it(`should handle query invalidation and refetch properly`, async () => { + const queryKey = [`invalidation-test`] + let items = [{ id: `1`, name: `Item 1` }] + const queryFn = vi.fn().mockImplementation(() => Promise.resolve(items)) + + const config: QueryCollectionConfig = { + id: `invalidation-test`, + queryClient, + queryKey, + queryFn, + getKey, + startSync: true, + } + + const options = queryCollectionOptions(config) + const collection = createCollection(options) + + // Wait for initial data + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(collection.size).toBe(1) + }) + + // Update data for next fetch + items = [ + { id: `1`, name: `Updated Item 1` }, + { id: `2`, name: `Item 2` }, + ] + + // Invalidate and refetch + await queryClient.invalidateQueries({ queryKey }) + + // Wait for refetch to complete + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(collection.size).toBe(2) + }) + + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Updated Item 1`, + }) + expect(stripVirtualProps(collection.get(`2`))).toEqual({ + id: `2`, + name: `Item 2`, + }) + }) + + describe(`invalidation behavior`, () => { + it(`rematerializes an active eager query after exact invalidation`, async () => { + const queryKey = [`invalidation-exact-eager-test`] + let items: Array = [{ id: `1`, name: `Item 1` }] + const queryFn = vi.fn().mockImplementation(() => Promise.resolve(items)) + + const collection = createCollection( + queryCollectionOptions({ + id: `invalidation-exact-eager-test`, + queryClient, + queryKey, + queryFn, + getKey, + startSync: true, + }), + ) + + try { + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Item 1`, + }) + }) + + items = [{ id: `1`, name: `Updated Item 1` }] + await queryClient.invalidateQueries({ queryKey, exact: true }) + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Updated Item 1`, + }) + }) + } finally { + await collection.cleanup() + } + + expect(queryClient.getQueryCache().find({ queryKey })).toBeUndefined() + }) + + it(`rematerializes an active eager query after prefix invalidation`, async () => { + const rootQueryKey = [`invalidation-prefix-eager-test`] + const queryKey = [...rootQueryKey, `child`] + let items: Array = [{ id: `1`, name: `Item 1` }] + const queryFn = vi.fn().mockImplementation(() => Promise.resolve(items)) + + const collection = createCollection( + queryCollectionOptions({ + id: `invalidation-prefix-eager-test`, + queryClient, + queryKey, + queryFn, + getKey, + startSync: true, + }), + ) + + try { + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Item 1`, + }) + }) + + items = [{ id: `1`, name: `Updated Item 1` }] + await queryClient.invalidateQueries({ queryKey: rootQueryKey }) + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Updated Item 1`, + }) + }) + } finally { + await collection.cleanup() + } + }) + + it(`rematerializes an active on-demand subset after exact invalidation`, async () => { + const queryKey = [`invalidation-exact-on-demand-test`] + let items: Array = [{ id: `1`, name: `Item 1` }] + const queryFn = vi.fn().mockImplementation(() => Promise.resolve(items)) + + const collection = createCollection( + queryCollectionOptions({ + id: `invalidation-exact-on-demand-test`, + queryClient, + queryKey, + queryFn, + getKey, + syncMode: `on-demand`, + }), + ) + const liveQuery = createLiveQueryCollection({ + query: (q) => + q + .from({ item: collection }) + .select(({ item }) => ({ id: item.id, name: item.name })), + }) + + try { + await liveQuery.preload() + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Item 1`, + }) + }) + + items = [{ id: `1`, name: `Updated Item 1` }] + await queryClient.invalidateQueries({ queryKey, exact: true }) + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Updated Item 1`, + }) + }) + } finally { + await liveQuery.cleanup() + } + }) + + it(`rematerializes an active on-demand subset after root prefix invalidation`, async () => { + const queryKey = [`invalidation-prefix-on-demand-test`] + let items: Array = [{ id: `1`, name: `Item 1` }] + const observedQueryKeys: Array> = [] + const queryFn = vi + .fn() + .mockImplementation( + (ctx: QueryFunctionContext>) => { + observedQueryKeys.push(ctx.queryKey) + return Promise.resolve(items) + }, + ) + + const collection = createCollection( + queryCollectionOptions({ + id: `invalidation-prefix-on-demand-test`, + queryClient, + queryKey, + queryFn, + getKey, + syncMode: `on-demand`, + }), + ) + const liveQuery = createLiveQueryCollection({ + query: (q) => + q + .from({ item: collection }) + .where(({ item }) => eq(item.id, `1`)) + .select(({ item }) => ({ id: item.id, name: item.name })), + }) + + try { + await liveQuery.preload() + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(observedQueryKeys[0]?.length).toBeGreaterThan( + queryKey.length, + ) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Item 1`, + }) + }) + + items = [{ id: `1`, name: `Updated Item 1` }] + await queryClient.invalidateQueries({ queryKey }) + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Updated Item 1`, + }) + }) + } finally { + await liveQuery.cleanup() + } + }) + + it(`keeps overlapping on-demand subset rows materialized when one subset is invalidated`, async () => { + const queryKey = [`invalidation-overlap-on-demand-test`] + let firstSubset: Array = [ + { id: `1`, name: `Item 1` }, + { id: `2`, name: `Shared Item` }, + ] + const secondSubset: Array = [ + { id: `2`, name: `Shared Item` }, + { id: `3`, name: `Item 3` }, + ] + const observedQueryKeys: Array> = [] + const queryFn = vi + .fn() + .mockImplementation( + (ctx: QueryFunctionContext>) => { + const firstObservedKey = observedQueryKeys[0] + observedQueryKeys.push(ctx.queryKey) + return Promise.resolve( + firstObservedKey === undefined || + hashKey(ctx.queryKey) === hashKey(firstObservedKey) + ? firstSubset + : secondSubset, + ) + }, + ) + + const collection = createCollection( + queryCollectionOptions({ + id: `invalidation-overlap-on-demand-test`, + queryClient, + queryKey, + queryFn, + getKey, + syncMode: `on-demand`, + }), + ) + const firstLiveQuery = createLiveQueryCollection({ + query: (q) => + q + .from({ item: collection }) + .where(({ item }) => inArray(item.id, [`1`, `2`])) + .select(({ item }) => ({ id: item.id, name: item.name })), + }) + const secondLiveQuery = createLiveQueryCollection({ + query: (q) => + q + .from({ item: collection }) + .where(({ item }) => inArray(item.id, [`2`, `3`])) + .select(({ item }) => ({ id: item.id, name: item.name })), + }) + + try { + await firstLiveQuery.preload() + await secondLiveQuery.preload() + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(collection.size).toBe(3) + }) + + firstSubset = [ + { id: `1`, name: `Updated Item 1` }, + { id: `2`, name: `Updated Shared Item` }, + ] + await queryClient.invalidateQueries({ + queryKey: observedQueryKeys[0], + exact: true, + }) + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(3) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Updated Item 1`, + }) + expect(stripVirtualProps(collection.get(`2`))).toEqual({ + id: `2`, + name: `Updated Shared Item`, + }) + }) + expect(stripVirtualProps(collection.get(`3`))).toEqual({ + id: `3`, + name: `Item 3`, + }) + } finally { + await firstLiveQuery.cleanup() + await secondLiveQuery.cleanup() + } + }) + + it(`retains existing rows when an invalidation refetch fails`, async () => { + const queryKey = [`invalidation-failed-refetch-test`] + const initialItems: Array = [{ id: `1`, name: `Item 1` }] + const refetchError = new Error(`refetch failed`) + const queryFn = vi + .fn() + .mockResolvedValueOnce(initialItems) + .mockRejectedValueOnce(refetchError) + + const collection = createCollection( + queryCollectionOptions({ + id: `invalidation-failed-refetch-test`, + queryClient, + queryKey, + queryFn, + getKey, + startSync: true, + retry: false, + }), + ) + + try { + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Item 1`, + }) + }) - // Verify cleanup methods were called again for the restarted sync - expect(cancelQueriesSpy).toHaveBeenCalledWith({ - queryKey, - exact: true, + await queryClient.invalidateQueries({ queryKey, exact: true }) + + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(collection.utils.lastError).toBe(refetchError) + }) + expect(collection.size).toBe(1) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Item 1`, + }) + } finally { + await collection.cleanup() + } }) - expect(removeQueriesSpy).toHaveBeenCalledWith({ queryKey, exact: true }) - // Restore spies - cancelQueriesSpy.mockRestore() - removeQueriesSpy.mockRestore() - }) + // Retained/persisted invalidation is intentionally not covered here: the + // existing unit fixtures do not exercise the full persisted retention path + // without introducing broader persistence setup. This PR characterizes active, + // inactive, removed, overlapping, and failed-refetch behavior first. + it(`does not refetch a cleaned-up query after invalidation`, async () => { + const retainedQueryClient = new QueryClient({ + defaultOptions: { + queries: { + staleTime: 0, + gcTime: 60_000, + retry: false, + }, + }, + }) + const queryKey = [`invalidation-inactive-cached-test`] + let items: Array = [{ id: `1`, name: `Item 1` }] + const queryFn = vi.fn().mockImplementation(() => Promise.resolve(items)) + + const collection = createCollection( + queryCollectionOptions({ + id: `invalidation-inactive-cached-test`, + queryClient: retainedQueryClient, + queryKey, + queryFn, + getKey, + startSync: true, + }), + ) - it(`should handle query invalidation and refetch properly`, async () => { - const queryKey = [`invalidation-test`] - let items = [{ id: `1`, name: `Item 1` }] - const queryFn = vi.fn().mockImplementation(() => Promise.resolve(items)) + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(collection.size).toBe(1) + }) - const config: QueryCollectionConfig = { - id: `invalidation-test`, - queryClient, - queryKey, - queryFn, - getKey, - startSync: true, - } + await collection.cleanup() + expect(collection.status).toBe(`cleaned-up`) + expect( + retainedQueryClient.getQueryCache().find({ queryKey }), + ).toBeUndefined() - const options = queryCollectionOptions(config) - const collection = createCollection(options) + items = [{ id: `1`, name: `Updated Item 1` }] + await retainedQueryClient.invalidateQueries({ queryKey, exact: true }) - // Wait for initial data - await vi.waitFor(() => { expect(queryFn).toHaveBeenCalledTimes(1) - expect(collection.size).toBe(1) + expect(collection.size).toBe(0) + retainedQueryClient.clear() }) - // Update data for next fetch - items = [ - { id: `1`, name: `Updated Item 1` }, - { id: `2`, name: `Item 2` }, - ] + it(`does not refetch a removed query after invalidation`, async () => { + const queryKey = [`invalidation-removed-query-test`] + let items: Array = [{ id: `1`, name: `Item 1` }] + const queryFn = vi.fn().mockImplementation(() => Promise.resolve(items)) - // Invalidate and refetch - await queryClient.invalidateQueries({ queryKey }) + const collection = createCollection( + queryCollectionOptions({ + id: `invalidation-removed-query-test`, + queryClient, + queryKey, + queryFn, + getKey, + startSync: true, + }), + ) - // Wait for refetch to complete - await vi.waitFor(() => { - expect(queryFn).toHaveBeenCalledTimes(2) - expect(collection.size).toBe(2) - }) + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(collection.size).toBe(1) + }) - expect(stripVirtualProps(collection.get(`1`))).toEqual({ - id: `1`, - name: `Updated Item 1`, - }) - expect(stripVirtualProps(collection.get(`2`))).toEqual({ - id: `2`, - name: `Item 2`, + await collection.cleanup() + queryClient.removeQueries({ queryKey, exact: true }) + expect(queryClient.getQueryCache().find({ queryKey })).toBeUndefined() + + items = [{ id: `1`, name: `Updated Item 1` }] + await queryClient.invalidateQueries({ queryKey, exact: true }) + + expect(queryFn).toHaveBeenCalledTimes(1) + expect(collection.size).toBe(0) }) }) @@ -1719,6 +3053,260 @@ describe(`QueryCollection`, () => { expect(collection.size).toBe(0) }) + describe(`query cancellation and subset cleanup lifecycle`, () => { + const createSubset = (collection: Collection) => + createLiveQueryCollection({ + query: (q) => + q + .from({ item: collection }) + .where(({ item }) => eq(item.id, `1`)) + .select(({ item }) => ({ id: item.id, name: item.name })), + }) + + it(`forwards Query Core's signal and aborts an in-flight eager query on collection cleanup`, async () => { + const deferred = createDeferred>() + let signal: AbortSignal | undefined + const collection = createCollection( + queryCollectionOptions({ + id: `signal-forwarding-cleanup-test`, + queryClient, + queryKey: [`signal-forwarding-cleanup-test`], + queryFn: (context) => { + signal = context.signal + // Reading the signal makes Query Core treat the request as cancellable. + void context.signal.aborted + return deferred.promise + }, + getKey, + startSync: true, + }), + ) + + await vi.waitFor(() => expect(signal).toBeDefined()) + expect(signal?.aborted).toBe(false) + + await collection.cleanup() + + expect(signal?.aborted).toBe(true) + expect(collection.size).toBe(0) + // Query Core may retain the cancelled cache entry, but cleanup releases every observer. + expect( + queryClient + .getQueryCache() + .find({ queryKey: [`signal-forwarding-cleanup-test`] }) + ?.getObserversCount() ?? 0, + ).toBe(0) + }) + + it(`cleans listeners immediately and rejects the abandoned preload before its request settles`, async () => { + const deferred = createDeferred>() + const collection = createCollection( + queryCollectionOptions({ + id: `late-subset-result-test`, + queryClient, + queryKey: [`late-subset-result-test`], + queryFn: () => deferred.promise, + getKey, + syncMode: `on-demand`, + }), + ) + const liveQuery = createSubset(collection) + let preloadError: unknown + const preloadOutcome = liveQuery.preload().then( + () => undefined, + (error: unknown) => { + preloadError = error + return error + }, + ) + + await vi.waitFor(() => expect(queryClient.isFetching()).toBe(1)) + await liveQuery.cleanup() + + const subsetQuery = queryClient.getQueryCache().findAll({ + queryKey: [`late-subset-result-test`], + })[0] + // This assertion runs while the request is unresolved and directly guards the + // ready-listener bookkeeping bug: unload must synchronously detach its observer. + expect(subsetQuery?.getObserversCount() ?? 0).toBe(0) + // Cleanup cancels the caller's wait even while Query Core keeps fetching. + expect(preloadError).toMatchObject({ name: `AbortError` }) + + deferred.resolve([{ id: `1`, name: `Late item` }]) + await vi.waitFor(() => expect(queryClient.isFetching()).toBe(0)) + + expect(collection.size).toBe(0) + expect(await preloadOutcome).toBe(preloadError) + expect(subsetQuery?.getObserversCount() ?? 0).toBe(0) + await collection.cleanup() + }) + + it(`keeps the cleanup error when an abandoned preload's request later rejects`, async () => { + const deferred = createDeferred>() + const collection = createCollection( + queryCollectionOptions({ + id: `late-subset-rejection-test`, + queryClient, + queryKey: [`late-subset-rejection-test`], + queryFn: () => deferred.promise, + getKey, + syncMode: `on-demand`, + }), + ) + const liveQuery = createSubset(collection) + const preloadOutcome = liveQuery.preload().then( + () => undefined, + (error: unknown) => error, + ) + + await vi.waitFor(() => expect(queryClient.isFetching()).toBe(1)) + await liveQuery.cleanup() + + const subsetQuery = queryClient.getQueryCache().findAll({ + queryKey: [`late-subset-rejection-test`], + })[0] + expect(subsetQuery?.getObserversCount() ?? 0).toBe(0) + const preloadError = await preloadOutcome + expect(preloadError).toMatchObject({ name: `AbortError` }) + + deferred.reject(new Error(`Late query failure`)) + await vi.waitFor(() => expect(queryClient.isFetching()).toBe(0)) + + expect(collection.size).toBe(0) + expect(await preloadOutcome).toBe(preloadError) + expect(subsetQuery?.getObserversCount() ?? 0).toBe(0) + await collection.cleanup() + }) + + it(`preserves an active subset across cache removal and accepts a late notification from its detached observer`, async () => { + const queryKey = [`cache-removal-late-notification-test`] + const collection = createCollection( + queryCollectionOptions({ + id: `cache-removal-late-notification-test`, + queryClient, + queryKey, + queryFn: () => Promise.resolve([{ id: `1`, name: `Initial item` }]), + getKey, + syncMode: `on-demand`, + }), + ) + const liveQuery = createSubset(collection) + await liveQuery.preload() + const subsetQuery = queryClient.getQueryCache().findAll({ queryKey })[0] + expect(subsetQuery).toBeDefined() + + // A Query Core `removed` event can arrive before this collection's observer + // is detached. Existing semantics retain the active rows and observer. + queryClient.getQueryCache().remove(subsetQuery!) + expect(queryClient.getQueryCache().findAll({ queryKey })).toHaveLength( + 0, + ) + expect(collection.get(`1`)?.name).toBe(`Initial item`) + expect(subsetQuery!.getObserversCount()).toBe(1) + + // The retained observer can still notify after its query left the cache. + subsetQuery!.setData([{ id: `1`, name: `Late notification` }]) + await vi.waitFor(() => + expect(collection.get(`1`)?.name).toBe(`Late notification`), + ) + + await liveQuery.cleanup() + expect(subsetQuery!.getObserversCount()).toBe(0) + expect(collection.size).toBe(0) + await collection.cleanup() + }) + + it(`deterministically materializes a shared in-flight result after fast subset unmount and remount`, async () => { + const deferred = createDeferred>() + const queryFn = vi.fn(() => deferred.promise) + const collection = createCollection( + queryCollectionOptions({ + id: `fast-subset-remount-test`, + queryClient, + queryKey: [`fast-subset-remount-test`], + queryFn, + getKey, + syncMode: `on-demand`, + }), + ) + const firstLiveQuery = createSubset(collection) + void firstLiveQuery.preload().catch(() => undefined) + await vi.waitFor(() => expect(queryFn).toHaveBeenCalledTimes(1)) + + await firstLiveQuery.cleanup() + const secondLiveQuery = createSubset(collection) + const secondPreload = secondLiveQuery.preload() + deferred.resolve([{ id: `1`, name: `Remounted item` }]) + await secondPreload + + expect(queryFn).toHaveBeenCalledTimes(1) + expect(stripVirtualProps(collection.get(`1`))).toEqual({ + id: `1`, + name: `Remounted item`, + }) + await secondLiveQuery.cleanup() + expect(collection.size).toBe(0) + await collection.cleanup() + }) + + it(`keeps invalidate-unsubscribe-resubscribe compatible while removing stale subset rows and observers`, async () => { + let items: Array = [{ id: `1`, name: `Initial item` }] + const queryFn = vi.fn(() => Promise.resolve(items)) + const collection = createCollection( + queryCollectionOptions({ + id: `subset-invalidation-remount-test`, + queryClient, + queryKey: [`subset-invalidation-remount-test`], + queryFn, + getKey, + syncMode: `on-demand`, + }), + ) + const firstLiveQuery = createSubset(collection) + await firstLiveQuery.preload() + const subsetQuery = queryClient.getQueryCache().findAll({ + queryKey: [`subset-invalidation-remount-test`], + })[0] + expect(subsetQuery).toBeDefined() + + items = [{ id: `1`, name: `Invalidated item` }] + await queryClient.invalidateQueries({ + queryKey: subsetQuery!.queryKey, + exact: true, + }) + await vi.waitFor(() => + expect(collection.get(`1`)?.name).toBe(`Invalidated item`), + ) + + await firstLiveQuery.cleanup() + expect(collection.size).toBe(0) + expect(subsetQuery!.getObserversCount()).toBe(0) + + items = [{ id: `1`, name: `Remounted item` }] + const secondLiveQuery = createSubset(collection) + await secondLiveQuery.preload() + await queryClient.invalidateQueries({ + queryKey: subsetQuery!.queryKey, + exact: true, + }) + await vi.waitFor(() => + expect(collection.get(`1`)?.name).toBe(`Remounted item`), + ) + + await secondLiveQuery.cleanup() + expect(collection.size).toBe(0) + expect( + queryClient + .getQueryCache() + .findAll({ + queryKey: [`subset-invalidation-remount-test`], + })[0] + ?.getObserversCount() ?? 0, + ).toBe(0) + await collection.cleanup() + }) + }) + it(`should maintain data consistency during rapid updates`, async () => { const queryKey = [`rapid-updates-test`] let updateCount = 0 @@ -3053,6 +4641,88 @@ describe(`QueryCollection`, () => { return createCollection(options) } + it.each([`select`, `getKey`, `write`] as const)( + `reports an error when %s throws while applying a successful result`, + async (failureStage) => { + const applicationError = new Error(`${failureStage} failed`) + const consoleErrorSpy = vi + .spyOn(console, `error`) + .mockImplementation(() => {}) + let keyCalls = 0 + const throwingGetKey = (item: TestItem) => { + keyCalls++ + if ( + failureStage === `getKey` || + (failureStage === `write` && keyCalls === 2) + ) { + throw applicationError + } + return item.id + } + + const options = queryCollectionOptions({ + id: `successful-result-${failureStage}-error-test`, + queryClient, + queryKey: [`successful-result-${failureStage}-error-test`], + queryFn: vi.fn().mockResolvedValue([{ id: `1`, name: `Item 1` }]), + getKey: throwingGetKey, + select: + failureStage === `select` + ? () => { + throw applicationError + } + : undefined, + startSync: true, + retry: false, + }) + const collection = createCollection(options) + + await expect(collection.preload()).rejects.toBe(applicationError) + expect(collection.status).toBe(`error`) + expect(collection.utils.lastError).toBe(applicationError) + expect(collection.utils.errorCount).toBe(1) + expect(collection.size).toBe(0) + + await collection.cleanup() + consoleErrorSpy.mockRestore() + }, + ) + + it(`does not treat a failed application as established coverage`, async () => { + const applicationError = new Error(`application failed`) + const consoleErrorSpy = vi + .spyOn(console, `error`) + .mockImplementation(() => {}) + const demand = { where: eq(`id`, `1`) } + const collection = createCollection( + queryCollectionOptions({ + id: `failed-application-coverage`, + queryClient, + queryKey: [`failed-application-coverage`], + queryFn: vi.fn().mockResolvedValue([{ id: `1`, name: `Item 1` }]), + getKey: () => { + throw applicationError + }, + syncMode: `on-demand`, + startSync: true, + retry: false, + }), + ) + + try { + const firstLoad = collection._sync.loadSubset(demand) + await expect(Promise.resolve(firstLoad)).rejects.toBe(applicationError) + + const repeatedLoad = collection._sync.loadSubset(demand) + await expect(Promise.resolve(repeatedLoad)).rejects.toBe( + applicationError, + ) + } finally { + await collection.cleanup() + consoleErrorSpy.mockRestore() + } + }) + it(`should track error state, count, and support recovery`, async () => { const initialData = [{ id: `1`, name: `Item 1` }] const updatedData = [{ id: `1`, name: `Updated Item 1` }] @@ -3241,9 +4911,9 @@ describe(`QueryCollection`, () => { const options = queryCollectionOptions(config) const collection = createCollection(options) - // Wait for collection to be ready (even with error) + // No initial snapshot exists, so the collection reports an error. await vi.waitFor(() => { - expect(collection.status).toBe(`ready`) + expect(collection.status).toBe(`error`) expect(collection.utils.isError).toBe(true) }) @@ -3267,9 +4937,9 @@ describe(`QueryCollection`, () => { queryFn, ) - // Wait for collection to be ready (even with error) + // No initial snapshot exists, so the collection reports an error. await vi.waitFor(() => { - expect(collection.status).toBe(`ready`) + expect(collection.status).toBe(`error`) expect(collection.utils.isError).toBe(true) }) @@ -3313,7 +4983,7 @@ describe(`QueryCollection`, () => { // Wait for all retry attempts to complete and final failure await vi.waitFor( () => { - expect(collection.status).toBe(`ready`) // Should be ready even with error + expect(collection.status).toBe(`error`) expect(queryFn).toHaveBeenCalledTimes(totalAttempts) expect(collection.utils.isError).toBe(true) }, @@ -4302,24 +5972,27 @@ describe(`QueryCollection`, () => { it(`should handle GC correctly when queries are ordered and have a LIMIT`, async () => { const baseQueryKey = [`deduplication-gc-test`] - // Mock queryFn to return different data based on predicates + const items = [ + { id: `1`, name: `Item 1`, category: `A` }, + { id: `2`, name: `Item 2`, category: `A` }, + { id: `3`, name: `Item 3`, category: `A` }, + ] + // Honor the complete pushed predicate so an exact tie request does not + // masquerade as another full category load. const queryFn = vi.fn().mockImplementation((context) => { const { meta } = context const loadSubsetOptions = meta?.loadSubsetOptions ?? {} - const { where, limit } = loadSubsetOptions - - // Query 1: all items with category A (no limit) - if (isCategory(`A`, where)) { - const items = [ - { id: `1`, name: `Item 1`, category: `A` }, - { id: `2`, name: `Item 2`, category: `A` }, - { id: `3`, name: `Item 3`, category: `A` }, - ] - // Slice to limit if provided - return Promise.resolve(limit ? items.slice(0, limit) : items) - } - - return Promise.resolve([]) + const { where, offset = 0, limit } = loadSubsetOptions + + const matching = where + ? items.filter((item) => evaluateReferenceExpression(where, item)) + : items + return Promise.resolve( + matching.slice( + offset, + limit === undefined ? undefined : offset + limit, + ), + ) }) const config: QueryCollectionConfig = { @@ -4389,8 +6062,8 @@ describe(`QueryCollection`, () => { await flushPromises() - // queryFn should have been called twice - // because we do not dedupe the 2nd query + // The initial complete category load already proves that no unseen row + // ties the ordered boundary, so the second demand needs only its prefix. expect(queryFn).toHaveBeenCalledTimes(2) // Collection should still have all 3 items (deduplication doesn't remove data) @@ -4402,22 +6075,201 @@ describe(`QueryCollection`, () => { // GC the first query (all category A without limit) await query1.cleanup() - // Wait for async GC to complete - await vi.waitFor(() => { - expect(collection.size).toBe(2) // Should only have items 1 and 2 because they are still referenced by query 2 + // Wait for async GC to complete + await vi.waitFor(() => { + // Query 2 shares the already-complete category acquisition so it can + // refill locally. It may retain row 3 even though its visible window + // contains only rows 1 and 2. + expect(collection.size).toBe(3) + }) + + expect(collection.has(`1`)).toBe(true) + expect(collection.has(`2`)).toBe(true) + expect(collection.has(`3`)).toBe(true) + + // GC the second query (category A with limit 2) + await query2.cleanup() + + // Wait for final GC to process + await vi.waitFor(() => { + expect(collection.size).toBe(0) + }) + }) + + describe(`ownership lifecycle characterization`, () => { + it(`removes only rows whose final subset owner is unloaded`, async () => { + const queryFn = vi + .fn() + .mockResolvedValueOnce([ + { id: `1`, name: `First only` }, + { id: `2`, name: `Shared` }, + ]) + .mockResolvedValueOnce([ + { id: `2`, name: `Shared` }, + { id: `3`, name: `Second only` }, + ]) + const options = queryCollectionOptions({ + id: `ownership-overlapping-subsets-test`, + queryClient, + queryKey: [`ownership-overlapping-subsets-test`], + queryFn, + getKey, + syncMode: `on-demand`, + }) + const collection = createCollection(options) + const firstSubset = createLiveQueryCollection({ + query: (q) => + q + .from({ item: collection }) + .where(({ item }) => inArray(item.id, [`1`, `2`])), + }) + const secondSubset = createLiveQueryCollection({ + query: (q) => + q + .from({ item: collection }) + .where(({ item }) => inArray(item.id, [`2`, `3`])), + }) + + await firstSubset.preload() + await secondSubset.preload() + expect(collection.size).toBe(3) + + await firstSubset.cleanup() + await vi.waitFor(() => { + expect(collection.has(`1`)).toBe(false) + expect(collection.has(`2`)).toBe(true) + expect(collection.has(`3`)).toBe(true) + }) + await secondSubset.cleanup() + await vi.waitFor(() => { + expect(collection.size).toBe(0) + }) + }) + + it(`expires the Query cache entry after unload without restoring deleted rows`, async () => { + const expiringQueryClient = new QueryClient({ + defaultOptions: { + queries: { gcTime: 100, retry: false, staleTime: Infinity }, + }, + }) + const queryKey = [`ownership-query-cache-expiry-test`] + const collection = createCollection( + queryCollectionOptions({ + id: `ownership-query-cache-expiry-test`, + queryClient: expiringQueryClient, + queryKey, + queryFn: async () => [{ id: `1`, name: `Only row` }], + getKey, + syncMode: `on-demand`, + }), + ) + const liveQuery = createLiveQueryCollection({ + query: (q) => q.from({ item: collection }), + }) + + await liveQuery.preload() + expect(collection.has(`1`)).toBe(true) + + vi.useFakeTimers() + try { + await liveQuery.cleanup() + expect(collection.size).toBe(0) + expect( + expiringQueryClient.getQueryCache().find({ queryKey }), + ).toBeDefined() + + await vi.advanceTimersByTimeAsync(101) + + expect( + expiringQueryClient.getQueryCache().find({ queryKey }), + ).toBeUndefined() + expect(collection.size).toBe(0) + } finally { + vi.useRealTimers() + expiringQueryClient.clear() + } }) - // Verify that only row 3 is removed (it was only referenced by query 1) - expect(collection.has(`1`)).toBe(true) // Still present (referenced by query 2) - expect(collection.has(`2`)).toBe(true) // Still present (referenced by query 2) - expect(collection.has(`3`)).toBe(false) // Removed (only referenced by query 1) + it(`hydrates a retained row and deletes it when its query revalidates empty`, async () => { + const queryKey = [`ownership-retained-hydration-test`] + const queryHash = hashKey(queryKey) + const retainedRow: CategorisedItem = { + id: `1`, + name: `Retained row`, + category: `A`, + } + let releaseRevalidation!: () => void + const revalidationReleased = new Promise((resolve) => { + releaseRevalidation = resolve + }) + const queryFn = vi.fn(async () => { + await revalidationReleased + return [] + }) + const baseOptions = queryCollectionOptions({ + id: `ownership-retained-hydration-test`, + queryClient, + queryKey, + queryFn, + getKey: (item) => item.id, + startSync: true, + }) + const originalSync = baseOptions.sync + const metadataHarness = createInMemorySyncMetadataApi< + string | number, + CategorisedItem + >({ + persistedRows: new Map([[retainedRow.id, retainedRow]]), + rowMetadata: new Map([ + [ + retainedRow.id, + { + queryCollection: { owners: { [queryHash]: true } }, + }, + ], + ]), + collectionMetadata: new Map([ + [ + `queryCollection:gc:${queryHash}`, + { queryHash, mode: `until-revalidated` }, + ], + ]), + }) + const collection = createCollection({ + ...baseOptions, + sync: { + sync: (params: Parameters[0]) => { + params.begin({ immediate: true }) + params.write({ type: `insert`, value: retainedRow }) + params.commit() + return originalSync.sync({ + ...params, + metadata: metadataHarness.api, + }) + }, + }, + }) - // GC the second query (category A with limit 2) - await query2.cleanup() + expect(collection.has(retainedRow.id)).toBe(true) + expect( + metadataHarness.collectionMetadata.has( + `queryCollection:gc:${queryHash}`, + ), + ).toBe(true) - // Wait for final GC to process - await vi.waitFor(() => { - expect(collection.size).toBe(0) + releaseRevalidation() + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + expect(collection.has(retainedRow.id)).toBe(false) + }) + expect(metadataHarness.rowMetadata.get(retainedRow.id)).toBeUndefined() + expect( + metadataHarness.collectionMetadata.has( + `queryCollection:gc:${queryHash}`, + ), + ).toBe(false) + + await collection.cleanup() }) }) @@ -4501,6 +6353,89 @@ describe(`QueryCollection`, () => { }) }) + it(`should not let initial data satisfy persisted revalidation`, async () => { + const queryKey = [`persisted-initial-data-revalidation`] + const queryHash = hashKey(queryKey) + const retainedRow = { + id: `retained`, + name: `Retained`, + category: `A`, + } + const initialRow = { + id: `initial`, + name: `Initial`, + category: `A`, + } + const serverRow = { id: `server`, name: `Server`, category: `A` } + const serverResult = createDeferred>() + const queryFn = vi.fn(() => serverResult.promise) + const adapter = createPersistedQueryAdapter({ + rows: new Map([[retainedRow.id, retainedRow]]), + rowMetadata: new Map([ + [ + retainedRow.id, + { + queryCollection: { + owners: { + [queryHash]: true, + }, + }, + }, + ], + ]), + collectionMetadata: new Map([ + [ + `queryCollection:gc:${queryHash}`, + { + queryHash, + mode: `until-revalidated`, + }, + ], + ]), + }) + + const collection = createCollection( + persistedCollectionOptions({ + ...(queryCollectionOptions({ + id: `persisted-initial-data-revalidation`, + queryClient, + queryKey, + queryFn, + getKey: (item) => item.id, + initialData: [initialRow], + staleTime: Infinity, + syncMode: `eager`, + startSync: true, + }) as any), + persistence: { + adapter, + }, + }) as any, + ) + + try { + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(1) + }) + expect(adapter.rows.has(retainedRow.id)).toBe(true) + expect(adapter.rows.has(initialRow.id)).toBe(false) + expect( + adapter.collectionMetadata.has(`queryCollection:gc:${queryHash}`), + ).toBe(true) + + serverResult.resolve([serverRow]) + await vi.waitFor(() => { + expect(adapter.rows.has(retainedRow.id)).toBe(false) + expect(adapter.rows.get(serverRow.id)).toEqual(serverRow) + expect( + adapter.collectionMetadata.has(`queryCollection:gc:${queryHash}`), + ).toBe(false) + }) + } finally { + await collection.cleanup() + } + }) + it(`should diff against retained query-owned rows on warm start`, async () => { const baseQueryKey = [`persisted-baseline-test`] const queryFn = vi.fn().mockResolvedValue([]) @@ -4578,6 +6513,65 @@ describe(`QueryCollection`, () => { ).toBe(false) }) + it(`does not apply a retained-query result after its subset is released`, async () => { + const queryKey = [`stale-retained-reconciliation`] + const queryHash = hashKey(queryKey) + const item = { id: `1`, name: `Stale result`, category: `A` } + const persistedScan = + createDeferred< + Array<{ key: string; value: CategorisedItem; metadata?: unknown }> + >() + const metadataHarness = createInMemorySyncMetadataApi({ + collectionMetadata: new Map([ + [ + `queryCollection:gc:${queryHash}`, + { queryHash, mode: `until-revalidated` }, + ], + ]), + }) + const scanPersisted = vi.fn().mockReturnValue(persistedScan.promise) + const metadataApi = { + ...metadataHarness.api, + row: { + ...metadataHarness.api.row, + scanPersisted, + }, + } as SyncMetadataApi + + const baseOptions = queryCollectionOptions({ + id: `stale-retained-reconciliation`, + queryClient, + queryKey: () => queryKey, + queryFn: async () => [item], + getKey: (value) => value.id, + syncMode: `on-demand`, + startSync: true, + }) + const originalSync = baseOptions.sync + const collection = createCollection({ + ...baseOptions, + sync: { + sync: (params: Parameters[0]) => + originalSync.sync({ + ...params, + metadata: metadataApi, + }), + }, + }) + const load = collection._sync.loadSubset({}) + await vi.waitFor(() => { + expect(scanPersisted).toHaveBeenCalledOnce() + }) + + collection._sync.unloadSubset({}) + persistedScan.resolve([]) + await load + await flushPromises() + + expect(collection.has(item.id)).toBe(false) + await collection.cleanup() + }) + it(`should clean up expired persisted ttl placeholders on startup`, async () => { const baseQueryKey = [`persisted-ttl-cleanup-test`] const queryFn = vi.fn().mockResolvedValue([]) @@ -4888,6 +6882,113 @@ describe(`QueryCollection`, () => { ).toBe(false) }) + it(`should clean up an inserted row dropped by the query after a reload`, async () => { + // A row inserted while the collection is mounted is persisted. After the + // app reloads, if the query no longer returns that row, it must be removed + // from both the live collection and the persisted store. + const queryKey = [`reload-insert-cleanup`] + const makeQueryClient = () => + new QueryClient({ + defaultOptions: { + queries: { staleTime: 0, gcTime: 5 * 60 * 1000, retry: false }, + }, + }) + + // The shared on-device store, surviving across the two sessions. + const adapter = createPersistedQueryAdapter({}) + + // ---- First session: insert an item that the server then returns ---- + let serverRows: Array = [] + const firstQueryClient = makeQueryClient() + const collection1 = createCollection( + persistedCollectionOptions({ + ...(queryCollectionOptions({ + id: `reload-insert-cleanup`, + queryClient: firstQueryClient, + queryKey, + queryFn: async () => serverRows, + getKey: (item: CategorisedItem): string => item.id, + syncMode: `eager`, + startSync: true, + onInsert: async ({ transaction }) => { + // The mutation reaches the server: the item now appears in the + // API response for subsequent fetches. + for (const mutation of transaction.mutations) { + serverRows = [...serverRows, mutation.modified] + } + }, + }) as any), + persistence: { adapter }, + }) as any, + ) + + await collection1.stateWhenReady() + await flushPromises() + + await collection1.insert({ id: `1`, name: `Buy milk`, category: `A` }) + // The query refetches and sees the now-synced row. + await firstQueryClient.invalidateQueries({ queryKey }) + await flushPromises() + await flushPromises() + + expect(adapter.rows.has(`1`)).toBe(true) + + // ---- App closes; the row is removed on the server out of band ---- + serverRows = [] + + // ---- Second session: reload from the persisted store ---- + const secondQueryClient = makeQueryClient() + const adapter2 = createPersistedQueryAdapter({ + rows: adapter.rows, + rowMetadata: adapter.rowMetadata, + collectionMetadata: adapter.collectionMetadata, + }) + let releaseSecondFetch!: () => void + const secondFetchReleased = new Promise((resolve) => { + releaseSecondFetch = resolve + }) + const collection2 = createCollection( + persistedCollectionOptions({ + ...(queryCollectionOptions({ + id: `reload-insert-cleanup`, + queryClient: secondQueryClient, + queryKey, + queryFn: async () => { + await secondFetchReleased + return serverRows + }, + getKey: (item: CategorisedItem): string => item.id, + syncMode: `eager`, + startSync: true, + }) as any), + persistence: { adapter: adapter2 }, + }) as any, + ) + + await vi.waitFor(() => { + expect(collection2.has(`1`)).toBe(true) + expect(adapter2.rows.has(`1`)).toBe(true) + }) + + const liveQuery = createLiveQueryCollection({ + query: (q) => q.from({ item: collection2 }), + }) + releaseSecondFetch() + await liveQuery.preload() + // The query responds on launch (after persisted rows have hydrated). + await flushPromises() + await flushPromises() + + await vi.waitFor(() => { + expect(collection2.has(`1`)).toBe(false) + expect(adapter2.rows.has(`1`)).toBe(false) + expect(liveQuery.size).toBe(0) + }) + + firstQueryClient.clear() + secondQueryClient.clear() + }) + it(`should expire retained ttl placeholders while the app stays open`, async () => { vi.useFakeTimers() try { @@ -4967,6 +7068,60 @@ describe(`QueryCollection`, () => { } }) + it(`should clear retained ownership during explicit collection cleanup`, async () => { + const baseQueryKey = [`explicit-retained-cleanup-test`] + const retainedQueryHash = hashKey(baseQueryKey) + const items: Array = [ + { id: `1`, name: `Retained`, category: `A` }, + ] + const config: QueryCollectionConfig = { + id: `explicit-retained-cleanup-test`, + queryClient, + queryKey: () => baseQueryKey, + queryFn: async () => items, + getKey: (item) => item.id, + syncMode: `on-demand`, + startSync: true, + persistedGcTime: 60_000, + } + const baseOptions = queryCollectionOptions(config) + const originalSync = baseOptions.sync + const metadataHarness = createInMemorySyncMetadataApi< + string | number, + CategorisedItem + >({ persistedRows: new Map(items.map((item) => [item.id, item])) }) + const collection = createCollection({ + ...baseOptions, + sync: { + sync: (params: Parameters[0]) => + originalSync.sync({ ...params, metadata: metadataHarness.api }), + }, + }) + const liveQuery = createLiveQueryCollection({ + query: (q) => + q + .from({ item: collection }) + .where(({ item }) => eq(item.category, `A`)), + }) + + await liveQuery.preload() + await liveQuery.cleanup() + expect( + metadataHarness.collectionMetadata.has( + `queryCollection:gc:${retainedQueryHash}`, + ), + ).toBe(true) + + await collection.cleanup() + + expect(collection.size).toBe(0) + expect( + metadataHarness.collectionMetadata.has( + `queryCollection:gc:${retainedQueryHash}`, + ), + ).toBe(false) + }) + it(`should default persisted retention ttl to query gcTime when persistedGcTime is undefined`, async () => { vi.useFakeTimers() const gcTime = 120 @@ -5220,13 +7375,17 @@ describe(`QueryCollection`, () => { expect(collection.size).toBe(2) }) - // Force GC by calling removeQueries (simulates gcTime expiry) + // Release the first acquisition before its cache entry is removed. + // Cache events do not revoke active collection ownership. + await query1.cleanup() + await vi.waitFor(() => { + expect(collection.size).toBe(0) + }) + + // Force GC by calling removeQueries (simulates gcTime expiry). queryClient.removeQueries({ queryKey: baseQueryKey }) await flushPromises() - // BUG: queryRefCounts still has stale count, wasn't cleaned up by cleanupQuery - // When we load again, the refcount will be wrong (starts at 1 instead of 0, or accumulates) - // Reload the same query const query2 = createLiveQueryCollection({ query: (q) => @@ -5243,14 +7402,11 @@ describe(`QueryCollection`, () => { expect(collection.size).toBe(2) }) - // Cleanup - this should properly decrement from 1 to 0 and clean up + // Cleanup should decrement the new acquisition from one to zero. await query2.cleanup() await vi.waitFor(() => { expect(collection.size).toBe(0) // Should be cleaned up }) - - // BUG SYMPTOM: If refcount was stale (e.g. was 2, decremented to 1), - // the observer won't be destroyed and data won't be cleaned up }) it(`should handle mount/unmount/remount without breaking cache (destroyed observer bug)`, async () => { @@ -5692,6 +7848,76 @@ describe(`QueryCollection`, () => { }) }) + describe(`On-demand query persistence`, () => { + const containsFunction = (value: unknown): boolean => { + if (typeof value === `function`) { + return true + } + + if (Array.isArray(value)) { + return value.some(containsFunction) + } + + if (value && typeof value === `object`) { + return Object.values(value as Record).some( + containsFunction, + ) + } + + return false + } + + it(`should keep dehydrated query state structured-clone safe after loading an on-demand subset with subscription state`, async () => { + const queryClient = new QueryClient() + const items: Array = [ + { id: `1`, name: `Item 1`, category: `A` }, + { id: `2`, name: `Item 2`, category: `B` }, + ] + + const collection = createCollection( + queryCollectionOptions({ + id: `on-demand-persistence-clone-safe-test`, + queryClient, + queryKey: [`on-demand-persistence-clone-safe-test`], + queryFn: vi.fn().mockResolvedValue(items), + getKey: (item) => item.id, + syncMode: `on-demand`, + startSync: true, + }), + ) + + try { + await collection._sync.loadSubset({ + where: eq(`category`, `A`), + subscription: { + options: { + onUnsubscribe: () => {}, + }, + } as unknown as NonNullable, + }) + + const cachedQuery = queryClient.getQueryCache().findAll()[0] + expect(cachedQuery?.meta?.loadSubsetOptions).toBeDefined() + expect( + cachedQuery?.meta?.loadSubsetOptions?.subscription, + ).toBeUndefined() + + const dehydrated = dehydrate(queryClient) + expect(dehydrated.queries.length).toBeGreaterThan(0) + expect(() => structuredClone(dehydrated)).not.toThrow() + + for (const query of dehydrated.queries) { + expect(containsFunction(query.meta)).toBe(false) + expect( + containsFunction(query.state.data as Record), + ).toBe(false) + } + } finally { + queryClient.clear() + } + }) + }) + describe(`Static queryKey with on-demand mode`, () => { it(`should automatically append serialized predicates to static queryKey in on-demand mode`, async () => { const items: Array = [ diff --git a/packages/query-db-collection/tests/server-pagination-boundary.test.ts b/packages/query-db-collection/tests/server-pagination-boundary.test.ts new file mode 100644 index 0000000000..dc6b236c40 --- /dev/null +++ b/packages/query-db-collection/tests/server-pagination-boundary.test.ts @@ -0,0 +1,21 @@ +import { describe, expect, it } from 'vitest' +import { createServerPaginationFixture } from './server-pagination-fixture' + +describe(`manual server page ownership`, () => { + it(`an explicit refetch replaces directly appended eager rows despite infinite stale time`, async () => { + const fixture = createServerPaginationFixture({ + rows: [{ id: 1, rank: 1 }], + syncMode: `eager`, + }) + try { + await fixture.collection.preload() + fixture.collection.utils.writeUpsert({ id: 2, rank: 2 }) + expect([...fixture.collection.keys()]).toEqual([1, 2]) + await fixture.collection.utils.refetch() + expect([...fixture.collection.keys()]).toEqual([1]) + } finally { + await fixture.collection.cleanup() + fixture.client.clear() + } + }) +}) diff --git a/packages/query-db-collection/tests/server-pagination-fixture.ts b/packages/query-db-collection/tests/server-pagination-fixture.ts new file mode 100644 index 0000000000..3828c27f8c --- /dev/null +++ b/packages/query-db-collection/tests/server-pagination-fixture.ts @@ -0,0 +1,75 @@ +import { QueryClient } from '@tanstack/query-core' +import { BTreeIndex, createCollection } from '@tanstack/db' +import { queryCollectionOptions } from '../src/index' +import { evaluateReferenceExpression } from '../../db/tests/reference-expression' +import type { LoadSubsetOptions } from '@tanstack/db' + +export type ServerRow = { id: number; rank: number } + +// An ordinary QueryObserver, not InfiniteQueryObserver. This fixture models +// numeric rows supplied in the query's requested order; it is not a general sorter. +// The endpoint either fulfills the request or returns one nonconforming cap. +export function createServerPaginationFixture(options: { + syncMode: `eager` | `on-demand` + rows: Array + cap?: number + serverPageSize?: number +}) { + const client = new QueryClient({ + defaultOptions: { queries: { retry: false, staleTime: Infinity } }, + }) + const requests: Array<{ + pageParam: unknown + subset: LoadSubsetOptions | undefined + }> = [] + const serverPages: Array = [] + const collection = createCollection( + queryCollectionOptions({ + queryClient: client, + queryKey: [`server-pagination-probe`], + syncMode: options.syncMode, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + getKey: (row: ServerRow) => row.id, + queryFn: async (context): Promise> => { + requests.push({ + pageParam: `pageParam` in context ? context.pageParam : undefined, + subset: context.meta?.loadSubsetOptions, + }) + const subset = context.meta?.loadSubsetOptions + const start = subset?.offset ?? 0 + const limit = options.cap ?? subset?.limit ?? options.rows.length + const matching = options.rows.filter((row) => + subset?.where + ? evaluateReferenceExpression(subset.where, row) === true + : true, + ) + if (options.serverPageSize !== undefined) { + const pageSize = options.serverPageSize + const firstPage = Math.floor(start / pageSize) + const prefixSkip = start % pageSize + const gathered: Array = [] + let page: number | undefined = firstPage + while (page !== undefined && gathered.length < prefixSkip + limit) { + serverPages.push(page) + // Server authority stays inside the adapter. Query DB receives only + // the completed row array, not this endpoint-specific continuation. + const response: { + rows: Array + nextPage: number | undefined + } = await Promise.resolve({ + rows: matching.slice(page * pageSize, (page + 1) * pageSize), + nextPage: + (page + 1) * pageSize < matching.length ? page + 1 : undefined, + }) + gathered.push(...response.rows) + page = response.nextPage + } + return gathered.slice(prefixSkip, prefixSkip + limit) + } + return matching.slice(start, start + limit) + }, + }), + ) + return { collection, requests, serverPages, client } +} diff --git a/packages/react-db/CHANGELOG.md b/packages/react-db/CHANGELOG.md index d8d077ea50..f75c431839 100644 --- a/packages/react-db/CHANGELOG.md +++ b/packages/react-db/CHANGELOG.md @@ -1,5 +1,192 @@ # @tanstack/react-db +## 0.3.8 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 0.3.7 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 0.3.6 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 0.3.5 + +### Patch Changes + +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 0.3.4 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 0.3.3 + +### Patch Changes + +- Support disabling live queries declared with the `{ query }` config syntax by returning `undefined` or `null` from the query callback. ([#1757](https://github.com/TanStack/db/pull/1757)) + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 0.3.2 + +### Patch Changes + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 0.3.1 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 0.3.0 + +### Minor Changes + +- Add SSR through request-scoped `DbClient` instances, collection descriptors, ([#1564](https://github.com/TanStack/db/pull/1564)) + explicit collection-row hydration, live-query result snapshots, adapter sync + metadata, and React and Svelte descriptor resolution. + + React live queries now derive identity from structured query IR. Opaque queries + can provide `queryKey`; legacy dependency arrays and unkeyed opaque queries keep + working with development warnings until 1.0. + + Add TanStack Router integration that streams live queries discovered during a + Suspense render as pending promises which resolve to ordered result snapshots. + The browser starts normal source sync and atomically replaces the snapshot when + its live result is ready. + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 0.2.1 + +### Patch Changes + +- Update agent skills to match current APIs and behavior. ([#1696](https://github.com/TanStack/db/pull/1696)) + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 0.2.0 + +### Minor Changes + +- Add `useLiveInfiniteQuery` as a Vue binding over the shared live-query window controller. Align infinite-query behavior across React, Vue, and Svelte, including awaitable page fetches, safe page sizes, reactive page-depth preservation, ordered collection validation, shared input resolution, and shared-window cleanup. ([#1724](https://github.com/TanStack/db/pull/1724)) + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 0.1.96 + +### Patch Changes + +- Add an internal shared live-query observer and migrate all five framework adapters to it ([#1642](https://github.com/TanStack/db/pull/1642)) + + Introduces `createLiveQueryObserver` in `@tanstack/db`: given a resolved live-query collection (or `null` for a disabled query) it owns the subscription lifecycle every adapter used to re-implement — change and status subscriptions, a snapshot with stable identity per state revision for wholesale consumers, and delivery of the raw `ChangeMessage[]` for granular consumers. React, Vue, Svelte, Solid, and Angular's live-query hooks now materialize from the observer instead of their own hand-rolled subscription/status/snapshot machinery, keeping each adapter's native reactivity and each adapter's data-loading policy (wholesale adapters subscribe without initial state; granular adapters seed from it). + + The observer is an **internal, unstable contract** for TanStack DB's official adapters — it is exported so the adapter packages can consume it, but it is not a public extension point yet and its API may change in any release. + + The migration also fixes several live-query lifecycle defects: status-only transitions (`error`, `cleaned-up`) now reach mounted consumers; snapshot identity is stable across unsubscribe/resubscribe and stays fresh while detached; dispatch is FIFO and non-reentrant with subscriptions identified by record rather than callback; disposing during the synchronous initial replay no longer leaks the collection subscription; subscribing after dispose throws instead of registering a dead listener; Solid guards its async resource continuations against superseded collections; and constructing an observer no longer activates sync. Observers activate on their first committed subscription unless an adapter has already started a pre-created collection supplied directly or returned from a callback. + +- Add the unstable, internal `createLiveQueryWindowController` primitive for ([#1675](https://github.com/TanStack/db/pull/1675)) + forward pagination. It coordinates collection-scoped window leases, commits + pages only after subset loads succeed, restores windows after failures and + cleanup, and lets React's `useLiveInfiniteQuery` become a thin binding without + changing its public API or resetting pages for structurally equal dependencies. +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 0.1.95 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 0.1.94 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 0.1.93 + +### Patch Changes + +- Extract shared live-query adapter helpers into `@tanstack/db` ([#1641](https://github.com/TanStack/db/pull/1641)) + + Adds `isCollection`, `isSingleResultCollection`, and `getLiveQueryStatusFlags` to `@tanstack/db` and migrates all five framework adapters to use them. `isCollection` replaces the per-adapter duck-typing and Solid's `instanceof CollectionImpl` with one structural, multi-realm-safe guard (the `instanceof` form gave false negatives across dual-package boundaries). No behavior change; internal deduplication only. + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 0.1.92 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 0.1.91 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 0.1.90 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 0.1.89 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 0.1.88 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 0.1.87 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 0.1.86 ### Patch Changes diff --git a/packages/react-db/README.md b/packages/react-db/README.md index 12c86adeff..c1637d17ca 100644 --- a/packages/react-db/README.md +++ b/packages/react-db/README.md @@ -1,3 +1,32 @@ +
                      + + + + TanStack React DB + +
                      # @tanstack/react-db React hooks for TanStack DB. See [TanStack/db](https://github.com/TanStack/db) for more details. + +```tsx +import { useLiveQuery } from '@tanstack/react-db' + +function TodoList() { + const { data: todos } = useLiveQuery({ + query: (q) => q.from({ todo: todoCollection }), + }) + + return todos.map((todo) =>
                      {todo.text}
                      ) +} +``` diff --git a/packages/react-db/package.json b/packages/react-db/package.json index 65f6f2f315..8dc99a60f8 100644 --- a/packages/react-db/package.json +++ b/packages/react-db/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/react-db", - "version": "0.1.86", + "version": "0.3.8", "description": "React integration for @tanstack/db", "author": "Kyle Mathews", "license": "MIT", diff --git a/packages/react-db/skills/react-db/SKILL.md b/packages/react-db/skills/react-db/SKILL.md index fb864ac8f7..273fad8cba 100644 --- a/packages/react-db/skills/react-db/SKILL.md +++ b/packages/react-db/skills/react-db/SKILL.md @@ -1,9 +1,10 @@ --- name: react-db description: > - React bindings for TanStack DB. useLiveQuery hook with dependency arrays - (8 overloads: query function, config object, pre-created collection, - disabled state via returning undefined/null). useLiveSuspenseQuery for + React bindings for TanStack DB. Prefer useLiveQuery({ query }) with + derived structured query identity. Provide queryKey only for opaque + functional query variants or very hot render paths. Dependency arrays are + legacy and warn before 1.0 removal. useLiveSuspenseQuery for React Suspense with Error Boundaries (data always defined). useLiveInfiniteQuery for cursor-based pagination (pageSize, fetchNextPage, hasNextPage, isFetchingNextPage). usePacedMutations for debounced React @@ -13,7 +14,7 @@ description: > type: framework library: db framework: react -library_version: '0.6.0' +library_version: '0.6.17' requires: - db-core sources: @@ -30,15 +31,16 @@ This skill builds on db-core. Read it first for collection setup, query builder, ## Setup ```tsx -import { useLiveQuery, eq, not } from '@tanstack/react-db' +import { eq, not, useLiveQuery } from '@tanstack/react-db' function TodoList() { - const { data: todos, isLoading } = useLiveQuery((q) => - q - .from({ todo: todoCollection }) - .where(({ todo }) => not(todo.completed)) - .orderBy(({ todo }) => todo.created_at, 'asc'), - ) + const { data: todos, isLoading } = useLiveQuery({ + query: (q) => + q + .from({ todo: todoCollection }) + .where(({ todo }) => not(todo.completed)) + .orderBy(({ todo }) => todo.created_at, 'asc'), + }) if (isLoading) return
                      Loading...
                      @@ -59,7 +61,7 @@ function TodoList() { ### useLiveQuery ```tsx -// Query function with dependency array +// Preferred config object with derived query identity const { data, state, @@ -70,15 +72,14 @@ const { isError, isIdle, isCleanedUp, -} = useLiveQuery( - (q) => +} = useLiveQuery({ + query: (q) => q .from({ todo: todoCollection }) .where(({ todo }) => eq(todo.userId, userId)), - [userId], -) +}) -// Config object +// Static query const { data } = useLiveQuery({ query: (q) => q.from({ todo: todoCollection }), gcTime: 60000, @@ -87,16 +88,15 @@ const { data } = useLiveQuery({ // Pre-created collection (from route loader) const { data } = useLiveQuery(preloadedCollection) -// Conditional query — return undefined/null to disable -const { data, status } = useLiveQuery( - (q) => { +// Conditional query — derived identity handles enabled/disabled transitions +const { data, status } = useLiveQuery({ + query: (q) => { if (!userId) return undefined return q .from({ todo: todoCollection }) .where(({ todo }) => eq(todo.userId, userId)) }, - [userId], -) +}) // When disabled: status='disabled', data=undefined ``` @@ -106,9 +106,9 @@ const { data, status } = useLiveQuery( // data is ALWAYS defined — never undefined // Must wrap in and function TodoList() { - const { data: todos } = useLiveSuspenseQuery((q) => - q.from({ todo: todoCollection }), - ) + const { data: todos } = useLiveSuspenseQuery({ + query: (q) => q.from({ todo: todoCollection }), + }) return (
                        @@ -119,14 +119,13 @@ function TodoList() { ) } -// With deps — re-suspends when deps change -const { data } = useLiveSuspenseQuery( - (q) => +// Structured captured values are part of the derived identity and re-suspend when changed +const { data } = useLiveSuspenseQuery({ + query: (q) => q .from({ todo: todoCollection }) .where(({ todo }) => eq(todo.category, category)), - [category], -) +}) ``` ### useLiveInfiniteQuery @@ -137,9 +136,11 @@ const { data, fetchNextPage, hasNextPage, isFetchingNextPage } = (q) => q .from({ posts: postsCollection }) + .where(({ posts }) => eq(posts.category, category)) .orderBy(({ posts }) => posts.createdAt, 'desc'), - { pageSize: 20 }, - [category], + { + pageSize: 20, + }, ) // data is the flat array of all loaded pages @@ -174,16 +175,17 @@ When a query uses includes (subqueries in `select`), each child field is a live ```tsx function ProjectList() { - const { data: projects } = useLiveQuery((q) => - q.from({ p: projectsCollection }).select(({ p }) => ({ - id: p.id, - name: p.name, - issues: q - .from({ i: issuesCollection }) - .where(({ i }) => eq(i.projectId, p.id)) - .select(({ i }) => ({ id: i.id, title: i.title })), - })), - ) + const { data: projects } = useLiveQuery({ + query: (q) => + q.from({ p: projectsCollection }).select(({ p }) => ({ + id: p.id, + name: p.name, + issues: q + .from({ i: issuesCollection }) + .where(({ i }) => eq(i.projectId, p.id)) + .select(({ i }) => ({ id: i.id, title: i.title })), + })), + }) return (
                          @@ -217,19 +219,20 @@ With `toArray()`, child results are plain arrays and the parent re-renders on ch ```tsx import { toArray, eq } from '@tanstack/react-db' -const { data: projects } = useLiveQuery((q) => - q.from({ p: projectsCollection }).select(({ p }) => ({ - id: p.id, - name: p.name, - issues: toArray( - q - .from({ i: issuesCollection }) - .where(({ i }) => eq(i.projectId, p.id)) - .select(({ i }) => ({ id: i.id, title: i.title })), - ), - })), -) -// project.issues is string[] — no subcomponent needed +const { data: projects } = useLiveQuery({ + query: (q) => + q.from({ p: projectsCollection }).select(({ p }) => ({ + id: p.id, + name: p.name, + issues: toArray( + q + .from({ i: issuesCollection }) + .where(({ i }) => eq(i.projectId, p.id)) + .select(({ i }) => ({ id: i.id, title: i.title })), + ), + })), +}) +// project.issues is Array<{ id: string; title: string }> — no subcomponent needed ``` See db-core/live-queries/SKILL.md for full includes rules (correlation conditions, nested includes, aggregates). @@ -238,7 +241,8 @@ See db-core/live-queries/SKILL.md for full includes rules (correlation condition Live query results include computed, read-only virtual properties on every row: -- `$synced`: `true` when the row is confirmed by sync; `false` when it is still optimistic. +- `$synced`: `true` when no pending local optimistic write affects the row; + `false` while one does. It does not prove backend confirmation. - `$origin`: `"local"` if the last confirmed change came from this client, otherwise `"remote"`. - `$key`: the row key for the result. - `$collectionId`: the source collection ID. @@ -246,38 +250,54 @@ Live query results include computed, read-only virtual properties on every row: These props are added automatically and can be used in `where`, `select`, and `orderBy` clauses. Do not persist them back to storage. ```tsx -const { data } = useLiveQuery( - (q) => +const { data } = useLiveQuery({ + query: (q) => q .from({ todo: todoCollection }) .where(({ todo }) => eq(todo.$synced, false)), - [], -) -// Shows only optimistic (unconfirmed) todos +}) +// Shows rows with pending local optimistic writes ``` ## React-Specific Patterns -### Dependency arrays +### Query identity ```tsx -// Include ALL external reactive values -const { data } = useLiveQuery( - (q) => +// Structured captured values are included in the derived identity +const { data } = useLiveQuery({ + query: (q) => q .from({ todo: todoCollection }) .where(({ todo }) => and(eq(todo.userId, userId), eq(todo.status, filter)), ), - [userId, filter], -) +}) -// Empty array = static query, never re-runs -const { data } = useLiveQuery((q) => q.from({ todo: todoCollection }), []) +// Static query +const { data } = useLiveQuery({ + query: (q) => q.from({ todo: todoCollection }), +}) +``` + +Use `queryKey` only when DB cannot derive identity from structured IR, such as +`.fn.where`, `.fn.select`, `.fn.having`, or as a deliberate performance escape +hatch on a hot render path: -// No array = re-runs on every render (usually wrong) +```tsx +const { data } = useLiveQuery({ + queryKey: [todoCollection.id, 'search', search], + query: (q) => + q.from({ todo: todoCollection }).fn.where(({ todo }) => { + return fuzzyMatch(todo.title, search) + }), +}) ``` +Before 1.0, opaque IR warns and keeps legacy mount-stable identity. Slow or +repeated derived identity work also warns once. Both point to the same +`queryKey` escape hatch; unhashable IR without a key will throw in 1.0. + ### Suspense + Error Boundary ```tsx @@ -295,36 +315,42 @@ const { data } = useLiveQuery((q) => q.from({ todo: todoCollection }), []) await todoCollection.preload() // In component — data available immediately: -const { data } = useLiveQuery((q) => q.from({ todo: todoCollection })) +const { data } = useLiveQuery({ + query: (q) => q.from({ todo: todoCollection }), +}) ``` See meta-framework/SKILL.md for full preloading patterns. ## Common Mistakes -### CRITICAL Missing external values in dependency array +### CRITICAL Using opaque query logic without queryKey Wrong: ```tsx -const { data } = useLiveQuery((q) => - q.from({ todo: todoCollection }).where(({ todo }) => eq(todo.userId, userId)), -) +const { data } = useLiveQuery({ + query: (q) => + q.from({ todo: todoCollection }).fn.where(({ todo }) => { + return fuzzyMatch(todo.title, search) + }), +}) ``` Correct: ```tsx -const { data } = useLiveQuery( - (q) => - q - .from({ todo: todoCollection }) - .where(({ todo }) => eq(todo.userId, userId)), - [userId], -) +const { data } = useLiveQuery({ + queryKey: [todoCollection.id, 'search', search], + query: (q) => + q.from({ todo: todoCollection }).fn.where(({ todo }) => { + return fuzzyMatch(todo.title, search) + }), +}) ``` -When the query uses external state not in the deps array, the query won't re-run when that value changes, showing stale results. +Structured expressions are hashable by default. Functional query variants are +opaque runtime code, so they need an explicit key to say when identity changes. Source: docs/framework/react/overview.md @@ -354,7 +380,12 @@ Source: docs/guides/live-queries.md ### HIGH "Not a Collection" error from duplicate @tanstack/db -If `useLiveQuery` throws `InvalidSourceError: The value provided for alias "todo" is not a Collection`, it usually means two copies of `@tanstack/db` are installed. The collection was created by one copy, but `useLiveQuery` checks `instanceof` against the other. +If a query-builder alias throws +`InvalidSourceError: The value provided for alias "todo" is not a Collection`, +it can mean two copies of `@tanstack/db` are installed. Direct +`useLiveQuery(preCreatedCollection)` detection is structural and works across +package copies or realms, but `q.from({ todo: collection })` still validates +the source with the core collection class. In dev mode, TanStack DB also throws `DuplicateDbInstanceError` if two instances are detected. @@ -372,7 +403,7 @@ If multiple versions appear, fix with one of: { "pnpm": { "overrides": { - "@tanstack/db": "^0.6.0" + "@tanstack/db": "^0.6.17" } } } diff --git a/packages/react-db/src/DbProvider.tsx b/packages/react-db/src/DbProvider.tsx new file mode 100644 index 0000000000..f817dd1a2a --- /dev/null +++ b/packages/react-db/src/DbProvider.tsx @@ -0,0 +1,32 @@ +'use client' + +import { createContext, useContext } from 'react' +import type { DbClient } from '@tanstack/db' +import type { ReactNode } from 'react' + +const DbContext = createContext(undefined) + +export type DbProviderProps = { + client: DbClient + children?: ReactNode +} + +export function DbProvider(props: DbProviderProps) { + return ( + + {props.children} + + ) +} + +export function useDbClient(): DbClient { + const client = useContext(DbContext) + if (!client) { + throw new Error(`useDbClient must be used within a DbProvider.`) + } + return client +} + +export function useOptionalDbClient(): DbClient | undefined { + return useContext(DbContext) +} diff --git a/packages/react-db/src/HydrationBoundary.tsx b/packages/react-db/src/HydrationBoundary.tsx new file mode 100644 index 0000000000..e3fdd0b1db --- /dev/null +++ b/packages/react-db/src/HydrationBoundary.tsx @@ -0,0 +1,25 @@ +'use client' + +import { useRef } from 'react' +import { useDbClient } from './DbProvider' +import type { DehydratedDbState } from '@tanstack/db' +import type { ReactNode } from 'react' + +export type HydrationBoundaryProps = { + state: DehydratedDbState + children?: ReactNode +} + +export function HydrationBoundary({ state, children }: HydrationBoundaryProps) { + const client = useDbClient() + const hydrated = useRef< + { client: typeof client; state: DehydratedDbState } | undefined + >(undefined) + + if (hydrated.current?.client !== client || hydrated.current.state !== state) { + client.hydrate(state) + hydrated.current = { client, state } + } + + return children +} diff --git a/packages/react-db/src/index.ts b/packages/react-db/src/index.ts index 96db7e2796..b616297a83 100644 --- a/packages/react-db/src/index.ts +++ b/packages/react-db/src/index.ts @@ -1,5 +1,13 @@ // Re-export all public APIs -export * from './useLiveQuery' +export { useLiveQuery } from './useLiveQuery' +export type { + ConditionalUseLiveQueryConfig, + LiveQueryKey, + UseLiveQueryConfig, + UseLiveQueryStatus, +} from './useLiveQuery' +export * from './DbProvider' +export * from './HydrationBoundary' export * from './useLiveSuspenseQuery' export * from './usePacedMutations' export * from './useLiveInfiniteQuery' diff --git a/packages/react-db/src/live-query-internals.ts b/packages/react-db/src/live-query-internals.ts new file mode 100644 index 0000000000..b68cdf1abc --- /dev/null +++ b/packages/react-db/src/live-query-internals.ts @@ -0,0 +1,36 @@ +import type { + DbClient, + LiveQueryObserver, + UnhashableQueryIRError, +} from '@tanstack/db' + +const liveQueryResultInfo = Symbol(`liveQueryResultInfo`) + +export type LiveQueryResultInfo = { + client: DbClient | undefined + queryHash: string | undefined + identityError: UnhashableQueryIRError | undefined + observer: LiveQueryObserver +} + +type ResultWithInfo = { + [liveQueryResultInfo]?: LiveQueryResultInfo +} + +export function setLiveQueryResultInfo( + result: object, + info: LiveQueryResultInfo, +): void { + Object.defineProperty(result, liveQueryResultInfo, { + configurable: true, + value: info, + }) +} + +export function getLiveQueryResultInfo(result: object): LiveQueryResultInfo { + const info = (result as ResultWithInfo)[liveQueryResultInfo] + if (!info) { + throw new Error(`Missing internal live query result information.`) + } + return info +} diff --git a/packages/react-db/src/useLiveInfiniteQuery.ts b/packages/react-db/src/useLiveInfiniteQuery.ts index 99c77c7397..a9a657a669 100644 --- a/packages/react-db/src/useLiveInfiniteQuery.ts +++ b/packages/react-db/src/useLiveInfiniteQuery.ts @@ -1,39 +1,59 @@ -import { useCallback, useEffect, useMemo, useRef, useState } from 'react' -import { CollectionImpl } from '@tanstack/db' -import { useLiveQuery } from './useLiveQuery' +'use client' + +import { useCallback, useRef, useSyncExternalStore } from 'react' +import { + assertLiveQueryWindowManyResult, + compareLiveQueryWindowDependencies, + createLiveQueryCollection, + createLiveQueryWindowController, + fetchNextLiveQueryWindowPage, + getLiveQueryWindowCollectionWarning, + getLiveQueryWindowInputKind, + normalizeLiveQueryWindowPageSize, + resolveLiveQueryWindowInput, + shouldPreserveLiveQueryWindowPageCount, +} from '@tanstack/db' +import { useOptionalDbClient } from './DbProvider' +import { + prepareDerivedQuery, + prepareQueryValue, + warnDeprecatedDepsArray, + warnUnhashableDerivedIdentity, +} from './useLiveQuery' +import type { + DerivedIdentityProfiler, + LiveQueryKey, + useLiveQuery, +} from './useLiveQuery' import type { Collection, + CollectionImpl as CollectionImplType, Context, + DbClient, InferResultType, InitialQueryBuilder, - LiveQueryCollectionUtils, + LiveQueryWindowController, NonSingleResult, QueryBuilder, } from '@tanstack/db' -/** - * Type guard to check if utils object has setWindow method (LiveQueryCollectionUtils) - */ -function isLiveQueryCollectionUtils( - utils: unknown, -): utils is LiveQueryCollectionUtils { - return typeof (utils as any).setWindow === `function` -} +// Live queries created here are cleaned up immediately (0 disables GC). +const DEFAULT_GC_TIME_MS = 1 +const unpreparedQueryValue = Symbol(`unpreparedQueryValue`) -export type UseLiveInfiniteQueryConfig = { - pageSize?: number - initialPageParam?: number +// Keep the generic parameter for existing typed config wrappers. +export type UseLiveInfiniteQueryConfig<_TContext extends Context> = { /** - * @deprecated This callback is not used by the current implementation. - * Pagination is determined internally via a peek-ahead strategy. - * Provided for API compatibility with TanStack Query conventions. + * Explicit identity for queries that contain opaque functional variants or + * are hot enough that deriving identity from structured IR is too expensive. + * Structured queries should omit this so DB can derive identity directly. */ - getNextPageParam?: ( - lastPage: Array[number]>, - allPages: Array[number]>>, - lastPageParam: number, - allPageParams: Array, - ) => number | undefined + queryKey?: LiveQueryKey + /** Override the nearest DbProvider for this query. */ + client?: DbClient + pageSize?: number + /** First result-page label, not a server cursor or remote offset. */ + initialPageParam?: number } export type UseLiveInfiniteQueryReturn = Omit< @@ -43,75 +63,44 @@ export type UseLiveInfiniteQueryReturn = Omit< data: InferResultType pages: Array[number]>> pageParams: Array - fetchNextPage: () => void + fetchNextPage: () => Promise hasNextPage: boolean isFetchingNextPage: boolean + error: unknown +} + +type EnabledLiveQueryReturn = ReturnType< + typeof useLiveQuery +> + +type InfiniteQueryRenderState = { + inputKind: `collection` | `query` + inputCollection: Collection | null + inputQuery: unknown + client: DbClient | undefined + identityMode: `collection` | `queryKey` | `legacyDeps` | `derived` + dependencies: Array | null + pageSize: number + initialPageParam: number + collection: Collection + controller: LiveQueryWindowController + warning: string | null + warned: boolean + deferredCollections: Set< + CollectionImplType + > } /** - * Create an infinite query using a query function with live updates + * Create an infinite query using a query function with live updates. * * Uses `utils.setWindow()` to dynamically adjust the limit/offset window * without recreating the live query collection on each page change. * * @param queryFn - Query function that defines what data to fetch. Must include `.orderBy()` for setWindow to work. - * @param config - Configuration including pageSize and getNextPageParam - * @param deps - Array of dependencies that trigger query re-execution when changed + * @param config - Configuration including pageSize and an optional initial page label + * @param deps - Deprecated array of dependencies that trigger query re-execution when changed * @returns Object with pages, data, and pagination controls - * - * @example - * // Basic infinite query - * const { data, pages, fetchNextPage, hasNextPage } = useLiveInfiniteQuery( - * (q) => q - * .from({ posts: postsCollection }) - * .orderBy(({ posts }) => posts.createdAt, 'desc') - * .select(({ posts }) => ({ - * id: posts.id, - * title: posts.title - * })), - * { - * pageSize: 20, - * getNextPageParam: (lastPage, allPages) => - * lastPage.length === 20 ? allPages.length : undefined - * } - * ) - * - * @example - * // With dependencies - * const { pages, fetchNextPage } = useLiveInfiniteQuery( - * (q) => q - * .from({ posts: postsCollection }) - * .where(({ posts }) => eq(posts.category, category)) - * .orderBy(({ posts }) => posts.createdAt, 'desc'), - * { - * pageSize: 10, - * getNextPageParam: (lastPage) => - * lastPage.length === 10 ? lastPage.length : undefined - * }, - * [category] - * ) - * - * @example - * // Router loader pattern with pre-created collection - * // In loader: - * const postsQuery = createLiveQueryCollection({ - * query: (q) => q - * .from({ posts: postsCollection }) - * .orderBy(({ posts }) => posts.createdAt, 'desc') - * .limit(20) - * }) - * await postsQuery.preload() - * return { postsQuery } - * - * // In component: - * const { postsQuery } = useLoaderData() - * const { data, fetchNextPage, hasNextPage } = useLiveInfiniteQuery( - * postsQuery, - * { - * pageSize: 20, - * getNextPageParam: (lastPage) => lastPage.length === 20 ? lastPage.length : undefined - * } - * ) */ // Overload for pre-created collection (non-single result) @@ -135,196 +124,224 @@ export function useLiveInfiniteQuery( export function useLiveInfiniteQuery( queryFnOrCollection: any, config: UseLiveInfiniteQueryConfig, - deps: Array = [], + deps?: Array, ): UseLiveInfiniteQueryReturn { - const pageSize = config.pageSize || 20 - const initialPageParam = config.initialPageParam ?? 0 - - // Detect if input is a collection or query function - const isCollection = queryFnOrCollection instanceof CollectionImpl - - // Validate input type - if (!isCollection && typeof queryFnOrCollection !== `function`) { + if (`getNextPageParam` in config) { throw new Error( - `useLiveInfiniteQuery: First argument must be either a pre-created live query collection (CollectionImpl) ` + - `or a query function. Received: ${typeof queryFnOrCollection}`, + `getNextPageParam is not supported by useLiveInfiniteQuery. Use an on-demand collection and fulfill meta.loadSubsetOptions in queryFn for server pagination.`, ) } - - // Track how many pages have been loaded - const [loadedPageCount, setLoadedPageCount] = useState(1) - const [isFetchingNextPage, setIsFetchingNextPage] = useState(false) - - // Track collection instance and whether we've validated it (only for pre-created collections) - const collectionRef = useRef(isCollection ? queryFnOrCollection : null) - const hasValidatedCollectionRef = useRef(false) - - // Track deps for query functions (stringify for comparison) - let depsKey: string - try { - depsKey = JSON.stringify(deps) - } catch { - throw new Error( - `useLiveInfiniteQuery: dependency array contains values that cannot be serialized (e.g. circular references). ` + - `Ensure all dependency values are JSON-serializable.`, - ) - } - const prevDepsKeyRef = useRef(depsKey) - - // Reset pagination when inputs change - useEffect(() => { - let shouldReset = false - - if (isCollection) { - // Reset if collection instance changed - if (collectionRef.current !== queryFnOrCollection) { - collectionRef.current = queryFnOrCollection - hasValidatedCollectionRef.current = false - shouldReset = true - } + const pageSize = normalizeLiveQueryWindowPageSize(config.pageSize) + const initialPageParam = config.initialPageParam ?? 0 + const contextDbClient = useOptionalDbClient() + const dbClient = config.client ?? contextDbClient + + const inputIsCollection = + getLiveQueryWindowInputKind(queryFnOrCollection) === `collection` + + const committedRef = useRef(null) + const committed = committedRef.current + const inputKind = inputIsCollection ? `collection` : `query` + const derivedIdentityProfilerRef = useRef({ + renderCount: 0, + totalMs: 0, + maxMs: 0, + warned: false, + }) + const legacyUnhashableIdentityRef = useRef>([ + `legacy-unhashable`, + ]) + const deferredCollections = new Set< + CollectionImplType + >() + + let preparedQueryValue: unknown | typeof unpreparedQueryValue = + unpreparedQueryValue + let identityDeps: ReadonlyArray = [] + let identityMode: InfiniteQueryRenderState[`identityMode`] = `collection` + + if (!inputIsCollection) { + if (config.queryKey !== undefined) { + identityMode = `queryKey` + identityDeps = config.queryKey + } else if (deps !== undefined) { + identityMode = `legacyDeps` + identityDeps = deps + warnDeprecatedDepsArray(`useLiveInfiniteQuery`) + } else if ( + committed?.identityMode === `derived` && + committed.inputQuery === queryFnOrCollection && + committed.client === dbClient + ) { + identityMode = `derived` + identityDeps = committed.dependencies ?? [] } else { - // Reset if deps changed (for query functions) - if (prevDepsKeyRef.current !== depsKey) { - prevDepsKeyRef.current = depsKey - shouldReset = true - } - } - - if (shouldReset) { - setLoadedPageCount(1) - } - }, [isCollection, queryFnOrCollection, depsKey]) - - // Create a live query with initial limit and offset - // Either pass collection directly or wrap query function - // Use pageSize + 1 for peek-ahead detection (to know if there are more pages) - const queryResult = isCollection - ? useLiveQuery(queryFnOrCollection) - : useLiveQuery( - (q) => - queryFnOrCollection(q) - .limit(pageSize + 1) - .offset(0), - deps, + identityMode = `derived` + const preparation = prepareDerivedQuery( + queryFnOrCollection, + dbClient, + derivedIdentityProfilerRef.current, + deferredCollections, ) - - // Adjust window when pagination changes - useEffect(() => { - const utils = queryResult.collection.utils - const expectedOffset = 0 - const expectedLimit = loadedPageCount * pageSize + 1 // +1 for peek ahead - - // Check if collection has orderBy (required for setWindow) - if (!isLiveQueryCollectionUtils(utils)) { - // For pre-created collections, throw an error if no orderBy - if (isCollection) { - throw new Error( - `useLiveInfiniteQuery: Pre-created live query collection must have an orderBy clause for infinite pagination to work. ` + - `Please add .orderBy() to your createLiveQueryCollection query.`, - ) + preparedQueryValue = preparation.value + if (preparation.status === `hashable`) { + identityDeps = preparation.identityDeps + } else { + warnUnhashableDerivedIdentity(preparation.error) + identityDeps = legacyUnhashableIdentityRef.current } - return } + } - // For pre-created collections, validate window on first check - if (isCollection && !hasValidatedCollectionRef.current) { - const currentWindow = utils.getWindow() - if ( - currentWindow && - (currentWindow.offset !== expectedOffset || - currentWindow.limit !== expectedLimit) - ) { - console.warn( - `useLiveInfiniteQuery: Pre-created collection has window {offset: ${currentWindow.offset}, limit: ${currentWindow.limit}} ` + - `but hook expects {offset: ${expectedOffset}, limit: ${expectedLimit}}. Adjusting window now.`, - ) + const usesLegacyDeps = + !inputIsCollection && config.queryKey === undefined && deps !== undefined + const dependencyComparison = compareLiveQueryWindowDependencies( + committed?.dependencies, + identityDeps, + ) + const sameClient = committed?.client === dbClient + const dependenciesChanged = + !inputIsCollection && + (!sameClient || + (usesLegacyDeps + ? dependencyComparison.changed + : !dependencyComparison.structurallyEqual)) + const dependenciesStructurallyEqual = + usesLegacyDeps && sameClient && dependencyComparison.structurallyEqual + const needsNewCollection = + committed === null || + committed.inputKind !== inputKind || + (inputIsCollection && committed.inputCollection !== queryFnOrCollection) || + dependenciesChanged + const pageShapeChanged = + committed === null || + committed.pageSize !== pageSize || + committed.initialPageParam !== initialPageParam + const needsNewController = + committed === null || needsNewCollection || pageShapeChanged + + let renderState = committed + if (needsNewController) { + let collection = committed?.collection + let warning: string | null = null + + if (needsNewCollection) { + let inputValue = queryFnOrCollection + if (!inputIsCollection) { + if (preparedQueryValue === unpreparedQueryValue) { + preparedQueryValue = prepareQueryValue( + queryFnOrCollection, + dbClient, + deferredCollections, + ) + } + inputValue = () => preparedQueryValue } - hasValidatedCollectionRef.current = true - } - - // For query functions, wait until collection is ready - if (!isCollection && !queryResult.isReady) return - - // Adjust the window - let cancelled = false - const result = utils.setWindow({ - offset: expectedOffset, - limit: expectedLimit, - }) - - if (result !== true) { - setIsFetchingNextPage(true) - result - .catch((error: unknown) => { - if (!cancelled) - console.error(`useLiveInfiniteQuery: setWindow failed:`, error) - }) - .finally(() => { - if (!cancelled) setIsFetchingNextPage(false) + const input = resolveLiveQueryWindowInput(inputValue) + if (input.kind === `collection`) { + collection = input.collection + } else { + // Wrap the query with the first page's peek-ahead window; the controller + // grows the limit from here via setWindow. + collection = createLiveQueryCollection({ + query: input.query.limit(pageSize + 1).offset(0), + // Construction happens during render. Synchronization starts only when + // useSyncExternalStore commits the controller subscription. + startSync: false, + gcTime: DEFAULT_GC_TIME_MS, }) - } else { - setIsFetchingNextPage(false) + } } - return () => { - cancelled = true + if (!collection) { + throw new Error(`useLiveInfiniteQuery: Failed to create a collection.`) } - }, [ - isCollection, - queryResult.collection, - queryResult.isReady, - loadedPageCount, - pageSize, - ]) - - // Split the data array into pages and determine if there's a next page - const { pages, pageParams, hasNextPage, flatData } = useMemo(() => { - const dataArray = ( - Array.isArray(queryResult.data) ? queryResult.data : [] - ) as InferResultType - const totalItemsRequested = loadedPageCount * pageSize - - // Check if we have more data than requested (the peek ahead item) - const hasMore = dataArray.length > totalItemsRequested - // Build pages array (without the peek ahead item) - const pagesResult: Array[number]>> = [] - const pageParamsResult: Array = [] - - for (let i = 0; i < loadedPageCount; i++) { - const pageData = dataArray.slice(i * pageSize, (i + 1) * pageSize) - pagesResult.push(pageData) - pageParamsResult.push(initialPageParam + i) + if (inputIsCollection) { + warning = + getLiveQueryWindowCollectionWarning(collection, pageSize + 1) ?? null + } else { + assertLiveQueryWindowManyResult(collection) } - // Flatten the pages for the data return (without peek ahead item) - const flatDataResult = dataArray.slice( - 0, - totalItemsRequested, - ) as InferResultType - - return { - pages: pagesResult, - pageParams: pageParamsResult, - hasNextPage: hasMore, - flatData: flatDataResult, + const canPreservePageCount = shouldPreserveLiveQueryWindowPageCount({ + hasPreviousController: committed !== null, + previousInputKind: committed?.inputKind, + inputKind, + sameCollection: + inputIsCollection && committed?.inputCollection === collection, + dependenciesChanged, + dependenciesStructurallyEqual, + pageShapeChanged, + }) + const previousPageCount = committed + ? Math.max(1, committed.controller.getSnapshot().pages.length) + : 1 + const initialPageCount = canPreservePageCount ? previousPageCount : 1 + renderState = { + inputKind, + inputCollection: inputIsCollection ? collection : null, + inputQuery: inputIsCollection ? null : queryFnOrCollection, + client: dbClient, + identityMode, + dependencies: inputIsCollection ? null : [...identityDeps], + pageSize, + initialPageParam, + collection, + controller: createLiveQueryWindowController(collection, { + pageSize, + initialPageParam, + initialPageCount, + }), + warning, + warned: false, + deferredCollections, } - }, [queryResult.data, loadedPageCount, pageSize, initialPageParam]) - - // Fetch next page - const fetchNextPage = useCallback(() => { - if (!hasNextPage || isFetchingNextPage) return - - setLoadedPageCount((prev) => prev + 1) - }, [hasNextPage, isFetchingNextPage]) + } + const currentRenderState = renderState! + const controller = currentRenderState.controller + + const subscribe = useCallback( + (onStoreChange: () => void) => { + const unsubscribe = controller.subscribe(onStoreChange) + committedRef.current = currentRenderState + if (currentRenderState.warning && !currentRenderState.warned) { + currentRenderState.warned = true + console.warn(currentRenderState.warning) + } + for (const collection of currentRenderState.deferredCollections) { + collection._resumeSyncStart() + } + currentRenderState.deferredCollections.clear() + return unsubscribe + }, + [controller, currentRenderState], + ) + const getSnapshot = useCallback(() => controller.getSnapshot(), [controller]) + const snapshot = useSyncExternalStore(subscribe, getSnapshot, getSnapshot) + + const fetchNextPage = useCallback( + () => fetchNextLiveQueryWindowPage(controller), + [controller], + ) return { - ...queryResult, - data: flatData, - pages, - pageParams, + data: snapshot.data as InferResultType, + state: snapshot.state as EnabledLiveQueryReturn[`state`], + status: snapshot.status as EnabledLiveQueryReturn[`status`], + isLoading: snapshot.isLoading, + isReady: snapshot.isReady, + isIdle: snapshot.isIdle, + isError: snapshot.isError, + isCleanedUp: snapshot.isCleanedUp, + collection: + snapshot.collection as EnabledLiveQueryReturn[`collection`], + isEnabled: snapshot.isEnabled, + pages: snapshot.pages as Array[number]>>, + pageParams: snapshot.pageParams as Array, fetchNextPage, - hasNextPage, - isFetchingNextPage, - } as UseLiveInfiniteQueryReturn + hasNextPage: snapshot.hasNextPage, + isFetchingNextPage: snapshot.isFetchingNextPage, + error: snapshot.error, + } } diff --git a/packages/react-db/src/useLiveQuery.ts b/packages/react-db/src/useLiveQuery.ts index 331ff3a279..c9c0126465 100644 --- a/packages/react-db/src/useLiveQuery.ts +++ b/packages/react-db/src/useLiveQuery.ts @@ -1,75 +1,351 @@ +'use client' + import { useRef, useSyncExternalStore } from 'react' import { BaseQueryBuilder, - CollectionImpl, + UnhashableQueryIRError, createLiveQueryCollection, + createLiveQueryObserver, + deepEquals, + getPreparedLiveQueryIdentity, + getStableValueHash, + isCollection, + prepareLiveQueryValue, } from '@tanstack/db' +import { useOptionalDbClient } from './DbProvider' +import { setLiveQueryResultInfo } from './live-query-internals' import type { Collection, - CollectionConfigSingleRowOption, + CollectionImpl, CollectionStatus, Context, + DbClient, GetResult, InferResultType, InitialQueryBuilder, LiveQueryCollectionConfig, + LiveQueryObserver, NonSingleResult, QueryBuilder, SingleResult, } from '@tanstack/db' const DEFAULT_GC_TIME_MS = 1 // Live queries created by useLiveQuery are cleaned up immediately (0 disables GC) +const DERIVED_IDENTITY_SINGLE_RENDER_WARN_MS = 16 +const DERIVED_IDENTITY_RENDER_COUNT_WARN_THRESHOLD = 10 +const DERIVED_IDENTITY_TOTAL_WARN_MS = 50 +const warnedDepsCallsites = new Set() +const warnedDerivedIdentityCallsites = new Set() +const warnedUnhashableIdentityCallsites = new Set() +const unpreparedQueryValue = Symbol(`unpreparedQueryValue`) + +export type DerivedIdentityProfiler = { + renderCount: number + totalMs: number + maxMs: number + warned: boolean +} export type UseLiveQueryStatus = CollectionStatus | `disabled` +export type LiveQueryKey = ReadonlyArray +type UseLiveQueryConfigOptions = Omit< + LiveQueryCollectionConfig, + `query` +> & { + /** + * Explicit identity for queries that contain opaque functional variants or + * are hot enough that deriving identity from structured IR is too expensive. + * Structured queries should omit this so DB can derive identity directly. + */ + queryKey?: LiveQueryKey + /** Override the nearest DbProvider for this query. */ + client?: DbClient +} + +type ConfiguredQueryBuilder = Extract< + LiveQueryCollectionConfig[`query`], + QueryBuilder +> + +export type UseLiveQueryConfig = + UseLiveQueryConfigOptions & + Pick, `query`> + +export type ConditionalUseLiveQueryConfig = + UseLiveQueryConfigOptions & { + query: + | ConfiguredQueryBuilder + | (( + q: InitialQueryBuilder, + ) => ConfiguredQueryBuilder | undefined | null) + } + +export function warnDeprecatedDepsArray( + hookName: `useLiveQuery` | `useLiveInfiniteQuery` = `useLiveQuery`, +): void { + if (!shouldWarnInDevelopment(`TANSTACK_DB_DISABLE_DEPRECATION_WARNINGS`)) { + return + } + + const callsite = getWarningCallsite(4) + if (warnedDepsCallsites.has(callsite)) { + return + } + warnedDepsCallsites.add(callsite) + const replacement = + hookName === `useLiveQuery` + ? `useLiveQuery({ query })` + : `useLiveInfiniteQuery(query, { queryKey })` + console.warn( + `[${hookName}] The dependency-array form is deprecated and will be removed in 1.0. Use ${replacement} instead. Provide queryKey only for functional/opaque queries or to avoid deriving identity from structured query IR on render.`, + ) +} + +function shouldWarnInDevelopment(disableEnvVar: string): boolean { + if (typeof process === `undefined`) { + return false + } + + return ( + process.env.NODE_ENV !== `production` && process.env[disableEnvVar] !== `1` + ) +} + +function getCurrentTime(): number { + return typeof performance !== `undefined` && + typeof performance.now === `function` + ? performance.now() + : Date.now() +} + +function getWarningCallsite(stackIndex: number): string { + const stack = new Error().stack ?? `unknown` + return stack.split(`\n`)[stackIndex]?.trim() ?? stack +} + +function warnDerivedIdentityHotPath( + profiler: DerivedIdentityProfiler, + durationMs: number, +): void { + if ( + profiler.warned || + !shouldWarnInDevelopment(`TANSTACK_DB_DISABLE_QUERY_IDENTITY_WARNINGS`) + ) { + return + } + + const isSlowSingleRender = + durationMs >= DERIVED_IDENTITY_SINGLE_RENDER_WARN_MS + const isHotRenderPath = + profiler.renderCount >= DERIVED_IDENTITY_RENDER_COUNT_WARN_THRESHOLD && + profiler.totalMs >= DERIVED_IDENTITY_TOTAL_WARN_MS + + if (!isSlowSingleRender && !isHotRenderPath) { + return + } + + const callsite = getWarningCallsite(5) + if (warnedDerivedIdentityCallsites.has(callsite)) { + profiler.warned = true + return + } + + warnedDerivedIdentityCallsites.add(callsite) + profiler.warned = true + + const reason = isSlowSingleRender + ? `one render took ${durationMs.toFixed(1)}ms` + : `${profiler.renderCount} renders took ${profiler.totalMs.toFixed(1)}ms` + + console.warn( + `[useLiveQuery] Deriving live query identity from structured query IR is running on a hot render path (${reason}, max ${profiler.maxMs.toFixed(1)}ms). ` + + `Provide an explicit queryKey to skip rebuilding and hashing the IR on every render: useLiveQuery({ queryKey: [...], query }).`, + ) +} + +function getExplicitQueryKey(value: unknown): LiveQueryKey | undefined { + return value && + typeof value === `object` && + Array.isArray((value as { queryKey?: unknown }).queryKey) + ? (value as { queryKey: LiveQueryKey }).queryKey + : undefined +} + +function getExplicitDbClient(value: unknown): DbClient | undefined { + return value && + typeof value === `object` && + `client` in value && + (value as { client?: unknown }).client !== undefined + ? (value as { client: DbClient }).client + : undefined +} + +export function prepareQueryValue( + value: unknown, + dbClient: DbClient | undefined, + deferredCollections: Set>, +): unknown { + return prepareLiveQueryValue(value, dbClient, deferredCollections) +} + +type DerivedQueryPreparation = + | { + status: `hashable` + value: unknown + identityDeps: Array + } + | { + status: `unhashable` + value: unknown + error: UnhashableQueryIRError + } + +export function prepareDerivedQuery( + value: unknown, + dbClient: DbClient | undefined, + profiler: DerivedIdentityProfiler, + deferredCollections: Set>, +): DerivedQueryPreparation { + const shouldProfile = shouldWarnInDevelopment( + `TANSTACK_DB_DISABLE_QUERY_IDENTITY_WARNINGS`, + ) + const start = shouldProfile ? getCurrentTime() : 0 + const preparedValue = prepareQueryValue(value, dbClient, deferredCollections) + + try { + const identity = getPreparedLiveQueryIdentity(preparedValue) + return { + status: `hashable`, + value: preparedValue, + identityDeps: [`derived`, identity], + } + } catch (error) { + if (error instanceof UnhashableQueryIRError) { + return { status: `unhashable`, value: preparedValue, error } + } + + throw error + } finally { + if (shouldProfile) { + const durationMs = getCurrentTime() - start + profiler.renderCount += 1 + profiler.totalMs += durationMs + profiler.maxMs = Math.max(profiler.maxMs, durationMs) + warnDerivedIdentityHotPath(profiler, durationMs) + } + } +} + +export function warnUnhashableDerivedIdentity( + error: UnhashableQueryIRError, +): void { + if (!shouldWarnInDevelopment(`TANSTACK_DB_DISABLE_QUERY_IDENTITY_WARNINGS`)) { + return + } + + const callsite = getWarningCallsite(4) + if (warnedUnhashableIdentityCallsites.has(callsite)) { + return + } + warnedUnhashableIdentityCallsites.add(callsite) + + console.warn( + `[useLiveQuery] This query cannot derive a stable identity because ${error.reason} at ${error.path}. ` + + `It will keep the legacy mount-stable behavior for now. Add queryKey: [...] to make captured values reactive. ` + + `Unhashable queries without queryKey will throw in 1.0.`, + ) +} + +function createCollectionFromPreparedQuery(value: unknown) { + if (value === undefined || value === null) { + return null + } + + if (isCollection(value)) { + value.startSyncImmediate() + return value + } + + if (value instanceof BaseQueryBuilder) { + return createLiveQueryCollection({ + query: value, + startSync: true, + gcTime: DEFAULT_GC_TIME_MS, + }) + } + + if (typeof value === `object`) { + return createLiveQueryCollection({ + startSync: true, + gcTime: DEFAULT_GC_TIME_MS, + ...(value as LiveQueryCollectionConfig), + }) + } + + throw new Error( + `useLiveQuery callback must return a QueryBuilder, LiveQueryCollectionConfig, Collection, undefined, or null. Got: ${typeof value}`, + ) +} /** - * Create a live query using a query function + * Create a live query using a query function. * @param queryFn - Query function that defines what data to fetch - * @param deps - Array of dependencies that trigger query re-execution when changed + * @param deps - Deprecated array of dependencies that trigger query re-execution when changed * @returns Object with reactive data, state, and status information * @example - * // Basic query with object syntax - * const { data, isLoading } = useLiveQuery((q) => - * q.from({ todos: todosCollection }) - * .where(({ todos }) => eq(todos.completed, false)) - * .select(({ todos }) => ({ id: todos.id, text: todos.text })) - * ) + * // Prefer config object syntax + * const { data, isLoading } = useLiveQuery({ + * query: (q) => + * q.from({ todos: todosCollection }) + * .where(({ todos }) => eq(todos.completed, false)) + * .select(({ todos }) => ({ id: todos.id, text: todos.text })) + * }) * * @example * // Single result query - * const { data } = useLiveQuery( - * (q) => q.from({ todos: todosCollection }) + * const { data } = useLiveQuery({ + * query: (q) => q.from({ todos: todosCollection }) * .where(({ todos }) => eq(todos.id, 1)) * .findOne() - * ) + * }) * * @example - * // With dependencies that trigger re-execution - * const { data, state } = useLiveQuery( - * (q) => q.from({ todos: todosCollection }) + * // Structured captured values are included in derived query identity + * const { data, state } = useLiveQuery({ + * query: (q) => q.from({ todos: todosCollection }) * .where(({ todos }) => gt(todos.priority, minPriority)), - * [minPriority] // Re-run when minPriority changes - * ) + * }) + * + * @example + * // Return undefined or null to disable a query + * const { data, isEnabled } = useLiveQuery({ + * query: (q) => { + * if (!userId) return undefined + * return q.from({ todos: todosCollection }) + * .where(({ todos }) => eq(todos.userId, userId)) + * }, + * }) * * @example * // Join pattern - * const { data } = useLiveQuery((q) => - * q.from({ issues: issueCollection }) - * .join({ persons: personCollection }, ({ issues, persons }) => - * eq(issues.userId, persons.id) - * ) - * .select(({ issues, persons }) => ({ - * id: issues.id, - * title: issues.title, - * userName: persons.name - * })) - * ) + * const { data } = useLiveQuery({ + * query: (q) => + * q.from({ issues: issueCollection }) + * .join({ persons: personCollection }, ({ issues, persons }) => + * eq(issues.userId, persons.id) + * ) + * .select(({ issues, persons }) => ({ + * id: issues.id, + * title: issues.title, + * userName: persons.name + * })) + * }) * * @example * // Handle loading and error states - * const { data, isLoading, isError, status } = useLiveQuery((q) => - * q.from({ todos: todoCollection }) - * ) + * const { data, isLoading, isError, status } = useLiveQuery({ + * query: (q) => q.from({ todos: todoCollection }) + * }) * * if (isLoading) return
                          Loading...
                          * if (isError) return
                          Error: {status}
                          @@ -196,7 +472,7 @@ export function useLiveQuery< /** * Create a live query using configuration object * @param config - Configuration object with query and options - * @param deps - Array of dependencies that trigger query re-execution when changed + * @param deps - Deprecated array of dependencies that trigger query re-execution when changed * @returns Object with reactive data, state, and status information * @example * // Basic config object usage @@ -212,7 +488,9 @@ export function useLiveQuery< * .where(({ persons }) => gt(persons.age, 30)) * .select(({ persons }) => ({ id: persons.id, name: persons.name })) * - * const { data, isReady } = useLiveQuery({ query: queryBuilder }) + * const { data, isReady } = useLiveQuery({ + * query: queryBuilder, + * }) * * @example * // Handle all states uniformly @@ -227,6 +505,38 @@ export function useLiveQuery< * return
                          {data.length} items loaded
                          */ // Overload 6: Accept config object +export function useLiveQuery( + config: UseLiveQueryConfig, +): { + state: Map> + data: InferResultType + collection: Collection, string | number, {}> + status: CollectionStatus // Can't be disabled when query always returns a builder + isLoading: boolean + isReady: boolean + isIdle: boolean + isError: boolean + isCleanedUp: boolean + isEnabled: true // Always true when query always returns a builder +} + +// Overload 7: Accept config object with a query that can return undefined/null +export function useLiveQuery( + config: ConditionalUseLiveQueryConfig, +): { + state: Map> | undefined + data: InferResultType | undefined + collection: Collection, string | number, {}> | undefined + status: UseLiveQueryStatus + isLoading: boolean + isReady: boolean + isIdle: boolean + isError: boolean + isCleanedUp: boolean + isEnabled: boolean +} + +// Overload 8: Accept config object with legacy deps export function useLiveQuery( config: LiveQueryCollectionConfig, deps?: Array, @@ -234,13 +544,30 @@ export function useLiveQuery( state: Map> data: InferResultType collection: Collection, string | number, {}> - status: CollectionStatus // Can't be disabled for config objects + status: CollectionStatus // Can't be disabled when query always returns a builder + isLoading: boolean + isReady: boolean + isIdle: boolean + isError: boolean + isCleanedUp: boolean + isEnabled: true // Always true when query always returns a builder +} + +// Overload 9: Accept config object with legacy deps and a query that can return undefined/null +export function useLiveQuery( + config: ConditionalUseLiveQueryConfig, + deps: Array, +): { + state: Map> | undefined + data: InferResultType | undefined + collection: Collection, string | number, {}> | undefined + status: UseLiveQueryStatus isLoading: boolean isReady: boolean isIdle: boolean isError: boolean isCleanedUp: boolean - isEnabled: true // Always true for config objects + isEnabled: boolean } /** @@ -272,7 +599,7 @@ export function useLiveQuery( * * return
                          {data.map(item => )}
                          */ -// Overload 7: Accept pre-created live query collection +// Overload 10: Accept pre-created live query collection export function useLiveQuery< TResult extends object, TKey extends string | number, @@ -292,7 +619,7 @@ export function useLiveQuery< isEnabled: true // Always true for pre-created live query collections } -// Overload 8: Accept pre-created live query collection with singleResult: true +// Overload 10: Accept pre-created live query collection with singleResult: true export function useLiveQuery< TResult extends object, TKey extends string | number, @@ -315,15 +642,15 @@ export function useLiveQuery< // Implementation - use function overloads to infer the actual collection type export function useLiveQuery( configOrQueryOrCollection: any, - deps: Array = [], + deps?: Array, ) { - // Check if it's already a collection by checking for specific collection methods - const isCollection = - configOrQueryOrCollection && - typeof configOrQueryOrCollection === `object` && - typeof configOrQueryOrCollection.subscribeChanges === `function` && - typeof configOrQueryOrCollection.startSyncImmediate === `function` && - typeof configOrQueryOrCollection.id === `string` + const contextDbClient = useOptionalDbClient() + // Check if it's already a collection + const inputIsCollection = isCollection(configOrQueryOrCollection) + const dbClient = inputIsCollection + ? contextDbClient + : (getExplicitDbClient(configOrQueryOrCollection) ?? contextDbClient) + const resolvedDeps = deps ?? [] // Use refs to cache collection and track dependencies const collectionRef = useRef | null>( @@ -331,37 +658,132 @@ export function useLiveQuery( ) const depsRef = useRef | null>(null) const configRef = useRef(null) + const clientRef = useRef(dbClient) + const legacyUnhashableIdentityRef = useRef>([ + `legacy-unhashable`, + ]) - // Use refs to track version and memoized snapshot - const versionRef = useRef(0) - const snapshotRef = useRef<{ - collection: Collection | null - version: number - } | null>(null) + const derivedIdentityProfilerRef = useRef({ + renderCount: 0, + totalMs: 0, + maxMs: 0, + warned: false, + }) + const deferredCollectionsRef = useRef( + new Set>(), + ) + const observerRef = useRef | null>( + null, + ) + const queryHashRef = useRef(undefined) + const identityErrorRef = useRef(undefined) + + const queryKey = !inputIsCollection + ? getExplicitQueryKey(configOrQueryOrCollection) + : undefined + let preparedQueryValue: unknown | typeof unpreparedQueryValue = + unpreparedQueryValue + let identityDeps: ReadonlyArray + let streamIdentity: unknown = undefined + let identityError: UnhashableQueryIRError | undefined + + if (queryKey) { + identityDeps = queryKey + streamIdentity = [`queryKey`, queryKey] + } else if (deps !== undefined) { + identityDeps = resolvedDeps + try { + preparedQueryValue = prepareQueryValue( + configOrQueryOrCollection, + dbClient, + deferredCollectionsRef.current, + ) + streamIdentity = [ + `deps`, + resolvedDeps, + getPreparedLiveQueryIdentity(preparedQueryValue), + ] + } catch (error) { + if (!(error instanceof UnhashableQueryIRError)) throw error + warnUnhashableDerivedIdentity(error) + identityError = error + } + } else if (inputIsCollection) { + identityDeps = [] + streamIdentity = [`collection`, configOrQueryOrCollection.id] + } else { + const preparation = prepareDerivedQuery( + configOrQueryOrCollection, + dbClient, + derivedIdentityProfilerRef.current, + deferredCollectionsRef.current, + ) + preparedQueryValue = preparation.value + if (preparation.status === `hashable`) { + identityDeps = preparation.identityDeps + streamIdentity = preparation.identityDeps + } else { + warnUnhashableDerivedIdentity(preparation.error) + identityDeps = legacyUnhashableIdentityRef.current + identityError = preparation.error + } + } + + let queryHash: string | undefined + if (streamIdentity !== undefined) { + try { + queryHash = getStableValueHash(streamIdentity, `queryKey`) + } catch (error) { + if (error instanceof UnhashableQueryIRError) { + if (queryKey !== undefined) throw error + identityError = error + } else { + throw error + } + } + } + + if (deps !== undefined) { + warnDeprecatedDepsArray() + } + + const identityChanged = + depsRef.current === null || + (deps !== undefined + ? depsRef.current.length !== identityDeps.length || + depsRef.current.some((dep, index) => dep !== identityDeps[index]) + : !deepEquals(depsRef.current, identityDeps)) // Check if we need to create/recreate the collection const needsNewCollection = !collectionRef.current || - (isCollection && configRef.current !== configOrQueryOrCollection) || - (!isCollection && - (depsRef.current === null || - depsRef.current.length !== deps.length || - depsRef.current.some((dep, i) => dep !== deps[i]))) + (inputIsCollection && configRef.current !== configOrQueryOrCollection) || + (!inputIsCollection && (clientRef.current !== dbClient || identityChanged)) + + const resumeDeferredCollections = () => { + for (const collection of deferredCollectionsRef.current) { + collection._resumeSyncStart() + } + deferredCollectionsRef.current.clear() + } if (needsNewCollection) { - if (isCollection) { + if (inputIsCollection) { // Warn when passing a collection directly with on-demand sync mode // In on-demand mode, data is only loaded when queries with predicates request it // Passing the collection directly doesn't provide any predicates, so no data loads const syncMode = ( configOrQueryOrCollection as { config?: { syncMode?: string } } ).config?.syncMode - if (syncMode === `on-demand`) { + if ( + syncMode === `on-demand` && + shouldWarnInDevelopment(`TANSTACK_DB_DISABLE_QUERY_IDENTITY_WARNINGS`) + ) { console.warn( `[useLiveQuery] Warning: Passing a collection with syncMode "on-demand" directly to useLiveQuery ` + `will not load any data. In on-demand mode, data is only loaded when queries with predicates request it.\n\n` + `Instead, use a query builder function:\n` + - ` const { data } = useLiveQuery((q) => q.from({ c: myCollection }).select(({ c }) => c))\n\n` + + ` const { data } = useLiveQuery({ query: (q) => q.from({ c: myCollection }).select(({ c }) => c) })\n\n` + `Or switch to syncMode "eager" if you want all data to sync automatically.`, ) } @@ -370,185 +792,69 @@ export function useLiveQuery( collectionRef.current = configOrQueryOrCollection configRef.current = configOrQueryOrCollection } else { - // Handle different callback return types - if (typeof configOrQueryOrCollection === `function`) { - // Call the function with a query builder to see what it returns - const queryBuilder = new BaseQueryBuilder() as InitialQueryBuilder - const result = configOrQueryOrCollection(queryBuilder) - - if (result === undefined || result === null) { - // Callback returned undefined/null - disabled query - collectionRef.current = null - } else if (result instanceof CollectionImpl) { - // Callback returned a Collection instance - use it directly - result.startSyncImmediate() - collectionRef.current = result - } else if (result instanceof BaseQueryBuilder) { - // Callback returned QueryBuilder - create live query collection using the original callback - // (not the result, since the result might be from a different query builder instance) - collectionRef.current = createLiveQueryCollection({ - query: configOrQueryOrCollection, - startSync: true, - gcTime: DEFAULT_GC_TIME_MS, - }) - } else if (result && typeof result === `object`) { - // Assume it's a LiveQueryCollectionConfig - collectionRef.current = createLiveQueryCollection({ - startSync: true, - gcTime: DEFAULT_GC_TIME_MS, - ...result, - }) - } else { - // Unexpected return type - throw new Error( - `useLiveQuery callback must return a QueryBuilder, LiveQueryCollectionConfig, Collection, undefined, or null. Got: ${typeof result}`, - ) - } - depsRef.current = [...deps] - } else { - // Original logic for config objects - collectionRef.current = createLiveQueryCollection({ - startSync: true, - gcTime: DEFAULT_GC_TIME_MS, - ...configOrQueryOrCollection, - }) - depsRef.current = [...deps] + if (preparedQueryValue === unpreparedQueryValue) { + preparedQueryValue = prepareQueryValue( + configOrQueryOrCollection, + dbClient, + deferredCollectionsRef.current, + ) } + collectionRef.current = createCollectionFromPreparedQuery( + preparedQueryValue, + ) as Collection + configRef.current = configOrQueryOrCollection + depsRef.current = [...identityDeps] } + clientRef.current = dbClient + queryHashRef.current = queryHash + identityErrorRef.current = identityError } - // Reset refs when collection changes + // Recreate the observer when the underlying collection changes. The observer + // is not disposed explicitly here or on unmount: `useSyncExternalStore` + // unsubscribes it when the subscribe changes or the component unmounts, which + // detaches the collection subscription; the observer is then GC'd. (An unmount + // effect that disposed it would misfire under StrictMode/offscreen effect + // replay, leaving a disposed observer in the ref.) if (needsNewCollection) { - versionRef.current = 0 - snapshotRef.current = null + // Defer the initial notify: useSyncExternalStore must not be notified + // synchronously during subscribe. + // Wholesale mode: React re-reads getSnapshot() on notify, keeps the + // hook's pre-observer loading policy, and — because wholesale delivers + // nothing synchronously during subscribe — never notifies + // useSyncExternalStore inside its own subscribe call. + observerRef.current = createLiveQueryObserver(collectionRef.current, { + mode: `wholesale`, + client: dbClient, + queryHash: queryHashRef.current, + onPreload: resumeDeferredCollections, + }) } + const observer = observerRef.current! - // Create stable subscribe function using ref + // Stable subscribe bound to the current observer; the observer owns the + // subscription, ready-race, and disposal. const subscribeRef = useRef< ((onStoreChange: () => void) => () => void) | null >(null) if (!subscribeRef.current || needsNewCollection) { subscribeRef.current = (onStoreChange: () => void) => { - // If no collection, return a no-op unsubscribe function - if (!collectionRef.current) { - return () => {} - } - - const subscription = collectionRef.current.subscribeChanges(() => { - // Bump version on any change; getSnapshot will rebuild next time - versionRef.current += 1 - onStoreChange() - }) - // Collection may be ready and will not receive initial `subscribeChanges()` - if (collectionRef.current.status === `ready`) { - versionRef.current += 1 - onStoreChange() - } - return () => { - subscription.unsubscribe() - } - } - } - - // Create stable getSnapshot function using ref - const getSnapshotRef = useRef< - | (() => { - collection: Collection | null - version: number - }) - | null - >(null) - if (!getSnapshotRef.current || needsNewCollection) { - getSnapshotRef.current = () => { - const currentVersion = versionRef.current - const currentCollection = collectionRef.current - - // Recreate snapshot object only if version/collection changed - if ( - !snapshotRef.current || - snapshotRef.current.version !== currentVersion || - snapshotRef.current.collection !== currentCollection - ) { - snapshotRef.current = { - collection: currentCollection, - version: currentVersion, - } - } - - return snapshotRef.current + const unsubscribe = observer.subscribe(() => onStoreChange()) + resumeDeferredCollections() + return unsubscribe } } - // Use useSyncExternalStore to subscribe to collection changes - const snapshot = useSyncExternalStore( + const returned = useSyncExternalStore( subscribeRef.current, - getSnapshotRef.current, + () => observer.getSnapshot(), + () => observer.getServerSnapshot(), ) - - // Track last snapshot (from useSyncExternalStore) and the returned value separately - const returnedSnapshotRef = useRef<{ - collection: Collection | null - version: number - } | null>(null) - // Keep implementation return loose to satisfy overload signatures - const returnedRef = useRef(null) - - // Rebuild returned object only when the snapshot changes (version or collection identity) - if ( - !returnedSnapshotRef.current || - returnedSnapshotRef.current.version !== snapshot.version || - returnedSnapshotRef.current.collection !== snapshot.collection - ) { - // Handle null collection case (when callback returns undefined/null) - if (!snapshot.collection) { - returnedRef.current = { - state: undefined, - data: undefined, - collection: undefined, - status: `disabled`, - isLoading: false, - isReady: true, - isIdle: false, - isError: false, - isCleanedUp: false, - isEnabled: false, - } - } else { - // Capture a stable view of entries for this snapshot to avoid tearing - const entries = Array.from(snapshot.collection.entries()) - const config: CollectionConfigSingleRowOption = - snapshot.collection.config - const singleResult = config.singleResult - let stateCache: Map | null = null - let dataCache: Array | null = null - - returnedRef.current = { - get state() { - if (!stateCache) { - stateCache = new Map(entries) - } - return stateCache - }, - get data() { - if (!dataCache) { - dataCache = entries.map(([, value]) => value) - } - return singleResult ? dataCache[0] : dataCache - }, - collection: snapshot.collection, - status: snapshot.collection.status, - isLoading: snapshot.collection.status === `loading`, - isReady: snapshot.collection.status === `ready`, - isIdle: snapshot.collection.status === `idle`, - isError: snapshot.collection.status === `error`, - isCleanedUp: snapshot.collection.status === `cleaned-up`, - isEnabled: true, - } - } - - // Remember the snapshot that produced this returned value - returnedSnapshotRef.current = snapshot - } - - return returnedRef.current! + setLiveQueryResultInfo(returned, { + client: dbClient, + queryHash: queryHashRef.current, + identityError: identityErrorRef.current, + observer, + }) + return returned as any } diff --git a/packages/react-db/src/useLiveSuspenseQuery.ts b/packages/react-db/src/useLiveSuspenseQuery.ts index 162bf1f3fe..a4e8dec5b1 100644 --- a/packages/react-db/src/useLiveSuspenseQuery.ts +++ b/packages/react-db/src/useLiveSuspenseQuery.ts @@ -1,5 +1,9 @@ +'use client' + import { useRef } from 'react' import { useLiveQuery } from './useLiveQuery' +import { getLiveQueryResultInfo } from './live-query-internals' +import type { UseLiveQueryConfig } from './useLiveQuery' import type { Collection, Context, @@ -15,18 +19,19 @@ import type { /** * Create a live query with React Suspense support * @param queryFn - Query function that defines what data to fetch - * @param deps - Array of dependencies that trigger query re-execution when changed + * @param deps - Deprecated array of dependencies that trigger query re-execution when changed * @returns Object with reactive data and state - data is guaranteed to be defined * @throws Promise when data is loading (caught by Suspense boundary) * @throws Error when collection fails (caught by Error boundary) * @example * // Basic usage with Suspense * function TodoList() { - * const { data } = useLiveSuspenseQuery((q) => - * q.from({ todos: todosCollection }) - * .where(({ todos }) => eq(todos.completed, false)) - * .select(({ todos }) => ({ id: todos.id, text: todos.text })) - * ) + * const { data } = useLiveSuspenseQuery({ + * query: (q) => + * q.from({ todos: todosCollection }) + * .where(({ todos }) => eq(todos.completed, false)) + * .select(({ todos }) => ({ id: todos.id, text: todos.text })) + * }) * * return ( *
                            @@ -53,12 +58,11 @@ import type { * // data is guaranteed to be the single item (or undefined if not found) * * @example - * // With dependencies that trigger re-suspension - * const { data } = useLiveSuspenseQuery( - * (q) => q.from({ todos: todosCollection }) + * // Structured captured values are included in derived query identity and trigger re-suspension + * const { data } = useLiveSuspenseQuery({ + * query: (q) => q.from({ todos: todosCollection }) * .where(({ todos }) => gt(todos.priority, minPriority)), - * [minPriority] // Re-suspends when minPriority changes - * ) + * }) * * @example * // With Error boundary @@ -87,9 +91,9 @@ import type { * ✅ **Use conditional rendering instead:** * ```ts * function Profile({ userId }: { userId: string }) { - * const { data } = useLiveSuspenseQuery( - * (q) => q.from({ users }).where(({ users }) => eq(users.id, userId)) - * ) + * const { data } = useLiveSuspenseQuery({ + * query: (q) => q.from({ users }).where(({ users }) => eq(users.id, userId)), + * }) * return
                            {data.name}
                            * } * @@ -97,12 +101,9 @@ import type { * {userId ? :
                            No user
                            } * ``` * - * ✅ **Or use useLiveQuery for conditional queries:** + * ✅ **For optional inputs, conditionally render a component with complete query inputs:** * ```ts - * const { data, isEnabled } = useLiveQuery( - * (q) => userId ? q.from({ users }) : undefined, // ✅ Supported! - * [userId] - * ) + * {userId ? :
                            No user
                            } * ``` */ // Overload 1: Accept query function that always returns QueryBuilder @@ -116,6 +117,15 @@ export function useLiveSuspenseQuery( } // Overload 2: Accept config object +export function useLiveSuspenseQuery( + config: UseLiveQueryConfig, +): { + state: Map> + data: InferResultType + collection: Collection, string | number, {}> +} + +// Overload 3: Accept legacy config object export function useLiveSuspenseQuery( config: LiveQueryCollectionConfig, deps?: Array, @@ -125,7 +135,7 @@ export function useLiveSuspenseQuery( collection: Collection, string | number, {}> } -// Overload 3: Accept pre-created live query collection +// Overload 4: Accept pre-created live query collection export function useLiveSuspenseQuery< TResult extends object, TKey extends string | number, @@ -138,7 +148,7 @@ export function useLiveSuspenseQuery< collection: Collection } -// Overload 4: Accept pre-created live query collection with singleResult: true +// Overload 5: Accept pre-created live query collection with singleResult: true export function useLiveSuspenseQuery< TResult extends object, TKey extends string | number, @@ -154,16 +164,20 @@ export function useLiveSuspenseQuery< // Implementation - uses useLiveQuery internally and adds Suspense logic export function useLiveSuspenseQuery( configOrQueryOrCollection: any, - deps: Array = [], + deps?: Array, ) { const promiseRef = useRef | null>(null) const collectionRef = useRef | null>(null) const hasBeenReadyRef = useRef(false) // Use useLiveQuery to handle collection management and reactivity - const result = useLiveQuery(configOrQueryOrCollection, deps) + const result = + deps === undefined + ? useLiveQuery(configOrQueryOrCollection) + : useLiveQuery(configOrQueryOrCollection, deps) + const queryInfo = getLiveQueryResultInfo(result) - // Reset promise and ready state when collection changes (deps changed) + // Reset promise and ready state when query identity changes if (collectionRef.current !== result.collection) { promiseRef.current = null collectionRef.current = result.collection @@ -183,16 +197,20 @@ export function useLiveSuspenseQuery( ) } - // It’s not recommended to suspend a render based on a store value returned by useSyncExternalStore. - // result.status is the snapshot from syncExternalStore. We read the fresh status from the collection reference instead. const collectionStatus = result.collection.status // Track when we reach ready state - if (collectionStatus === `ready`) { + if (result.isReady) { hasBeenReadyRef.current = true promiseRef.current = null } + const observerError = queryInfo.observer.getError() + if (observerError !== undefined && !hasBeenReadyRef.current) { + promiseRef.current = null + throw observerError + } + // Only throw errors during initial load (before first ready) // After success, errors surface as stale data (matches TanStack Query behavior) if (collectionStatus === `error` && !hasBeenReadyRef.current) { @@ -202,10 +220,19 @@ export function useLiveSuspenseQuery( throw new Error(`Collection "${result.collection.id}" failed to load`) } - if (collectionStatus === `loading` || collectionStatus === `idle`) { + if (!hasBeenReadyRef.current && (result.isLoading || result.isIdle)) { + if (queryInfo.client?._isSsrStreamingEnabled() && !queryInfo.queryHash) { + const reason = queryInfo.identityError + ? `${queryInfo.identityError.reason} at ${queryInfo.identityError.path}` + : `the query has no stable identity` + throw new Error( + `Cannot stream this live query during SSR because ${reason}. Provide an explicit serializable queryKey.`, + ) + } + // Create or reuse promise for current collection if (!promiseRef.current) { - promiseRef.current = result.collection.preload() + promiseRef.current = queryInfo.observer.preload() } // THROW PROMISE - React Suspense catches this (React 18+ required) // Note: We don't check React version here. In React <18, this will be caught diff --git a/packages/react-db/tests/DbProvider.test.tsx b/packages/react-db/tests/DbProvider.test.tsx new file mode 100644 index 0000000000..59e1a88eb5 --- /dev/null +++ b/packages/react-db/tests/DbProvider.test.tsx @@ -0,0 +1,24 @@ +import { describe, expect, it } from 'vitest' +import { renderHook } from '@testing-library/react' +import { DbClient } from '@tanstack/db' +import { DbProvider, useDbClient } from '../src/DbProvider' +import type { ReactNode } from 'react' + +describe(`DbProvider`, () => { + it(`provides a DbClient to hooks`, () => { + const client = new DbClient() + const wrapper = ({ children }: { children: ReactNode }) => ( + {children} + ) + + const { result } = renderHook(() => useDbClient(), { wrapper }) + + expect(result.current).toBe(client) + }) + + it(`throws without a provider`, () => { + expect(() => renderHook(() => useDbClient())).toThrow( + /useDbClient must be used within a DbProvider/, + ) + }) +}) diff --git a/packages/react-db/tests/HydrationBoundary.test.tsx b/packages/react-db/tests/HydrationBoundary.test.tsx new file mode 100644 index 0000000000..1daeac0dc1 --- /dev/null +++ b/packages/react-db/tests/HydrationBoundary.test.tsx @@ -0,0 +1,63 @@ +import { describe, expect, it, vi } from 'vitest' +import { render, screen } from '@testing-library/react' +import { DbClient, collectionOptions } from '@tanstack/db' +import { DbProvider, useDbClient } from '../src/DbProvider' +import { HydrationBoundary } from '../src/HydrationBoundary' + +type Person = { + id: string + name: string +} + +const people = collectionOptions(`hydration-boundary-people`, () => ({ + id: `hydration-boundary-people`, + getKey: (person: Person) => person.id, + sync: { + sync: ({ markReady }) => markReady(), + }, +})) + +const state = { + collections: [ + { + collectionId: people.id, + rows: [{ key: `1`, value: { id: `1`, name: `Hydrated` } }], + }, + ], +} + +function PersonName() { + const client = useDbClient() + return {client.collection(people).get(`1`)?.name} +} + +function App({ client }: { client: DbClient }) { + return ( + + + + + + ) +} + +describe(`HydrationBoundary`, () => { + it(`hydrates before children render and follows the provider client`, () => { + const firstClient = new DbClient() + const firstHydrate = vi.spyOn(firstClient, `hydrate`) + const view = render() + + expect(screen.getByText(`Hydrated`)).toBeInTheDocument() + expect(firstHydrate).toHaveBeenCalledTimes(1) + + view.rerender() + expect(firstHydrate).toHaveBeenCalledTimes(1) + + const secondClient = new DbClient() + const secondHydrate = vi.spyOn(secondClient, `hydrate`) + view.rerender() + + expect(screen.getByText(`Hydrated`)).toBeInTheDocument() + expect(secondHydrate).toHaveBeenCalledTimes(1) + }) +}) diff --git a/packages/react-db/tests/conformance.test.tsx b/packages/react-db/tests/conformance.test.tsx new file mode 100644 index 0000000000..3916d67b79 --- /dev/null +++ b/packages/react-db/tests/conformance.test.tsx @@ -0,0 +1,214 @@ +/** + * React reference driver for the shared live-query conformance suite. + * + * Everything realm-sensitive — collection creation and query operators — is + * imported here (React package's `@tanstack/db`) and handed to the shared + * scenarios, so instances match what this package's `useLiveQuery` expects. + * + * `knownGaps` is populated empirically from the run below, NOT from the coverage + * matrix: only keys that actually fail belong here. Today that's just the + * universal order-only-move case (handled by the shared suite), so the list is empty. + */ +import { act, renderHook } from '@testing-library/react' +import { + coalesce, + count, + createCollection, + createLiveQueryCollection, + createOptimisticAction, + eq, + gt, + sum, +} from '@tanstack/db' +import { + mockSyncCollectionOptions, + mockSyncCollectionOptionsNoInitialState, +} from '../../db/tests/utils' +import { useLiveQuery } from '../src/useLiveQuery' +import { runSuite } from '../../db/tests/conformance/suite' +import type { RenderHookResult } from '@testing-library/react' +import type { + ConformanceResult, + ControllableHandle, + DeferredSourceHandle, + LiveQueryDriver, + QueryBuild, + SourceHandle, +} from '../../db/tests/conformance/contract' + +let sourceSeq = 0 + +function writer(collection: any) { + return (type: `insert` | `update` | `delete`, value: T) => { + collection.utils.begin() + collection.utils.write({ type, value }) + collection.utils.commit() + } +} + +function makeSource( + initialData: ReadonlyArray, +): SourceHandle { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `conformance-react-${sourceSeq++}`, + getKey: (r) => r.id, + initialData: [...initialData], + }), + ) + const write = writer(collection) + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + } +} + +function makeDeferredSource< + T extends { id: string }, +>(): DeferredSourceHandle { + const collection = createCollection( + mockSyncCollectionOptionsNoInitialState({ + id: `conformance-react-${sourceSeq++}`, + getKey: (r) => r.id, + }), + ) + // Start sync so the sync fn binds utils and the collection sits in `loading` + // (NoInitialState never calls markReady on its own). + collection.startSyncImmediate() + const write = writer(collection) + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + emit: (rows) => { + collection.utils.begin() + rows.forEach((value) => collection.utils.write({ type: `insert`, value })) + collection.utils.commit() + }, + markReady: () => collection.utils.markReady(), + } +} + +function makePrecreated(build: QueryBuild, opts?: { startSync?: boolean }) { + const collection = createLiveQueryCollection({ + query: build as any, + startSync: opts?.startSync ?? true, + }) + return { collection } +} + +function makeErrorSource() { + const collection = createCollection<{ id: string }>({ + id: `conformance-react-err-${sourceSeq++}`, + getKey: (r) => r.id, + startSync: false, + sync: { + sync: () => { + throw new Error(`conformance: sync failure`) + }, + }, + }) + // Starting sync throws → engine catches and sets status to `error`. + try { + collection.startSyncImmediate() + } catch { + // expected: the rethrown sync error; status is already `error` + } + return { collection } +} + +function mount(build: QueryBuild) { + const hook = renderHook(() => useLiveQuery(build as any)) + return makeHandle(hook) +} + +function mountCollection(collection: any) { + const hook = renderHook(() => useLiveQuery(collection)) + return makeHandle(hook) +} + +function mountConfig(build: QueryBuild) { + const hook = renderHook(() => useLiveQuery({ query: build as any })) + return makeHandle(hook) +} + +function mountDisabled() { + // React's disabled convention: the query callback returns null. + const hook = renderHook(() => useLiveQuery(() => null as any)) + return makeHandle(hook) +} + +function mountControllable

                            ( + build: (q: any, param: P) => any, + initial: P, +): ControllableHandle

                            { + const hook = renderHook( + ({ param }: { param: P }) => + // Param goes in the dependency list so the hook recompiles when it changes. + useLiveQuery((q: any) => build(q, param), [param]), + { initialProps: { param: initial } }, + ) + const handle = makeHandle(hook) + return { + ...handle, + async setParam(param: P) { + await act(async () => { + hook.rerender({ param }) + }) + await handle.flush() + }, + } +} + +function makeHandle(hook: RenderHookResult) { + return { + current(): ConformanceResult { + const r: any = hook.result.current + return { + data: r?.data, + state: r?.state, + status: r?.status ?? `idle`, + isReady: Boolean(r?.isReady), + isError: Boolean(r?.isError), + // Read react-db's real `isEnabled` field so the suite catches a broken + // one (deriving from status would mask it). + isEnabled: Boolean(r?.isEnabled), + } + }, + async flush() { + await act(async () => { + await new Promise((resolve) => setTimeout(resolve, 0)) + }) + }, + async apply(fn: () => void) { + await act(async () => { + fn() + await new Promise((resolve) => setTimeout(resolve, 0)) + }) + }, + unmount() { + hook.unmount() + }, + } +} + +const reactDriver: LiveQueryDriver = { + name: `react`, + ops: { eq, gt, count, sum, coalesce, createOptimisticAction }, + makeSource, + makeDeferredSource, + makePrecreated, + makeErrorSource, + mount, + mountControllable, + mountCollection, + mountConfig, + mountDisabled, + knownGaps: [], + features: { serverSnapshot: true, suspense: true }, +} + +runSuite(reactDriver) diff --git a/packages/react-db/tests/infinite-query-conformance.test.tsx b/packages/react-db/tests/infinite-query-conformance.test.tsx new file mode 100644 index 0000000000..bf7bd198eb --- /dev/null +++ b/packages/react-db/tests/infinite-query-conformance.test.tsx @@ -0,0 +1,204 @@ +/** React driver for the shared infinite-query conformance suite. */ +import { act, renderHook } from '@testing-library/react' +import { + BTreeIndex, + createCollection, + createLiveQueryCollection, + gt, +} from '@tanstack/db' +import { mockSyncCollectionOptions } from '../../db/tests/utils' +import { runInfiniteQuerySuite } from '../../db/tests/conformance/infinite-suite' +import { makeInfiniteOnDemandSource } from '../../db/tests/conformance/infinite-on-demand' +import { useLiveInfiniteQuery } from '../src/useLiveInfiniteQuery' +import type { RenderHookResult } from '@testing-library/react' +import type { + InfiniteQueryConfig, + InfiniteQueryDriver, + InfiniteQueryHandle, +} from '../../db/tests/conformance/infinite-contract' +import type { + QueryBuild, + SourceHandle, +} from '../../db/tests/conformance/contract' + +let sourceSequence = 0 + +function makeSource( + initialData: ReadonlyArray, +): SourceHandle { + const collection = createCollection( + mockSyncCollectionOptions({ + autoIndex: `eager`, + id: `infinite-conformance-react-${sourceSequence++}`, + getKey: (row) => row.id, + initialData: [...initialData], + }), + ) + const write = (type: `insert` | `update` | `delete`, value: T) => { + collection.utils.begin() + collection.utils.write({ type, value }) + collection.utils.commit() + } + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + } +} + +function makePrecreated(build: QueryBuild) { + return { + collection: createLiveQueryCollection({ query: build as any }), + } +} + +function makeHandle(hook: RenderHookResult): InfiniteQueryHandle { + return { + current() { + const result = hook.result.current + return { + data: result.data, + pages: result.pages, + pageParams: result.pageParams, + hasNextPage: result.hasNextPage, + isFetchingNextPage: result.isFetchingNextPage, + error: result.error, + status: result.status, + collection: result.collection, + } + }, + fetchNextPage() { + let request!: Promise + act(() => { + request = hook.result.current.fetchNextPage() + }) + return request + }, + async flush() { + await act(async () => { + await new Promise((resolve) => setTimeout(resolve, 0)) + }) + }, + async apply(fn) { + await act(async () => { + fn() + await Promise.resolve() + }) + }, + unmount() { + hook.unmount() + }, + } +} + +function mount(build: QueryBuild, config: InfiniteQueryConfig = {}) { + return makeHandle( + renderHook(() => useLiveInfiniteQuery(build as any, config as any)), + ) +} + +function mountControllable

                            ( + build: (q: any, param: P) => any, + initial: P, + config: InfiniteQueryConfig = {}, +) { + const hook = renderHook( + ({ param }: { param: P }) => + useLiveInfiniteQuery((q: any) => build(q, param), config as any, [param]), + { initialProps: { param: initial } }, + ) + const handle = makeHandle(hook) + return { + ...handle, + setParamSync(param: P) { + act(() => hook.rerender({ param })) + }, + } +} + +function mountCollection(collection: any, config: InfiniteQueryConfig = {}) { + return makeHandle( + renderHook(() => useLiveInfiniteQuery(collection, config as any)), + ) +} + +function mountCollectionControllable( + initial: any, + config: InfiniteQueryConfig = {}, +) { + const hook = renderHook( + ({ collection }) => useLiveInfiniteQuery(collection, config as any), + { initialProps: { collection: initial } }, + ) + const handle = makeHandle(hook) + return { + ...handle, + replaceCollectionSync(collection: any) { + act(() => hook.rerender({ collection })) + }, + } +} + +function mountConfigControllable( + build: QueryBuild, + initial: InfiniteQueryConfig, +) { + const hook = renderHook( + ({ config }: { config: InfiniteQueryConfig }) => + useLiveInfiniteQuery(build as any, config as any), + { initialProps: { config: initial } }, + ) + const handle = makeHandle(hook) + return { + ...handle, + setConfigSync(config: InfiniteQueryConfig) { + act(() => hook.rerender({ config })) + }, + } +} + +function mountInputControllable( + collection: any, + build: QueryBuild, + config: InfiniteQueryConfig = {}, +) { + const hook = renderHook( + ({ kind }: { kind: `collection` | `query` }) => + useLiveInfiniteQuery( + kind === `collection` ? collection : build, + config as any, + [kind], + ), + { + initialProps: { + kind: `collection` as `collection` | `query`, + }, + }, + ) + const handle = makeHandle(hook) + return { + ...handle, + setInputKindSync(kind: `collection` | `query`) { + act(() => hook.rerender({ kind })) + }, + } +} + +const reactInfiniteDriver: InfiniteQueryDriver = { + name: `react`, + gt, + makeSource, + makeOnDemandSource: (data, delay) => + makeInfiniteOnDemandSource({ createCollection, BTreeIndex }, data, delay), + makePrecreated, + mount, + mountControllable, + mountCollection, + mountCollectionControllable, + mountConfigControllable, + mountInputControllable, + knownGaps: [], +} + +runInfiniteQuerySuite(reactInfiniteDriver) diff --git a/packages/react-db/tests/server-pagination-probe.test.tsx b/packages/react-db/tests/server-pagination-probe.test.tsx new file mode 100644 index 0000000000..e7d94bc8ab --- /dev/null +++ b/packages/react-db/tests/server-pagination-probe.test.tsx @@ -0,0 +1,222 @@ +import { act, renderHook, waitFor } from '@testing-library/react' +import { describe, expect, it, vi } from 'vitest' +import { useLiveInfiniteQuery } from '../src/useLiveInfiniteQuery' +import { createServerPaginationFixture } from '../../query-db-collection/tests/server-pagination-fixture' + +const rows = Array.from({ length: 8 }, (_, id) => ({ id, rank: id })) +const pageCases = [1, 2, 3, 5].flatMap((serverPageSize) => + [1, 2, 5].flatMap((pageSize) => + [0, 1, 8].map((rowCount) => ({ serverPageSize, pageSize, rowCount })), + ), +) + +describe(`server pagination contract probes`, () => { + it(`rejects a server-page callback before constructing a query`, () => { + const queryFn = vi.fn(() => { + throw new Error(`query must not be constructed`) + }) + const config = { pageSize: 2, getNextPageParam: () => 1 } + const error = vi.spyOn(console, `error`).mockImplementation(() => {}) + try { + expect(() => + renderHook(() => useLiveInfiniteQuery(queryFn, config)), + ).toThrow(`getNextPageParam is not supported`) + expect(queryFn).not.toHaveBeenCalled() + } finally { + error.mockRestore() + } + }) + + it.each(pageCases)( + `drains server pages of $serverPageSize for UI pages of $pageSize over $rowCount rows`, + async ({ serverPageSize, pageSize, rowCount }) => { + const sourceRows = rows.slice(0, rowCount) + const fixture = createServerPaginationFixture({ + rows: sourceRows, + syncMode: `on-demand`, + serverPageSize, + }) + const { result, unmount } = renderHook(() => + useLiveInfiniteQuery( + (q) => + q.from({ row: fixture.collection }).orderBy(({ row }) => row.id), + { pageSize }, + ), + ) + try { + await waitFor(() => expect(result.current.isReady).toBe(true)) + for (let size = pageSize; ; size += pageSize) { + expect(result.current.data.map((row) => row.id)).toEqual( + sourceRows.slice(0, size).map((row) => row.id), + ) + expect(result.current.hasNextPage).toBe(size < rowCount) + if (size >= rowCount) break + await act(() => result.current.fetchNextPage()) + } + if (rowCount > serverPageSize) + expect(fixture.serverPages.length).toBeGreaterThan(1) + if (rowCount > pageSize + 1) { + expect( + fixture.requests.some( + (request) => (request.subset?.offset ?? 0) > 0, + ), + ).toBe(true) + } + } finally { + unmount() + await result.current.collection?.cleanup() + await fixture.collection.cleanup() + fixture.client.clear() + } + }, + ) + + it(`retains a tie group across backend and UI page boundaries`, async () => { + // Already sorted by rank, then id. IDs decrease between groups so an + // accidental id-first order cannot produce the expected result. + const sourceRows = [ + ...Array.from({ length: 6 }, (_, index) => ({ id: index + 10, rank: 1 })), + ...Array.from({ length: 3 }, (_, index) => ({ id: index + 1, rank: 2 })), + ] + const fixture = createServerPaginationFixture({ + rows: sourceRows, + syncMode: `on-demand`, + serverPageSize: 2, + }) + const { result, unmount } = renderHook(() => + useLiveInfiniteQuery( + (q) => + q + .from({ row: fixture.collection }) + .orderBy(({ row }) => row.rank) + .orderBy(({ row }) => row.id), + { pageSize: 3 }, + ), + ) + try { + await waitFor(() => expect(result.current.isReady).toBe(true)) + for (const size of [3, 6, 9]) { + expect(result.current.data.map((row) => row.id)).toEqual( + sourceRows.slice(0, size).map((row) => row.id), + ) + expect(result.current.hasNextPage).toBe(size < sourceRows.length) + if (size < sourceRows.length) + await act(() => result.current.fetchNextPage()) + } + expect(new Set(fixture.serverPages)).toEqual(new Set([0, 1, 2, 3, 4])) + const requestCount = fixture.requests.length + await act(() => result.current.fetchNextPage()) + expect(fixture.requests).toHaveLength(requestCount) + } finally { + unmount() + await result.current.collection?.cleanup() + await fixture.collection.cleanup() + fixture.client.clear() + } + }) + + it(`eager page responses remain local data, not an infinite-query transport`, async () => { + const fixture = createServerPaginationFixture({ + rows, + syncMode: `eager`, + cap: 4, + }) + const { result, unmount } = renderHook(() => + useLiveInfiniteQuery( + (q) => q.from({ row: fixture.collection }).orderBy(({ row }) => row.id), + { pageSize: 2, initialPageParam: 10 }, + ), + ) + try { + await waitFor(() => expect(result.current.isReady).toBe(true)) + expect(result.current.data.map((row) => row.id)).toEqual([0, 1]) + expect(result.current.pageParams).toEqual([10]) + await act(() => result.current.fetchNextPage()) + expect(result.current.data.map((row) => row.id)).toEqual([0, 1, 2, 3]) + expect(result.current.pageParams).toEqual([10, 11]) + expect(result.current.hasNextPage).toBe(false) + await act(() => result.current.fetchNextPage()) + expect(fixture.requests).toHaveLength(1) + expect(fixture.requests[0]?.pageParam).toBeUndefined() + } finally { + unmount() + await result.current.collection?.cleanup() + await fixture.collection.cleanup() + fixture.client.clear() + } + }) + + it(`on-demand prefixes grow through Query DB and retain earlier rows`, async () => { + const fixture = createServerPaginationFixture({ + rows, + syncMode: `on-demand`, + }) + const { result, unmount } = renderHook(() => + useLiveInfiniteQuery( + (q) => + q + .from({ row: fixture.collection }) + .orderBy(({ row }) => row.id) + .orderBy(({ row }) => row.rank), + { pageSize: 2 }, + ), + ) + try { + await waitFor(() => expect(result.current.isReady).toBe(true)) + for (const size of [2, 4, 6, 8]) { + expect(result.current.data.map((row) => row.id)).toEqual( + rows.slice(0, size).map((row) => row.id), + ) + expect(result.current.hasNextPage).toBe(size < rows.length) + if (size < rows.length) await act(() => result.current.fetchNextPage()) + } + expect( + fixture.requests.flatMap((request) => + request.subset?.limit === undefined ? [] : [request.subset.limit], + ), + ).toEqual([3, 5, 7, 9]) + expect( + fixture.requests.every((request) => request.pageParam === undefined), + ).toBe(true) + } finally { + unmount() + await result.current.collection?.cleanup() + await fixture.collection.cleanup() + fixture.client.clear() + } + }) + + // Deliberately nonconforming provider: this is a protocol boundary control, + // not an oracle accepting truncated responses as successful pagination. + it(`a capped response can underfill locally while the server still has rows`, async () => { + const fixture = createServerPaginationFixture({ + rows, + syncMode: `on-demand`, + cap: 2, + }) + const { result, unmount } = renderHook(() => + useLiveInfiniteQuery( + (q) => + q + .from({ row: fixture.collection }) + .orderBy(({ row }) => row.id) + .orderBy(({ row }) => row.rank), + { pageSize: 2 }, + ), + ) + try { + await waitFor(() => expect(result.current.isReady).toBe(true)) + expect(result.current.data.map((row) => row.id)).toEqual([0, 1]) + expect(rows.length).toBeGreaterThan(result.current.data.length) + expect(result.current.hasNextPage).toBe(false) + const requestCount = fixture.requests.length + await act(() => result.current.fetchNextPage()) + expect(fixture.requests).toHaveLength(requestCount) + } finally { + unmount() + await result.current.collection?.cleanup() + await fixture.collection.cleanup() + fixture.client.clear() + } + }) +}) diff --git a/packages/react-db/tests/useLiveInfiniteQuery.test-d.tsx b/packages/react-db/tests/useLiveInfiniteQuery.test-d.tsx new file mode 100644 index 0000000000..dd3923af7a --- /dev/null +++ b/packages/react-db/tests/useLiveInfiniteQuery.test-d.tsx @@ -0,0 +1,29 @@ +import { describe, expectTypeOf, it } from 'vitest' +import type { + UseLiveInfiniteQueryConfig, + UseLiveInfiniteQueryReturn, +} from '../src/useLiveInfiniteQuery' +import type { Context } from '@tanstack/db' + +describe(`useLiveInfiniteQuery type assertions`, () => { + it(`does not advertise a server-page callback`, () => { + expectTypeOf< + Extract, `getNextPageParam`> + >().toEqualTypeOf() + }) + + it(`keeps legacy generic wrappers source-compatible`, () => { + function acceptsContext( + _config: UseLiveInfiniteQueryConfig, + _result: UseLiveInfiniteQueryReturn, + ): void {} + + void acceptsContext + }) + + it(`exposes the controller fetch promise`, () => { + expectTypeOf< + UseLiveInfiniteQueryReturn[`fetchNextPage`] + >().toEqualTypeOf<() => Promise>() + }) +}) diff --git a/packages/react-db/tests/useLiveInfiniteQuery.test.tsx b/packages/react-db/tests/useLiveInfiniteQuery.test.tsx index 9aa63244e7..0ac0c88ba8 100644 --- a/packages/react-db/tests/useLiveInfiniteQuery.test.tsx +++ b/packages/react-db/tests/useLiveInfiniteQuery.test.tsx @@ -1,11 +1,15 @@ -import { describe, expect, it } from 'vitest' -import { act, renderHook, waitFor } from '@testing-library/react' -import { createCollection, createLiveQueryCollection, eq } from '@tanstack/db' -import { BTreeIndex } from '@tanstack/db' +import { describe, expect, it, vi } from 'vitest' +import { act, render, renderHook, waitFor } from '@testing-library/react' +import { Suspense } from 'react' +import { + createCollection, + createLiveQueryCollection, + eq, + gt, +} from '@tanstack/db' import { useLiveInfiniteQuery } from '../src/useLiveInfiniteQuery' import { mockSyncCollectionOptions } from '../../db/tests/utils' -import { createFilterFunctionFromExpression } from '../../db/src/collection/change-events' -import type { LoadSubsetOptions } from '@tanstack/db' +import type { ReactNode } from 'react' type Post = { id: string @@ -29,1873 +33,491 @@ function createMockPosts(count: number): Array { return posts } -type OnDemandCollectionOptions = { - id: string - allPosts: Array - autoIndex?: `off` | `eager` - asyncDelay?: number -} - -/** - * Creates an on-demand collection with a loadSubset handler that supports - * sorting, cursor-based pagination, and limit. Returns the collection and - * a reference to recorded loadSubset calls for test assertions. - */ -function createOnDemandCollection(opts: OnDemandCollectionOptions) { - const loadSubsetCalls: Array = [] - const { id, allPosts, autoIndex, asyncDelay } = opts - - const collection = createCollection({ - id, - getKey: (post: Post) => post.id, - syncMode: `on-demand`, - startSync: true, - autoIndex: autoIndex ?? `eager`, - defaultIndexType: BTreeIndex, - sync: { - sync: ({ markReady, begin, write, commit }) => { - markReady() +describe(`useLiveInfiniteQuery`, () => { + it(`does not activate a query-function collection for an abandoned render`, async () => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `abandoned-infinite-query`, + getKey: (post) => post.id, + initialData: createMockPosts(10), + }), + ) + const never = new Promise(() => {}) - return { - loadSubset: (subsetOpts: LoadSubsetOptions) => { - loadSubsetCalls.push({ ...subsetOpts }) + function AbandonedQuery(): ReactNode { + useLiveInfiniteQuery( + (q) => + q + .from({ post: source }) + .orderBy(({ post }) => post.createdAt, `desc`), + { pageSize: 3 }, + ) + throw never + } - let filtered = [...allPosts].sort( - (a, b) => b.createdAt - a.createdAt, - ) + const rendered = render( + + + , + ) + await new Promise((resolve) => setTimeout(resolve, 0)) - if (subsetOpts.cursor) { - const whereFromFn = createFilterFunctionFromExpression( - subsetOpts.cursor.whereFrom, - ) - filtered = filtered.filter(whereFromFn) - } + expect(source.subscriberCount).toBe(0) + rendered.unmount() + }) - if (subsetOpts.limit !== undefined) { - filtered = filtered.slice(0, subsetOpts.limit) - } + it(`does not activate a supplied collection for an abandoned render`, async () => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `abandoned-supplied-infinite-query`, + getKey: (post) => post.id, + initialData: createMockPosts(10), + }), + ) + const liveQuery = createLiveQueryCollection({ + query: (q) => + q.from({ post: source }).orderBy(({ post }) => post.createdAt, `desc`), + }) + const never = new Promise(() => {}) - function writeAll(): void { - begin() - for (const post of filtered) { - write({ type: `insert`, value: post }) - } - commit() - } + function AbandonedQuery(): ReactNode { + useLiveInfiniteQuery(liveQuery, { pageSize: 3 }) + throw never + } - if (asyncDelay !== undefined) { - return new Promise((resolve) => { - setTimeout(() => { - writeAll() - resolve() - }, asyncDelay) - }) - } + const rendered = render( + + + , + ) + await new Promise((resolve) => setTimeout(resolve, 0)) - writeAll() - return true - }, - } - }, - }, + expect(source.subscriberCount).toBe(0) + rendered.unmount() }) - return { collection, loadSubsetCalls } -} - -describe(`useLiveInfiniteQuery`, () => { - it(`should fetch initial page of data`, async () => { - const posts = createMockPosts(50) - const collection = createCollection( + it(`preserves committed pages across an abandoned dependency update`, async () => { + const source = createCollection( mockSyncCollectionOptions({ autoIndex: `eager`, - id: `initial-page-test`, - getKey: (post: Post) => post.id, - initialData: posts, + id: `abandoned-infinite-query-update`, + getKey: (post) => post.id, + initialData: createMockPosts(20), }), ) + const never = new Promise(() => {}) + let shouldSuspend = false + let current: + | { + isReady: boolean + pages: Array> + fetchNextPage: () => Promise + } + | undefined - const { result } = renderHook(() => { - return useLiveInfiniteQuery( + function Query({ minimum }: { minimum: number }): ReactNode { + current = useLiveInfiniteQuery( (q) => q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`) - .select(({ posts: p }) => ({ - id: p.id, - title: p.title, - createdAt: p.createdAt, - })), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }, + .from({ post: source }) + .where(({ post }) => gt(post.createdAt, minimum)) + .orderBy(({ post }) => post.createdAt, `desc`), + { pageSize: 3 }, + [minimum], ) - }) + if (shouldSuspend) throw never + return null + } + + function App({ minimum }: { minimum: number }): ReactNode { + return ( + + + + ) + } - await waitFor(() => { - expect(result.current.isReady).toBe(true) + const rendered = render() + await waitFor(() => expect(current?.isReady).toBe(true)) + await act(async () => { + await current!.fetchNextPage() + await current!.fetchNextPage() }) + expect(current?.pages.map((page) => page.length)).toEqual([3, 3, 3]) - // Should have 1 page initially - expect(result.current.pages).toHaveLength(1) - expect(result.current.pages[0]).toHaveLength(10) + shouldSuspend = true + rendered.rerender() + await Promise.resolve() - // Data should be flattened - expect(result.current.data).toHaveLength(10) + shouldSuspend = false + rendered.rerender() + await waitFor(() => expect(current?.isReady).toBe(true)) + expect(current?.pages.map((page) => page.length)).toEqual([3, 3, 3]) + rendered.unmount() + }) - // Should have next page since we have 50 items total - expect(result.current.hasNextPage).toBe(true) + it(`recognizes a structurally valid collection from another realm`, () => { + const foreignCollection = { + id: `foreign-live-query`, + subscribeChanges: () => () => {}, + startSyncImmediate: () => {}, + utils: { + setWindow: () => true as const, + getWindow: () => undefined, + }, + } - // First item should be Post 1 (most recent by createdAt) - expect(result.current.pages[0]![0]).toMatchObject({ - id: `1`, - title: `Post 1`, - }) + expect(() => + renderHook(() => + useLiveInfiniteQuery(foreignCollection as any, { pageSize: 3 }), + ), + ).toThrow(/orderBy/) }) - it(`should fetch multiple pages`, async () => { - const posts = createMockPosts(50) - const collection = createCollection( + it(`resolves the fetch promise and exposes pagination failures in state`, async () => { + const source = createCollection( mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `multiple-pages-test`, - getKey: (post: Post) => post.id, - initialData: posts, + id: `infinite-query-pagination-failure`, + getKey: (post) => post.id, + initialData: createMockPosts(10), }), ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }, - ) + const query = createLiveQueryCollection({ + query: (q) => + q.from({ post: source }).orderBy(({ post }) => post.createdAt, `desc`), }) + const { result } = renderHook(() => + useLiveInfiniteQuery(query, { pageSize: 2 }), + ) await waitFor(() => { expect(result.current.isReady).toBe(true) + expect(result.current.hasNextPage).toBe(true) }) - // Initially 1 page - expect(result.current.pages).toHaveLength(1) - expect(result.current.hasNextPage).toBe(true) + const failure = new Error(`window load failed`) + vi.spyOn(query.utils, `setWindow`).mockRejectedValueOnce(failure) - // Fetch next page + let request!: Promise act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) + request = result.current.fetchNextPage() }) - - expect(result.current.pages[0]).toHaveLength(10) - expect(result.current.pages[1]).toHaveLength(10) - expect(result.current.data).toHaveLength(20) - expect(result.current.hasNextPage).toBe(true) - - // Fetch another page - act(() => { - result.current.fetchNextPage() + await act(async () => { + await expect(request).resolves.toBeUndefined() }) await waitFor(() => { - expect(result.current.pages).toHaveLength(3) + expect(result.current.isError).toBe(true) + expect(result.current.error).toBe(failure) + expect(result.current.isFetchingNextPage).toBe(false) }) - - expect(result.current.data).toHaveLength(30) + expect(result.current.pages).toHaveLength(1) expect(result.current.hasNextPage).toBe(true) }) - it(`should detect when no more pages available`, async () => { - const posts = createMockPosts(25) + it(`should derive query identity from structured captured values`, async () => { + const posts = createMockPosts(50) const collection = createCollection( mockSyncCollectionOptions({ autoIndex: `eager`, - id: `no-more-pages-test`, + id: `derived-identity-change-test`, getKey: (post: Post) => post.id, initialData: posts, }), ) - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }, - ) - }) + const { result, rerender } = renderHook( + ({ category }: { category: string }) => { + return useLiveInfiniteQuery( + (q) => + q + .from({ posts: collection }) + .where(({ posts: p }) => eq(p.category, category)) + .orderBy(({ posts: p }) => p.createdAt, `desc`), + { + pageSize: 5, + }, + ) + }, + { initialProps: { category: `tech` } }, + ) await waitFor(() => { expect(result.current.isReady).toBe(true) }) - // Page 1: 10 items, has more - expect(result.current.pages).toHaveLength(1) - expect(result.current.hasNextPage).toBe(true) - - // Fetch page 2 - act(() => { - result.current.fetchNextPage() + await act(async () => { + await result.current.fetchNextPage() }) await waitFor(() => { expect(result.current.pages).toHaveLength(2) }) - // Page 2: 10 items, has more - expect(result.current.pages[1]).toHaveLength(10) - expect(result.current.hasNextPage).toBe(true) - - // Fetch page 3 act(() => { - result.current.fetchNextPage() + rerender({ category: `life` }) }) await waitFor(() => { - expect(result.current.pages).toHaveLength(3) - }) - - // Page 3: 5 items, no more - expect(result.current.pages[2]).toHaveLength(5) - expect(result.current.data).toHaveLength(25) - expect(result.current.hasNextPage).toBe(false) - }) - - it(`should handle empty results`, async () => { - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `empty-results-test`, - getKey: (post: Post) => post.id, - initialData: [], - }), - ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }, - ) + expect(result.current.pages).toHaveLength(1) }) - await waitFor(() => { - expect(result.current.isReady).toBe(true) + result.current.pages[0]!.forEach((post) => { + expect(post.category).toBe(`life`) }) - - // With no data, we still have 1 page (which is empty) - expect(result.current.pages).toHaveLength(1) - expect(result.current.pages[0]).toHaveLength(0) - expect(result.current.data).toHaveLength(0) - expect(result.current.hasNextPage).toBe(false) }) - it(`should update pages when underlying data changes`, async () => { - const posts = createMockPosts(30) - const collection = createCollection( + it(`uses structural queryKey identity without rerunning a stable query`, async () => { + const source = createCollection( mockSyncCollectionOptions({ autoIndex: `eager`, - id: `live-updates-test`, - getKey: (post: Post) => post.id, - initialData: posts, + id: `infinite-query-key-identity`, + getKey: (post) => post.id, + initialData: createMockPosts(20), }), ) + let queryExecutions = 0 - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) + const { result, rerender } = renderHook( + ({ filter }: { filter: { category: string } }) => + useLiveInfiniteQuery( + (q) => { + queryExecutions += 1 + return q + .from({ post: source }) + .where(({ post }) => eq(post.category, filter.category)) + .orderBy(({ post }) => post.createdAt, `desc`) + }, + { + pageSize: 2, + queryKey: [source.id, `category`, filter], + }, + ), + { initialProps: { filter: { category: `tech` } } }, + ) - // Fetch 2 pages - act(() => { - result.current.fetchNextPage() - }) + await waitFor(() => expect(result.current.isReady).toBe(true)) + const firstCollection = result.current.collection + expect(queryExecutions).toBe(1) - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) - }) + rerender({ filter: { category: `tech` } }) - expect(result.current.data).toHaveLength(20) + expect(result.current.collection).toBe(firstCollection) + expect(queryExecutions).toBe(1) - // Insert a new post with most recent timestamp - act(() => { - collection.utils.begin() - collection.utils.write({ - type: `insert`, - value: { - id: `new-1`, - title: `New Post`, - content: `New Content`, - createdAt: 1000001, // Most recent - category: `tech`, - }, - }) - collection.utils.commit() - }) + rerender({ filter: { category: `life` } }) await waitFor(() => { - // New post should be first - expect(result.current.pages[0]![0]).toMatchObject({ - id: `new-1`, - title: `New Post`, - }) + expect(result.current.collection).not.toBe(firstCollection) + expect( + result.current.data.every((post) => post.category === `life`), + ).toBe(true) }) - - // Still showing 2 pages (20 items), but content has shifted - // The new item is included, pushing the last item out of view - expect(result.current.pages).toHaveLength(2) - expect(result.current.data).toHaveLength(20) - expect(result.current.pages[0]).toHaveLength(10) - expect(result.current.pages[1]).toHaveLength(10) + expect(queryExecutions).toBe(2) }) - it(`should handle deletions across pages`, async () => { - const posts = createMockPosts(25) - const collection = createCollection( + it(`uses queryKey to make captured values in opaque queries reactive`, async () => { + const source = createCollection( mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `deletions-test`, - getKey: (post: Post) => post.id, - initialData: posts, + id: `infinite-opaque-query-key`, + getKey: (post) => post.id, + initialData: createMockPosts(20), }), ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - // Fetch 2 pages - act(() => { - result.current.fetchNextPage() - }) + const { result, rerender } = renderHook( + ({ category }: { category: string }) => + useLiveInfiniteQuery( + (q) => + q + .from({ post: source }) + .fn.where(({ post }) => post.category === category) + .orderBy(({ post }) => post.createdAt, `desc`), + { + pageSize: 2, + queryKey: [source.id, `category-fn`, category], + }, + ), + { initialProps: { category: `tech` } }, + ) await waitFor(() => { - expect(result.current.pages).toHaveLength(2) + expect(result.current.data.length).toBeGreaterThan(0) + expect( + result.current.data.every((post) => post.category === `tech`), + ).toBe(true) }) + const firstCollection = result.current.collection - expect(result.current.data).toHaveLength(20) - const firstItemId = result.current.data[0]!.id - - // Delete the first item - act(() => { - collection.utils.begin() - collection.utils.write({ - type: `delete`, - value: posts[0]!, - }) - collection.utils.commit() - }) + rerender({ category: `life` }) await waitFor(() => { - // First item should have changed - expect(result.current.data[0]!.id).not.toBe(firstItemId) + expect(result.current.collection).not.toBe(firstCollection) + expect(result.current.data.length).toBeGreaterThan(0) + expect( + result.current.data.every((post) => post.category === `life`), + ).toBe(true) }) - - // Still showing 2 pages, each pulls from remaining 24 items - // Page 1: items 0-9 (10 items) - // Page 2: items 10-19 (10 items) - // Total: 20 items (item 20-23 are beyond our loaded pages) - expect(result.current.pages).toHaveLength(2) - expect(result.current.data).toHaveLength(20) - expect(result.current.pages[0]).toHaveLength(10) - expect(result.current.pages[1]).toHaveLength(10) }) - it(`should handle deletion from partial page with descending order`, async () => { - // Create only 5 items - fewer than the pageSize of 20 - const posts = createMockPosts(5) - const collection = createCollection( + it(`compares dependencies by identity instead of serialization`, async () => { + const source = createCollection( mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `partial-page-deletion-desc-test`, - getKey: (post: Post) => post.id, - initialData: posts, + id: `infinite-query-map-deps`, + getKey: (post) => post.id, + initialData: createMockPosts(10), }), ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 20, - getNextPageParam: (lastPage) => - lastPage.length === 20 ? lastPage.length : undefined, + const { result, rerender } = renderHook( + ({ filter }: { filter: Map }) => + useLiveInfiniteQuery( + (q) => + q + .from({ post: source }) + .where(({ post }) => eq(post.category, filter.get(`category`))) + .orderBy(({ post }) => post.createdAt, `desc`), + { pageSize: 3 }, + [filter], + ), + { + initialProps: { + filter: new Map([[`category`, `tech`]]), }, - ) - }) + }, + ) await waitFor(() => { expect(result.current.isReady).toBe(true) + expect(result.current.data.length).toBeGreaterThan(0) + expect( + result.current.data.every((post) => post.category === `tech`), + ).toBe(true) }) - // Should have all 5 items on one page (partial page) - expect(result.current.pages).toHaveLength(1) - expect(result.current.data).toHaveLength(5) - expect(result.current.hasNextPage).toBe(false) - - // Verify the first item (most recent by createdAt descending) - const firstItemId = result.current.data[0]!.id - expect(firstItemId).toBe(`1`) // Post 1 has the highest createdAt - - // Delete the first item (the one that appears first in descending order) - act(() => { - collection.utils.begin() - collection.utils.write({ - type: `delete`, - value: posts[0]!, // Post 1 - }) - collection.utils.commit() - }) + rerender({ filter: new Map([[`category`, `life`]]) }) - // The deleted item should disappear from the result await waitFor(() => { - expect(result.current.data).toHaveLength(4) + expect(result.current.data.length).toBeGreaterThan(0) + expect( + result.current.data.every((post) => post.category === `life`), + ).toBe(true) }) - - // Verify the deleted item is no longer in the data - expect( - result.current.data.find((p) => p.id === firstItemId), - ).toBeUndefined() - - // Verify the new first item is Post 2 - expect(result.current.data[0]!.id).toBe(`2`) - - // Still should have 1 page with 4 items - expect(result.current.pages).toHaveLength(1) - expect(result.current.pages[0]).toHaveLength(4) - expect(result.current.hasNextPage).toBe(false) }) - it(`should handle deletion from partial page with ascending order`, async () => { - // Create only 5 items - fewer than the pageSize of 20 - const posts = createMockPosts(5) - const collection = createCollection( + it(`preserves loaded pages when dependencies are structurally unchanged`, async () => { + const source = createCollection( mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `partial-page-deletion-asc-test`, - getKey: (post: Post) => post.id, - initialData: posts, + id: `infinite-query-structurally-equal-deps`, + getKey: (post) => post.id, + initialData: createMockPosts(20), }), ) + const { result, rerender } = renderHook( + ({ filter }: { filter: { category: string } }) => + useLiveInfiniteQuery( + (q) => + q + .from({ post: source }) + .where(({ post }) => eq(post.category, filter.category)) + .orderBy(({ post }) => post.createdAt, `desc`), + { pageSize: 2 }, + [filter], + ), + { initialProps: { filter: { category: `tech` } } }, + ) - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `asc`), // ascending order - { - pageSize: 20, - getNextPageParam: (lastPage) => - lastPage.length === 20 ? lastPage.length : undefined, - }, - ) + await waitFor(() => expect(result.current.isReady).toBe(true)) + await act(async () => { + await result.current.fetchNextPage() }) + await waitFor(() => expect(result.current.pages).toHaveLength(2)) + const firstCollection = result.current.collection + + rerender({ filter: { category: `tech` } }) await waitFor(() => { + expect(result.current.collection).not.toBe(firstCollection) expect(result.current.isReady).toBe(true) }) + expect(result.current.pages).toHaveLength(2) + }) - // Should have all 5 items on one page (partial page) - expect(result.current.pages).toHaveLength(1) - expect(result.current.data).toHaveLength(5) - expect(result.current.hasNextPage).toBe(false) - - // In ascending order, Post 5 has the lowest createdAt and appears first - const firstItemId = result.current.data[0]!.id - expect(firstItemId).toBe(`5`) // Post 5 has the lowest createdAt - - // Delete the first item (the one that appears first in ascending order) - act(() => { - collection.utils.begin() - collection.utils.write({ - type: `delete`, - value: posts[4]!, // Post 5 (index 4 in array) - }) - collection.utils.commit() + it(`releases a replaced controller through the external-store unsubscribe`, async () => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `infinite-query-controller-replacement`, + getKey: (post) => post.id, + initialData: createMockPosts(20), + }), + ) + const query = createLiveQueryCollection({ + query: (q) => + q.from({ post: source }).orderBy(({ post }) => post.createdAt, `desc`), }) + const { result, rerender, unmount } = renderHook( + ({ pageSize }) => useLiveInfiniteQuery(query, { pageSize }), + { initialProps: { pageSize: 2 } }, + ) - // The deleted item should disappear from the result - await waitFor(() => { - expect(result.current.data).toHaveLength(4) - }) + await waitFor(() => expect(result.current.isReady).toBe(true)) + expect(query.subscriberCount).toBe(1) - // Verify the deleted item is no longer in the data - expect( - result.current.data.find((p) => p.id === firstItemId), - ).toBeUndefined() + rerender({ pageSize: 3 }) + await waitFor(() => expect(result.current.pages[0]).toHaveLength(3)) + expect(query.subscriberCount).toBe(1) - // Still should have 1 page with 4 items - expect(result.current.pages).toHaveLength(1) - expect(result.current.pages[0]).toHaveLength(4) - expect(result.current.hasNextPage).toBe(false) + unmount() + expect(query.subscriberCount).toBe(0) }) - it(`should work with where clauses`, async () => { - const posts = createMockPosts(50) - const collection = createCollection( + it(`binds fetchNextPage to the controller that returned it`, async () => { + const sourceA = createCollection( mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `where-clause-test`, - getKey: (post: Post) => post.id, - initialData: posts, + id: `infinite-query-generation-a`, + getKey: (post) => post.id, + initialData: createMockPosts(10), }), ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .where(({ posts: p }) => eq(p.category, `tech`)) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 5, - getNextPageParam: (lastPage) => - lastPage.length === 5 ? lastPage.length : undefined, - }, - ) + const sourceB = createCollection( + mockSyncCollectionOptions({ + id: `infinite-query-generation-b`, + getKey: (post) => post.id, + initialData: createMockPosts(10), + }), + ) + const queryA = createLiveQueryCollection({ + query: (q) => + q.from({ post: sourceA }).orderBy(({ post }) => post.createdAt, `desc`), }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) + const queryB = createLiveQueryCollection({ + query: (q) => + q.from({ post: sourceB }).orderBy(({ post }) => post.createdAt, `desc`), }) + const { result, rerender } = renderHook( + ({ query }) => useLiveInfiniteQuery(query, { pageSize: 2 }), + { initialProps: { query: queryA } }, + ) - // Should only have tech posts (every even ID) - expect(result.current.pages).toHaveLength(1) - expect(result.current.pages[0]).toHaveLength(5) + await waitFor(() => expect(result.current.isReady).toBe(true)) + const fetchFromA = result.current.fetchNextPage - // All items should be tech category - result.current.pages[0]!.forEach((post) => { - expect(post.category).toBe(`tech`) + rerender({ query: queryB }) + await waitFor(() => { + expect(result.current.collection).toBe(queryB) + expect(result.current.isReady).toBe(true) + expect(result.current.pages).toHaveLength(1) }) - // Should have more pages - expect(result.current.hasNextPage).toBe(true) - - // Fetch next page act(() => { - result.current.fetchNextPage() + void fetchFromA() }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) + await act(async () => { + await new Promise((resolve) => setTimeout(resolve, 0)) }) + expect(result.current.pages).toHaveLength(1) - expect(result.current.data).toHaveLength(10) - }) - - it(`should re-execute query when dependencies change`, async () => { - const posts = createMockPosts(50) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `deps-change-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const { result, rerender } = renderHook( - ({ category }: { category: string }) => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .where(({ posts: p }) => eq(p.category, category)) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 5, - getNextPageParam: (lastPage) => - lastPage.length === 5 ? lastPage.length : undefined, - }, - [category], - ) - }, - { initialProps: { category: `tech` } }, - ) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - // Fetch 2 pages of tech posts - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) - }) - - // Change category to life - act(() => { - rerender({ category: `life` }) - }) - - await waitFor(() => { - // Should reset to 1 page with life posts - expect(result.current.pages).toHaveLength(1) - }) - - // All items should be life category - result.current.pages[0]!.forEach((post) => { - expect(post.category).toBe(`life`) - }) - }) - - it(`should track pageParams correctly`, async () => { - const posts = createMockPosts(30) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `page-params-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - initialPageParam: 0, - getNextPageParam: (lastPage, _allPages, lastPageParam) => - lastPage.length === 10 ? lastPageParam + 1 : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - expect(result.current.pageParams).toEqual([0]) - - // Fetch next page - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pageParams).toEqual([0, 1]) - }) - - // Fetch another page - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pageParams).toEqual([0, 1, 2]) - }) - }) - - it(`should handle exact page size boundaries`, async () => { - const posts = createMockPosts(20) // Exactly 2 pages - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `exact-boundary-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - // Better getNextPageParam that checks against total data available - getNextPageParam: (lastPage, allPages) => { - // If last page is not full, we're done - if (lastPage.length < 10) return undefined - // Check if we've likely loaded all data (this is a heuristic) - // In a real app with backend, you'd check response metadata - const totalLoaded = allPages.flat().length - // If we have less than a full page left, no more pages - return totalLoaded - }, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - expect(result.current.hasNextPage).toBe(true) - - // Fetch page 2 - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) - }) - - expect(result.current.pages[1]).toHaveLength(10) - // With setWindow peek-ahead, we can now detect no more pages immediately - // We request 21 items (2 * 10 + 1 peek) but only get 20, so we know there's no more - expect(result.current.hasNextPage).toBe(false) - - // Verify total data - expect(result.current.data).toHaveLength(20) - }) - - it(`should not fetch when already fetching`, async () => { - const posts = createMockPosts(50) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `concurrent-fetch-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - expect(result.current.pages).toHaveLength(1) - - // With sync data, all fetches complete immediately, so all 3 calls will succeed - // The key is that they won't cause race conditions or errors - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) - }) - - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(3) - }) - - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(4) - }) - - // All fetches should have succeeded - expect(result.current.pages).toHaveLength(4) - expect(result.current.data).toHaveLength(40) - }) - - it(`should not fetch when hasNextPage is false`, async () => { - const posts = createMockPosts(5) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `no-fetch-when-done-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - expect(result.current.hasNextPage).toBe(false) - expect(result.current.pages).toHaveLength(1) - - // Try to fetch when there's no next page - act(() => { - result.current.fetchNextPage() - }) - - await new Promise((resolve) => setTimeout(resolve, 50)) - - // Should still have only 1 page - expect(result.current.pages).toHaveLength(1) - }) - - it(`should support custom initialPageParam`, async () => { - const posts = createMockPosts(30) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `initial-param-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - initialPageParam: 100, - getNextPageParam: (lastPage, _allPages, lastPageParam) => - lastPage.length === 10 ? lastPageParam + 1 : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - expect(result.current.pageParams).toEqual([100]) - - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pageParams).toEqual([100, 101]) - }) - }) - - it(`should detect hasNextPage change when new items are synced`, async () => { - // Start with exactly 20 items (2 pages) - const posts = createMockPosts(20) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `sync-detection-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - // Load both pages - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) - }) - - // Should have no next page (exactly 20 items, 2 full pages, peek returns nothing) - expect(result.current.hasNextPage).toBe(false) - expect(result.current.data).toHaveLength(20) - - // Add 5 more items to the collection - act(() => { - collection.utils.begin() - for (let i = 0; i < 5; i++) { - collection.utils.write({ - type: `insert`, - value: { - id: `new-${i}`, - title: `New Post ${i}`, - content: `Content ${i}`, - createdAt: Date.now() + i, - category: `tech`, - }, - }) - } - collection.utils.commit() - }) - - // Should now detect that there's a next page available - await waitFor(() => { - expect(result.current.hasNextPage).toBe(true) - }) - - // Data should still be 20 items (we haven't fetched the next page yet) - expect(result.current.data).toHaveLength(20) - expect(result.current.pages).toHaveLength(2) - - // Fetch the next page - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(3) - }) - - // Third page should have the new items - expect(result.current.pages[2]).toHaveLength(5) - expect(result.current.data).toHaveLength(25) - - // No more pages available now - expect(result.current.hasNextPage).toBe(false) - }) - - it(`should set isFetchingNextPage to false when data is immediately available`, async () => { - const posts = createMockPosts(50) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `immediate-data-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - // Initially 1 page and not fetching - expect(result.current.pages).toHaveLength(1) - expect(result.current.isFetchingNextPage).toBe(false) - - // Fetch next page - should remain false because data is immediately available - act(() => { - result.current.fetchNextPage() - }) - - // Since data is *synchronously* available, isFetchingNextPage should be false - expect(result.current.pages).toHaveLength(2) - expect(result.current.isFetchingNextPage).toBe(false) - }) - - it(`should request limit+1 (peek-ahead) from loadSubset for hasNextPage detection`, async () => { - // Verifies that useLiveInfiniteQuery requests pageSize+1 items from loadSubset - // to detect whether there are more pages available (peek-ahead strategy) - const PAGE_SIZE = 10 - const { collection, loadSubsetCalls } = createOnDemandCollection({ - id: `peek-ahead-limit-test`, - allPosts: createMockPosts(PAGE_SIZE), // Exactly PAGE_SIZE posts - }) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: PAGE_SIZE, - getNextPageParam: (lastPage) => - lastPage.length === PAGE_SIZE ? lastPage.length : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - const callWithLimit = loadSubsetCalls.find( - (call) => call.limit !== undefined, - ) - expect(callWithLimit).toBeDefined() - expect(callWithLimit!.limit).toBe(PAGE_SIZE + 1) - - // With exactly PAGE_SIZE posts, hasNextPage should be false (no peek-ahead item returned) - expect(result.current.hasNextPage).toBe(false) - expect(result.current.data).toHaveLength(PAGE_SIZE) - }) - - it(`should detect hasNextPage via peek-ahead with exactly pageSize+1 items in on-demand collection`, async () => { - // Boundary test: with exactly pageSize+1 items, the peek-ahead item should - // signal hasNextPage=true but NOT appear in user-visible data - const PAGE_SIZE = 10 - const { collection } = createOnDemandCollection({ - id: `peek-ahead-boundary-test`, - allPosts: createMockPosts(PAGE_SIZE + 1), - }) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: PAGE_SIZE, - getNextPageParam: (lastPage) => - lastPage.length === PAGE_SIZE ? lastPage.length : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - // Peek-ahead item detected: hasNextPage should be true - expect(result.current.hasNextPage).toBe(true) - // But user-visible data should be exactly pageSize (peek-ahead excluded) - expect(result.current.data).toHaveLength(PAGE_SIZE) - expect(result.current.pages).toHaveLength(1) - expect(result.current.pages[0]).toHaveLength(PAGE_SIZE) - }) - - it(`should work with on-demand collection and fetch multiple pages`, async () => { - // End-to-end test: on-demand collection where ALL data comes from loadSubset - // (no initial data). Simulates the real Electric on-demand scenario. - const PAGE_SIZE = 10 - const { collection, loadSubsetCalls } = createOnDemandCollection({ - id: `on-demand-e2e-test`, - allPosts: createMockPosts(25), // 2 full pages + 5 items - autoIndex: `eager`, - }) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: PAGE_SIZE, - getNextPageParam: (lastPage) => - lastPage.length === PAGE_SIZE ? lastPage.length : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - // Page 1: 10 items - expect(result.current.pages).toHaveLength(1) - expect(result.current.data).toHaveLength(PAGE_SIZE) - expect(result.current.hasNextPage).toBe(true) - expect(result.current.data[0]!.id).toBe(`1`) - expect(result.current.data[9]!.id).toBe(`10`) - - // Fetch page 2 - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) - }) - - expect(loadSubsetCalls.length).toBeGreaterThan(1) - expect(result.current.data).toHaveLength(20) - expect(result.current.hasNextPage).toBe(true) - expect(result.current.pages[1]![0]!.id).toBe(`11`) - expect(result.current.pages[1]![9]!.id).toBe(`20`) - - // Fetch page 3 (partial page) - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(3) - }) - - expect(result.current.data).toHaveLength(25) - expect(result.current.pages[2]).toHaveLength(5) - expect(result.current.hasNextPage).toBe(false) - expect(result.current.pages[2]![0]!.id).toBe(`21`) - expect(result.current.pages[2]![4]!.id).toBe(`25`) - }) - - it(`should work with on-demand collection with async loadSubset`, async () => { - // Same as the sync on-demand test, but loadSubset returns a Promise - // to simulate async network requests (the real Electric scenario). - const PAGE_SIZE = 10 - const { collection, loadSubsetCalls } = createOnDemandCollection({ - id: `on-demand-async-test`, - allPosts: createMockPosts(25), - autoIndex: `eager`, - asyncDelay: 10, - }) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: PAGE_SIZE, - getNextPageParam: (lastPage) => - lastPage.length === PAGE_SIZE ? lastPage.length : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - await waitFor(() => { - expect(result.current.data).toHaveLength(PAGE_SIZE) - }) - - expect(result.current.pages).toHaveLength(1) - expect(result.current.hasNextPage).toBe(true) - - const initialCallCount = loadSubsetCalls.length - - // Fetch page 2 - act(() => { - result.current.fetchNextPage() - }) - - expect(result.current.isFetchingNextPage).toBe(true) - - await waitFor( - () => { - expect(result.current.data).toHaveLength(20) - }, - { timeout: 500 }, - ) - - expect(result.current.pages).toHaveLength(2) - expect(loadSubsetCalls.length).toBeGreaterThan(initialCallCount) - expect(result.current.hasNextPage).toBe(true) - - // Fetch page 3 (partial page) to verify async path handles end-of-data - const callCountBeforePage3 = loadSubsetCalls.length - - act(() => { - result.current.fetchNextPage() - }) - - await waitFor( - () => { - expect(result.current.data).toHaveLength(25) - }, - { timeout: 500 }, - ) - - expect(result.current.pages).toHaveLength(3) - expect(result.current.pages[2]).toHaveLength(5) - expect(loadSubsetCalls.length).toBeGreaterThan(callCountBeforePage3) - expect(result.current.hasNextPage).toBe(false) - }) - - it(`should track isFetchingNextPage when async loading is triggered`, async () => { - // Define all data upfront - const allPosts = createMockPosts(30) - - const collection = createCollection({ - id: `async-loading-test`, - getKey: (post: Post) => post.id, - syncMode: `on-demand`, - startSync: true, - autoIndex: `eager`, - defaultIndexType: BTreeIndex, - sync: { - sync: ({ markReady, begin, write, commit }) => { - // Provide initial data by slicing the first 15 elements - begin() - const initialPosts = allPosts.slice(0, 15) - for (const post of initialPosts) { - write({ - type: `insert`, - value: post, - }) - } - commit() - markReady() - - return { - loadSubset: (opts: LoadSubsetOptions) => { - // Filter the data array based on opts - let filtered = allPosts - - // Apply where clause if provided - if (opts.where) { - const filterFn = createFilterFunctionFromExpression(opts.where) - filtered = filtered.filter(filterFn) - } - - // Sort by createdAt descending if orderBy is provided - if (opts.orderBy && opts.orderBy.length > 0) { - filtered = filtered.sort((a, b) => { - // We know ordering is always by createdAt descending - return b.createdAt - a.createdAt - }) - } - - // Apply cursor expressions if present (new cursor-based pagination) - if (opts.cursor) { - const { whereFrom, whereCurrent } = opts.cursor - try { - const whereFromFn = - createFilterFunctionFromExpression(whereFrom) - const fromData = filtered.filter(whereFromFn) - - const whereCurrentFn = - createFilterFunctionFromExpression(whereCurrent) - const currentData = filtered.filter(whereCurrentFn) - - // Combine current (ties) with from (next page), deduplicate - const seenIds = new Set() - filtered = [] - for (const item of currentData) { - if (!seenIds.has(item.id)) { - seenIds.add(item.id) - filtered.push(item) - } - } - // Apply limit only to fromData - const limitedFromData = opts.limit - ? fromData.slice(0, opts.limit) - : fromData - for (const item of limitedFromData) { - if (!seenIds.has(item.id)) { - seenIds.add(item.id) - filtered.push(item) - } - } - // Re-sort after combining - filtered.sort((a, b) => b.createdAt - a.createdAt) - } catch (e) { - throw new Error(`Test loadSubset: cursor parsing failed`, { - cause: e, - }) - } - } else if (opts.limit !== undefined) { - // Apply limit only if no cursor (cursor handles limit internally) - filtered = filtered.slice(0, opts.limit) - } - - // Subsequent calls simulate async loading with a real timeout - const loadPromise = new Promise((resolve) => { - setTimeout(() => { - begin() - - // Insert the requested posts - for (const post of filtered) { - write({ - type: `insert`, - value: post, - }) - } - - commit() - resolve() - }, 50) - }) - - return loadPromise - }, - } - }, - }, - }) - - const { result } = renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }, - ) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - // Wait for initial window setup to complete - await waitFor(() => { - expect(result.current.isFetchingNextPage).toBe(false) - }) - - expect(result.current.pages).toHaveLength(1) - - // Fetch next page which will trigger async loading act(() => { - result.current.fetchNextPage() - }) - - // Should be fetching now and so isFetchingNextPage should be true *synchronously!* - expect(result.current.isFetchingNextPage).toBe(true) - - // Wait for loading to complete - await waitFor( - () => { - expect(result.current.isFetchingNextPage).toBe(false) - }, - { timeout: 200 }, - ) - - // Should have 2 pages now - expect(result.current.pages).toHaveLength(2) - expect(result.current.data).toHaveLength(20) - }, 10000) - - describe(`pre-created collections`, () => { - it(`should accept pre-created live query collection`, async () => { - const posts = createMockPosts(50) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `pre-created-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const liveQueryCollection = createLiveQueryCollection({ - query: (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`) - .limit(5), // Initial limit - }) - - await liveQueryCollection.preload() - - const { result } = renderHook(() => { - return useLiveInfiniteQuery(liveQueryCollection, { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - // Should have 1 page initially - expect(result.current.pages).toHaveLength(1) - expect(result.current.pages[0]).toHaveLength(10) - expect(result.current.data).toHaveLength(10) - expect(result.current.hasNextPage).toBe(true) - - // First item should be Post 1 (most recent by createdAt) - expect(result.current.pages[0]![0]).toMatchObject({ - id: `1`, - title: `Post 1`, - }) - }) - - it(`should fetch multiple pages with pre-created collection`, async () => { - const posts = createMockPosts(50) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `pre-created-multi-page-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const liveQueryCollection = createLiveQueryCollection({ - query: (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`) - .limit(10) - .offset(0), - }) - - await liveQueryCollection.preload() - - const { result } = renderHook(() => { - return useLiveInfiniteQuery(liveQueryCollection, { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - expect(result.current.pages).toHaveLength(1) - expect(result.current.hasNextPage).toBe(true) - - // Fetch next page - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) - }) - - expect(result.current.pages[0]).toHaveLength(10) - expect(result.current.pages[1]).toHaveLength(10) - expect(result.current.data).toHaveLength(20) - expect(result.current.hasNextPage).toBe(true) - }) - - it(`should reset pagination when collection instance changes`, async () => { - const posts1 = createMockPosts(30) - const collection1 = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `pre-created-reset-1`, - getKey: (post: Post) => post.id, - initialData: posts1, - }), - ) - - const liveQueryCollection1 = createLiveQueryCollection({ - query: (q) => - q - .from({ posts: collection1 }) - .orderBy(({ posts: p }) => p.createdAt, `desc`) - .limit(10) - .offset(0), - }) - - await liveQueryCollection1.preload() - - const posts2 = createMockPosts(40) - const collection2 = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `pre-created-reset-2`, - getKey: (post: Post) => post.id, - initialData: posts2, - }), - ) - - const liveQueryCollection2 = createLiveQueryCollection({ - query: (q) => - q - .from({ posts: collection2 }) - .orderBy(({ posts: p }) => p.createdAt, `desc`) - .limit(10) - .offset(0), - }) - - await liveQueryCollection2.preload() - - const { result, rerender } = renderHook( - ({ coll }: { coll: any }) => { - return useLiveInfiniteQuery(coll, { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }) - }, - { initialProps: { coll: liveQueryCollection1 } }, - ) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - // Fetch 2 pages - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) - }) - - expect(result.current.data).toHaveLength(20) - - // Switch to second collection - act(() => { - rerender({ coll: liveQueryCollection2 }) - }) - - await waitFor(() => { - // Should reset to 1 page - expect(result.current.pages).toHaveLength(1) - }) - - expect(result.current.data).toHaveLength(10) - }) - - it(`should throw error if collection lacks orderBy`, async () => { - const posts = createMockPosts(50) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `no-orderby-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - // Create collection WITHOUT orderBy - const liveQueryCollection = createLiveQueryCollection({ - query: (q) => q.from({ posts: collection }), - }) - - await liveQueryCollection.preload() - - // Should throw error when trying to use it with useLiveInfiniteQuery - expect(() => { - renderHook(() => { - return useLiveInfiniteQuery(liveQueryCollection, { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }) - }) - }).toThrow(/ORDER BY/) - }) - - it(`should throw error if first argument is not a collection or function`, () => { - // Should throw error when passing invalid types - expect(() => { - renderHook(() => { - return useLiveInfiniteQuery(`not a collection or function` as any, { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }) - }) - }).toThrow(/must be either a pre-created live query collection/) - - expect(() => { - renderHook(() => { - return useLiveInfiniteQuery(123 as any, { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }) - }) - }).toThrow(/must be either a pre-created live query collection/) - - expect(() => { - renderHook(() => { - return useLiveInfiniteQuery(null as any, { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }) - }) - }).toThrow(/must be either a pre-created live query collection/) - }) - - it(`should work correctly even if pre-created collection has different initial limit`, async () => { - const posts = createMockPosts(50) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `mismatched-window-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const liveQueryCollection = createLiveQueryCollection({ - query: (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`) - .limit(5) // Different from pageSize - .offset(0), - }) - - await liveQueryCollection.preload() - - const { result } = renderHook(() => { - return useLiveInfiniteQuery(liveQueryCollection, { - pageSize: 10, // Different from the initial limit of 5 - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - // Should work correctly despite different initial limit - // The window will be adjusted to match pageSize - expect(result.current.pages).toHaveLength(1) - expect(result.current.pages[0]).toHaveLength(10) - expect(result.current.data).toHaveLength(10) - expect(result.current.hasNextPage).toBe(true) - }) - - it(`should handle live updates with pre-created collection`, async () => { - const posts = createMockPosts(30) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `pre-created-live-updates-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const liveQueryCollection = createLiveQueryCollection({ - query: (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`) - .limit(10) - .offset(0), - }) - - await liveQueryCollection.preload() - - const { result } = renderHook(() => { - return useLiveInfiniteQuery(liveQueryCollection, { - pageSize: 10, - getNextPageParam: (lastPage) => - lastPage.length === 10 ? lastPage.length : undefined, - }) - }) - - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - // Fetch 2 pages - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) - }) - - expect(result.current.data).toHaveLength(20) - - // Insert a new post with most recent timestamp - act(() => { - collection.utils.begin() - collection.utils.write({ - type: `insert`, - value: { - id: `new-1`, - title: `New Post`, - content: `New Content`, - createdAt: 1000001, // Most recent - category: `tech`, - }, - }) - collection.utils.commit() - }) - - await waitFor(() => { - // New post should be first - expect(result.current.pages[0]![0]).toMatchObject({ - id: `new-1`, - title: `New Post`, - }) - }) - - // Still showing 2 pages (20 items), but content has shifted - expect(result.current.pages).toHaveLength(2) - expect(result.current.data).toHaveLength(20) - }) - - it(`should work with router loader pattern (preloaded collection)`, async () => { - const posts = createMockPosts(50) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `router-loader-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - // Simulate router loader: create and preload collection - const loaderQuery = createLiveQueryCollection({ - query: (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`) - .limit(20), - }) - - // Preload in loader - await loaderQuery.preload() - - // Simulate component receiving preloaded collection - const { result } = renderHook(() => { - return useLiveInfiniteQuery(loaderQuery, { - pageSize: 20, - getNextPageParam: (lastPage) => - lastPage.length === 20 ? lastPage.length : undefined, - }) - }) - - // Should be immediately ready since it was preloaded - await waitFor(() => { - expect(result.current.isReady).toBe(true) - }) - - expect(result.current.pages).toHaveLength(1) - expect(result.current.pages[0]).toHaveLength(20) - expect(result.current.data).toHaveLength(20) - expect(result.current.hasNextPage).toBe(true) - - // Can still fetch more pages - act(() => { - result.current.fetchNextPage() - }) - - await waitFor(() => { - expect(result.current.pages).toHaveLength(2) - }) - - expect(result.current.data).toHaveLength(40) + void result.current.fetchNextPage() }) - }) - - it(`throws a descriptive error when deps contain non-serializable values`, () => { - const posts = createMockPosts(10) - const collection = createCollection( - mockSyncCollectionOptions({ - autoIndex: `eager`, - id: `circular-deps-test`, - getKey: (post: Post) => post.id, - initialData: posts, - }), - ) - - const circular: Record = { a: 1 } - circular.self = circular - - expect(() => { - renderHook(() => { - return useLiveInfiniteQuery( - (q) => - q - .from({ posts: collection }) - .orderBy(({ posts: p }) => p.createdAt, `desc`), - { - pageSize: 5, - getNextPageParam: (lastPage) => - lastPage.length === 5 ? lastPage.length : undefined, - }, - [circular], - ) - }) - }).toThrow(/useLiveInfiniteQuery.*dependency/) + await waitFor(() => expect(result.current.pages).toHaveLength(2)) }) }) diff --git a/packages/react-db/tests/useLiveQuery.eager-onstorechange.test.tsx b/packages/react-db/tests/useLiveQuery.eager-onstorechange.test.tsx new file mode 100644 index 0000000000..29f801e837 --- /dev/null +++ b/packages/react-db/tests/useLiveQuery.eager-onstorechange.test.tsx @@ -0,0 +1,78 @@ +import { describe, expect, it, vi } from 'vitest' + +import { renderHook } from '@testing-library/react' +import { createCollection, createLiveQueryCollection } from '@tanstack/db' +import { useLiveQuery } from '../src/useLiveQuery' +import { mockSyncCollectionOptions } from '../../db/tests/utils' +import type * as ReactNS from 'react' + +// Intercept React.useSyncExternalStore so we can capture the `subscribe` +// callback that `useLiveQuery` registers and assert that it does not invoke +// `onStoreChange` synchronously when the collection is already ready. +let capturedSubscribe: ((cb: () => void) => () => void) | null = null + +vi.mock('react', async () => { + const actual = await vi.importActual('react') + return { + ...actual, + default: (actual as any).default ?? actual, + useSyncExternalStore: (subscribe: any, getSnapshot: any) => { + capturedSubscribe = subscribe + return getSnapshot() + }, + } +}) + +type Person = { id: string; name: string; age: number } + +const initialPersons: Array = [ + { id: `1`, name: `A`, age: 10 }, + { id: `2`, name: `B`, age: 20 }, +] + +describe(`useLiveQuery: eager onStoreChange must not fire synchronously during subscribe`, () => { + it(`defers the initial ready-state onStoreChange to a microtask`, async () => { + const base = createCollection( + mockSyncCollectionOptions({ + id: `eager-onstorechange-persons`, + getKey: (p) => p.id, + initialData: initialPersons, + }), + ) + + const lqc = createLiveQueryCollection({ + startSync: true, + query: (q) => q.from({ persons: base }), + }) + await lqc.preload() + expect(lqc.status).toBe(`ready`) + + capturedSubscribe = null + renderHook(() => useLiveQuery(lqc)) + expect(capturedSubscribe).toBeTypeOf(`function`) + + const onStoreChange = vi.fn() + const unsub = capturedSubscribe!(onStoreChange) + + // onStoreChange must not be invoked synchronously inside subscribe — + // useSyncExternalStore's own post-subscribe getSnapshot re-read covers a + // ready transition that happened between render and subscribe, so an + // already-ready unchanged collection needs no wake-up at all. + expect(onStoreChange).not.toHaveBeenCalled() + + await Promise.resolve() + expect(onStoreChange).not.toHaveBeenCalled() + + // A real delta does wake the store. + base.utils.begin() + base.utils.write({ + type: `insert`, + value: { id: `3`, name: `C`, age: 30 }, + }) + base.utils.commit() + await Promise.resolve() + expect(onStoreChange).toHaveBeenCalled() + + unsub() + }) +}) diff --git a/packages/react-db/tests/useLiveQuery.strictmode.test.tsx b/packages/react-db/tests/useLiveQuery.strictmode.test.tsx new file mode 100644 index 0000000000..874d020625 --- /dev/null +++ b/packages/react-db/tests/useLiveQuery.strictmode.test.tsx @@ -0,0 +1,44 @@ +import { describe, expect, it } from 'vitest' +import { StrictMode } from 'react' +import { act, renderHook, waitFor } from '@testing-library/react' +import { createCollection } from '@tanstack/db' +import { useLiveQuery } from '../src/useLiveQuery' +import { mockSyncCollectionOptions } from '../../db/tests/utils' + +type Person = { id: string; name: string } + +describe(`useLiveQuery under StrictMode`, () => { + it(`keeps the subscription alive across StrictMode effect replay`, async () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `strictmode-persons`, + getKey: (p) => p.id, + initialData: [{ id: `1`, name: `A` }], + }), + ) + + // StrictMode double-invokes effects (mount → cleanup → mount). A dispose in + // the unmount effect would tear the observer down and never recreate it, + // leaving a dead subscription. + const { result } = renderHook( + () => + useLiveQuery((q) => + q + .from({ p: collection }) + .select(({ p }) => ({ id: p.id, name: p.name })), + ), + { wrapper: StrictMode }, + ) + + await waitFor(() => expect(result.current.data).toHaveLength(1)) + + // A mutation after the StrictMode replay must still reach the hook. + act(() => { + collection.utils.begin() + collection.utils.write({ type: `insert`, value: { id: `2`, name: `B` } }) + collection.utils.commit() + }) + + await waitFor(() => expect(result.current.data).toHaveLength(2)) + }) +}) diff --git a/packages/react-db/tests/useLiveQuery.test-d.tsx b/packages/react-db/tests/useLiveQuery.test-d.tsx index 43747284d8..c541e71e74 100644 --- a/packages/react-db/tests/useLiveQuery.test-d.tsx +++ b/packages/react-db/tests/useLiveQuery.test-d.tsx @@ -1,15 +1,29 @@ import { describe, expectTypeOf, it } from 'vitest' import { renderHook } from '@testing-library/react' import { createCollection } from '../../db/src/collection/index' +import { collectionOptions } from '../../db/src/index' import { mockSyncCollectionOptions } from '../../db/tests/utils' import { + Query, createLiveQueryCollection, eq, liveQueryCollectionOptions, } from '../../db/src/query/index' import { useLiveQuery } from '../src/useLiveQuery' +import { useLiveInfiniteQuery } from '../src/useLiveInfiniteQuery' +import { useLiveSuspenseQuery } from '../src/useLiveSuspenseQuery' +import { useDbClient } from '../src/DbProvider' +import { HydrationBoundary } from '../src/HydrationBoundary' +import type { DbClient, DehydratedDbState } from '../../db/src/index' +import type { JSX } from 'react' import type { OutputWithVirtual } from '../../db/tests/utils' import type { SingleResult } from '../../db/src/types' +import type { QueryBuilder } from '../../db/src/query/index' +import type { + ConditionalUseLiveQueryConfig, + UseLiveQueryConfig, + UseLiveQueryStatus, +} from '../src/index' type Person = { id: string @@ -21,6 +35,22 @@ type Person = { } describe(`useLiveQuery type assertions`, () => { + it(`should type useDbClient as DbClient`, () => { + const client = useDbClient() + expectTypeOf(client).toEqualTypeOf() + }) + + it(`types HydrationBoundary state`, () => { + const state: DehydratedDbState = { collections: [] } + const boundary = ( + +

                            + + ) + + expectTypeOf(boundary).toEqualTypeOf() + }) + it(`should type findOne query builder to return a single row`, () => { const collection = createCollection( mockSyncCollectionOptions({ @@ -68,6 +98,247 @@ describe(`useLiveQuery type assertions`, () => { >() }) + it(`types a conditional findOne config object as disabled-capable`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `test-conditional-person-config`, + getKey: (person: Person) => person.id, + initialData: [], + }), + ) + const enabled = null as unknown as boolean + const query = new Query() + .from({ collection }) + .where(({ collection: c }) => eq(c.id, `3`)) + .findOne() + type QueryContext = + typeof query extends QueryBuilder ? TContext : never + const config: ConditionalUseLiveQueryConfig = { + queryKey: [collection.id, enabled], + query: () => (enabled ? query : undefined), + } + + const { result } = renderHook(() => { + return useLiveQuery(config) + }) + + expectTypeOf(result.current.data).toMatchTypeOf< + OutputWithVirtual | undefined + >() + expectTypeOf(result.current.status).toEqualTypeOf() + expectTypeOf(result.current.isEnabled).toEqualTypeOf() + }) + + it(`accepts an annotated enabled config in useLiveSuspenseQuery`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `test-annotated-suspense-config`, + getKey: (person: Person) => person.id, + initialData: [], + }), + ) + const query = new Query().from({ collection }) + type QueryContext = + typeof query extends QueryBuilder ? TContext : never + const config: UseLiveQueryConfig = { + query: () => query, + } + + const { result } = renderHook(() => useLiveSuspenseQuery(config)) + + expectTypeOf(result.current.data).toMatchTypeOf< + Array> + >() + }) + + it(`types a conditional config with deprecated dependencies`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `test-conditional-config-deps`, + getKey: (person: Person) => person.id, + initialData: [], + }), + ) + const enabled = null as unknown as boolean + + const { result } = renderHook(() => + useLiveQuery( + { + query: (q) => + enabled ? q.from({ collection }).findOne() : undefined, + }, + [enabled], + ), + ) + + expectTypeOf(result.current.data).toMatchTypeOf< + OutputWithVirtual | undefined + >() + expectTypeOf(result.current.status).toEqualTypeOf() + expectTypeOf(result.current.isEnabled).toEqualTypeOf() + }) + + it(`rejects a conditional config with a top-level scalar result`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `test-conditional-scalar-config`, + getKey: (person: Person) => person.id, + initialData: [], + }), + ) + const enabled = null as unknown as boolean + + useLiveQuery({ + // @ts-expect-error - top-level scalar results are not supported + query: (q) => { + if (!enabled) return undefined + return q.from({ collection }).select(({ collection: c }) => c.name) + }, + }) + }) + + it(`should type config object to return query rows without queryKey`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `test-persons-query-key`, + getKey: (person: Person) => person.id, + initialData: [], + }), + ) + + const { result } = renderHook(() => { + return useLiveQuery({ + query: (q) => + q + .from({ collection }) + .where(({ collection: c }) => eq(c.team, `team-1`)), + }) + }) + + expectTypeOf(result.current.data).toMatchTypeOf< + Array> + >() + }) + + it(`should type queryKey and a per-call DbClient override`, () => { + const descriptor = collectionOptions( + mockSyncCollectionOptions({ + id: `test-persons-client-override`, + getKey: (person: Person) => person.id, + initialData: [], + }), + ) + const client = null as unknown as DbClient + + const { result } = renderHook(() => { + return useLiveQuery({ + client, + queryKey: [descriptor.id, `team`, `team-1`], + query: (q) => + q + .from({ person: descriptor }) + .where(({ person }) => eq(person.team, `team-1`)), + }) + }) + + expectTypeOf(result.current.data).toMatchTypeOf< + Array> + >() + }) + + it(`keeps the deprecated dependency-array overload typed`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `test-persons-deprecated-deps`, + getKey: (person: Person) => person.id, + initialData: [], + }), + ) + + const { result } = renderHook(() => + useLiveQuery( + (q) => + q + .from({ person: collection }) + .where(({ person }) => eq(person.team, `team-1`)), + [`team-1`], + ), + ) + + expectTypeOf(result.current.data).toMatchTypeOf< + Array> + >() + }) + + it(`should type collection descriptors in query sources`, () => { + const collection = collectionOptions( + mockSyncCollectionOptions({ + id: `test-persons-descriptor-query-source`, + getKey: (person: Person) => person.id, + initialData: [], + }), + ) + + const { result } = renderHook(() => { + return useLiveQuery({ + query: (q) => + q + .from({ collection }) + .where(({ collection: c }) => eq(c.team, `team-1`)), + }) + }) + + expectTypeOf(result.current.data).toMatchTypeOf< + Array> + >() + }) + + it(`should type suspense config object to return query rows without queryKey`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `test-persons-suspense-query-key`, + getKey: (person: Person) => person.id, + initialData: [], + }), + ) + + const { result } = renderHook(() => { + return useLiveSuspenseQuery({ + query: (q) => + q + .from({ collection }) + .where(({ collection: c }) => eq(c.team, `team-1`)), + }) + }) + + expectTypeOf(result.current.data).toMatchTypeOf< + Array> + >() + }) + + it(`should type infinite config object to return query rows without queryKey`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `test-persons-infinite-query-key`, + getKey: (person: Person) => person.id, + initialData: [], + }), + ) + + const { result } = renderHook(() => { + return useLiveInfiniteQuery( + (q) => q.from({ collection }).orderBy(({ collection: c }) => c.name), + { + pageSize: 10, + }, + ) + }) + + expectTypeOf(result.current.data).toMatchTypeOf< + Array> + >() + }) + it(`should type findOne collection using liveQueryCollectionOptions to return a single row`, () => { const collection = createCollection( mockSyncCollectionOptions({ diff --git a/packages/react-db/tests/useLiveQuery.test.tsx b/packages/react-db/tests/useLiveQuery.test.tsx index fbb48d882c..a04c835a44 100644 --- a/packages/react-db/tests/useLiveQuery.test.tsx +++ b/packages/react-db/tests/useLiveQuery.test.tsx @@ -1,8 +1,10 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' import { act, renderHook, waitFor } from '@testing-library/react' import { + DbClient, Query, coalesce, + collectionOptions, count, createCollection, createLiveQueryCollection, @@ -11,13 +13,18 @@ import { gt, lte, sum, + toArray, } from '@tanstack/db' import { useEffect } from 'react' import { useLiveQuery } from '../src/useLiveQuery' +import { getLiveQueryResultInfo } from '../src/live-query-internals' +import { DbProvider } from '../src/DbProvider' import { mockSyncCollectionOptions, stripVirtualProps, } from '../../db/tests/utils' +import type { DehydratedDbState } from '@tanstack/db' +import type { ReactNode } from 'react' type Person = { id: string @@ -1975,32 +1982,191 @@ describe(`Query Collections`, () => { }) }) - describe(`callback variants with conditional returns`, () => { - it(`should handle callback returning undefined with proper state`, async () => { + describe(`conditional returns`, () => { + it(`disables a config query that returns undefined`, async () => { const collection = createCollection( mockSyncCollectionOptions({ - id: `undefined-callback-test`, + id: `undefined-config-query-test`, getKey: (person: Person) => person.id, initialData: initialPersons, }), ) const { result, rerender } = renderHook( - ({ enabled }: { enabled: boolean }) => { - return useLiveQuery( - (q) => { + ({ enabled }: { enabled: boolean }) => + useLiveQuery({ + query: (q) => { if (!enabled) return undefined return q .from({ persons: collection }) .where(({ persons }) => gt(persons.age, 30)) - .select(({ persons }) => ({ - id: persons.id, - name: persons.name, - age: persons.age, - })) + }, + }), + { initialProps: { enabled: false } }, + ) + + expect(result.current.data).toBeUndefined() + expect(result.current.collection).toBeUndefined() + expect(result.current.status).toBe(`disabled`) + expect(result.current.isEnabled).toBe(false) + expect(result.current.isReady).toBe(true) + + rerender({ enabled: true }) + + await waitFor(() => expect(result.current.data).toHaveLength(1)) + expect(result.current.status).toBe(`ready`) + expect(result.current.isEnabled).toBe(true) + + rerender({ enabled: false }) + + expect(result.current.data).toBeUndefined() + expect(result.current.collection).toBeUndefined() + expect(result.current.status).toBe(`disabled`) + expect(result.current.isEnabled).toBe(false) + expect(result.current.isReady).toBe(true) + }) + + it(`disables a config query that returns null`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `null-config-query-test`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const { result } = renderHook( + ({ enabled }: { enabled: boolean }) => + useLiveQuery({ + query: (q) => { + if (!enabled) return null + return q.from({ persons: collection }) + }, + }), + { initialProps: { enabled: false } }, + ) + + expect(result.current.data).toBeUndefined() + expect(result.current.collection).toBeUndefined() + expect(result.current.status).toBe(`disabled`) + expect(result.current.isEnabled).toBe(false) + }) + + it(`disables a config query with deprecated dependencies`, async () => { + const warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const collection = createCollection( + mockSyncCollectionOptions({ + id: `conditional-config-deps-test`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const { result, rerender } = renderHook( + ({ enabled }: { enabled: boolean }) => + useLiveQuery( + { + query: (q) => { + if (!enabled) return undefined + return q + .from({ persons: collection }) + .where(({ persons }) => gt(persons.age, 30)) + }, }, [enabled], - ) + ), + { initialProps: { enabled: false } }, + ) + + expect(result.current.status).toBe(`disabled`) + expect(result.current.isEnabled).toBe(false) + + rerender({ enabled: true }) + + await waitFor(() => expect(result.current.data).toHaveLength(1)) + expect(result.current.status).toBe(`ready`) + expect(result.current.isEnabled).toBe(true) + + rerender({ enabled: false }) + + expect(result.current.status).toBe(`disabled`) + expect(result.current.isEnabled).toBe(false) + warnSpy.mockRestore() + }) + + it(`stays disabled when the prior query becomes ready`, async () => { + let finishSync: (() => void) | undefined + const collection = createCollection({ + id: `conditional-config-pending-sync-test`, + getKey: (person) => person.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + finishSync = () => { + begin() + write({ type: `insert`, value: initialPersons[2]! }) + commit() + markReady() + } + }, + }, + onInsert: async () => {}, + onUpdate: async () => {}, + onDelete: async () => {}, + }) + + const { result, rerender } = renderHook( + ({ enabled }: { enabled: boolean }) => + useLiveQuery({ + query: (q) => { + if (!enabled) return undefined + return q + .from({ persons: collection }) + .where(({ persons }) => gt(persons.age, 30)) + }, + }), + { initialProps: { enabled: true } }, + ) + + await waitFor(() => expect(finishSync).toBeDefined()) + expect(result.current.isLoading).toBe(true) + + rerender({ enabled: false }) + expect(result.current.status).toBe(`disabled`) + + await act(async () => { + finishSync!() + await Promise.resolve() + }) + + expect(collection.status).toBe(`ready`) + expect(collection.state.size).toBe(1) + expect(result.current.status).toBe(`disabled`) + expect(result.current.data).toBeUndefined() + expect(result.current.collection).toBeUndefined() + }) + + it(`should handle callback returning undefined without a dependency array`, async () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `undefined-callback-test`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const { result, rerender } = renderHook( + ({ enabled }: { enabled: boolean }) => { + return useLiveQuery((q) => { + if (!enabled) return undefined + return q + .from({ persons: collection }) + .where(({ persons }) => gt(persons.age, 30)) + .select(({ persons }) => ({ + id: persons.id, + name: persons.name, + age: persons.age, + })) + }) }, { initialProps: { enabled: false } }, ) @@ -2056,7 +2222,7 @@ describe(`Query Collections`, () => { expect(result.current.isCleanedUp).toBe(false) }) - it(`should handle callback returning null with proper state`, async () => { + it(`should handle callback returning null without a dependency array`, async () => { const collection = createCollection( mockSyncCollectionOptions({ id: `null-callback-test`, @@ -2067,20 +2233,17 @@ describe(`Query Collections`, () => { const { result, rerender } = renderHook( ({ enabled }: { enabled: boolean }) => { - return useLiveQuery( - (q) => { - if (!enabled) return null - return q - .from({ persons: collection }) - .where(({ persons }) => gt(persons.age, 30)) - .select(({ persons }) => ({ - id: persons.id, - name: persons.name, - age: persons.age, - })) - }, - [enabled], - ) + return useLiveQuery((q) => { + if (!enabled) return null + return q + .from({ persons: collection }) + .where(({ persons }) => gt(persons.age, 30)) + .select(({ persons }) => ({ + id: persons.id, + name: persons.name, + age: persons.age, + })) + }) }, { initialProps: { enabled: false } }, ) @@ -2520,7 +2683,7 @@ describe(`Query Collections`, () => { { id: `i3`, title: `Bug in Beta`, projectId: `p2` }, ] - it(`should render includes results and reactively update child collections`, async () => { + it(`renders only the affected child hook once when an include changes`, async () => { const projectsCollection = createCollection( mockSyncCollectionOptions({ id: `includes-react-projects`, @@ -2537,9 +2700,14 @@ describe(`Query Collections`, () => { }), ) + let parentRenderCount = 0 + let alphaRenderCount = 0 + let betaRenderCount = 0 + // Parent hook: runs includes query that produces child Collections - const { result: parentResult } = renderHook(() => - useLiveQuery((q) => + const { result: parentResult } = renderHook(() => { + parentRenderCount += 1 + return useLiveQuery((q) => q.from({ p: projectsCollection }).select(({ p }) => ({ id: p.id, name: p.name, @@ -2551,8 +2719,8 @@ describe(`Query Collections`, () => { title: i.title, })), })), - ), - ) + ) + }) // Wait for parent to be ready await waitFor(() => { @@ -2562,24 +2730,38 @@ describe(`Query Collections`, () => { const alphaProject = parentResult.current.data.find( (p: any) => p.id === `p1`, )! + const betaProject = parentResult.current.data.find( + (p: any) => p.id === `p2`, + )! expect(alphaProject.name).toBe(`Alpha`) - // Child hook: subscribes to the child Collection from the parent row, - // simulating a subcomponent using useLiveQuery(project.issues) - const { result: childResult } = renderHook(() => - useLiveQuery((alphaProject as any).issues), - ) + // Child hooks simulate sibling subcomponents subscribing to the child + // Collections from their parent rows. + const { result: alphaResult } = renderHook(() => { + alphaRenderCount += 1 + return useLiveQuery((alphaProject as any).issues) + }) + const { result: betaResult } = renderHook(() => { + betaRenderCount += 1 + return useLiveQuery((betaProject as any).issues) + }) await waitFor(() => { - expect(childResult.current.data).toHaveLength(2) + expect(alphaResult.current.data).toHaveLength(2) + expect(alphaResult.current.isReady).toBe(true) + expect(betaResult.current.data).toHaveLength(1) + expect(betaResult.current.isReady).toBe(true) }) - expect(childResult.current.data).toEqual( + expect(alphaResult.current.data).toEqual( expect.arrayContaining([ expect.objectContaining({ id: `i1`, title: `Bug in Alpha` }), expect.objectContaining({ id: `i2`, title: `Feature for Alpha` }), ]), ) + const settledParentRenders = parentRenderCount + const settledAlphaRenders = alphaRenderCount + const settledBetaRenders = betaRenderCount // Add a new issue to Alpha — the child hook should reactively update act(() => { @@ -2592,16 +2774,820 @@ describe(`Query Collections`, () => { }) await waitFor(() => { - expect(childResult.current.data).toHaveLength(3) + expect(alphaResult.current.data).toHaveLength(3) }) - expect(childResult.current.data).toEqual( + expect(alphaResult.current.data).toEqual( expect.arrayContaining([ expect.objectContaining({ id: `i1`, title: `Bug in Alpha` }), expect.objectContaining({ id: `i2`, title: `Feature for Alpha` }), expect.objectContaining({ id: `i4`, title: `New Alpha issue` }), ]), ) + expect(parentRenderCount).toBe(settledParentRenders) + expect(alphaRenderCount).toBe(settledAlphaRenders + 1) + expect(betaRenderCount).toBe(settledBetaRenders) + }) + + it(`keeps nested array includes on the render after a parent update`, async () => { + type Document = { + id: string + name: string + schemaId: string + } + type Schema = { + id: string + name: string + } + type Field = { + id: string + schemaId: string + name: string + } + + const documents = createCollection( + mockSyncCollectionOptions({ + id: `includes-react-documents`, + getKey: (document) => document.id, + initialData: [{ id: `d1`, name: `Before`, schemaId: `s1` }], + }), + ) + const schemas = createCollection( + mockSyncCollectionOptions({ + id: `includes-react-schemas`, + getKey: (schema) => schema.id, + initialData: [{ id: `s1`, name: `Schema` }], + }), + ) + const fields = createCollection( + mockSyncCollectionOptions({ + id: `includes-react-fields`, + getKey: (field) => field.id, + initialData: [{ id: `f1`, schemaId: `s1`, name: `Title` }], + }), + ) + + const { result } = renderHook(() => + useLiveQuery((q) => + q.from({ document: documents }).select(({ document }) => ({ + id: document.id, + name: document.name, + schema: toArray( + q + .from({ schema: schemas }) + .where(({ schema }) => eq(schema.id, document.schemaId)) + .select(({ schema }) => ({ + id: schema.id, + fields: toArray( + q + .from({ field: fields }) + .where(({ field }) => eq(field.schemaId, schema.id)) + .select(({ field }) => ({ + id: field.id, + name: field.name, + })), + ), + })), + ), + })), + ), + ) + + await waitFor(() => { + expect(result.current.data[0]).toMatchObject({ + name: `Before`, + schema: [{ id: `s1`, fields: [{ id: `f1`, name: `Title` }] }], + }) + }) + + act(() => { + documents.utils.begin() + documents.utils.write({ + type: `update`, + value: { id: `d1`, name: `After`, schemaId: `s1` }, + }) + documents.utils.commit() + }) + + await waitFor(() => { + expect(result.current.data[0]).toMatchObject({ + name: `After`, + schema: [{ id: `s1`, fields: [{ id: `f1`, name: `Title` }] }], + }) + }) + }) + }) + + describe(`SSR hydration`, () => { + it(`round-trips collection rows into React and applies streamed chunks incrementally`, async () => { + const peopleCollectionId = `ssr-react-people` + const peopleCollection = collectionOptions(peopleCollectionId, () => ({ + id: peopleCollectionId, + getKey: (person: Person) => person.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + + return { + loadSubset: () => { + begin({ immediate: true }) + for (const person of initialPersons) { + write({ + type: `insert`, + value: person, + }) + } + commit() + return true + }, + } + }, + }, + })) + const serverClient = new DbClient() + const serverPeople = serverClient.collection(peopleCollection) + const serverLiveQuery = createLiveQueryCollection((q) => + q + .from({ people: serverPeople }) + .where(({ people }) => eq(people.team, `team1`)), + ) + + await serverLiveQuery.preload() + + expect(serverLiveQuery.toArray.map((person) => person.id)).toEqual([ + `1`, + `3`, + ]) + const dehydratedState = serverClient.dehydrate() + expect( + dehydratedState.collections + .flatMap((collection) => collection.rows.map((row) => row.key)) + .sort(), + ).toEqual([`1`, `2`, `3`]) + + const transferredState = JSON.parse( + JSON.stringify(dehydratedState), + ) as DehydratedDbState + const clientClient = new DbClient() + clientClient.hydrate(transferredState) + const wrapper = ({ children }: { children: ReactNode }) => ( + {children} + ) + + const { result } = renderHook( + () => + useLiveQuery({ + query: (q) => + q + .from({ people: peopleCollection }) + .where(({ people }) => eq(people.team, `team1`)), + }), + { wrapper }, + ) + + const resultIds = () => result.current.data.map((person) => person.id) + + await waitFor(() => { + expect(resultIds()).toEqual([`1`, `3`]) + }) + const hydratedLiveQuery = result.current.collection + + act(() => { + clientClient.applyCollectionChunk({ + collectionId: peopleCollectionId, + rows: [ + { + key: `4`, + value: { + id: `4`, + name: `Kyle Doe`, + age: 40, + email: `kyle.doe@example.com`, + isActive: true, + team: `team1`, + }, + }, + ], + }) + }) + + await waitFor(() => { + expect(resultIds()).toEqual([`1`, `3`, `4`]) + }) + expect(result.current.collection).toBe(hydratedLiveQuery) + }) + }) + + describe(`derived query identity`, () => { + it(`resolves collection descriptors from DbProvider`, async () => { + const dbClient = new DbClient() + const peopleCollection = collectionOptions( + mockSyncCollectionOptions({ + id: `descriptor-people`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + const wrapper = ({ children }: { children: ReactNode }) => ( + {children} + ) + + const { result } = renderHook( + () => + useLiveQuery({ + query: (q) => + q + .from({ people: peopleCollection }) + .where(({ people }) => eq(people.team, `team1`)), + }), + { wrapper }, + ) + + await waitFor(() => { + expect(result.current.data).toHaveLength(2) + }) + + const people = dbClient.collection(peopleCollection) + + act(() => { + people.insert({ + id: `4`, + name: `Kyle Doe`, + age: 40, + email: `kyle.doe@example.com`, + isActive: true, + team: `team1`, + }) + }) + + await waitFor(() => { + expect(result.current.data).toHaveLength(3) + }) + }) + + it(`reuses dynamically-created collection descriptors by id`, async () => { + const dbClient = new DbClient() + const materialize = vi.fn(() => + mockSyncCollectionOptions({ + id: `dynamic-descriptor-people`, + getKey: (person) => person.id, + initialData: initialPersons, + }), + ) + const descriptors = new Set() + const wrapper = ({ children }: { children: ReactNode }) => ( + {children} + ) + + const { result, rerender } = renderHook( + ({ team }) => + useLiveQuery({ + query: (q) => { + const descriptor = collectionOptions( + `dynamic-descriptor-people`, + materialize, + ) + descriptors.add(descriptor) + + return q + .from({ people: descriptor }) + .where(({ people }) => eq(people.team, team)) + }, + }), + { initialProps: { team: `team1` }, wrapper }, + ) + + await waitFor(() => { + expect(result.current.data).toHaveLength(2) + }) + const firstCollection = result.current.collection + + rerender({ team: `team1` }) + + expect(descriptors.size).toBeGreaterThan(1) + expect(materialize).toHaveBeenCalledOnce() + expect(result.current.collection).toBe(firstCollection) + }) + + it(`keeps the same live query collection when derived identity is stable`, async () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `query-key-stable`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const { result, rerender } = renderHook( + ({ minAge }) => + useLiveQuery({ + query: (q) => + q + .from({ people: collection }) + .where(({ people }) => gt(people.age, minAge)), + }), + { initialProps: { minAge: 30 } }, + ) + + await waitFor(() => { + expect(result.current.data).toHaveLength(1) + }) + const firstCollection = result.current.collection + + rerender({ minAge: 30 }) + + expect(result.current.collection).toBe(firstCollection) + }) + + it(`evaluates a derived query once per render`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `derived-identity-single-evaluation`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + let queryExecutions = 0 + + const { result, rerender } = renderHook( + ({ minAge }) => + useLiveQuery({ + query: (q) => { + queryExecutions += 1 + return q + .from({ people: collection }) + .where(({ people }) => gt(people.age, minAge)) + }, + }), + { initialProps: { minAge: 25 } }, + ) + + expect(queryExecutions).toBe(1) + const firstCollection = result.current.collection + + rerender({ minAge: 25 }) + + expect(queryExecutions).toBe(2) + expect(result.current.collection).toBe(firstCollection) + + rerender({ minAge: 30 }) + + expect(queryExecutions).toBe(3) + expect(result.current.collection).not.toBe(firstCollection) + }) + + it(`rebinds descriptors when the DbProvider client changes`, async () => { + const peopleCollection = collectionOptions( + `provider-swap-people`, + (client) => + mockSyncCollectionOptions({ + id: `provider-swap-people`, + getKey: (person) => person.id, + initialData: client.requireDependency>(`people`), + }), + ) + const clientA = new DbClient({ + people: [{ ...initialPersons[0]!, name: `Client A` }], + }) + const clientB = new DbClient({ + people: [{ ...initialPersons[0]!, name: `Client B` }], + }) + let currentClient = clientA + const wrapper = ({ children }: { children: ReactNode }) => ( + {children} + ) + const { result, rerender } = renderHook( + () => + useLiveQuery({ + query: (q) => q.from({ people: peopleCollection }), + }), + { wrapper }, + ) + + await waitFor(() => { + expect(result.current.data[0]?.name).toBe(`Client A`) + }) + const firstCollection = result.current.collection + + currentClient = clientB + rerender() + + await waitFor(() => { + expect(result.current.data[0]?.name).toBe(`Client B`) + }) + expect(result.current.collection).not.toBe(firstCollection) + }) + + it(`recreates the live query collection when derived identity changes`, async () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `query-key-change`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const { result, rerender } = renderHook( + ({ minAge }) => + useLiveQuery({ + query: (q) => + q + .from({ people: collection }) + .where(({ people }) => gt(people.age, minAge)), + }), + { initialProps: { minAge: 25 } }, + ) + + await waitFor(() => { + expect(result.current.data).toHaveLength(2) + }) + const firstCollection = result.current.collection + + rerender({ minAge: 30 }) + + await waitFor(() => { + expect(result.current.data).toHaveLength(1) + }) + expect(result.current.collection).not.toBe(firstCollection) + }) + + it(`warns and preserves legacy behavior when a functional query has no queryKey`, () => { + const warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const collection = createCollection( + mockSyncCollectionOptions({ + id: `derived-identity-functional-missing-key`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + expect(() => + renderHook( + ({ minAge }) => + useLiveQuery({ + query: (q) => + q + .from({ people: collection }) + .fn.where(({ people }) => people.age > minAge), + }), + { initialProps: { minAge: 25 } }, + ), + ).not.toThrow() + + const warnings = warnSpy.mock.calls.filter(([message]) => + String(message).includes(`cannot derive a stable identity`), + ) + expect(warnings).toHaveLength(1) + expect(warnings[0]![0]).toContain(`queryKey`) + expect(warnings[0]![0]).toContain(`1.0`) + warnSpy.mockRestore() + }) + + it(`uses runtime identity for opaque values in a structured query without queryKey`, () => { + const warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const collection = createCollection( + mockSyncCollectionOptions({ + id: `derived-identity-opaque-value`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const runtimeValue = () => `John Doe` + const { result, rerender } = renderHook( + ({ value }) => + useLiveQuery({ + query: (q) => + q + .from({ people: collection }) + .where(({ people }) => eq(people.name, value as never)), + }), + { initialProps: { value: runtimeValue } }, + ) + const firstCollection = result.current.collection + rerender({ value: runtimeValue }) + expect(result.current.collection).toBe(firstCollection) + rerender({ value: () => `John Doe` }) + expect(result.current.collection).not.toBe(firstCollection) + + const warnings = warnSpy.mock.calls.filter(([message]) => + String(message).includes(`function value`), + ) + expect(warnings).toHaveLength(0) + warnSpy.mockRestore() + }) + + it(`does not emit identity warnings in production`, () => { + const previousNodeEnv = process.env.NODE_ENV + process.env.NODE_ENV = `production` + const warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const collection = createCollection( + mockSyncCollectionOptions({ + id: `derived-identity-production-warning`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + let unmount: (() => void) | undefined + try { + ;({ unmount } = renderHook(() => + useLiveQuery({ + query: (q) => + q + .from({ people: collection }) + .fn.where(({ people }) => people.age > 25), + }), + )) + + expect( + warnSpy.mock.calls.some(([message]) => + String(message).includes(`cannot derive a stable identity`), + ), + ).toBe(false) + } finally { + unmount?.() + process.env.NODE_ENV = previousNodeEnv + warnSpy.mockRestore() + } + }) + + it(`uses explicit queryKey for functional query variants`, async () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `derived-identity-functional-explicit-key`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const { result, rerender } = renderHook( + ({ minAge }) => + useLiveQuery({ + queryKey: [collection.id, `fn`, minAge], + query: (q) => + q + .from({ people: collection }) + .fn.where(({ people }) => people.age > minAge), + }), + { initialProps: { minAge: 25 } }, + ) + + await waitFor(() => { + expect(result.current.data).toHaveLength(2) + }) + const firstCollection = result.current.collection + + rerender({ minAge: 30 }) + + await waitFor(() => { + expect(result.current.data).toHaveLength(1) + }) + expect(result.current.collection).not.toBe(firstCollection) + }) + + it(`throws when an explicit queryKey cannot be stably hashed`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `unhashable-explicit-query-key`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + expect(() => + renderHook(() => + useLiveQuery({ + queryKey: [collection.id, () => `opaque`], + query: (q) => q.from({ people: collection }), + }), + ), + ).toThrow(/queryKey.*function value/) + }) + + it(`keeps an explicit queryKey stable across structurally equal values`, async () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `explicit-query-key-structural-equality`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + let queryExecutions = 0 + + const { result, rerender } = renderHook( + ({ filter }: { filter: { minAge: number } }) => + useLiveQuery({ + queryKey: [collection.id, `minimum-age`, filter], + query: (q) => { + queryExecutions += 1 + return q + .from({ people: collection }) + .where(({ people }) => gt(people.age, filter.minAge)) + }, + }), + { initialProps: { filter: { minAge: 25 } } }, + ) + + await waitFor(() => expect(result.current.data).toHaveLength(2)) + const firstCollection = result.current.collection + + rerender({ filter: { minAge: 25 } }) + + expect(result.current.collection).toBe(firstCollection) + expect(queryExecutions).toBe(1) + }) + + it(`preserves reference semantics for deprecated dependency arrays`, async () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `legacy-deps-reference-semantics`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const { result, rerender } = renderHook( + ({ filter }: { filter: { minAge: number } }) => + useLiveQuery( + (q) => + q + .from({ people: collection }) + .where(({ people }) => gt(people.age, filter.minAge)), + [filter], + ), + { initialProps: { filter: { minAge: 25 } } }, + ) + + await waitFor(() => expect(result.current.data).toHaveLength(2)) + const firstCollection = result.current.collection + + rerender({ filter: { minAge: 25 } }) + + expect(result.current.collection).not.toBe(firstCollection) + }) + + it(`warns when derived query identity is slow enough to need queryKey`, () => { + const warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const nowSpy = vi.spyOn(globalThis.performance, `now`) + let currentTime = 0 + nowSpy.mockImplementation(() => { + const value = currentTime + currentTime += 20 + return value + }) + + const collection = createCollection( + mockSyncCollectionOptions({ + id: `derived-identity-slow-warning`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const { rerender } = renderHook(() => + useLiveQuery({ + query: (q) => + q + .from({ people: collection }) + .where(({ people }) => gt(people.age, 25)), + }), + ) + + rerender() + + const hotPathWarnings = warnSpy.mock.calls.filter(([message]) => + String(message).includes(`hot render path`), + ) + expect(hotPathWarnings).toHaveLength(1) + expect(hotPathWarnings[0]![0]).toContain(`queryKey`) + + nowSpy.mockRestore() + warnSpy.mockRestore() + }) + + it(`warns when repeated derived query identity work accumulates on a hot render path`, () => { + const warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const nowSpy = vi.spyOn(globalThis.performance, `now`) + let currentTime = 0 + nowSpy.mockImplementation(() => { + const value = currentTime + currentTime += 6 + return value + }) + + const collection = createCollection( + mockSyncCollectionOptions({ + id: `derived-identity-accumulated-warning`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const { rerender } = renderHook( + ({ renderCount }) => { + void renderCount + return useLiveQuery({ + query: (q) => + q + .from({ people: collection }) + .where(({ people }) => gt(people.age, 25)), + }) + }, + { initialProps: { renderCount: 0 } }, + ) + + for (let renderCount = 1; renderCount < 10; renderCount++) { + rerender({ renderCount }) + } + + const hotPathWarnings = warnSpy.mock.calls.filter(([message]) => + String(message).includes(`renders took`), + ) + expect(hotPathWarnings).toHaveLength(1) + expect(hotPathWarnings[0]![0]).toContain(`queryKey`) + + nowSpy.mockRestore() + warnSpy.mockRestore() + }) + + it(`warns once for the deprecated dependency-array form`, () => { + const warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const collection = createCollection( + mockSyncCollectionOptions({ + id: `deps-warning`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const { rerender } = renderHook( + ({ minAge }) => + useLiveQuery( + (q) => + q + .from({ people: collection }) + .where(({ people }) => gt(people.age, minAge)), + [minAge], + ), + { initialProps: { minAge: 25 } }, + ) + + rerender({ minAge: 30 }) + rerender({ minAge: 30 }) + + expect(warnSpy).toHaveBeenCalledTimes(1) + expect(warnSpy).toHaveBeenCalledWith( + expect.stringContaining(`will be removed in 1.0`), + ) + + warnSpy.mockRestore() + }) + + it(`warns for an explicitly passed empty dependency array`, () => { + const warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const collection = createCollection( + mockSyncCollectionOptions({ + id: `empty-deps-warning`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + renderHook(() => useLiveQuery((q) => q.from({ people: collection }), [])) + + expect(warnSpy).toHaveBeenCalledWith( + expect.stringContaining(`useLiveQuery({ query })`), + ) + + warnSpy.mockRestore() + }) + + it(`includes the query in legacy dependency-array SSR identity`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `legacy-deps-query-identity`, + getKey: (person) => person.id, + initialData: initialPersons, + }), + ) + const first = renderHook(() => + useLiveQuery((q) => q.from({ people: collection }), [1]), + ) + const second = renderHook(() => + useLiveQuery( + (q) => + q + .from({ people: collection }) + .where(({ people }) => gt(people.age, 30)), + [1], + ), + ) + + expect(getLiveQueryResultInfo(first.result.current).queryHash).not.toBe( + getLiveQueryResultInfo(second.result.current).queryHash, + ) }) }) }) diff --git a/packages/react-db/tests/useLiveQuery.uncommitted-render.test.tsx b/packages/react-db/tests/useLiveQuery.uncommitted-render.test.tsx new file mode 100644 index 0000000000..7b1126cb2a --- /dev/null +++ b/packages/react-db/tests/useLiveQuery.uncommitted-render.test.tsx @@ -0,0 +1,192 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { Component, Suspense } from 'react' +import { act, cleanup, render } from '@testing-library/react' +import { createCollection, createLiveQueryCollection } from '@tanstack/db' +import { useLiveQuery } from '../src/useLiveQuery' +import { useLiveSuspenseQuery } from '../src/useLiveSuspenseQuery' +import { + mockSyncCollectionOptions, + resetCleanupQueue, +} from '../../db/tests/utils' +import type { ReactNode } from 'react' + +type Person = { id: string; name: string } + +const collections: Array<{ cleanup: () => Promise }> = [] + +function makeSource(id: string) { + const source = createCollection( + mockSyncCollectionOptions({ + id, + getKey: (p) => p.id, + initialData: [{ id: `1`, name: `A` }], + }), + ) + collections.push(source) + return source +} + +async function advanceTime(ms: number) { + await act(async () => { + await vi.advanceTimersByTimeAsync(ms) + }) +} + +class Boundary extends Component<{ children: ReactNode }, { failed: boolean }> { + state = { failed: false } + static getDerivedStateFromError() { + return { failed: true } + } + render() { + return this.state.failed ?
                            Failed
                            : this.props.children + } +} + +describe(`live queries across uncommitted renders`, () => { + beforeEach(() => { + resetCleanupQueue() + vi.useFakeTimers() + }) + + afterEach(async () => { + cleanup() + await advanceTime(100) + for (const collection of collections.splice(0).reverse()) { + await collection.cleanup() + } + resetCleanupQueue() + vi.restoreAllMocks() + vi.useRealTimers() + }) + + it(`releases the source when the subtree suspends after the hook ran`, async () => { + const source = makeSource(`uncommitted-suspend`) + const neverResolves = new Promise(() => {}) + + const Suspender = () => { + throw neverResolves + } + + const Route = () => { + useLiveQuery((q) => + q.from({ p: source }).select(({ p }) => ({ id: p.id, name: p.name })), + ) + return + } + + render( + Loading}> + + , + ) + expect(source.subscriberCount).toBeGreaterThan(0) + + await advanceTime(100) + + expect(source.subscriberCount).toBe(0) + }) + + it(`releases the source when the render throws after the hook ran`, async () => { + const source = makeSource(`uncommitted-throw`) + const renderError = new Error(`render discarded`) + const logError = console.error.bind(console) + vi.spyOn(console, `error`).mockImplementation((...args: Array) => { + if (args.includes(renderError)) return + logError(...args) + }) + + const Thrower = () => { + useLiveQuery((q) => + q.from({ p: source }).select(({ p }) => ({ id: p.id, name: p.name })), + ) + throw renderError + } + + const view = render( + + + , + ) + expect(view.getByText(`Failed`)).toBeDefined() + expect(source.subscriberCount).toBeGreaterThan(0) + + await advanceTime(100) + + expect(source.subscriberCount).toBe(0) + }) + + it(`keeps a committed query active until unmount`, async () => { + const source = makeSource(`uncommitted-control`) + + const Ok = () => { + const { data } = useLiveQuery((q) => + q.from({ p: source }).select(({ p }) => ({ id: p.id, name: p.name })), + ) + return
                            {data.length}
                            + } + + const { unmount } = render() + expect(source.subscriberCount).toBeGreaterThan(0) + await advanceTime(100) + expect(source.subscriberCount).toBeGreaterThan(0) + + unmount() + await advanceTime(2) + + expect(source.subscriberCount).toBe(0) + }) + + it(`keeps a suspense preload alive until slow source data arrives`, async () => { + let completeLoad = () => {} + const source = createCollection({ + getKey: (person) => person.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + completeLoad = () => { + begin() + write({ type: `insert`, value: { id: `1`, name: `Alice` } }) + commit() + markReady() + } + }, + }, + }) + const live = createLiveQueryCollection({ + gcTime: 1, + query: (q) => q.from({ person: source }).select(({ person }) => person), + }) + collections.push(source, live) + const reclaimed = vi.fn() + live.on(`status:cleaned-up`, reclaimed) + + const People = () => { + const { data } = useLiveSuspenseQuery(live) + return
                            {data.map((person) => person.name).join(`, `)}
                            + } + + const view = render( + Loading}> + + , + ) + expect(view.getByText(`Loading`)).toBeDefined() + expect(source.subscriberCount).toBeGreaterThan(0) + + await advanceTime(150) + + // React can retry an aborted preload and return to loading, hiding an + // intervening cleanup if we only check the current status. + expect(reclaimed).not.toHaveBeenCalled() + expect(live.status).toBe(`loading`) + expect(source.subscriberCount).toBeGreaterThan(0) + await act(async () => { + completeLoad() + await Promise.resolve() + }) + expect(view.getByText(`Alice`)).toBeDefined() + + view.unmount() + await advanceTime(2) + expect(source.subscriberCount).toBe(0) + }) +}) diff --git a/packages/react-db/tests/useLiveSuspenseQuery.test.tsx b/packages/react-db/tests/useLiveSuspenseQuery.test.tsx index fc78b07968..ef2c3fe49b 100644 --- a/packages/react-db/tests/useLiveSuspenseQuery.test.tsx +++ b/packages/react-db/tests/useLiveSuspenseQuery.test.tsx @@ -1,13 +1,17 @@ -import { describe, expect, it } from 'vitest' -import { renderHook, waitFor } from '@testing-library/react' +import { describe, expect, it, vi } from 'vitest' +import { act, renderHook, waitFor } from '@testing-library/react' import { + DbClient, + collectionOptions, createCollection, createLiveQueryCollection, eq, + getStableValueHash, gt, } from '@tanstack/db' import { StrictMode, Suspense } from 'react' import { useLiveSuspenseQuery } from '../src/useLiveSuspenseQuery' +import { DbProvider } from '../src/DbProvider' import { mockSyncCollectionOptions } from '../../db/tests/utils' import type { ReactNode } from 'react' @@ -53,6 +57,195 @@ function SuspenseWrapper({ children }: { children: ReactNode }) { } describe(`useLiveSuspenseQuery`, () => { + it(`renders a streamed query snapshot until browser sync is authoritative`, async () => { + let resolveServerLoad!: () => void + const serverLoad = new Promise((resolve) => { + resolveServerLoad = resolve + }) + let resolveBrowserLoad!: () => void + let finishBrowserLoad!: () => void + const browserLoadPromise = new Promise((resolve) => { + finishBrowserLoad = resolve + }) + const browserLoad = vi.fn() + const descriptor = collectionOptions(`streamed-people`, (client) => { + const runtime = client.requireDependency<`server` | `browser`>(`runtime`) + + return { + id: `streamed-people`, + getKey: (person: Person) => person.id, + syncMode: `on-demand` as const, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: + runtime === `server` + ? async () => { + await serverLoad + begin({ immediate: true }) + write({ type: `insert`, value: initialPersons[0]! }) + commit() + } + : () => { + browserLoad() + resolveBrowserLoad = () => { + begin({ immediate: true }) + write({ + type: `insert`, + value: initialPersons[1]!, + }) + commit() + finishBrowserLoad() + } + return browserLoadPromise + }, + } + }, + }, + } + }) + const serverClient = new DbClient({ runtime: `server` }) + serverClient._setSsrStreamingEnabled(true) + const serverWrapper = ({ children }: { children: ReactNode }) => ( + + Loading...}>{children} + + ) + const serverHook = renderHook( + () => + useLiveSuspenseQuery({ + query: (q) => q.from({ people: descriptor }), + }), + { wrapper: serverWrapper }, + ) + + const dehydrated = serverClient.dehydrate({ + shouldDehydrateCollection: () => false, + shouldDehydrateLiveQuery: () => true, + }) + const dehydratedQuery = dehydrated.liveQueries?.[0] + expect(dehydratedQuery).toBeDefined() + + const browserClient = new DbClient({ runtime: `browser` }) + browserClient._setSsrStreamingEnabled(true) + browserClient.hydrate(dehydrated) + const browserWrapper = ({ children }: { children: ReactNode }) => ( + + Loading...}>{children} + + ) + const browserHook = renderHook( + () => + useLiveSuspenseQuery({ + query: (q) => q.from({ people: descriptor }), + }), + { wrapper: browserWrapper }, + ) + + expect(browserLoad).not.toHaveBeenCalled() + + await act(async () => { + resolveServerLoad() + await browserClient._getLiveQuery(dehydratedQuery!.queryHash)?.promise + }) + + await waitFor(() => { + expect(browserHook.result.current.data).toHaveLength(1) + expect(browserHook.result.current.data[0]).toMatchObject( + initialPersons[0]!, + ) + }) + expect(browserLoad).toHaveBeenCalled() + + await act(async () => resolveBrowserLoad()) + + await waitFor(() => { + expect(browserHook.result.current.data).toHaveLength(1) + expect(browserHook.result.current.data[0]).toMatchObject( + initialPersons[1]!, + ) + }) + serverHook.unmount() + browserHook.unmount() + }) + + it(`requires queryKey for an opaque query during SSR streaming`, () => { + const warn = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const descriptor = collectionOptions(`streamed-people`, () => ({ + id: `streamed-people`, + getKey: (person: Person) => person.id, + syncMode: `on-demand` as const, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset: () => new Promise(() => {}) } + }, + }, + })) + const client = new DbClient() + client._setSsrStreamingEnabled(true) + const wrapper = ({ children }: { children: ReactNode }) => ( + {children} + ) + + expect(() => + renderHook( + () => + useLiveSuspenseQuery({ + query: (q) => + q + .from({ people: descriptor }) + .fn.where(({ people }) => people.age > 20), + }), + { wrapper }, + ), + ).toThrow(/Provide an explicit serializable queryKey/) + expect(warn).toHaveBeenCalledWith( + expect.stringContaining(`cannot derive a stable identity`), + ) + warn.mockRestore() + }) + + it(`throws the original streamed query error`, async () => { + const error = new Error(`Server load failed`) + const queryKey = [`streamed-people-error`] as const + const queryHash = getStableValueHash([`queryKey`, queryKey], `queryKey`) + const descriptor = collectionOptions(`streamed-people-error`, () => ({ + id: `streamed-people-error`, + getKey: (person: Person) => person.id, + syncMode: `on-demand` as const, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset: () => new Promise(() => {}) } + }, + }, + })) + const client = new DbClient() + client._setSsrStreamingEnabled(true) + await expect( + client._registerLiveQuery(queryHash, Promise.reject(error)), + ).rejects.toBe(error) + const wrapper = ({ children }: { children: ReactNode }) => ( + {children} + ) + const consoleError = vi.spyOn(console, `error`).mockImplementation(() => {}) + + expect(() => + renderHook( + () => + useLiveSuspenseQuery({ + queryKey, + query: (q) => q.from({ people: descriptor }), + }), + { wrapper }, + ), + ).toThrow(error) + + consoleError.mockRestore() + }) + it(`should suspend while loading and return data when ready`, async () => { const collection = createCollection( mockSyncCollectionOptions({ @@ -185,7 +378,7 @@ describe(`useLiveSuspenseQuery`, () => { }) }) - it(`should re-suspend when deps change`, async () => { + it(`should re-suspend when derived query identity changes`, async () => { const collection = createCollection( mockSyncCollectionOptions({ id: `test-persons-suspense-5`, @@ -196,13 +389,12 @@ describe(`useLiveSuspenseQuery`, () => { const { result, rerender } = renderHook( ({ minAge }) => { - return useLiveSuspenseQuery( - (q) => + return useLiveSuspenseQuery({ + query: (q) => q .from({ persons: collection }) .where(({ persons }) => gt(persons.age, minAge)), - [minAge], - ) + }) }, { wrapper: SuspenseWrapper, @@ -216,7 +408,7 @@ describe(`useLiveSuspenseQuery`, () => { }) expect(result.current.data[0]?.age).toBe(35) - // Change deps - age > 20 + // Change derived identity - age > 20 rerender({ minAge: 20 }) // Should re-suspend and load new data diff --git a/packages/react-native-db-sqlite-persistence/CHANGELOG.md b/packages/react-native-db-sqlite-persistence/CHANGELOG.md index 4f72cb892d..9756571ab3 100644 --- a/packages/react-native-db-sqlite-persistence/CHANGELOG.md +++ b/packages/react-native-db-sqlite-persistence/CHANGELOG.md @@ -1,5 +1,152 @@ # @tanstack/react-native-db-sqlite-persistence +## 0.2.21 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db-sqlite-persistence-core@0.2.21 + +## 0.2.20 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.20 + +## 0.2.19 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.19 + +## 0.2.18 + +### Patch Changes + +- Updated dependencies [[`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db-sqlite-persistence-core@0.2.18 + +## 0.2.17 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.17 + +## 0.2.16 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.16 + +## 0.2.15 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.15 + +## 0.2.14 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.14 + +## 0.2.13 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db-sqlite-persistence-core@0.2.13 + +## 0.2.12 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.12 + +## 0.2.11 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.11 + +## 0.2.10 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.10 + +## 0.2.9 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.9 + +## 0.2.8 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.8 + +## 0.2.7 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.7 + +## 0.2.6 + +### Patch Changes + +- Updated dependencies [[`f7da776`](https://github.com/TanStack/db/commit/f7da77660b16cbfe30817fb5c938267d696c8d1c)]: + - @tanstack/db-sqlite-persistence-core@0.2.6 + +## 0.2.5 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.5 + +## 0.2.4 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.4 + +## 0.2.3 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.3 + +## 0.2.2 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.2 + +## 0.2.1 + +### Patch Changes + +- Updated dependencies [[`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db-sqlite-persistence-core@0.2.1 + ## 0.2.0 ### Minor Changes diff --git a/packages/react-native-db-sqlite-persistence/package.json b/packages/react-native-db-sqlite-persistence/package.json index 36fb14352c..46733ff76b 100644 --- a/packages/react-native-db-sqlite-persistence/package.json +++ b/packages/react-native-db-sqlite-persistence/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/react-native-db-sqlite-persistence", - "version": "0.2.0", + "version": "0.2.21", "description": "React Native and Expo SQLite persisted collection adapter for TanStack DB", "author": "TanStack Team", "license": "MIT", diff --git a/packages/react-router-with-db/CHANGELOG.md b/packages/react-router-with-db/CHANGELOG.md new file mode 100644 index 0000000000..de46d8d59f --- /dev/null +++ b/packages/react-router-with-db/CHANGELOG.md @@ -0,0 +1,18 @@ +# @tanstack/react-router-with-db + +## 0.1.0 + +### Minor Changes + +- Add SSR through request-scoped `DbClient` instances, collection descriptors, ([#1564](https://github.com/TanStack/db/pull/1564)) + explicit collection-row hydration, live-query result snapshots, adapter sync + metadata, and React and Svelte descriptor resolution. + + React live queries now derive identity from structured query IR. Opaque queries + can provide `queryKey`; legacy dependency arrays and unkeyed opaque queries keep + working with development warnings until 1.0. + + Add TanStack Router integration that streams live queries discovered during a + Suspense render as pending promises which resolve to ordered result snapshots. + The browser starts normal source sync and atomically replaces the snapshot when + its live result is ready. diff --git a/packages/react-router-with-db/README.md b/packages/react-router-with-db/README.md new file mode 100644 index 0000000000..de90857371 --- /dev/null +++ b/packages/react-router-with-db/README.md @@ -0,0 +1,21 @@ +# @tanstack/react-router-with-db + +TanStack Router and TanStack Start SSR integration for TanStack DB. + +```tsx +const dbClient = new DbClient() +const router = createRouter({ + routeTree, + context: { dbClient }, +}) + +export default routerWithDbClient(router, dbClient) +``` + +The adapter provides the client, hydrates critical DB state, and streams +`useLiveSuspenseQuery` calls discovered during server rendering. Streamed +promises resolve to ordered live-query result snapshots. Source collections +start normally in the browser and replace the snapshot when their live result is +ready. + +See the [SSR and Hydration guide](../../docs/guides/ssr.md). diff --git a/packages/react-router-with-db/package.json b/packages/react-router-with-db/package.json new file mode 100644 index 0000000000..81203e42d0 --- /dev/null +++ b/packages/react-router-with-db/package.json @@ -0,0 +1,66 @@ +{ + "name": "@tanstack/react-router-with-db", + "version": "0.1.0", + "description": "TanStack Router SSR integration for TanStack DB", + "author": "Kyle Mathews", + "license": "MIT", + "repository": { + "type": "git", + "url": "git+https://github.com/TanStack/db.git", + "directory": "packages/react-router-with-db" + }, + "homepage": "https://tanstack.com/db", + "keywords": [ + "database", + "react", + "router", + "ssr", + "streaming", + "tanstack" + ], + "scripts": { + "build": "vite build", + "build:minified": "vite build --minify", + "dev": "vite build --watch", + "lint": "eslint . --fix", + "test": "vitest --run" + }, + "type": "module", + "main": "dist/cjs/index.cjs", + "module": "dist/esm/index.js", + "types": "dist/esm/index.d.ts", + "exports": { + ".": { + "import": { + "types": "./dist/esm/index.d.ts", + "default": "./dist/esm/index.js" + }, + "require": { + "types": "./dist/cjs/index.d.cts", + "default": "./dist/cjs/index.cjs" + } + }, + "./package.json": "./package.json" + }, + "sideEffects": false, + "files": [ + "dist", + "src" + ], + "peerDependencies": { + "@tanstack/react-db": ">=0.2.1", + "@tanstack/react-router": ">=1.43.2", + "@tanstack/router-core": ">=1.127.0", + "react": ">=18.0.0", + "react-dom": ">=18.0.0" + }, + "devDependencies": { + "@tanstack/react-db": "workspace:*", + "@tanstack/react-router": "^1.159.5", + "@tanstack/router-core": "^1.159.4", + "@vitejs/plugin-react": "^5.1.3", + "@vitest/coverage-istanbul": "^3.2.4", + "react": "^19.2.4", + "react-dom": "^19.2.4" + } +} diff --git a/packages/react-router-with-db/src/index.tsx b/packages/react-router-with-db/src/index.tsx new file mode 100644 index 0000000000..3124966fed --- /dev/null +++ b/packages/react-router-with-db/src/index.tsx @@ -0,0 +1,196 @@ +import { Fragment } from 'react' +import { DbProvider } from '@tanstack/react-db' +import '@tanstack/router-core/ssr/client' +import type { AnyRouter } from '@tanstack/react-router' +import type { DbClient, DehydratedDbState } from '@tanstack/react-db' +import type { ReactNode } from 'react' + +type AdditionalOptions = { + WrapProvider?: (props: { children: ReactNode }) => React.JSX.Element +} + +export type DehydratedRouterDbState = { + dehydratedDbClient: DehydratedDbState + dbStream: ReadableStream +} + +export type ValidateRouter = + NonNullable extends { dbClient: DbClient } + ? TRouter + : never + +export function routerWithDbClient( + router: ValidateRouter, + dbClient: DbClient, + additionalOptions?: AdditionalOptions, +): TRouter { + const originalOptions = router.options + + router.options = { + ...router.options, + context: { + ...originalOptions.context, + dbClient, + }, + Wrap: ({ children }) => { + const OuterWrapper = additionalOptions?.WrapProvider ?? Fragment + const OriginalWrap = originalOptions.Wrap ?? Fragment + + return ( + + + {children} + + + ) + }, + } + + if (router.isServer) { + dbClient._setSsrStreamingEnabled(true) + dbClient._setSsrServerCleanupEnabled(true) + const dbStream = createPushableStream() + const bufferedQueryHashes = new Set() + const streamedQueryHashes = new Set() + let criticalStateCaptured = false + let renderFinishRegistered = false + + const streamLiveQuery = (queryHash: string) => { + if (streamedQueryHashes.has(queryHash)) return + streamedQueryHashes.add(queryHash) + + const enqueued = dbStream.enqueue( + dbClient.dehydrate({ + shouldDehydrateCollection: () => false, + shouldDehydrateLiveQuery: (query) => query.queryHash === queryHash, + }), + ) + if (!enqueued) { + console.warn( + `Tried to stream live query ${queryHash} after the DB stream was closed.`, + ) + } + } + + const unsubscribe = dbClient.subscribe((event) => { + if (event.type !== `liveQueryAdded`) return + + if (!criticalStateCaptured) { + bufferedQueryHashes.add(event.query.queryHash) + return + } + + streamLiveQuery(event.query.queryHash) + }) + + router.options.dehydrate = async (): Promise => { + const originalDehydrated = await originalOptions.dehydrate?.() + const dehydratedDbClient = dbClient.dehydrate({ + shouldDehydrateLiveQuery: () => true, + }) + const criticalQueryHashes = new Set( + dehydratedDbClient.liveQueries?.map((query) => query.queryHash), + ) + criticalStateCaptured = true + + if (!renderFinishRegistered) { + renderFinishRegistered = true + router.serverSsr!.onRenderFinished(() => { + unsubscribe() + dbStream.close() + void dbClient + .cleanup() + .catch((error) => + console.error(`Error cleaning up DbClient:`, error), + ) + }) + } + + for (const queryHash of bufferedQueryHashes) { + if (!criticalQueryHashes.has(queryHash)) streamLiveQuery(queryHash) + } + bufferedQueryHashes.clear() + + return { + ...originalDehydrated, + dehydratedDbClient, + dbStream: dbStream.stream, + } + } + } else { + router.options.hydrate = async (dehydrated: DehydratedRouterDbState) => { + dbClient._setSsrStreamingEnabled(true) + try { + await originalOptions.hydrate?.(dehydrated) + dbClient.hydrate(dehydrated.dehydratedDbClient) + + const reader = dehydrated.dbStream.getReader() + void readDbStream(reader, dbClient) + .catch((error) => console.error(`Error reading DB stream:`, error)) + .finally(() => { + dbClient._setSsrStreamingEnabled(false) + }) + } catch (error) { + dbClient._setSsrStreamingEnabled(false) + throw error + } + } + } + + return router +} + +async function readDbStream( + reader: ReadableStreamDefaultReader, + dbClient: DbClient, +): Promise { + try { + let entry = await reader.read() + while (!entry.done) { + dbClient.hydrate(entry.value) + entry = await reader.read() + } + } catch (error) { + dbClient._failPendingLiveQueries(error) + throw error + } +} + +type PushableStream = { + stream: ReadableStream + enqueue: (chunk: T) => boolean + close: () => void + error: (error: unknown) => void +} + +function createPushableStream(): PushableStream { + let controllerRef!: ReadableStreamDefaultController + let state: `open` | `closed` | `errored` | `cancelled` = `open` + const stream = new ReadableStream({ + start(controller) { + controllerRef = controller + }, + cancel() { + state = `cancelled` + }, + }) + + return { + stream, + enqueue: (chunk) => { + if (state !== `open`) return false + controllerRef.enqueue(chunk) + return true + }, + close: () => { + if (state !== `open`) return + state = `closed` + controllerRef.close() + }, + error: (error) => { + if (state !== `open`) return + state = `errored` + controllerRef.error(error) + }, + } +} diff --git a/packages/react-router-with-db/tests/index.test-d.ts b/packages/react-router-with-db/tests/index.test-d.ts new file mode 100644 index 0000000000..edb42fa95d --- /dev/null +++ b/packages/react-router-with-db/tests/index.test-d.ts @@ -0,0 +1,27 @@ +import { expectTypeOf, test } from 'vitest' +import { + createRootRouteWithContext, + createRouter, +} from '@tanstack/react-router' +import { DbClient } from '@tanstack/react-db' +import { routerWithDbClient } from '../src' + +test(`requires DbClient in router context and preserves the router type`, () => { + const dbClient = new DbClient() + const rootRoute = createRootRouteWithContext<{ dbClient: DbClient }>()() + const router = createRouter({ + routeTree: rootRoute, + context: { dbClient }, + }) + + expectTypeOf(routerWithDbClient(router, dbClient)).toEqualTypeOf(router) + + const invalidRootRoute = createRootRouteWithContext<{}>()() + const invalidRouter = createRouter({ + routeTree: invalidRootRoute, + context: {}, + }) + + // @ts-expect-error router context must contain dbClient + routerWithDbClient(invalidRouter, dbClient) +}) diff --git a/packages/react-router-with-db/tests/index.test.ts b/packages/react-router-with-db/tests/index.test.ts new file mode 100644 index 0000000000..d60b45716f --- /dev/null +++ b/packages/react-router-with-db/tests/index.test.ts @@ -0,0 +1,296 @@ +import { readFileSync } from 'node:fs' +import { describe, expect, it, vi } from 'vitest' +import { DbClient, collectionOptions } from '@tanstack/react-db' +import { routerWithDbClient } from '../src' +import type { AnyRouter } from '@tanstack/react-router' +import type { DehydratedDbState } from '@tanstack/react-db' +import type { DehydratedRouterDbState } from '../src' + +type Todo = { + id: string + text: string +} + +const adaptRouter = routerWithDbClient as unknown as ( + router: AnyRouter, + dbClient: DbClient, +) => AnyRouter + +function createTodoDescriptor() { + return collectionOptions(`todos`, () => ({ + id: `todos`, + getKey: (todo: Todo) => todo.id, + sync: { + sync: ({ markReady }) => markReady(), + }, + })) +} + +describe(`routerWithDbClient`, () => { + it(`declares peer floors that contain the imported SSR APIs`, () => { + const packageJson = JSON.parse(readFileSync(`package.json`, `utf8`)) as { + peerDependencies: Record + } + + expect(packageJson.peerDependencies).toMatchObject({ + '@tanstack/react-db': `>=0.2.1`, + '@tanstack/router-core': `>=1.127.0`, + }) + }) + + it(`leaves SSR streaming disabled in the browser until hydration starts`, () => { + const dbClient = new DbClient() + const router = { + options: { context: { dbClient } }, + isServer: false, + } as unknown as AnyRouter + + adaptRouter(router, dbClient) + + expect(dbClient._isSsrStreamingEnabled()).toBe(false) + }) + + it(`cleans up server collections when rendering finishes`, async () => { + const cleanup = vi.fn() + const dbClient = new DbClient() + const collection = dbClient.collection( + collectionOptions(`server-cleanup`, () => ({ + id: `server-cleanup`, + getKey: (todo: Todo) => todo.id, + sync: { + sync: ({ markReady }) => { + markReady() + return cleanup + }, + }, + })), + ) + await collection.preload() + let finishRender = () => {} + const router = { + options: { context: { dbClient } }, + isServer: true, + serverSsr: { + isDehydrated: () => false, + onRenderFinished: (callback: () => void) => { + finishRender = callback + }, + }, + } as unknown as AnyRouter + + adaptRouter(router, dbClient) + expect(dbClient._isSsrServerCleanupEnabled()).toBe(true) + await router.options.dehydrate?.() + finishRender() + + await vi.waitFor(() => expect(cleanup).toHaveBeenCalledOnce()) + expect(dbClient._isSsrServerCleanupEnabled()).toBe(false) + }) + + it(`streams live queries registered after critical dehydration`, async () => { + const dbClient = new DbClient() + let isDehydrated = false + let finishRender = () => {} + const router = { + options: { context: { dbClient } }, + isServer: true, + serverSsr: { + isDehydrated: () => isDehydrated, + onRenderFinished: (callback: () => void) => { + finishRender = callback + }, + }, + } as unknown as AnyRouter + + adaptRouter(router, dbClient) + const initialState = (await router.options.dehydrate?.()) as + | DehydratedRouterDbState + | undefined + expect(initialState).toBeDefined() + expect(initialState!.dehydratedDbClient.liveQueries).toBeUndefined() + + isDehydrated = true + let resolveQuery!: (snapshot: { + rows: Array<{ key: string; value: Todo }> + }) => void + const queryPromise = new Promise<{ + rows: Array<{ key: string; value: Todo }> + }>((resolve) => { + resolveQuery = resolve + }) + dbClient._registerLiveQuery(`open-todos`, queryPromise) + + const reader = initialState!.dbStream.getReader() + const streamedState = await reader.read() + expect(streamedState.done).toBe(false) + expect(streamedState.value?.collections).toEqual([]) + expect(streamedState.value?.liveQueries?.[0]?.queryHash).toBe(`open-todos`) + + resolveQuery({ + rows: [{ key: `1`, value: { id: `1`, text: `Streamed` } }], + }) + + await expect( + streamedState.value!.liveQueries![0]!.promise, + ).resolves.toEqual({ + rows: [{ key: `1`, value: { id: `1`, text: `Streamed` } }], + }) + + finishRender() + await expect(reader.read()).resolves.toEqual({ + done: true, + value: undefined, + }) + }) + + it(`includes queries registered while critical dehydration is pending`, async () => { + const dbClient = new DbClient() + let releaseOriginalDehydrate!: () => void + const originalDehydrate = new Promise((resolve) => { + releaseOriginalDehydrate = resolve + }) + let finishRender = () => {} + const router = { + options: { + context: { dbClient }, + dehydrate: () => originalDehydrate, + }, + isServer: true, + serverSsr: { + isDehydrated: () => false, + onRenderFinished: (callback: () => void) => { + finishRender = callback + }, + }, + } as unknown as AnyRouter + + adaptRouter(router, dbClient) + const statePromise = router.options.dehydrate?.() + dbClient._registerLiveQuery( + `during-critical`, + Promise.resolve({ rows: [] }), + ) + releaseOriginalDehydrate() + + const state = (await statePromise) as DehydratedRouterDbState + expect( + state.dehydratedDbClient.liveQueries?.map((query) => query.queryHash), + ).toEqual([`during-critical`]) + + const reader = state.dbStream.getReader() + finishRender() + await expect(reader.read()).resolves.toEqual({ + done: true, + value: undefined, + }) + }) + + it(`rejects pending live queries when the client stream fails`, async () => { + const dbClient = new DbClient() + const router = { + options: { context: { dbClient } }, + isServer: false, + } as unknown as AnyRouter + const error = new Error(`transport failed`) + const pendingSnapshot = new Promise<{ rows: [] }>(() => {}) + const dbStream = new ReadableStream({ + start(controller) { + controller.error(error) + }, + }) + const consoleError = vi.spyOn(console, `error`).mockImplementation(() => {}) + + adaptRouter(router, dbClient) + await router.options.hydrate?.({ + dehydratedDbClient: { + collections: [], + liveQueries: [ + { + queryHash: `pending`, + dehydratedAt: 1, + promise: pendingSnapshot, + }, + ], + }, + dbStream, + } satisfies DehydratedRouterDbState) + + await expect(dbClient._getLiveQuery(`pending`)?.promise).rejects.toBe(error) + await vi.waitFor(() => { + expect(dbClient._isSsrStreamingEnabled()).toBe(false) + }) + consoleError.mockRestore() + }) + + it(`does not enqueue after the stream is cancelled`, async () => { + const dbClient = new DbClient() + let finishRender = () => {} + const router = { + options: { context: { dbClient } }, + isServer: true, + serverSsr: { + isDehydrated: () => true, + onRenderFinished: (callback: () => void) => { + finishRender = callback + }, + }, + } as unknown as AnyRouter + const warning = vi.spyOn(console, `warn`).mockImplementation(() => {}) + + adaptRouter(router, dbClient) + const state = + (await router.options.dehydrate?.()) as DehydratedRouterDbState + await state.dbStream.cancel() + + expect(() => + dbClient._registerLiveQuery(`late`, Promise.resolve({ rows: [] })), + ).not.toThrow() + expect(warning).toHaveBeenCalledWith( + expect.stringContaining(`after the DB stream was closed`), + ) + + finishRender() + warning.mockRestore() + }) + + it(`hydrates every client stream entry`, async () => { + const dbClient = new DbClient() + const todoDescriptor = createTodoDescriptor() + const originalHydrate = vi.fn() + const router = { + options: { + context: { dbClient }, + hydrate: originalHydrate, + }, + isServer: false, + } as unknown as AnyRouter + const dbStream = new ReadableStream({ + start(controller) { + controller.enqueue({ + collections: [ + { + collectionId: `todos`, + rows: [{ key: `1`, value: { id: `1`, text: `From the stream` } }], + }, + ], + }) + controller.close() + }, + }) + + adaptRouter(router, dbClient) + await router.options.hydrate?.({ + dehydratedDbClient: { collections: [] }, + dbStream, + } satisfies DehydratedRouterDbState) + + expect(originalHydrate).toHaveBeenCalledOnce() + await vi.waitFor(() => { + expect(dbClient.collection(todoDescriptor).get(`1`)).toMatchObject({ + id: `1`, + text: `From the stream`, + }) + expect(dbClient._isSsrStreamingEnabled()).toBe(false) + }) + }) +}) diff --git a/packages/react-router-with-db/tsconfig.json b/packages/react-router-with-db/tsconfig.json new file mode 100644 index 0000000000..5a0367056f --- /dev/null +++ b/packages/react-router-with-db/tsconfig.json @@ -0,0 +1,22 @@ +{ + "extends": "../../tsconfig.json", + "compilerOptions": { + "target": "ES2020", + "module": "ESNext", + "moduleResolution": "Bundler", + "declaration": true, + "outDir": "dist", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "jsx": "react-jsx", + "paths": { + "@tanstack/db": ["../db/src"], + "@tanstack/db-ivm": ["../db-ivm/src"], + "@tanstack/react-db": ["../react-db/src"] + } + }, + "include": ["src/**/*", "tests", "vite.config.ts"], + "exclude": ["node_modules", "dist"] +} diff --git a/packages/react-router-with-db/vite.config.ts b/packages/react-router-with-db/vite.config.ts new file mode 100644 index 0000000000..922872e859 --- /dev/null +++ b/packages/react-router-with-db/vite.config.ts @@ -0,0 +1,24 @@ +import { defineConfig, mergeConfig } from 'vitest/config' +import { tanstackViteConfig } from '@tanstack/vite-config' +import react from '@vitejs/plugin-react' +import packageJson from './package.json' + +export default defineConfig(async () => { + const tanstack = await tanstackViteConfig({ + entry: `./src/index.tsx`, + srcDir: `./src`, + }) + + const base = { + plugins: [react()], + test: { + name: packageJson.name, + dir: `./tests`, + environment: `jsdom`, + coverage: { enabled: true, provider: `istanbul`, include: [`src/**/*`] }, + typecheck: { enabled: true }, + }, + } + + return mergeConfig(tanstack, base) +}) diff --git a/packages/rxdb-db-collection/CHANGELOG.md b/packages/rxdb-db-collection/CHANGELOG.md index 2e6f39e24f..eff6a3c769 100644 --- a/packages/rxdb-db-collection/CHANGELOG.md +++ b/packages/rxdb-db-collection/CHANGELOG.md @@ -1,5 +1,172 @@ # @tanstack/rxdb-db-collection +## 0.1.95 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 0.1.94 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 0.1.93 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 0.1.92 + +### Patch Changes + +- Settle subset loads only after their committed rows and events are visible. A ([#1769](https://github.com/TanStack/db/pull/1769)) + commit receipt now rejects with `AbortError` when cancellation wins before + application and ignores later aborts. Preserve causal publication, + cancellation, persistence, and error handling across the affected sync + adapters. +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 0.1.91 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 0.1.90 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 0.1.89 + +### Patch Changes + +- Propagate initial query sync failures through dependent live queries and readiness promises, including recovery and late subscribers, while preserving a ready cached snapshot on later refetch failures. Let sync adapters pass the original failure to `markError(error)` so readiness promises reject with that cause. Isolate adapter callbacks by sync session, preserve synchronous startup errors, and prevent rejected deduplicated subset requests from creating detached promise rejections. ([#1751](https://github.com/TanStack/db/pull/1751)) + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 0.1.88 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 0.1.87 + +### Patch Changes + +- Add SSR through request-scoped `DbClient` instances, collection descriptors, ([#1564](https://github.com/TanStack/db/pull/1564)) + explicit collection-row hydration, live-query result snapshots, adapter sync + metadata, and React and Svelte descriptor resolution. + + React live queries now derive identity from structured query IR. Opaque queries + can provide `queryKey`; legacy dependency arrays and unkeyed opaque queries keep + working with development warnings until 1.0. + + Add TanStack Router integration that streams live queries discovered during a + Suspense render as pending promises which resolve to ordered result snapshots. + The browser starts normal source sync and atomically replaces the snapshot when + its live result is ready. + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 0.1.86 + +### Patch Changes + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 0.1.85 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 0.1.84 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 0.1.83 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 0.1.82 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 0.1.81 + +### Patch Changes + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 0.1.80 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 0.1.79 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 0.1.78 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 0.1.77 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 0.1.76 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 0.1.75 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 0.1.74 ### Patch Changes diff --git a/packages/rxdb-db-collection/package.json b/packages/rxdb-db-collection/package.json index edd6a3474b..32edfbc4e3 100644 --- a/packages/rxdb-db-collection/package.json +++ b/packages/rxdb-db-collection/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/rxdb-db-collection", - "version": "0.1.74", + "version": "0.1.95", "description": "Reactive, Offline-First adapter for TanStack DB using RxDB. Sync, Replication and Local-First support.", "author": "Kyle Mathews", "license": "MIT", diff --git a/packages/rxdb-db-collection/src/rxdb.ts b/packages/rxdb-db-collection/src/rxdb.ts index 6ec59cfb69..bd8d1bd89d 100644 --- a/packages/rxdb-db-collection/src/rxdb.ts +++ b/packages/rxdb-db-collection/src/rxdb.ts @@ -7,6 +7,7 @@ import { rxStorageWriteErrorToRxError, } from 'rxdb/plugins/core' import DebugModule from 'debug' +import { withCollectionConfigFactory } from '@tanstack/db' import { stripRxdbFields } from './helper' import type { FilledMangoQuery, @@ -101,7 +102,9 @@ export function rxdbCollectionOptions( schema?: never // no schema in the result } -export function rxdbCollectionOptions(config: RxDBCollectionConfig) { +export function rxdbCollectionOptions( + config: RxDBCollectionConfig, +): CollectionConfig { type Row = Record type Key = string // because RxDB primary keys must be strings @@ -124,9 +127,9 @@ export function rxdbCollectionOptions(config: RxDBCollectionConfig) { type SyncParams = Parameters[`sync`]>[0] const sync: SyncConfig = { sync: (params: SyncParams) => { - const { begin, write, commit, markReady } = params + const { begin, write, commit, markReady, markError, collection } = params - let ready = false + let initialFetchComplete = false async function initialFetch() { /** * RxDB stores a last-write-time @@ -137,7 +140,7 @@ export function rxdbCollectionOptions(config: RxDBCollectionConfig) { const syncBatchSize = config.syncBatchSize ? config.syncBatchSize : 1000 begin() - while (!ready) { + while (!initialFetchComplete) { let query: FilledMangoQuery if (cursor) { query = { @@ -181,7 +184,7 @@ export function rxdbCollectionOptions(config: RxDBCollectionConfig) { cursor = lastOfArray(docs) if (docs.length === 0) { - ready = true + initialFetchComplete = true break } @@ -192,13 +195,14 @@ export function rxdbCollectionOptions(config: RxDBCollectionConfig) { }) }) } - commit() + await commit() } type WriteMessage = Parameters[0] const buffer: Array = [] + let buffering = true const queue = (msg: WriteMessage) => { - if (!ready) { + if (buffering) { buffer.push(msg) return } @@ -207,7 +211,19 @@ export function rxdbCollectionOptions(config: RxDBCollectionConfig) { commit() } - let sub: Subscription + let sub: Subscription | undefined + function stopOngoingFetch() { + buffer.length = 0 + if (!sub) return + getFromMapOrCreate( + OPEN_RXDB_SUBSCRIPTIONS, + rxCollection, + () => new Set(), + ).delete(sub) + sub.unsubscribe() + sub = undefined + } + function startOngoingFetch() { // Subscribe early and buffer live changes during initial load and ongoing sub = rxCollection.$.subscribe((ev) => { @@ -234,30 +250,42 @@ export function rxdbCollectionOptions(config: RxDBCollectionConfig) { } async function start() { + const isCleanedUp = () => collection.status === `cleaned-up` + startOngoingFetch() await initialFetch() + if (isCleanedUp()) { + return + } - if (buffer.length) { + // Take one finite snapshot of changes observed during the initial + // fetch, then route newer events through the normal live path. The + // core transaction queue preserves their order without letting a + // continuous event stream postpone readiness forever. + const pending = buffer.splice(0) + buffering = false + if (pending.length > 0) { begin() - for (const msg of buffer) write(msg) - commit() - buffer.length = 0 + for (const msg of pending) write(msg) + await commit() + if (isCleanedUp()) { + return + } } - markReady() + if (!isCleanedUp()) { + markReady() + } } - start() + void start().catch((error: unknown) => { + stopOngoingFetch() + if (collection.status === `loading`) { + markError(error) + } + }) - return () => { - const subs = getFromMapOrCreate( - OPEN_RXDB_SUBSCRIPTIONS, - rxCollection, - () => new Set(), - ) - subs.delete(sub) - sub.unsubscribe() - } + return stopOngoingFetch }, // Expose the getSyncMetadata function getSyncMetadata: undefined, @@ -309,5 +337,7 @@ export function rxdbCollectionOptions(config: RxDBCollectionConfig) { }) }, } - return collectionConfig + return withCollectionConfigFactory(collectionConfig, () => + rxdbCollectionOptions(config), + ) } diff --git a/packages/rxdb-db-collection/tests/rxdb.test.ts b/packages/rxdb-db-collection/tests/rxdb.test.ts index c3ff49a760..a8dfbe82dd 100644 --- a/packages/rxdb-db-collection/tests/rxdb.test.ts +++ b/packages/rxdb-db-collection/tests/rxdb.test.ts @@ -1,5 +1,5 @@ -import { describe, expect, it } from 'vitest' -import { createCollection } from '@tanstack/db' +import { describe, expect, it, vi } from 'vitest' +import { createCollection, createTransaction } from '@tanstack/db' import { addRxPlugin, createRxDatabase, @@ -22,6 +22,14 @@ type RxCollections = { test: RxCollection } // Helper to advance timers and allow microtasks to flush const flushPromises = () => new Promise((resolve) => setTimeout(resolve, 0)) +function createDeferred() { + let resolve!: (value: T) => void + const promise = new Promise((resolvePromise) => { + resolve = resolvePromise + }) + return { promise, resolve } +} + describe(`RxDB Integration`, () => { addRxPlugin(RxDBDevModePlugin) @@ -103,6 +111,168 @@ describe(`RxDB Integration`, () => { } describe(`sync`, () => { + it(`reports an initial storage query failure`, async () => { + const db = await getDatababase() + const rxCollection: RxCollection = db.test + const initialError = new Error(`initial RxDB query failed`) + const query = vi + .spyOn(rxCollection.storageInstance, `query`) + .mockRejectedValueOnce(initialError) + const collection = createCollection( + rxdbCollectionOptions({ + rxCollection, + startSync: true, + syncBatchSize: 10, + }), + ) + + try { + await expect(collection.preload()).rejects.toBe(initialError) + expect(collection.status).toBe(`error`) + expect(OPEN_RXDB_SUBSCRIPTIONS.get(rxCollection)?.size ?? 0).toBe(0) + + await rxCollection.insert({ id: `after-failure`, name: `failed` }) + await flushPromises() + expect(OPEN_RXDB_SUBSCRIPTIONS.get(rxCollection)?.size ?? 0).toBe(0) + expect(collection.has(`after-failure`)).toBe(false) + } finally { + query.mockRestore() + await collection.cleanup() + await db.remove() + } + }) + + it(`marks initial sync ready only after its rows are applied`, async () => { + const db = await getDatababase([{ id: `server`, name: `Server` }]) + const rxCollection: RxCollection = db.test + const releaseInitialQuery = createDeferred() + const initialQueryStarted = createDeferred() + const storageQuery = rxCollection.storageInstance.query.bind( + rxCollection.storageInstance, + ) + const query = vi + .spyOn(rxCollection.storageInstance, `query`) + .mockImplementationOnce(async (preparedQuery) => { + const result = await storageQuery(preparedQuery) + initialQueryStarted.resolve() + await releaseInitialQuery.promise + return result + }) + const collection = createCollection( + rxdbCollectionOptions({ + rxCollection, + startSync: true, + syncBatchSize: 10, + }), + ) + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + + try { + await initialQueryStarted.promise + transaction.mutate(() => + collection.insert({ id: `local`, name: `Local` }), + ) + const buffered = await rxCollection.insert({ + id: `buffered`, + name: `Buffered`, + }) + releaseInitialQuery.resolve() + + const ready = collection.preload() + await flushPromises() + + // The initial receipt is still parked. A later live change for the + // same row must not overtake the older buffered insert. + await buffered.getLatest().patch({ name: `Newest` }) + await flushPromises() + + expect(collection.status).toBe(`loading`) + expect(collection.get(`server`)).toBeUndefined() + expect(collection.get(`buffered`)).toBeUndefined() + + persistence.resolve() + await transaction.isPersisted.promise + await ready + + expect(collection.get(`server`)).toEqual( + expect.objectContaining({ id: `server`, name: `Server` }), + ) + expect(collection.get(`buffered`)).toEqual( + expect.objectContaining({ id: `buffered`, name: `Newest` }), + ) + expect(collection.status).toBe(`ready`) + } finally { + releaseInitialQuery.resolve() + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + query.mockRestore() + await collection.cleanup() + await db.remove() + } + }) + + it(`does not let later live traffic extend the startup readiness boundary`, async () => { + const db = await getDatababase() + const rxCollection: RxCollection = db.test + const initialQueryStarted = createDeferred() + const releaseInitialQuery = createDeferred() + const bufferedApplied = createDeferred() + const laterLiveApplied = createDeferred() + const storageQuery = rxCollection.storageInstance.query.bind( + rxCollection.storageInstance, + ) + const query = vi + .spyOn(rxCollection.storageInstance, `query`) + .mockImplementationOnce(async (preparedQuery) => { + const result = await storageQuery(preparedQuery) + initialQueryStarted.resolve() + await releaseInitialQuery.promise + return result + }) + const options = rxdbCollectionOptions({ rxCollection }) + const begin = vi.fn() + const write = vi.fn() + const commit = vi + .fn() + .mockReturnValueOnce(true) + .mockReturnValueOnce(bufferedApplied.promise) + .mockReturnValueOnce(laterLiveApplied.promise) + const markReady = vi.fn() + const markError = vi.fn() + const cleanup = options.sync.sync({ + begin, + write, + commit, + markReady, + markError, + collection: { status: `loading` }, + } as never) + + try { + await initialQueryStarted.promise + await rxCollection.insert({ id: `buffered`, name: `Buffered` }) + releaseInitialQuery.resolve() + await vi.waitFor(() => expect(commit).toHaveBeenCalledTimes(2)) + + await rxCollection.insert({ id: `later`, name: `Later` }) + await vi.waitFor(() => expect(commit).toHaveBeenCalledTimes(3)) + expect(markReady).not.toHaveBeenCalled() + + bufferedApplied.resolve() + await vi.waitFor(() => expect(markReady).toHaveBeenCalledOnce()) + } finally { + releaseInitialQuery.resolve() + bufferedApplied.resolve() + laterLiveApplied.resolve() + if (typeof cleanup === `function`) cleanup() + query.mockRestore() + await db.remove() + } + }) + it(`should initialize and fetch initial data`, async () => { const initialItems = getTestData(2) diff --git a/packages/solid-db/CHANGELOG.md b/packages/solid-db/CHANGELOG.md index 28f10b3eff..fa61ab8fea 100644 --- a/packages/solid-db/CHANGELOG.md +++ b/packages/solid-db/CHANGELOG.md @@ -1,5 +1,166 @@ # @tanstack/react-db +## 0.2.43 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 0.2.42 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 0.2.41 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 0.2.40 + +### Patch Changes + +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 0.2.39 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 0.2.38 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 0.2.37 + +### Patch Changes + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 0.2.36 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 0.2.35 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 0.2.34 + +### Patch Changes + +- Update agent skills to match current APIs and behavior. ([#1696](https://github.com/TanStack/db/pull/1696)) + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 0.2.33 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 0.2.32 + +### Patch Changes + +- Add an internal shared live-query observer and migrate all five framework adapters to it ([#1642](https://github.com/TanStack/db/pull/1642)) + + Introduces `createLiveQueryObserver` in `@tanstack/db`: given a resolved live-query collection (or `null` for a disabled query) it owns the subscription lifecycle every adapter used to re-implement — change and status subscriptions, a snapshot with stable identity per state revision for wholesale consumers, and delivery of the raw `ChangeMessage[]` for granular consumers. React, Vue, Svelte, Solid, and Angular's live-query hooks now materialize from the observer instead of their own hand-rolled subscription/status/snapshot machinery, keeping each adapter's native reactivity and each adapter's data-loading policy (wholesale adapters subscribe without initial state; granular adapters seed from it). + + The observer is an **internal, unstable contract** for TanStack DB's official adapters — it is exported so the adapter packages can consume it, but it is not a public extension point yet and its API may change in any release. + + The migration also fixes several live-query lifecycle defects: status-only transitions (`error`, `cleaned-up`) now reach mounted consumers; snapshot identity is stable across unsubscribe/resubscribe and stays fresh while detached; dispatch is FIFO and non-reentrant with subscriptions identified by record rather than callback; disposing during the synchronous initial replay no longer leaks the collection subscription; subscribing after dispose throws instead of registering a dead listener; Solid guards its async resource continuations against superseded collections; and constructing an observer no longer activates sync. Observers activate on their first committed subscription unless an adapter has already started a pre-created collection supplied directly or returned from a callback. + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 0.2.31 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 0.2.30 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 0.2.29 + +### Patch Changes + +- Extract shared live-query adapter helpers into `@tanstack/db` ([#1641](https://github.com/TanStack/db/pull/1641)) + + Adds `isCollection`, `isSingleResultCollection`, and `getLiveQueryStatusFlags` to `@tanstack/db` and migrates all five framework adapters to use them. `isCollection` replaces the per-adapter duck-typing and Solid's `instanceof CollectionImpl` with one structural, multi-realm-safe guard (the `instanceof` form gave false negatives across dual-package boundaries). No behavior change; internal deduplication only. + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 0.2.28 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 0.2.27 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 0.2.26 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 0.2.25 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 0.2.24 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 0.2.23 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 0.2.22 ### Patch Changes diff --git a/packages/solid-db/README.md b/packages/solid-db/README.md index 652e582e5f..7856b9bfc5 100644 --- a/packages/solid-db/README.md +++ b/packages/solid-db/README.md @@ -1,3 +1,20 @@ +
                            + + + + TanStack Solid DB + +
                            # @tanstack/solid-db Solidjs hooks for TanStack DB. See [TanStack/db](https://github.com/TanStack/db) for more details. diff --git a/packages/solid-db/package.json b/packages/solid-db/package.json index 50debdc474..8803eece53 100644 --- a/packages/solid-db/package.json +++ b/packages/solid-db/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/solid-db", - "version": "0.2.22", + "version": "0.2.43", "description": "Solid integration for @tanstack/db", "author": "Kyle Mathews", "license": "MIT", diff --git a/packages/solid-db/skills/solid-db/SKILL.md b/packages/solid-db/skills/solid-db/SKILL.md index 291d3855c7..0039c6cc89 100644 --- a/packages/solid-db/skills/solid-db/SKILL.md +++ b/packages/solid-db/skills/solid-db/SKILL.md @@ -5,12 +5,13 @@ description: > doubles as data access (call as function) with state/status properties. Fine-grained reactivity: signal reads MUST happen inside the query function for tracking. Config passed as Accessor (() => config). Built-in Suspense - support via createResource. ReactiveMap for state. Import from + support via createResource and errors through Solid ErrorBoundary. + ReactiveMap for state. Import from @tanstack/solid-db (re-exports all of @tanstack/db). type: framework library: db framework: solid -library_version: '0.6.0' +library_version: '0.6.17' requires: - db-core sources: @@ -26,7 +27,7 @@ This skill builds on db-core. Read it first for collection setup, query builder, ```tsx import { useLiveQuery, eq, not } from '@tanstack/solid-db' -import { For, Show, Suspense } from 'solid-js' +import { ErrorBoundary, For, Show, Suspense } from 'solid-js' function TodoList() { const todosQuery = useLiveQuery((q) => @@ -131,12 +132,16 @@ return {(user) =>
                            {user().name}
                            }                             ### Suspense integration ```tsx -Loading...}> - {(todo) =>
                          • {todo.text}
                            • }                             -                             +
                            {error.message}
                            }> + Loading...}> + {(todo) =>
                          • {todo.text}
                            • }                             +                             +                             ``` -`useLiveQuery` integrates with Solid's `createResource` — wrap in `` for loading states. +`useLiveQuery` integrates with Solid's `createResource`. Use `` for +loading and `` for errors. Reading an errored query throws +through the resource, so do not rely on reading `isError` after failure. ## Includes (Hierarchical Data) diff --git a/packages/solid-db/src/useLiveQuery.ts b/packages/solid-db/src/useLiveQuery.ts index 5380bae00d..cf8d84f43d 100644 --- a/packages/solid-db/src/useLiveQuery.ts +++ b/packages/solid-db/src/useLiveQuery.ts @@ -9,15 +9,16 @@ import { import { ReactiveMap } from '@solid-primitives/map' import { BaseQueryBuilder, - CollectionImpl, createLiveQueryCollection, + createLiveQueryObserver, + isCollection, + isSingleResultCollection, } from '@tanstack/db' import { createStore, reconcile } from 'solid-js/store' import type { Accessor } from 'solid-js' import type { ChangeMessage, Collection, - CollectionConfigSingleRowOption, CollectionStatus, Context, GetResult, @@ -311,7 +312,7 @@ export function useLiveQuery( return null } - if (innerCollection instanceof CollectionImpl) { + if (isCollection(innerCollection)) { innerCollection.startSyncImmediate() return innerCollection as Collection } @@ -350,9 +351,16 @@ export function useLiveQuery( ) } + // Generation guard for the resource's async continuations: Solid discards a + // superseded fetch's *return value*, but the writes below are side effects + // into hook-scoped state and would still run — resurrecting rows/status from + // a collection that has already been replaced. + let resourceGeneration = 0 + const [getDataResource] = createResource( () => ({ currentCollection: collection() }), async ({ currentCollection }) => { + const generation = ++resourceGeneration if (!currentCollection) { return [] } @@ -360,9 +368,12 @@ export function useLiveQuery( try { await currentCollection.toArrayWhenReady() } catch (error) { - setStatus(`error`) + if (generation === resourceGeneration) setStatus(`error`) throw error } + if (generation !== resourceGeneration) { + return data + } // Initialize state with current collection data batch(() => { state.clear() @@ -389,36 +400,54 @@ export function useLiveQuery( setData([]) return } - const subscription = currentCollection.subscribeChanges( - (changes: Array>) => { - // Apply each change individually to the reactive state + + // The shared observer owns subscription, the ready-race, and status; Solid + // materializes into its keyed ReactiveMap (granular) + reconciled store. + const observer = createLiveQueryObserver(currentCollection) + // Clear any keys carried over from a previous collection before the new + // observer re-seeds via `includeInitialState` (which only inserts current + // rows, never deletes stale ones). Without this, switching collections + // leaves the dropped keys in `state` until the async resource reconciles. + state.clear() + const unsubscribe = observer.subscribe( + (changes: Array> | undefined) => { batch(() => { - for (const change of changes) { - switch (change.type) { - case `insert`: - case `update`: - state.set(change.key, change.value) - break - case `delete`: - state.delete(change.key) - break + if (changes) { + for (const change of changes) { + switch (change.type) { + case `insert`: + case `update`: + state.set(change.key, change.value) + break + case `delete`: + state.delete(change.key) + break + } + } + } else { + // Cleanup and other status-only publications carry no row deltas. + // Rebuild the keyed view so it cannot diverge from ordered data. + state.clear() + for (const [key, value] of observer.getSnapshot().state ?? []) { + state.set(key, value) } } - syncDataFromCollection(currentCollection) - - // Update status ref on every change - setStatus(currentCollection.status) + setStatus(observer.getSnapshot().status) }) }, - { - // Include initial state to ensure immediate population for pre-created collections - includeInitialState: true, - }, ) + // An already-ready empty collection produces no initial row batch. Bring + // ordered data and status in line synchronously instead of waiting for the + // resource continuation to correct the previous collection's rows. + batch(() => { + syncDataFromCollection(currentCollection) + setStatus(observer.getSnapshot().status) + }) onCleanup(() => { - subscription.unsubscribe() + unsubscribe() + observer.dispose() }) }) @@ -426,9 +455,7 @@ export function useLiveQuery( function getData() { const currentCollection = collection() if (currentCollection) { - const config: CollectionConfigSingleRowOption = - currentCollection.config - if (config.singleResult) { + if (isSingleResultCollection(currentCollection)) { // Force resource tracking so Suspense works getDataResource() return data[0] diff --git a/packages/solid-db/tests/conformance.test.tsx b/packages/solid-db/tests/conformance.test.tsx new file mode 100644 index 0000000000..ddb1cfbd08 --- /dev/null +++ b/packages/solid-db/tests/conformance.test.tsx @@ -0,0 +1,227 @@ +/** + * Solid driver for the shared live-query conformance suite. + * + * Each mount runs inside a `createRoot` so `unmount` disposes via the captured + * dispose fn; the root stays alive between mount and reads so Solid's reactive + * getters stay current. Solid auto-tracks signals, so controllable inputs use a + * signal read inside the query fn (no deps array). Collection/config inputs are + * passed as accessors, per Solid's arity-based input detection. + * + * `knownGaps` is populated empirically from the run below. + */ +import { + coalesce, + count, + createCollection, + createLiveQueryCollection, + createOptimisticAction, + eq, + gt, + sum, +} from '@tanstack/db' +import { createRoot, createSignal } from 'solid-js' +import { + mockSyncCollectionOptions, + mockSyncCollectionOptionsNoInitialState, +} from '../../db/tests/utils' +import { useLiveQuery } from '../src/useLiveQuery' +import { runSuite } from '../../db/tests/conformance/suite' +import type { + ConformanceResult, + ControllableHandle, + DeferredSourceHandle, + LiveQueryDriver, + LiveQueryHandle, + QueryBuild, + SourceHandle, +} from '../../db/tests/conformance/contract' + +let sourceSeq = 0 + +function writer(collection: any) { + return (type: `insert` | `update` | `delete`, value: T) => { + collection.utils.begin() + collection.utils.write({ type, value }) + collection.utils.commit() + } +} + +function makeSource( + initialData: ReadonlyArray, +): SourceHandle { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `conformance-solid-${sourceSeq++}`, + getKey: (r) => r.id, + initialData: [...initialData], + }), + ) + const write = writer(collection) + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + } +} + +function makeDeferredSource< + T extends { id: string }, +>(): DeferredSourceHandle { + const collection = createCollection( + mockSyncCollectionOptionsNoInitialState({ + id: `conformance-solid-${sourceSeq++}`, + getKey: (r) => r.id, + }), + ) + collection.startSyncImmediate() + const write = writer(collection) + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + emit: (rows) => { + collection.utils.begin() + rows.forEach((value) => collection.utils.write({ type: `insert`, value })) + collection.utils.commit() + }, + markReady: () => collection.utils.markReady(), + } +} + +function makePrecreated(build: QueryBuild, opts?: { startSync?: boolean }) { + const collection = createLiveQueryCollection({ + query: build as any, + startSync: opts?.startSync ?? true, + }) + return { collection } +} + +function makeErrorSource() { + const collection = createCollection<{ id: string }>({ + id: `conformance-solid-err-${sourceSeq++}`, + getKey: (r) => r.id, + startSync: false, + sync: { + sync: () => { + throw new Error(`conformance: sync failure`) + }, + }, + }) + try { + collection.startSyncImmediate() + } catch { + // expected: engine catches the sync error and sets status to `error` + } + return { collection } +} + +async function settle() { + await new Promise((resolve) => setTimeout(resolve, 10)) +} + +function makeHandle( + getResult: () => any, + dispose: () => void, +): LiveQueryHandle { + return { + current(): ConformanceResult { + const result = getResult() + return { + data: result?.data, + state: result?.state, + status: result?.status ?? `idle`, + isReady: Boolean(result?.isReady), + isError: Boolean(result?.isError), + // solid-db exposes no `isEnabled`; derive it from status (status-derived). + isEnabled: result?.status !== `disabled`, + } + }, + flush: settle, + async apply(fn: () => void) { + fn() + await settle() + }, + unmount() { + dispose() + }, + } +} + +function inRoot(fn: () => any): { getResult: () => any; dispose: () => void } { + let result: any + let dispose!: () => void + createRoot((d) => { + dispose = d + result = fn() + }) + return { getResult: () => result, dispose } +} + +function mount(build: QueryBuild) { + const { getResult, dispose } = inRoot(() => useLiveQuery(build as any)) + return makeHandle(getResult, dispose) +} + +function mountCollection(collection: any) { + // Solid accepts a pre-created collection via an accessor. + const { getResult, dispose } = inRoot(() => useLiveQuery(() => collection)) + return makeHandle(getResult, dispose) +} + +function mountConfig(build: QueryBuild) { + // Solid accepts the config-object form via an accessor. + const { getResult, dispose } = inRoot(() => + useLiveQuery(() => ({ query: build })), + ) + return makeHandle(getResult, dispose) +} + +function mountDisabled() { + // Disabled: an accessor returning null. + const { getResult, dispose } = inRoot(() => useLiveQuery(() => null)) + return makeHandle(getResult, dispose) +} + +function mountControllable

                            ( + build: (q: any, param: P) => any, + initial: P, +): ControllableHandle

                            { + const [param, setParam] = createSignal

                            (initial) + const { getResult, dispose } = inRoot(() => + // Reading param() inside the query fn makes Solid recompute on change. + useLiveQuery((q: any) => build(q, param())), + ) + const handle = makeHandle(getResult, dispose) + return { + ...handle, + async setParam(next: P) { + setParam(() => next) + await settle() + }, + } +} + +const solidDriver: LiveQueryDriver = { + name: `solid`, + ops: { eq, gt, count, sum, coalesce, createOptimisticAction }, + makeSource, + makeDeferredSource, + makePrecreated, + makeErrorSource, + mount, + mountControllable, + mountCollection, + mountConfig, + mountDisabled, + // solid-db routes errors through its createResource/Suspense path: reading an + // errored query throws (CollectionStateError) for an to catch, + // rather than exposing a readable isError flag. That's a framework idiom, not a + // gap — the error-status scenario is parametrized to assert it via the boundary. + errorSurface: `throw`, + knownGaps: [], + features: { serverSnapshot: false, suspense: true }, +} + +runSuite(solidDriver) diff --git a/packages/solid-db/tests/useLiveQuery.test.tsx b/packages/solid-db/tests/useLiveQuery.test.tsx index 378c2a0fc7..85f9144fc1 100644 --- a/packages/solid-db/tests/useLiveQuery.test.tsx +++ b/packages/solid-db/tests/useLiveQuery.test.tsx @@ -8,6 +8,7 @@ import { createOptimisticAction, eq, gt, + toArray, } from '@tanstack/db' import { For, @@ -85,6 +86,37 @@ const initialIssues: Array = [ ] describe(`Query Collections`, () => { + it(`clears data immediately when switching to an already-ready empty collection`, async () => { + return createRoot(async (dispose) => { + const populated = createCollection( + mockSyncCollectionOptions({ + id: `solid-populated-switch`, + getKey: (person) => person.id, + initialData: initialPersons, + }), + ) + const empty = createCollection( + mockSyncCollectionOptions({ + id: `solid-empty-switch`, + getKey: (person) => person.id, + initialData: [], + }), + ) + populated.startSyncImmediate() + empty.startSyncImmediate() + + const [current, setCurrent] = createSignal(populated) + const result = useLiveQuery(current) + await waitFor(() => expect(result()).toHaveLength(3)) + + setCurrent(empty) + + expect(result()).toHaveLength(0) + expect(result.state.size).toBe(0) + dispose() + }) + }) + it(`should work with basic collection and select`, async () => { const collection = createCollection( mockSyncCollectionOptions({ @@ -523,6 +555,125 @@ describe(`Query Collections`, () => { }) }) + it(`should drop stale keys from state synchronously when parameters narrow`, async () => { + // Narrowing recompiles into a *new* collection with fewer keys. The + // observer re-seeds via `includeInitialState`, which only inserts current + // rows and never deletes the previous collection's keys. `state` must be + // cleared synchronously so the dropped keys don't linger in the window + // before the async resource reconciles (this reads `state` with no settle; + // `data`, rebuilt wholesale, stays correct either way). + return createRoot(async (dispose) => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `stale-keys-on-narrow-test`, + getKey: (person: Person) => person.id, + initialData: initialPersons, + }), + ) + + const [minAge, setMinAge] = createSignal(10) + const rendered = renderHook( + (props: { minAge: Accessor }) => { + return useLiveQuery((q) => + q + .from({ collection }) + .where(({ collection: c }) => gt(c.age, props.minAge())) + .select(({ collection: c }) => ({ id: c.id })), + ) + }, + { initialProps: [{ minAge }] }, + ) + + await new Promise((resolve) => setTimeout(resolve, 50)) + expect(rendered.result.state.size).toBe(3) // all three ages > 10 + + // Narrow to only John Smith (age 35); ids 1 and 2 must not linger. + setMinAge(32) + + expect(rendered.result.state.size).toBe(1) + expect(rendered.result.state.has(`1`)).toBe(false) + expect(rendered.result.state.has(`2`)).toBe(false) + + dispose() + }) + }) + + it(`does not resurrect state from a superseded collection's async continuation`, async () => { + // The resource fetcher awaits toArrayWhenReady(); if the collection is + // switched while that await is pending, the old continuation must not + // write its (now stale) rows/status over the new collection's. + return createRoot(async (dispose) => { + let beginA: (() => void) | undefined + let writeA: ((msg: any) => void) | undefined + let commitA: (() => void) | undefined + let markReadyA: (() => void) | undefined + + const slowCollection = createCollection({ + id: `superseded-async-slow`, + getKey: (person: Person) => person.id, + startSync: false, + sync: { + sync: ({ begin, write, commit, markReady }) => { + beginA = begin + writeA = write + commitA = commit + markReadyA = markReady + // Stays loading until markReady is called manually. + }, + }, + }) + const fastCollection = createCollection( + mockSyncCollectionOptions({ + id: `superseded-async-fast`, + getKey: (person: Person) => person.id, + initialData: [initialPersons[0]!], + }), + ) + + const [useSlow, setUseSlow] = createSignal(true) + const rendered = renderHook(() => { + return useLiveQuery((q) => + q + .from({ persons: useSlow() ? slowCollection : fastCollection }) + .select(({ persons }) => ({ id: persons.id, name: persons.name })), + ) + }) + + await new Promise((resolve) => setTimeout(resolve, 10)) + expect(rendered.result.isLoading).toBe(true) + + // Switch collections while the slow fetch is still awaiting readiness. + setUseSlow(false) + await new Promise((resolve) => setTimeout(resolve, 10)) + expect(rendered.result.state.has(`1`)).toBe(true) + + // The superseded collection now becomes ready with different rows; its + // continuation resolves but must not clobber the current state. + beginA!() + writeA!({ + type: `insert`, + value: { + id: `stale`, + name: `Stale Row`, + age: 99, + email: `stale@example.com`, + isActive: false, + team: `none`, + }, + }) + commitA!() + markReadyA!() + await new Promise((resolve) => setTimeout(resolve, 20)) + + expect(rendered.result.state.has(`stale`)).toBe(false) + expect(rendered.result.state.has(`1`)).toBe(true) + expect(rendered.result.data.map((p: any) => p.id)).toEqual([`1`]) + expect(rendered.result.status).toBe(`ready`) + + dispose() + }) + }) + it(`should be able to query a result collection with live updates`, async () => { const collection = createCollection( mockSyncCollectionOptions({ @@ -2588,3 +2739,152 @@ describe(`Query Collections`, () => { }) }) }) + +describe(`includes subqueries`, () => { + type Project = { + id: number + name: string + } + + type ProjectIssue = { + id: number + projectId: number + title: string + } + + function includedIssues(value: unknown): Array { + if (Array.isArray(value)) { + return value as Array + } + if ( + value !== null && + typeof value === `object` && + `toArray` in value && + Array.isArray(value.toArray) + ) { + return value.toArray as Array + } + return [] + } + + it(`updates a rendered array include after a child insert`, async () => { + const projects = createCollection( + mockSyncCollectionOptions({ + id: `includes-solid-array-projects`, + getKey: (project) => project.id, + initialData: [ + { id: 1, name: `Alpha` }, + { id: 2, name: `Beta` }, + ], + }), + ) + const issues = createCollection( + mockSyncCollectionOptions({ + id: `includes-solid-array-issues`, + getKey: (issue) => issue.id, + initialData: [ + { id: 10, projectId: 1, title: `Bug in Alpha` }, + { id: 20, projectId: 2, title: `Bug in Beta` }, + ], + }), + ) + + function TestComponent() { + const query = useLiveQuery((q) => + q.from({ project: projects }).select(({ project }) => ({ + id: project.id, + issueTitles: toArray( + q + .from({ issue: issues }) + .where(({ issue }) => eq(issue.projectId, project.id)) + .select(({ issue }) => ({ + id: issue.id, + title: issue.title, + })), + ), + })), + ) + + return ( + + {(project) => ( +

                            + {project.issueTitles.map((issue) => issue.title).join(`|`)} +

                            + )} + + ) + } + + const rendered = render(() => ) + await waitFor(() => { + expect(rendered.getByTestId(`project-1`).textContent).toBe(`Bug in Alpha`) + }) + + issues.utils.begin() + issues.utils.write({ + type: `insert`, + value: { id: 11, projectId: 1, title: `Feature for Alpha` }, + }) + issues.utils.commit() + + await waitFor(() => { + expect(rendered.getByTestId(`project-1`).textContent).toBe( + `Bug in Alpha|Feature for Alpha`, + ) + }) + }) + + it(`populates an initially empty collection include after its first child insert`, async () => { + const projects = createCollection( + mockSyncCollectionOptions({ + id: `includes-solid-empty-projects`, + getKey: (project) => project.id, + initialData: [{ id: 1, name: `Alpha` }], + }), + ) + const issues = createCollection( + mockSyncCollectionOptions({ + id: `includes-solid-empty-issues`, + getKey: (issue) => issue.id, + initialData: [], + }), + ) + + const rendered = renderHook(() => + useLiveQuery((q) => + q.from({ project: projects }).select(({ project }) => ({ + id: project.id, + issues: q + .from({ issue: issues }) + .where(({ issue }) => eq(issue.projectId, project.id)) + .select(({ issue }) => ({ + id: issue.id, + projectId: issue.projectId, + title: issue.title, + })), + })), + ), + ) + + await waitFor(() => { + expect(rendered.result.isReady).toBe(true) + expect(includedIssues(rendered.result()[0]?.issues)).toEqual([]) + }) + + issues.utils.begin() + issues.utils.write({ + type: `insert`, + value: { id: 10, projectId: 1, title: `Bug in Alpha` }, + }) + issues.utils.commit() + + await waitFor(() => { + expect( + includedIssues(rendered.result()[0]?.issues).map( + (issue) => issue.title, + ), + ).toEqual([`Bug in Alpha`]) + }) + }) +}) diff --git a/packages/svelte-db/CHANGELOG.md b/packages/svelte-db/CHANGELOG.md index a740729e17..d171da18f6 100644 --- a/packages/svelte-db/CHANGELOG.md +++ b/packages/svelte-db/CHANGELOG.md @@ -1,5 +1,191 @@ # @tanstack/svelte-db +## 0.3.8 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 0.3.7 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 0.3.6 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 0.3.5 + +### Patch Changes + +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 0.3.4 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 0.3.3 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 0.3.2 + +### Patch Changes + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 0.3.1 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 0.3.0 + +### Minor Changes + +- Add SSR through request-scoped `DbClient` instances, collection descriptors, ([#1564](https://github.com/TanStack/db/pull/1564)) + explicit collection-row hydration, live-query result snapshots, adapter sync + metadata, and React and Svelte descriptor resolution. + + React live queries now derive identity from structured query IR. Opaque queries + can provide `queryKey`; legacy dependency arrays and unkeyed opaque queries keep + working with development warnings until 1.0. + + Add TanStack Router integration that streams live queries discovered during a + Suspense render as pending promises which resolve to ordered result snapshots. + The browser starts normal source sync and atomically replaces the snapshot when + its live result is ready. + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 0.2.2 + +### Patch Changes + +- Update agent skills to match current APIs and behavior. ([#1696](https://github.com/TanStack/db/pull/1696)) + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 0.2.1 + +### Patch Changes + +- Add `useLiveInfiniteQuery` as a Vue binding over the shared live-query window controller. Align infinite-query behavior across React, Vue, and Svelte, including awaitable page fetches, safe page sizes, reactive page-depth preservation, ordered collection validation, shared input resolution, and shared-window cleanup. ([#1724](https://github.com/TanStack/db/pull/1724)) + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 0.2.0 + +### Minor Changes + +- Add `useLiveInfiniteQuery` as a Svelte binding over the shared live-query window controller. ([#1723](https://github.com/TanStack/db/pull/1723)) + +### Patch Changes + +- Add an internal shared live-query observer and migrate all five framework adapters to it ([#1642](https://github.com/TanStack/db/pull/1642)) + + Introduces `createLiveQueryObserver` in `@tanstack/db`: given a resolved live-query collection (or `null` for a disabled query) it owns the subscription lifecycle every adapter used to re-implement — change and status subscriptions, a snapshot with stable identity per state revision for wholesale consumers, and delivery of the raw `ChangeMessage[]` for granular consumers. React, Vue, Svelte, Solid, and Angular's live-query hooks now materialize from the observer instead of their own hand-rolled subscription/status/snapshot machinery, keeping each adapter's native reactivity and each adapter's data-loading policy (wholesale adapters subscribe without initial state; granular adapters seed from it). + + The observer is an **internal, unstable contract** for TanStack DB's official adapters — it is exported so the adapter packages can consume it, but it is not a public extension point yet and its API may change in any release. + + The migration also fixes several live-query lifecycle defects: status-only transitions (`error`, `cleaned-up`) now reach mounted consumers; snapshot identity is stable across unsubscribe/resubscribe and stays fresh while detached; dispatch is FIFO and non-reentrant with subscriptions identified by record rather than callback; disposing during the synchronous initial replay no longer leaks the collection subscription; subscribing after dispose throws instead of registering a dead listener; Solid guards its async resource continuations against superseded collections; and constructing an observer no longer activates sync. Observers activate on their first committed subscription unless an adapter has already started a pre-created collection supplied directly or returned from a callback. + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 0.1.94 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 0.1.93 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 0.1.92 + +### Patch Changes + +- Extract shared live-query adapter helpers into `@tanstack/db` ([#1641](https://github.com/TanStack/db/pull/1641)) + + Adds `isCollection`, `isSingleResultCollection`, and `getLiveQueryStatusFlags` to `@tanstack/db` and migrates all five framework adapters to use them. `isCollection` replaces the per-adapter duck-typing and Solid's `instanceof CollectionImpl` with one structural, multi-realm-safe guard (the `instanceof` form gave false negatives across dual-package boundaries). No behavior change; internal deduplication only. + +- fix(svelte-db): a disabled `useLiveQuery` (query callback returning `null`/`undefined`) no longer crashes ([#1637](https://github.com/TanStack/db/pull/1637)) + + The reactive-getter unwrapping (`toValue`) called the query callback and, when it returned `null`/`undefined` to signal a disabled query, passed the unwrapped `null` into `createLiveQueryCollection`, throwing in `getQueryIR`. A `null`/`undefined` resolved value is now treated as a disabled query (returns the `disabled` state) as it is in the other adapters. + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 0.1.91 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 0.1.90 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 0.1.89 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 0.1.88 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 0.1.87 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 0.1.86 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 0.1.85 ### Patch Changes diff --git a/packages/svelte-db/README.md b/packages/svelte-db/README.md index 13ae2725d0..6280b39747 100644 --- a/packages/svelte-db/README.md +++ b/packages/svelte-db/README.md @@ -1,3 +1,20 @@ +
                            + + + + TanStack Svelte DB + +
                            # @tanstack/svelte-db Svelte helpers for TanStack DB. See [TanStack/db](https://github.com/TanStack/db) for more details. diff --git a/packages/svelte-db/package.json b/packages/svelte-db/package.json index 79cfab6a16..c8afd4c181 100644 --- a/packages/svelte-db/package.json +++ b/packages/svelte-db/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/svelte-db", - "version": "0.1.85", + "version": "0.3.8", "description": "Svelte integration for @tanstack/db", "author": "Kyle Mathews", "license": "MIT", diff --git a/packages/svelte-db/skills/svelte-db/SKILL.md b/packages/svelte-db/skills/svelte-db/SKILL.md index 5976c43e7a..295103a302 100644 --- a/packages/svelte-db/skills/svelte-db/SKILL.md +++ b/packages/svelte-db/skills/svelte-db/SKILL.md @@ -10,7 +10,7 @@ description: > type: framework library: db framework: svelte -library_version: '0.6.0' +library_version: '0.6.17' requires: - db-core sources: diff --git a/packages/svelte-db/src/DbProvider.svelte b/packages/svelte-db/src/DbProvider.svelte new file mode 100644 index 0000000000..789d3ae0e1 --- /dev/null +++ b/packages/svelte-db/src/DbProvider.svelte @@ -0,0 +1,17 @@ + + +{@render children?.()} diff --git a/packages/svelte-db/src/db-context.ts b/packages/svelte-db/src/db-context.ts new file mode 100644 index 0000000000..f7f531318e --- /dev/null +++ b/packages/svelte-db/src/db-context.ts @@ -0,0 +1,26 @@ +import { getContext, setContext } from 'svelte' +import type { DbClient } from '@tanstack/db' + +const dbClientContext = Symbol.for(`@tanstack/svelte-db.DbClient`) +type DbClientContext = () => DbClient + +export function setDbClientContext(client: DbClientContext): DbClientContext { + return setContext(dbClientContext, client) +} + +export function useDbClient(): DbClient { + const client = useOptionalDbClient() + if (!client) { + throw new Error(`useDbClient must be used within a DbProvider.`) + } + return client +} + +export function useOptionalDbClient(): DbClient | undefined { + try { + return getContext(dbClientContext)?.() + } catch { + // Legacy helpers may be called from a rune root rather than a component. + return undefined + } +} diff --git a/packages/svelte-db/src/index.ts b/packages/svelte-db/src/index.ts index a07f98a64c..185d1bd5ac 100644 --- a/packages/svelte-db/src/index.ts +++ b/packages/svelte-db/src/index.ts @@ -1,5 +1,8 @@ // Re-export all public APIs export * from './useLiveQuery.svelte.js' +export * from './useLiveInfiniteQuery.svelte.js' +export { useDbClient, useOptionalDbClient } from './db-context.js' +export { default as DbProvider } from './DbProvider.svelte' // Re-export everything from @tanstack/db export * from '@tanstack/db' diff --git a/packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts b/packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts new file mode 100644 index 0000000000..51675dfa26 --- /dev/null +++ b/packages/svelte-db/src/useLiveInfiniteQuery.svelte.ts @@ -0,0 +1,267 @@ +import { + assertLiveQueryWindowManyResult, + compareLiveQueryWindowDependencies, + createLiveQueryCollection, + createLiveQueryWindowController, + fetchNextLiveQueryWindowPage, + getLiveQueryWindowCollectionWarning, + normalizeLiveQueryWindowPageSize, + resolveLiveQueryWindowInput, + shouldPreserveLiveQueryWindowPageCount, +} from '@tanstack/db' +import { tick, untrack } from 'svelte' +// Type-only: used in `ReturnType` below. +import type { + UseLiveQueryReturnWithCollection, + useLiveQuery, +} from './useLiveQuery.svelte.js' +import type { + Collection, + Context, + InferResultType, + InitialQueryBuilder, + NonSingleResult, + QueryBuilder, + UtilsRecord, +} from '@tanstack/db' + +const DEFAULT_GC_TIME_MS = 1 + +type MaybeGetter = T | (() => T) + +type InternalCollection = Collection + +type PreviousController = { + getSnapshot: () => { pages: ReadonlyArray> } +} + +type InfiniteQueryOptions = { + pageSize?: number + /** First result-page label, not a server cursor or remote offset. */ + initialPageParam?: number +} + +// Keep the generic parameter for existing typed config wrappers. +export type LiveInfiniteQueryConfig<_TRow> = InfiniteQueryOptions + +export type UseLiveInfiniteQueryConfig = + LiveInfiniteQueryConfig[number]> + +export type UseLiveInfiniteQueryReturn = Omit< + ReturnType>, + `data` +> & { + data: InferResultType + pages: Array[number]>> + pageParams: Array + fetchNextPage: () => Promise + hasNextPage: boolean + isFetchingNextPage: boolean + error: unknown +} + +export type UseLiveInfiniteQueryReturnWithCollection< + TResult extends object, + TKey extends string | number, + TUtils extends Record, +> = Omit< + UseLiveQueryReturnWithCollection>, + `data` +> & { + data: Array + pages: Array> + pageParams: Array + fetchNextPage: () => Promise + hasNextPage: boolean + isFetchingNextPage: boolean + error: unknown +} + +type EnabledLiveQueryReturn = ReturnType< + typeof useLiveQuery +> + +/** + * Create a Svelte-native reactive view over the shared live-query window + * controller. The query must include an `orderBy` clause. + */ +export function useLiveInfiniteQuery< + TResult extends object, + TKey extends string | number, + TUtils extends Record, +>( + liveQueryCollection: MaybeGetter< + Collection & NonSingleResult + >, + config: LiveInfiniteQueryConfig, +): UseLiveInfiniteQueryReturnWithCollection + +export function useLiveInfiniteQuery( + queryFn: (q: InitialQueryBuilder) => QueryBuilder, + config: UseLiveInfiniteQueryConfig, + deps?: Array<() => unknown>, +): UseLiveInfiniteQueryReturn + +export function useLiveInfiniteQuery( + queryFnOrCollection: unknown, + config: InfiniteQueryOptions, + deps: Array<() => unknown> = [], +): UseLiveInfiniteQueryReturn { + if (`getNextPageParam` in config) { + throw new Error( + `getNextPageParam is not supported by useLiveInfiniteQuery. Use an on-demand collection and fulfill meta.loadSubsetOptions in queryFn for server pagination.`, + ) + } + let validatedCollection: InternalCollection | null = null + let previousController: PreviousController | null = null + let previousInput: ReturnType< + typeof resolveLiveQueryWindowInput + > | null = null + let previousDependencies: Array | null = null + let previousPageSize: number | null = null + let previousInitialPageParam: number | null = null + + const pageSize = $derived(normalizeLiveQueryWindowPageSize(config.pageSize)) + const initialPageParam = $derived(config.initialPageParam ?? 0) + + const controller = $derived.by(() => { + const dependencies = deps.map((dependency) => dependency()) + + const input = resolveLiveQueryWindowInput(queryFnOrCollection) + const dependencyComparison = compareLiveQueryWindowDependencies( + previousDependencies, + dependencies, + ) + const dependenciesChanged = dependencyComparison.changed + const dependenciesStructurallyEqual = dependencyComparison.structurallyEqual + const pageShapeChanged = + previousPageSize !== pageSize || + previousInitialPageParam !== initialPageParam + const sameCollection = + input.kind === `collection` && + previousInput?.kind === `collection` && + previousInput.collection === input.collection + const canPreservePageCount = shouldPreserveLiveQueryWindowPageCount({ + hasPreviousController: previousController !== null, + previousInputKind: previousInput?.kind, + inputKind: input.kind, + sameCollection, + dependenciesChanged, + dependenciesStructurallyEqual, + pageShapeChanged, + }) + const previousPageCount = previousController + ? Math.max(1, previousController.getSnapshot().pages.length) + : 1 + const initialPageCount = canPreservePageCount ? previousPageCount : 1 + + previousInput = input + previousDependencies = [...dependencies] + previousPageSize = pageSize + previousInitialPageParam = initialPageParam + + if (input.kind === `collection`) { + const collection = input.collection + const warning = getLiveQueryWindowCollectionWarning( + collection, + pageSize + 1, + ) + + if (validatedCollection !== collection) { + validatedCollection = collection + if (warning) console.warn(warning) + } + const currentController = createLiveQueryWindowController(collection, { + pageSize, + initialPageParam, + initialPageCount, + }) + previousController = currentController + return currentController + } + + const collection = createLiveQueryCollection({ + query: input.query.limit(pageSize + 1).offset(0), + startSync: false, + gcTime: DEFAULT_GC_TIME_MS, + }) + assertLiveQueryWindowManyResult(collection) + const currentController = createLiveQueryWindowController(collection, { + pageSize, + initialPageParam, + initialPageCount, + }) + previousController = currentController + return currentController + }) + + let snapshot = $state.raw(untrack(() => controller.getSnapshot())) + + $effect(() => { + const currentController = controller + snapshot = currentController.getSnapshot() + const unsubscribe = currentController.subscribe(() => { + snapshot = currentController.getSnapshot() + }) + snapshot = currentController.getSnapshot() + + return () => { + unsubscribe() + currentController.dispose() + } + }) + + const fetchNextPage = async () => { + // A dependency can invalidate the derived controller before Svelte runs the + // effect that subscribes it. Queue the imperative call until that handoff + // has completed so it cannot target an inactive controller. + await tick() + await fetchNextLiveQueryWindowPage(controller) + } + + return { + get state() { + return snapshot.state as EnabledLiveQueryReturn[`state`] + }, + get data() { + return snapshot.data as InferResultType + }, + get collection() { + return snapshot.collection as EnabledLiveQueryReturn[`collection`] + }, + get status() { + return snapshot.status as EnabledLiveQueryReturn[`status`] + }, + get isLoading() { + return snapshot.isLoading + }, + get isReady() { + return snapshot.isReady + }, + get isIdle() { + return snapshot.isIdle + }, + get isError() { + return snapshot.isError + }, + get isCleanedUp() { + return snapshot.isCleanedUp + }, + get pages() { + return snapshot.pages as Array[number]>> + }, + get pageParams() { + return snapshot.pageParams as Array + }, + get hasNextPage() { + return snapshot.hasNextPage + }, + get isFetchingNextPage() { + return snapshot.isFetchingNextPage + }, + get error() { + return snapshot.error + }, + fetchNextPage, + } +} diff --git a/packages/svelte-db/src/useLiveQuery.svelte.ts b/packages/svelte-db/src/useLiveQuery.svelte.ts index 76dedd3c78..5af53f6b55 100644 --- a/packages/svelte-db/src/useLiveQuery.svelte.ts +++ b/packages/svelte-db/src/useLiveQuery.svelte.ts @@ -2,17 +2,31 @@ import { untrack } from 'svelte' // eslint-disable-next-line import/no-duplicates -- See https://github.com/un-ts/eslint-plugin-import-x/issues/308 import { SvelteMap } from 'svelte/reactivity' -import { BaseQueryBuilder, createLiveQueryCollection } from '@tanstack/db' +import { + BaseQueryBuilder, + UnhashableQueryIRError, + createLiveQueryCollection, + createLiveQueryObserver, + getLiveQueryHash, + getStableValueHash, + isCollection, + isSingleResultCollection, + prepareLiveQueryValue, +} from '@tanstack/db' +import { useOptionalDbClient } from './db-context.js' import type { ChangeMessage, Collection, - CollectionConfigSingleRowOption, CollectionStatus, Context, + DbClient, + DeferredLiveQueryCollections, GetResult, InferResultType, InitialQueryBuilder, LiveQueryCollectionConfig, + LiveQueryKey, + LiveQueryObserver, NonSingleResult, QueryBuilder, SingleResult, @@ -61,6 +75,12 @@ export interface UseLiveQueryReturnWithCollection< type MaybeGetter = T | (() => T) +export type UseLiveQueryConfig = + LiveQueryCollectionConfig & { + queryKey?: MaybeGetter + client?: DbClient + } + function toValue(value: MaybeGetter): T { if (typeof value === `function`) { return (value as () => T)() @@ -212,7 +232,7 @@ export function useLiveQuery( */ // Overload 2: Accept config object export function useLiveQuery( - config: LiveQueryCollectionConfig, + config: UseLiveQueryConfig, deps?: Array<() => unknown>, ): UseLiveQueryReturn, InferResultType> @@ -286,7 +306,9 @@ export function useLiveQuery( configOrQueryOrCollection: any, deps: Array<() => unknown> = [], ): UseLiveQueryReturn | UseLiveQueryReturnWithCollection { - const collection = $derived.by(() => { + const contextDbClient = useOptionalDbClient() + + const resolved = $derived.by(() => { // First check if the original parameter might be a getter // by seeing if toValue returns something different than the original let unwrappedParam = configOrQueryOrCollection @@ -301,14 +323,13 @@ export function useLiveQuery( } // Check if it's already a collection by checking for specific collection methods - const isCollection = - unwrappedParam && - typeof unwrappedParam === `object` && - typeof unwrappedParam.subscribeChanges === `function` && - typeof unwrappedParam.startSyncImmediate === `function` && - typeof unwrappedParam.id === `string` + const inputIsCollection = isCollection(unwrappedParam) + const dbClient = inputIsCollection + ? contextDbClient + : ((unwrappedParam as { client?: DbClient } | null)?.client ?? + contextDbClient) - if (isCollection) { + if (inputIsCollection) { // Warn when passing a collection directly with on-demand sync mode // In on-demand mode, data is only loaded when queries with predicates request it // Passing the collection directly doesn't provide any predicates, so no data loads @@ -328,143 +349,153 @@ export function useLiveQuery( if (unwrappedParam.status === `idle`) { unwrappedParam.startSyncImmediate() } - return unwrappedParam + return { + collection: unwrappedParam, + client: dbClient, + queryHash: getStableValueHash( + [`collection`, unwrappedParam.id], + `queryKey`, + ), + resumeDeferredCollections: () => {}, + } } // Reference deps to make computed reactive to them - deps.forEach((dep) => toValue(dep)) - - // Ensure we always start sync for Svelte helpers - if (typeof unwrappedParam === `function`) { - // Check if query function returns null/undefined (disabled query) - const queryBuilder = new BaseQueryBuilder() as InitialQueryBuilder - const result = unwrappedParam(queryBuilder) - - if (result === undefined || result === null) { - // Disabled query - return null - return null - } + const dependencyValues = deps.map((dep) => toValue(dep)) + const deferredCollections: DeferredLiveQueryCollections = new Set() + const preparedValue = prepareLiveQueryValue( + unwrappedParam, + dbClient, + deferredCollections, + ) + const configuredQueryKey = ( + unwrappedParam as { queryKey?: MaybeGetter } | null + )?.queryKey + const queryKey = configuredQueryKey + ? toValue(configuredQueryKey) + : undefined + + let queryHash: string | undefined + try { + queryHash = + deps.length > 0 && !queryKey + ? getStableValueHash( + [`deps`, dependencyValues, getLiveQueryHash(preparedValue)], + `queryKey`, + ) + : getLiveQueryHash(preparedValue, queryKey) + } catch (error) { + if (!(error instanceof UnhashableQueryIRError)) throw error + if (queryKey !== undefined) throw error + } - return createLiveQueryCollection({ - query: unwrappedParam, + let collection: Collection | null + if (preparedValue === undefined || preparedValue === null) { + collection = null + } else if (isCollection(preparedValue)) { + collection = preparedValue + } else if (preparedValue instanceof BaseQueryBuilder) { + collection = createLiveQueryCollection({ + query: preparedValue, startSync: true, }) } else { - return createLiveQueryCollection({ - ...unwrappedParam, + collection = createLiveQueryCollection({ + ...(preparedValue as LiveQueryCollectionConfig), startSync: true, }) } + + return { + collection, + client: dbClient, + queryHash, + resumeDeferredCollections: () => { + for (const deferredCollection of deferredCollections) { + deferredCollection._resumeSyncStart() + } + deferredCollections.clear() + }, + } + }) + + let currentResolved = untrack(() => resolved) + let currentObserver = createLiveQueryObserver(currentResolved.collection, { + client: currentResolved.client, + queryHash: currentResolved.queryHash, + onPreload: currentResolved.resumeDeferredCollections, }) + const initialSnapshot = currentObserver.getServerSnapshot() // Reactive state that gets updated granularly through change events - const state = new SvelteMap() + const state = new SvelteMap(initialSnapshot.state ?? []) // Reactive data array that maintains sorted order - let internalData = $state>([]) + let internalData = $state>( + Array.from(initialSnapshot.state?.values() ?? []), + ) // Track collection status reactively - let status = $state(collection ? collection.status : (`disabled` as const)) + let status = $state(initialSnapshot.status) - // Helper to sync data array from collection in correct order - const syncDataFromCollection = ( - currentCollection: Collection, + const syncFromObserver = ( + observer: LiveQueryObserver, + changes?: Array>, ) => { + const snapshot = observer.getSnapshot() + status = snapshot.status as CollectionStatus untrack(() => { - internalData = [] - internalData.push(...Array.from(currentCollection.values())) + if (changes && changes.length > 0) { + for (const change of changes) { + switch (change.type) { + case `insert`: + case `update`: + state.set(change.key, change.value) + break + case `delete`: + state.delete(change.key) + break + } + } + } else { + state.clear() + for (const [key, value] of snapshot.state ?? []) { + state.set(key, value) + } + } + internalData = Array.from(snapshot.state?.values() ?? []) }) } - // Track current unsubscribe function - let currentUnsubscribe: (() => void) | null = null - // Watch for collection changes and subscribe to updates $effect(() => { - const currentCollection = collection - - // Handle null collection (disabled query) - if (!currentCollection) { - status = `disabled` as const - untrack(() => { - state.clear() - internalData = [] + const nextResolved = resolved + + if (nextResolved !== currentResolved) { + currentObserver.dispose() + currentResolved = nextResolved + currentObserver = createLiveQueryObserver(nextResolved.collection, { + client: nextResolved.client, + queryHash: nextResolved.queryHash, + onPreload: nextResolved.resumeDeferredCollections, }) - if (currentUnsubscribe) { - currentUnsubscribe() - currentUnsubscribe = null - } - return + syncFromObserver(currentObserver) } - // Update status state whenever the effect runs - status = currentCollection.status - - // Clean up previous subscription - if (currentUnsubscribe) { - currentUnsubscribe() - } - - // Initialize state with current collection data - untrack(() => { - state.clear() - for (const [key, value] of currentCollection.entries()) { - state.set(key, value) - } - }) - - // Initialize data array in correct order - syncDataFromCollection(currentCollection) + const observer = currentObserver - // Listen for the first ready event to catch status transitions - // that might not trigger change events (fixes async status transition bug) - currentCollection.onFirstReady(() => { - // Update status directly - Svelte's reactivity system handles the update automatically - // Note: We cannot use flushSync here as it's disallowed inside effects in async mode - status = currentCollection.status - }) - - // Subscribe to collection changes with granular updates - const subscription = currentCollection.subscribeChanges( - (changes: Array>) => { - // Apply each change individually to the reactive state - untrack(() => { - for (const change of changes) { - switch (change.type) { - case `insert`: - case `update`: - state.set(change.key, change.value) - break - case `delete`: - state.delete(change.key) - break - } - } - }) - - // Update the data array to maintain sorted order - syncDataFromCollection(currentCollection) - // Update status state on every change - status = currentCollection.status - }, - { - includeInitialState: true, + const unsubscribe = observer.subscribe( + (changes: Array> | undefined) => { + syncFromObserver(observer, changes) }, ) - - currentUnsubscribe = subscription.unsubscribe.bind(subscription) - - // Preload collection data if not already started - if (currentCollection.status === `idle`) { - currentCollection.preload().catch(console.error) - } + currentResolved.resumeDeferredCollections() + syncFromObserver(observer) // Cleanup when effect is invalidated return () => { - if (currentUnsubscribe) { - currentUnsubscribe() - currentUnsubscribe = null - } + unsubscribe() + if (observer === currentObserver) observer.dispose() } }) @@ -473,25 +504,17 @@ export function useLiveQuery( return state }, get data() { - const currentCollection = collection - if (currentCollection) { - const config = - currentCollection.config as CollectionConfigSingleRowOption< - any, - any, - any - > - if (config.singleResult) { - return internalData[0] - } + const currentCollection = resolved.collection + if (currentCollection && isSingleResultCollection(currentCollection)) { + return internalData[0] } return internalData }, get collection() { - return collection + return resolved.collection }, get status() { - return status + return status as CollectionStatus }, get isLoading() { return status === `loading` diff --git a/packages/svelte-db/tests/SsrDbApp.svelte b/packages/svelte-db/tests/SsrDbApp.svelte new file mode 100644 index 0000000000..adc6823847 --- /dev/null +++ b/packages/svelte-db/tests/SsrDbApp.svelte @@ -0,0 +1,23 @@ + + + + + diff --git a/packages/svelte-db/tests/SsrDbQuery.svelte b/packages/svelte-db/tests/SsrDbQuery.svelte new file mode 100644 index 0000000000..6e0c9a58da --- /dev/null +++ b/packages/svelte-db/tests/SsrDbQuery.svelte @@ -0,0 +1,30 @@ + + + + {#each people.data as person (person.id)} + {person.name} + {/each} + diff --git a/packages/svelte-db/tests/conformance.svelte.test.ts b/packages/svelte-db/tests/conformance.svelte.test.ts new file mode 100644 index 0000000000..b68d412d16 --- /dev/null +++ b/packages/svelte-db/tests/conformance.svelte.test.ts @@ -0,0 +1,218 @@ +/** + * Svelte driver for the shared live-query conformance suite. + * + * Svelte 5 runes: each mount runs inside a persistent `$effect.root` so the + * internal `$effect` keeps updating rune state after mount; `unmount` disposes + * the root. Reads happen after `flushSync()`. Realm-sensitive pieces come from + * Svelte's `@tanstack/db`. + * + * `knownGaps` is populated empirically from the run below. + */ +import { + coalesce, + count, + createCollection, + createLiveQueryCollection, + createOptimisticAction, + eq, + gt, + sum, +} from '@tanstack/db' +import { flushSync } from 'svelte' +import { + mockSyncCollectionOptions, + mockSyncCollectionOptionsNoInitialState, +} from '../../db/tests/utils' +import { useLiveQuery } from '../src/useLiveQuery.svelte.js' +import { runSuite } from '../../db/tests/conformance/suite' +import type { + ConformanceResult, + ControllableHandle, + DeferredSourceHandle, + LiveQueryDriver, + LiveQueryHandle, + QueryBuild, + SourceHandle, +} from '../../db/tests/conformance/contract' + +let sourceSeq = 0 + +function writer(collection: any) { + return (type: `insert` | `update` | `delete`, value: T) => { + collection.utils.begin() + collection.utils.write({ type, value }) + collection.utils.commit() + } +} + +function makeSource( + initialData: ReadonlyArray, +): SourceHandle { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `conformance-svelte-${sourceSeq++}`, + getKey: (r) => r.id, + initialData: [...initialData], + }), + ) + const write = writer(collection) + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + } +} + +function makeDeferredSource< + T extends { id: string }, +>(): DeferredSourceHandle { + const collection = createCollection( + mockSyncCollectionOptionsNoInitialState({ + id: `conformance-svelte-${sourceSeq++}`, + getKey: (r) => r.id, + }), + ) + collection.startSyncImmediate() + const write = writer(collection) + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + emit: (rows) => { + collection.utils.begin() + rows.forEach((value) => collection.utils.write({ type: `insert`, value })) + collection.utils.commit() + }, + markReady: () => collection.utils.markReady(), + } +} + +function makePrecreated(build: QueryBuild, opts?: { startSync?: boolean }) { + const collection = createLiveQueryCollection({ + query: build as any, + startSync: opts?.startSync ?? true, + }) + return { collection } +} + +function makeErrorSource() { + const collection = createCollection<{ id: string }>({ + id: `conformance-svelte-err-${sourceSeq++}`, + getKey: (r) => r.id, + startSync: false, + sync: { + sync: () => { + throw new Error(`conformance: sync failure`) + }, + }, + }) + try { + collection.startSyncImmediate() + } catch { + // expected: engine catches the sync error and sets status to `error` + } + return { collection } +} + +async function settle() { + flushSync() + await new Promise((resolve) => setTimeout(resolve, 10)) + flushSync() +} + +function makeHandle(getQuery: () => any, dispose: () => void): LiveQueryHandle { + return { + current(): ConformanceResult { + const query = getQuery() + return { + data: query?.data, + state: query?.state, + status: query?.status ?? `idle`, + isReady: Boolean(query?.isReady), + isError: Boolean(query?.isError), + // svelte-db exposes no `isEnabled`; derive it from status (status-derived). + isEnabled: query?.status !== `disabled`, + } + }, + flush: settle, + async apply(fn: () => void) { + fn() + await settle() + }, + unmount() { + dispose() + }, + } +} + +function mount(build: QueryBuild) { + let query: any + const dispose = $effect.root(() => { + query = useLiveQuery(build as any) + }) + return makeHandle(() => query, dispose) +} + +function mountCollection(collection: any) { + let query: any + const dispose = $effect.root(() => { + query = useLiveQuery(collection) + }) + return makeHandle(() => query, dispose) +} + +function mountConfig(build: QueryBuild) { + let query: any + const dispose = $effect.root(() => { + query = useLiveQuery({ query: build } as any) + }) + return makeHandle(() => query, dispose) +} + +function mountDisabled() { + // Svelte's disabled convention: the query callback returns null. + let query: any + const dispose = $effect.root(() => { + query = useLiveQuery(() => null as any) + }) + return makeHandle(() => query, dispose) +} + +function mountControllable

                            ( + build: (q: any, param: P) => any, + initial: P, +): ControllableHandle

                            { + let param = $state(initial) + let query: any + const dispose = $effect.root(() => { + query = useLiveQuery((q: any) => build(q, param), [() => param]) + }) + const handle = makeHandle(() => query, dispose) + return { + ...handle, + async setParam(next: P) { + param = next + await settle() + }, + } +} + +const svelteDriver: LiveQueryDriver = { + name: `svelte`, + ops: { eq, gt, count, sum, coalesce, createOptimisticAction }, + makeSource, + makeDeferredSource, + makePrecreated, + makeErrorSource, + mount, + mountControllable, + mountCollection, + mountConfig, + mountDisabled, + knownGaps: [], + features: { serverSnapshot: false, suspense: false }, +} + +runSuite(svelteDriver) diff --git a/packages/svelte-db/tests/hydration.svelte.test.ts b/packages/svelte-db/tests/hydration.svelte.test.ts new file mode 100644 index 0000000000..37c37a7534 --- /dev/null +++ b/packages/svelte-db/tests/hydration.svelte.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, it, vi } from 'vitest' +import { flushSync } from 'svelte' +import { DbClient } from '@tanstack/db' +import { useLiveQuery } from '../src/useLiveQuery.svelte.js' +import { createPeopleDescriptor, peopleQuery } from './ssr-test-utils.js' + +describe(`Svelte hydration`, () => { + it(`keeps the hydrated result until the browser source is authoritative`, async () => { + const { descriptor, resolveBrowserLoad } = createPeopleDescriptor() + const serverClient = new DbClient({ runtime: `server` }) + await serverClient.preloadLiveQuery(peopleQuery(descriptor)) + const dehydrated = serverClient.dehydrate({ + shouldDehydrateCollection: () => false, + shouldDehydrateLiveQuery: () => true, + }) + + const browserClient = new DbClient({ runtime: `browser` }) + browserClient.hydrate(dehydrated) + const createQuery = () => + useLiveQuery({ + ...peopleQuery(descriptor), + client: browserClient, + }) + let query!: ReturnType + const dispose = $effect.root(() => { + query = createQuery() + }) + + flushSync() + expect(query.data).toEqual([ + expect.objectContaining({ id: `server`, name: `Server snapshot` }), + ]) + + resolveBrowserLoad() + await vi.waitFor(() => { + flushSync() + expect(query.data).toEqual([ + expect.objectContaining({ id: `browser`, name: `Browser source` }), + ]) + }) + dispose() + }) +}) diff --git a/packages/svelte-db/tests/infinite-query-conformance.svelte.test.ts b/packages/svelte-db/tests/infinite-query-conformance.svelte.test.ts new file mode 100644 index 0000000000..0e3718838e --- /dev/null +++ b/packages/svelte-db/tests/infinite-query-conformance.svelte.test.ts @@ -0,0 +1,206 @@ +/** Svelte driver for the shared infinite-query conformance suite. */ +import { + BTreeIndex, + createCollection, + createLiveQueryCollection, + gt, +} from '@tanstack/db' +import { flushSync } from 'svelte' +import { mockSyncCollectionOptions } from '../../db/tests/utils' +import { runInfiniteQuerySuite } from '../../db/tests/conformance/infinite-suite' +import { makeInfiniteOnDemandSource } from '../../db/tests/conformance/infinite-on-demand' +import { useLiveInfiniteQuery } from '../src/useLiveInfiniteQuery.svelte.js' +import type { + InfiniteQueryConfig, + InfiniteQueryDriver, + InfiniteQueryHandle, +} from '../../db/tests/conformance/infinite-contract' +import type { + QueryBuild, + SourceHandle, +} from '../../db/tests/conformance/contract' + +let sourceSequence = 0 + +function makeSource( + initialData: ReadonlyArray, +): SourceHandle { + const collection = createCollection( + mockSyncCollectionOptions({ + autoIndex: `eager`, + id: `infinite-conformance-svelte-${sourceSequence++}`, + getKey: (row) => row.id, + initialData: [...initialData], + }), + ) + const write = (type: `insert` | `update` | `delete`, value: T) => { + collection.utils.begin() + collection.utils.write({ type, value }) + collection.utils.commit() + } + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + } +} + +function makePrecreated(build: QueryBuild) { + return { + collection: createLiveQueryCollection({ query: build as any }), + } +} + +async function settle(): Promise { + flushSync() + await new Promise((resolve) => setTimeout(resolve, 0)) + flushSync() +} + +function makeHandle( + getResult: () => any, + dispose: () => void, +): InfiniteQueryHandle { + return { + current() { + const result = getResult() + return { + data: result.data, + pages: result.pages, + pageParams: result.pageParams, + hasNextPage: result.hasNextPage, + isFetchingNextPage: result.isFetchingNextPage, + error: result.error, + status: result.status, + collection: result.collection, + } + }, + fetchNextPage: () => getResult().fetchNextPage(), + flush: settle, + async apply(fn) { + fn() + await settle() + }, + unmount: dispose, + } +} + +function mount(build: QueryBuild, config: InfiniteQueryConfig = {}) { + let result: any + const dispose = $effect.root(() => { + result = useLiveInfiniteQuery(build as any, config as any) + }) + flushSync() + return makeHandle(() => result, dispose) +} + +function mountControllable

                            ( + build: (q: any, param: P) => any, + initial: P, + config: InfiniteQueryConfig = {}, +) { + let result: any + let setParam!: (next: P) => void + const dispose = $effect.root(() => { + // The contract treats dependency values as caller-owned identities. Avoid + // deep proxying, which rewrites circular object identity before it reaches + // the adapter. + let param = $state.raw(initial) + result = useLiveInfiniteQuery((q: any) => build(q, param), config as any, [ + () => param, + ]) + setParam = (next) => { + param = next + } + }) + flushSync() + const handle = makeHandle(() => result, dispose) + return { ...handle, setParamSync: setParam } +} + +function mountCollection(collection: any, config: InfiniteQueryConfig = {}) { + let result: any + const dispose = $effect.root(() => { + result = useLiveInfiniteQuery(collection, config as any) + }) + flushSync() + return makeHandle(() => result, dispose) +} + +function mountCollectionControllable( + initial: any, + config: InfiniteQueryConfig = {}, +) { + let result: any + let replaceCollection!: (next: any) => void + const dispose = $effect.root(() => { + let collection = $state(initial) + result = useLiveInfiniteQuery(() => collection, config as any) + replaceCollection = (next) => { + collection = next + } + }) + flushSync() + const handle = makeHandle(() => result, dispose) + return { ...handle, replaceCollectionSync: replaceCollection } +} + +function mountConfigControllable( + build: QueryBuild, + initial: InfiniteQueryConfig, +) { + let result: any + let setConfig!: (next: InfiniteQueryConfig) => void + const dispose = $effect.root(() => { + const config = $state({ ...initial }) + result = useLiveInfiniteQuery(build as any, config as any) + setConfig = (next) => { + Object.assign(config, next) + } + }) + flushSync() + const handle = makeHandle(() => result, dispose) + return { ...handle, setConfigSync: setConfig } +} + +function mountInputControllable( + collection: any, + build: QueryBuild, + config: InfiniteQueryConfig = {}, +) { + let result: any + let setInputKind!: (next: `collection` | `query`) => void + const dispose = $effect.root(() => { + let kind = $state<`collection` | `query`>(`collection`) + result = useLiveInfiniteQuery( + (q: any) => (kind === `collection` ? collection : build(q)), + config as any, + [() => kind], + ) + setInputKind = (next) => { + kind = next + } + }) + flushSync() + const handle = makeHandle(() => result, dispose) + return { ...handle, setInputKindSync: setInputKind } +} + +const svelteInfiniteDriver: InfiniteQueryDriver = { + name: `svelte`, + gt, + makeSource, + makeOnDemandSource: (data, delay) => + makeInfiniteOnDemandSource({ createCollection, BTreeIndex }, data, delay), + makePrecreated, + mount, + mountControllable, + mountCollection, + mountCollectionControllable, + mountConfigControllable, + mountInputControllable, + knownGaps: [], +} + +runInfiniteQuerySuite(svelteInfiniteDriver) diff --git a/packages/svelte-db/tests/ssr-test-utils.ts b/packages/svelte-db/tests/ssr-test-utils.ts new file mode 100644 index 0000000000..d1dfdab2fd --- /dev/null +++ b/packages/svelte-db/tests/ssr-test-utils.ts @@ -0,0 +1,68 @@ +import { collectionOptions } from '@tanstack/db' +import type { InitialQueryBuilder } from '@tanstack/db' + +export type Person = { + id: string + name: string + sourcePayload: string +} + +export const serverPerson: Person = { + id: `server`, + name: `Server snapshot`, + sourcePayload: `SOURCE_ONLY_SERVER_PAYLOAD`, +} + +export const browserPerson: Person = { + id: `browser`, + name: `Browser source`, + sourcePayload: `SOURCE_ONLY_BROWSER_PAYLOAD`, +} + +export function createPeopleDescriptor() { + let resolveBrowserLoad: (() => void) | undefined + const descriptor = collectionOptions(`svelte-ssr-people`, (client) => { + const runtime = client.requireDependency<`server` | `browser`>(`runtime`) + + return { + id: `svelte-ssr-people`, + getKey: (person: Person) => person.id, + syncMode: `on-demand` as const, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: async () => { + if (runtime === `browser`) { + await new Promise((resolve) => { + resolveBrowserLoad = resolve + }) + } + begin({ immediate: true }) + write({ + type: `insert`, + value: runtime === `server` ? serverPerson : browserPerson, + }) + commit() + }, + } + }, + }, + } + }) + + return { + descriptor, + resolveBrowserLoad: () => resolveBrowserLoad?.(), + } +} + +export const peopleQuery = ( + descriptor: ReturnType[`descriptor`], +) => ({ + query: (q: InitialQueryBuilder) => + q.from({ people: descriptor }).select(({ people }) => ({ + id: people.id, + name: people.name, + })), +}) diff --git a/packages/svelte-db/tests/ssr.svelte.test.ts b/packages/svelte-db/tests/ssr.svelte.test.ts new file mode 100644 index 0000000000..6af046c49a --- /dev/null +++ b/packages/svelte-db/tests/ssr.svelte.test.ts @@ -0,0 +1,34 @@ +// @vitest-environment node + +import { describe, expect, it } from 'vitest' +import { render } from 'svelte/server' +import { DbClient } from '@tanstack/db' +import SsrDbApp from './SsrDbApp.svelte' +import { + createPeopleDescriptor, + peopleQuery, + serverPerson, +} from './ssr-test-utils.js' + +describe(`Svelte SSR`, () => { + it(`renders a hydrated live-query result without hydrating source rows`, async () => { + const { descriptor } = createPeopleDescriptor() + const serverClient = new DbClient({ runtime: `server` }) + await serverClient.preloadLiveQuery(peopleQuery(descriptor)) + const dehydrated = serverClient.dehydrate({ + shouldDehydrateCollection: () => false, + shouldDehydrateLiveQuery: () => true, + }) + expect(dehydrated.collections).toEqual([]) + + const browserClient = new DbClient({ runtime: `browser` }) + browserClient.hydrate(dehydrated) + const { body } = render(SsrDbApp, { + props: { client: browserClient, descriptor }, + }) + + expect(body).toContain(`Server snapshot`) + expect(body).toContain(`data-status="ready"`) + expect(body).not.toContain(serverPerson.sourcePayload) + }) +}) diff --git a/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts b/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts new file mode 100644 index 0000000000..12f847bd46 --- /dev/null +++ b/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts @@ -0,0 +1,139 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { flushSync } from 'svelte' +import { createCollection, createLiveQueryCollection } from '@tanstack/db' +import { useLiveInfiniteQuery } from '../src/useLiveInfiniteQuery.svelte.js' +import { mockSyncCollectionOptions } from '../../db/tests/utils' + +type Post = { + id: string + title: string + createdAt: number +} + +function createPosts(count: number): Array { + return Array.from({ length: count }, (_, index) => ({ + id: String(index + 1), + title: `Post ${index + 1}`, + createdAt: count - index, + })) +} + +function createPostsCollection(id: string, count: number) { + return createCollection( + mockSyncCollectionOptions({ + autoIndex: `eager`, + id, + getKey: (post) => post.id, + initialData: createPosts(count), + }), + ) +} + +function createPostsLiveQuery(posts: ReturnType) { + return createLiveQueryCollection({ + query: (q) => + q + .from({ posts }) + .orderBy(({ posts: post }) => post.createdAt, `desc`) + .limit(4), + }) +} + +function usePostsCollectionInfiniteQuery( + getCollection: () => ReturnType, +) { + return useLiveInfiniteQuery(getCollection, { pageSize: 3 }) +} + +describe(`useLiveInfiniteQuery`, () => { + it(`rejects a server-page callback before constructing a query`, () => { + const queryFn = vi.fn(() => { + throw new Error(`query must not be constructed`) + }) + const config = { pageSize: 2, getNextPageParam: () => 1 } + const stop = $effect.root(() => { + expect(() => useLiveInfiniteQuery(queryFn, config)).toThrow( + `getNextPageParam is not supported`, + ) + expect(queryFn).not.toHaveBeenCalled() + }) + stop() + }) + + let cleanup: (() => void) | undefined + + afterEach(() => { + cleanup?.() + cleanup = undefined + vi.restoreAllMocks() + }) + + it(`accepts a reactive getter for a pre-created ordered collection`, async () => { + const posts = createPostsCollection(`svelte-infinite-precreated`, 7) + const livePosts = createLiveQueryCollection({ + query: (q) => + q + .from({ posts }) + .orderBy(({ posts: post }) => post.createdAt, `desc`) + .limit(2) + .offset(1), + }) + await livePosts.preload() + const warning = vi.spyOn(console, `warn`).mockImplementation(() => {}) + + let query!: ReturnType + cleanup = $effect.root(() => { + query = useLiveInfiniteQuery(() => livePosts, { + pageSize: 3, + }) + }) + flushSync() + // flushSync starts the subscription but does not settle its window load. + await vi.waitFor(() => + expect(livePosts.utils.getWindow()).toEqual({ offset: 0, limit: 4 }), + ) + flushSync() + + expect(query.collection).toBe(livePosts) + expect(query.data.map((post) => post.id)).toEqual([`1`, `2`, `3`]) + expect(query.state.get(`1`)?.title).toBe(`Post 1`) + expect(query.hasNextPage).toBe(true) + expect(warning).toHaveBeenCalledOnce() + expect(livePosts.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) + }) + + it(`resets to the first page when a collection getter changes`, async () => { + const firstPosts = createPostsCollection(`svelte-infinite-swap-first`, 8) + const secondPosts = createPostsCollection(`svelte-infinite-swap-second`, 4) + const firstQuery = createPostsLiveQuery(firstPosts) + const secondQuery = createPostsLiveQuery(secondPosts) + await Promise.all([firstQuery.preload(), secondQuery.preload()]) + + let query: ReturnType | undefined + let replaceCollection: + | ((collection: typeof secondQuery) => void) + | undefined + cleanup = $effect.root(() => { + let selectedQuery = $state(firstQuery) + query = usePostsCollectionInfiniteQuery(() => selectedQuery) + replaceCollection = (collection) => { + selectedQuery = collection + } + }) + flushSync() + if (!query || !replaceCollection) { + throw new Error(`Failed to mount infinite query`) + } + + await query.fetchNextPage() + flushSync() + expect(query.pages).toHaveLength(2) + + replaceCollection(secondQuery) + flushSync() + + expect(query.collection).toBe(secondQuery) + expect(query.pages).toHaveLength(1) + expect(query.data.map((post) => post.createdAt)).toEqual([4, 3, 2]) + }) +}) diff --git a/packages/svelte-db/tests/useLiveInfiniteQuery.test-d.ts b/packages/svelte-db/tests/useLiveInfiniteQuery.test-d.ts new file mode 100644 index 0000000000..edc1f5e075 --- /dev/null +++ b/packages/svelte-db/tests/useLiveInfiniteQuery.test-d.ts @@ -0,0 +1,38 @@ +import { describe, expectTypeOf, it } from 'vitest' +import { useLiveInfiniteQuery } from '../src/useLiveInfiniteQuery.svelte.js' +import type { + UseLiveInfiniteQueryConfig, + UseLiveInfiniteQueryReturn, +} from '../src/useLiveInfiniteQuery.svelte.js' +import type { Context, InitialQueryBuilder } from '@tanstack/db' + +describe(`useLiveInfiniteQuery type assertions`, () => { + it(`does not advertise a server-page callback`, () => { + expectTypeOf< + Extract, `getNextPageParam`> + >().toEqualTypeOf() + }) + + it(`keeps legacy generic wrappers source-compatible`, () => { + function acceptsContext( + _config: UseLiveInfiniteQueryConfig, + _result: UseLiveInfiniteQueryReturn, + ): void {} + + void acceptsContext + }) + + it(`preserves the awaitable fetch callback`, () => { + expectTypeOf< + UseLiveInfiniteQueryReturn[`fetchNextPage`] + >().toEqualTypeOf<() => Promise>() + }) + + it(`does not advertise disabled null queries`, () => { + useLiveInfiniteQuery( + // @ts-expect-error Infinite queries do not support disabled null queries. + (_q: InitialQueryBuilder) => null, + { pageSize: 5 }, + ) + }) +}) diff --git a/packages/svelte-db/tests/useLiveQuery.svelte.test.ts b/packages/svelte-db/tests/useLiveQuery.svelte.test.ts index cb16e85797..5d490f3036 100644 --- a/packages/svelte-db/tests/useLiveQuery.svelte.test.ts +++ b/packages/svelte-db/tests/useLiveQuery.svelte.test.ts @@ -1,9 +1,13 @@ import { afterEach, describe, expect, it } from 'vitest' import { + BaseQueryBuilder, + DbClient, count, createCollection, createLiveQueryCollection, eq, + getLiveQueryHash, + getStableValueHash, gt, } from '@tanstack/db' import { flushSync } from 'svelte' @@ -75,12 +79,106 @@ const initialIssues: Array = [ ] describe(`Query Collections`, () => { + it(`includes the query in legacy dependency-array SSR identity`, () => { + const client = new DbClient() + const collection = createCollection({ + id: `svelte-legacy-query-identity`, + getKey: (person) => person.id, + startSync: false, + sync: { sync: () => {} }, + }) + const firstPrepared = new BaseQueryBuilder().from({ people: collection }) + const secondPrepared = new BaseQueryBuilder() + .from({ people: collection }) + .where(({ people }) => gt(people.age, 30)) + const firstHash = getStableValueHash( + [`deps`, [1], getLiveQueryHash({ query: firstPrepared })], + `queryKey`, + ) + const secondHash = getStableValueHash( + [`deps`, [1], getLiveQueryHash({ query: secondPrepared })], + `queryKey`, + ) + client.hydrate({ + collections: [], + liveQueries: [ + { + queryHash: firstHash, + dehydratedAt: 1, + snapshot: { + rows: [ + { key: `first`, value: { ...initialPersons[0]!, id: `first` } }, + ], + }, + }, + { + queryHash: secondHash, + dehydratedAt: 1, + snapshot: { + rows: [ + { key: `second`, value: { ...initialPersons[2]!, id: `second` } }, + ], + }, + }, + ], + }) + let firstId: string | undefined + let secondId: string | undefined + + cleanup = $effect.root(() => { + const first = useLiveQuery( + { client, query: (q) => q.from({ people: collection }) }, + [() => 1], + ) + const second = useLiveQuery( + { + client, + query: (q) => + q + .from({ people: collection }) + .where(({ people }) => gt(people.age, 30)), + }, + [() => 1], + ) + flushSync() + firstId = first.data[0]?.id + secondId = second.data[0]?.id + }) + + expect(firstId).toBe(`first`) + expect(secondId).toBe(`second`) + }) + let cleanup: (() => void) | null = null afterEach(() => { cleanup?.() }) + it(`keeps data and keyed state aligned after collection cleanup`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `cleanup-alignment-svelte`, + getKey: (person) => person.id, + initialData: initialPersons, + }), + ) + + cleanup = $effect.root(() => { + const query = useLiveQuery(collection) + flushSync() + + expect(query.data).toHaveLength(3) + expect(query.state.size).toBe(3) + + void collection.cleanup() + flushSync() + + expect(query.data).toHaveLength(0) + expect(query.state.size).toBe(0) + }) + }) + it(`should work with basic collection and select`, () => { const collection = createCollection( mockSyncCollectionOptions({ @@ -116,6 +214,26 @@ describe(`Query Collections`, () => { }) }) + it(`throws when an explicit queryKey cannot be stably hashed`, () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `unhashable-explicit-query-key-svelte`, + getKey: (person) => person.id, + initialData: initialPersons, + }), + ) + + expect(() => { + cleanup = $effect.root(() => { + useLiveQuery({ + queryKey: [collection.id, () => `opaque`], + query: (q) => q.from({ people: collection }), + }) + flushSync() + }) + }).toThrow(/queryKey.*function value/) + }) + it(`should maintain reactivity when destructuring return values with $derived`, () => { const collection = createCollection( mockSyncCollectionOptions({ diff --git a/packages/tauri-db-sqlite-persistence/CHANGELOG.md b/packages/tauri-db-sqlite-persistence/CHANGELOG.md index 25f620e40b..52a3ff37b0 100644 --- a/packages/tauri-db-sqlite-persistence/CHANGELOG.md +++ b/packages/tauri-db-sqlite-persistence/CHANGELOG.md @@ -1,5 +1,152 @@ # @tanstack/tauri-db-sqlite-persistence +## 0.2.21 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db-sqlite-persistence-core@0.2.21 + +## 0.2.20 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.20 + +## 0.2.19 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.19 + +## 0.2.18 + +### Patch Changes + +- Updated dependencies [[`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db-sqlite-persistence-core@0.2.18 + +## 0.2.17 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.17 + +## 0.2.16 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.16 + +## 0.2.15 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.15 + +## 0.2.14 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.14 + +## 0.2.13 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db-sqlite-persistence-core@0.2.13 + +## 0.2.12 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.12 + +## 0.2.11 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.11 + +## 0.2.10 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.10 + +## 0.2.9 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.9 + +## 0.2.8 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.8 + +## 0.2.7 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.7 + +## 0.2.6 + +### Patch Changes + +- Updated dependencies [[`f7da776`](https://github.com/TanStack/db/commit/f7da77660b16cbfe30817fb5c938267d696c8d1c)]: + - @tanstack/db-sqlite-persistence-core@0.2.6 + +## 0.2.5 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.5 + +## 0.2.4 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.4 + +## 0.2.3 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.3 + +## 0.2.2 + +### Patch Changes + +- Updated dependencies []: + - @tanstack/db-sqlite-persistence-core@0.2.2 + +## 0.2.1 + +### Patch Changes + +- Updated dependencies [[`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db-sqlite-persistence-core@0.2.1 + ## 0.2.0 ### Minor Changes diff --git a/packages/tauri-db-sqlite-persistence/e2e/app/CHANGELOG.md b/packages/tauri-db-sqlite-persistence/e2e/app/CHANGELOG.md index 254ae4ded8..81e802bb3d 100644 --- a/packages/tauri-db-sqlite-persistence/e2e/app/CHANGELOG.md +++ b/packages/tauri-db-sqlite-persistence/e2e/app/CHANGELOG.md @@ -1,5 +1,173 @@ # @tanstack/tauri-db-sqlite-persistence-e2e-app +## 0.0.33 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + - @tanstack/tauri-db-sqlite-persistence@0.2.21 + +## 0.0.32 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + - @tanstack/tauri-db-sqlite-persistence@0.2.20 + +## 0.0.31 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + - @tanstack/tauri-db-sqlite-persistence@0.2.19 + +## 0.0.30 + +### Patch Changes + +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + - @tanstack/tauri-db-sqlite-persistence@0.2.18 + +## 0.0.29 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + - @tanstack/tauri-db-sqlite-persistence@0.2.17 + +## 0.0.28 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + - @tanstack/tauri-db-sqlite-persistence@0.2.16 + +## 0.0.27 + +### Patch Changes + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + - @tanstack/tauri-db-sqlite-persistence@0.2.15 + +## 0.0.26 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + - @tanstack/tauri-db-sqlite-persistence@0.2.14 + +## 0.0.25 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + - @tanstack/tauri-db-sqlite-persistence@0.2.13 + +## 0.0.24 + +### Patch Changes + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + - @tanstack/tauri-db-sqlite-persistence@0.2.12 + +## 0.0.23 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + - @tanstack/tauri-db-sqlite-persistence@0.2.11 + +## 0.0.22 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + - @tanstack/tauri-db-sqlite-persistence@0.2.10 + +## 0.0.21 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + - @tanstack/tauri-db-sqlite-persistence@0.2.9 + +## 0.0.20 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + - @tanstack/tauri-db-sqlite-persistence@0.2.8 + +## 0.0.19 + +### Patch Changes + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + - @tanstack/tauri-db-sqlite-persistence@0.2.7 + +## 0.0.18 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + - @tanstack/tauri-db-sqlite-persistence@0.2.6 + +## 0.0.17 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + - @tanstack/tauri-db-sqlite-persistence@0.2.5 + +## 0.0.16 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + - @tanstack/tauri-db-sqlite-persistence@0.2.4 + +## 0.0.15 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + - @tanstack/tauri-db-sqlite-persistence@0.2.3 + +## 0.0.14 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + - @tanstack/tauri-db-sqlite-persistence@0.2.2 + +## 0.0.13 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + - @tanstack/tauri-db-sqlite-persistence@0.2.1 + ## 0.0.12 ### Patch Changes diff --git a/packages/tauri-db-sqlite-persistence/e2e/app/package.json b/packages/tauri-db-sqlite-persistence/e2e/app/package.json index 795d1406cd..24a3c50646 100644 --- a/packages/tauri-db-sqlite-persistence/e2e/app/package.json +++ b/packages/tauri-db-sqlite-persistence/e2e/app/package.json @@ -1,7 +1,7 @@ { "name": "@tanstack/tauri-db-sqlite-persistence-e2e-app", "private": true, - "version": "0.0.12", + "version": "0.0.33", "type": "module", "scripts": { "build": "vite build", diff --git a/packages/tauri-db-sqlite-persistence/package.json b/packages/tauri-db-sqlite-persistence/package.json index 796c62efa3..9bffea8982 100644 --- a/packages/tauri-db-sqlite-persistence/package.json +++ b/packages/tauri-db-sqlite-persistence/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/tauri-db-sqlite-persistence", - "version": "0.2.0", + "version": "0.2.21", "description": "Tauri SQLite persisted collection adapter for TanStack DB", "author": "TanStack Team", "license": "MIT", diff --git a/packages/trailbase-db-collection/CHANGELOG.md b/packages/trailbase-db-collection/CHANGELOG.md index da6baa3e67..a298e6125d 100644 --- a/packages/trailbase-db-collection/CHANGELOG.md +++ b/packages/trailbase-db-collection/CHANGELOG.md @@ -1,5 +1,174 @@ # @tanstack/trailbase-db-collection +## 0.1.107 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 0.1.106 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 0.1.105 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 0.1.104 + +### Patch Changes + +- Settle subset loads only after their committed rows and events are visible. A ([#1769](https://github.com/TanStack/db/pull/1769)) + commit receipt now rejects with `AbortError` when cancellation wins before + application and ignores later aborts. Preserve causal publication, + cancellation, persistence, and error handling across the affected sync + adapters. +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 0.1.103 + +### Patch Changes + +- Report incremental subset-load failures through subscriptions, live-query utilities, and effects while keeping cached source rows available. Recover cleanly from failed or overlapping must-refetch replays, collection cleanup, effect teardown errors, and cooperative adapter cancellation. Electric's shared-stream snapshot path still depends on upstream request identity or cancellation support to prevent rows from an aborted request from arriving before the request Promise settles. ([#1756](https://github.com/TanStack/db/pull/1756)) + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 0.1.102 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 0.1.101 + +### Patch Changes + +- Propagate initial query sync failures through dependent live queries and readiness promises, including recovery and late subscribers, while preserving a ready cached snapshot on later refetch failures. Let sync adapters pass the original failure to `markError(error)` so readiness promises reject with that cause. Isolate adapter callbacks by sync session, preserve synchronous startup errors, and prevent rejected deduplicated subset requests from creating detached promise rejections. ([#1751](https://github.com/TanStack/db/pull/1751)) + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 0.1.100 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 0.1.99 + +### Patch Changes + +- Add SSR through request-scoped `DbClient` instances, collection descriptors, ([#1564](https://github.com/TanStack/db/pull/1564)) + explicit collection-row hydration, live-query result snapshots, adapter sync + metadata, and React and Svelte descriptor resolution. + + React live queries now derive identity from structured query IR. Opaque queries + can provide `queryKey`; legacy dependency arrays and unkeyed opaque queries keep + working with development warnings until 1.0. + + Add TanStack Router integration that streams live queries discovered during a + Suspense render as pending promises which resolve to ordered result snapshots. + The browser starts normal source sync and atomically replaces the snapshot when + its live result is ready. + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 0.1.98 + +### Patch Changes + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 0.1.97 + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 0.1.96 + +### Patch Changes + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 0.1.95 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 0.1.94 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 0.1.93 + +### Patch Changes + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 0.1.92 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 0.1.91 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 0.1.90 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 0.1.89 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 0.1.88 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 0.1.87 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 0.1.86 ### Patch Changes diff --git a/packages/trailbase-db-collection/e2e/trailbase.e2e.test.ts b/packages/trailbase-db-collection/e2e/trailbase.e2e.test.ts index 47cdf0cea1..375426d802 100644 --- a/packages/trailbase-db-collection/e2e/trailbase.e2e.test.ts +++ b/packages/trailbase-db-collection/e2e/trailbase.e2e.test.ts @@ -6,7 +6,7 @@ */ import { describe, expect, inject } from 'vitest' -import { createCollection, BTreeIndex } from '@tanstack/db' +import { BTreeIndex, createCollection } from '@tanstack/db' import { initClient } from 'trailbase' import { trailBaseCollectionOptions } from '../src/trailbase' import { diff --git a/packages/trailbase-db-collection/package.json b/packages/trailbase-db-collection/package.json index 01ac435b6b..f0a934cf82 100644 --- a/packages/trailbase-db-collection/package.json +++ b/packages/trailbase-db-collection/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/trailbase-db-collection", - "version": "0.1.86", + "version": "0.1.107", "description": "TrailBase collection for TanStack DB", "author": "Sebastian Jeltsch", "license": "MIT", @@ -21,6 +21,7 @@ "dev": "vite build --watch", "lint": "eslint . --fix", "test": "vitest --run", + "test:oracles": "vitest run tests/lifecycle-oracle.property.test.ts", "test:e2e": "vitest --run --config vitest.e2e.config.ts" }, "type": "module", diff --git a/packages/trailbase-db-collection/src/trailbase.ts b/packages/trailbase-db-collection/src/trailbase.ts index b47728d861..51c2ccebbf 100644 --- a/packages/trailbase-db-collection/src/trailbase.ts +++ b/packages/trailbase-db-collection/src/trailbase.ts @@ -1,5 +1,6 @@ /* eslint-disable @typescript-eslint/no-unnecessary-condition */ import { Store } from '@tanstack/store' +import { withCollectionConfigFactory } from '@tanstack/db' import { ExpectedDeleteTypeError, ExpectedInsertTypeError, @@ -170,7 +171,9 @@ export function trailBaseCollectionOptions< let eventReader: ReadableStreamDefaultReader | undefined const cancelEventReader = () => { if (eventReader) { - eventReader.cancel() + // An already-errored stream rejects cancellation too. Cleanup still + // retires its reader; that rejection must not escape as detached work. + void eventReader.cancel().catch(() => undefined) eventReader.releaseLock() eventReader = undefined } @@ -179,7 +182,18 @@ export function trailBaseCollectionOptions< type SyncParams = Parameters[`sync`]>[0] const sync = { sync: (params: SyncParams) => { - const { begin, write, commit, markReady } = params + const { begin, write, commit, markReady, markError, collection } = params + let cancelled = false + let periodicCleanupTask: ReturnType | undefined + + const cleanup = () => { + cancelled = true + cancelEventReader() + if (periodicCleanupTask !== undefined) { + clearInterval(periodicCleanupTask) + periodicCleanupTask = undefined + } + } // NOTE: We cache cursors from prior fetches. TanStack/db expects that // cursors can be derived from a key, which is not true for TB, since @@ -188,6 +202,8 @@ export function trailBaseCollectionOptions< // Load (more) data. async function load(opts: LoadSubsetOptions) { + if (cancelled || opts.signal?.aborted) return + const lastKey = opts.cursor?.lastKey let cursor: string | undefined = lastKey !== undefined ? cursors.get(lastKey) : undefined @@ -204,18 +220,26 @@ export function trailBaseCollectionOptions< if (remaining <= 0) { return } + const appliedPages: Array> = [] while (true) { const limit = Math.min(remaining, 256) - const response = await config.recordApi.list({ - pagination: { - limit, - offset, - cursor, - }, - order, - filters, - }) + let response + try { + response = await config.recordApi.list({ + pagination: { + limit, + offset, + cursor, + }, + order, + filters, + }) + } catch (error) { + if (cancelled || opts.signal?.aborted) return + throw error + } + if (cancelled || opts.signal?.aborted) return const length = response.records.length if (length === 0) { @@ -232,7 +256,11 @@ export function trailBaseCollectionOptions< }) } - commit() + const applied = commit(opts.signal) + if (applied !== true) { + appliedPages.push(applied) + } + if (cancelled || opts.signal?.aborted) return remaining -= length @@ -254,6 +282,8 @@ export function trailBaseCollectionOptions< cursor = response.cursor } } + + await Promise.all(appliedPages) } // Afterwards subscribe. @@ -262,8 +292,6 @@ export function trailBaseCollectionOptions< const { done, value: event } = await reader.read() if (done || !event) { - reader.releaseLock() - eventReader = undefined return } @@ -281,7 +309,7 @@ export function trailBaseCollectionOptions< } else { console.error(`Error: ${event.Error}`) } - commit() + void commit() if (value) { seenIds.setState((curr: Map) => { @@ -294,32 +322,62 @@ export function trailBaseCollectionOptions< } async function start() { - const eventStream = await config.recordApi.subscribe(`*`) - const reader = (eventReader = eventStream.getReader()) - - // Start listening for subscriptions first. Otherwise, we'd risk a gap - // between the initial fetch and starting to listen. - listen(reader) - + let reader: ReadableStreamDefaultReader | undefined try { + const eventStream = await config.recordApi.subscribe(`*`) + if (cancelled) { + await eventStream.cancel() + return + } + const subscribedReader = eventStream.getReader() + reader = eventReader = subscribedReader + + // Start listening for subscriptions first. Otherwise, we'd risk a gap + // between the initial fetch and starting to listen. + void listen(subscribedReader) + .finally(() => { + // A closed stream can still have a final event being processed. + // Release only after the listener has finished draining it. + // A processing failure can leave the stream open; cancel it too. + // Preserve the original failure if the stream already errored. + // Error settlement must not wait for transport cleanup. + void subscribedReader.cancel().catch(() => undefined) + subscribedReader.releaseLock() + if (eventReader === subscribedReader) eventReader = undefined + }) + .catch((error: unknown) => { + if (!cancelled && collection.status === `loading`) { + markError(error) + } else if (!cancelled) { + console.error(`TrailBase subscription failed`, error) + } + }) + // Eager mode: perform initial fetch to populate everything if (internalSyncMode === `eager`) { // Load everything on initial load. await load({}) + if (cancelled) return fullSyncCompleted = true } - } catch (e) { + if (!cancelled && collection.status === `loading`) { + markReady() + } + } catch (error) { + // An abandoned startup must not cancel a replacement session's reader. + if (cancelled) return cancelEventReader() - throw e - } finally { - // Mark ready both if everything went well or if there's an error to - // avoid blocking apps waiting for `.preload()` to finish. - markReady() + if (collection.status === `loading`) { + markError(error) + } + return } // Lastly, start a periodic cleanup task that will be removed when the // reader closes. - const periodicCleanupTask = setInterval(() => { + if (cancelled || !reader) return + + periodicCleanupTask = setInterval(() => { seenIds.setState((curr) => { const now = Date.now() let anyExpired = false @@ -337,17 +395,25 @@ export function trailBaseCollectionOptions< }) }, 120 * 1000) - reader.closed.finally(() => clearInterval(periodicCleanupTask)) + const clearCleanupTask = () => { + if (periodicCleanupTask !== undefined) { + clearInterval(periodicCleanupTask) + periodicCleanupTask = undefined + } + } + // listen() reports read errors. Observe this separate promise too. + void reader.closed.then(clearCleanupTask, clearCleanupTask) } - start() + void start() // Eager mode doesn't need subset loading if (internalSyncMode === `eager`) { - return + return { cleanup } } return { + cleanup, loadSubset: load, getSyncMetadata: () => ({ @@ -363,7 +429,7 @@ export function trailBaseCollectionOptions< }) as const, } - return { + const options = { ...config, sync, getKey, @@ -428,6 +494,11 @@ export function trailBaseCollectionOptions< cancel: cancelEventReader, }, } + + return withCollectionConfigFactory( + options, + () => trailBaseCollectionOptions(config) as typeof options, + ) } function buildOrder(opts: LoadSubsetOptions): undefined | Array { diff --git a/packages/trailbase-db-collection/tests/ORACLE.md b/packages/trailbase-db-collection/tests/ORACLE.md new file mode 100644 index 0000000000..c3df466642 --- /dev/null +++ b/packages/trailbase-db-collection/tests/ORACLE.md @@ -0,0 +1,74 @@ +# TrailBase lifecycle oracle + +`lifecycle-oracle.property.test.ts` runs the real adapter and collection against +a controlled RecordApi and native ReadableStream. Only network I/O is mocked. +An independent Map models rows; explicit gates control subscribe/list settlement. +The same interpreter runs a fixed corpus and generated histories. + +## Laws and scope + +- Published inserts, updates and deletes agree with the model after each event. +- Eager startup waits for its list; on-demand startup does not claim to load rows. + Required startup failures reject. Later stream failures retain rows and report + the error, matching the adapter's current policy. +- Graceful closure drains buffered events before releasing the reader. +- Processing failures cancel an open source. Terminal paths release readers and + timers without detached rejections, including same-turn cleanup. +- A startup processing error rejects readiness before asynchronous cancellation + settles. Two fixed controls hold cancellation through later list completion, + then resolve or reject it; cleanup cannot turn failed startup into readiness. +- Cleanup clears the collection. Late work from an old session cannot publish + into, cancel, or report errors against a replacement session. + +Histories contain one to three sessions, eager/on-demand modes, delayed startup +and list resolve/reject, zero to eight row edits, five stream endings, and +immediate versus settled cleanup. Thirty-two fixed cases pin the boundaries; +ordinary runs add 30 fixed-seed and 50 fresh-seed histories with shrinking. + +This is not a model of pagination, filtered subsets, optimistic mutation +acknowledgements, service reconnects, or arbitrary event/list interleavings. +The generated event phase starts after loading; existing loading-time unit +matrices and the two held-cancellation startup controls remain valuable. Those +controls were both RED when listener failure awaited cancellation, and GREEN +when cancellation was observed separately. The on-demand driver calls the core subset boundary, +not a live query. Service-backed E2E coverage remains separate. + +## Run and replay + +Run from `packages/trailbase-db-collection`: + +```sh +pnpm test:oracles --coverage.enabled=false +TANSTACK_DB_ORACLE_RUNS_MULTIPLIER=10 pnpm test:oracles --coverage.enabled=false --testTimeout=60000 +``` + +For a failing random campaign, use its reported seed and shrink path: + +```sh +TANSTACK_DB_ORACLE_SEED=123 TANSTACK_DB_ORACLE_PATH=0:1 TANSTACK_DB_ORACLE_PROPERTY=trailbase.lifecycle pnpm test:oracles --coverage.enabled=false -t 'random or replayed' +``` + +Replace the example seed/path with the failure's values. Shared configuration +lives in `packages/db/tests/oracle-config.ts`; do not copy its replay parser. + +## Evidence against false greens + +Before the stale-startup fix, both campaigns independently shrank to: cancel a +pending subscription, restart and load a healthy stream, reject the old subscribe. +The old startup canceled the replacement reader. The fixed corpus pins this in +both modes and sends further edits through the replacement stream. + +Five isolated source-transform mutations were each rejected by the fixed corpus: + +| Reintroduced fault | Failing cases | Detecting assertion | +| ------------------------------------------------- | ------------: | ------------------------------------------------- | +| Unobserved `reader.closed.finally()` rejection | 4 | No detached rejections | +| Release reader as soon as `closed` settles | 2 | Buffered close reports no error | +| Omit source cancellation after processing failure | 2 | Underlying source canceled | +| Ignore cleanup's cancellation rejection | 2 | Same-turn cleanup has no detached rejection | +| Let abandoned startup cancel the current reader | 2 | Replacement remains locked, uncanceled and usable | + +These targeted mutations test known failure classes, not overall mutation +coverage or a proof of completeness. The 10× campaign passed 800 generated +histories (fixed seed 714203, random seed 127535183) plus the 32 corpus cases. +No service-backed E2E run is claimed. diff --git a/packages/trailbase-db-collection/tests/lifecycle-oracle.property.test.ts b/packages/trailbase-db-collection/tests/lifecycle-oracle.property.test.ts new file mode 100644 index 0000000000..04f74ac39a --- /dev/null +++ b/packages/trailbase-db-collection/tests/lifecycle-oracle.property.test.ts @@ -0,0 +1,422 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { expect, it, vi } from 'vitest' +import { createCollection } from '@tanstack/db' +import { oraclePropertyOptions, oracleRuns } from '../../db/tests/oracle-config' +import { trailBaseCollectionOptions } from '../src/trailbase' +import { MockRecordApi } from './mock-record-api' +import type { Event, ListResponse } from 'trailbase' + +type Row = { id: number; value: number } +type Change = { operation: `set` | `delete`; id: number; value: number } +type Ending = + | `close` + | `buffered-close` + | `read-error` + | `parse-error` + | `cleanup` +type Session = + | { kind: `cancel-subscribe`; late: `resolve` | `reject` } + | { kind: `cancel-load`; late: `resolve` | `reject` } + | { kind: `reject-subscribe` } + | { kind: `reject-load` } + | { + kind: `stream` + changes: Array + ending: Ending + immediate: boolean + } +type Scenario = { mode: `eager` | `on-demand`; sessions: Array } + +function deferred() { + let resolve!: (value: T) => void + let reject!: (error: unknown) => void + const promise = new Promise((yes, no) => { + resolve = yes + reject = no + }) + // Every adapter promise is observed even when its session is abandoned. + void promise.catch(() => undefined) + return { promise, resolve, reject } +} + +function observe(promise: Promise) { + let result: `pending` | `fulfilled` | `rejected` = `pending` + let error: unknown + void promise.then( + () => { + result = `fulfilled` + }, + (failure: unknown) => { + result = `rejected` + error = failure + }, + ) + return { + get result() { + return result + }, + get error() { + return error + }, + } +} + +// All adapter I/O is gated. One host turn drains native stream microtasks and +// exposes detached rejections; it does not stand in for a network completion. +const turn = () => new Promise((resolve) => setTimeout(resolve, 0)) +const failure = new Error(`oracle stream failure`) +const invalidValue = -10_000 + +function source() { + let controller!: ReadableStreamDefaultController + const cancel = vi.fn((): void | Promise => {}) + const stream = new ReadableStream({ + start(value) { + controller = value + }, + cancel, + }) + return { + stream, + cancel, + controller, + subscription: deferred>(), + list: deferred>(), + listCalls: 0, + } +} + +/** + * Reference: rows are an independent key/value relation. Before readiness a + * required load failure rejects startup; afterward a broken stream retains + * the last rows and reports an error. Cleanup clears rows and retires work. + * The model never reads adapter bookkeeping, pending transactions or caches. + */ +async function checkLifecycle({ mode, sessions }: Scenario) { + const api = new MockRecordApi() + let current = source() + api.subscribe.mockImplementation(() => current.subscription.promise) + api.list.mockImplementation(() => { + current.listCalls++ + return current.list.promise + }) + const allSources: Array> = [] + const retired: Array<() => void> = [] + const unhandled: Array = [] + const onUnhandled = (error: unknown) => unhandled.push(error) + const reported = vi.spyOn(console, `error`).mockImplementation(() => {}) + const intervals = vi.spyOn(globalThis, `setInterval`) + const cleared = vi.spyOn(globalThis, `clearInterval`) + process.on(`unhandledRejection`, onUnhandled) + const collection = createCollection( + trailBaseCollectionOptions({ + recordApi: api, + getKey: (row: Row) => row.id, + syncMode: mode, + parse: { + value: (value) => { + if (value === invalidValue) throw failure + return value + }, + }, + serialize: {}, + }), + ) + let expected = new Map() + const publicRows = () => + Array.from(collection.values(), ({ id, value }) => ({ id, value })).sort( + (a, b) => a.id - b.id, + ) + const assertRows = () => + expect(publicRows()).toEqual( + [...expected.values()].sort((a, b) => a.id - b.id), + ) + const assertNoTimers = () => { + for (const timer of intervals.mock.results) { + if (timer.type === `return`) + expect(cleared).toHaveBeenCalledWith(timer.value) + } + } + const flushRetired = async () => { + retired.splice(0).forEach((settle) => settle()) + await turn() + assertRows() + expect(unhandled).toEqual([]) + } + try { + for (const [epoch, plan] of sessions.entries()) { + current = source() + const active = current + allSources.push(active) + expected = new Map() + reported.mockClear() + collection.startSyncImmediate() + const preload = observe(collection.preload()) + await turn() + expect(api.subscribe).toHaveBeenCalledTimes(epoch + 1) + expect(collection.status).toBe(`loading`) + expect(preload.result).toBe(`pending`) + assertRows() + + if (plan.kind === `cancel-subscribe`) { + await collection.cleanup() + retired.push(() => + plan.late === `resolve` + ? active.subscription.resolve(active.stream) + : active.subscription.reject(failure), + ) + } else if (plan.kind === `reject-subscribe`) { + active.subscription.reject(failure) + await turn() + expect(preload.result).toBe(`rejected`) + expect(preload.error).toBe(failure) + expect(collection.status).toBe(`error`) + expect(active.listCalls).toBe(0) + await collection.cleanup() + } else { + active.subscription.resolve(active.stream) + await turn() + const load = + mode === `eager` + ? preload + : observe(Promise.resolve(collection._sync.loadSubset({}))) + await turn() + expect(active.listCalls).toBe(1) + expect(load.result).toBe(`pending`) + expect(preload.result).toBe(mode === `eager` ? `pending` : `fulfilled`) + + if (plan.kind === `cancel-load`) { + await collection.cleanup() + retired.push(() => + plan.late === `resolve` + ? active.list.resolve({ records: [{ id: 99, value: epoch }] }) + : active.list.reject(failure), + ) + } else if (plan.kind === `reject-load`) { + active.list.reject(failure) + await turn() + expect(load.result).toBe(`rejected`) + expect(load.error).toBe(failure) + expect(collection.status).toBe(mode === `eager` ? `error` : `ready`) + assertRows() + await collection.cleanup() + } else { + const baseline = { id: 0, value: epoch } + active.list.resolve({ records: [baseline] }) + expected.set(0, baseline) + await turn() + expect(load.result).toBe(`fulfilled`) + expect(preload.result).toBe(`fulfilled`) + expect(collection.status).toBe(`ready`) + assertRows() + + // Old subscribe/list work finishes only once the replacement owns + // a live stream and baseline, not merely after the old cleanup. + await flushRetired() + expect(active.stream.locked).toBe(true) + expect(active.cancel).not.toHaveBeenCalled() + + for (const change of plan.changes) { + const previous = expected.get(change.id) + if (change.operation === `delete`) { + if (!previous) continue // Only issue legal deletes from the model. + active.controller.enqueue({ Delete: previous }) + expected.delete(change.id) + } else { + const row = { id: change.id, value: change.value } + active.controller.enqueue( + previous ? { Update: row } : { Insert: row }, + ) + expected.set(change.id, row) + } + await turn() + assertRows() + expect(collection.status).toBe(`ready`) + } + + if (plan.ending === `buffered-close`) { + for (const id of [10, 11]) { + const row = { id, value: epoch } + active.controller.enqueue({ Insert: row }) + expected.set(id, row) + } + } + if (plan.ending === `close` || plan.ending === `buffered-close`) + active.controller.close() + if (plan.ending === `read-error`) active.controller.error(failure) + if (plan.ending === `parse-error`) + active.controller.enqueue({ + Insert: { id: 12, value: invalidValue }, + }) + if (plan.immediate || plan.ending === `cleanup`) { + await collection.cleanup() + expected.clear() + } + await turn() + assertRows() + expect(active.stream.locked).toBe(false) + assertNoTimers() + const isFailure = + plan.ending === `read-error` || plan.ending === `parse-error` + if (isFailure && !plan.immediate) { + expect(reported).toHaveBeenCalledExactlyOnceWith( + `TrailBase subscription failed`, + failure, + ) + } else expect(reported).not.toHaveBeenCalled() + if (plan.ending === `cleanup` || plan.ending === `parse-error`) + expect(active.cancel).toHaveBeenCalledOnce() + if (!plan.immediate && plan.ending !== `cleanup`) + expect(collection.status).toBe(`ready`) + await collection.cleanup() + } + } + expected.clear() + await turn() + assertRows() + expect(collection.status).toBe(`cleaned-up`) + expect(unhandled).toEqual([]) + assertNoTimers() + } + await flushRetired() + for (const active of allSources) expect(active.stream.locked).toBe(false) + } finally { + await collection.cleanup() + // Release all controlled gates even when a mutant fails an early assertion. + for (const active of allSources) { + active.subscription.resolve(active.stream) + active.list.resolve({ records: [] }) + } + await turn() + process.off(`unhandledRejection`, onUnhandled) + reported.mockRestore() + intervals.mockRestore() + cleared.mockRestore() + } +} + +const changeArb = fc.record({ + operation: fc.constantFrom(`set`, `delete`), + id: fc.integer({ min: 0, max: 3 }), + value: fc.integer({ min: -20, max: 20 }), +}) +const sessionArb: fc.Arbitrary = fc.oneof( + fc.record({ + kind: fc.constantFrom(`cancel-subscribe` as const, `cancel-load` as const), + late: fc.constantFrom(`resolve` as const, `reject` as const), + }), + fc.record({ + kind: fc.constantFrom(`reject-subscribe` as const, `reject-load` as const), + }), + fc.record({ + kind: fc.constant(`stream` as const), + changes: fc.array(changeArb, { maxLength: 8 }), + ending: fc.constantFrom( + `close`, + `buffered-close`, + `read-error`, + `parse-error`, + `cleanup`, + ), + immediate: fc.boolean(), + }), +) +const scenarioArb = fc.record({ + mode: fc.constantFrom(`eager` as const, `on-demand` as const), + sessions: fc.array(sessionArb, { minLength: 1, maxLength: 3 }), +}) +const live = (ending: Ending, immediate = false): Session => ({ + kind: `stream`, + ending, + immediate, + changes: [ + { operation: `set`, id: 1, value: 3 }, + { operation: `set`, id: 1, value: 4 }, + { operation: `delete`, id: 1, value: 0 }, + ], +}) + +// Fixed witnesses ensure every lifecycle boundary is exercised, independently +// of the random distribution. The same interpreter runs corpus and fuzz cases. +const corpus: Array<{ name: string; sessions: Array }> = [ + ...( + [`close`, `buffered-close`, `read-error`, `parse-error`, `cleanup`] as const + ).flatMap((ending) => + [false, true].map((immediate) => ({ + name: `${ending}, immediate=${immediate}`, + sessions: [live(ending, immediate)], + })), + ), + ...([`cancel-subscribe`, `cancel-load`] as const).flatMap((kind) => + ([`resolve`, `reject`] as const).map((late) => ({ + name: `${kind}, stale ${late} after restart`, + sessions: [{ kind, late }, live(`close`)], + })), + ), + ...([`reject-subscribe`, `reject-load`] as const).map((kind) => ({ + name: kind, + sessions: [{ kind }, live(`close`)], + })), +] +it.each([`resolve`, `reject`] as const)( + `rejects startup before stream cancellation can %s`, + async (cancellation) => { + const active = source() + const cancelled = deferred() + active.cancel.mockImplementation(() => cancelled.promise) + const api = new MockRecordApi() + api.subscribe.mockResolvedValue(active.stream) + api.list.mockImplementation(() => active.list.promise) + const collection = createCollection( + trailBaseCollectionOptions({ + recordApi: api, + getKey: (row: Row) => row.id, + syncMode: `eager`, + parse: { + value: () => { + throw failure + }, + }, + serialize: {}, + }), + ) + const preload = observe(collection.preload()) + try { + await turn() + expect(api.list).toHaveBeenCalledOnce() + active.controller.enqueue({ Insert: { id: 1, value: invalidValue } }) + await turn() + expect(active.cancel).toHaveBeenCalledOnce() + // Cleanup I/O is still pending, but it cannot postpone the load error. + expect(preload.result).toBe(`rejected`) + expect(preload.error).toBe(failure) + expect(collection.status).toBe(`error`) + active.list.resolve({ records: [] }) + await turn() + expect(preload.result).toBe(`rejected`) + expect(collection.status).toBe(`error`) + } finally { + if (cancellation === `resolve`) cancelled.resolve() + else cancelled.reject(new Error(`cancel failure`)) + active.list.resolve({ records: [] }) + await collection.cleanup() + await turn() + } + }, +) + +it.each( + corpus.flatMap((entry) => + ([`eager`, `on-demand`] as const).map((mode) => ({ ...entry, mode })), + ), +)(`preserves lifecycle laws: $mode / $name`, ({ mode, sessions }) => + checkLifecycle({ mode, sessions }), +) +fcTest.prop([scenarioArb], { seed: 714_203, numRuns: oracleRuns(30) })( + `matches generated lifecycle histories with a fixed seed`, + checkLifecycle, +) +fcTest.prop([scenarioArb], oraclePropertyOptions(50, `trailbase.lifecycle`))( + `matches generated lifecycle histories with a random or replayed seed`, + checkLifecycle, +) diff --git a/packages/trailbase-db-collection/tests/mock-record-api.ts b/packages/trailbase-db-collection/tests/mock-record-api.ts new file mode 100644 index 0000000000..04e0e2ea15 --- /dev/null +++ b/packages/trailbase-db-collection/tests/mock-record-api.ts @@ -0,0 +1,90 @@ +import { vi } from 'vitest' +import type { + CreateOperation, + DeleteOperation, + Event, + FilterOrComposite, + ListOperation, + ListOpts, + ListResponse, + Pagination, + ReadOperation, + ReadOpts, + RecordApi, + RecordId, + SubscribeOpts, + UpdateOperation, +} from 'trailbase' + +export class MockRecordApi implements RecordApi { + list = vi.fn( + (_opts?: { + pagination?: Pagination + order?: Array + filters?: Array + count?: boolean + expand?: Array + }): Promise> => { + return Promise.resolve({ records: [] }) + }, + ) + listOp = vi.fn((_opts?: ListOpts): ListOperation => { + throw `listOp` + }) + listGeoOp = vi.fn((_geometryColumn: string, _opts?: ListOpts) => { + throw `listGeoOp` + }) + + read = vi.fn( + ( + _id: string | number, + _opt?: { + expand?: Array + }, + ): Promise => { + throw `read` + }, + ) + readOp = vi.fn((_id: RecordId, _opt?: ReadOpts): ReadOperation => { + throw `readOp` + }) + + create = vi.fn((_record: T): Promise => { + throw `create` + }) + createBulk = vi.fn((_records: Array): Promise> => { + throw `createBulk` + }) + createOp = vi.fn((_record: T): CreateOperation => { + throw `createOp` + }) + + update = vi.fn((_id: string | number, _record: Partial): Promise => { + throw `update` + }) + updateOp = vi.fn((_id: RecordId, _record: Partial): UpdateOperation => { + throw `updateOp` + }) + + delete = vi.fn((_id: string | number): Promise => { + throw `delete` + }) + deleteOp = vi.fn((_id: RecordId): DeleteOperation => { + throw `deleteOp` + }) + + subscribe = vi.fn((_id: string | number): Promise> => { + return Promise.resolve( + new ReadableStream({ + start: (controller: ReadableStreamDefaultController) => { + controller.close() + }, + }), + ) + }) + subscribeAll = vi.fn( + (_opts?: SubscribeOpts): Promise> => { + throw `subscribeAll` + }, + ) +} diff --git a/packages/trailbase-db-collection/tests/trailbase.test.ts b/packages/trailbase-db-collection/tests/trailbase.test.ts index c2c0845147..7065486e24 100644 --- a/packages/trailbase-db-collection/tests/trailbase.test.ts +++ b/packages/trailbase-db-collection/tests/trailbase.test.ts @@ -1,23 +1,9 @@ import { describe, expect, it, vi } from 'vitest' -import { createCollection } from '@tanstack/db' +import { createCollection, createTransaction } from '@tanstack/db' import { trailBaseCollectionOptions } from '../src/trailbase' import { stripVirtualProps } from '../../db/tests/utils' -import type { - CreateOperation, - DeleteOperation, - Event, - FilterOrComposite, - ListOperation, - ListOpts, - ListResponse, - Pagination, - ReadOperation, - ReadOpts, - RecordApi, - RecordId, - SubscribeOpts, - UpdateOperation, -} from 'trailbase' +import { MockRecordApi } from './mock-record-api' +import type { Event, ListResponse } from 'trailbase' type Data = { id: number | null @@ -33,94 +19,456 @@ const stripState = (state: Map) => ]), ) -class MockRecordApi implements RecordApi { - list = vi.fn( - (_opts?: { - pagination?: Pagination - order?: Array - filters?: Array - count?: boolean - expand?: Array - }): Promise> => { - return Promise.resolve({ records: [] }) +function setUp(recordApi: MockRecordApi) { + // Get the options with utilities + const options = trailBaseCollectionOptions({ + recordApi, + getKey: (item: Data): number | number => + item.id ?? Math.round(Math.random() * 100000), + startSync: true, + parse: {}, + serialize: {}, + }) + + return options +} + +async function expectWildcardFailureSettlesPreload(): Promise { + const failure = new Error(`wildcard subscription denied`) + const recordApi = new MockRecordApi() + recordApi.subscribe.mockRejectedValue(failure) + + const collection = createCollection(setUp(recordApi)) + const preload = collection.preload() + + try { + await expect(preload).rejects.toBe(failure) + expect(collection.status).toBe(`error`) + } finally { + await collection.cleanup() + await Promise.allSettled([preload]) + } +} + +describe(`TrailBase Integration`, () => { + it.each( + ([`loading`, `ready`] as const).flatMap((phase) => + ([`close`, `error`] as const).map((ending) => ({ phase, ending })), + ), + )( + `handles same-turn $ending and cleanup while $phase`, + async ({ phase, ending }) => { + const recordApi = new MockRecordApi() + const row: Data = { id: 1, updated: 0, data: `loaded` } + let resolveList!: (response: ListResponse) => void + recordApi.list.mockReturnValue( + new Promise((resolve) => { + resolveList = resolve + }), + ) + let controller!: ReadableStreamDefaultController + const stream = new ReadableStream({ + start(value) { + controller = value + }, + }) + recordApi.subscribe.mockResolvedValue(stream) + const errors: Array = [] + const recordUnhandled = (error: unknown) => errors.push(error) + process.on(`unhandledRejection`, recordUnhandled) + const collection = createCollection(setUp(recordApi)) + const preload = collection.preload().then( + () => `ready`, + (error: unknown) => error, + ) + try { + await vi.waitFor(() => expect(recordApi.list).toHaveBeenCalledOnce()) + if (phase === `ready`) { + resolveList({ records: [row] }) + expect(await preload).toBe(`ready`) + expect(collection.get(1)).toMatchObject(row) + } + if (ending === `error`) controller.error(new Error(`connection lost`)) + else controller.close() + // No microtask between stream termination and cleanup. + await collection.cleanup() + resolveList({ records: [row] }) + if (phase === `loading`) + expect(await preload).toMatchObject({ name: `AbortError` }) + await new Promise((resolve) => setTimeout(resolve, 0)) + expect(errors).toEqual([]) + expect(stream.locked).toBe(false) + expect(collection.status).toBe(`cleaned-up`) + expect(collection.size).toBe(0) + expect(recordApi.subscribe).toHaveBeenCalledOnce() + expect(recordApi.list).toHaveBeenCalledOnce() + } finally { + resolveList({ records: [] }) + await collection.cleanup() + await preload + await new Promise((resolve) => setTimeout(resolve, 0)) + process.off(`unhandledRejection`, recordUnhandled) + } }, ) - listOp = vi.fn((_opts?: ListOpts): ListOperation => { - throw `listOp` - }) - listGeoOp = vi.fn((_geometryColumn: string, _opts?: ListOpts) => { - throw `listGeoOp` - }) - read = vi.fn( - ( - _id: string | number, - _opt?: { - expand?: Array + it(`cancels an open stream when processing an event fails`, async () => { + const recordApi = new MockRecordApi() + let controller!: ReadableStreamDefaultController + const cancel = vi.fn() + const stream = new ReadableStream({ + start(value) { + controller = value }, - ): Promise => { - throw `read` + cancel, + }) + recordApi.subscribe.mockResolvedValue(stream) + const failure = new Error(`parse rejected row`) + const reported = vi.spyOn(console, `error`).mockImplementation(() => {}) + const collection = createCollection( + trailBaseCollectionOptions({ + recordApi, + getKey: (row: Data) => row.id!, + parse: { + id: () => { + throw failure + }, + }, + serialize: {}, + }), + ) + try { + await collection.preload() + controller.enqueue({ Insert: { id: 1, updated: 0, data: `invalid` } }) + await new Promise((resolve) => setTimeout(resolve, 0)) + expect(reported).toHaveBeenCalledExactlyOnceWith( + `TrailBase subscription failed`, + failure, + ) + expect(cancel).toHaveBeenCalledOnce() + expect(stream.locked).toBe(false) + await collection.cleanup() + expect(cancel).toHaveBeenCalledOnce() + } finally { + await collection.cleanup() + reported.mockRestore() + } + }) + + it.each([`close`, `buffered-close`, `error`] as const)( + `releases a settled stream without unhandled rejection after %s`, + async (ending) => { + const recordApi = new MockRecordApi() + const row: Data = { id: 1, updated: 0, data: `retained` } + recordApi.list.mockResolvedValue({ records: [row] }) + let controller!: ReadableStreamDefaultController + const stream = new ReadableStream({ + start(value) { + controller = value + }, + }) + recordApi.subscribe.mockResolvedValue(stream) + const failure = new Error(`connection lost`) + const errors: Array = [] + const recordUnhandled = (error: unknown) => errors.push(error) + const reported = vi.spyOn(console, `error`).mockImplementation(() => {}) + const intervals = vi.spyOn(globalThis, `setInterval`) + const clear = vi.spyOn(globalThis, `clearInterval`) + process.on(`unhandledRejection`, recordUnhandled) + const collection = createCollection(setUp(recordApi)) + + try { + await collection.preload() + const timer = intervals.mock.results.at(-1)?.value + expect(timer).toBeDefined() + const expectedRows = new Map([[1, row]]) + if (ending === `buffered-close`) { + for (const id of [2, 3]) { + const value: Data = { id, updated: 0, data: `buffered-${id}` } + expectedRows.set(id, value) + controller.enqueue({ Insert: value }) + } + } + if (ending === `error`) controller.error(failure) + else controller.close() + await new Promise((resolve) => setTimeout(resolve, 0)) + + expect(errors).toEqual([]) + expect(clear).toHaveBeenCalledWith(timer) + expect(stream.locked).toBe(false) + expect(collection.status).toBe(`ready`) + expect(stripState(collection.state)).toEqual(expectedRows) + expect(recordApi.subscribe).toHaveBeenCalledOnce() + expect(recordApi.list).toHaveBeenCalledOnce() + if (ending === `error`) { + expect(reported).toHaveBeenCalledExactlyOnceWith( + `TrailBase subscription failed`, + failure, + ) + } else expect(reported).not.toHaveBeenCalled() + + await collection.cleanup() + await new Promise((resolve) => setTimeout(resolve, 0)) + expect(errors).toEqual([]) + } finally { + await collection.cleanup() + await new Promise((resolve) => setTimeout(resolve, 0)) + process.off(`unhandledRejection`, recordUnhandled) + reported.mockRestore() + intervals.mockRestore() + clear.mockRestore() + } }, ) - readOp = vi.fn((_id: RecordId, _opt?: ReadOpts): ReadOperation => { - throw `readOp` - }) - create = vi.fn((_record: T): Promise => { - throw `create` - }) - createBulk = vi.fn((_records: Array): Promise> => { - throw `createBulk` - }) - createOp = vi.fn((_record: T): CreateOperation => { - throw `createOp` - }) + it(`marks initial sync ready only after its rows are applied`, async () => { + const recordApi = new MockRecordApi() + let resolveList!: (response: ListResponse) => void + recordApi.list.mockReturnValue( + new Promise>((resolve) => { + resolveList = resolve + }), + ) + recordApi.subscribe.mockResolvedValue(new TransformStream().readable) + + let resolvePersistence!: () => void + const persistence = new Promise((resolve) => { + resolvePersistence = resolve + }) + const collection = createCollection(setUp(recordApi)) + const transaction = createTransaction({ + mutationFn: () => persistence, + }) + const preload = collection.preload() + + try { + await vi.waitFor(() => expect(recordApi.list).toHaveBeenCalledOnce()) + transaction.mutate(() => + collection.insert({ id: 2, updated: 0, data: `local` }), + ) + expect(transaction.state).toBe(`persisting`) - update = vi.fn((_id: string | number, _record: Partial): Promise => { - throw `update` + resolveList({ + records: [{ id: 1, updated: 0, data: `server` }], + }) + await Promise.resolve() + await Promise.resolve() + + expect(collection.status).toBe(`loading`) + expect(collection.get(1)).toBeUndefined() + + resolvePersistence() + await transaction.isPersisted.promise + await preload + + expect(collection.status).toBe(`ready`) + expect(collection.get(1)).toEqual( + expect.objectContaining({ + id: 1, + updated: 0, + data: `server`, + }), + ) + } finally { + resolveList({ records: [] }) + resolvePersistence() + await transaction.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + await Promise.allSettled([preload]) + } }) - updateOp = vi.fn((_id: RecordId, _record: Partial): UpdateOperation => { - throw `updateOp` + + it(`settles preload when wildcard subscription startup fails`, async () => { + await expectWildcardFailureSettlesPreload() }) - delete = vi.fn((_id: string | number): Promise => { - throw `delete` + it(`cancels its event subscription when the collection is cleaned up`, async () => { + const recordApi = new MockRecordApi() + const cancel = vi.fn() + recordApi.subscribe.mockResolvedValue(new ReadableStream({ cancel })) + const collection = createCollection(setUp(recordApi)) + + await vi.waitFor(() => expect(recordApi.subscribe).toHaveBeenCalledOnce()) + await collection.cleanup() + + await vi.waitFor(() => expect(cancel).toHaveBeenCalledOnce()) }) - deleteOp = vi.fn((_id: RecordId): DeleteOperation => { - throw `deleteOp` + + it(`ignores an initial fetch that resolves after cleanup`, async () => { + const recordApi = new MockRecordApi() + let resolveList!: (response: ListResponse) => void + recordApi.list.mockReturnValue( + new Promise>((resolve) => { + resolveList = resolve + }), + ) + recordApi.subscribe.mockResolvedValue(new TransformStream().readable) + const options = setUp(recordApi) + const collection = createCollection(options) + + await vi.waitFor(() => expect(recordApi.list).toHaveBeenCalledOnce()) + await collection.cleanup() + resolveList({ + records: [{ id: 1, updated: 0, data: `late` }], + }) + await new Promise((resolve) => setTimeout(resolve, 0)) + + expect(stripState(collection.state)).toEqual(new Map()) + expect(options.sync.getSyncMetadata?.()).toMatchObject({ + fullSyncComplete: false, + }) }) - subscribe = vi.fn((_id: string | number): Promise> => { - return Promise.resolve( - new ReadableStream({ - start: (controller: ReadableStreamDefaultController) => { - controller.close() - }, + it(`ignores a subset page that resolves after its request is aborted`, async () => { + const recordApi = new MockRecordApi() + let resolveList!: (response: ListResponse) => void + recordApi.list.mockReturnValue( + new Promise>((resolve) => { + resolveList = resolve }), ) + recordApi.subscribe.mockResolvedValue(new TransformStream().readable) + const collection = createCollection( + trailBaseCollectionOptions({ + recordApi, + getKey: (item: Data) => item.id ?? -1, + startSync: true, + syncMode: `on-demand`, + parse: {}, + serialize: {}, + }), + ) + const abortController = new AbortController() + + try { + await vi.waitFor(() => expect(collection.status).toBe(`ready`)) + const load = collection._sync.loadSubset({ + signal: abortController.signal, + }) + expect(recordApi.list).toHaveBeenCalledOnce() + abortController.abort() + resolveList({ + records: [{ id: 1, updated: 0, data: `obsolete` }], + }) + if (load instanceof Promise) await load + + expect(stripState(collection.state)).toEqual(new Map()) + } finally { + resolveList({ records: [] }) + await collection.cleanup() + } }) - subscribeAll = vi.fn( - (_opts?: SubscribeOpts): Promise> => { - throw `subscribeAll` - }, - ) -} -function setUp(recordApi: MockRecordApi) { - // Get the options with utilities - const options = trailBaseCollectionOptions({ - recordApi, - getKey: (item: Data): number | number => - item.id ?? Math.round(Math.random() * 100000), - startSync: true, - parse: {}, - serialize: {}, + it(`does not publish a parked subset page after its request is aborted`, async () => { + const recordApi = new MockRecordApi() + recordApi.list.mockResolvedValue({ + records: [{ id: 1, updated: 0, data: `obsolete` }], + }) + recordApi.subscribe.mockResolvedValue(new TransformStream().readable) + const collection = createCollection( + trailBaseCollectionOptions({ + recordApi, + getKey: (item: Data) => item.id ?? -1, + startSync: true, + syncMode: `on-demand`, + parse: {}, + serialize: {}, + }), + ) + let resolvePersistence!: () => void + const persistence = new Promise((resolve) => { + resolvePersistence = resolve + }) + const transaction = createTransaction({ + mutationFn: () => persistence, + }) + const abortController = new AbortController() + + try { + await vi.waitFor(() => expect(collection.status).toBe(`ready`)) + transaction.mutate(() => + collection.insert({ id: 2, updated: 0, data: `local` }), + ) + const load = collection._sync.loadSubset({ + signal: abortController.signal, + }) + await vi.waitFor(() => expect(recordApi.list).toHaveBeenCalledOnce()) + await Promise.resolve() + await Promise.resolve() + + expect(collection.get(1)).toBeUndefined() + abortController.abort() + resolvePersistence() + await transaction.isPersisted.promise + if (load === true) { + throw new Error(`Expected a pending applied receipt`) + } + await expect(load).rejects.toMatchObject({ name: `AbortError` }) + + expect(collection.get(1)).toBeUndefined() + expect(recordApi.list).toHaveBeenCalledOnce() + } finally { + abortController.abort() + resolvePersistence() + await transaction.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } }) - return options -} + it(`fetches later subset pages while earlier pages wait to apply`, async () => { + const recordApi = new MockRecordApi() + recordApi.list.mockImplementation(async () => { + const start = recordApi.list.mock.calls.length === 1 ? 1 : 257 + const count = start === 1 ? 256 : 1 + return { + records: Array.from({ length: count }, (_, index) => ({ + id: start + index, + updated: 0, + data: `remote`, + })), + cursor: `page-${start}`, + } + }) + recordApi.subscribe.mockResolvedValue(new TransformStream().readable) + const collection = createCollection( + trailBaseCollectionOptions({ + recordApi, + getKey: (item: Data) => item.id ?? -1, + startSync: true, + syncMode: `on-demand`, + parse: {}, + serialize: {}, + }), + ) + let resolvePersistence!: () => void + const persistence = new Promise((resolve) => { + resolvePersistence = resolve + }) + const transaction = createTransaction({ mutationFn: () => persistence }) + + try { + await vi.waitFor(() => expect(collection.status).toBe(`ready`)) + transaction.mutate(() => + collection.insert({ id: 999, updated: 0, data: `local` }), + ) + const load = collection._sync.loadSubset({ limit: 257 }) + + await vi.waitFor(() => expect(recordApi.list).toHaveBeenCalledTimes(2)) + expect(collection.get(1)).toBeUndefined() + + resolvePersistence() + await transaction.isPersisted.promise + if (load instanceof Promise) await load + + expect(collection.get(1)?.data).toBe(`remote`) + expect(collection.get(257)?.data).toBe(`remote`) + } finally { + resolvePersistence() + await transaction.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } + }) -describe(`TrailBase Integration`, () => { it(`initial fetch, receive update and cancel`, async () => { const records: Array = [ { diff --git a/packages/vue-db/CHANGELOG.md b/packages/vue-db/CHANGELOG.md index 7a4bebe4eb..d2b6999c63 100644 --- a/packages/vue-db/CHANGELOG.md +++ b/packages/vue-db/CHANGELOG.md @@ -1,5 +1,170 @@ # @tanstack/vue-db +## 0.1.10 + +### Patch Changes + +- Updated dependencies [[`cfb01ce`](https://github.com/TanStack/db/commit/cfb01cee34de7d0378e008dc8c01c1df5253c1e2)]: + - @tanstack/db@0.9.0 + +## 0.1.9 + +### Patch Changes + +- Updated dependencies [[`bbc9edf`](https://github.com/TanStack/db/commit/bbc9edf28ef707c8fb3c451fe2c4724039a0037a)]: + - @tanstack/db@0.8.7 + +## 0.1.8 + +### Patch Changes + +- Updated dependencies [[`ae2fe74`](https://github.com/TanStack/db/commit/ae2fe74e4cb4e74500a90034e6db7987bbd90bd8)]: + - @tanstack/db@0.8.6 + +## 0.1.7 + +### Patch Changes + +- Updated dependencies [[`d8defd2`](https://github.com/TanStack/db/commit/d8defd2a8eb96162cbd4e24970d519eac217bb95), [`9ad882f`](https://github.com/TanStack/db/commit/9ad882f71872aa2210b93ea93d084bd08bedb6a4), [`8c5838d`](https://github.com/TanStack/db/commit/8c5838ddd5f08b3c298d4458cae1ce599af80624)]: + - @tanstack/db@0.8.5 + +## 0.1.6 + +### Patch Changes + +- Updated dependencies [[`3131de1`](https://github.com/TanStack/db/commit/3131de14507006f72631947a61e040b1523d417f), [`8f432ba`](https://github.com/TanStack/db/commit/8f432ba226df2a27d67498ddd1df8468f93ff776)]: + - @tanstack/db@0.8.4 + +## 0.1.5 + +### Patch Changes + +- Updated dependencies [[`99ba511`](https://github.com/TanStack/db/commit/99ba5113b21fd850a8f3e517e5d44ea42ac9f984), [`43cc741`](https://github.com/TanStack/db/commit/43cc741842ae3689128c308be19069b062642f12)]: + - @tanstack/db@0.8.3 + +## 0.1.4 + +### Patch Changes + +- Updated dependencies [[`c521b5d`](https://github.com/TanStack/db/commit/c521b5d6503d8fdf03574b9f9791143e59d34204)]: + - @tanstack/db@0.8.2 + +## 0.1.3 + +### Patch Changes + +- Updated dependencies [[`5d9335d`](https://github.com/TanStack/db/commit/5d9335d0d42c1cc1ec2b92be8ce40ae8abe42827), [`a20352a`](https://github.com/TanStack/db/commit/a20352a9a7b64c9708bef9a1dfb90c96426b6730)]: + - @tanstack/db@0.8.1 + +## 0.1.2 + +### Patch Changes + +- Updated dependencies [[`4b9e8cd`](https://github.com/TanStack/db/commit/4b9e8cdf79551734cf526e6fa4bbdba42ec94575)]: + - @tanstack/db@0.8.0 + +## 0.1.1 + +### Patch Changes + +- Update agent skills to match current APIs and behavior. ([#1696](https://github.com/TanStack/db/pull/1696)) + +- Updated dependencies [[`5f63996`](https://github.com/TanStack/db/commit/5f63996b0febd4775fb641f50975f8f0d442dc00)]: + - @tanstack/db@0.7.2 + +## 0.1.0 + +### Minor Changes + +- Add `useLiveInfiniteQuery` as a Vue binding over the shared live-query window controller. Align infinite-query behavior across React, Vue, and Svelte, including awaitable page fetches, safe page sizes, reactive page-depth preservation, ordered collection validation, shared input resolution, and shared-window cleanup. ([#1724](https://github.com/TanStack/db/pull/1724)) + +### Patch Changes + +- Updated dependencies [[`424382b`](https://github.com/TanStack/db/commit/424382b3a80c6b3556701b433c26c8a60fc8d1af)]: + - @tanstack/db@0.7.1 + +## 0.0.129 + +### Patch Changes + +- Add an internal shared live-query observer and migrate all five framework adapters to it ([#1642](https://github.com/TanStack/db/pull/1642)) + + Introduces `createLiveQueryObserver` in `@tanstack/db`: given a resolved live-query collection (or `null` for a disabled query) it owns the subscription lifecycle every adapter used to re-implement — change and status subscriptions, a snapshot with stable identity per state revision for wholesale consumers, and delivery of the raw `ChangeMessage[]` for granular consumers. React, Vue, Svelte, Solid, and Angular's live-query hooks now materialize from the observer instead of their own hand-rolled subscription/status/snapshot machinery, keeping each adapter's native reactivity and each adapter's data-loading policy (wholesale adapters subscribe without initial state; granular adapters seed from it). + + The observer is an **internal, unstable contract** for TanStack DB's official adapters — it is exported so the adapter packages can consume it, but it is not a public extension point yet and its API may change in any release. + + The migration also fixes several live-query lifecycle defects: status-only transitions (`error`, `cleaned-up`) now reach mounted consumers; snapshot identity is stable across unsubscribe/resubscribe and stays fresh while detached; dispatch is FIFO and non-reentrant with subscriptions identified by record rather than callback; disposing during the synchronous initial replay no longer leaks the collection subscription; subscribing after dispose throws instead of registering a dead listener; Solid guards its async resource continuations against superseded collections; and constructing an observer no longer activates sync. Observers activate on their first committed subscription unless an adapter has already started a pre-created collection supplied directly or returned from a callback. + +- Updated dependencies [[`ad88d07`](https://github.com/TanStack/db/commit/ad88d0751db9723dfb9f164ebfcef88d52b6efa3), [`7e7abda`](https://github.com/TanStack/db/commit/7e7abda73a7ab313f9ec6a413fad00f300e79fb3), [`dc53f0e`](https://github.com/TanStack/db/commit/dc53f0ecbc38e173af68d829ff2de97531494722)]: + - @tanstack/db@0.7.0 + +## 0.0.128 + +### Patch Changes + +- Updated dependencies [[`8ee783d`](https://github.com/TanStack/db/commit/8ee783d7aed9bd5585c182607581305374b8904f)]: + - @tanstack/db@0.6.17 + +## 0.0.127 + +### Patch Changes + +- Updated dependencies [[`8258d09`](https://github.com/TanStack/db/commit/8258d0955ab47c8510bd49ea59bcdbefd2ae054d), [`286964d`](https://github.com/TanStack/db/commit/286964d72612b59e3e427baabd9870f5a71a4281)]: + - @tanstack/db@0.6.16 + +## 0.0.126 + +### Patch Changes + +- Extract shared live-query adapter helpers into `@tanstack/db` ([#1641](https://github.com/TanStack/db/pull/1641)) + + Adds `isCollection`, `isSingleResultCollection`, and `getLiveQueryStatusFlags` to `@tanstack/db` and migrates all five framework adapters to use them. `isCollection` replaces the per-adapter duck-typing and Solid's `instanceof CollectionImpl` with one structural, multi-realm-safe guard (the `instanceof` form gave false negatives across dual-package boundaries). No behavior change; internal deduplication only. + +- Updated dependencies [[`eabcea7`](https://github.com/TanStack/db/commit/eabcea743fdfa045a2db01e12bef87403613102a), [`6d4c096`](https://github.com/TanStack/db/commit/6d4c096395b7ff3f428122ea8842bbead551a8c9)]: + - @tanstack/db@0.6.15 + +## 0.0.125 + +### Patch Changes + +- Updated dependencies [[`397e12a`](https://github.com/TanStack/db/commit/397e12a1224ad563e20a331eebcbe904cd4af948)]: + - @tanstack/db@0.6.14 + +## 0.0.124 + +### Patch Changes + +- Updated dependencies [[`99e9afe`](https://github.com/TanStack/db/commit/99e9afed46ab4083d66609a3e37ee44103c2177f), [`816b667`](https://github.com/TanStack/db/commit/816b6671c2cc9806715f6e6ed4410b3f4efb5afb)]: + - @tanstack/db@0.6.13 + +## 0.0.123 + +### Patch Changes + +- Updated dependencies [[`2b27dd1`](https://github.com/TanStack/db/commit/2b27dd1448da71c78a48e2390cb71b0ada1b1488)]: + - @tanstack/db@0.6.12 + +## 0.0.122 + +### Patch Changes + +- Updated dependencies [[`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`36fb29a`](https://github.com/TanStack/db/commit/36fb29ad7e906d39b6afdba2fd31e369c601bbb0), [`d79b0cd`](https://github.com/TanStack/db/commit/d79b0cd3fd20c1f7e2525e90121752fb6bee314c), [`ac09b11`](https://github.com/TanStack/db/commit/ac09b1177a100eafa85cba3cd09dd1f53f933ded)]: + - @tanstack/db@0.6.11 + +## 0.0.121 + +### Patch Changes + +- Updated dependencies [[`307fdf8`](https://github.com/TanStack/db/commit/307fdf80f522a39a50e316316b3b75ba27fd5e84)]: + - @tanstack/db@0.6.10 + +## 0.0.120 + +### Patch Changes + +- Updated dependencies [[`2147345`](https://github.com/TanStack/db/commit/2147345236ceee6e73d9fc6c0cdc2385833199fc), [`00389a4`](https://github.com/TanStack/db/commit/00389a47b258ad58fc3a03c5cc6f66957b9bd2d1)]: + - @tanstack/db@0.6.9 + ## 0.0.119 ### Patch Changes diff --git a/packages/vue-db/README.md b/packages/vue-db/README.md index c468f120b4..2ebddcff54 100644 --- a/packages/vue-db/README.md +++ b/packages/vue-db/README.md @@ -1,3 +1,20 @@ +

                            + + + + TanStack Vue DB + +
                            # @tanstack/vue-db Vue composables for TanStack DB. See [TanStack/db](https://github.com/TanStack/db) for more details. diff --git a/packages/vue-db/package.json b/packages/vue-db/package.json index f35759d05d..42bb6a6bf1 100644 --- a/packages/vue-db/package.json +++ b/packages/vue-db/package.json @@ -1,6 +1,6 @@ { "name": "@tanstack/vue-db", - "version": "0.0.119", + "version": "0.1.10", "description": "Vue integration for @tanstack/db", "author": "Kyle Mathews", "license": "MIT", diff --git a/packages/vue-db/skills/vue-db/SKILL.md b/packages/vue-db/skills/vue-db/SKILL.md index 650af931c1..3779be9a5d 100644 --- a/packages/vue-db/skills/vue-db/SKILL.md +++ b/packages/vue-db/skills/vue-db/SKILL.md @@ -10,7 +10,7 @@ description: > type: framework library: db framework: vue -library_version: '0.6.0' +library_version: '0.6.17' requires: - db-core sources: diff --git a/packages/vue-db/src/index.ts b/packages/vue-db/src/index.ts index 681078b9c2..9016993064 100644 --- a/packages/vue-db/src/index.ts +++ b/packages/vue-db/src/index.ts @@ -1,5 +1,6 @@ // Re-export all public APIs export * from './useLiveQuery' +export * from './useLiveInfiniteQuery' // Re-export everything from @tanstack/db export * from '@tanstack/db' diff --git a/packages/vue-db/src/useLiveInfiniteQuery.ts b/packages/vue-db/src/useLiveInfiniteQuery.ts new file mode 100644 index 0000000000..dd012ecbc3 --- /dev/null +++ b/packages/vue-db/src/useLiveInfiniteQuery.ts @@ -0,0 +1,263 @@ +import { computed, shallowRef, toValue, watchEffect } from 'vue' +import { + assertLiveQueryWindowManyResult, + compareLiveQueryWindowDependencies, + createLiveQueryCollection, + createLiveQueryWindowController, + fetchNextLiveQueryWindowPage, + getLiveQueryWindowCollectionWarning, + normalizeLiveQueryWindowPageSize, + resolveLiveQueryWindowInput, + shouldPreserveLiveQueryWindowPageCount, +} from '@tanstack/db' +import type { + Collection, + CollectionStatus, + Context, + GetResult, + InferResultType, + InitialQueryBuilder, + NonSingleResult, + QueryBuilder, + UtilsRecord, +} from '@tanstack/db' +import type { ComputedRef, MaybeRefOrGetter } from 'vue' + +const DEFAULT_GC_TIME_MS = 1 + +type InternalCollection = Collection + +type PreviousController = { + getSnapshot: () => { pages: ReadonlyArray> } +} + +type InfiniteQueryOptions = { + pageSize?: number + /** First result-page label, not a server cursor or remote offset. */ + initialPageParam?: number +} + +// Keep the generic parameter for existing typed config wrappers. +export type LiveInfiniteQueryConfig<_TRow> = InfiniteQueryOptions + +export type UseLiveInfiniteQueryConfig< + TContext extends Context & NonSingleResult, +> = LiveInfiniteQueryConfig[number]> + +export interface UseLiveInfiniteQueryReturn< + TContext extends Context & NonSingleResult, +> { + state: ComputedRef>> + data: ComputedRef> + collection: ComputedRef< + Collection, string | number, UtilsRecord> + > + status: ComputedRef + isLoading: ComputedRef + isReady: ComputedRef + isIdle: ComputedRef + isError: ComputedRef + isCleanedUp: ComputedRef + pages: ComputedRef[number]>>> + pageParams: ComputedRef> + fetchNextPage: () => Promise + hasNextPage: ComputedRef + isFetchingNextPage: ComputedRef + error: ComputedRef +} + +export interface UseLiveInfiniteQueryReturnWithCollection< + TResult extends object, + TKey extends string | number, + TUtils extends UtilsRecord, +> { + state: ComputedRef> + data: ComputedRef> + collection: ComputedRef> + status: ComputedRef + isLoading: ComputedRef + isReady: ComputedRef + isIdle: ComputedRef + isError: ComputedRef + isCleanedUp: ComputedRef + pages: ComputedRef>> + pageParams: ComputedRef> + fetchNextPage: () => Promise + hasNextPage: ComputedRef + isFetchingNextPage: ComputedRef + error: ComputedRef +} + +/** + * Create a Vue-native reactive view over the shared live-query window + * controller. The query must include an `orderBy` clause. + */ +export function useLiveInfiniteQuery< + TResult extends object, + TKey extends string | number, + TUtils extends UtilsRecord, +>( + liveQueryCollection: MaybeRefOrGetter< + Collection & NonSingleResult + >, + config: LiveInfiniteQueryConfig, +): UseLiveInfiniteQueryReturnWithCollection + +export function useLiveInfiniteQuery< + TContext extends Context & NonSingleResult, +>( + queryFn: (q: InitialQueryBuilder) => QueryBuilder, + config: UseLiveInfiniteQueryConfig, + deps?: Array>, +): UseLiveInfiniteQueryReturn + +export function useLiveInfiniteQuery< + TContext extends Context & NonSingleResult, +>( + queryFnOrCollection: unknown, + config: InfiniteQueryOptions, + deps: Array> = [], +): UseLiveInfiniteQueryReturn { + if (`getNextPageParam` in config) { + throw new Error( + `getNextPageParam is not supported by useLiveInfiniteQuery. Use an on-demand collection and fulfill meta.loadSubsetOptions in queryFn for server pagination.`, + ) + } + let validatedCollection: InternalCollection | null = null + let previousController: PreviousController | null = null + let previousInput: ReturnType< + typeof resolveLiveQueryWindowInput + > | null = null + let previousDependencies: Array | null = null + let previousPageSize: number | null = null + let previousInitialPageParam: number | null = null + + const controller = computed(() => { + const dependencies = deps.map((dependency) => toValue(dependency)) + + const pageSize = normalizeLiveQueryWindowPageSize(config.pageSize) + const initialPageParam = config.initialPageParam ?? 0 + const unwrappedInput = + typeof queryFnOrCollection === `function` + ? queryFnOrCollection + : toValue(queryFnOrCollection) + const input = resolveLiveQueryWindowInput(unwrappedInput) + const dependencyComparison = compareLiveQueryWindowDependencies( + previousDependencies, + dependencies, + ) + const dependenciesChanged = dependencyComparison.changed + const dependenciesStructurallyEqual = dependencyComparison.structurallyEqual + const pageShapeChanged = + previousPageSize !== pageSize || + previousInitialPageParam !== initialPageParam + const sameCollection = + input.kind === `collection` && + previousInput?.kind === `collection` && + previousInput.collection === input.collection + const canPreservePageCount = shouldPreserveLiveQueryWindowPageCount({ + hasPreviousController: previousController !== null, + previousInputKind: previousInput?.kind, + inputKind: input.kind, + sameCollection, + dependenciesChanged, + dependenciesStructurallyEqual, + pageShapeChanged, + }) + const previousPageCount = previousController + ? Math.max(1, previousController.getSnapshot().pages.length) + : 1 + const initialPageCount = canPreservePageCount ? previousPageCount : 1 + + previousInput = input + previousDependencies = [...dependencies] + previousPageSize = pageSize + previousInitialPageParam = initialPageParam + + if (input.kind === `collection`) { + const collection = input.collection + const warning = getLiveQueryWindowCollectionWarning( + collection, + pageSize + 1, + ) + + if (validatedCollection !== collection) { + validatedCollection = collection + if (warning) console.warn(warning) + } + + const currentController = createLiveQueryWindowController(collection, { + pageSize, + initialPageParam, + initialPageCount, + }) + previousController = currentController + return currentController + } + + const collection = createLiveQueryCollection({ + query: input.query.limit(pageSize + 1).offset(0), + startSync: false, + gcTime: DEFAULT_GC_TIME_MS, + }) + assertLiveQueryWindowManyResult(collection) + const currentController = createLiveQueryWindowController(collection, { + pageSize, + initialPageParam, + initialPageCount, + }) + previousController = currentController + return currentController + }) + + const snapshot = shallowRef(controller.value.getSnapshot()) + + watchEffect( + (onInvalidate) => { + const currentController = controller.value + const updateSnapshot = () => { + snapshot.value = currentController.getSnapshot() + } + + updateSnapshot() + const unsubscribe = currentController.subscribe(updateSnapshot) + updateSnapshot() + + onInvalidate(() => { + unsubscribe() + currentController.dispose() + }) + }, + { flush: `sync` }, + ) + + return { + state: computed( + () => snapshot.value.state as Map>, + ), + data: computed(() => snapshot.value.data as InferResultType), + collection: computed( + () => + snapshot.value.collection as Collection< + GetResult, + string | number, + UtilsRecord + >, + ), + status: computed(() => snapshot.value.status as CollectionStatus), + isLoading: computed(() => snapshot.value.isLoading), + isReady: computed(() => snapshot.value.isReady), + isIdle: computed(() => snapshot.value.isIdle), + isError: computed(() => snapshot.value.isError), + isCleanedUp: computed(() => snapshot.value.isCleanedUp), + pages: computed( + () => + snapshot.value.pages as Array[number]>>, + ), + pageParams: computed(() => snapshot.value.pageParams as Array), + fetchNextPage: () => fetchNextLiveQueryWindowPage(controller.value), + hasNextPage: computed(() => snapshot.value.hasNextPage), + isFetchingNextPage: computed(() => snapshot.value.isFetchingNextPage), + error: computed(() => snapshot.value.error), + } +} diff --git a/packages/vue-db/src/useLiveQuery.ts b/packages/vue-db/src/useLiveQuery.ts index 479c92b2b0..40a2fafb9d 100644 --- a/packages/vue-db/src/useLiveQuery.ts +++ b/packages/vue-db/src/useLiveQuery.ts @@ -1,24 +1,28 @@ import { computed, getCurrentInstance, - nextTick, onUnmounted, reactive, ref, toValue, watchEffect, } from 'vue' -import { createLiveQueryCollection } from '@tanstack/db' +import { + createLiveQueryCollection, + createLiveQueryObserver, + isCollection, + isSingleResultCollection, +} from '@tanstack/db' import type { ChangeMessage, Collection, - CollectionConfigSingleRowOption, CollectionStatus, Context, GetResult, InferResultType, InitialQueryBuilder, LiveQueryCollectionConfig, + LiveQueryObserver, NonSingleResult, QueryBuilder, SingleResult, @@ -265,15 +269,10 @@ export function useLiveQuery( } } - // Check if it's already a collection by checking for specific collection methods - const isCollection = - unwrappedParam && - typeof unwrappedParam === `object` && - typeof unwrappedParam.subscribeChanges === `function` && - typeof unwrappedParam.startSyncImmediate === `function` && - typeof unwrappedParam.id === `string` + // Check if it's already a collection + const inputIsCollection = isCollection(unwrappedParam) - if (isCollection) { + if (inputIsCollection) { // Warn when passing a collection directly with on-demand sync mode // In on-demand mode, data is only loaded when queries with predicates request it // Passing the collection directly doesn't provide any predicates, so no data loads @@ -347,9 +346,9 @@ export function useLiveQuery( if (!currentCollection) { return internalData } - const config: CollectionConfigSingleRowOption = - currentCollection.config - return config.singleResult ? internalData[0] : internalData + return isSingleResultCollection(currentCollection) + ? internalData[0] + : internalData }) // Track collection status reactively @@ -365,101 +364,80 @@ export function useLiveQuery( internalData.push(...Array.from(currentCollection.values())) } - // Track current unsubscribe function - let currentUnsubscribe: (() => void) | null = null + // The shared observer owns subscription, the ready-race, and status; Vue + // materializes into its own reactive map (granular) + ordered array. + let currentObserver: LiveQueryObserver | null = null + + const syncFromObserver = ( + observer: LiveQueryObserver, + currentCollection: Collection, + ) => { + status.value = observer.getSnapshot().status as CollectionStatus + syncDataFromCollection(currentCollection) + } // Watch for collection changes and subscribe to updates watchEffect((onInvalidate) => { const currentCollection = collection.value + // Tear down any previous observer. + currentObserver?.dispose() + currentObserver = null + // Handle null collection (disabled query) if (!currentCollection) { status.value = `disabled` as const state.clear() internalData.length = 0 - if (currentUnsubscribe) { - currentUnsubscribe() - currentUnsubscribe = null - } return } - // Update status ref whenever the effect runs - status.value = currentCollection.status - - // Clean up previous subscription - if (currentUnsubscribe) { - currentUnsubscribe() - } + const observer = createLiveQueryObserver(currentCollection) + currentObserver = observer - // Initialize state with current collection data + // Initial rows arrive as the observer's first delta (includeInitialState); + // apply them and every subsequent delta granularly to the reactive map. state.clear() - for (const [key, value] of currentCollection.entries()) { - state.set(key, value) - } - - // Initialize data array in correct order - syncDataFromCollection(currentCollection) - - // Listen for the first ready event to catch status transitions - // that might not trigger change events (fixes async status transition bug) - currentCollection.onFirstReady(() => { - // Use nextTick to ensure Vue reactivity updates properly - nextTick(() => { - status.value = currentCollection.status - }) - }) - // Subscribe to collection changes with granular updates - const subscription = currentCollection.subscribeChanges( - (changes: Array>) => { - // Apply each change individually to the reactive state - for (const change of changes) { - switch (change.type) { - case `insert`: - case `update`: - state.set(change.key, change.value) - break - case `delete`: - state.delete(change.key) - break + const unsubscribe = observer.subscribe( + (changes: Array> | undefined) => { + if (changes) { + for (const change of changes) { + switch (change.type) { + case `insert`: + case `update`: + state.set(change.key, change.value) + break + case `delete`: + state.delete(change.key) + break + } + } + } else { + // Cleanup and other status-only publications carry no row deltas. + // Rebuild the keyed view so it cannot diverge from ordered data. + state.clear() + for (const [key, value] of observer.getSnapshot().state ?? []) { + state.set(key, value) } } - - // Update the data array to maintain sorted order - syncDataFromCollection(currentCollection) - // Update status ref on every change - status.value = currentCollection.status - }, - { - includeInitialState: true, + syncFromObserver(observer, currentCollection) }, ) - - currentUnsubscribe = subscription.unsubscribe.bind(subscription) - - // Preload collection data if not already started - if (currentCollection.status === `idle`) { - currentCollection.preload().catch(console.error) - } + syncFromObserver(observer, currentCollection) // Cleanup when effect is invalidated onInvalidate(() => { - if (currentUnsubscribe) { - currentUnsubscribe() - currentUnsubscribe = null - } + unsubscribe() + observer.dispose() + currentObserver = null }) }) // Cleanup on unmount (only if we're in a component context) const instance = getCurrentInstance() if (instance) { - onUnmounted(() => { - if (currentUnsubscribe) { - currentUnsubscribe() - } - }) + onUnmounted(() => currentObserver?.dispose()) } return { diff --git a/packages/vue-db/tests/conformance.test.ts b/packages/vue-db/tests/conformance.test.ts new file mode 100644 index 0000000000..eef6e0a2f6 --- /dev/null +++ b/packages/vue-db/tests/conformance.test.ts @@ -0,0 +1,220 @@ +/** + * Vue driver for the shared live-query conformance suite. + * + * Realm-sensitive pieces (collection factories, query operators) are imported + * from Vue's `@tanstack/db` and handed to the shared scenarios. Vue composables + * run inside an `effectScope` so `unmount` can dispose them via `scope.stop()`, + * which triggers the `watchEffect` `onInvalidate` cleanup. + * + * `knownGaps` is populated empirically from the run below. + */ +import { + coalesce, + count, + createCollection, + createLiveQueryCollection, + createOptimisticAction, + eq, + gt, + sum, +} from '@tanstack/db' +import { effectScope, nextTick, ref } from 'vue' +import { + mockSyncCollectionOptions, + mockSyncCollectionOptionsNoInitialState, +} from '../../db/tests/utils' +import { useLiveQuery } from '../src/useLiveQuery' +import { runSuite } from '../../db/tests/conformance/suite' +import type { + ConformanceResult, + ControllableHandle, + DeferredSourceHandle, + LiveQueryDriver, + LiveQueryHandle, + QueryBuild, + SourceHandle, +} from '../../db/tests/conformance/contract' + +let sourceSeq = 0 + +function writer(collection: any) { + return (type: `insert` | `update` | `delete`, value: T) => { + collection.utils.begin() + collection.utils.write({ type, value }) + collection.utils.commit() + } +} + +function makeSource( + initialData: ReadonlyArray, +): SourceHandle { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `conformance-vue-${sourceSeq++}`, + getKey: (r) => r.id, + initialData: [...initialData], + }), + ) + const write = writer(collection) + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + } +} + +function makeDeferredSource< + T extends { id: string }, +>(): DeferredSourceHandle { + const collection = createCollection( + mockSyncCollectionOptionsNoInitialState({ + id: `conformance-vue-${sourceSeq++}`, + getKey: (r) => r.id, + }), + ) + collection.startSyncImmediate() + const write = writer(collection) + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + emit: (rows) => { + collection.utils.begin() + rows.forEach((value) => collection.utils.write({ type: `insert`, value })) + collection.utils.commit() + }, + markReady: () => collection.utils.markReady(), + } +} + +function makePrecreated(build: QueryBuild, opts?: { startSync?: boolean }) { + const collection = createLiveQueryCollection({ + query: build as any, + startSync: opts?.startSync ?? true, + }) + return { collection } +} + +function makeErrorSource() { + const collection = createCollection<{ id: string }>({ + id: `conformance-vue-err-${sourceSeq++}`, + getKey: (r) => r.id, + startSync: false, + sync: { + sync: () => { + throw new Error(`conformance: sync failure`) + }, + }, + }) + try { + collection.startSyncImmediate() + } catch { + // expected: engine catches the sync error and sets status to `error` + } + return { collection } +} + +async function settle() { + await nextTick() + await new Promise((resolve) => setTimeout(resolve, 10)) +} + +function makeHandle(result: any, scope: ReturnType) { + const handle: LiveQueryHandle = { + current(): ConformanceResult { + return { + data: result.data?.value, + state: result.state?.value, + status: result.status?.value ?? `idle`, + isReady: Boolean(result.isReady?.value), + isError: Boolean(result.isError?.value), + // vue-db exposes no `isEnabled`; derive it from status (status-derived). + isEnabled: result.status?.value !== `disabled`, + } + }, + flush: settle, + async apply(fn: () => void) { + fn() + await settle() + }, + unmount() { + scope.stop() + }, + } + return handle +} + +function runInScope(fn: () => R): { + result: R + scope: ReturnType +} { + const scope = effectScope() + let result!: R + scope.run(() => { + result = fn() + }) + return { result, scope } +} + +function mount(build: QueryBuild) { + const { result, scope } = runInScope(() => useLiveQuery(build as any)) + return makeHandle(result, scope) +} + +function mountCollection(collection: any) { + const { result, scope } = runInScope(() => useLiveQuery(collection)) + return makeHandle(result, scope) +} + +function mountConfig(build: QueryBuild) { + const { result, scope } = runInScope(() => + useLiveQuery({ query: build } as any), + ) + return makeHandle(result, scope) +} + +function mountDisabled() { + // Vue's disabled convention: the query callback returns undefined. + const { result, scope } = runInScope(() => + useLiveQuery(() => undefined as any), + ) + return makeHandle(result, scope) +} + +function mountControllable

                            ( + build: (q: any, param: P) => any, + initial: P, +): ControllableHandle

                            { + const param = ref(initial) as { value: P } + const { result, scope } = runInScope(() => + useLiveQuery((q: any) => build(q, param.value), [() => param.value]), + ) + const handle = makeHandle(result, scope) + return { + ...handle, + async setParam(next: P) { + param.value = next + await settle() + }, + } +} + +const vueDriver: LiveQueryDriver = { + name: `vue`, + ops: { eq, gt, count, sum, coalesce, createOptimisticAction }, + makeSource, + makeDeferredSource, + makePrecreated, + makeErrorSource, + mount, + mountControllable, + mountCollection, + mountConfig, + mountDisabled, + knownGaps: [], + features: { serverSnapshot: false, suspense: false }, +} + +runSuite(vueDriver) diff --git a/packages/vue-db/tests/infinite-query-conformance.test.ts b/packages/vue-db/tests/infinite-query-conformance.test.ts new file mode 100644 index 0000000000..f41bbbfd61 --- /dev/null +++ b/packages/vue-db/tests/infinite-query-conformance.test.ts @@ -0,0 +1,204 @@ +/** Vue driver for the shared infinite-query conformance suite. */ +import { + BTreeIndex, + createCollection, + createLiveQueryCollection, + gt, +} from '@tanstack/db' +import { effectScope, nextTick, reactive, ref, shallowRef } from 'vue' +import { mockSyncCollectionOptions } from '../../db/tests/utils' +import { runInfiniteQuerySuite } from '../../db/tests/conformance/infinite-suite' +import { makeInfiniteOnDemandSource } from '../../db/tests/conformance/infinite-on-demand' +import { useLiveInfiniteQuery } from '../src/useLiveInfiniteQuery' +import type { + InfiniteQueryConfig, + InfiniteQueryDriver, + InfiniteQueryHandle, +} from '../../db/tests/conformance/infinite-contract' +import type { + QueryBuild, + SourceHandle, +} from '../../db/tests/conformance/contract' + +let sourceSequence = 0 + +function makeSource( + initialData: ReadonlyArray, +): SourceHandle { + const collection = createCollection( + mockSyncCollectionOptions({ + autoIndex: `eager`, + id: `infinite-conformance-vue-${sourceSequence++}`, + getKey: (row) => row.id, + initialData: [...initialData], + }), + ) + const write = (type: `insert` | `update` | `delete`, value: T) => { + collection.utils.begin() + collection.utils.write({ type, value }) + collection.utils.commit() + } + return { + collection, + insert: (row) => write(`insert`, row), + update: (row) => write(`update`, row), + remove: (row) => write(`delete`, row), + } +} + +function makePrecreated(build: QueryBuild) { + return { + collection: createLiveQueryCollection({ query: build as any }), + } +} + +async function settle(): Promise { + await nextTick() + await new Promise((resolve) => setTimeout(resolve, 0)) +} + +function runInScope(fn: () => R) { + const scope = effectScope() + let result!: R + scope.run(() => { + result = fn() + }) + return { result, scope } +} + +function makeHandle( + result: any, + scope: ReturnType, +): InfiniteQueryHandle { + return { + current() { + return { + data: result.data.value, + pages: result.pages.value, + pageParams: result.pageParams.value, + hasNextPage: result.hasNextPage.value, + isFetchingNextPage: result.isFetchingNextPage.value, + error: result.error.value, + status: result.status.value, + collection: result.collection.value, + } + }, + fetchNextPage: () => result.fetchNextPage(), + flush: settle, + async apply(fn) { + fn() + await settle() + }, + unmount() { + scope.stop() + }, + } +} + +function mount(build: QueryBuild, config: InfiniteQueryConfig = {}) { + const { result, scope } = runInScope(() => + useLiveInfiniteQuery(build as any, config as any), + ) + return makeHandle(result, scope) +} + +function mountControllable

                            ( + build: (q: any, param: P) => any, + initial: P, + config: InfiniteQueryConfig = {}, +) { + const param = ref(initial) as { value: P } + const { result, scope } = runInScope(() => + useLiveInfiniteQuery((q: any) => build(q, param.value), config as any, [ + param, + ]), + ) + const handle = makeHandle(result, scope) + return { + ...handle, + setParamSync(next: P) { + param.value = next + }, + } +} + +function mountCollection(collection: any, config: InfiniteQueryConfig = {}) { + const { result, scope } = runInScope(() => + useLiveInfiniteQuery(collection, config as any), + ) + return makeHandle(result, scope) +} + +function mountCollectionControllable( + initial: any, + config: InfiniteQueryConfig = {}, +) { + const collection = shallowRef(initial) + const { result, scope } = runInScope(() => + useLiveInfiniteQuery(collection, config as any), + ) + const handle = makeHandle(result, scope) + return { + ...handle, + replaceCollectionSync(next: any) { + collection.value = next + }, + } +} + +function mountConfigControllable( + build: QueryBuild, + initial: InfiniteQueryConfig, +) { + const config = reactive({ ...initial }) + const { result, scope } = runInScope(() => + useLiveInfiniteQuery(build as any, config as any), + ) + const handle = makeHandle(result, scope) + return { + ...handle, + setConfigSync(next: InfiniteQueryConfig) { + Object.assign(config, next) + }, + } +} + +function mountInputControllable( + collection: any, + build: QueryBuild, + config: InfiniteQueryConfig = {}, +) { + const kind = ref<`collection` | `query`>(`collection`) + const { result, scope } = runInScope(() => + useLiveInfiniteQuery( + (q: any) => (kind.value === `collection` ? collection : build(q)), + config as any, + [kind], + ), + ) + const handle = makeHandle(result, scope) + return { + ...handle, + setInputKindSync(next: `collection` | `query`) { + kind.value = next + }, + } +} + +const vueInfiniteDriver: InfiniteQueryDriver = { + name: `vue`, + gt, + makeSource, + makeOnDemandSource: (data, delay) => + makeInfiniteOnDemandSource({ createCollection, BTreeIndex }, data, delay), + makePrecreated, + mount, + mountControllable, + mountCollection, + mountCollectionControllable, + mountConfigControllable, + mountInputControllable, + knownGaps: [], +} + +runInfiniteQuerySuite(vueInfiniteDriver) diff --git a/packages/vue-db/tests/useLiveInfiniteQuery.test-d.ts b/packages/vue-db/tests/useLiveInfiniteQuery.test-d.ts new file mode 100644 index 0000000000..ab14a64dd8 --- /dev/null +++ b/packages/vue-db/tests/useLiveInfiniteQuery.test-d.ts @@ -0,0 +1,70 @@ +import { describe, expectTypeOf, it } from 'vitest' +import { shallowRef } from 'vue' +import { createCollection, createLiveQueryCollection } from '@tanstack/db' +import { useLiveInfiniteQuery } from '../src/useLiveInfiniteQuery' +import { mockSyncCollectionOptions } from '../../db/tests/utils' +import type { InitialQueryBuilder } from '@tanstack/db' +import type { LiveInfiniteQueryConfig } from '../src/useLiveInfiniteQuery' + +type Post = { + id: string + createdAt: number +} + +describe(`useLiveInfiniteQuery type assertions`, () => { + it(`does not advertise a server-page callback`, () => { + expectTypeOf< + Extract, `getNextPageParam`> + >().toEqualTypeOf() + }) + + it(`preserves query and pre-created collection result types`, () => { + const posts = createCollection( + mockSyncCollectionOptions({ + id: `vue-infinite-types`, + getKey: (post) => post.id, + initialData: [], + }), + ) + + const queryResult = useLiveInfiniteQuery( + (q: InitialQueryBuilder) => + q.from({ posts }).orderBy(({ posts: post }) => post.createdAt, `desc`), + { pageSize: 5 }, + ) + expectTypeOf(queryResult.data.value[0]!.id).toEqualTypeOf() + expectTypeOf(queryResult.data.value[0]!.createdAt).toEqualTypeOf() + expectTypeOf(queryResult.fetchNextPage()).toEqualTypeOf>() + + const livePosts = createLiveQueryCollection((q: InitialQueryBuilder) => + q.from({ posts }).orderBy(({ posts: post }) => post.createdAt, `desc`), + ) + const collectionResult = useLiveInfiniteQuery(shallowRef(livePosts), { + pageSize: 5, + }) + + expectTypeOf(collectionResult.data.value[0]!.id).toEqualTypeOf() + expectTypeOf( + collectionResult.data.value[0]!.createdAt, + ).toEqualTypeOf() + expectTypeOf(collectionResult.state.value.get(`1`)?.id).toEqualTypeOf< + string | undefined + >() + + useLiveInfiniteQuery( + // @ts-expect-error Infinite queries cannot use a single-result query. + (q: InitialQueryBuilder) => + q + .from({ posts }) + .orderBy(({ posts: post }) => post.createdAt, `desc`) + .findOne(), + { pageSize: 5 }, + ) + + useLiveInfiniteQuery( + // @ts-expect-error Infinite queries do not support disabled null queries. + (_q: InitialQueryBuilder) => null, + { pageSize: 5 }, + ) + }) +}) diff --git a/packages/vue-db/tests/useLiveInfiniteQuery.test.ts b/packages/vue-db/tests/useLiveInfiniteQuery.test.ts new file mode 100644 index 0000000000..26a453fed9 --- /dev/null +++ b/packages/vue-db/tests/useLiveInfiniteQuery.test.ts @@ -0,0 +1,164 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { effectScope, nextTick, shallowRef } from 'vue' +import { createCollection, createLiveQueryCollection } from '@tanstack/db' +import { useLiveInfiniteQuery } from '../src/useLiveInfiniteQuery' +import { mockSyncCollectionOptions } from '../../db/tests/utils' +import type { InitialQueryBuilder } from '@tanstack/db' + +type Post = { + id: string + title: string + createdAt: number +} + +function createPosts(count: number): Array { + return Array.from({ length: count }, (_, index) => ({ + id: String(index + 1), + title: `Post ${index + 1}`, + createdAt: count - index, + })) +} + +function createPostsCollection(id: string, count: number) { + return createCollection( + mockSyncCollectionOptions({ + autoIndex: `eager`, + id, + getKey: (post) => post.id, + initialData: createPosts(count), + }), + ) +} + +async function flushVue(): Promise { + await nextTick() + await Promise.resolve() +} + +describe(`useLiveInfiniteQuery`, () => { + it(`rejects a server-page callback before constructing a query`, () => { + const queryFn = vi.fn(() => { + throw new Error(`query must not be constructed`) + }) + const config = { pageSize: 2, getNextPageParam: () => 1 } + const scope = effectScope() + try { + expect(() => + scope.run(() => useLiveInfiniteQuery(queryFn, config)), + ).toThrow(`getNextPageParam is not supported`) + expect(queryFn).not.toHaveBeenCalled() + } finally { + scope.stop() + } + }) + + let cleanup: (() => void) | undefined + + afterEach(() => { + cleanup?.() + cleanup = undefined + vi.restoreAllMocks() + }) + + it(`accepts a reactive ref for a pre-created ordered collection`, async () => { + const posts = createPostsCollection(`vue-infinite-precreated`, 7) + const livePosts = createLiveQueryCollection({ + query: (q) => + q + .from({ posts }) + .orderBy(({ posts: post }) => post.createdAt, `desc`) + .limit(2) + .offset(1), + }) + await livePosts.preload() + const warning = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const scope = effectScope() + const collection = shallowRef(livePosts) + const query = scope.run(() => + useLiveInfiniteQuery(collection, { + pageSize: 3, + }), + ) + cleanup = () => scope.stop() + if (!query) throw new Error(`Failed to mount infinite query`) + // A framework tick does not settle asynchronous window normalization. + await vi.waitFor(() => + expect(livePosts.utils.getWindow()).toEqual({ offset: 0, limit: 4 }), + ) + await flushVue() + + expect(query.collection.value).toBe(livePosts) + expect(query.data.value.map((post) => post.id)).toEqual([`1`, `2`, `3`]) + expect(query.state.value.get(`1`)?.title).toBe(`Post 1`) + expect(query.hasNextPage.value).toBe(true) + expect(warning).toHaveBeenCalledOnce() + expect(livePosts.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) + }) + + it(`resets to the first page when a collection ref changes`, async () => { + const firstPosts = createPostsCollection(`vue-infinite-swap-first`, 8) + const secondPosts = createPostsCollection(`vue-infinite-swap-second`, 4) + const firstQuery = createLiveQueryCollection({ + query: (q) => + q + .from({ posts: firstPosts }) + .orderBy(({ posts: post }) => post.createdAt, `desc`) + .limit(4), + }) + const secondQuery = createLiveQueryCollection({ + query: (q) => + q + .from({ posts: secondPosts }) + .orderBy(({ posts: post }) => post.createdAt, `desc`) + .limit(4), + }) + await Promise.all([firstQuery.preload(), secondQuery.preload()]) + + const scope = effectScope() + const selectedQuery = shallowRef(firstQuery) + const query = scope.run(() => + useLiveInfiniteQuery(selectedQuery, { pageSize: 3 }), + ) + cleanup = () => scope.stop() + if (!query) throw new Error(`Failed to mount infinite query`) + await flushVue() + + await query.fetchNextPage() + await flushVue() + expect(query.pages.value).toHaveLength(2) + + selectedQuery.value = secondQuery + await flushVue() + + expect(query.collection.value).toBe(secondQuery) + expect(query.pages.value).toHaveLength(1) + expect(query.data.value.map((post) => post.createdAt)).toEqual([4, 3, 2]) + }) + + it(`does not recreate a controller through a retained callback after unmount`, async () => { + const posts = createPostsCollection(`vue-infinite-retained-fetch`, 7) + let queryBuilds = 0 + const scope = effectScope() + const query = scope.run(() => + useLiveInfiniteQuery( + (q: InitialQueryBuilder) => { + queryBuilds++ + return q + .from({ posts }) + .orderBy(({ posts: post }) => post.createdAt, `desc`) + }, + { pageSize: 3 }, + ), + ) + if (!query) throw new Error(`Failed to mount infinite query`) + await flushVue() + expect(queryBuilds).toBe(1) + + const retainedFetch = query.fetchNextPage + scope.stop() + await retainedFetch() + await retainedFetch() + + expect(queryBuilds).toBe(1) + }) +}) diff --git a/packages/vue-db/tests/useLiveQuery.test.ts b/packages/vue-db/tests/useLiveQuery.test.ts index 57b8ae57b0..657aec902f 100644 --- a/packages/vue-db/tests/useLiveQuery.test.ts +++ b/packages/vue-db/tests/useLiveQuery.test.ts @@ -99,6 +99,27 @@ async function waitFor(fn: () => void, timeout = 2000, interval = 20) { } describe(`Query Collections`, () => { + it(`keeps data and keyed state aligned after collection cleanup`, async () => { + const collection = createCollection( + mockSyncCollectionOptions({ + id: `cleanup-alignment-vue`, + getKey: (person) => person.id, + initialData: initialPersons, + }), + ) + const result = useLiveQuery(collection) + + await waitForVueUpdate() + expect(result.data.value).toHaveLength(3) + expect(result.state.value.size).toBe(3) + + await collection.cleanup() + await nextTick() + + expect(result.data.value).toHaveLength(0) + expect(result.state.value.size).toBe(0) + }) + it(`should work with basic collection and select`, async () => { const collection = createCollection( mockSyncCollectionOptions({ diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index ad174b46b5..c9512eaf06 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -148,10 +148,10 @@ importers: specifier: ^20.3.16 version: 20.3.16(@angular/common@20.3.16(@angular/core@20.3.17(@angular/compiler@20.3.16)(rxjs@7.8.2)(zone.js@0.15.1))(rxjs@7.8.2))(@angular/core@20.3.17(@angular/compiler@20.3.16)(rxjs@7.8.2)(zone.js@0.15.1))(@angular/platform-browser@20.3.16(@angular/common@20.3.16(@angular/core@20.3.17(@angular/compiler@20.3.16)(rxjs@7.8.2)(zone.js@0.15.1))(rxjs@7.8.2))(@angular/core@20.3.17(@angular/compiler@20.3.16)(rxjs@7.8.2)(zone.js@0.15.1)))(rxjs@7.8.2) '@tanstack/angular-db': - specifier: ^0.1.68 + specifier: ^0.1.89 version: link:../../../packages/angular-db '@tanstack/db': - specifier: ^0.6.8 + specifier: ^0.9.0 version: link:../../../packages/db rxjs: specifier: ^7.8.2 @@ -209,19 +209,19 @@ importers: examples/electron/offline-first: dependencies: '@tanstack/electron-db-sqlite-persistence': - specifier: ^0.1.12 + specifier: ^0.1.33 version: link:../../../packages/electron-db-sqlite-persistence '@tanstack/node-db-sqlite-persistence': - specifier: ^0.2.0 + specifier: ^0.2.21 version: link:../../../packages/node-db-sqlite-persistence '@tanstack/offline-transactions': - specifier: ^1.0.33 + specifier: ^1.0.54 version: link:../../../packages/offline-transactions '@tanstack/query-db-collection': - specifier: ^1.0.40 + specifier: ^1.2.13 version: link:../../../packages/query-db-collection '@tanstack/react-db': - specifier: ^0.1.86 + specifier: ^0.3.8 version: link:../../../packages/react-db '@tanstack/react-query': specifier: ^5.90.20 @@ -300,19 +300,19 @@ importers: specifier: 11.4.1 version: 11.4.1(react-native@0.79.6(@babel/core@7.29.0)(@types/react@19.2.13)(react@19.0.0)) '@tanstack/db': - specifier: ^0.6.8 + specifier: ^0.9.0 version: link:../../../packages/db '@tanstack/offline-transactions': - specifier: ^1.0.33 + specifier: ^1.0.54 version: link:../../../packages/offline-transactions '@tanstack/query-db-collection': - specifier: ^1.0.40 + specifier: ^1.2.13 version: link:../../../packages/query-db-collection '@tanstack/react-db': - specifier: ^0.1.86 + specifier: ^0.3.8 version: link:../../../packages/react-db '@tanstack/react-native-db-sqlite-persistence': - specifier: ^0.2.0 + specifier: ^0.2.21 version: link:../../../packages/react-native-db-sqlite-persistence '@tanstack/react-query': specifier: ^5.90.20 @@ -397,19 +397,19 @@ importers: specifier: 11.4.1 version: 11.4.1(react-native@0.79.6(@babel/core@7.29.0)(@types/react@19.2.13)(react@19.0.0)) '@tanstack/db': - specifier: ^0.6.8 + specifier: ^0.9.0 version: link:../../../packages/db '@tanstack/electric-db-collection': - specifier: ^0.3.6 + specifier: ^0.4.8 version: link:../../../packages/electric-db-collection '@tanstack/offline-transactions': - specifier: ^1.0.33 + specifier: ^1.0.54 version: link:../../../packages/offline-transactions '@tanstack/react-db': - specifier: ^0.1.86 + specifier: ^0.3.8 version: link:../../../packages/react-db '@tanstack/react-native-db-sqlite-persistence': - specifier: ^0.2.0 + specifier: ^0.2.21 version: link:../../../packages/react-native-db-sqlite-persistence '@tanstack/react-query': specifier: ^5.90.20 @@ -479,22 +479,56 @@ importers: specifier: ^5.9.2 version: 5.9.3 + examples/react/next-ssr-e2e: + dependencies: + '@tanstack/db': + specifier: ^0.9.0 + version: link:../../../packages/db + '@tanstack/react-db': + specifier: ^0.3.8 + version: link:../../../packages/react-db + next: + specifier: ^16.3.1 + version: 16.3.1(@babel/core@7.29.0)(@playwright/test@1.60.0)(@types/node@25.2.2)(babel-plugin-react-compiler@1.0.0)(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(sass@1.90.0) + react: + specifier: ^19.2.4 + version: 19.2.4 + react-dom: + specifier: ^19.2.4 + version: 19.2.4(react@19.2.4) + devDependencies: + '@playwright/test': + specifier: ^1.60.0 + version: 1.60.0 + '@types/node': + specifier: ^25.2.2 + version: 25.2.2 + '@types/react': + specifier: ^19.2.13 + version: 19.2.13 + '@types/react-dom': + specifier: ^19.2.3 + version: 19.2.3(@types/react@19.2.13) + typescript: + specifier: ^5.9.2 + version: 5.9.3 + examples/react/offline-transactions: dependencies: '@tanstack/browser-db-sqlite-persistence': - specifier: ^0.2.0 + specifier: ^0.2.21 version: link:../../../packages/browser-db-sqlite-persistence '@tanstack/db': - specifier: ^0.6.8 + specifier: ^0.9.0 version: link:../../../packages/db '@tanstack/offline-transactions': - specifier: ^1.0.33 + specifier: ^1.0.54 version: link:../../../packages/offline-transactions '@tanstack/query-db-collection': - specifier: ^1.0.40 + specifier: ^1.2.13 version: link:../../../packages/query-db-collection '@tanstack/react-db': - specifier: ^0.1.86 + specifier: ^0.3.8 version: link:../../../packages/react-db '@tanstack/react-query': specifier: ^5.90.20 @@ -552,10 +586,10 @@ importers: examples/react/paced-mutations-demo: dependencies: '@tanstack/db': - specifier: ^0.6.8 + specifier: ^0.9.0 version: link:../../../packages/db '@tanstack/react-db': - specifier: ^0.1.86 + specifier: ^0.3.8 version: link:../../../packages/react-db mitt: specifier: ^3.0.1 @@ -592,10 +626,10 @@ importers: specifier: ^5.90.20 version: 5.90.20 '@tanstack/query-db-collection': - specifier: ^1.0.40 + specifier: ^1.2.13 version: link:../../../packages/query-db-collection '@tanstack/react-db': - specifier: ^0.1.86 + specifier: ^0.3.8 version: link:../../../packages/react-db '@tanstack/react-router': specifier: ^1.159.5 @@ -620,7 +654,7 @@ importers: version: 11.10.0(typescript@5.9.3) better-auth: specifier: ^1.4.18 - version: 1.4.18(b4f55ef685357933f61fbe0b4095cc16) + version: 1.4.18(dd19f6838762949983acd690aa0ac646) dotenv: specifier: ^17.2.4 version: 17.2.4 @@ -722,19 +756,65 @@ importers: specifier: ^5.1.0 version: 5.1.0 + examples/react/start-ssr-e2e: + dependencies: + '@tanstack/react-db': + specifier: ^0.3.8 + version: link:../../../packages/react-db + '@tanstack/react-router': + specifier: ^1.159.5 + version: 1.159.5(react-dom@19.2.4(react@19.2.4))(react@19.2.4) + '@tanstack/react-router-with-db': + specifier: ^0.1.0 + version: link:../../../packages/react-router-with-db + '@tanstack/react-start': + specifier: ^1.159.5 + version: 1.159.5(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(vite-plugin-solid@2.11.10(@testing-library/jest-dom@6.9.1)(solid-js@1.9.11)(vite@7.3.2(@types/node@25.2.2)(jiti@2.6.1)(lightningcss@1.30.2)(sass@1.90.0)(terser@5.44.0)(tsx@4.21.0)(yaml@2.8.1)))(vite@7.3.2(@types/node@25.2.2)(jiti@2.6.1)(lightningcss@1.30.2)(sass@1.90.0)(terser@5.44.0)(tsx@4.21.0)(yaml@2.8.1)) + react: + specifier: ^19.2.4 + version: 19.2.4 + react-dom: + specifier: ^19.2.4 + version: 19.2.4(react@19.2.4) + vite-tsconfig-paths: + specifier: ^5.1.4 + version: 5.1.4(typescript@5.9.3)(vite@7.3.2(@types/node@25.2.2)(jiti@2.6.1)(lightningcss@1.30.2)(sass@1.90.0)(terser@5.44.0)(tsx@4.21.0)(yaml@2.8.1)) + devDependencies: + '@playwright/test': + specifier: ^1.60.0 + version: 1.60.0 + '@types/node': + specifier: ^25.2.2 + version: 25.2.2 + '@types/react': + specifier: ^19.2.13 + version: 19.2.13 + '@types/react-dom': + specifier: ^19.2.3 + version: 19.2.3(@types/react@19.2.13) + '@vitejs/plugin-react': + specifier: ^5.1.3 + version: 5.1.3(vite@7.3.2(@types/node@25.2.2)(jiti@2.6.1)(lightningcss@1.30.2)(sass@1.90.0)(terser@5.44.0)(tsx@4.21.0)(yaml@2.8.1)) + typescript: + specifier: ^5.9.2 + version: 5.9.3 + vite: + specifier: ^7.3.0 + version: 7.3.2(@types/node@25.2.2)(jiti@2.6.1)(lightningcss@1.30.2)(sass@1.90.0)(terser@5.44.0)(tsx@4.21.0)(yaml@2.8.1) + examples/react/todo: dependencies: '@tanstack/electric-db-collection': - specifier: ^0.3.6 + specifier: ^0.4.8 version: link:../../../packages/electric-db-collection '@tanstack/query-core': specifier: ^5.90.20 version: 5.90.20 '@tanstack/query-db-collection': - specifier: ^1.0.40 + specifier: ^1.2.13 version: link:../../../packages/query-db-collection '@tanstack/react-db': - specifier: ^0.1.86 + specifier: ^0.3.8 version: link:../../../packages/react-db '@tanstack/react-router': specifier: ^1.159.5 @@ -743,7 +823,7 @@ importers: specifier: ^1.159.5 version: 1.159.5(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(vite-plugin-solid@2.11.10(@testing-library/jest-dom@6.9.1)(solid-js@1.9.11)(vite@7.3.2(@types/node@25.2.2)(jiti@2.6.1)(lightningcss@1.30.2)(sass@1.90.0)(terser@5.44.0)(tsx@4.21.0)(yaml@2.8.1)))(vite@7.3.2(@types/node@25.2.2)(jiti@2.6.1)(lightningcss@1.30.2)(sass@1.90.0)(terser@5.44.0)(tsx@4.21.0)(yaml@2.8.1)) '@tanstack/trailbase-db-collection': - specifier: ^0.1.86 + specifier: ^0.1.107 version: link:../../../packages/trailbase-db-collection cors: specifier: ^2.8.6 @@ -846,16 +926,16 @@ importers: examples/solid/todo: dependencies: '@tanstack/electric-db-collection': - specifier: ^0.3.6 + specifier: ^0.4.8 version: link:../../../packages/electric-db-collection '@tanstack/query-core': specifier: ^5.90.20 version: 5.90.20 '@tanstack/query-db-collection': - specifier: ^1.0.40 + specifier: ^1.2.13 version: link:../../../packages/query-db-collection '@tanstack/solid-db': - specifier: ^0.2.22 + specifier: ^0.2.43 version: link:../../../packages/solid-db '@tanstack/solid-router': specifier: ^1.159.5 @@ -864,7 +944,7 @@ importers: specifier: ^1.159.5 version: 1.159.5(@tanstack/react-router@1.159.5(react-dom@19.2.4(react@19.2.4))(react@19.2.4))(solid-js@1.9.11)(vite-plugin-solid@2.11.10(@testing-library/jest-dom@6.9.1)(solid-js@1.9.11)(vite@7.3.2(@types/node@25.2.2)(jiti@2.6.1)(lightningcss@1.30.2)(sass@1.90.0)(terser@5.44.0)(tsx@4.21.0)(yaml@2.8.1)))(vite@7.3.2(@types/node@25.2.2)(jiti@2.6.1)(lightningcss@1.30.2)(sass@1.90.0)(terser@5.44.0)(tsx@4.21.0)(yaml@2.8.1)) '@tanstack/trailbase-db-collection': - specifier: ^0.1.86 + specifier: ^0.1.107 version: link:../../../packages/trailbase-db-collection cors: specifier: ^2.8.6 @@ -1196,13 +1276,13 @@ importers: '@tanstack/db': specifier: workspace:* version: link:../db - '@tanstack/store': - specifier: ^0.9.2 - version: 0.9.2 debug: specifier: ^4.4.3 version: 4.4.3 devDependencies: + '@tanstack/query-core': + specifier: ^5.90.20 + version: 5.90.20 '@types/debug': specifier: ^4.1.12 version: 4.1.12 @@ -1338,11 +1418,11 @@ importers: version: 4.0.1 devDependencies: '@powersync/common': - specifier: 1.49.0 - version: 1.49.0 + specifier: 1.57.0 + version: 1.57.0 '@powersync/node': - specifier: 0.18.1 - version: 0.18.1(@powersync/common@1.49.0)(better-sqlite3@12.8.0) + specifier: 0.19.2 + version: 0.19.2(@powersync/common@1.57.0)(better-sqlite3@12.8.0) '@types/debug': specifier: ^4.1.12 version: 4.1.12 @@ -1431,6 +1511,30 @@ importers: specifier: ^12.6.2 version: 12.8.0 + packages/react-router-with-db: + devDependencies: + '@tanstack/react-db': + specifier: workspace:* + version: link:../react-db + '@tanstack/react-router': + specifier: ^1.159.5 + version: 1.159.5(react-dom@19.2.4(react@19.2.4))(react@19.2.4) + '@tanstack/router-core': + specifier: ^1.159.4 + version: 1.159.4 + '@vitejs/plugin-react': + specifier: ^5.1.3 + version: 5.1.3(vite@7.3.2(@types/node@25.2.2)(jiti@2.6.1)(lightningcss@1.30.2)(sass@1.90.0)(terser@5.44.0)(tsx@4.21.0)(yaml@2.8.1)) + '@vitest/coverage-istanbul': + specifier: ^3.2.4 + version: 3.2.4(vitest@3.2.4) + react: + specifier: ^19.2.4 + version: 19.2.4 + react-dom: + specifier: ^19.2.4 + version: 19.2.4(react@19.2.4) + packages/rxdb-db-collection: dependencies: '@standard-schema/spec': @@ -2586,6 +2690,9 @@ packages: '@emnapi/core@1.7.1': resolution: {integrity: sha512-o1uhUASyo921r2XtHYOHy7gdkGLge8ghBEQHMWmyJFoXlpU58kIrhhN3w26lpQb6dspetweapMn2CSNwQ8I4wg==} + '@emnapi/runtime@1.11.3': + resolution: {integrity: sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==} + '@emnapi/runtime@1.7.1': resolution: {integrity: sha512-PVtJr5CmLwYAU9PZDMITZoR5iAOShYREoR45EyyLrbntV50mdePTgUn4AmOw90Ifcj+x2kRjdzr1HP3RrNiHGA==} @@ -3905,70 +4012,145 @@ packages: cpu: [arm64] os: [darwin] + '@img/sharp-darwin-arm64@0.35.3': + resolution: {integrity: sha512-RMnFX7YQsMoh7lWfcM4NEHHymBX/rLuKNPVM84XE9ONPcaSCDgE7CHIHpSgPcO2xcRthgBy1HfNO319mwhIAkg==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [darwin] + '@img/sharp-darwin-x64@0.34.5': resolution: {integrity: sha512-YNEFAF/4KQ/PeW0N+r+aVVsoIY0/qxxikF2SWdp+NRkmMB7y9LBZAVqQ4yhGCm/H3H270OSykqmQMKLBhBJDEw==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [x64] os: [darwin] + '@img/sharp-darwin-x64@0.35.3': + resolution: {integrity: sha512-Xo+5uFBtLN0BKqieTxiFzFPQAUlBbbH5iBKyRX/z1JrbnYsHTfKJnUfL8+p2TPXr1pXqao4eeL4Rl144uDpK9w==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [darwin] + + '@img/sharp-freebsd-wasm32@0.35.3': + resolution: {integrity: sha512-lUxcqWIj2wMQ9BrwNjngcr1gWUr5xgaGThBRqPPalIC2n67Cqj1uPh8NnA/ZhAg8hUbKl+kVHKwgUIwe6ZYPrg==} + engines: {node: '>=20.9.0'} + os: [freebsd] + '@img/sharp-libvips-darwin-arm64@1.2.4': resolution: {integrity: sha512-zqjjo7RatFfFoP0MkQ51jfuFZBnVE2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==} cpu: [arm64] os: [darwin] + '@img/sharp-libvips-darwin-arm64@1.3.2': + resolution: {integrity: sha512-9J6ypZFpQBj4YnePGoq/S38w6nz+vqg5WZLrLGY4YuSemdMq47GMLBPO42MzwdGwpg/agZ7xzZcFHa48xlywfg==} + cpu: [arm64] + os: [darwin] + '@img/sharp-libvips-darwin-x64@1.2.4': resolution: {integrity: sha512-1IOd5xfVhlGwX+zXv2N93k0yMONvUlANylbJw1eTah8K/Jtpi15KC+WSiaX/nBmbm2HxRM1gZ0nSdjSsrZbGKg==} cpu: [x64] os: [darwin] + '@img/sharp-libvips-darwin-x64@1.3.2': + resolution: {integrity: sha512-m2pW1n6cns9VaubNwsZ+c3CRYjxNQWgJ5gPlnL1nbBcpkBvFm6SCFN5o0psFHI8w9n11NKhFkeEDns98tiqbEw==} + cpu: [x64] + os: [darwin] + '@img/sharp-libvips-linux-arm64@1.2.4': resolution: {integrity: sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==} cpu: [arm64] os: [linux] libc: [glibc] + '@img/sharp-libvips-linux-arm64@1.3.2': + resolution: {integrity: sha512-dqVSFynCox4C/J8kT16V7SIFAns0IjgLwkvYT7p8LQVmJ5OS5b6tI9IGflxTeuBS//zXeFIUbwt5dwxyZ17cnA==} + cpu: [arm64] + os: [linux] + libc: [glibc] + '@img/sharp-libvips-linux-arm@1.2.4': resolution: {integrity: sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==} cpu: [arm] os: [linux] libc: [glibc] + '@img/sharp-libvips-linux-arm@1.3.2': + resolution: {integrity: sha512-1eMLzy92I4J6rmi4mAT8yC3HxOtniyGELlzGbNMLLeqe052ahFQ0h6LFq+lh5DsDIdYViIDst08abvSbcEdLXQ==} + cpu: [arm] + os: [linux] + libc: [glibc] + '@img/sharp-libvips-linux-ppc64@1.2.4': resolution: {integrity: sha512-FMuvGijLDYG6lW+b/UvyilUWu5Ayu+3r2d1S8notiGCIyYU/76eig1UfMmkZ7vwgOrzKzlQbFSuQfgm7GYUPpA==} cpu: [ppc64] os: [linux] libc: [glibc] + '@img/sharp-libvips-linux-ppc64@1.3.2': + resolution: {integrity: sha512-3z0NHDxD6n5I9gc05U1eW1AyRm+Gznzq3naMrthPNqE6oYykcogW0l/jfpJdjYnuNl8R7yI9pNbE1XiUeyq0Aw==} + cpu: [ppc64] + os: [linux] + libc: [glibc] + '@img/sharp-libvips-linux-riscv64@1.2.4': resolution: {integrity: sha512-oVDbcR4zUC0ce82teubSm+x6ETixtKZBh/qbREIOcI3cULzDyb18Sr/Wcyx7NRQeQzOiHTNbZFF1UwPS2scyGA==} cpu: [riscv64] os: [linux] libc: [glibc] + '@img/sharp-libvips-linux-riscv64@1.3.2': + resolution: {integrity: sha512-bsb4rI+NldGOsXuej2r8OdSS8+zXDVaCWxyWrcv6kneTOlgAHtZABRzBBCwdsPiD90J4myNJuHpg6kA20ImW/w==} + cpu: [riscv64] + os: [linux] + libc: [glibc] + '@img/sharp-libvips-linux-s390x@1.2.4': resolution: {integrity: sha512-qmp9VrzgPgMoGZyPvrQHqk02uyjA0/QrTO26Tqk6l4ZV0MPWIW6LTkqOIov+J1yEu7MbFQaDpwdwJKhbJvuRxQ==} cpu: [s390x] os: [linux] libc: [glibc] + '@img/sharp-libvips-linux-s390x@1.3.2': + resolution: {integrity: sha512-/ABshyj8gCpyIrNXnHn4LorDJ0HHm1VhXPBlxZ8zAtfVPAaSafXPGn+sUSIRiwaSBy0mmFjSjiXI5mkcwdChKQ==} + cpu: [s390x] + os: [linux] + libc: [glibc] + '@img/sharp-libvips-linux-x64@1.2.4': resolution: {integrity: sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==} cpu: [x64] os: [linux] libc: [glibc] + '@img/sharp-libvips-linux-x64@1.3.2': + resolution: {integrity: sha512-ITPEtgffGJ0S6G9dRyw/366tJQqFRcHWPHhC+Stpg3Z8AEMrDrTr2lhdz4f/Y/HMbRh//7Z5mBzEpVdi62Oc3w==} + cpu: [x64] + os: [linux] + libc: [glibc] + '@img/sharp-libvips-linuxmusl-arm64@1.2.4': resolution: {integrity: sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==} cpu: [arm64] os: [linux] libc: [musl] + '@img/sharp-libvips-linuxmusl-arm64@1.3.2': + resolution: {integrity: sha512-zE9EdiUzUmg5mDT5a1rk5fYJ6GWPloTwWBYDS14naqHsL+EaMpDj1AWnpLgh3u0YCORv2Tt50wrcrpYqkP97Kw==} + cpu: [arm64] + os: [linux] + libc: [musl] + '@img/sharp-libvips-linuxmusl-x64@1.2.4': resolution: {integrity: sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==} cpu: [x64] os: [linux] libc: [musl] + '@img/sharp-libvips-linuxmusl-x64@1.3.2': + resolution: {integrity: sha512-m0lrLiUt+lBYnCFr8qV/65yMR4E/c7/wf78I5eKTdkEakFAlZ9QlzEM3QIhhAwVeUhLAHLcCq7a7Vszq/oFNZQ==} + cpu: [x64] + os: [linux] + libc: [musl] + '@img/sharp-linux-arm64@0.34.5': resolution: {integrity: sha512-bKQzaJRY/bkPOXyKx5EVup7qkaojECG6NLYswgktOZjaXecSAeCWiZwwiFf3/Y+O1HrauiE3FVsGxFg8c24rZg==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} @@ -3976,6 +4158,13 @@ packages: os: [linux] libc: [glibc] + '@img/sharp-linux-arm64@0.35.3': + resolution: {integrity: sha512-QgKDspHPnrU+GQ55XPhGwyhC8acLVOOSyAvo1oVfFmrIXLkDNmGWzAfDZ4xK8oSA1qBQrALcHX0G5UZni/SuFQ==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [linux] + libc: [glibc] + '@img/sharp-linux-arm@0.34.5': resolution: {integrity: sha512-9dLqsvwtg1uuXBGZKsxem9595+ujv0sJ6Vi8wcTANSFpwV/GONat5eCkzQo/1O6zRIkh0m/8+5BjrRr7jDUSZw==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} @@ -3983,6 +4172,13 @@ packages: os: [linux] libc: [glibc] + '@img/sharp-linux-arm@0.35.3': + resolution: {integrity: sha512-affVWCTLooy8TSxbDx2qkzuDeaWLNVBA+P//FNBirHsXpP2fuBhk5AuboYUnrDnzoXes8GFjpTx0SBFOCRg+FA==} + engines: {node: '>=20.9.0'} + cpu: [arm] + os: [linux] + libc: [glibc] + '@img/sharp-linux-ppc64@0.34.5': resolution: {integrity: sha512-7zznwNaqW6YtsfrGGDA6BRkISKAAE1Jo0QdpNYXNMHu2+0dTrPflTLNkpc8l7MUP5M16ZJcUvysVWWrMefZquA==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} @@ -3990,6 +4186,13 @@ packages: os: [linux] libc: [glibc] + '@img/sharp-linux-ppc64@0.35.3': + resolution: {integrity: sha512-sMd8rDxmpLOwv/7N44klFjOD5DUO7FLdjiXDI0hoxYaf7Ar262dQIEkosE98bps+5HPLtp/EvNqeqQtOycP/IA==} + engines: {node: '>=20.9.0'} + cpu: [ppc64] + os: [linux] + libc: [glibc] + '@img/sharp-linux-riscv64@0.34.5': resolution: {integrity: sha512-51gJuLPTKa7piYPaVs8GmByo7/U7/7TZOq+cnXJIHZKavIRHAP77e3N2HEl3dgiqdD/w0yUfiJnII77PuDDFdw==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} @@ -3997,6 +4200,13 @@ packages: os: [linux] libc: [glibc] + '@img/sharp-linux-riscv64@0.35.3': + resolution: {integrity: sha512-0Eob78yjlYPfL5vMNWAW55l3R9Y6BQS/gOfe0ZcP9mEz9ohhKSt4im1hayiknXgf8AWrFqMvJcKIdmLmEe7yeQ==} + engines: {node: '>=20.9.0'} + cpu: [riscv64] + os: [linux] + libc: [glibc] + '@img/sharp-linux-s390x@0.34.5': resolution: {integrity: sha512-nQtCk0PdKfho3eC5MrbQoigJ2gd1CgddUMkabUj+rBevs8tZ2cULOx46E7oyX+04WGfABgIwmMC0VqieTiR4jg==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} @@ -4004,6 +4214,13 @@ packages: os: [linux] libc: [glibc] + '@img/sharp-linux-s390x@0.35.3': + resolution: {integrity: sha512-KgAxQ0DxpNOq1rG2t5cgTgShJFGSuU7XO45cqC+1NVOuZnP6tlgZRuSYOfNupGkHID0o3cJOsw4DVeJpMovcGw==} + engines: {node: '>=20.9.0'} + cpu: [s390x] + os: [linux] + libc: [glibc] + '@img/sharp-linux-x64@0.34.5': resolution: {integrity: sha512-MEzd8HPKxVxVenwAa+JRPwEC7QFjoPWuS5NZnBt6B3pu7EG2Ge0id1oLHZpPJdn3OQK+BQDiw9zStiHBTJQQQQ==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} @@ -4011,6 +4228,13 @@ packages: os: [linux] libc: [glibc] + '@img/sharp-linux-x64@0.35.3': + resolution: {integrity: sha512-8pqvxubL2PGdhlPy6GLqzDYMUjyRmKAwKHYKixpdJYBUK7PJ0C029XdsnpFIdgRZG68fZiGdHVWcKPvtiPB4cA==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [linux] + libc: [glibc] + '@img/sharp-linuxmusl-arm64@0.34.5': resolution: {integrity: sha512-fprJR6GtRsMt6Kyfq44IsChVZeGN97gTD331weR1ex1c1rypDEABN6Tm2xa1wE6lYb5DdEnk03NZPqA7Id21yg==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} @@ -4018,6 +4242,13 @@ packages: os: [linux] libc: [musl] + '@img/sharp-linuxmusl-arm64@0.35.3': + resolution: {integrity: sha512-Vz0iQjzzcSX3HCbfwFfCSG/9SCIqyO0mH2sXyiHaAYfBk0cRsCWXRyQYX0ovCK/PAQBbTzQ0dsPQHh5MAFL59w==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [linux] + libc: [musl] + '@img/sharp-linuxmusl-x64@0.34.5': resolution: {integrity: sha512-Jg8wNT1MUzIvhBFxViqrEhWDGzqymo3sV7z7ZsaWbZNDLXRJZoRGrjulp60YYtV4wfY8VIKcWidjojlLcWrd8Q==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} @@ -4025,29 +4256,63 @@ packages: os: [linux] libc: [musl] + '@img/sharp-linuxmusl-x64@0.35.3': + resolution: {integrity: sha512-6O1NPKcDVj9QEdg7Hx549EX8U0rp6yXQERqru6yRN7fGBn32UvIRJUlWnk+8xDCiG76hXVBbX82NZ/ZKr0euIg==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [linux] + libc: [musl] + '@img/sharp-wasm32@0.34.5': resolution: {integrity: sha512-OdWTEiVkY2PHwqkbBI8frFxQQFekHaSSkUIJkwzclWZe64O1X4UlUjqqqLaPbUpMOQk6FBu/HtlGXNblIs0huw==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [wasm32] + '@img/sharp-wasm32@0.35.3': + resolution: {integrity: sha512-cZ0XkcYGpHZkqW6iCkqTcmUC0CD9DhD5d/qeZlZkfRBn6GnHniZXLUo5+9xw8Iv76YE6LQFN9YNBlKREcCG76w==} + engines: {node: '>=20.9.0'} + + '@img/sharp-webcontainers-wasm32@0.35.3': + resolution: {integrity: sha512-2rnq7bX3NzeR2T4YWgz8qiG4h3TSdMe+vN1iQXpJleSJ3SM5zQ8Fy2SyyXAWlbxpEZ2Y+Z4u1BePgJEYbSy80Q==} + engines: {node: '>=20.9.0'} + cpu: [wasm32] + '@img/sharp-win32-arm64@0.34.5': resolution: {integrity: sha512-WQ3AgWCWYSb2yt+IG8mnC6Jdk9Whs7O0gxphblsLvdhSpSTtmu69ZG1Gkb6NuvxsNACwiPV6cNSZNzt0KPsw7g==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [arm64] os: [win32] + '@img/sharp-win32-arm64@0.35.3': + resolution: {integrity: sha512-4bPwFdMbeC4JQ8L8LOyWp6nsHcboP5fxkp6iPOXz2Vg49R42TuMs2whkJ5OAP4/Ul035qOzy0AecOF9VOscn4w==} + engines: {node: '>=20.9.0'} + cpu: [arm64] + os: [win32] + '@img/sharp-win32-ia32@0.34.5': resolution: {integrity: sha512-FV9m/7NmeCmSHDD5j4+4pNI8Cp3aW+JvLoXcTUo0IqyjSfAZJ8dIUmijx1qaJsIiU+Hosw6xM5KijAWRJCSgNg==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [ia32] os: [win32] + '@img/sharp-win32-ia32@0.35.3': + resolution: {integrity: sha512-r53mXsBN6lFUDiST764SvgwUdHAqM4rPAiDzAmf4fLoB6X/rkfyTrLCg6+g17wJJiCmB3JYgHuUldCWUIRFSXw==} + engines: {node: ^20.9.0} + cpu: [ia32] + os: [win32] + '@img/sharp-win32-x64@0.34.5': resolution: {integrity: sha512-+29YMsqY2/9eFEiW93eqWnuLcWcufowXewwSNIT6UwZdUUCrM3oFjMWH/Z6/TMmb4hlFenmfAVbpWeup2jryCw==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [x64] os: [win32] + '@img/sharp-win32-x64@0.35.3': + resolution: {integrity: sha512-D4y1vNeZrIIJCN+uHaWVtH86B+aCrdMYYjicy9pXHvbGZeGYLLSd3wdVuC37FxVXlU1ARsk84eKWfWMXGYEqvA==} + engines: {node: '>=20.9.0'} + cpu: [x64] + os: [win32] + '@inquirer/checkbox@4.2.2': resolution: {integrity: sha512-E+KExNurKcUJJdxmjglTl141EwxWyAHplvsYJQgSwXf8qiNWkTxTuCCqmhFEmbIXd4zLaGMfQFJ6WrZ7fSeV3g==} engines: {node: '>=18'} @@ -4524,6 +4789,61 @@ packages: '@napi-rs/wasm-runtime@1.1.0': resolution: {integrity: sha512-Fq6DJW+Bb5jaWE69/qOE0D1TUN9+6uWhCeZpdnSBk14pjLcCWR7Q8n49PTSPHazM37JqrsdpEthXy2xn6jWWiA==} + '@next/env@16.3.1': + resolution: {integrity: sha512-35G3xwkQUb2oETSDjFXGrVugknoayLFBh7vSE+yAcl9IP2zT9wyGwq7297AYHR11kJld807t5f8AJBs6WBzXsQ==} + + '@next/swc-darwin-arm64@16.3.1': + resolution: {integrity: sha512-ABMIu2zQ7cnNIHm5ivKGwZwUrm0pAai3yiJ/gK/rF1c1VP9UOnj7XECbMKFdVKp9I9eMYq9NoDs1WXOoowxzJw==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [darwin] + + '@next/swc-darwin-x64@16.3.1': + resolution: {integrity: sha512-gNG21e/UnrroeScbY/QndUEdl0mF1FRibW7BBeYUz/5ABCepjqDdEdgr592vpzMtCn/m7FTjYq3TN4TpyDnutw==} + engines: {node: '>= 10'} + cpu: [x64] + os: [darwin] + + '@next/swc-linux-arm64-gnu@16.3.1': + resolution: {integrity: sha512-6B6Lw016iwNUQuaJoraMMTLh6TwHzFUtxipSScD1F3YyymcrRWkobodRS2ftIOkF5vrs4zNlyUrTC5YZQ9Lz5w==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [linux] + libc: [glibc] + + '@next/swc-linux-arm64-musl@16.3.1': + resolution: {integrity: sha512-JUiPXZKK9wOhjf4MgDiH29GZLxfqOesbLtHq2pDxwH/WwscTRV2ToymnOTh1egzaZf0ueUf8T2+CeYTGHjW0Iw==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [linux] + libc: [musl] + + '@next/swc-linux-x64-gnu@16.3.1': + resolution: {integrity: sha512-Uog9jsrmIRIL/lfvIp9htmskSNC7JcQsMVucXL2V2YY1y/D9IUN3LPEafqy0zRJ2cIU1SQ0V6F6TlffQ+pLAGg==} + engines: {node: '>= 10'} + cpu: [x64] + os: [linux] + libc: [glibc] + + '@next/swc-linux-x64-musl@16.3.1': + resolution: {integrity: sha512-6yy3FT13KgUFOj5H8bl8w/6nKiJwHIvbtwh1V+1acsu+7y4tJjnemSa6mhsh53BeoVrlozE+fMgZhXH46WmjMA==} + engines: {node: '>= 10'} + cpu: [x64] + os: [linux] + libc: [musl] + + '@next/swc-win32-arm64-msvc@16.3.1': + resolution: {integrity: sha512-iOoN1QecUoGNZik536U/vtK43YwgyrCsGIkth52yIkl612n+0C9MjSnJbQAikISpb+WYRooBVhaDlUW7iZoKog==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [win32] + + '@next/swc-win32-x64-msvc@16.3.1': + resolution: {integrity: sha512-d/k+PpAriUPaeMJJOG7HUSdqfEX46FEPWU1p3/nm2ACmXhj9hFEWdFODUBIpkuijXYkfL90qZzTqVPRp4BW/hw==} + engines: {node: '>= 10'} + cpu: [x64] + os: [win32] + '@noble/ciphers@2.1.1': resolution: {integrity: sha512-bysYuiVfhxNJuldNXlFEitTVdNnYUc+XNJZd7Qm2a5j1vZHgY+fazadNFWFaMK/2vye0JVlxV3gHmC0WDfAOQw==} engines: {node: '>= 20.19.0'} @@ -4819,6 +5139,11 @@ packages: resolution: {integrity: sha512-QNqXyfVS2wm9hweSYD2O7F0G06uurj9kZ96TRQE5Y9hU7+tgdZwIkbAKc5Ocy1HxEY2kuDQa6cQ1WRs/O5LFKA==} engines: {node: ^12.20.0 || ^14.18.0 || >=16.0.0} + '@playwright/test@1.60.0': + resolution: {integrity: sha512-O71yZIbAh/PxDMNGns37GHBIfrVkEVyn+AXyIa5dOTfb4/xNvRWV+Vv/NMbNCtODB/pO7vLlF2OTmMVLhmr7Ag==} + engines: {node: '>=18'} + hasBin: true + '@polka/url@1.0.0-next.29': resolution: {integrity: sha512-wwQAWhWSuHaag8c4q/KN/vCoeOJYshAIvMQwD4GpSb3OiZklFfvAgmj0VCBBImRpuF/aFgIRzllXlVX93Jevww==} @@ -4831,13 +5156,13 @@ packages: '@poppinss/exception@1.2.3': resolution: {integrity: sha512-dCED+QRChTVatE9ibtoaxc+WkdzOSjYTKi/+uacHWIsfodVfpsueo3+DKpgU5Px8qXjgmXkSvhXvSCz3fnP9lw==} - '@powersync/common@1.49.0': - resolution: {integrity: sha512-g6uonubvtmtyx8hS/G5trg9LsBvzHY3tAKHiV7SIQV3Xyz9ONM6NNnjDMP2vcLZVmsOSi8x/QJZmy/ig1YtBMg==} + '@powersync/common@1.57.0': + resolution: {integrity: sha512-uYccCxK5mwahELRouY3YY584TZgjFU8wPPKZQQ6sAOUoMikV8D/+v+UYsNI280MKMnhFqLkxk4TPZIG7ArIzTQ==} - '@powersync/node@0.18.1': - resolution: {integrity: sha512-fcTICgs61CAEb39xiC7pedYsPgbjUInJ/47dr7RIdnEHpAgjWH8bW95/b70qK1fQUANy9lKBBF3PcmfswVgfCw==} + '@powersync/node@0.19.2': + resolution: {integrity: sha512-lF7v/rkiLujAojn7Vjgvs1AibhL5zlEQVYO0iCUGoE1S1Hw7lxfUvAa1mTneKWCEmj0EC9yQBHkPUyBDZXVdLA==} peerDependencies: - '@powersync/common': ^1.49.0 + '@powersync/common': ^1.57.0 better-sqlite3: 12.x peerDependenciesMeta: better-sqlite3: @@ -5603,6 +5928,9 @@ packages: resolution: {integrity: sha512-08eKiDAjj4zLug1taXSIJ0kGL5cawjVCyJkBb6EWSg5fEPX6L+Wtr0CH2If4j5KYylz85iaZiFlUItvgJvll5g==} engines: {node: ^14.13.1 || ^16.0.0 || >=18} + '@swc/helpers@0.5.23': + resolution: {integrity: sha512-5lSsMOTXURePglDfvuAQUqkGek9Hg2kksOYay2m0+XR++b2NWYL/4sWyuvVBIs8oKnJaxkdi9whaL/sqN13afw==} + '@szmarczak/http-timer@4.0.6': resolution: {integrity: sha512-4BAffykYOgO+5nzBWYwE3W90sBgLJoUPRWWcL8wlyiM8IB8ipJz3UMJ9KXQd1RKQXpKp8Tutn80HZtWsu2u76w==} engines: {node: '>=10'} @@ -6799,9 +7127,6 @@ packages: async-limiter@1.0.1: resolution: {integrity: sha512-csOlWGAcRFJaI6m+F2WKdnMKr4HhdhFVBk0H/QbJFMCr+uO2kwohwXQPxw/9OCxp05r5ghVBFSyioixx3gfkNQ==} - async-mutex@0.5.0: - resolution: {integrity: sha512-1A94B18jkJ3DYq284ohPxoXbfTA5HsQ7/Mf4DEhcyLx3Bz27Rh59iScbB6EPiP+B+joue6YCxcMXSbFC1tZKwA==} - asynckit@0.4.0: resolution: {integrity: sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==} @@ -8173,9 +8498,6 @@ packages: resolution: {integrity: sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==} engines: {node: '>= 0.6'} - event-iterator@2.0.0: - resolution: {integrity: sha512-KGft0ldl31BZVV//jj+IAIGCxkvvUkkON+ScH6zfoX+l+omX6001ggyRSpI0Io2Hlro0ThXotswCtfzS8UkIiQ==} - event-reduce-js@5.2.7: resolution: {integrity: sha512-Vi6aIiAmakzx81JAwhw8L988aSX5a3ZqqVjHyZa9xFU6P4oT1IotoDreWtjNlS+fvEnASvyIQT565nmkOtns/Q==} engines: {node: '>=16'} @@ -8586,6 +8908,11 @@ packages: fs.realpath@1.0.0: resolution: {integrity: sha512-OO0pH2lK6a0hZnAdau5ItzHPI6pUlvI7jMVnxUQRtw4owF2wk8lOSabtGDCTP4Ggrg2MbGnWO9X8K1t4+fGMDw==} + fsevents@2.3.2: + resolution: {integrity: sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==} + engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} + os: [darwin] + fsevents@2.3.3: resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==} engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} @@ -9372,6 +9699,9 @@ packages: js-base64@3.7.8: resolution: {integrity: sha512-hNngCeKxIUQiEUN3GPJOkz4wF/YvdUdbNL9hsBcMQTkKzboD7T/q3OYOuuPZLUE6dBxSGpwhk5mwuDud7JVAow==} + js-logger@1.6.1: + resolution: {integrity: sha512-yTgMCPXVjhmg28CuUH8CKjU+cIKL/G+zTu4Fn4lQxs8mRFH/03QTNvEFngcxfg/gRDiQAOoyCKmMTOm9ayOzXA==} + js-tokens@4.0.0: resolution: {integrity: sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==} @@ -10195,6 +10525,11 @@ packages: engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} hasBin: true + nanoid@3.3.18: + resolution: {integrity: sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w==} + engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} + hasBin: true + nanostores@1.1.0: resolution: {integrity: sha512-yJBmDJr18xy47dbNVlHcgdPrulSn1nhSE6Ns9vTG+Nx9VPT6iV1MD6aQFp/t52zpf82FhLLTXAXr30NuCnxvwA==} engines: {node: ^20.0.0 || >=22.0.0} @@ -10234,6 +10569,27 @@ packages: nested-error-stacks@2.0.1: resolution: {integrity: sha512-SrQrok4CATudVzBS7coSz26QRSmlK9TzzoFbeKfcPBUFPjcQM9Rqvr/DlJkOrwI/0KcgvMub1n1g5Jt9EgRn4A==} + next@16.3.1: + resolution: {integrity: sha512-hsAp0i7Rh+/dhe7DGIeN2YlpLM1DP4MNxti9EtDMtqcO612X81MvvEj388/oTce9U1EcEIOWDlGq0zRwrBKvuA==} + engines: {node: '>=20.9.0'} + hasBin: true + peerDependencies: + '@opentelemetry/api': ^1.1.0 + '@playwright/test': ^1.51.1 + babel-plugin-react-compiler: '*' + react: ^18.2.0 || 19.0.0-rc-de68d2f4-20241204 || ^19.0.0 + react-dom: ^18.2.0 || 19.0.0-rc-de68d2f4-20241204 || ^19.0.0 + sass: ^1.3.0 + peerDependenciesMeta: + '@opentelemetry/api': + optional: true + '@playwright/test': + optional: true + babel-plugin-react-compiler: + optional: true + sass: + optional: true + nice-try@1.0.5: resolution: {integrity: sha512-1nh45deeb5olNY7eX82BkPO7SSxR5SSYJiPTrTdFUVYwAl8CKMA5N9PjTYkHiRjisVcxcQ1HXdLhx2qxxJzLNQ==} @@ -10701,6 +11057,16 @@ packages: pkg-types@1.3.1: resolution: {integrity: sha512-/Jm5M4RvtBFVkKWRu2BLUTNP8/M2a+UwuAX+ae4770q1qVGtfjG+WTCupoZixokjmHiry8uI+dlY8KXYV5HVVQ==} + playwright-core@1.60.0: + resolution: {integrity: sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA==} + engines: {node: '>=18'} + hasBin: true + + playwright@1.60.0: + resolution: {integrity: sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA==} + engines: {node: '>=18'} + hasBin: true + plist@3.1.0: resolution: {integrity: sha512-uysumyrvkUX0rX/dEVqt8gC3sTBzd4zoWfLeS29nb53imdaXVvLINYXTI2GNqzaMuvacNx4uJQ8+b3zXR0pkgQ==} engines: {node: '>=10.4.0'} @@ -10724,6 +11090,10 @@ packages: resolution: {integrity: sha512-pMMHxBOZKFU6HgAZ4eyGnwXF/EvPGGqUr0MnZ5+99485wwW41kW91A4LOGxSHhgugZmSChL5AlElNdwlNgcnLQ==} engines: {node: ^10 || ^12 || >=14} + postcss@8.5.23: + resolution: {integrity: sha512-g50586zr4bZmwFiTlflMu8E0bDTb5I5gertgwAKmsdUlTQIhZtunzUlD1WSzwcVWPoAVpsrA6vlfCD7oXvRwgg==} + engines: {node: ^10 || ^12 || >=14} + postgres-array@2.0.0: resolution: {integrity: sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==} engines: {node: '>=4'} @@ -11288,6 +11658,11 @@ packages: engines: {node: '>=10'} hasBin: true + semver@7.8.5: + resolution: {integrity: sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==} + engines: {node: '>=10'} + hasBin: true + send@0.19.0: resolution: {integrity: sha512-dW41u5VfLXu8SJh5bwRmyYUbAoSB3c9uQh6L8h/KtsFREPWpbX1lrljJo186Jc4nmci/sGUZ9a0a0J2zgfq2hw==} engines: {node: '>= 0.8.0'} @@ -11361,6 +11736,15 @@ packages: resolution: {integrity: sha512-Ou9I5Ft9WNcCbXrU9cMgPBcCK8LiwLqcbywW3t4oDV37n1pzpuNLsYiAV8eODnjbtQlSDwZ2cUEeQz4E54Hltg==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} + sharp@0.35.3: + resolution: {integrity: sha512-ej0zVHuZGHCiABXcNxeYhpRnPNPAcvbG8RMdBAhDAxLKkCRVSpK3Iyu7qbqw3JMzoj0REeM6f3tJLtVwl0023Q==} + engines: {node: '>=20.9.0'} + peerDependencies: + '@types/node': '*' + peerDependenciesMeta: + '@types/node': + optional: true + shebang-command@1.2.0: resolution: {integrity: sha512-EV3L1+UQWGor21OmnvojK36mhg+TyIKDh3iFBKBohr5xeXIhNBcx8oWdgkTEEQ+BEFFYdLRuqMfd5L84N1V5Vg==} engines: {node: '>=0.10.0'} @@ -11762,6 +12146,19 @@ packages: style-to-object@1.0.9: resolution: {integrity: sha512-G4qppLgKu/k6FwRpHiGiKPaPTFcG3g4wNVX/Qsfu+RqQM30E7Tyu/TEgxcL9PNLF5pdRLwQdE3YKKf+KF2Dzlw==} + styled-jsx@5.1.6: + resolution: {integrity: sha512-qSVyDTeMotdvQYoHWLNGwRFJHC+i+ZvdBRYosOFgC+Wg1vx4frN2/RG/NA7SYqqvKNLf39P2LSRA2pu6n0XYZA==} + engines: {node: '>= 12.0.0'} + peerDependencies: + '@babel/core': '*' + babel-plugin-macros: '*' + react: '>= 16.8.0 || 17.x.x || ^18.0.0-0 || ^19.0.0-0' + peerDependenciesMeta: + '@babel/core': + optional: true + babel-plugin-macros: + optional: true + sucrase@3.35.0: resolution: {integrity: sha512-8EbVDiu9iN/nESwxeSxDKe0dunta1GOlHufmSSXxMD2z2/tMZpDMpvXQGsc+ajGo8y2uYUmixaSRUc/QPoQ0GA==} engines: {node: '>=16 || 14 >=14.17'} @@ -14045,7 +14442,7 @@ snapshots: make-fetch-happen: 10.2.1 nopt: 6.0.0 proc-log: 2.0.1 - semver: 7.7.4 + semver: 7.8.5 tar: 6.2.1 which: 2.0.2 transitivePeerDependencies: @@ -14078,6 +14475,11 @@ snapshots: tslib: 2.8.1 optional: true + '@emnapi/runtime@1.11.3': + dependencies: + tslib: 2.8.1 + optional: true + '@emnapi/runtime@1.7.1': dependencies: tslib: 2.8.1 @@ -15596,95 +15998,199 @@ snapshots: '@img/sharp-libvips-darwin-arm64': 1.2.4 optional: true + '@img/sharp-darwin-arm64@0.35.3': + optionalDependencies: + '@img/sharp-libvips-darwin-arm64': 1.3.2 + optional: true + '@img/sharp-darwin-x64@0.34.5': optionalDependencies: '@img/sharp-libvips-darwin-x64': 1.2.4 optional: true + '@img/sharp-darwin-x64@0.35.3': + optionalDependencies: + '@img/sharp-libvips-darwin-x64': 1.3.2 + optional: true + + '@img/sharp-freebsd-wasm32@0.35.3': + dependencies: + '@img/sharp-wasm32': 0.35.3 + optional: true + '@img/sharp-libvips-darwin-arm64@1.2.4': optional: true + '@img/sharp-libvips-darwin-arm64@1.3.2': + optional: true + '@img/sharp-libvips-darwin-x64@1.2.4': optional: true + '@img/sharp-libvips-darwin-x64@1.3.2': + optional: true + '@img/sharp-libvips-linux-arm64@1.2.4': optional: true + '@img/sharp-libvips-linux-arm64@1.3.2': + optional: true + '@img/sharp-libvips-linux-arm@1.2.4': optional: true + '@img/sharp-libvips-linux-arm@1.3.2': + optional: true + '@img/sharp-libvips-linux-ppc64@1.2.4': optional: true + '@img/sharp-libvips-linux-ppc64@1.3.2': + optional: true + '@img/sharp-libvips-linux-riscv64@1.2.4': optional: true + '@img/sharp-libvips-linux-riscv64@1.3.2': + optional: true + '@img/sharp-libvips-linux-s390x@1.2.4': optional: true + '@img/sharp-libvips-linux-s390x@1.3.2': + optional: true + '@img/sharp-libvips-linux-x64@1.2.4': optional: true + '@img/sharp-libvips-linux-x64@1.3.2': + optional: true + '@img/sharp-libvips-linuxmusl-arm64@1.2.4': optional: true + '@img/sharp-libvips-linuxmusl-arm64@1.3.2': + optional: true + '@img/sharp-libvips-linuxmusl-x64@1.2.4': optional: true + '@img/sharp-libvips-linuxmusl-x64@1.3.2': + optional: true + '@img/sharp-linux-arm64@0.34.5': optionalDependencies: '@img/sharp-libvips-linux-arm64': 1.2.4 optional: true + '@img/sharp-linux-arm64@0.35.3': + optionalDependencies: + '@img/sharp-libvips-linux-arm64': 1.3.2 + optional: true + '@img/sharp-linux-arm@0.34.5': optionalDependencies: '@img/sharp-libvips-linux-arm': 1.2.4 optional: true + '@img/sharp-linux-arm@0.35.3': + optionalDependencies: + '@img/sharp-libvips-linux-arm': 1.3.2 + optional: true + '@img/sharp-linux-ppc64@0.34.5': optionalDependencies: '@img/sharp-libvips-linux-ppc64': 1.2.4 optional: true + '@img/sharp-linux-ppc64@0.35.3': + optionalDependencies: + '@img/sharp-libvips-linux-ppc64': 1.3.2 + optional: true + '@img/sharp-linux-riscv64@0.34.5': optionalDependencies: '@img/sharp-libvips-linux-riscv64': 1.2.4 optional: true + '@img/sharp-linux-riscv64@0.35.3': + optionalDependencies: + '@img/sharp-libvips-linux-riscv64': 1.3.2 + optional: true + '@img/sharp-linux-s390x@0.34.5': optionalDependencies: '@img/sharp-libvips-linux-s390x': 1.2.4 optional: true + '@img/sharp-linux-s390x@0.35.3': + optionalDependencies: + '@img/sharp-libvips-linux-s390x': 1.3.2 + optional: true + '@img/sharp-linux-x64@0.34.5': optionalDependencies: '@img/sharp-libvips-linux-x64': 1.2.4 optional: true + '@img/sharp-linux-x64@0.35.3': + optionalDependencies: + '@img/sharp-libvips-linux-x64': 1.3.2 + optional: true + '@img/sharp-linuxmusl-arm64@0.34.5': optionalDependencies: '@img/sharp-libvips-linuxmusl-arm64': 1.2.4 optional: true + '@img/sharp-linuxmusl-arm64@0.35.3': + optionalDependencies: + '@img/sharp-libvips-linuxmusl-arm64': 1.3.2 + optional: true + '@img/sharp-linuxmusl-x64@0.34.5': optionalDependencies: '@img/sharp-libvips-linuxmusl-x64': 1.2.4 optional: true + '@img/sharp-linuxmusl-x64@0.35.3': + optionalDependencies: + '@img/sharp-libvips-linuxmusl-x64': 1.3.2 + optional: true + '@img/sharp-wasm32@0.34.5': dependencies: '@emnapi/runtime': 1.7.1 optional: true + '@img/sharp-wasm32@0.35.3': + dependencies: + '@emnapi/runtime': 1.11.3 + optional: true + + '@img/sharp-webcontainers-wasm32@0.35.3': + dependencies: + '@img/sharp-wasm32': 0.35.3 + optional: true + '@img/sharp-win32-arm64@0.34.5': optional: true + '@img/sharp-win32-arm64@0.35.3': + optional: true + '@img/sharp-win32-ia32@0.34.5': optional: true + '@img/sharp-win32-ia32@0.35.3': + optional: true + '@img/sharp-win32-x64@0.34.5': optional: true + '@img/sharp-win32-x64@0.35.3': + optional: true + '@inquirer/checkbox@4.2.2(@types/node@25.2.2)': dependencies: '@inquirer/core': 10.2.0(@types/node@25.2.2) @@ -16224,6 +16730,32 @@ snapshots: '@tybys/wasm-util': 0.10.1 optional: true + '@next/env@16.3.1': {} + + '@next/swc-darwin-arm64@16.3.1': + optional: true + + '@next/swc-darwin-x64@16.3.1': + optional: true + + '@next/swc-linux-arm64-gnu@16.3.1': + optional: true + + '@next/swc-linux-arm64-musl@16.3.1': + optional: true + + '@next/swc-linux-x64-gnu@16.3.1': + optional: true + + '@next/swc-linux-x64-musl@16.3.1': + optional: true + + '@next/swc-win32-arm64-msvc@16.3.1': + optional: true + + '@next/swc-win32-x64-msvc@16.3.1': + optional: true + '@noble/ciphers@2.1.1': {} '@noble/hashes@2.0.1': {} @@ -16255,11 +16787,11 @@ snapshots: '@npmcli/fs@2.1.2': dependencies: '@gar/promisify': 1.1.3 - semver: 7.7.4 + semver: 7.8.5 '@npmcli/fs@5.0.0': dependencies: - semver: 7.7.4 + semver: 7.8.5 '@npmcli/git@7.0.1': dependencies: @@ -16466,6 +16998,10 @@ snapshots: '@pkgr/core@0.2.9': {} + '@playwright/test@1.60.0': + dependencies: + playwright: 1.60.0 + '@polka/url@1.0.0-next.29': {} '@poppinss/colors@4.1.6': @@ -16480,16 +17016,13 @@ snapshots: '@poppinss/exception@1.2.3': {} - '@powersync/common@1.49.0': + '@powersync/common@1.57.0': dependencies: - async-mutex: 0.5.0 - event-iterator: 2.0.0 + js-logger: 1.6.1 - '@powersync/node@0.18.1(@powersync/common@1.49.0)(better-sqlite3@12.8.0)': + '@powersync/node@0.19.2(@powersync/common@1.57.0)(better-sqlite3@12.8.0)': dependencies: - '@powersync/common': 1.49.0 - async-mutex: 0.5.0 - bson: 6.10.4 + '@powersync/common': 1.57.0 comlink: 4.4.2 undici: 7.24.4 optionalDependencies: @@ -17494,6 +18027,10 @@ snapshots: transitivePeerDependencies: - encoding + '@swc/helpers@0.5.23': + dependencies: + tslib: 2.8.1 + '@szmarczak/http-timer@4.0.6': dependencies: defer-to-connect: 2.0.1 @@ -18957,10 +19494,6 @@ snapshots: async-limiter@1.0.1: {} - async-mutex@0.5.0: - dependencies: - tslib: 2.8.1 - asynckit@0.4.0: {} at-least-node@1.0.0: {} @@ -19193,7 +19726,7 @@ snapshots: postcss: 8.5.10 postcss-media-query-parser: 0.2.3 - better-auth@1.4.18(b4f55ef685357933f61fbe0b4095cc16): + better-auth@1.4.18(dd19f6838762949983acd690aa0ac646): dependencies: '@better-auth/core': 1.4.18(@better-auth/utils@0.3.0)(@better-fetch/fetch@1.1.21)(better-call@1.1.8(zod@4.3.6))(jose@6.1.3)(kysely@0.28.11)(nanostores@1.1.0) '@better-auth/telemetry': 1.4.18(@better-auth/core@1.4.18(@better-auth/utils@0.3.0)(@better-fetch/fetch@1.1.21)(better-call@1.1.8(zod@4.3.6))(jose@6.1.3)(kysely@0.28.11)(nanostores@1.1.0)) @@ -19214,6 +19747,7 @@ snapshots: drizzle-kit: 0.31.9 drizzle-orm: 0.45.1(@op-engineering/op-sqlite@15.2.7(react-native@0.83.2(@babel/core@7.29.0)(@types/react@19.2.13)(react@19.2.4))(react@19.2.4))(@types/better-sqlite3@7.6.13)(@types/pg@8.16.0)(better-sqlite3@12.8.0)(expo-sqlite@55.0.11(expo@55.0.8)(react-native@0.83.2(@babel/core@7.29.0)(@types/react@19.2.13)(react@19.2.4))(react@19.2.4))(kysely@0.28.11)(pg@8.20.0)(postgres@3.4.8)(sql.js@1.14.1) mongodb: 6.21.0(socks@2.8.7) + next: 16.3.1(@babel/core@7.29.0)(@playwright/test@1.60.0)(@types/node@25.2.2)(babel-plugin-react-compiler@1.0.0)(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(sass@1.90.0) pg: 8.20.0 react: 19.2.4 react-dom: 19.2.4(react@19.2.4) @@ -20374,7 +20908,7 @@ snapshots: eslint-compat-utils@0.5.1(eslint@9.39.4(jiti@2.6.1)): dependencies: eslint: 9.39.4(jiti@2.6.1) - semver: 7.7.4 + semver: 7.8.5 eslint-config-prettier@10.1.8(eslint@9.39.4(jiti@2.6.1)): dependencies: @@ -20577,8 +21111,6 @@ snapshots: etag@1.8.1: {} - event-iterator@2.0.0: {} - event-reduce-js@5.2.7: dependencies: array-push-at-sort-position: 4.0.1 @@ -21315,6 +21847,9 @@ snapshots: fs.realpath@1.0.0: {} + fsevents@2.3.2: + optional: true + fsevents@2.3.3: optional: true @@ -22133,6 +22668,8 @@ snapshots: js-base64@3.7.8: {} + js-logger@1.6.1: {} + js-tokens@4.0.0: {} js-tokens@9.0.1: {} @@ -23085,6 +23622,8 @@ snapshots: nanoid@3.3.11: {} + nanoid@3.3.18: {} + nanostores@1.1.0: {} napi-build-utils@2.0.0: {} @@ -23121,6 +23660,34 @@ snapshots: nested-error-stacks@2.0.1: {} + next@16.3.1(@babel/core@7.29.0)(@playwright/test@1.60.0)(@types/node@25.2.2)(babel-plugin-react-compiler@1.0.0)(react-dom@19.2.4(react@19.2.4))(react@19.2.4)(sass@1.90.0): + dependencies: + '@next/env': 16.3.1 + '@swc/helpers': 0.5.23 + baseline-browser-mapping: 2.9.19 + caniuse-lite: 1.0.30001769 + postcss: 8.5.23 + react: 19.2.4 + react-dom: 19.2.4(react@19.2.4) + styled-jsx: 5.1.6(@babel/core@7.29.0)(react@19.2.4) + optionalDependencies: + '@next/swc-darwin-arm64': 16.3.1 + '@next/swc-darwin-x64': 16.3.1 + '@next/swc-linux-arm64-gnu': 16.3.1 + '@next/swc-linux-arm64-musl': 16.3.1 + '@next/swc-linux-x64-gnu': 16.3.1 + '@next/swc-linux-x64-musl': 16.3.1 + '@next/swc-win32-arm64-msvc': 16.3.1 + '@next/swc-win32-x64-msvc': 16.3.1 + '@playwright/test': 1.60.0 + babel-plugin-react-compiler: 1.0.0 + sass: 1.90.0 + sharp: 0.35.3(@types/node@25.2.2) + transitivePeerDependencies: + - '@babel/core' + - '@types/node' + - babel-plugin-macros + nice-try@1.0.5: {} nkeys.js@1.1.0: @@ -23167,7 +23734,7 @@ snapshots: make-fetch-happen: 15.0.5 nopt: 9.0.0 proc-log: 6.1.0 - semver: 7.7.4 + semver: 7.8.5 tar: 7.5.7 tinyglobby: 0.2.16 which: 6.0.0 @@ -23196,7 +23763,7 @@ snapshots: npm-install-checks@8.0.0: dependencies: - semver: 7.7.4 + semver: 7.8.5 npm-normalize-package-bin@5.0.0: {} @@ -23639,6 +24206,14 @@ snapshots: mlly: 1.8.0 pathe: 2.0.3 + playwright-core@1.60.0: {} + + playwright@1.60.0: + dependencies: + playwright-core: 1.60.0 + optionalDependencies: + fsevents: 2.3.2 + plist@3.1.0: dependencies: '@xmldom/xmldom': 0.8.11 @@ -23663,6 +24238,12 @@ snapshots: picocolors: 1.1.1 source-map-js: 1.2.1 + postcss@8.5.23: + dependencies: + nanoid: 3.3.18 + picocolors: 1.1.1 + source-map-js: 1.2.1 + postgres-array@2.0.0: {} postgres-bytea@1.0.0: {} @@ -24545,6 +25126,8 @@ snapshots: semver@7.7.4: {} + semver@7.8.5: {} + send@0.19.0: dependencies: debug: 2.6.9 @@ -24693,6 +25276,40 @@ snapshots: '@img/sharp-win32-ia32': 0.34.5 '@img/sharp-win32-x64': 0.34.5 + sharp@0.35.3(@types/node@25.2.2): + dependencies: + '@img/colour': 1.1.0 + detect-libc: 2.1.2 + semver: 7.8.5 + optionalDependencies: + '@img/sharp-darwin-arm64': 0.35.3 + '@img/sharp-darwin-x64': 0.35.3 + '@img/sharp-freebsd-wasm32': 0.35.3 + '@img/sharp-libvips-darwin-arm64': 1.3.2 + '@img/sharp-libvips-darwin-x64': 1.3.2 + '@img/sharp-libvips-linux-arm': 1.3.2 + '@img/sharp-libvips-linux-arm64': 1.3.2 + '@img/sharp-libvips-linux-ppc64': 1.3.2 + '@img/sharp-libvips-linux-riscv64': 1.3.2 + '@img/sharp-libvips-linux-s390x': 1.3.2 + '@img/sharp-libvips-linux-x64': 1.3.2 + '@img/sharp-libvips-linuxmusl-arm64': 1.3.2 + '@img/sharp-libvips-linuxmusl-x64': 1.3.2 + '@img/sharp-linux-arm': 0.35.3 + '@img/sharp-linux-arm64': 0.35.3 + '@img/sharp-linux-ppc64': 0.35.3 + '@img/sharp-linux-riscv64': 0.35.3 + '@img/sharp-linux-s390x': 0.35.3 + '@img/sharp-linux-x64': 0.35.3 + '@img/sharp-linuxmusl-arm64': 0.35.3 + '@img/sharp-linuxmusl-x64': 0.35.3 + '@img/sharp-webcontainers-wasm32': 0.35.3 + '@img/sharp-win32-arm64': 0.35.3 + '@img/sharp-win32-ia32': 0.35.3 + '@img/sharp-win32-x64': 0.35.3 + '@types/node': 25.2.2 + optional: true + shebang-command@1.2.0: dependencies: shebang-regex: 1.0.0 @@ -25145,6 +25762,13 @@ snapshots: dependencies: inline-style-parser: 0.2.4 + styled-jsx@5.1.6(@babel/core@7.29.0)(react@19.2.4): + dependencies: + client-only: 0.0.1 + react: 19.2.4 + optionalDependencies: + '@babel/core': 7.29.0 + sucrase@3.35.0: dependencies: '@jridgewell/gen-mapping': 0.3.13 diff --git a/todo-1785-reconciliation.md b/todo-1785-reconciliation.md new file mode 100644 index 0000000000..8fc5691ab1 --- /dev/null +++ b/todo-1785-reconciliation.md @@ -0,0 +1,295 @@ +# Reconcile lifecycle/resume work onto current main + +Base: origin/main ad043b745. Published PR #1785 head: 97e5c642a. +Main includes merged #1797 and #1800; #1800's final head passed every CI check. +Use a normal merge, never rewrite published history or restore an old tree. + +## Acceptance queue + +- [x] Update RFC #1657: #1800 merged and verified. +- [x] Compare the old PR with main before porting. +- [x] Resolve the merge while preserving main's newer contracts. +- [x] Inventory every old test/fix: retained, already shipped, or retired with reason. +- [x] Run retained Electric/persistence lifecycle tests against main for RED evidence. +- [x] Verify per-collection Electric evidence/utilities, lazy startup and GC cleanup. +- [x] Verify resume presence, callback partitioning, move-out, and reset conflicts. +- [x] Verify persistence startup/hydration/invalidation generation fences. +- [x] Preserve useful space tests without adding production diagnostic APIs. +- [x] Rebuilt-core Electric, persistence, Query DB, framework and core gates. +- [x] Review resulting diff and size; narrow changeset to unshipped packages. +- [x] Publish normal merge commit and refresh PR body against it (368a1f24c). +- [x] Audit RFC contracts/docs and retain explicit owners for unresolved reports. +- [ ] CI and service-backed adapter E2E on the published reconciliation; RFC closure remains blocked. + +## Reconciliation decisions + +- Main already contains lazy runtime-reference identity and its regression. + Keep main's symbol support and implementation; do not reapply the old variant. +- Main's ownership oracle replaces removed internal-map test seams with public + behavior checks, including eager cache removal, exact acquisition release, + and overlapping persisted owners. Keep these stronger tests. +- Do not restore BucketFacadeMetrics or a retained builder pointer for tests. + Preserve the old nested-space law using test instrumentation. +- Persistence conflicts must preserve #1800's object-identity acquisitions, + upstream rejection/peer ownership contract, and one-shot refresh behavior. +- Core cleanup must preserve main's reentrant-cleanup guard and status revisions. + +## File and contract reconciliation + +All 30 paths in the old merge-base diff are accounted for: + +| Old area | Disposition | +| ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Core Collection construction and lifecycle (2 files) | Retain per-instance sync materialization and pre-start cleanup, preserving main's cleanup/reentrancy guards. | +| Persistence runtime and tests (2 files) | Retain generation and hydration fences; preserve newer object-identity acquisitions, failure handling, and one-shot refresh behavior. | +| Electric runtime, package, three test files and mutation ledger (6 files) | Retain lifecycle owner, sparse presence validation, reset reconciliation, and all tests; remove the replaced Store dependency. | +| Query runtime and two ownership test files (3 files) | Keep main: it already has the refcount guard, persisted ownership ordering, and stronger public ownership laws. Textual merge had duplicated a refcount guard; removed that duplicate. | +| Runtime identity code, test, and its changeset (3 files) | Already shipped. Keep main's lazy initialization plus symbol support and test; remove duplicate release note. | +| Facade adapter, builder, internal utils, architecture (4 files) | Keep main; do not restore metrics API or retained builder pointer. Add only a test-contract reference in docs. | +| Nested space fixture, test, benchmark, core package (4 files) | Retain tests and command entry points; count returned facade entries and retained maps through test-only instrumentation. | +| getKey planning and React/Solid tests (3 files) | Retain added behavior tests; no production changes to these boundaries. | +| AGENTS.md, lifecycle changeset, lockfile (3 files) | Keep independent-oracle rules; narrow release note to unshipped packages; retain Store dependency removal. | + +## RED/GREEN evidence + +Temporarily replaced the five changed runtime files with exact origin/main +versions and rebuilt db-ivm/core, leaving retained tests in place. Restored the +reconciled runtime afterward and rebuilt core again. + +- Electric baseline: 41 failures / 15 passes, 56 cases, plus one unhandled + cleanup rejection. This is a case count, not a distinct-bug count. + Log: /private/tmp/1785-electric-main-red.log. +- Persistence baseline: two stale lifecycle tests RED, late-write guard already + GREEN, and the resume-baseline test cannot run because main lacks its hook. + The hook failure alone is not a reproduced bug. Electric's public persisted + resume/hydration cases supply behavioral evidence. + Log: /private/tmp/1785-persistence-main-red.log. +- Reconciled Electric: all 56 oracle cases GREEN; full package 15 test/type files + pass with no type errors. Log: /private/tmp/1785-electric-full.log. +- Reconciled persistence: 148 runtime/type checks GREEN, six files. + Log: /private/tmp/1785-persistence-full.log. +- Nested space: passes on current main without any production metrics API. + Log: /private/tmp/1785-space-test.log. + +Two old Electric GC/startup probes awaited successful preload after cleanup. +Updated them to observe rejection immediately and assert AbortError, matching +main's documented cleanup contract. The waiter retirement assertions remain. +No skips, weakened classifiers, or timeout increases. + +The worktree's old pnpm installation tried to purge dependencies after the +package-manager version changed. Used existing local vite/vitest/tsc binaries +instead; no lockfile regeneration. For the main-only RED run, restored the +already-installed Store 0.9.2 dependency link required by main's Electric code. + +## Reconciled-main verification checkpoint + +- Core: 4,863 runtime tests / 150 files, all pass. Standalone tsc passes. +- Electric: 56 oracle cases pass; full package 15 runtime/type files pass, + 622 reported checks with no errors (type/runtime totals overlap). +- Persistence: 148 runtime/type checks, all pass. +- Query DB: 368 reported passes / 369 discovered checks, no failures or type + errors; same count shape as the #1800 verification. +- React useLiveQuery: 57 tests pass. Solid useLiveQuery: 40 tests pass. +- Focused ESLint: zero errors, six existing require-await warnings. +- Logs: /private/tmp/1785-{core-full,core-types,electric-full,persistence-full, + query-full,react,solid,lint}.log. + +The current production delta is limited to core Collection lifecycle setup, +Electric, and persistence. Query DB runtime matches main exactly. Remaining: +final diff/size review, refreshed PR description, and the RFC-wide docs audit. + +## Final size check + +Compared exact origin/main ad043b745 runtime with the reconciled runtime using +esbuild 0.27.7, bundle + minify, browser, ESM, ES2022, identical installed +dependencies, and source aliases for core and db-ivm. Baseline source for each +changed runtime file was supplied from git show, without editing the worktree. +These are diagnostic entry bundles, not application download-size estimates. + +| Entry | Main minified / gzip | Reconciled minified / gzip | Gzip delta | +| -------------------- | -------------------- | -------------------------- | ---------- | +| Core all exports | 347,757 / 98,632 | 348,349 / 98,793 | +161 bytes | +| Electric all exports | 96,396 / 31,749 | 95,220 / 31,022 | -727 bytes | + +Runtime TypeScript delta: +410 lines (core +69, persistence +83, Electric +258). +No Query DB runtime changes or facade metrics remain in this PR. + +## RFC contract/docs audit + +- Applied settlement and request-scoped cancellation: core LoadSubsetFn and + SyncConfig contracts plus transaction/refinement oracles; add the missing + plain-language explanation to the adapter guide. +- Exact acquisition release: core UnloadSubsetFn contract, current Query DB + ownership oracle, persisted failed-peer isolation tests; guide now states + synchronous failure cleanup and asynchronous failure release obligations. +- Private replay, stale rows and bounded repair: error-handling guide and + replay/ordered lifecycle suites agree; do not reinterpret transport completion + as proof of source exhaustion. +- Electric lifecycle/resume: all 56 oracle cases pass. Add docs distinguishing + invalid eager/progressive resume (error plus reset for next sync), unverifiable + persisted hydration (fresh snapshot), and ignored out-of-subset partial rows. +- PowerSync: full package gate passes, 131 checks / eight files, no type errors. + Log: /private/tmp/1785-powersync-full.log. +- #1017 is OPEN and remains pinned as an exact expected assertion failure in + load-subset-oracle.property.test.ts: source rows are still hidden while a + derived optimistic mutation persists. The full core run exercised that pin; + passing the suite does not mean the desired behavior passes. +- #968 is OPEN. React still declares but does not invoke getNextPageParam; + server-page bridging/source extent remain separate from this resume fix. +- #836/#1521/#1615/#1659/#1741 and the feature reports remain separately owned + as listed in the RFC. No broad cross-adapter conformance or service-backed E2E + claim follows from these local gates. RFC closure is not justified yet. + +## External review fixes — 2026-09-10 + +Starting head: `885ed1c9165fbbebd5dfb278189d2452ff25574e`. Changes below are +local follow-ups to that head, not a new published verification claim. + +- [x] Fresh eager recovery replaces the hydrated cache at commit. The integrated + recovery oracle crosses eager/progressive, empty/nonempty replacement, and + hydration before/after the stream callback. It asserts public and persisted + rows. Four eager cases were RED before the fix; all ten cases, including two + valid-resume controls, are GREEN. +- [x] Subset acquisition cannot resurrect logically removed row presence. The + fix reads the existing pending sync queue and progressive buffer instead of + maintaining another history. A nine-cell reset/delete/move-out × acquisition + timing matrix and a generated acquisition-history oracle cover the path. + Reintroducing unconditional baseline refresh makes the generated oracle RED; + its shrunk trace is retained as a committed example alongside random runs. +- [x] Resumed invalid-update validation follows executed visibility changes. + Removed the separate preflight planner, which omitted move-outs. Generated + delete/move-out histories now cross eager/progressive and every contiguous + callback partition, checking rows, error state and persisted reset together. + Cancellation preserves the prior public snapshot and discards staged evidence. +- [x] Reusing raw or once-spread options does not rebind another collection's + utilities. Only adapter-factory utilities are copied, preserving descriptors + and prototype; ordinary utilities retain their existing object identity. +- [x] Tag visibility belongs to each Collection, not the reusable descriptor. + A first per-session fix failed a compatible persisted-restart probe. The + final Collection-keyed tracker retains that state across compatible resume + and clears it on fresh snapshot/reset. All 20 descriptor/restart tests pass. +- [x] Corrected the mutation ledger's collection-local evidence attribution. + Fresh-descriptor process generation does not prove shared-descriptor safety. + Added all new suites to the package's `test:oracles` command and documented + authoritative fresh replacement and descriptor reuse in the adapter guide. + +Lessons: stream markers do not substitute for actual acquisition calls; +published presence can lag logical deletion; fresh transport startup does not +itself replace a durable snapshot. Partition laws must include tag events. +Ownership tests must cross both peer collections and compatible same-owner +restart, not assume every session should discard every state cell. + +Scope: retaining in-memory tags for the same Collection does not add cold-start +restoration of tag indexes from persisted metadata. That pre-existing limitation +and different-schema reuse of a static shape were not established as new PR +bugs and are not claimed fixed by these tests. The review's original sandbox +artifacts were unavailable; all three reported traces were independently rebuilt +against the actual Collection/Electric/persistence path. + +Verification: 4,865 core runtime tests (150 files) and 355 Electric runtime tests +(9 files) pass. Core and Electric standalone TypeScript checks and focused lint +pass. Final persistence rerun and loss-audit closeout are recorded below. +Logs: `/private/tmp/1785-review-fixes-{core,electric,persistence}-final.log`. +Detailed source-order evidence: `/private/tmp/evaluate-1785-external-fd82-ledger.md`. + +Final persistence rerun: 75/75 tests pass (two files), bringing these runtime +gates to 5,295 passing tests. The final bounded peer review confirms all its +findings are accounted for: descriptor tests 20/20, collateral probes 3/3, and +original lifecycle probes 7/7 GREEN. External source-order loss audit: seven +items fixed, one deferred original-artifact retrieval only; no unresolved +behavioral evidence gap. Utilities/tag isolation and mutation attribution from +the prior review are also fixed. Net production change versus reviewed head: +nine added TypeScript lines; no compressed-size measurement claimed here. + +## Second external review follow-up — 2026-09-10 + +Still local to reviewed head `885ed1c9165fbbebd5dfb278189d2452ff25574e`. + +- [x] R1: partial updates use the applied baseline plus pending writes. Removed + the retained `knownKeys` copy. Independent persistence publications now + reach Electric without an intervening subset acquisition. All six + eager/progressive/on-demand × targeted/full-reload cases were RED before + the fix and are GREEN now. The generated history crosses peer insertion, + deletion, reload, and subsequent partial updates, asserting public and + durable rows after each transition. It waits on a coordinator publication + marker even when the row set is unchanged; equality alone would false-green. +- [x] R6: new and deduplicated acquisitions both perform zero applied-key scans + in deterministic 10/100-row work tests. Stream callbacks inspect the + pending sync queue and progressive buffer once, then use keyed lookups. + This avoids O(applied rows) copies, not all work on queued operations. +- [x] R3: warn once per options descriptor when an older persistence wrapper + cannot attest hydration for a saved resume. Keep the safe fresh-fetch + fallback and give explicit package-update guidance. Compatible cleanup/ + restart does not repeat the warning. No mandatory persistence dependency. +- [x] R7: remove the stale row-returning capability type. Hydration is a barrier, + not a second persisted-row query; `scanPersisted` is a presence marker. +- [x] R4: retain the review's exact persisted-wrapper/real-insert path as a + regression, in addition to raw/once-spread utility tests. An acknowledgement + on A resolves A's insert after B starts; the prior shared-utils mutant + rejected that insert despite A receiving its txid. +- [x] Keep R8's requested todo record. R2's unknown-partial-resume error remains + intentional: main's apparent success materialized an incomplete row. + +Test-integrity checks: removing the applied-baseline fallback makes the peer +publication property RED; ignoring the pending overlay makes parked delete and +move-out RED (reset remains a passing control because truncation drains at once). +The random differential property also exposed an oracle-domain bug: its partition +filter checked only the first reset, admitting reset/subset/reset in one callback. +Validate every reset and pin that history. Do not change production semantics or +increase timeouts to accommodate an illegal publication-epoch partition. + +Limits retained from the source-order audit: cold-new-Collection tag restoration +is not added; different schemas for the same static shape and hand-copied wrappers +remain unproven paths, not refuted supported cases. Bounded match-buffer work and +live-session snapshot-evidence growth remain separate performance questions. +No service-backed or installed mixed-version conformance claim follows from the +mocked stream and capability-shape tests. + +Electric: 368/368 runtime tests in nine files; persistence: 75/75 runtime tests +in two files. Electric tsc and focused lint pass. The 92-test descriptor/oracle +rerun passes after the final diagnostic wording/type adjustment. Detailed logs: +`/private/tmp/1785-second-{electric,persistence,types,lint}-final.log`, +`/private/tmp/1785-second-final-focused.log`, and the two +`/private/tmp/1785-no-{baseline,overlay}-mutant-final-red.log` files. +Updated mutation recipes live in `packages/electric-db-collection/tests/ORACLE_MUTATIONS.md`. +Full source-order ledger: `/private/tmp/evaluate-1785-second-2bf032-ledger.md`. +Combined production diff versus the reviewed head: three fewer TypeScript lines +(core +5, Electric -8); this is not a bundle-size measurement. No commit or push. + +Final core rerun: 4,865/4,865 runtime tests, 150 files, at the same local runtime +(`/private/tmp/1785-second-core-final.log`). Total core/Electric/persistence: +5,308 passing runtime tests. No test was skipped to clear a failure. + +## CI follow-up: buffered move-outs orphan the progressive swap + +- [x] Reproduce CI's two stale-title assertions with the full real Electric + E2E suite: 143 pass, two progressive Moves cases fail. Focused cases alone + pass because earlier tests supply the tagged stream history that reaches + the initial buffering path. Log: /private/tmp/pr-1785-e2e-full-baseline.log. +- [x] Rule out mere replication delay: the tagged title remains stale under a + condition-based wait too. Discard the temporary wait and diagnostic edits; + retain both original E2E assertions and their timing. +- [x] Oracle first: add a nine-cell mode × move-out-count matrix plus generated + IDs/values/counts. Test every callback partition before initial up-to-date, + followed by a new insert and partial update. Two progressive cells and the + property RED; seven controls GREEN. Seed -1632249566, path 0:1. + Log: /private/tmp/pr-1785-buffered-moveout-oracle-red.log. +- [x] Fix the atomic swap's buffered move-out call to acknowledge its existing + transaction. The normal-stream flag is false there; passing it opened a + second transaction and stranded the original truncate. No new state or + weaker presence rule is needed. +- [x] GREEN: all 145 service-backed Electric E2E tests, all 378 Electric runtime + tests, package type checks, and focused ESLint. Logs: + /private/tmp/pr-1785-e2e-full-green.log, + /private/tmp/pr-1785-electric-green.log, + /private/tmp/pr-1785-moveout-lint.log. + +Local E2E used CI's Node 22.13 and a current Electric canary in isolated containers +on ports 55432/53000. The existing app containers were not modified. Node 24's +fetch rejects jsdom AbortSignals before tests start; the initially cached Electric +image also rejected offset=now. Neither setup failure is the PR correctness bug. + +Why earlier oracles missed it: post-ready move-outs and initial snapshot row +operations were tested separately. Neither followed a buffered initial move-out +with independent live work after the swap. The new property crosses that boundary +and checks public rows, rather than reading transaction flags into its model. diff --git a/typedoc.json b/typedoc.json new file mode 100644 index 0000000000..3bd16c29b0 --- /dev/null +++ b/typedoc.json @@ -0,0 +1,3 @@ +{ + "hidePageTitle": true +}