Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/export-with-publication-context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@tanstack/db': minor
---

Export `withPublicationContext`. A sync source that commits sync transactions to several Collections for one source state, such as one server checkpoint, can commit them inside one call: every live query that reads those Collections then runs after the last commit and publishes one batch, instead of running after each commit and possibly publishing intermediate results. A commit made while an optimistic transaction is persisting still applies when that transaction settles, in a later publication. Each Collection still notifies its own change subscribers when it commits, so this is not an atomic publication across the Collections.
25 changes: 25 additions & 0 deletions docs/guides/collection-options-creator.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,31 @@ The sync process follows this lifecycle:
6. **reportError(error)** - After the Collection is ready, report a sync problem while you keep trying. It sets the node's `failureReason` without changing its status or rows
7. **reportHealthy()** - After the Collection is ready, report that sync works again. It clears the `failureReason`, and does nothing when nothing is wrong

### Committing several Collections together

A source that syncs several Collections over one connection often receives one source state for all of them, such as one server checkpoint. Committed one by one, every live query that joins those Collections runs after each commit, over Collections that only some of the commits have reached, and can publish those intermediate results. Commit them inside one `withPublicationContext` call instead.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

The callback must be synchronous: the publication ends when it returns, so a commit after an `await` in it publishes on its own, and a rejected Promise is not handled as a throw. Do not commit inside a `transaction.mutate` callback within it either: live-query work inside `mutate` belongs to the transaction and publishes when `mutate` returns, before the commits after it. Await the source's input first, then commit inside the call:

```typescript
import { withPublicationContext } from '@tanstack/db'

const collectionsInCheckpoint = await receiveCheckpoint()
withPublicationContext(() => {
for (const { sync, changes } of collectionsInCheckpoint) {
try {
sync.begin()
for (const change of changes) sync.write(change)
sync.commit()
} catch (error) {
sync.fail(error)
}
}
})
```

Live queries then run after the last commit and publish one batch, with every commit that applied at once. A commit made while an optimistic transaction is persisting applies when that transaction settles, in a later publication: `commit()` returns a receipt instead of `true`, and the call does not wait for it. Each Collection still notifies its own change subscribers when it commits, so this is not an atomic publication across the Collections. Catch each commit's failure inside the callback: if the callback throws, the live-query work of the commits it already made is discarded, and those live queries miss them until their sources change again.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

### Reporting failures

Users see two separate signals: whether data can be shown, and whether it is
Expand Down
1 change: 1 addition & 0 deletions packages/db/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ export {
deleteDatabase,
} from './indexed-db-wrapper'
export * from './errors'
export { withPublicationContext } from './scheduler'
/** @internal Unstable protocol for persistence-aware collection adapters. */
export * from './sync-persistence'
export * from './sync-receipt'
Expand Down
28 changes: 28 additions & 0 deletions packages/db/src/scheduler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,34 @@ export function recordPublicationError(error: unknown): void {
* Runs one synchronous Collection publication inside a scheduler context.
* Nested publications share the outer context, so downstream live queries run
* only after every subscriber to the original committed batch has observed it.
*
* A sync source that commits sync transactions to several Collections for one
* source state, such as one server checkpoint, can commit them inside one call.
* Every live query that observes those Collections then runs after `publish`
* returns, with every commit that applied at once, and publishes one batch.
* Committed one by one, a live query that joins them runs after each commit,
* over Collections that only some of the commits have reached, and can publish
* those intermediate results.
*
* A commit that does not apply at once, because an optimistic transaction is
* persisting, applies when that transaction settles, in a later publication.
* This call does not wait for it, so live queries that read its Collection can
* publish without it first.
*
* Each Collection still applies its sync transaction and notifies its own
* change subscribers when it commits, so this is not an atomic publication
* across the Collections. A failure that a change subscriber reports is thrown
* from this call after the graphs ran. When `publish` itself throws, the queued
* graph work is discarded, so live queries miss the commits that `publish`
* already made: catch each commit's own failure inside `publish`.
*
* `publish` must be synchronous. The publication ends when it returns, so a
* commit after an `await` in it publishes on its own, and a rejected Promise is
* not handled as a throw. Await the source's input before this call.
*
* Inside `transaction.mutate`, live-query work belongs to the transaction and
* publishes when `mutate` returns. A commit made in a `mutate` callback inside
* `publish` therefore publishes before the commits after it.
*/
export function withPublicationContext<T>(publish: () => T): T {
if (activePublicationContext !== undefined) return publish()
Expand Down
200 changes: 199 additions & 1 deletion packages/db/tests/query/scheduler.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,11 @@ import {
import { CollectionConfigBuilder } from '../../src/query/live/collection-config-builder.js'
import { getCollectionBuilder } from '../../src/query/live/collection-registry.js'
import { CollectionSubscriber } from '../../src/query/live/collection-subscriber.js'
import { Query, createEffect } from '../../src/index.js'
import {
Query,
createEffect,
withPublicationContext as publicWithPublicationContext,
} from '../../src/index.js'
import {
flushPromises,
mockSyncCollectionOptions,
Expand Down Expand Up @@ -341,6 +345,200 @@ describe(`Collection publication scheduler context`, () => {
)
})

describe(`sync transactions of several Collections in one publication`, () => {
type SyncOperations<T extends object> = Parameters<SyncConfig<T>[`sync`]>[0]

// A left join shows a row for a user whose tasks have not arrived, so a live
// query that runs between the two commits publishes that intermediate row
async function setupJoinedSources(id: string) {
let usersSync!: SyncOperations<User>
let tasksSync!: SyncOperations<Task>
const users = createCollection<User>({
id: `${id}-users`,
getKey: (user) => user.id,
startSync: true,
sync: {
sync: (operations) => {
usersSync = operations
operations.markReady()
},
},
})
const tasks = createCollection<Task>({
id: `${id}-tasks`,
getKey: (task) => task.id,
startSync: true,
sync: {
sync: (operations) => {
tasksSync = operations
operations.markReady()
},
},
})
const assignments = createLiveQueryCollection({
id: `${id}-assignments`,
startSync: true,
query: (q) =>
q
.from({ user: users })
.join(
{ task: tasks },
({ user, task }) => eq(user.id, task.userId),
`left`,
)
.select(({ user, task }) => ({
name: user.name,
title: task.title,
})),
})
await assignments.preload()
const commitUser = (user: User) => {
usersSync.begin()
usersSync.write({ type: `insert`, value: user })
usersSync.commit()
}
const commitTask = (task: Task) => {
tasksSync.begin()
tasksSync.write({ type: `insert`, value: task })
tasksSync.commit()
}
const rows = () => assignments.toArray.map((row) => stripVirtualProps(row))
return { users, tasks, assignments, commitUser, commitTask, rows }
}

it(`publishes a joining live query once, after every commit applied`, async () => {
const { users, tasks, assignments, commitUser, commitTask, rows } =
await setupJoinedSources(`publication-batch`)
const order: Array<string> = []
const subscriptions = [
users.subscribeChanges(() => order.push(`users`)),
tasks.subscribeChanges(() => order.push(`tasks`)),
assignments.subscribeChanges((changes) =>
order.push(
`assignments ${JSON.stringify(changes.map(({ type }) => type))}`,
),
),
]

publicWithPublicationContext(() => {
commitUser({ id: 1, name: `Ada` })
commitTask({ id: 1, userId: 1, title: `Review` })
expect(rows()).toEqual([])
})

expect(rows()).toEqual([{ name: `Ada`, title: `Review` }])
// Not an atomic publication: each Collection notifies its own subscribers
// when it commits
expect(order).toEqual([`users`, `tasks`, `assignments ["insert"]`])
for (const subscription of subscriptions) subscription.unsubscribe()
})

it(`publishes the intermediate join of each commit without one`, async () => {
const { assignments, commitUser, commitTask, rows } =
await setupJoinedSources(`publication-per-commit`)
const published: Array<unknown> = []
const subscription = assignments.subscribeChanges((changes) =>
published.push(
changes.map(({ type, value }) => ({
type,
...stripVirtualProps(value),
})),
),
)

commitUser({ id: 1, name: `Ada` })
commitTask({ id: 1, userId: 1, title: `Review` })

expect(published).toEqual([
[{ type: `insert`, name: `Ada`, title: undefined }],
[
{ type: `insert`, name: `Ada`, title: `Review` },
{ type: `delete`, name: `Ada`, title: undefined },
],
])
expect(rows()).toEqual([{ name: `Ada`, title: `Review` }])
subscription.unsubscribe()
})

it(`discards the live-query work of earlier commits when publish throws`, async () => {
const { users, commitUser, commitTask, rows } =
await setupJoinedSources(`publication-throws`)
const failure = new Error(`source failed after a commit`)

expect(() =>
publicWithPublicationContext(() => {
commitUser({ id: 1, name: `Ada` })
commitTask({ id: 1, userId: 1, title: `Review` })
throw failure
}),
).toThrow(failure)

expect(users.size).toBe(1)
expect(rows()).toEqual([])
})

it(`publishes a commit held behind a persisting transaction in a later batch`, async () => {
const { tasks, assignments, commitUser, commitTask, rows } =
await setupJoinedSources(`publication-held`)
const persisting = createDeferred<void>()
const transaction = createTransaction({
mutationFn: () => persisting.promise,
})
transaction.mutate(() =>
tasks.insert({ id: 99, userId: 99, title: `optimistic` }),
)
const published: Array<Array<string>> = []
const subscription = assignments.subscribeChanges((changes) =>
published.push(
changes.map(({ type, value }) => `${type} ${value.title}`),
),
)

publicWithPublicationContext(() => {
commitUser({ id: 1, name: `Ada` })
commitTask({ id: 1, userId: 1, title: `Review` })
})
expect(published).toEqual([[`insert undefined`]])

persisting.resolve()
await transaction.when(`settled`)
expect(published).toEqual([
[`insert undefined`],
[`insert Review`, `delete undefined`],
])
expect(rows()).toEqual([{ name: `Ada`, title: `Review` }])
subscription.unsubscribe()
})

it(`publishes a commit made inside transaction.mutate before the commits after it`, async () => {
const { assignments, commitUser, commitTask, rows } =
await setupJoinedSources(`publication-in-mutate`)
const transaction = createTransaction({
mutationFn: async () => {},
autoCommit: false,
})
const published: Array<Array<string>> = []
const subscription = assignments.subscribeChanges((changes) =>
published.push(
changes.map(({ type, value }) => `${type} ${value.title}`),
),
)

publicWithPublicationContext(() => {
transaction.mutate(() => commitUser({ id: 1, name: `Ada` }))
commitTask({ id: 1, userId: 1, title: `Review` })
})

expect(published).toEqual([
[`insert undefined`],
[`insert Review`, `delete undefined`],
])
expect(rows()).toEqual([{ name: `Ada`, title: `Review` }])
subscription.unsubscribe()
transaction.rollback()
})
})

describe(`live query scheduler`, () => {
it(`does not deliver a source batch after a snapshotted listener unsubscribes`, async () => {
let begin!: () => void
Expand Down
Loading