diff --git a/architecture/consistency.mdx b/architecture/consistency.mdx index f717cdad3..756d09a72 100644 --- a/architecture/consistency.mdx +++ b/architecture/consistency.mdx @@ -24,7 +24,7 @@ While mutations are present in the upload queue, the client does not advance to Only once all the client-side mutations have been acknowledged by the server, and the data for that new checkpoint is downloaded by the client, does the client advance to the next checkpoint. This ensures that the operations are always ordered correctly on the client. -There is one nuanced case here, which is buckets with [Priority 0](/sync/advanced/prioritized-sync#special-case-priority-0) if you are using [Prioritized Syncing](/sync/advanced/prioritized-sync). +There is one nuanced case here, which is buckets with [Priority 0](/sync/streams/prioritized-sync#special-case-priority-0) if you are using [Prioritized Syncing](/sync/streams/prioritized-sync). ## Types of Client-Side Mutations/Operations diff --git a/client-sdks/advanced/raw-tables.mdx b/client-sdks/advanced/raw-tables.mdx index de17713bd..ebe568c96 100644 --- a/client-sdks/advanced/raw-tables.mdx +++ b/client-sdks/advanced/raw-tables.mdx @@ -507,7 +507,7 @@ Raw tables support advanced table constraints including foreign keys. When enabl 1. While PowerSync will always apply synced data in a transaction, there is no way to control the order in which rows get applied. For this reason, foreign keys need to be configured with `DEFERRABLE INITIALLY DEFERRED`. -2. When using [stream priorities](/sync/advanced/prioritized-sync), you need to ensure you don't have foreign keys from high-priority +2. When using [stream priorities](/sync/streams/prioritized-sync), you need to ensure you don't have foreign keys from high-priority rows to lower-priority data. PowerSync applies data in one transaction per priority, so these foreign keys would not work. 3. As usual when using foreign keys, note that they need to be explicitly enabled with `pragma foreign_keys = on`. diff --git a/configuration/app-backend/client-side-integration.mdx b/configuration/app-backend/client-side-integration.mdx index 8768f56b6..c8c76618c 100644 --- a/configuration/app-backend/client-side-integration.mdx +++ b/configuration/app-backend/client-side-integration.mdx @@ -10,7 +10,7 @@ After you've [instantiated](/intro/setup-guide#instantiate-the-powersync-databas | Purpose | Description | |---------|-------------| -| **Uploading mutations to your backend:** | Mutations that are made to the client-side SQLite database are uploaded to your backend application, where you control how they're applied to your backend source database (Postgres, MongoDB, MySQL, SQL Server, or Convex). This is how PowerSync achieves bi-directional syncing of data: The [PowerSync Service](/architecture/powersync-service) provides the _server-to-client read path_ based on your [Sync Streams or Sync Rules (legacy)](/sync/overview), and the _client-to-server write path_ goes via your backend. | +| **Uploading mutations to your backend:** | Mutations that are made to the client-side SQLite database are uploaded to your backend application, where you control how they're applied to your backend source database (Postgres, MongoDB, MySQL, SQL Server, or Convex). This is how PowerSync achieves bi-directional syncing of data: The [PowerSync Service](/architecture/powersync-service) provides the _server-to-client read path_ based on your [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)), and the _client-to-server write path_ goes via your backend. | | **Authentication integration:** (optional) | PowerSync uses JWTs for authentication between the Client SDK and PowerSync Service. Some [authentication providers](/configuration/auth/overview#common-authentication-providers) generate JWTs for users which PowerSync can verify directly. For others, some code must be [added to your application backend](/configuration/auth/custom) to generate the JWTs. | diff --git a/configuration/source-db/postgres-maintenance.mdx b/configuration/source-db/postgres-maintenance.mdx index f5ea3e3ef..263486cc4 100644 --- a/configuration/source-db/postgres-maintenance.mdx +++ b/configuration/source-db/postgres-maintenance.mdx @@ -7,7 +7,7 @@ description: "Manage Postgres replication slots and WAL lag for reliable PowerSy Postgres logical replication slots are used to keep track of [replication](/architecture/powersync-service#replication-from-the-source-database) progress (recorded as a [LSN](https://www.postgresql.org/docs/current/datatype-pg-lsn.html)). -Every time a new version of [Sync Streams or Sync Rules](/sync/overview) is deployed, PowerSync creates a new replication slot. Once the new version is fully processed, PowerSync switches to use the new slot and deletes the old one. The Service logs these steps and, during a snapshot, how much WAL budget remains. See [Postgres Replication Slots and WAL Budget](/debugging/log-reference#postgres-replication-slots-and-wal-budget) in the Log Reference. +Every time a new version of [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)) is deployed, PowerSync creates a new replication slot. Once the new version is fully processed, PowerSync switches to use the new slot and deletes the old one. The Service logs these steps and, during a snapshot, how much WAL budget remains. See [Postgres Replication Slots and WAL Budget](/debugging/log-reference#postgres-replication-slots-and-wal-budget) in the Log Reference. The replication slots can be viewed using this query: diff --git a/debugging/error-codes.mdx b/debugging/error-codes.mdx index affbf86b9..4bf69b795 100644 --- a/debugging/error-codes.mdx +++ b/debugging/error-codes.mdx @@ -9,7 +9,7 @@ This reference documents PowerSync error codes organized by component, with trou ## PSYNC_Rxxxx: Sync Config issues - **PSYNC_R0001**: - Catch-all [Sync Config](/sync/overview) parsing error, if no more specific error is available + Catch-all [Sync Config](/sync/streams/quickstart#defining-streams) parsing error, if no more specific error is available - **PSYNC_R2201**: A table or schema wildcard (`%`) is not supported by the configured source connector. diff --git a/debugging/troubleshooting.mdx b/debugging/troubleshooting.mdx index 89605bdb7..7967620d1 100644 --- a/debugging/troubleshooting.mdx +++ b/debugging/troubleshooting.mdx @@ -131,6 +131,6 @@ Sync & API logs in the [PowerSync Dashboard](https://dashboard.powersync.com/) r #### Common Causes of Latency * **Large initial sync**: if your Sync Streams/Rules result in a large dataset, the first sync after connecting will be slow. Inspect bucket sizes and sync state with the [Sync Diagnostics Client](/tools/diagnostics-client). -* **Upload queue blocking downloads**: by default, uploads are processed before downloads, so a backlogged upload queue delays receiving new data. Buckets and streams at [priority 0](/sync/advanced/prioritized-sync) are not blocked by uploads, but come with the trade-off of potential sync inconsistencies. +* **Upload queue blocking downloads**: by default, uploads are processed before downloads, so a backlogged upload queue delays receiving new data. Buckets and streams at [priority 0](/sync/streams/prioritized-sync) are not blocked by uploads, but come with the trade-off of potential sync inconsistencies. * **Replication lag on the source database**: high write volume, long-running transactions, bulk updates, or backfills can cause replication to fall behind faster than the service can drain it. See [Replication Lag](/maintenance-ops/replication-lag) for source-specific causes and fixes. * **Too many buckets per user**: incremental sync overhead scales roughly linearly with the number of buckets per user. See [Too Many Buckets](#psync_s2305-too-many-buckets-/-parameter-query-results) above. diff --git a/docs.json b/docs.json index c57a8c129..50a4eec91 100644 --- a/docs.json +++ b/docs.json @@ -183,42 +183,29 @@ ] }, { - "group": "Sync Streams & Rules", + "group": "Sync Streams", "icon": "arrows-rotate", "pages": [ - "sync/overview", + "sync/streams/overview", + "sync/streams/quickstart", { - "group": "Sync Streams", + "group": "Usage", "pages": [ - "sync/streams/overview", - "sync/streams/parameters", "sync/streams/queries", + "sync/streams/parameters", "sync/streams/ctes", "sync/streams/bucket-count", - "sync/streams/examples", "sync/streams/client-usage", - "sync/streams/migration" + "sync/streams/prioritized-sync", + "sync/streams/examples" ] }, { - "group": "Sync Rules (Legacy)", - "pages": [ - "sync/rules/overview", - "sync/rules/organize-data-into-buckets", - "sync/rules/global-buckets", - "sync/rules/parameter-queries", - "sync/rules/data-queries", - "sync/rules/many-to-many-join-tables", - "sync/rules/client-parameters" - ] - }, - "sync/types", - { - "group": "Supported SQL", + "group": "Reference", "pages": [ "sync/supported-sql", "sync/grammar/sync-streams/index", - "sync/grammar/sync-rules/index" + "sync/types" ] }, { @@ -226,7 +213,6 @@ "pages": [ "sync/advanced/overview", "sync/advanced/reducing-bucket-count", - "sync/advanced/prioritized-sync", "sync/advanced/client-id", "sync/advanced/case-sensitivity", "sync/advanced/compatibility", @@ -237,6 +223,20 @@ "sync/advanced/partitioned-tables", "sync/advanced/sharded-databases" ] + }, + { + "group": "Sync Rules (Legacy)", + "pages": [ + "sync/rules/migrate-to-sync-streams", + "sync/rules/overview", + "sync/rules/organize-data-into-buckets", + "sync/rules/global-buckets", + "sync/rules/parameter-queries", + "sync/rules/data-queries", + "sync/rules/many-to-many-join-tables", + "sync/rules/client-parameters", + "sync/grammar/sync-rules/index" + ] } ] }, @@ -784,6 +784,14 @@ "source": "/usage/sync-streams", "destination": "/sync/streams/overview" }, + { + "source": "/sync/overview", + "destination": "/sync/streams/overview" + }, + { + "source": "/sync/streams/migration", + "destination": "/sync/rules/migrate-to-sync-streams" + }, { "source": "/usage/sync-rules/types", "destination": "/sync/types" @@ -802,7 +810,11 @@ }, { "source": "/usage/use-case-examples/prioritized-sync", - "destination": "/sync/advanced/prioritized-sync" + "destination": "/sync/streams/prioritized-sync" + }, + { + "source": "/sync/advanced/prioritized-sync", + "destination": "/sync/streams/prioritized-sync" }, { "source": "/usage/sync-rules/client-id", diff --git a/snippets/binary-type.mdx b/snippets/binary-type.mdx index ace29f838..530cededc 100644 --- a/snippets/binary-type.mdx +++ b/snippets/binary-type.mdx @@ -1,3 +1,3 @@ - Binary data can be accessed in the Sync Streams / Sync Rules, but cannot be used as [parameters](/sync/overview#how-it-works). To sync binary columns/fields to clients, those columns need to be converted to hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). + Binary data can be accessed in Sync Streams (or legacy Sync Rules), but cannot be used as a parameter (see [Sync Streams parameters](/sync/streams/parameters) or [Sync Rules client parameters](/sync/rules/client-parameters)). To sync binary columns/fields to clients, those columns need to be converted to hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). \ No newline at end of file diff --git a/snippets/stream-definition-reference.mdx b/snippets/stream-definition-reference.mdx index 8740faf00..af5e93310 100644 --- a/snippets/stream-definition-reference.mdx +++ b/snippets/stream-definition-reference.mdx @@ -28,5 +28,5 @@ streams: | `queries` | — | Array of queries defining which data to sync. More efficient than defining separate streams: the client manages one subscription and PowerSync merges the data from all queries (see [Multiple Queries per Stream](/sync/streams/queries#multiple-queries-per-stream)). | | `with` | — | [CTEs](/sync/streams/ctes) available to this stream's queries. Define the `with` block inside each stream. | | `auto_subscribe` | `false` | When `true`, clients automatically subscribe on connect. | -| `priority` | — | Sync priority (lower value = higher priority). See [Prioritized Sync](/sync/advanced/prioritized-sync). | +| `priority` | — | Sync priority (lower value = higher priority). See [Prioritized Sync](/sync/streams/prioritized-sync). | | `accept_potentially_dangerous_queries` | `false` | Silences security warnings when queries use client-controlled parameters (i.e. _connection parameters_ and _subscription parameters_), as opposed to _authentication parameters_ that are signed as part of the JWT. Set to `true` only if you've verified the query is safe. See [Using Parameters](/sync/streams/parameters). | diff --git a/sync/advanced/overview.mdx b/sync/advanced/overview.mdx index de09cbf17..070471c18 100644 --- a/sync/advanced/overview.mdx +++ b/sync/advanced/overview.mdx @@ -1,15 +1,17 @@ --- title: "Advanced Topics" -description: "Advanced Sync Streams and Sync Rules topics." +description: "Sync Streams topics you only need when a specific condition applies to your data, source database, or deployment." sidebarTitle: Overview --- +These pages cover situations that do not apply to every project. Use them when a specific condition applies to your data, source database, or deployment. For pages that every project needs, see [Writing Queries](/sync/streams/queries) and the other Usage pages. + - + diff --git a/sync/advanced/storage-version-4.mdx b/sync/advanced/storage-version-4.mdx index 56d5922ba..2b6b4b9ed 100644 --- a/sync/advanced/storage-version-4.mdx +++ b/sync/advanced/storage-version-4.mdx @@ -23,7 +23,7 @@ The PowerSync Cloud and self-hosted columns below apply during the Beta only. On | Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | | S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | -Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/streams/migration). +Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). ## Opt In diff --git a/sync/grammar/sync-streams/index.mdx b/sync/grammar/sync-streams/index.mdx index cc3e7f591..a4aab0fcb 100644 --- a/sync/grammar/sync-streams/index.mdx +++ b/sync/grammar/sync-streams/index.mdx @@ -1,5 +1,5 @@ --- -title: "Grammar Reference (Sync Streams)" +title: "Grammar Reference" description: "Railroad diagram reference for the SQL grammar supported in Sync Streams queries." --- diff --git a/sync/overview.mdx b/sync/overview.mdx deleted file mode 100644 index b2024c134..000000000 --- a/sync/overview.mdx +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: "Sync Streams and Sync Rules" -sidebarTitle: "Overview" -description: "PowerSync Sync Streams and the legacy Sync Rules allow developers to control which data syncs to which clients/devices (i.e. they enable partial sync)." ---- - - -## Sync Streams — Recommended - -With [Sync Streams](/sync/streams/overview), you write SQL-like queries to define streams of data. Clients subscribe to the streams they need, either on-demand or automatically on connect. Sync Streams are the recommended path to achieve partial sync for both new and existing projects. - -Key improvements in Sync Streams over legacy Sync Rules include: -- **On-demand syncing**: You define Sync Streams on the PowerSync Service, and a client can then subscribe to them one or more times with different parameters, on-demand. You still have the option of auto-subscribing streams when a client connects, for "sync data upfront" behavior. -- **Temporary caching-like behavior**: Each subscription includes a configurable TTL that keeps data active after the client unsubscribes, acting as a warm cache for re-subscribing. -- **Simpler developer experience**: Simplified syntax and mental model, and capabilities such as your UI components automatically managing subscriptions (for example, React hooks). - -If you're on Sync Rules, you can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See the [migration guide](/sync/streams/migration) for details. - - - - -## Sync Rules (Legacy) - -Sync Rules is the legacy approach for controlling data sync. It remains available and supported for existing projects: - - - -If you're currently using Sync Rules and want to migrate to Sync Streams, see our [migration docs](/sync/streams/migration). - - -## How It Works - -You may also find it useful to look at the [PowerSync Service architecture](/architecture/powersync-service) for background. - -Each [PowerSync Service](/architecture/powersync-service) instance has a deployed _Sync Streams_ (or legacy _Sync Rules_) configuration. This takes the form of a YAML file which contains: -- **In the case of Sync Streams:** Definitions of the streams that exist, with a SQL-like query (which can also contain limited subqueries), which defines the data in the stream, and references the necessary parameters. -- **In the case of Sync Rules:** Definitions of the different [buckets](/architecture/powersync-service#bucket-system) that exist, with SQL-like queries to specify the parameters used by each bucket (if any), as well as the data contained in each bucket. - - - - - - -A _parameter_ is a value that can be used in Sync Streams (or legacy Sync Rules) to create dynamic sync behavior for each user/client. Each client syncs only the relevant [_buckets_](/architecture/powersync-service#bucket-system) based on the parameters for that client. -* Sync Streams can make use of _authentication parameters_ from the JWT token (such as the user ID or other JWT claims), _connection parameters_ (specified at connection), and _subscription parameters_ (specified by the client when it subscribes to a stream at any time). See [Using Parameters](/sync/streams/parameters). -* Sync Rules can make use of _authentication parameters_ from the JWT token, as well as [_client parameters_](/sync/rules/client-parameters) (passed directly from the client when it connects to the PowerSync Service). - -It is also possible to have buckets/streams with no parameters. In the case of Sync Rules, these buckets sync to all users/clients automatically. - - - -The concept of _buckets_ is core to PowerSync and key to its performance and scalability. The [PowerSync Service architecture overview](/architecture/powersync-service) provides more background on this. -* In _Sync Streams_, buckets and parameters are implicit — they are automatically created based on the streams, their queries and subqueries. You don't need to explicitly define the buckets that exist. -* In legacy _Sync Rules_, buckets and their parameters are [explicitly defined](/sync/rules/overview#bucket-definition). - - -There are limitations on the SQL syntax and functionality that is supported in Sync Streams and Sync Rules. See [Supported SQL](/sync/supported-sql) for details and limitations. - -In addition to filtering data based on parameters, Sync Streams and Sync Rules also enable: - -* Selecting only specific tables/collections and columns/fields to sync. -* Filtering data based on static conditions. -* Transforming column/field names and values. - - -### Sync Streams/Rules Determine Replication From the Source Database - -A PowerSync Service instance [replicates and transforms](/architecture/powersync-service#replication-from-the-source-database) relevant data from your backend source database according to your Sync Streams/Sync Rules. During replication, data and metadata are persisted in [buckets](/architecture/powersync-service#bucket-system) on the PowerSync Service. Buckets are incrementally updated so that they contain the latest state as well as a history of changes (operations). This is key to how PowerSync achieves efficient delta syncing — having the operation history for each bucket allows clients to sync only the deltas that they need to get up to date (see [Protocol](/architecture/powersync-protocol#protocol) for more details). - - -As a practical example, let's say you have a bucket named `user_todo_lists` that contains the to-do lists for a user, and that bucket utilizes a `user_id` parameter (which will be embedded in the JWT). Now let's say users with IDs `A` and `B` exist in the source database. PowerSync will then replicate data from the source database and create individual buckets with IDs `user_todo_lists["A"]` and `user_todo_lists["B"]`. When the user with ID `A` connects, they can efficiently sync just the bucket with ID `user_todo_lists["A"]`. - - - - - - - -### Sync Streams/Rules Determine Real-Time Streaming Sync to Clients - -Whenever buckets are updated (buckets added or removed, or operations added to existing buckets), these changes are [streamed in real-time](/architecture/powersync-service#streaming-sync) to clients based on the Sync Streams/Sync Rules. - -This syncing behavior can be highly dynamic: in the case of Sync Streams, syncing will dynamically adjust based on the stream subscriptions (which can make use of _subscription parameters_), as well as _connection parameters_ and _authentication parameters_ (from the JWT). In the case of Sync Rules, syncing will dynamically adjust based on changes in _client parameters_ and _authentication parameters_. - -The bucket data is persisted in SQLite on the client-side, where it is easily queryable based on the [client-side schema](/intro/setup-guide#define-your-client-side-schema), which corresponds to the Sync Streams/Rules. - -For more information on the client-side SQLite database structure, see [Client Architecture](/architecture/client-architecture#client-side-schema-and-sqlite-database-structure). - - - - - - - diff --git a/sync/rules/client-parameters.mdx b/sync/rules/client-parameters.mdx index 08a814109..cdd33224b 100644 --- a/sync/rules/client-parameters.mdx +++ b/sync/rules/client-parameters.mdx @@ -16,7 +16,7 @@ PowerSync already supports using **token parameters** in parameter queries. An e [Sync Streams](/sync/streams/overview) make it easier to manage dynamic parameters, especially for apps where parameters are managed across different UI components and tabs. Sync Streams offer _subscription parameters_ (specified when subscribing to a stream) and _connection parameters_ (the equivalent of client parameters). - We recommend Sync Streams for new projects, and [migrating](/sync/streams/migration) existing projects. + We recommend Sync Streams for new projects, and [migrating](/sync/rules/migrate-to-sync-streams) existing projects. ### Usage diff --git a/sync/rules/migrate-to-sync-streams.mdx b/sync/rules/migrate-to-sync-streams.mdx new file mode 100644 index 000000000..69a2c4f11 --- /dev/null +++ b/sync/rules/migrate-to-sync-streams.mdx @@ -0,0 +1,286 @@ +--- +title: "Migrate to Sync Streams" +description: "Convert legacy Sync Rules to Sync Streams without changing what your app syncs, then adopt on-demand syncing over time." +--- + +import StreamDefinitionReference from '/snippets/stream-definition-reference.mdx'; + + +Sync Streams support everything Sync Rules do (and more), and migrating does not change what your app syncs. The [migration tool](#migrate-with-the-migration-tool) converts your bucket definitions into streams with the same behavior: + +- Every generated stream has `auto_subscribe: true`, so clients keep syncing all their data when they connect, exactly as they do with Sync Rules. +- [Client Parameters](/sync/rules/client-parameters) become [connection parameters](/sync/streams/parameters#connection-parameters). Your app passes them the same way when it connects. +- Once your SDKs meet the [minimum versions](#requirements), no client-side code changes are needed. + +In most cases you migrate to stay compatible first, then [adopt Sync Streams features](#adopt-sync-streams-features) such as on-demand syncing one stream at a time. + +If your Sync Config has a `bucket_definitions:` section, you use Sync Rules and this guide applies to you. If it only has `streams:`, you already use Sync Streams and no action is needed. + +## Why Migrate? + +Beyond matching Sync Rules, Sync Streams add: + +- **More expressive queries:** Stream queries support JOINs, [CTEs](/sync/streams/ctes), subqueries, and [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream), with syntax closer to plain SQL. You write one query instead of separate `parameters:` and `data:` blocks. +- **On-demand syncing:** Define a stream once, then subscribe from your app one or more times with different parameters. Each subscription has its own lifecycle, so two screens or browser tabs can subscribe to the same stream independently. With Sync Rules, Client Parameters approximate this. You have to aggregate the parameter values yourself across screens and tabs, and remove them when they are no longer needed. +- **Built-in caching:** Each subscription has a configurable `ttl` that keeps data on the device after unsubscribing. When users return to a screen, the data is often already available. +- **Framework integration:** [React hooks, Vue composables, TanStack Query, and Kotlin Compose extensions](/sync/streams/client-usage#framework-integrations) let UI components manage subscriptions based on what is rendered. +- **Access to new features:** Newer PowerSync Service features such as [incremental reprocessing](/sync/advanced/storage-version-4) require Sync Streams. + +## Requirements + +- PowerSync Service v1.20.0+ (Cloud instances already meet this) +- An SDK version that supports Sync Streams (see table). Streams run on the [Rust-based sync client](https://releases.powersync.com/announcements/improved-sync-performance-in-our-client-sdks), which is the default in current SDKs. If your version is between the two columns, enable it manually. +- `config: edition: 3` in your Sync Config (the migration tool sets this) + + + +| SDK | Minimum Version | Rust Client Default | +|-----|-----------------|---------------------| +| JS Web | v1.27.0 | v1.32.0 | +| React Native | v1.25.0 | v1.29.0 | +| React hooks | v1.8.0 | — | +| Node.js | v0.11.0 | v0.16.0 | +| Capacitor | v0.0.1 | v0.3.0 | +| Tauri | v0.0.1 | Always (Rust client only) | +| Dart/Flutter | v1.16.0 | v1.17.0 | +| Kotlin | v1.7.0 | v1.9.0 | +| Swift | v1.11.0 | v1.8.0 | +| .NET | v0.0.8-alpha.1 | v0.0.5-alpha.1 | + + + +If you're on an SDK version below the "Rust Client Default" version, enable the Rust client manually: + +**JavaScript:** +```js +await db.connect(new MyConnector(), { + clientImplementation: SyncClientImplementation.RUST +}); +``` + +**Dart:** +```dart +database.connect( + connector: YourConnector(), + options: const SyncOptions( + syncImplementation: SyncClientImplementation.rust, + ), +); +``` + +**Kotlin:** +```kotlin +database.connect(MyConnector(), options = SyncOptions( + newClientImplementation = true, +)) +``` + +**Swift:** +```swift +import PowerSync + +try await db.connect(connector: connector, options: ConnectOptions( + newClientImplementation: true, +)) +``` + + + +## Migrate With the Migration Tool + + + + Use one of the following: + + - **PowerSync Dashboard:** Click **Migrate to Sync Streams**. The Dashboard converts the instance's deployed Sync Rules and opens the result as a draft for you to review. + - **CLI:** Run `powersync migrate sync-rules`. By default the command reads `sync-config.yaml` in your `powersync` config directory and overwrites it with the result. Use `--input-file` and `--output-file` to read from and write to other paths. See the [command reference](https://github.com/powersync-ja/powersync-cli/blob/main/cli/README.md#powersync-migrate-sync-rules) for all flags. + + + Compare the draft with your Sync Rules. See [What the Tool Generates](#what-the-tool-generates) for how the output maps to your bucket definitions, and [What to Check Before You Deploy](#what-to-check-before-you-deploy) for the items that need your attention. + + + Deploy the draft from the Dashboard or with `powersync deploy sync-config`. This works like any other Sync Config deploy: the Service reprocesses your data in the background while the current version keeps serving clients, then switches over without downtime. After the switch, each client does a one-time full re-sync. + + + +### What the Tool Generates + +The following Sync Rules define global data, user-scoped data, a parameter query that reads from a table, and a Client Parameter: + +```yaml +bucket_definitions: + global: + data: + - SELECT * FROM categories + user_lists: + parameters: SELECT request.user_id() as user_id + data: + - SELECT * FROM lists WHERE owner_id = bucket.user_id + list_todos: + parameters: SELECT id as list_id FROM lists WHERE owner_id = request.user_id() + data: + - SELECT * FROM todos WHERE list_id = bucket.list_id + page_posts: + parameters: SELECT request.parameters() ->> 'page_number' as page_number + data: + - SELECT * FROM posts WHERE page_number = bucket.page_number +``` + +The migration tool converts them to: + +```yaml +config: + edition: 3 +streams: + migrated_to_streams: + auto_subscribe: true + with: + list_todos_param: SELECT id AS list_id FROM lists WHERE owner_id = auth.user_id() + queries: + # Translated from "global" bucket definition. + - SELECT * FROM categories + # Translated from "user_lists" bucket definition. + - SELECT * FROM lists WHERE owner_id = auth.user_id() + # Translated from "list_todos" bucket definition. + - "SELECT todos.* FROM todos,list_todos_param AS bucket WHERE todos.list_id = bucket.list_id" + # Translated from "page_posts" bucket definition. + - SELECT * FROM posts WHERE page_number = connection.parameter('page_number') +``` + +The tool applies these rules: + +- **Compatibility edition:** It sets `config: edition: 3`, which Sync Streams require, and keeps any other options in your `config` block. +- **One stream per priority:** Bucket definitions with the same [priority](/sync/streams/prioritized-sync) are merged into one stream named `migrated_to_streams`. Comments mark which bucket definition each group of queries came from. If your bucket definitions use different priorities, the tool creates one stream per priority, named `migrated_to_streams_prio_`. +- **Same sync behavior:** Every stream has `auto_subscribe: true`. Queries are always written as a `queries:` list so that you can add more. +- **Parameters:** `request.*` functions become `auth.*` and `connection.*` functions. See [Parameter Syntax Changes](#parameter-syntax-changes) for the full mapping. Parameter queries that only select request values, such as `SELECT request.user_id() as user_id`, are replaced by those values in the data queries: `bucket.user_id` becomes `auth.user_id()`. Parameter queries that read from a table become CTEs in a `with:` block, named `_param`, and the data queries join them under the alias `bucket`. +- **Cleanup:** The `bucket_definitions:` section is removed. + +### What to Check Before You Deploy + +- **Compatibility edition:** If your Sync Rules had no `edition` set, `edition: 3` also turns on the edition 2 fixes, such as ISO 8601 timestamp formatting and custom Postgres type handling. These change how some values look in the client database. See [Compatibility](/sync/advanced/compatibility) for the full list. To keep the old behavior for a fix, set its option to `false` next to the edition: + + ```yaml + config: + edition: 3 + timestamps_iso8601: false + ``` + +- **Queries the tool cannot convert:** This is rare. When it happens, the tool stops and reports the query it could not parse. The Dashboard shows the error and its line in the validation panel, and the CLI prints it. Convert that bucket definition by hand using [Parameter Syntax Changes](#parameter-syntax-changes), or ask on [Discord](https://discord.gg/powersync). + +## Adopt Sync Streams Features + +After the deploy, the generated streams behave like your bucket definitions did. You can then make the following changes one stream at a time. Changes that keep `auto_subscribe: true` need no client changes. Changes that remove it or change the parameter type need an app update, because clients only receive that data once they subscribe. + +| Change | Client changes | +|--------|----------------| +| [Split the merged stream](#split-the-merged-stream) into named streams | None | +| [Replace parameter CTEs with subqueries](#replace-parameter-ctes-with-subqueries) or JOINs | None | +| [Sync data on demand](#sync-data-on-demand) instead of on connect | Subscribe to the stream from the app | +| [Convert connection parameters to subscription parameters](#convert-connection-parameters-to-subscription-parameters) | Replace connect-time `params` with subscriptions | + +When old and new app versions coexist, keep the old stream and add the changed one under a new name. Newer app versions [opt out of auto-subscribed streams](/sync/streams/client-usage#opting-out-of-auto-subscribed-streams) and subscribe explicitly. Remove the old stream when the older app versions are retired. + +### Split the Merged Stream + +The tool merges your bucket definitions into one stream. Splitting them into named streams makes each stream's purpose visible and lets you change each one independently later. Global data, which syncs the same rows to every user, and user-scoped data both keep `auto_subscribe: true`. Set [`priority`](/sync/streams/prioritized-sync) per stream where needed: + +```yaml +streams: + categories: + auto_subscribe: true + query: SELECT * FROM categories + user_lists: + auto_subscribe: true + priority: 1 + query: SELECT * FROM lists WHERE owner_id = auth.user_id() +``` + +Both streams still sync on connect, so no client changes are needed. + +### Replace Parameter CTEs With Subqueries + +A parameter query that read from a table becomes a CTE that the data query joins. A subquery expresses the same filter in one statement. The generated `list_todos` queries above become: + +```yaml +streams: + list_todos: + auto_subscribe: true + query: SELECT * FROM todos WHERE list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) +``` + +The same rows sync, so no client changes are needed. See [Writing Queries](/sync/streams/queries) for JOINs, nested subqueries, and multiple queries per stream. + +### Sync Data On Demand + +A stream without `auto_subscribe: true` syncs only while the app is subscribed to it. Use this for data that a user needs on one screen, such as the todos of the list they opened. Add a [subscription parameter](/sync/streams/parameters#subscription-parameters) for the value the screen provides, and keep an `auth.*` filter so that clients can only subscribe to data they may access: + +```yaml +streams: + list_todos: + query: | + SELECT * FROM todos + WHERE list_id = subscription.parameter('list_id') + AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) +``` + +The app subscribes when the screen opens and unsubscribes when it closes. The subscription's [TTL](/sync/streams/client-usage#ttl-time-to-live) keeps the data on the device afterwards, so returning to the screen is instant: + +```js +const sub = await db.syncStream('list_todos', { list_id: listId }).subscribe(); +await sub.waitForFirstSync(); + +// When the screen closes +sub.unsubscribe(); +``` + +See [Client-Side Usage](/sync/streams/client-usage) for each SDK and for [framework integrations](/sync/streams/client-usage#framework-integrations) that manage subscriptions from UI components. + +### Convert Connection Parameters to Subscription Parameters + +The tool converts Client Parameters to connection parameters because they behave the same way: the app passes them in `connect()`, they apply to the whole connection, and the app has to reconnect to change them. This keeps your existing behavior, but it is not the best fit for on-demand syncing. Subscription parameters let the app subscribe to the same stream several times with different values, without reconnecting, and each subscription has its own lifecycle. If you prefer to keep passing values at connect time, keep the connection parameters. They need no client changes. + +Before, the migrated `page_posts` query syncs one page per connection: + +```yaml +streams: + page_posts: + auto_subscribe: true + query: SELECT * FROM posts WHERE page_number = connection.parameter('page_number') +``` + +```js +await db.connect(connector, { + params: { page_number: 1 } +}); +``` + +After, the app subscribes to the pages it needs: + +```yaml +streams: + page_posts: + query: SELECT * FROM posts WHERE page_number = subscription.parameter('page_number') +``` + +```js +await db.connect(connector); + +// Subscribe to multiple pages simultaneously +const page1 = await db.syncStream('page_posts', { page_number: 1 }).subscribe(); +const page2 = await db.syncStream('page_posts', { page_number: 2 }).subscribe(); +``` + +## Parameter Syntax Changes + +| Sync Rules | Sync Streams | +|------------|--------------| +| `request.user_id()` | `auth.user_id()` | +| `request.jwt() ->> 'claim'` | `auth.parameter('claim')` | +| `request.jwt()` | `auth.parameters()` | +| `request.parameters() ->> 'key'` | `connection.parameter('key')` ([connection parameter](/sync/streams/parameters#connection-parameters)). Use `subscription.parameter('key')` ([subscription parameter](/sync/streams/parameters#subscription-parameters)) when you convert the stream to on-demand syncing. | +| `request.parameters()` | `connection.parameters()` | +| `bucket.param_name` | Use the parameter directly in the query, for example `auth.user_id()`, or a subquery. See [Using Subqueries](/sync/streams/queries#using-subqueries). | + +## Stream Definition Reference + + diff --git a/sync/rules/overview.mdx b/sync/rules/overview.mdx index 6d85e0bcc..17921d0e2 100644 --- a/sync/rules/overview.mdx +++ b/sync/rules/overview.mdx @@ -11,7 +11,7 @@ Sync Rules are PowerSync's original system for results in fewer sync buckets. +If multiple streams share the same filtering logic, consider using [CTEs](/sync/streams/ctes) to avoid repetition and [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream) so the client only needs to manage one subscription instead of multiple. This is more efficient and results in fewer sync buckets. ### User's Default or Primary Item diff --git a/sync/streams/migration.mdx b/sync/streams/migration.mdx deleted file mode 100644 index 365cebf2b..000000000 --- a/sync/streams/migration.mdx +++ /dev/null @@ -1,252 +0,0 @@ ---- -title: "Migrating from Sync Rules" -description: "Migrate existing projects from legacy Sync Rules to Sync Streams." ---- - -import StreamDefinitionReference from '/snippets/stream-definition-reference.mdx'; - -## Why Migrate? - -PowerSync's original Sync Rules system was optimized for offline-first use cases where you want to "sync everything upfront" when the client connects, so data is available locally if the user goes offline. - -However, many developers are building apps where users are mostly online, and you don't want to make users wait to sync a lot of data upfront. This is especially true for **web apps**: users are mostly online, you often want to sync only the data needed for the current page, and users frequently have multiple browser tabs open — each needing different subsets of data. - -### The Problem with Client Parameters - -[Client Parameters](/sync/rules/client-parameters) in Sync Rules partially support on-demand syncing — for example, using a `project_ids` array to sync only specific projects. However, manually managing these arrays across different browser tabs becomes painful: - -- You need to aggregate IDs across all open tabs -- You need additional logic for different data types (tables) -- If you want to keep data around after a tab closes (caching), you need even more management - -### How Sync Streams Solve This - -Sync Streams address these limitations: - -1. **On-demand syncing**: Define streams once, then subscribe from your app one or more times with different parameters. No need to manage arrays of IDs — each subscription is independent. - -2. **Multi-tab support**: Each subscription manages its own lifecycle. Open the same list in two tabs? Each tab subscribes independently. Close one? The other keeps working. - -3. **Built-in caching**: Each subscription has a configurable `ttl` that keeps data cached after unsubscribing. When users return to a screen, data may already be available — no loading state needed. - -4. **Simpler, more powerful syntax**: Stream queries support JOINs, CTEs, subqueries, and multiple queries per stream, and the syntax is closer to plain SQL. For example, [parameter queries become inline subqueries](#data-with-subqueries-replaces-parameter-queries), so you write a single query instead of separate `parameters:` and `data:` blocks. - -5. **Framework integration**: [React hooks, Vue composables, TanStack Query, and Kotlin Compose extensions](/sync/streams/client-usage#framework-integrations) let your UI components automatically manage subscriptions based on what's rendered. - -### Still Need Offline-First? - -If you want "sync everything upfront" behavior (like Sync Rules), set [`auto_subscribe: true`](/sync/streams/overview#using-auto-subscribe) on your Sync Streams and clients will subscribe automatically when they connect. - -## Requirements - -- PowerSync Service v1.20.0+ (Cloud instances already meet this) -- Latest SDK versions with [Rust-based sync client](https://releases.powersync.com/announcements/improved-sync-performance-in-our-client-sdks) (enabled by default on latest SDKs) -- `config: edition: 3` in your Sync Config - - - -| SDK | Minimum Version | Rust Client Default | -|-----|-----------------|---------------------| -| JS Web | v1.27.0 | v1.32.0 | -| React Native | v1.25.0 | v1.29.0 | -| React hooks | v1.8.0 | — | -| Node.js | v0.11.0 | v0.16.0 | -| Capacitor | v0.0.1 | v0.3.0 | -| Tauri | v0.0.1 | Always (Rust client only) | -| Dart/Flutter | v1.16.0 | v1.17.0 | -| Kotlin | v1.7.0 | v1.9.0 | -| Swift | v1.11.0 | v1.8.0 | -| .NET | v0.0.8-alpha.1 | v0.0.5-alpha.1 | - - - -If you're on an SDK version below the "Rust Client Default" version, enable the Rust client manually: - -**JavaScript:** -```js -await db.connect(new MyConnector(), { - clientImplementation: SyncClientImplementation.RUST -}); -``` - -**Dart:** -```dart -database.connect( - connector: YourConnector(), - options: const SyncOptions( - syncImplementation: SyncClientImplementation.rust, - ), -); -``` - -**Kotlin:** -```kotlin -database.connect(MyConnector(), options = SyncOptions( - newClientImplementation = true, -)) -``` - -**Swift:** -```swift -import PowerSync - -try await db.connect(connector: connector, options: ConnectOptions( - newClientImplementation: true, -)) -``` - - - -## Migration Tool - -You can generate a Sync Streams draft from your existing Sync Rules in two ways: - -1. **Dashboard:** In the [PowerSync Dashboard](https://dashboard.powersync.com/), use the **Migrate to Sync Streams** button. It converts your Sync Rules into a Sync Streams draft that you can review before deploying. - -2. **CLI:** Run `powersync migrate sync-rules` to produce a Sync Streams draft from your current Sync Config. - -The output uses `auto_subscribe: true` by default, preserving your existing sync-everything-upfront behavior so no client-side changes are required when you first deploy. - -**Next steps:** Review the draft, then deploy it (via the Dashboard or `powersync deploy sync-config`). After that, you can optionally migrate individual streams to on-demand subscriptions over time — remove `auto_subscribe: true` from specific streams and update client code to use the `syncStream()` API where it makes sense for your app. - -## Stream Definition Reference - - - -## Migration Examples - -### Global Data (No Parameters) - -In Sync Rules, a ["global" bucket](/sync/rules/global-buckets) syncs the same data to all users. In Sync Streams, you achieve this with queries that have no parameters. Add [`auto_subscribe: true`](/sync/streams/overview#using-auto-subscribe) to maintain the Sync Rules behavior where data syncs automatically on connect. - -**Sync Rules:** -```yaml -bucket_definitions: - global: - data: - - SELECT * FROM todos - - SELECT * FROM lists WHERE archived = false -``` - -**Sync Streams:** -```yaml -config: - edition: 3 - -streams: - shared_data: - auto_subscribe: true # Sync automatically like Sync Rules - queries: - - SELECT * FROM todos - - SELECT * FROM lists WHERE archived = false -``` - - -Without `auto_subscribe: true`, clients would need to explicitly subscribe to these streams. This gives you flexibility to migrate incrementally or switch to on-demand syncing later. - - -### User-Scoped Data - -**Sync Rules:** -```yaml -bucket_definitions: - user_lists: - priority: 1 - parameters: SELECT request.user_id() as user_id - data: - - SELECT * FROM lists WHERE owner_id = bucket.user_id -``` - -**Sync Streams:** -```yaml -config: - edition: 3 - -streams: - user_lists: - auto_subscribe: true - priority: 1 - query: SELECT * FROM lists WHERE owner_id = auth.user_id() -``` - -### Data with Subqueries (Replaces Parameter Queries) - -**Sync Rules:** -```yaml -bucket_definitions: - owned_lists: - parameters: | - SELECT id as list_id FROM lists WHERE owner_id = request.user_id() - data: - - SELECT * FROM lists WHERE lists.id = bucket.list_id - - SELECT * FROM todos WHERE todos.list_id = bucket.list_id -``` - -**Sync Streams:** -```yaml -config: - edition: 3 - -streams: - owned_lists: - auto_subscribe: true - query: SELECT * FROM lists WHERE owner_id = auth.user_id() - list_todos: - query: | - SELECT * FROM todos - WHERE list_id = subscription.parameter('list_id') - AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) -``` - -### Client Parameters → Subscription Parameters - -**Sync Rules** used global [Client Parameters](/sync/rules/client-parameters): -```yaml -bucket_definitions: - posts: - parameters: SELECT (request.parameters() ->> 'current_page') as page_number - data: - - SELECT * FROM posts WHERE page_number = bucket.page_number -``` - -**Sync Streams** use Subscription Parameters, which are more flexible — you can subscribe multiple times with different values: -```yaml -config: - edition: 3 - -streams: - posts: - query: SELECT * FROM posts WHERE page_number = subscription.parameter('page_number') -``` - -```js -// Subscribe to multiple pages simultaneously -const page1 = await db.syncStream('posts', { page_number: 1 }).subscribe(); -const page2 = await db.syncStream('posts', { page_number: 2 }).subscribe(); -``` - -## Parameter Syntax Changes - -| Sync Rules | Sync Streams | -|------------|--------------| -| `request.user_id()` | `auth.user_id()` | -| `request.jwt() ->> 'claim'` | `auth.parameter('claim')` | -| `request.parameters() ->> 'key'` | `subscription.parameter('key')` ([subscription parameter](/sync/streams/parameters#subscription-parameters)) or `connection.parameter('key')` ([connection parameter](/sync/streams/parameters#connection-parameters)) | -| `bucket.param_name` | Use the parameter directly in the query e.g. `subscription.parameter('key')` | - -## Client-Side Changes - -After updating your Sync Config, update your client code to use subscriptions: - -```js -// Before (Sync Rules with Client Parameters) -await db.connect(connector, { - params: { current_project: projectId } -}); - -// After (Sync Streams with Subscriptions) -await db.connect(connector); -const sub = await db.syncStream('project_data', { project_id: projectId }).subscribe(); -``` - -See [Client-Side Usage](/sync/streams/client-usage) for detailed examples. diff --git a/sync/streams/overview.mdx b/sync/streams/overview.mdx index 3ec4013ca..5fbe3ba2a 100644 --- a/sync/streams/overview.mdx +++ b/sync/streams/overview.mdx @@ -1,287 +1,66 @@ --- title: "Sync Streams" -description: "Introduction to Sync Streams, the recommended way to define which data syncs to each client with SQL-based stream definitions." -sidebarTitle: "Quickstart" +description: "Sync Streams define which data syncs to each client. Learn what a stream is and how PowerSync replicates and streams the data." +sidebarTitle: "Overview" --- -import StreamDefinitionReference from '/snippets/stream-definition-reference.mdx'; +With Sync Streams, you write SQL-like queries to define streams of data, and your client app subscribes to the streams it needs. This enables _partial sync_: each client syncs only the relevant subset of data, instead of the entire database. PowerSync keeps subscribed data synced in real-time to a client-side SQLite database, where it stays available when the device is offline. -With Sync Streams, you write simple SQL-like queries to define streams of data, and your client app subscribes to the streams it needs. This enables _partial sync_: each client syncs only the relevant subset of data, instead of the entire database. PowerSync handles the rest, keeping subscribed data synced in real-time to a client-side SQLite database, where it stays available even when the device is offline. - -For example, you might define a stream that syncs only the current user's to-do items, another for shared projects they have access to, and another for reference data that everyone needs. Your app subscribes to these streams on demand, and only that data syncs to the device. Offline-first apps that need all relevant data available upfront can use `auto_subscribe: true` so streams sync automatically when clients connect. +For example, you might define a stream that syncs only the current user's to-do items, another for shared projects they have access to, and another for reference data that everyone needs. Your app subscribes to these streams on demand, and only that data syncs to the device. Apps that need all relevant data available upfront can set `auto_subscribe: true` so streams sync automatically when clients connect. **Are you still using Sync Rules?** Sync Streams support everything Sync Rules do, plus more expressive queries (including JOIN support), on-demand syncing, and a simpler developer experience (e.g. React hooks that manage subscriptions automatically). -You can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See the [migration guide](/sync/streams/migration) for details. +You can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See [Migrate to Sync Streams](/sync/rules/migrate-to-sync-streams) for details. -## Defining Streams - -Streams are defined in a YAML configuration file. Each stream has a **name** and a **query** that specifies which rows to sync using SQL-like syntax. The query can reference [parameters](/sync/overview#how-it-works) like the authenticated user's ID to personalize what each user receives. - - - -In the [PowerSync Dashboard](https://dashboard.powersync.com/): - -1. Select your project and instance -2. Go to **Sync Streams** -3. Edit the YAML directly in the dashboard -4. Click **Deploy** to validate and deploy - -```yaml -config: - edition: 3 - -streams: - todos: - query: SELECT * FROM todos WHERE owner_id = auth.user_id() -``` - - - -Add a `sync_config` section to your `service.yaml`. Using a **separate file** is recommended (e.g. `sync_config: path: sync-config.yaml`). Put the stream definition in that file: - -```yaml sync-config.yaml -config: - edition: 3 +## How It Works -streams: - todos: - query: SELECT * FROM todos WHERE owner_id = auth.user_id() -``` +Each PowerSync Service instance has a deployed Sync Streams configuration: a YAML file that defines the streams that exist. Each stream has a name and a SQL-like query that selects the tables and columns to sync, filters rows by static conditions or by parameters, and can rename or transform columns. The Service uses this configuration in two places: when it replicates data from your source database into buckets, and when it streams those buckets to clients. -You can also use inline `sync_config: content: |` with the YAML nested in your main config. See [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances#sync_config) for both options. - - +See the [PowerSync Service architecture](/architecture/powersync-service) for more background. -Available stream options: +### Buckets and Parameters - +PowerSync groups replicated data into [buckets](/architecture/powersync-service#bucket-system): partitions of data that are synced as a unit. A stream creates one bucket for each unique value of its filter, such as each user ID matched by `auth.user_id()` or each `list_id` that a client subscribes with. A stream without parameters creates a single bucket that syncs the same data to every subscriber. -## Basic Examples +Buckets are implicit in Sync Streams. The Service creates them from your stream queries, parameters, and subqueries, and you do not define or name them yourself. See [Bucket Count](/sync/streams/bucket-count) for how queries determine the number of buckets. -There are two independent concepts to understand: +### Replication From the Source Database -- _What_ data the stream returns. For example: - - *Global data*: No parameters. Same data for all users (e.g. reference tables like categories). - - *Filtered data*: Filters the data by a parameter value. This can make use of _auth parameters_ from the JWT token (such as the user ID or other JWT claims), _subscription parameters_ (specified by the client when it subscribes to a stream at any time), or _connection parameters_ (specified at connection). Different users will get different sets of data based on the parameters. See [Using Parameters](/sync/streams/parameters) for the full reference. -- _When_ the client syncs the data - - *Auto-subscribe*: Client automatically subscribes on connect (`auto_subscribe: true`) - - *On-demand*: Client explicitly subscribes when needed (default behavior) - -### Global Data - -Data without parameters is "global" data, meaning the same data goes to all users/clients. This is useful for reference tables: - -```yaml -config: - edition: 3 - -streams: - # Same categories for everyone - categories: - query: SELECT * FROM categories - - # Same active products for everyone - products: - query: SELECT * FROM products WHERE active = true -``` +The Service [replicates and transforms](/architecture/powersync-service#replication-from-the-source-database) data from your source database according to your stream queries, and persists the data and metadata in buckets. Buckets are updated incrementally, so they contain the latest state as well as a history of changes (operations). This operation history allows clients to sync only the deltas they need to get up to date. See [Protocol](/architecture/powersync-protocol#protocol) for details. -Global data streams still require clients to subscribe explicitly unless you set `auto_subscribe: true` +For example, a stream `user_lists` with the query `SELECT * FROM lists WHERE owner_id = auth.user_id()` creates one bucket per user. If users `A` and `B` exist in the source database, the Service creates a bucket for each of them. When user `A` connects and subscribes, they sync only their own bucket. -### Filtering Data by User - -Use `auth.user_id()` or other [JWT claims](/sync/streams/parameters#auth-parameters) to return different data per user: - -```yaml -config: - edition: 3 - -streams: - # Each user gets their own lists - my_lists: - query: SELECT * FROM lists WHERE owner_id = auth.user_id() - - # Each user gets their own orders - my_orders: - query: SELECT * FROM orders WHERE user_id = auth.user_id() -``` - -### Filtering Data Based on Subscription Parameters - -Use `subscription.parameter()` for data that clients subscribe to explicitly: - -```yaml -config: - edition: 3 - -streams: - # Sync todos for a specific list when the client subscribes with a list_id - list_todos: - query: | - SELECT * FROM todos - WHERE list_id = subscription.parameter('list_id') - AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) -``` - -```js -// Client subscribes with the list they want to view -const sub = await db.syncStream('list_todos', { list_id: 'abc123' }).subscribe(); -``` - -### Using Auto-Subscribe - -Set `auto_subscribe: true` to sync data automatically when clients connect. This is useful for: -- Reference data that all users need, or that are needed in many screens in the app. -- User data that should always be available offline -- Maintaining [Sync Rules](/sync/rules/overview) default behavior ("sync everything upfront") when migrating to Sync Streams - -```yaml -config: - edition: 3 - -streams: - # Global data, synced automatically - categories: - auto_subscribe: true - query: SELECT * FROM categories - - # User-scoped data, synced automatically - my_orders: - auto_subscribe: true - query: SELECT * FROM orders WHERE user_id = auth.user_id() - - # Parameterized data, subscribed on-demand (no auto_subscribe) - order_items: - query: | - SELECT * FROM order_items - WHERE order_id = subscription.parameter('order_id') - AND order_id IN (SELECT id FROM orders WHERE user_id = auth.user_id()) -``` - - -## Client-Side Usage - -Subscribe to streams from your client app: - - - -```js -const sub = await db.syncStream('list_todos', { list_id: 'abc123' }) - .subscribe({ ttl: 3600 }); - -// Wait for this subscription to have synced -await sub.waitForFirstSync(); - -// When the component needing the subscription is no longer active... -sub.unsubscribe(); -``` - -**React hooks:** - -```jsx -const stream = useSyncStream({ name: 'list_todos', parameters: { list_id: 'abc123' } }); -// Check download progress or subscription information -stream?.progress; -stream?.subscription.hasSynced; -``` - -The `useQuery` hook can wait for Sync Streams before running queries: - -```jsx -const { data } = useQuery( - 'SELECT * FROM todos WHERE list_id = ?', - [listId], - { streams: [{ name: 'list_todos', parameters: { list_id: listId }, waitForStream: true }] } -); -``` - - - -```dart -final sub = await db - .syncStream('list_todos', {'list_id': 'abc123'}) - .subscribe(ttl: const Duration(hours: 1)); - -// Wait for this subscription to have synced -await sub.waitForFirstSync(); - -// When the component needing the subscription is no longer active... -sub.unsubscribe(); -``` - - - -```kotlin -val sub = database.syncStream("list_todos", mapOf("list_id" to JsonParam.String("abc123"))) - .subscribe(ttl = 1.0.hours) - -// Wait for this subscription to have synced -sub.waitForFirstSync() - -// When the component needing the subscription is no longer active... -sub.unsubscribe() -``` - - - - -```swift -let sub = try await db.syncStream(name: "list_todos", params: ["list_id": JsonValue.string("abc123")]) - .subscribe(ttl: 60 * 60, priority: nil) // 1 hour - -// Wait for this subscription to have synced -try await sub.waitForFirstSync() - -// When the component needing the subscription is no longer active... -try await sub.unsubscribe() -``` - - - -```csharp -var sub = await db.SyncStream("list_todos", new() { ["list_id"] = "abc123" }) - .Subscribe(new SyncStreamSubscribeOptions { Ttl = TimeSpan.FromHours(1) }); - -// Wait for this subscription to have synced -await sub.WaitForFirstSync(); - -// When the component needing the subscription is no longer active... -sub.Unsubscribe(); -``` - - - -### TTL (Time-To-Live) - -Each subscription has a `ttl` that keeps data cached after unsubscribing. This enables warm cache behavior — when users return to a screen and you re-subscribe to relevant streams, data is already available on the client. Default TTL is 24 hours. See [Client-Side Usage](/sync/streams/client-usage) for details. - -```js -// Set TTL in seconds when subscribing -const sub = await db.syncStream('todos', { list_id: 'abc' }) - .subscribe({ ttl: 3600 }); // Cache for 1 hour after unsubscribe -``` -## Developer Notes - -- **SQL Syntax**: Stream queries use a SQL-like syntax with `SELECT` statements. You can use subqueries, `INNER JOIN`, and [CTEs](/sync/streams/ctes) for filtering. `GROUP BY`, `ORDER BY`, and `LIMIT` are not supported. See [Writing Queries](/sync/streams/queries) for details on joins, multiple queries per stream, and other features. - -- **Type Conversion**: Data types from your source database (Postgres, MongoDB, MySQL, SQL Server or Convex) are converted when synced to the client's SQLite database. SQLite has a limited type system, so most types become `text` and you may need to parse or cast values in your app code. See [Type Mapping](/sync/types) for details on how each type is handled. - -- **Primary Key**: PowerSync requires every synced table to have a primary key column named `id` of type `text`. If your backend uses a different column name or type, you'll need to map it. For MongoDB, collections use `_id` as the ID field; you must alias it in your stream queries (e.g. `SELECT *, _id as id FROM your_collection`). - -- **Case Sensitivity**: To avoid issues across different databases and platforms, use **lowercase identifiers** for all table and column names in your Sync Streams. If your backend uses mixed case, see [Case Sensitivity](/sync/advanced/case-sensitivity) for how to handle it. + + + -- **Bucket Limits**: PowerSync uses internal partitions called [buckets](/architecture/powersync-service#bucket-system) to efficiently sync data. Each user has a limit on how many buckets they can sync ([1,000 by default](/resources/performance-and-limits)), and the query pattern determines how many each stream creates. See [Bucket Count](/sync/streams/bucket-count) for how buckets are counted, and [Reducing Bucket Count](/sync/advanced/reducing-bucket-count) to resolve `PSYNC_S2305` errors. +### Streaming Sync to Clients -- **Troubleshooting**: If data isn't syncing as expected, the [Sync Diagnostics Client](/tools/diagnostics-client) helps you inspect what's happening for a specific user — you can see which buckets the user has and what data is being synced. +Whenever buckets change (buckets are added or removed, or operations are added to existing buckets), the Service [streams these changes in real-time](/architecture/powersync-service#streaming-sync) to the subscribed clients. The set of buckets a client receives adjusts as it subscribes to and unsubscribes from streams, and depends on its subscription, connection, and authentication parameters. -## Examples & Demos +On the client, bucket data is persisted in SQLite, where you query it through your [client-side schema](/intro/setup-guide#define-your-client-side-schema). See [Client Architecture](/architecture/client-architecture#client-side-schema-and-sqlite-database-structure) for the database structure. -See [Examples & Demos](/sync/streams/examples) for working demo apps and complete application patterns. + + + -## Migrating from Legacy Sync Rules +## Next Steps -If you have an existing project using legacy Sync Rules, see the [Migration Guide](/sync/streams/migration) for step-by-step instructions, syntax changes, and examples. + + + Define your first streams and subscribe to them from your app. + + + Filter data with auth, subscription, and connection parameters. + + + Query syntax, joins, subqueries, and multiple queries per stream. + + + Manage subscriptions in each SDK and UI framework. + + diff --git a/sync/advanced/prioritized-sync.mdx b/sync/streams/prioritized-sync.mdx similarity index 95% rename from sync/advanced/prioritized-sync.mdx rename to sync/streams/prioritized-sync.mdx index bc5a1340e..064c1dd56 100644 --- a/sync/advanced/prioritized-sync.mdx +++ b/sync/streams/prioritized-sync.mdx @@ -9,19 +9,6 @@ PowerSync supports defining sync priorities, which allows you to control the syn In Sync Streams, priorities are assigned to streams and PowerSync manages the underlying buckets internally. (In legacy Sync Rules, priorities were assigned to buckets explicitly.) - -**Availability** - -This feature was introduced in version **1.7.1** of the PowerSync Service, and in the following SDK versions: -- [Flutter v1.12.0](/client-sdks/reference/flutter) -- [React Native v1.18.1](/client-sdks/reference/react-native-and-expo) -- [JavaScript Web v1.14.2](/client-sdks/reference/javascript-web) -- [Kotlin v1.0.0-BETA26](/client-sdks/reference/kotlin) -- [Swift v1.0.0-Beta.8](/client-sdks/reference/swift) -- [.NET v0.0.6-alpha.1](/client-sdks/reference/dotnet) - - - ## Why Use Sync Priorities? PowerSync's standard sync protocol ensures that: diff --git a/sync/streams/quickstart.mdx b/sync/streams/quickstart.mdx new file mode 100644 index 000000000..c8ae8cd68 --- /dev/null +++ b/sync/streams/quickstart.mdx @@ -0,0 +1,276 @@ +--- +title: "Sync Streams Quickstart" +description: "Define your first Sync Streams and subscribe to them from your client app." +sidebarTitle: "Quickstart" +--- + +import StreamDefinitionReference from '/snippets/stream-definition-reference.mdx'; + +This page shows how to define streams and subscribe to them from your app. For what streams are and how PowerSync replicates and syncs them, see the [Sync Streams overview](/sync/streams/overview). + +## Defining Streams + +Streams are defined in a YAML configuration file. Each stream has a **name** and a **query** that specifies which rows to sync using SQL-like syntax. The query can reference [parameters](/sync/streams/parameters) like the authenticated user's ID to personalize what each user receives. + + + +In the [PowerSync Dashboard](https://dashboard.powersync.com/): + +1. Select your project and instance +2. Go to **Sync Streams** +3. Edit the YAML directly in the dashboard +4. Click **Deploy** to validate and deploy + +```yaml +config: + edition: 3 + +streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() +``` + + + +Add a `sync_config` section to your `service.yaml`. Using a **separate file** is recommended (e.g. `sync_config: path: sync-config.yaml`). Put the stream definition in that file: + +```yaml sync-config.yaml +config: + edition: 3 + +streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() +``` + +You can also use inline `sync_config: content: |` with the YAML nested in your main config. See [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances#sync_config) for both options. + + + +Available stream options: + + + +## Basic Examples + +There are two independent concepts to understand: + +- _What_ data the stream returns. For example: + - *Global data*: No parameters. Same data for all users (e.g. reference tables like categories). + - *Filtered data*: Filters the data by a parameter value. This can make use of _auth parameters_ from the JWT token (such as the user ID or other JWT claims), _subscription parameters_ (specified by the client when it subscribes to a stream at any time), or _connection parameters_ (specified at connection). Different users will get different sets of data based on the parameters. See [Using Parameters](/sync/streams/parameters) for the full reference. +- _When_ the client syncs the data + - *Auto-subscribe*: Client automatically subscribes on connect (`auto_subscribe: true`) + - *On-demand*: Client explicitly subscribes when needed (default behavior) + +### Global Data + +Data without parameters is "global" data, meaning the same data goes to all users/clients. This is useful for reference tables: + +```yaml +config: + edition: 3 + +streams: + # Same categories for everyone + categories: + query: SELECT * FROM categories + + # Same active products for everyone + products: + query: SELECT * FROM products WHERE active = true +``` + + +Global data streams still require clients to subscribe explicitly unless you set `auto_subscribe: true` + + +### Filtering Data by User + +Use `auth.user_id()` or other [JWT claims](/sync/streams/parameters#auth-parameters) to return different data per user: + +```yaml +config: + edition: 3 + +streams: + # Each user gets their own lists + my_lists: + query: SELECT * FROM lists WHERE owner_id = auth.user_id() + + # Each user gets their own orders + my_orders: + query: SELECT * FROM orders WHERE user_id = auth.user_id() +``` + +### Filtering Data Based on Subscription Parameters + +Use `subscription.parameter()` for data that clients subscribe to explicitly: + +```yaml +config: + edition: 3 + +streams: + # Sync todos for a specific list when the client subscribes with a list_id + list_todos: + query: | + SELECT * FROM todos + WHERE list_id = subscription.parameter('list_id') + AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) +``` + +```js +// Client subscribes with the list they want to view +const sub = await db.syncStream('list_todos', { list_id: 'abc123' }).subscribe(); +``` + +### Using Auto-Subscribe + +Set `auto_subscribe: true` to sync data automatically when clients connect. This is useful for: +- Reference data that all users need, or that is needed in many screens in the app. +- User data that should always be available offline. +- Keeping the "sync everything upfront" behavior of legacy [Sync Rules](/sync/rules/overview) when migrating to Sync Streams. + +```yaml +config: + edition: 3 + +streams: + # Global data, synced automatically + categories: + auto_subscribe: true + query: SELECT * FROM categories + + # User-scoped data, synced automatically + my_orders: + auto_subscribe: true + query: SELECT * FROM orders WHERE user_id = auth.user_id() + + # Parameterized data, subscribed on-demand (no auto_subscribe) + order_items: + query: | + SELECT * FROM order_items + WHERE order_id = subscription.parameter('order_id') + AND order_id IN (SELECT id FROM orders WHERE user_id = auth.user_id()) +``` + + +## Client-Side Usage + +Subscribe to streams from your client app: + + + +```js +const sub = await db.syncStream('list_todos', { list_id: 'abc123' }) + .subscribe({ ttl: 3600 }); + +// Wait for this subscription to have synced +await sub.waitForFirstSync(); + +// When the component needing the subscription is no longer active... +sub.unsubscribe(); +``` + +**React hooks:** + +```jsx +const stream = useSyncStream({ name: 'list_todos', parameters: { list_id: 'abc123' } }); +// Check download progress or subscription information +stream?.progress; +stream?.subscription.hasSynced; +``` + +The `useQuery` hook can wait for Sync Streams before running queries: + +```jsx +const { data } = useQuery( + 'SELECT * FROM todos WHERE list_id = ?', + [listId], + { streams: [{ name: 'list_todos', parameters: { list_id: listId }, waitForStream: true }] } +); +``` + + + +```dart +final sub = await db + .syncStream('list_todos', {'list_id': 'abc123'}) + .subscribe(ttl: const Duration(hours: 1)); + +// Wait for this subscription to have synced +await sub.waitForFirstSync(); + +// When the component needing the subscription is no longer active... +sub.unsubscribe(); +``` + + + +```kotlin +val sub = database.syncStream("list_todos", mapOf("list_id" to JsonParam.String("abc123"))) + .subscribe(ttl = 1.0.hours) + +// Wait for this subscription to have synced +sub.waitForFirstSync() + +// When the component needing the subscription is no longer active... +sub.unsubscribe() +``` + + + + +```swift +let sub = try await db.syncStream(name: "list_todos", params: ["list_id": JsonValue.string("abc123")]) + .subscribe(ttl: 60 * 60, priority: nil) // 1 hour + +// Wait for this subscription to have synced +try await sub.waitForFirstSync() + +// When the component needing the subscription is no longer active... +try await sub.unsubscribe() +``` + + + +```csharp +var sub = await db.SyncStream("list_todos", new() { ["list_id"] = "abc123" }) + .Subscribe(new SyncStreamSubscribeOptions { Ttl = TimeSpan.FromHours(1) }); + +// Wait for this subscription to have synced +await sub.WaitForFirstSync(); + +// When the component needing the subscription is no longer active... +sub.Unsubscribe(); +``` + + + +### TTL (Time-To-Live) + +Each subscription has a `ttl` that keeps data cached after unsubscribing. This enables warm cache behavior: when users return to a screen and you re-subscribe to relevant streams, data is already available on the client. Default TTL is 24 hours. See [Client-Side Usage](/sync/streams/client-usage) for details. + +```js +// Set TTL in seconds when subscribing +const sub = await db.syncStream('todos', { list_id: 'abc' }) + .subscribe({ ttl: 3600 }); // Cache for 1 hour after unsubscribe +``` + +## Developer Notes + +- **SQL Syntax**: Stream queries use a SQL-like syntax with `SELECT` statements. You can select specific columns, filter on static conditions, rename or transform columns, and use subqueries, `INNER JOIN`, and [CTEs](/sync/streams/ctes) for filtering. `GROUP BY`, `ORDER BY`, and `LIMIT` are not supported. See [Writing Queries](/sync/streams/queries) for details and [Supported SQL](/sync/supported-sql) for the operators and functions you can use. + +- **Type Conversion**: Data types from your source database (Postgres, MongoDB, MySQL, SQL Server or Convex) are converted when synced to the client's SQLite database. SQLite has a limited type system, so most types become `text` and you may need to parse or cast values in your app code. See [Type Mapping](/sync/types) for details on how each type is handled. + +- **Primary Key**: PowerSync requires every synced table to have a primary key column named `id` of type `text`. If your backend uses a different column name or type, you'll need to map it. For MongoDB, collections use `_id` as the ID field; you must alias it in your stream queries (e.g. `SELECT *, _id as id FROM your_collection`). + +- **Case Sensitivity**: To avoid issues across different databases and platforms, use **lowercase identifiers** for all table and column names in your Sync Streams. If your backend uses mixed case, see [Case Sensitivity](/sync/advanced/case-sensitivity) for how to handle it. + +- **Bucket Limits**: Each user has a limit on how many buckets they can sync ([1,000 by default](/resources/performance-and-limits)), and the query pattern determines how many each stream creates. See [Bucket Count](/sync/streams/bucket-count) for how buckets are counted, and [Reducing Bucket Count](/sync/advanced/reducing-bucket-count) to resolve `PSYNC_S2305` errors. + +- **Troubleshooting**: If data isn't syncing as expected, the [Sync Diagnostics Client](/tools/diagnostics-client) helps you inspect what's happening for a specific user. You can see which buckets the user has and what data is being synced. + +## Examples & Demos + +See [Examples & Demos](/sync/streams/examples) for working demo apps and complete application patterns. diff --git a/sync/supported-sql.mdx b/sync/supported-sql.mdx index a99508ebe..9287c428c 100644 --- a/sync/supported-sql.mdx +++ b/sync/supported-sql.mdx @@ -1,7 +1,6 @@ --- title: "Supported SQL" description: "Reference for SQL syntax, operators, built-in functions, and type casting supported in Sync Streams/Sync Rules queries." -sidebarTitle: "Guide" --- This guide explains the SQL supported in [Sync Streams](/sync/streams/overview) and [Sync Rules (legacy)](/sync/rules/overview): what you can write, with examples and restrictions.