Managed raw tables

[simolus3]

This adds a new type of table next to the existing ones based on JSON data + views (easy to manage and migrate, slower to query) and raw tables (fast queries and full customizability at the expense of a higher setup complexity). Direct tables (name tbd) have the shape of raw tables with proper SQLite columns that can be queried without decoding JSON on reads.

A direct table has a shape like this:

CREATE TABLE users (
  id TEXT NOT NULL PRIMARY KEY,
  -- Rest of this depends on schema set on PowerSync database
  name ANY,
  email ANY,
  points ANY
) STRICT;

The sync client treats this table like raw tables, automatically inferring statements to sync into it. There is no view to generate, but we generate triggers to record writes in ps_crud.

As an alternative to this table shape, I have considered keeping a ps_data__-like table that just adds GENERATED ALWAYS AS columns. This might seem simpler, but then you wouldn't be able to update these columns directly so we'd have to keep generating views and INSTEAD OF triggers. The current pattern is closer to the shape one would expect SQLite tables to have.

Direct tables are currently an opt-in, but we could make them the default in a future major release. An important difference in behavior is that JSON-based tables have INSTEAD OF triggers for tables that CAST values to their expected type. Direct tables don't have triggers, so inserting into them gives you the usual SQLite behavior where epxressions have a type affinity, but no enforced type. It is thus possible to sync, insert and update "wrong" types into direct tables. I have adopted STRICT tables with ANY columns for this reason: The type is dynamic either way, and this simplifies migrations because we don't have to care about changed column types.

Benchmarks

Despite the fact that we have to parse rows nows, enabling direct tables seems to improve sync_local_performance_test.dart. Using a full run with unique ids:

• Before: 1940ms Reads: 1291079 + 0 | Writes: 69031 + 0.
• After: 1408ms (-27.4%) Reads: 1286404 (-3.6%) + 0 | Writes: 42228 + 0 (-38.8%). This is an in-memory test with an instrumented VFS so times don't mean much, but this needing fewer writes should translate to better sync performance.

Since direct tables have the same structure as raw tables, they should result in similar performance benefits for queries (I did not benchmark direct tables for this specifically).

Migrations and tests

Raw tables already give us the infrastructure to sync into arbitrary tables, so most of this PR deals with migrations related to direct tables. The following things are considered and tested:

• Creating and removing direct tables, moving existing rows from or into ps_untyped.
• Support migrating from json-based to direct tables (and the other way around).
  • There are four combinations each, from the fact that the same migration might switch from local-only to synced tables too.
• Support local-only direct tables, and migrate between them the same way we migrate json-based tables.
• Handle schema changes on direct tables, like adding or removing columns as well as changing their type.
• Index creation and removal.

[rkistner]

I think this feature will be great, but I wonder if we could remove the _data / _rest column completely to reduce the overhead - same reasoning as this comment? #189 (comment)

Or would that make the migration paths with JSON-based tables too difficult? In that case, maybe we can consider another change later (potentially breaking or opt-in) to remove ps_untyped and _rest usage?

[simolus3]

I like that idea. Perhaps there could be a single flag (on the entire schema, not per-table) that does both.

1. When enabled for the first time, we create direct tables (copying rows from old ps_data__ tables and ps_untyped). We then delete old tables, clear ps_untyped and stop using it.
   1. If there is synced data for a table not covered by the schema, the client could emit a log. Otherwise things like typos between client/server state are very tricky to debug (we already had reports where people add aliases to their sync streams and then complain about data not syncing when it's in ps_untyped).
2. If a developer wants to disable this flag later, we can reconstruct ps_data__ and ps_untyped by forming a json structure of existing columns (read via pragma_table_info). That can miss columns, so a resync trigger is still necessary.