Repository navigation
New features and fixes for 2.6.0 - #135
Merged
Merged
Conversation
… relation (F7) 2.5.0 replaced the joins written as clauses of SELECT, such as table SELECT INNER_JOIN<R>(other) USING column, with joins of relations, which check the type their rows are read into against the joined relations, can join any number of relations, each with its own condition, and select result columns. It announced that the old ones would be removed in the next version, which this commit does: the functions CROSS_JOIN, JOIN, INNER_JOIN, NATURAL_JOIN, NATURAL_INNER_JOIN, LEFT_OUTER_JOIN and NATURAL_LEFT_OUTER_JOIN that took relations as arguments, the SELECT and SELECT_DISTINCT that took what they returned, and the types BaseJoinClause, JoinClause, NaturalJoinClause, JoinStatementWithoutCondition and JoinSelectStatement, with the ON, USING, WHERE, GROUP_BY, ORDER_BY and LIMIT that took them. It is a breaking change: such a join becomes (table INNER_JOIN other USING column) SELECT X<R>(). With them gone, a join can start with its first relation, as in PersonTable INNER_JOIN BookTable ON (...), without FROM: every join operator is now an infix function of a relation too, which starts the join FROM(relation) would. It couldn't be before, as such a function took the place of the old CROSS_JOIN(CategoryTable) in the lambda of a table, whose implicit receiver is a relation. FROM still starts a join. The test of the old joins now covers the joins that start from a relation, of every kind each platform has; the other tests and the sample use them too. The sample only keeps the joins that its Student type fits, as the new ones check it. The guide to advanced queries starts joins from their first relation, and describes how to migrate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2.5.0 deprecated the UNION {} and UNION_ALL {} blocks, and the
beginUnion, createUnionSelectStatement and endUnion functions they
used, for bugs that wouldn't be fixed: a block read its rows into its
first SELECT's result type while returning the table's row type, so a
union of projections, result columns or joins failed with a
ClassCastException when its results were used, and a nested block was
flattened into the SQL, which SQLite evaluates from left to right, so
that it changed its meaning. The infix UNION, UNION_ALL, INTERSECT and
EXCEPT replaced them. This removes them, with UnionSelectStatementGroup,
the container they collected their SELECTs in, so that a SELECT is now
always added to the scope, transaction or trigger body it is written
in. It is a breaking change: UNION { a; b } becomes (a) UNION (b).
The test of the block is removed, as the infix operators have their
own, and the guide to advanced queries describes the migration in place
of the deprecated block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An observed query gives results, while a reader that reads on its own
and only has to learn when the tables it read change, such as a
PagingSource that invalidates itself, needs a signal. And a version
read apart from the read can't be sound, as a table only has one once
it is tracked, which the read itself starts.
Database#versioned { } runs a scope of SELECTs, and returns its result
with a TableVersions token of the versions of the tables they read.
Like observe, it tracks the tables, captures their versions and runs
the SELECTs under the executive lock, in that order, so that no change
after the read is missed. A scope that writes is rejected before any of
its statements runs, as its token couldn't tell its own changes from
others'. Tokens combine with +, which takes the earlier version of a
table that both read, so that the combined token is stale when either
is, as for a query built from the results of another; tokens of
another Database are rejected.
awaitChange suspends until a table of the token changes. onChange
registers a one-shot callback, and registers before it checks: when
the token is already stale, it removes the registration and returns
null, so that the caller acts on the change itself, as a PagingSource
returns LoadResult.Invalid(). The registrations are a list replaced
atomically, so that a registration is either fired by a writer or
removed by onChange, never both. Callbacks run on the thread that
wrote, after the writing scope, blocking or suspended, has released the
lock, so that they may use the database.
The tracker's weak registrations for F13 are left to F13.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SQLlin had INSERT OR IGNORE and INSERT OR REPLACE. This adds the other
resolutions a statement can give: INSERT_OR_ABORT, INSERT_OR_FAIL and
INSERT_OR_ROLLBACK, which take objects, expressions, DEFAULT_VALUES and
a SELECT, as INSERT does, and UPDATE_OR_ABORT, UPDATE_OR_FAIL,
UPDATE_OR_IGNORE, UPDATE_OR_REPLACE and UPDATE_OR_ROLLBACK, written as
UPDATE. A statement's resolution takes the place of the conflict clause
of a constraint, which onConflict declares since F17: INSERT_OR_ABORT
fails where a UNIQUE ON CONFLICT IGNORE column would have skipped the
row, which is the use of a resolution that is otherwise the default.
Like INSERT_OR_IGNORE and INSERT_OR_REPLACE, the new INSERTs write the
primary key even when the database assigns it, as a conflict on it
couldn't be seen otherwise, so the three now share one builder,
Insert.insertOr, and Update.update takes the keyword it starts with.
The guide to modifying the database describes the resolutions in a new
section, including that in transaction { }, where a failing statement
rolls the transaction back, ABORT, FAIL and ROLLBACK only differ outside
one. The test checks that each failure is the constraint's, rather than
one of the SQL, and runs every form once successfully.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…VE (F27)
Since B5, every transaction begins IMMEDIATE, taking the write lock at
once. This lets a transaction begin in another TransactionMode:
EXCLUSIVE, which also keeps other connections from reading in DELETE
journal mode, and is IMMEDIATE in WAL mode; and DEFERRED, which takes
no lock until the transaction reads, nor the write lock until it
writes, but makes a transaction that reads before it writes fail at
once while another connection writes, without waiting for the busy
timeout.
In sqllin-driver, DatabaseConnection#beginTransaction(mode) begins one,
and withTransaction takes a mode. The new beginTransaction has a default
implementation, which only begins an IMMEDIATE transaction, so that a
DatabaseConnection implemented outside SQLlin still compiles; the JVM
and Kotlin/Native drivers run BEGIN with the mode, or a deferred BEGIN
on a read-only connection, as they did. Android's framework can't begin
a deferred transaction that writes, as it turns a BEGIN statement into
an EXCLUSIVE transaction of its own, so there DEFERRED begins an
IMMEDIATE one, which only takes the write lock earlier, while EXCLUSIVE
is its beginTransaction.
In the DSL, transaction(mode) { } begins one; a transaction in another
one joins it and keeps its mode. The guide to modifying the database
describes the modes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
INDEXED_BY makes a statement look its rows up through an index, and NOT_INDEXED through none, rather than through the one SQLite's query planner chooses: (PersonTable INDEXED_BY "idx_person_name") SELECT WHERE(...) and PersonTable.NOT_INDEXED DELETE WHERE(...). They return an IndexedTable, a Relation whose FROM is the table with the clause, and whose name and columns are the table's, so that a SELECT reads it as it reads the table, joins included, and an observed query or a versioned read tracks the table. It isn't a Table, so INSERT, which SQL doesn't allow it in, doesn't take it, which is checked at compile time; UPDATE, the UPDATE_OR_ ones and DELETE take it through overloads, which give the table with the clause to their builders as the target they write. The test shows the clauses are in the SQL of each statement by giving an index that doesn't exist, which fails with "no such index". The guide to modifying the database describes them with indexes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to 3.54.1.0 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sqllin-processor only gave its Java classes a JVM target, 21, through the java block, which left its Kotlin classes, all of its code, to the default: the JDK that runs the build. Built with JDK 25, they were Java 25 class files, major version 69, which a KSP run on an older JDK can't load, failing with UnsupportedClassVersionError; the Kotlin Gradle plugin warned of the inconsistent targets. CI, on JDK 21, built them for 21. The processor now uses the JVM toolchain of the other modules, JDK 21, which compiles its Java and Kotlin classes for it, whichever JDK runs the build: the jar's classes are major version 65. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EXPLAIN_QUERY_PLAN(select) tells how SQLite would run a SELECT without running it: which tables it scans, which indexes it searches, and where it sorts with a temporary B-tree. Any SELECT can be planned, joins, subqueries and compound ones included. It is taken out of the statements the scope runs, as a subquery is, and planned with its parameters, as it would run. The QueryPlanStatement it returns is a statement of its own, rather than a SELECT, so that it can't become a subquery or part of a compound, which SQL doesn't allow, and isn't counted among the tables a versioned read covers; it changes nothing. It reads the plan when it runs and closes its cursor, as the plan is small. getResults() returns a QueryPlan: its steps, each with its detail and its children, the details of all of them, and a toString() that writes them as a tree, as the sqlite3 shell does. The plan is read by the columns SQLite returns, rather than by its version, which no platform tells reliably: since SQLite 3.24.0 they are id, parent, notused and detail, and a step's parent is the id of the step it belongs to; before, as on Android below API 30, they tell no such relation, so the steps are listed without children. The details are SQLite's own text, which differs between versions, so they are left as they are, and the guide to advanced queries says to check a plan by the names of tables and indexes and by phrases such as USING INDEX, rather than by whole details. The helper that takes a SELECT out of the scope's statements is renamed takeOut(), as it no longer serves subqueries only, and gets back its documentation, which a later section had separated from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A clause appended to a statement, as ORDER_BY, LIMIT, OFFSET, GROUP_BY, HAVING, WHERE after result columns, and UPDATE's WHERE, replaced the last statement of its container, whatever it was, as long as it was a SELECT or an UPDATE without WHERE. When another statement had been built since, as when a statement is held in a variable and completed later, that statement was dropped and never ran, while the incomplete one still did: an UPDATE held as `val update = table UPDATE SET(...)`, followed by a SELECT and then completed with `update WHERE (...)`, updated every row of the table. UPDATE's WHERE also added its parameters to the list of the UPDATE without it, which then failed with too many parameters when its SET had some. The clauses now replace the statement they complete through replaceStatement, which finds it wherever it is, as CREATE INDEX's WHERE already did, and changeLastStatement is removed. A clause appended to a statement that a clause was already appended to, or that became part of another one, as a subquery, a compound SELECT or EXPLAIN QUERY PLAN, fails with an IllegalStateException that says so, while the scope is built, so that none of its statements runs, rather than replacing another statement. UPDATE's WHERE merges the parameters into a new list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A transaction in another one, which the documentation of DatabaseScope
says can be nested, ended the outer one when it ended, since 2.0.0 at
least: transaction { } called endTransaction() in its finally block
whether its beginTransaction() had begun a transaction or joined one,
and endTransaction() cleared the transaction being built. The
statements after the inner transaction were then no longer in the
transaction, so they ran on their own, and when one of them failed,
those before it weren't rolled back; a second transaction in the outer
one began a transaction of its own, in its own mode. The F27 test put
the inner transaction last, so it didn't show.
beginTransaction() and endTransaction() now count how deeply the
transactions are nested, and only the outermost endTransaction() ends
the transaction. The fix is in these two functions rather than in the
inline transaction { } and transaction(mode) { }, which call them, so
that it also applies to code compiled against an earlier version, whose
inlined bodies still call them as before, and to nested calls of the
two functions themselves. beginTransaction() still returns true only for
the outermost transaction. endTransaction() gets documentation that
says what it does.
The test rolls back an outer transaction whose statement after an inner
one fails, with two inner ones in a row, with modes, after an inner one
whose block threw, and with nested calls of beginTransaction() and
endTransaction(), and commits the statements of the inner ones with the
outer one; it fails without the fix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
transactionStatementsGroup and transactionDepth were @volatile, which suggested that a scope could be shared between threads. It can't: the statements are collected in unsynchronized lists, and beginning a transaction checks the depth, creates a group and adds it in several steps, which neither @volatile nor an atomic would make safe. A scope is used by the code that runs its block; a suspending block may resume on another thread, but a coroutine's steps run one after another, its dispatcher making the writes before a suspension visible after it, so the fields need no @volatile, as the other fields of the scope have none. The documentation of DatabaseScope now says that a scope isn't thread-safe, and that its functions mustn't be called from threads or coroutines that run at the same time, as coroutines launched on a dispatcher of several threads in the block of suspendedScope would. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t 1) The first part of F30: PRAGMA_INTEGRITY_CHECK(maxErrors) and PRAGMA_QUICK_CHECK(maxErrors), which return the problems, or an empty list rather than SQLite's "ok"; PRAGMA_FOREIGN_KEY_CHECK(), of all tables or of one; PRAGMA_TABLE_INFO(table), of a table, a view or a name, PRAGMA_INDEX_LIST(table), PRAGMA_INDEX_XINFO(index) and PRAGMA_FOREIGN_KEY_LIST(table); PRAGMA_WAL_CHECKPOINT(mode), which a transaction rejects, as SQLite fails it there with "database table is locked"; PRAGMA_PAGE_COUNT(), PRAGMA_FREELIST_COUNT() and PRAGMA_PAGE_SIZE(); and PRAGMA_DEFER_FOREIGN_KEYS(flag), which only a transaction takes, as SQLite turns it off when one ends. A PragmaStatement runs a PRAGMA as a query when it returns rows, and with execSQL when it returns none, as Android's framework doesn't run a statement that returns rows with execSQL, and sqlite-jdbc doesn't run one that returns none as a query. It reads the rows when it runs and closes its cursor; getResults() returns them once the scope has run, as ColumnInfo, IndexInfo, IndexColumn, ForeignKeyInfo, ForeignKeyViolation, a CheckpointResult or a value, in com.ctrip.sqllin.dsl.sql.pragma. A PRAGMA changes no table, so it can be in a versioned read. Names are written as string literals, and columns are looked up by name once there is a row, as an older SQLite names no columns for a PRAGMA that returns no rows. PRAGMA_RECURSIVE_TRIGGERS and reading PRAGMA_FOREIGN_KEYS(), which the first part was to have, are left out: on Android 13 and higher, a database in WAL mode has several connections, and the framework steps a PRAGMA that sqlite3_stmt_readonly() reports as read-only on another connection than the one that writes, which only prepares it, so the statements cached there keep running as they were compiled, which a test on Robolectric API 37 showed for both; they need the PRAGMA to run on every connection, which PRAGMA_FOREIGN_KEYS(flag) needs too. The test reads auto_vacuum before checking freelist_count after a delete, as Android's SQLite creates a database with auto_vacuum FULL, which leaves no unused pages. The guide to modifying the database describes the PRAGMAs in a new section. The ROADMAP leaves case_sensitive_like out of F30, as SQLite deprecates it, and adds F67, the PRAGMAs that set how each connection runs statements, as parameters of the configurations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nnection on Android (B12) On Android 13 and higher, PRAGMA foreign_keys = ON, and OFF, and PRAGMA recursive_triggers, run with execSQL, as the DSL's PRAGMA_FOREIGN_KEYS does, didn't apply to the statements whose SQL had run before: they kept checking foreign keys or not, and firing triggers recursively or not. There, a database in WAL mode has several connections, the one that writes and those that read outside a transaction. The framework prepares such a PRAGMA on the connection that writes, but steps it on one that reads, as sqlite3_stmt_readonly() reports it doesn't write. SQLite changes the setting when it prepares the PRAGMA, but only expires the statements the connection has compiled when it steps it, so the statements the connection that writes had cached kept running as they were compiled, while those it compiled afterwards had the new setting. On Robolectric API 37, an INSERT whose SQL had run before PRAGMA_FOREIGN_KEYS(true) inserted a row that violated a foreign key. The Android driver's execSQL now runs these two PRAGMAs, outside a transaction, on every connection of the database, those opened later included, with execPerConnectionSQL, as the busy timeout is set, so that each connection steps them. Inside a transaction, which runs on the connection that writes, it runs them there, where SQLite ignores a change of foreign_keys, as on the other platforms, and applies recursive_triggers to the statements after it. Below Android 13, a database that SQLlin opens has a single connection, and nothing changes. No API changes: the driver recognizes the two PRAGMAs in the SQL. The test fails without the fix on API 37, and passes on API 26 and the other platforms either way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second part of F30, with the two the first part left out, which B12 made work on every connection: - PRAGMA_FOREIGN_KEYS(), which reads what PRAGMA_FOREIGN_KEYS(flag) sets; PRAGMA_RECURSIVE_TRIGGERS(flag), PRAGMA_LEGACY_ALTER_TABLE(flag) and PRAGMA_IGNORE_CHECK_CONSTRAINTS(flag), which the Android driver now runs on every connection too, as it does foreign_keys; - PRAGMA_INTEGRITY_CHECK(table) and PRAGMA_QUICK_CHECK(table), which need SQLite 3.33.0, below which they check the whole database; - PRAGMA_TABLE_XINFO(table), with ColumnInfo.kind for the hidden and generated columns, which needs SQLite 3.26.0, and PRAGMA_TABLE_LIST(), which needs 3.37.0; - PRAGMA_COMPILE_OPTIONS(), which Android's SQLite is built without, PRAGMA_COLLATION_LIST(), PRAGMA_DATABASE_LIST(), and PRAGMA_FUNCTION_LIST(), PRAGMA_MODULE_LIST() and PRAGMA_PRAGMA_LIST(), which need SQLite 3.30.0; - PRAGMA_APPLICATION_ID() and PRAGMA_APPLICATION_ID(id), and PRAGMA_AUTO_VACUUM() and PRAGMA_AUTO_VACUUM(mode). SQLite ignores a PRAGMA it lacks rather than failing, so those that only newer versions or some builds have check that the rows SQLite returns have their first column, and throw an IllegalStateException that tells what they need rather than return nothing, without asking SQLite for its version; they are marked with @PlatformDependentSQLiteAPI. SQLite takes auto_vacuum and page_size from what the connection last read of the header of the file, which it reads again when a transaction begins, so they are read in a transaction of their own outside one: on Robolectric API 37, auto_vacuum read outside a transaction, on a connection that reads, still told FULL after INCREMENTAL and VACUUM. Left out: incremental_vacuum, which returns a row without columns for each page it removes, so that sqlite-jdbc runs one step of it and can't query it, and Android's framework below 13 rejects it in execSQL; journal_size_limit, wal_autocheckpoint, max_page_count, secure_delete and cell_size_check, which each connection has, and some of which Android's framework sets on each connection it opens, so they go to F67. Android's SQLite turns legacy_alter_table on by default, which the test and the documentation take into account, and its own test reads the auto_vacuum and the build options of the SQLite it runs on. The guide to modifying the database describes them; the ROADMAP moves F30 to Supported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Kotlin/Native and on Android 8.1 and lower, journalMode didn't take effect, since 1.2.0: a database stayed in the journal mode it was in, DELETE for a new one on Kotlin/Native and that of the framework on Android, rather than WAL, the default. updateJournalMode ran `PRAGMA journal_mode=WAL` as a query whose cursor it never read, and a query there only runs as its cursor is read: NativeCursor steps the statement in next(), and Android's framework fills its window when the cursor is first moved. On Robolectric API 26, the mode stayed `memory`, and became `wal` once the row was read. The JVM sets the mode through SQLiteConfig, and Android 9 and higher through OpenParams. The row is now read. A connection that is actually read-only, as one that isReadOnly opens where the file can't be written, keeps the mode the file is in, as SQLite writes WAL mode to the header of the file: switching a file in DELETE mode to WAL fails there with "attempt to write a readonly database". testJournalMode checks the mode after opening with the default, with DELETE, and with the default again, and fails on API 26 without the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PRAGMA incremental_vacuum removes a page each time it returns a row, which has no columns, so it has to run as a query whose rows are all read. F30 left it out as "no driver steps through it", which was wrong: measured, Android's framework runs it fully as a query, on API 26 and 37, though execSQL removes one page and then throws, and Kotlin/Native would, as it steps each row of a query. Only sqlite-jdbc can't: it doesn't run a statement that returns no columns as a query, failing with "Query does not return results", and runs it as a statement, as DB.exec does too, only as far as its first row, removing one page. The KDoc of PRAGMA_AUTO_VACUUM(mode), the guide to modifying the database, in English and Chinese, the CHANGELOG entry of F30 and the ROADMAP line of F30 now say so. The JVM would have to run `PRAGMA incremental_vacuum(1)` once for each page, in a transaction, and few databases are in INCREMENTAL mode, which Android's SQLite doesn't create them in, so it goes to a new part of the ROADMAP, Shelved, of the items that aren't planned until a need for them comes up, as F68. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n created A connection reads the schema once and keeps it, and when the PRAGMAs that read the schema of a table or an index by its name don't find it, SQLite doesn't always check that schema is still current: index_list, index_info, index_xinfo and foreign_key_list never do, up to its trunk, and table_info and table_xinfo don't before 3.33.0, Android 13 and lower. So a table or index that another connection, such as one in another process, has just created looks as if it didn't exist, until this connection runs a statement that reads or writes a table, which does check it. Measured on SQLite 3.43.2 with two connections, and on Robolectric API 33. A connection's own DDL updates its schema at once, and on Android 13 and higher, where a Database reads through several connections, the framework closes those that read after each DDL that runs through execSQL, so a single Database, which SQLlin's apps almost always are, doesn't meet it. It isn't worked around: that would cost an extra query, and a transaction on Android, for every such PRAGMA. The guide to modifying the database, in English and Chinese, now says so where it says a table or index that doesn't exist has none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
F46 already added the COLLATE operator, with BINARY, NOCASE and RTRIM, for conditions and ORDER BY, so two parts of F19 were left: - @CollateRTrim, experimental, for COLLATE RTRIM in a column's definition, which compares the column's values ignoring the spaces at their ends, as its UNIQUE constraint and indexes do, while storing them as they are written. The processor reports a property that has both it and @CollateNoCase, as a column has one collation, and one that isn't a String or a Char, and rejects it on views and FTS tables, as it does the other constraints. A column needs no @CollateBinary, as BINARY is its default. - COLLATE on the columns of an index. CREATE_INDEX already wrote the operator's expression without the table's name, and SQLite reads `(name COLLATE NOCASE)` in an index as the column with that collation, as index_xinfo tells, rather than as an expression, so it only needed saying and testing, with an order too, as `(name COLLATE NOCASE) to DESC`. testCollations checks the column's SQL, a lookup that ignores the trailing spaces of a stored value, which keeps them, the UNIQUE constraint that rejects the value without them, the collations and orders of two indexes through PRAGMA_INDEX_XINFO, one of them BINARY on a NOCASE column, and that a query comparing by NOCASE uses the NOCASE index, through EXPLAIN_QUERY_PLAN. The guides to getting started, to modifying the database and to SQL functions, in English and Chinese, the KDoc of CREATE_INDEX and of COLLATE, the CHANGELOG and the ROADMAP, where F19 moves to Supported, say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`PersonTable DELETE WHERE(PersonTable.age LT 18) ORDER_BY (PersonTable.age
to ASC) LIMIT 10`, and the same after DELETE X, UPDATE ... WHERE and an
UPDATE without WHERE, with OFFSET after LIMIT, experimental.
SQLite only takes ORDER BY and LIMIT in a DELETE or UPDATE when it is
built with SQLITE_ENABLE_UPDATE_DELETE_LIMIT. sqlite-jdbc and Apple's
SQLite are; Android's isn't, as AOSP's Android.bp shows, nor the
libraries SQLlin links on Linux and Windows, where the statement fails
with `near "ORDER": syntax error`, as it does on Python's SQLite 3.53.4.
When it is, sqlite3LimitWhere() turns them into a subquery of the rowids
of the rows they choose, so SQLlin writes that subquery itself, which
every SQLite runs:
DELETE FROM person WHERE rowid IN (SELECT rowid FROM person
WHERE person.age<? ORDER BY person.age ASC LIMIT 10)
EXPLAIN QUERY PLAN of the two is the same, for DELETE and UPDATE, and
with INDEXED BY, which SQLite keeps for the subquery while the statement
finds the rows by their rowids: SQLlin puts INDEXED BY and NOT INDEXED in
the subquery only, as with both the statement scans the index. Unlike
SQLite's syntax, the subquery is allowed in the body of a trigger. A
table with a column named rowid, which rowid then means, reads its rowid
as _rowid_ or oid, which FTS tables take too.
ORDER BY without LIMIT, which SQLite rejects, fails the scope with an
IllegalArgumentException before any of its statements runs: an
UPDATE or DELETE with ORDER BY is an OrderByUpdateDeleteStatement, whose
checkComplete() throws. checkComplete() moves to SingleStatement, which
SelectStatement overrides, so that the scope, a transaction and
CREATE_TRIGGER check every statement alike.
DELETE X and DELETE WHERE(...), of a Table and of an IndexedTable, now
return a WhereUpdateDeleteStatement, which ORDER_BY and LIMIT follow,
rather than Unit, which only needs code compiled against an earlier
version to be compiled again; UPDATE ... WHERE returns one too rather
than the String of its SQL, which is a Breaking change. Their parts are
kept in UpdateDeleteParts, from which each clause builds the next
statement. ROADMAP F15 now says that WITHOUT ROWID tables will have to
choose the rows by their primary key.
testUpdateDeleteOrderByLimit covers DELETE with WHERE and without,
UPDATE with WHERE and without, with the parameters of SET before those
of WHERE, OFFSET with a negative LIMIT, LIMIT without ORDER BY,
INDEXED BY, a column named rowid, a transaction, a trigger that keeps the
latest rows, and ORDER BY without LIMIT in a scope, in a transaction and
in a trigger, after which nothing ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new module, com.ctrip.kotlin:sqllin-dsl-paging, on every platform of
sqllin-dsl, which androidx.paging's paging-common 3.5.3, its latest
release, has too. `database.pagingSource { PersonTable SELECT
ORDER_BY(PersonTable.id to ASC) }`, experimental, returns a
PagingSource<Int, T> that loads the rows a page at a time by their
offsets, as Room's LimitOffsetPagingSource does:
- a page is `SELECT * FROM (select) LIMIT ... OFFSET ...`, so the SELECT
can have a WHERE, a join or compound operators; the first page also
counts the rows, `SELECT count(*) FROM (select)`, so that each page
tells how many rows are before and after it, and a refresh past the
last row loads the last rows;
- the keys are offsets at the borders between pages: a prepend loads at
most as many rows as there are before its key, so that two pages never
share a row wherever a refresh starts, which needs no grid of the page
size, as the ROADMAP said; a page after the first row can always
prepend, as after a RemoteMediator invalidates the source;
- getRefreshKey starts half the initial load before the row last read,
adding the first page's itemsBefore without placeholders, whose
anchorPosition doesn't count the rows before it.
It is built on the table version tokens of F66: it registers with
onChange from its first page, through a callback that only holds the
source weakly, so that a source its Pager drops can be collected, and
closes the registration when it is invalidated otherwise. A page read
after a change since the first is LoadResult.Invalid, and every Invalid
is returned through invalidate(), so that the source is invalid once it
returns one, also for a TestPager, which doesn't invalidate it as a
Pager does; a source invalidated before it loads reads nothing. Pages
load in Dispatchers.IO by default.
The module can only reach sqllin-dsl's public API, so sqllin-dsl marks
what it loads through with the new @InternalSQLlinAPI, an opt-in of
ERROR level for SQLlin's own modules: Database.loadPage, LoadedPage and
TableVersions.isStale, over the internal DatabaseScope.page() and
RowCountStatement, which reads its count under the lock it runs under.
The module's build script, which a template made, now follows
sqllin-dsl's: its package, targets without iosX64, the SDK levels of the
version catalog and Maven Central publishing; the template's lint
plugin, kotlin-stdlib, kotlin-test, test runner and junit entries, its
.gitignore, manifest and example tests are gone. CI builds it on each
host, and publish_all.sh publishes it.
CommonPagingTest, in sqllin-dsl-test, loads it through TestPager and a
Pager: the keys and counts of refreshes, appends and prepends, at the
start, the middle, the end and past it; the refresh key with
placeholders and without; invalidation by a change, not by one of
another table nor before the first page, before loading, between the
two reads of a refresh, and before a later page; WHERE, an outer join
and a compound SELECT; and a Pager that reloads around the position read
after a delete. Each of four fixes that are undone fails it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A property of a type SQLite doesn't store can be a column when its serializer encodes it as one value: the serializer kotlinx.serialization writes the property with is the converter, so that no annotation of SQLlin's is needed. That covers the built-in serializers of Uuid, Instant and Duration, value classes, types whose classes declare their serializers, as kotlinx-datetime's do, and any type, a List stored as JSON included, given a serializer with @serializable(with = ...) on the property, on the use of its type or in a type alias, or by @file:UseSerializers. - sqllin-processor finds that serializer as the plugin does, and generates a ClauseConverted element that holds it, with a SET {} setter that stores what it encodes; a type no column can hold, a collection or a class serialized as a structure, or a type serialized contextually, without a SerializersModule to find its serializer in, is an error whose message says how to give it one. - The column's type is the one the serializer stores, which the table's createSQL, then a getter, computes with Table#columnType, and which ALTER_ADD_COLUMN now computes too, through the kind of the descriptor; a serializer that encodes a structure fails CREATE, as does a collation of a column that doesn't store text. The tables of every other entity are generated as before, byte for byte. - ClauseConverted, experimental, converts the values of EQ, NEQ, LT, LTE, GT, GTE, IN, of values and of a subquery, BETWEEN, IS and ISNOT with the serializer, and checks that it is the property's, so that a serializer the processor took wrongly fails rather than compares other values than those stored. Comparisons and ORDER BY compare the stored values, which the guide warns the ISO-8601 strings of Instant aren't in the order of. B14: an enum whose class declares a serializer, as one storing its name, or any property given a serializer of its own, was written by it in INSERT but compared and set as its type is by default, as an enum's ordinal, in a column of INT, so that conditions didn't match and a row SET {} changed couldn't be read. Such a property is now converted by its serializer, which changes the type of its element: a breaking change. An enum with its default serializer, or serialized contextually, is still stored as its ordinal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An entity annotated with an opt-in marker, as @ExperimentalUuidApi on its class, requires every use of the class to opt in, which the object generated for it, a Table, a View or an FtsTable of it, didn't, so it didn't compile, with a converted column or without. The object now opts in to the markers that the class, its properties and its file are annotated with, the annotations that are themselves annotated with @RequiresOptIn, besides those that their @optins give, which a table with converted columns needs for their types. The objects of the other entities are generated as before. Found by the review of F12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It could let the table version tokens of F66, and the PagingSource of F13, see the commits of other connections and processes, which they don't, but it doesn't tell which tables changed, and each connection counts on its own, which Android's pool of connections needs a design for. It was left out of F30 without being written down. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A SELECT kept its cursor, and its statement, open until its results
were read, which they may never be, as the SELECTs of versioned { }
whose token alone is wanted. Meanwhile the statement kept reading its
tables, which couldn't be dropped, failing with "database table is
locked", and on Kotlin/Native it kept the connection open after
close(), as sqlite3_close_v2 waits for the statements to end, holding
the database file. Windows can't delete an open file, so on CI every
test after testTableVersions opened its database, with its rows and
without the tables of its own configuration, and 78 of 116 failed;
Linux and macOS delete it anyway.
The SELECT now reads and decodes its rows when it runs, under the lock
of its scope and in its transaction, and closes the cursor, so that its
statement ends with the scope; getResults() returns the rows, and an
error decoding them is thrown by the scope. testUnreadSelect drops the
table of an unread SELECT, which failed with SQLITE_LOCKED on the JVM,
and deletes the closed database, which failed on Windows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The macOS, Windows and Linux jobs uploaded sqllin-dsl/build/reports on failure, but the DSL's tests run in sqllin-dsl-test, so that the report of a failed run, as that of the Windows tests of #135, wasn't uploaded. The Android job already uploaded the right one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The new features of 2.6.0, and the bugs found while making them, from
feature/new-features-2.6.0, which started fromrelease/2.6.0after the first bug fixes and contains all of it. Every item has its entry inCHANGELOG.md, and the guides are updated; new DSL API is experimental, behind@ExperimentalDSLDatabaseAPI.New features
(PersonTable INNER_JOIN BookTable ON (...)) SELECT X<R>(), withoutFROM, with every join operator. (f05b95c)Database#versioned { },awaitChangeandonChange, which tell a reader when the tables it read change through the sameDatabase. (113cd45)INSERT_OR_ABORT,INSERT_OR_FAILandINSERT_OR_ROLLBACK. (c432843)transaction(mode) { }, which begins a transaction DEFERRED, IMMEDIATE or EXCLUSIVE. (5f7f480)INDEXED_BYandNOT_INDEXED. (15175d0)EXPLAIN_QUERY_PLAN(select), read as a tree ofQueryPlansteps. (36a5269)@CollateRTrim, and COLLATE on the columns of an index. (63eb2eb)ORDER_BY,LIMITandOFFSETafterDELETEandUPDATE, written as the subquery of rowids that SQLite turns them into, so that they run on every platform, Android's SQLite lackingSQLITE_ENABLE_UPDATE_DELETE_LIMIT. (5ae53ce)com.ctrip.kotlin:sqllin-dsl-paging, on every platform of sqllin-dsl:Database#pagingSource { }returns androidx.paging'sPagingSourceof a SELECT, built on F66, as Room'sLimitOffsetPagingSourceis. CI builds and publishes it. (8dd6317)Uuid,InstantandDuration, value classes, types whose classes declare their serializers, and any type given one with@Serializable(with = ...)or@file:UseSerializers, as aListstored as JSON. The column has the type the serializer stores, and its element,ClauseConverted<T>, converts the values of its conditions and ofSET {}with it. (e617d73)Fixes
transaction { }ended the outer transaction, since 2.0.0 at least. (59b86cd)PRAGMA_FOREIGN_KEYS(flag)and the other PRAGMAs that set how a connection runs statements didn't apply to statements whose SQL had run before. (12fbbbf)journalModehasn't taken effect since 1.2.0. (517491e)INSERTbut compared and set as its type is by default, so that conditions didn't match and a rowSET {}changed couldn't be read. It is now a column converted by its serializer. (e617d73)@ExperimentalUuidApi, didn't opt in to it, so it didn't compile. (21da315)close(), holding the database file, which Windows can't delete: the first CI run of this PR failed 78 of the 116 Windows tests, which all opened the database thattestTableVersionsleft. A SELECT now reads its rows when its scope runs it. (691685c)Breaking changes
UNION {}andUNION_ALL {}blocks, which 2.5.0 deprecated (0d8efdc).UPDATE ... WHEREreturns aWhereUpdateDeleteStatementrather than theStringof its SQL (F25);DELETEreturns a statement rather thanUnit, which only breaks binary compatibility.ClauseConvertedelement rather than aClauseEnum,ClauseStringorClauseNumber(B14), whose operators are experimental. Tables created before need no migration.Other
DatabaseScopeno longer has@Volatilefields, which suggested it could be shared between threads; its documentation says it isn't thread-safe. (63628a4)sqllin-dsl-teston macOS, Windows and Linux, rather than those ofsqllin-dsl, which has no tests, so that a failed run's report is kept. (de0df18)incremental_vacuum(F68, 340797b) anddata_version(F69, f454e0a).Testing
sqllin-dsl-testpassesjvmTest(117 tests) andtestAndroidHostTeston Robolectric API 26 and 37 (234 tests), andsqllin-driverits JVM (13) and Android (27) tests. Each fix but B9, a build setting checked by the version of the class files, has a test that fails without it.macosArm64,iosSimulatorArm64andwatchosArm32, and compiled forlinuxX64andmingwX64. The first CI run of this PR passed on macOS, Linux and Android, and failed on Windows for B16.🤖 Generated with Claude Code