Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
scripts/*.sh text eol=lf
50 changes: 50 additions & 0 deletions .github/actions/sqlite/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: SQLite library
description: >-
Provides the static SQLite library that the native tests link on Linux and Windows, libs/<target>/libsqlite3.a,
which isn't in the repository: from the cache, or built from the sources of the version with scripts/build-sqlite.sh

inputs:
version:
description: The SQLite version, one that scripts/build-sqlite.sh lists
required: true
target:
description: linux or windows
required: true

runs:
using: composite
steps:
# A version's library never changes unless how it is built does, so it is almost always here
- name: Restore SQLite ${{ inputs.version }} from the cache
id: cache
uses: actions/cache@v4
with:
path: libs/${{ inputs.target }}/libsqlite3.a
key: sqlite-${{ inputs.target }}-${{ inputs.version }}-${{ hashFiles('scripts/build-sqlite.sh', '.github/actions/sqlite/action.yml') }}

# zig comes from PyPI, which needs a Python that pip may install into, as that of Ubuntu isn't
- name: Set up Python
if: steps.cache.outputs.cache-hit != 'true'
uses: actions/setup-python@v5
with:
python-version: '3.13'

- name: Build SQLite ${{ inputs.version }}
if: steps.cache.outputs.cache-hit != 'true'
shell: bash
env:
ZIG_VERSION: 0.16.0
ZIG: python -m ziglang
SQLITE_VERSION: ${{ inputs.version }}
SQLITE_TARGET: ${{ inputs.target }}
run: |
python -m pip install --quiet "ziglang==$ZIG_VERSION"
bash scripts/build-sqlite.sh "$SQLITE_VERSION" "$SQLITE_TARGET"

- name: Show the SQLite library
shell: bash
env:
SQLITE_TARGET: ${{ inputs.target }}
run: |
ls -l "libs/$SQLITE_TARGET/libsqlite3.a"
if command -v sha256sum > /dev/null 2>&1; then sha256sum "libs/$SQLITE_TARGET/libsqlite3.a"; else shasum -a 256 "libs/$SQLITE_TARGET/libsqlite3.a"; fi
79 changes: 79 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,12 @@ jobs:
- name: Checkout
uses: actions/checkout@v4

- name: Build SQLite
uses: ./.github/actions/sqlite
with:
version: 3.51.1
target: windows

- name: Validate Gradle Wrapper
uses: gradle/actions/wrapper-validation@v3

Expand Down Expand Up @@ -146,6 +152,12 @@ jobs:
- name: Checkout
uses: actions/checkout@v4

- name: Build SQLite
uses: ./.github/actions/sqlite
with:
version: 3.51.1
target: linux

- name: Validate Gradle Wrapper
uses: gradle/actions/wrapper-validation@v3

Expand Down Expand Up @@ -204,6 +216,73 @@ jobs:
path: sqllin-dsl-test/build/reports
if: failure()

# The native tests on Linux again, linking the SQLite versions of the Android API levels that no other job runs:
# Robolectric's SQLite is 3.32.2 at its lowest. They tell what those versions of SQLite take, not what Android's
# build of them or its framework does.
test-on-older-sqlite:
name: test-on-sqlite-${{ matrix.sqlite }}
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
# Android API 24 and 25, 26, 28 and 29, and 30
sqlite: [ '3.9.2', '3.18.2', '3.22.0', '3.28.0' ]

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Build SQLite
uses: ./.github/actions/sqlite
with:
version: ${{ matrix.sqlite }}
target: linux

- name: Validate Gradle Wrapper
uses: gradle/actions/wrapper-validation@v3

- name: Set up JDK 21
uses: actions/setup-java@v4
with:
distribution: 'zulu'
java-version: 21

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v4

- name: Cache Kotlin/Native
uses: actions/cache@v4
with:
path: ~/.konan
key: ${{ runner.os }}-konan-${{ hashFiles('**/*.gradle.kts') }}
restore-keys: |
${{ runner.os }}-konan-

- name: Run sqllin-driver Linux X64 Tests on SQLite ${{ matrix.sqlite }}
run: ./gradlew :sqllin-driver:cleanLinuxX64Test :sqllin-driver:linuxX64Test --stacktrace

- name: Run sqllin-dsl Linux X64 Tests on SQLite ${{ matrix.sqlite }}
run: ./gradlew :sqllin-dsl-test:cleanLinuxX64Test :sqllin-dsl-test:linuxX64Test --stacktrace

# The library holds its version as a string, and no test has that string
- name: Check that the tests linked SQLite ${{ matrix.sqlite }}
run: strings -a sqllin-dsl-test/build/bin/linuxX64/debugTest/test.kexe | grep -Fx '${{ matrix.sqlite }}'

- name: Upload sqllin-driver Reports
uses: actions/upload-artifact@v4
with:
name: Test-Reports-SQLite-${{ matrix.sqlite }}-driver
path: sqllin-driver/build/reports
if: failure()

- name: Upload sqllin-dsl Reports
uses: actions/upload-artifact@v4
with:
name: Test-Reports-SQLite-${{ matrix.sqlite }}-dsl
path: sqllin-dsl-test/build/reports
if: failure()

build-android-and-test:
runs-on: ubuntu-latest
timeout-minutes: 60
Expand Down
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,5 @@ local.properties
/sqllin-dsl-test/build
/sample/build
*.podspec
.kotlin
.kotlin
/libs
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,22 @@

- Date format: YYYY-MM-dd

## 2.7.0 / 2026-xx-xx

### All

* The SQLite static libraries that the native tests on Linux and Windows link are no longer in the repository: CI builds them from the sources that sqlite.org publishes, whose SHA-256 it checks, with the new `scripts/build-sqlite.sh` and zig, and caches them, and the script builds them for whoever runs those tests on a Linux or Windows machine. For SQLite 3.51.1 it gives the bytes of the libraries that were in `libs`. On Windows, the tests now link this library before one that MSYS2 may have, as they do on Linux before the system's
* CI runs the native tests on Linux on older versions of SQLite too: 3.9.2, 3.18.2, 3.22.0 and 3.28.0, those of Android API 24, 26, 28 and 30, which no test ran on, as the SQLite of Robolectric is 3.32.2 at its lowest. The tests of what depends on the version of SQLite now ask for it: RENAME COLUMN, the tree of a query plan, the CHECK constraints that integrity_check verifies, a rename that rewrites the views that read the table, and the date and time functions in an index

### sqllin-dsl

* Fix: `PRAGMA_COMPILE_OPTIONS()` failed with "This SQLite doesn't have PRAGMA compile_options" on a SQLite before 3.16.0 that has the PRAGMA, which names its column `compile_option` rather than `compile_options`, by which SQLlin told whether the SQLite has it. It now takes either name. Android's SQLite, built without the PRAGMA, still fails, as the documentation says, and Apple's systems that SQLlin supports have newer versions, so it was a SQLite that an app links on Linux or Windows that met it
* Fix: an `INSERT` of entities that bind more than 999 values, as 334 rows of 3 columns, failed with "too many SQL variables" on a SQLite before 3.32.0, as on Android before API 31, which takes 999 parameters in a statement by default, as all the values were bound to one statement. Such an INSERT now runs in batches that bind at most 999 values each, whatever the SQLite, in a transaction, so that it inserts all the rows or none, as one statement does, but for `INSERT_OR_FAIL`, which keeps the rows inserted before the one that fails, as one statement does. Its SQL, which a trigger writes into its body, is still one statement. An `IN` of a list of more than 999 values still fails there, as the guide to advanced queries and the documentation of `IN` now say
* New experimental DSL API: `DROP_TABLE(name)`, `DROP_TABLE_IF_EXISTS(name)`, `DROP_VIEW(name)` and `DROP_VIEW_IF_EXISTS(name)`, for SQL syntax `DROP TABLE` and `DROP VIEW`, which drop a table or a view by its name, as `DROP_INDEX(name)` and `DROP_TRIGGER(name)` do an index and a trigger, so that a migration can drop one whose class is gone, where `DROP` and `DROP_IF_EXISTS` need its table object. The kind is in the name, as SQLite's `DROP TABLE` fails for a view and `DROP VIEW` for a table, with `IF EXISTS` too; an FTS table is dropped as a table; and a blank name is rejected with an `IllegalArgumentException` as the statement is built. The guide to modifying the database describes them
* New guide: [Testing](sqllin-dsl/doc/testing.md), in English and Chinese, about testing the code that uses SQLlin against a real SQLite database: where one runs in the tests of each platform; the setup of Android host tests with Robolectric, whose SDK level chooses the framework code that runs but not the SQLite version, which is 3.32.2 at every level up to 34 with Robolectric 4.17, so that what depends on the SQLite version needs an emulator or a device; closing and deleting the database of a test; `observe` and `pagingSource` under `runTest`; generating the table objects of the `@DBRow` classes of a test source set, by adding sqllin-processor to the KSP configuration of the test compilation, as `kspAndroidHostTest`; and testing a migration
* The guide to SQL functions says that a SQLite before 3.20.0, as Android's before API 28, rejects every date and time function in an index
* The guide to getting started says that the table object of a class with a column converted by its serializer computes its `createSQL` when it is read, so that the generated code holds no whole CREATE statement. The guide to advanced queries says that a page of `pagingSource` is the rows of one SELECT, so that it can't hold a one-to-many relation, which can be a converted column where no query filters by it, or a `PagingSource` of one's own on table version tokens otherwise

## 2.6.0 / 2026-11-08

### All
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ You can learn how to use _sqllin-dsl_ in these documentations:
- [Concurrency Safety](./sqllin-dsl/doc/concurrency-safety.md)
- [SQL Functions](./sqllin-dsl/doc/sql-functions.md)
- [Advanced Query](./sqllin-dsl/doc/advanced-query.md)
- [Testing](./sqllin-dsl/doc/testing.md)

I don't recommend using _sqllin-driver_ directly, but if you want to learn more about it, you can read:

Expand Down
1 change: 1 addition & 0 deletions README_CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ _sqllin-dsl_ 是 SQL 语句的 DSL 实现并且它基于 _sqllin-driver_。_sqll
- [并发安全](./sqllin-dsl/doc/concurrency-safety-cn.md)
- [SQL 函数](./sqllin-dsl/doc/sql-functions-cn.md)
- [高级查询](./sqllin-dsl/doc/advanced-query-cn.md)
- [测试](./sqllin-dsl/doc/testing-cn.md)

虽然不建议直接使用 _sqllin-driver_,但如果你想了解更多信息则可以阅读:

Expand Down
3 changes: 3 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ done. A new item takes the next number.
* F64: Support the JSON functions and JSON's `->` and `->>` operators (SQLite 3.38.0, Android API 34), left out of F46
* F65: Support the conflict clause of NOT NULL, as `NOT NULL ON CONFLICT REPLACE`, which writes the default value instead of NULL, left out of F17: a property's type declares NOT NULL, so no annotation can hold the clause yet; it could be a new annotation, or a parameter of `@Default`. SQLlin's statements rarely write NULL to a non-null column, so it is of little use for now
* F67: Support the PRAGMAs that set how each connection runs statements, such as cache_size, temp_store, mmap_size, query_only, automatic_index, trusted_schema, threads, cache_spill, analysis_limit, journal_size_limit, wal_autocheckpoint, max_page_count, secure_delete and cell_size_check, as parameters of `DatabaseConfiguration` and `DSLDBConfiguration` rather than as statements: Android's framework reads through several connections of a database in WAL mode from Android 13 on, while a statement sets a PRAGMA on the one connection that writes, so they have to be set on every connection, as `busyTimeout` is, and Android's framework sets journal_size_limit and wal_autocheckpoint on each connection it opens; and a change of temp_store deletes the TEMP tables and triggers, which observed queries count changes with
* F71: Support reads that depend on a page in the `PagingSource` of F13, as `pagingSource(query) { rows -> ... }`, which loads the rows that a one-to-many relation joins to the page's rows, such as tags filtered by, with one batched `IN` of the page's ids: each read takes a token of the table versions of F66, which the page's token takes in, so that a change of a related table invalidates the source too; the reads return one item for each row, in the same order, as the keys are offsets of the page's SELECT, and must not be skipped for what the page holds, but an empty page, or their tables could be declared up front, so that their versions are taken even when nothing reads them. The page and its reads run in scopes of their own, which the combined token keeps consistent; reading them in one transaction needs F31
* F72: Support `IN` of a list of more values than SQLite binds in one statement, 999 before SQLite 3.32.0, as on Android before API 31, by writing the values into the SQL as literals, as triggers, views and indexes have them, rather than binding them: such a list fails now, while an INSERT of as many values runs in batches (B2 of 2.7.0), which an `IN` can't, as `ORDER BY`, `LIMIT` and aggregates would change

## Shelved

Expand All @@ -44,6 +46,7 @@ Items that aren't planned, as they are of little use, or would cost more than th

## Supported

* F70: Support dropping a table or a view by its name, `DROP_TABLE(name)`, `DROP_TABLE_IF_EXISTS(name)`, `DROP_VIEW(name)` and `DROP_VIEW_IF_EXISTS(name)`, as `DROP_INDEX(name)` and `DROP_TRIGGER(name)` do, so that a migration can drop a table whose class is gone; the kind is in the name, as `DROP TABLE` fails on a view (2.7.0 ✅)
* F12: Support type converters: store a property of any type through a serializer that encodes it to a type SQLite supports, with type-safe WHERE and SET on its column. The serializer kotlinx.serialization writes the property with is the converter: built-in ones of `Uuid`, `Instant` and `Duration`, value classes, and any given with `@Serializable(with = ...)` or `@file:UseSerializers`; the column's element, `ClauseConverted`, converts the values of its conditions with it (2.6.0 ✅)
* F13: Provide a `PagingSource` for androidx.paging built on the table version tokens of F66, like Room's `LimitOffsetPagingSource`, in the new module sqllin-dsl-paging, `Database#pagingSource { }` (2.6.0 ✅)
* F25: Support ORDER BY and LIMIT in UPDATE and DELETE, which SQLite only takes with its `SQLITE_ENABLE_UPDATE_DELETE_LIMIT` option, which Android's lacks: SQLlin writes the subquery of rowids that SQLite turns them into, which every platform runs (2.6.0 ✅)
Expand Down
Binary file removed libs/linux/libsqlite3.a
Binary file not shown.
Binary file removed libs/windows/libsqlite3.a
Binary file not shown.
98 changes: 98 additions & 0 deletions scripts/build-sqlite.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
#!/usr/bin/env bash
#
# Builds the static SQLite library that SQLlin's native tests link on Linux and Windows, from the sources of a SQLite
# version that sqlite.org publishes. The libraries aren't in the repository: CI builds them with this script, see
# .github/actions/sqlite, and so does whoever runs the native tests on a Linux or Windows machine.
#
# Usage: scripts/build-sqlite.sh <version> <linux|windows> [<output>]
#
# scripts/build-sqlite.sh 3.51.1 linux # writes libs/linux/libsqlite3.a, which the build links
# scripts/build-sqlite.sh 3.9.2 linux /tmp/sqlite.a # an older version, as CI runs the tests on
#
# It needs curl, tar, sha256sum or shasum, and zig (https://ziglang.org), which compiles for both targets on any host:
# `zig` on the PATH, or the command in $ZIG, as ZIG="python3 -m ziglang" for the package of PyPI. CI uses zig 0.16.0,
# with which the same version and target give the same bytes on every host and in every directory.

set -euo pipefail

if [ $# -lt 2 ] || [ $# -gt 3 ]; then
sed -n '3,13p' "$0" | sed 's/^# \{0,1\}//' >&2
exit 2
fi

version=$1
target=$2
root=$(cd "$(dirname "$0")/.." && pwd)
output=${3:-$root/libs/$target/libsqlite3.a}
zig=${ZIG:-zig}

# The year of each version is part of its URL, and the SHA-256 of its sources is checked, so a version has to be
# listed here to be built: sqlite-autoconf-<number>.tar.gz, which sqlite.org has for every release, patch releases
# included.
case "$version" in
3.51.1) year=2025; sha256=4f2445cd70479724d32ad015ec7fd37fbb6f6130013bd4bfbc80c32beb42b7e0 ;; # What the tests run on
3.28.0) year=2019; sha256=d61b5286f062adfce5125eaf544d495300656908e61fca143517afcc0a89b7c3 ;; # Android API 30
3.22.0) year=2018; sha256=2824ab1238b706bc66127320afbdffb096361130e23291f26928a027b885c612 ;; # Android API 28 and 29
3.18.2) year=2017; sha256=0d40222ea818a559590c51994ac85570eff5c5f754cbe08c2d34a3791942c1ba ;; # Android API 26
3.9.2) year=2015; sha256=064c0abe9c9177534d4c770bca7a5902f9924b629ac886b4c08956be6dfbc36b ;; # Android API 24 and 25
*)
echo "Unknown SQLite version '$version': add its year and the SHA-256 of its sources to $0." >&2
exit 2
;;
esac

# What Kotlin/Native links the tests for: glibc no newer than that of its sysroot on Linux, and MinGW-w64 on Windows.
# clang records the name of the object file in a COFF object, which zig compiles to a file with a random name, so the
# name is given, for the same bytes each time.
case "$target" in
linux) triple=x86_64-linux-gnu.2.17; extra="" ;;
windows) triple=x86_64-windows-gnu; extra="-Xclang -object-file-name=sqlite3.o" ;;
*)
echo "Unknown target '$target': linux or windows." >&2
exit 2
;;
esac

# The full-text search modules, FTS3 and FTS4 with the enhanced query syntax, as the JVM's and Apple's SQLite have, and
# FTS5; R*Tree and Geopoly; the math functions; the dbstat virtual table; and column metadata. A version that lacks
# one ignores its option. -g0, as zig emits debug information by default, which holds the paths of the machine.
flags="-O2 -g0 -fPIC -DSQLITE_THREADSAFE=1 \
-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_FTS5 \
-DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_GEOPOLY -DSQLITE_ENABLE_MATH_FUNCTIONS \
-DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_COLUMN_METADATA"

# 3.51.1 -> 3510100
number=$(echo "$version" | awk -F. '{ printf "%d%02d%02d00", $1, $2, $3 }')
name=sqlite-autoconf-$number
url=https://www.sqlite.org/$year/$name.tar.gz

work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT

echo "Downloading $url"
curl -fsSL --retry 3 --retry-delay 5 -o "$work/$name.tar.gz" "$url"

if command -v sha256sum > /dev/null 2>&1; then
actual=$(sha256sum "$work/$name.tar.gz" | cut -d ' ' -f 1)
else
actual=$(shasum -a 256 "$work/$name.tar.gz" | cut -d ' ' -f 1)
fi
if [ "$actual" != "$sha256" ]; then
echo "The SHA-256 of $url is $actual, not $sha256." >&2
exit 1
fi

# Compiled in the directory of the sources, with relative paths, so that the host's kind of path doesn't matter
(
cd "$work"
tar -xzf "$name.tar.gz" "$name/sqlite3.c"
cd "$name"
echo "Compiling SQLite $version for $triple with zig $($zig version)"
# shellcheck disable=SC2086
$zig cc -target "$triple" -c sqlite3.c $flags $extra -o sqlite3.o
$zig ar rcs libsqlite3.a sqlite3.o
)

mkdir -p "$(dirname "$output")"
cp "$work/$name/libsqlite3.a" "$output"
echo "Wrote $output"
2 changes: 1 addition & 1 deletion sqllin-driver/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ fun KotlinNativeTarget.setupNativeConfig() {
binaries.all {
linkerOpts += when {
HostManager.hostIsLinux -> listOf("-lsqlite3", "-L$rootDir/libs/linux", "-L/usr/lib/x86_64-linux-gnu", "-L/usr/lib", "-L/usr/lib64")
HostManager.hostIsMingw -> listOf("-Lc:\\msys64\\mingw64\\lib", "-L$rootDir\\libs\\windows", "-lsqlite3")
HostManager.hostIsMingw -> listOf("-L$rootDir\\libs\\windows", "-Lc:\\msys64\\mingw64\\lib", "-lsqlite3")
else -> listOf("-lsqlite3")
}
}
Expand Down
Loading
Loading