Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -84,12 +84,10 @@ android/gradle/
!.yarn/sdks
!.yarn/versions

# Generated by the build pipelines from generate_tokenizers_header_file.js
# Copied out of the app's own c_sources by android/build.gradle
android/c_sources
# c_sources/ is also generated (copied out of the app's own c_sources by the
# podspec/Package.swift) but stays tracked: SwiftPM evaluates Package.swift in a
# sandbox that can deny that copy, so an SPM build off a fresh checkout needs the
# files to already be there. Android no longer depends on it.
# c_sources/ is also a copy (made by the podspec and react-native.config.js). It is
# tracked, but no build depends on that: they recreate it.
# c_sources/

scripts/sqlite-vec-*
Expand Down
159 changes: 45 additions & 114 deletions Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,11 @@ import PackageDescription

// Hand-authored SPM manifest for React Native's experimental SwiftPM support
// (RN 0.87, "Preview" — see scripts/spm/__doc__/spm-scripts.md in react-native).
// This is a direct translation of the logic in `op-sqlite.podspec` +
// `generate_tokenizers_header_file.rb`: CocoaPods evaluates that podspec (and
// its dynamic per-consumer-app configuration) as Ruby during `pod install`.
// This is a direct translation of the logic in `op-sqlite.podspec`:
// CocoaPods evaluates that podspec (and its dynamic per-consumer-app configuration) as Ruby during `pod install`.
// SwiftPM evaluates *this* file as Swift on every resolve, so the same
// dynamic logic — reading the consumer app's package.json, generating
// cpp/tokenizers.h, choosing which SQLite backend & xcframeworks to link —
// dynamic logic — reading the consumer app's package.json, finding its
// tokenizers, choosing which SQLite backend & xcframeworks to link —
// is reproduced here with Foundation instead of Ruby's FileUtils/JSON.
//
// Per RN's "hand-authored community library contract"
Expand All @@ -25,8 +24,12 @@ import PackageDescription

let packageRoot = URL(fileURLWithPath: #filePath).deletingLastPathComponent()

// #filePath keeps the autolinker's symlink path (see the header), which is not
// inside node_modules, so this has to look at the real directory.
let resolvedPackageRoot = packageRoot.resolvingSymlinksInPath()

// In the sample app the dir is not inside of node_modules.
let isUserApp = packageRoot.path.contains("node_modules")
let isUserApp = resolvedPackageRoot.path.contains("node_modules")

func readJSONObject(atPath path: String) -> [String: Any] {
guard let data = FileManager.default.contents(atPath: path),
Expand All @@ -38,15 +41,27 @@ func readJSONObject(atPath path: String) -> [String: Any] {
return dict
}

// When installed into node_modules, find the consumer's package.json by
// searching up through parent directories (starting one level above this
// package, i.e. outside node_modules/@op-engineering/op-sqlite). In the
// monorepo/dev checkout it's simply example/package.json.
let appPackageJSONPath: String = {
// Prefer the app, the same package.json Android reads. It is the folder above
// the one holding build/generated/autolinking; the search further down can't
// find it under pnpm or when hoisted, where it ends at the workspace root.
// Without the autolinker's symlink the app is unknown.
if let marker = packageRoot.path.range(of: "/build/generated/autolinking/libs/") {
let candidate = URL(fileURLWithPath: String(packageRoot.path[..<marker.lowerBound]))
.deletingLastPathComponent().appendingPathComponent("package.json").path
if readJSONObject(atPath: candidate)["op-sqlite"] != nil {
return candidate
}
}

// In the monorepo/dev checkout it's simply example/package.json.
guard isUserApp else {
return packageRoot.appendingPathComponent("example/package.json").path
}
var currentDir = packageRoot.deletingLastPathComponent()

// Otherwise search up through parent directories, starting one level above
// this package, i.e. outside node_modules/@op-engineering/op-sqlite.
var currentDir = resolvedPackageRoot.deletingLastPathComponent()
while true {
let candidate = currentDir.appendingPathComponent("package.json").path
if FileManager.default.fileExists(atPath: candidate) {
Expand Down Expand Up @@ -102,75 +117,6 @@ if !tokenizers.isEmpty && useTurso {

print("[OP-SQLITE] SPM configuration found at \(appPackageJSONPath)")

// MARK: - Tokenizer header generation
// The header contents come from generate_tokenizers_header_file.js, the one
// implementation shared by every build pipeline (CocoaPods and Gradle call the
// same script). This only has to find node and run it.

func generateTokenizersHeaderFile(names: [String], filePath: String) {
let scriptPath = packageRoot.appendingPathComponent("generate_tokenizers_header_file.js").path

// NODE_BINARY is React Native's own convention (it writes one into
// ios/.xcode.env). Xcode does not necessarily inherit the user's shell PATH,
// so fall back to a PATH lookup and then to the usual install locations.
var candidates: [String] = []
if let fromEnv = ProcessInfo.processInfo.environment["NODE_BINARY"], !fromEnv.isEmpty {
candidates.append(fromEnv)
}
candidates += ["node", "/opt/homebrew/bin/node", "/usr/local/bin/node", "/usr/bin/node"]

for node in candidates {
let process = Process()
// Going through `env` is what resolves a bare "node" against PATH.
process.executableURL = URL(fileURLWithPath: "/usr/bin/env")
process.arguments = [node, scriptPath, filePath] + names
process.standardOutput = FileHandle.standardError

do {
try process.run()
process.waitUntilExit()
if process.terminationStatus == 0 {
return
}
} catch {
continue
}
}

// Not fatal, on purpose. Manifest evaluation happens in contexts we do not
// control (Xcode's package resolution, `swift package dump-package`), and
// SwiftPM sandboxes writes there, so the generation can legitimately fail
// while an already up-to-date header sits on disk -- the script exits 0
// without writing in that case, so reaching this point means it really could
// not produce the file. Killing the manifest would be worse than letting the
// compiler report the missing header.
FileHandle.standardError.write(
Data(
("[OP-SQLITE] warning: could not generate \(filePath) with \(scriptPath). Make sure node is "
+ "on your PATH or set NODE_BINARY to its location.\n").utf8))
}

// Mirrors FileUtils.cp_r: copies the contents of `source` into `destination`,
// creating directories as needed and overwriting existing files.
func copyDirectoryContents(from source: URL, to destination: URL) {
let fm = FileManager.default
guard let enumerator = fm.enumerator(at: source, includingPropertiesForKeys: [.isDirectoryKey]) else {
return
}
try? fm.createDirectory(at: destination, withIntermediateDirectories: true)
for case let itemURL as URL in enumerator {
let relative = itemURL.path.replacingOccurrences(of: source.path + "/", with: "")
let destURL = destination.appendingPathComponent(relative)
let isDirectory = (try? itemURL.resourceValues(forKeys: [.isDirectoryKey]))?.isDirectory ?? false
if isDirectory {
try? fm.createDirectory(at: destURL, withIntermediateDirectories: true)
} else {
try? fm.removeItem(at: destURL)
try? fm.copyItem(at: itemURL, to: destURL)
}
}
}

// MARK: - Source file collection
// Mirrors Dir.glob("ios/**/*.{h,hpp,m,mm}") + Dir.glob("cpp/**/*.{hpp,h,cpp,c}")

Expand Down Expand Up @@ -222,28 +168,21 @@ if !useTurso {
}

// MARK: - Tokenizers
// Set to non-user-app value ("../example/c_sources/tokenizers.h") by default;
// swapped to "../c_sources/tokenizers.h" below when installed as a dependency.
// Both branches faithfully mirror op-sqlite.podspec, which points at the
// *original* c_sources dir in dev mode but the *copy* (below) once installed.
var tokenizersHeaderPath = "../example/c_sources/tokenizers.h"

// The manifest sandbox forbids writing anywhere, so react-native.config.js
// generates the app's c_sources/tokenizers.h and copies the folder here, as the
// podspec does. It isn't published with the package, so it may not exist yet.
if !tokenizers.isEmpty {
print("[OP-SQLITE] Tokenizers enabled: \(tokenizers)")

let cSourcesDir =
isUserApp
? packageRoot.deletingLastPathComponent().deletingLastPathComponent().deletingLastPathComponent()
.appendingPathComponent("c_sources")
: packageRoot.appendingPathComponent("example/c_sources")
let localCSourcesDir = packageRoot.appendingPathComponent("c_sources")

generateTokenizersHeaderFile(
names: tokenizers, filePath: cSourcesDir.appendingPathComponent("tokenizers.h").path)
copyDirectoryContents(from: cSourcesDir, to: localCSourcesDir)

sourceFiles += globFiles(under: "c_sources", extensions: ["h", "cpp"])
tokenizersHeaderPath = isUserApp ? "../c_sources/tokenizers.h" : "../example/c_sources/tokenizers.h"
let tokenizersHeader = packageRoot.appendingPathComponent("c_sources/tokenizers.h").path
if !FileManager.default.fileExists(atPath: tokenizersHeader) {
fatalError(
"[OP-SQLITE] Tokenizers \(tokenizers) are enabled, but \(tokenizersHeader) is missing. Run `npx react-native spm update` to create c_sources/tokenizers.h.")
}
if globFiles(under: "c_sources", extensions: ["cpp"]).isEmpty {
fatalError(
"[OP-SQLITE] Tokenizers \(tokenizers) are enabled, but there are no .cpp files in c_sources. Create c_sources/tokenizers.cpp with your implementation and run `npx react-native spm update`.")
}
}

// MARK: - Backend selection (exclude_files equivalent)
Expand Down Expand Up @@ -273,13 +212,6 @@ if phoneVersion {
excludedSources.formUnion(["cpp/sqlite3.c", "cpp/sqlite3.h"])
}

// A prior manifest evaluation (tokenizers previously enabled) may have left
// c_sources/*.{h,cpp} on disk even though tokenizers is empty now — exclude
// the whole dir so a stale copy doesn't get swept into the build.
if tokenizers.isEmpty {
excludedDirs.insert("c_sources")
}

sourceFiles.removeAll { excludedSources.contains($0) }
sourceFiles = Array(Set(sourceFiles)).sorted()

Expand Down Expand Up @@ -333,7 +265,7 @@ if !sqliteFlags.isEmpty {
defines += parseDefines(sqliteFlags)
}
if !tokenizers.isEmpty {
defines.append(("TOKENIZERS_HEADER_PATH", "\\\"\(tokenizersHeaderPath)\\\""))
defines.append(("TOKENIZERS_HEADER_PATH", "\\\"../c_sources/tokenizers.h\\\""))
}

let defineCSettings: [CSetting] = defines.map { name, value in
Expand Down Expand Up @@ -389,9 +321,8 @@ targetDependencies += [

// Best-effort translation of `s.dependency "OpenSSL-Universal"`. There's no
// SwiftPM release of the CocoaPods OpenSSL-Universal binary; this points at
// the closest maintained SwiftPM OpenSSL package instead. The example app
// doesn't enable sqlcipher, so this path is untested — verify ABI/symbol
// compatibility before shipping a sqlcipher-enabled SPM build.
// the closest maintained SwiftPM OpenSSL package instead. The ios-sqlcipher CI
// job builds and tests this path through the example app.
if useSqlcipher {
packageDependencies.append(
.package(url: "https://github.com/krzyzanowskim/OpenSSL.git", from: "3.3.3000"))
Expand All @@ -408,13 +339,13 @@ if useSqlcipher {
//
// NOTE: SwiftPM's manifest evaluation runs in a filesystem sandbox that
// forbids writes outside temp/build directories (confirmed empirically: an
// attempt to stage files into a repo-local scratch dir failed with EPERM,
// even though the tokenizer-header write above degrades gracefully via
// `try?`). That rules out physically relocating unwanted files at resolve
// attempt to stage files into a repo-local scratch dir failed with EPERM).
// That rules out physically relocating unwanted files at resolve
// time — the fix has to be purely declarative `exclude:` entries instead,
// which is why the backend-specific sources above are excluded by directory
// wherever the repo layout allows it.
let neededTopLevelDirs: Set<String> = ["ios", "cpp", "c_sources"]
// c_sources stays out when tokenizers are off, in case an earlier run left a copy.
let neededTopLevelDirs: Set<String> = tokenizers.isEmpty ? ["ios", "cpp"] : ["ios", "cpp", "c_sources"]
let topLevelExcludes: [String] =
(try? FileManager.default.contentsOfDirectory(atPath: packageRoot.path))?
.filter { !neededTopLevelDirs.contains($0) && $0 != "Package.swift" && $0 != "op-sqlite-spm-prefix.h" }
Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,14 @@ Some of the big supported features:

It also contains a simple [Key-Value store](https://op-engineering.github.io/op-sqlite/docs/key_value_storage) you can use without adding one more dependency to your app.

# Custom tokenizers

1. Add `"tokenizers": ["your_tokenizer"]` to the `"op-sqlite"` config in your app's `package.json`.
2. Run `npx react-native config` from your app's folder. It creates `c_sources/tokenizers.h`.
3. Write your implementation in `c_sources/tokenizers.cpp` and commit the `c_sources` folder.

Run the command again whenever you change `"tokenizers"`. See the [tokenizers docs](https://op-engineering.github.io/op-sqlite/docs/tokenizers) for the details.

# License

MIT License.
44 changes: 7 additions & 37 deletions android/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -143,38 +143,6 @@ if(!tokenizers.isEmpty()) {
println "[OP-SQLITE] Tokenizers enabled. Detected tokenizers: " + tokenizers
}

// Generates the tokenizers.h that cpp/OPBridge.cpp includes (through
// TOKENIZERS_HEADER_PATH) and that the copied tokenizers.cpp includes by name.
//
// The generation itself lives in generate_tokenizers_header_file.js, shared with
// the CocoaPods (generate_tokenizers_header_file.rb) and SwiftPM (Package.swift)
// pipelines, so all three emit the same header. Before this, Android had no
// generator of its own and could only build if an iOS pod install (or a header
// committed to the repo) had produced the file first.
//
// Uses a plain ProcessBuilder rather than project.exec: this runs at
// configuration time, where that API is deprecated/removed in recent Gradle.
def generateTokenizersHeaderFile(List<String> names, File headerFile) {
def packageDir = buildscript.sourceFile.parentFile.parentFile
def script = new File(packageDir, "generate_tokenizers_header_file.js")
// NODE_BINARY is React Native's own convention for pointing at a specific node.
def node = System.getenv("NODE_BINARY")
if (!node) {
node = "node"
}

def command = [node, script.absolutePath, headerFile.absolutePath] + names
def process = new ProcessBuilder(command).redirectErrorStream(true).start()
def output = process.inputStream.text
def status = process.waitFor()

if (status != 0) {
throw new GradleException(
"[OP-SQLITE] Could not generate ${headerFile}. Command: ${command.join(' ')}\n" +
"Make sure node is on your PATH or set NODE_BINARY to its location.\n${output}")
}
}

// Build the list of library-default SQLite compile flags driven by package.json
// toggles. These are emitted via CMake add_definitions() BEFORE the user-supplied
// sqliteFlags, so that user flags take precedence when both define the same macro
Expand Down Expand Up @@ -231,11 +199,13 @@ android {
if (!tokenizers.isEmpty()) {
def sourceDir = isUserApp ? file("$rootDir/../../../c_sources") : file("$rootDir/../c_sources")
def destDir = file("$buildscript.sourceFile.parentFile/c_sources")
// Generated into the app's own c_sources, exactly where the podspec and
// Package.swift put it, so the header the user's tokenizers.cpp includes
// is there no matter which platform was built last. The copy below then
// carries it over with the sources.
generateTokenizersHeaderFile(tokenizers, new File(sourceDir, "tokenizers.h"))
// tokenizers.h is generated by react-native.config.js. Autolinking loads
// it when the app's package.json changes, before this runs. The copy
// below carries the header over with the sources.
def tokenizersHeader = new File(sourceDir, "tokenizers.h")
if (!tokenizersHeader.exists()) {
throw new GradleException("[OP-SQLITE] Tokenizers ${tokenizers} are enabled, but ${tokenizersHeader} is missing. Run `npx react-native config` from your app's folder to create it.")
}
copy {
from sourceDir
into destDir
Expand Down
10 changes: 6 additions & 4 deletions docs/docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,12 @@ Required response headers (for worker/OPFS setup):
SQLite is very customizable on compilation level. op-sqlite also allows you add extensions or even change the base implementation. You can do this by adding the following to your `package.json`:

> [!IMPORTANT]
> When using a monorepo, be sure to add the "op-sqlite" config to the monorepo root `package.json` file.
> This is necessary because the `node_modules/@op-engineering/op-sqlite/op-sqlite.podspec` will search for first the `package.json` file in a parent directory.
> Alternative, you may be able to solve this by blocking this package from being hoisted to the root `node_modules`.
> Check your ios/Podfile.lock to see where it's being installed.
> In a monorepo, put the "op-sqlite" config in your app's `package.json` (the folder above `ios/` and `android/`). Android reads only that file.
> On iOS (CocoaPods and SwiftPM), if the app's `package.json` has no "op-sqlite" key, the build falls back to the first `package.json` above the installed package, usually the monorepo root. A config kept only in the root therefore applies to iOS but not to Android. If both have one, the app's is used and the two are not merged.
> `pod install` prints which file it used (`[OP-SQLITE] Configuration found at ...`). SwiftPM prints `[OP-SQLITE] SPM configuration found at ...`, but only when it evaluates the manifest rather than reusing its cache.

> [!WARNING]
> SwiftPM caches the evaluated manifest and doesn't notice changes to `package.json`, so with SwiftPM a config change can silently have no effect, even after a clean build. After changing the "op-sqlite" config, run `rm -rf ~/Library/Caches/org.swift.swiftpm/manifests` and do a clean build.

```json
{
Expand Down
Loading
Loading