Repository navigation
SqliteVec: support per-connection initialization callbacks - #81
rossdonald wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The store-level path runs the initializer before loading sqlite-vec, causing ListCollectionNamesAsync to fail with pooling disabled when the initializer calls vec_version().
1 open issue
What changed in this PR
Adds opt-in per-connection initialization to SqliteVec, enabling consumers to configure PRAGMAs and other connection settings as requested in #73.
Changes:
- Adds initializer callbacks to store and collection options, including propagation to store-created collections.
- Centralizes connection setup and disposal when initialization fails.
- Adds callback coverage and bumps the provider version to
1.0.4-preview.
| File | Description |
|---|---|
| MEVD/test/SqliteVec.ConformanceTests/SqliteConnectionInitializerTests.cs | Tests invocation, propagation, extension ordering, and failure cleanup. |
| MEVD/src/SqliteVec/SqliteVectorStoreOptions.cs | Adds and documents the store-level callback. |
| MEVD/src/SqliteVec/SqliteVectorStore.cs | Propagates callbacks and uses shared connection setup. |
| MEVD/src/SqliteVec/SqliteVec.csproj | Updates the provider package version. |
| MEVD/src/SqliteVec/SqliteConnectionFactory.cs | Centralizes connection opening, initialization, and failure cleanup. |
| MEVD/src/SqliteVec/SqliteCollectionOptions.cs | Adds and documents the collection-level callback. |
| MEVD/src/SqliteVec/SqliteCollection.cs | Routes collection connections through the shared factory. |
Review effort: Balanced
| using var connection = await SqliteConnectionFactory.OpenAsync( | ||
| _connectionString, | ||
| _connectionInitializer, | ||
| loadVector: false, |
adamsitnik
left a comment
There was a problem hiding this comment.
@rossdonald big thanks for your contribution!
The changes look good, but I think we should rename the callback and make it async, so we don't call sync code from async method. PTAL at my comments.
Similar public API exposed by BCL: https://learn.microsoft.com/en-us/dotnet/api/system.net.http.socketshttphandler.connectcallback?view=net-10.0
| /// <c>PRAGMA synchronous=NORMAL</c>, <c>PRAGMA cache_size</c> or <c>PRAGMA temp_store=MEMORY</c>, or for loading additional extensions. | ||
| /// </para> | ||
| /// <para> | ||
| /// When the collection is created from a <see cref="SqliteVectorStore"/>, the callback configured on the vector store options |
There was a problem hiding this comment.
Thank you for documenting this important information 👍
| connection.LoadVector(); | ||
| } | ||
|
|
||
| initializer?.Invoke(connection); |
There was a problem hiding this comment.
The callback is synchronous. Are there any SqliteConnection properties that require calling async methods to configure? I just want to be 100% sure the callback does not need to return a Task/ValueTask in order to avoid calling async methods from sync callback.
|
|
||
| public void Dispose() | ||
| { | ||
| File.Delete(_databasePath); |
There was a problem hiding this comment.
thank you for taking care of the cleanup 👍
|
|
||
| using var setCommand = connection.CreateCommand(); | ||
| setCommand.CommandText = "PRAGMA synchronous=NORMAL"; | ||
| setCommand.ExecuteNonQuery(); |
There was a problem hiding this comment.
If this is the typical usage, we are calling sync method from an async context. How about making the callback async and accepting a CancellationToken?
public Func<SqliteConnection, CancellationToken, ValueTask> ConnectionInitializer { get; set; }| /// thread-safe. | ||
| /// </para> | ||
| /// </remarks> | ||
| public Action<SqliteConnection>? ConnectionInitializer { get; set; } |
There was a problem hiding this comment.
I looked for some "prior art" in ASP.NET and EF Core and BCL and found "ConfigureHttpClient", "ConfigureEndpointDefaults", "ConfigureDataSource" and I think that "ConfigureConnection" would be better.
| public Action<SqliteConnection>? ConnectionInitializer { get; set; } | |
| public Action<SqliteConnection>? ConfigureConnection { get; set; } |
| connection.LoadVector(); | ||
| } | ||
|
|
||
| initializer?.Invoke(connection); |
There was a problem hiding this comment.
With pooling on (the Microsoft.Data.Sqlite default), this runs again every time a pooled physical connection is reused, and what it sets stays on that connection. The tests use Pooling=false, so they don't cover it. Two cases that fail on this branch with the default connection string and pass with Pooling=false:
- an
ATTACH DATABASEinitializer fails on the second operation:SQLite Error 1: 'database aux is already in use'; PRAGMA query_only=ONon one collection leaks to another collection with the same connection string:SQLite Error 8: 'attempt to write a readonly database'.
Maybe document that the callback must be safe to run more than once on the same connection, and add a test with pooling on?
Repro tests
[Fact]
public async Task AttachInitializerWorksOnEveryOperationWithPooling()
{
var options = new SqliteCollectionOptions
{
ConnectionInitializer = connection =>
{
using var command = connection.CreateCommand();
command.CommandText = $"ATTACH DATABASE '{_auxPath}' AS aux";
command.ExecuteNonQuery();
}
};
var collection = new SqliteCollection<string, ProbeRecord>(ConnectionString, "AttachCollection", options);
await collection.EnsureCollectionExistsAsync();
await collection.UpsertAsync(new ProbeRecord { Key = "k1", Text = "hello", Vector = new([1f, 2f, 3f, 4f]) });
}
[Fact]
public async Task QueryOnlyFromOneCollectionDoesNotLeakToAnotherWithPooling()
{
var readOnly = new SqliteCollection<string, ProbeRecord>(ConnectionString, "ReadOnlyCollection", new SqliteCollectionOptions
{
ConnectionInitializer = connection =>
{
using var command = connection.CreateCommand();
command.CommandText = "PRAGMA query_only=ON";
command.ExecuteNonQuery();
}
});
var writable = new SqliteCollection<string, ProbeRecord>(ConnectionString, "WritableCollection");
await writable.EnsureCollectionExistsAsync();
Assert.False(await readOnly.CollectionExistsAsync());
await writable.UpsertAsync(new ProbeRecord { Key = "k1", Text = "hello", Vector = new([1f, 2f, 3f, 4f]) });
}ConnectionString is $"Data Source={_databasePath}" (pooling on), and _auxPath is a second temp file. ProbeRecord has a string key, a Text data property and a 4-dimension vector.

Add an opt-in connection initialization callback to SqliteVectorStoreOptions and SqliteCollectionOptions. If defined, the connector invokes it right after it opens a connection and loads the vec0 extension allowing per-connection options to be set.
Fixes #73