Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ jobs:
test-integration:
needs: [ci-scope, test-unit]
if: needs.ci-scope.outputs.ci_required == 'true'
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
strategy:
fail-fast: true
matrix:
Expand Down Expand Up @@ -288,12 +288,22 @@ jobs:
sudo apt-get clean
df -h /

- name: Install SQL Server ODBC driver
run: |
set -eu
curl -sSL -O https://packages.microsoft.com/config/ubuntu/24.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
rm packages-microsoft-prod.deb
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 unixodbc-dev
odbcinst -q -d -n "ODBC Driver 18 for SQL Server"

- name: Cache docker images
id: docker-image-cache
uses: actions/cache@v6
with:
path: /tmp/docker-images
key: docker-images-v1-${{ hashFiles('.github/workflows/ci.yml') }}
key: docker-images-v2-${{ hashFiles('.github/workflows/ci.yml') }}

- name: Load cached docker images
if: steps.docker-image-cache.outputs.cache-hit == 'true'
Expand All @@ -312,6 +322,7 @@ jobs:
paradedb/paradedb:0.21.5-pg16
cockroachdb/cockroach:latest
mysql:8.4
mcr.microsoft.com/mssql/server:2022-latest
gvenzl/oracle-free:23-slim-faststart
ghcr.io/goccy/bigquery-emulator:latest
gcr.io/cloud-spanner-emulator/emulator:latest
Expand Down
6 changes: 4 additions & 2 deletions docs/extensions/adk/adapters.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,11 @@ Use async adapters for best performance with ADK runners:
- **DuckDB**: ``duckdb`` (analytics; reduced-scope for ADK)
- **ADBC**: ``adbc`` (Arrow-native portability; reduced-scope for ADK)
- **Spanner**: ``spanner`` (Google Cloud, globally distributed)
- **SQL Server over ODBC**: ``arrow_odbc`` with Microsoft ODBC Driver 18

Sync adapters (``psycopg`` sync mode, ``sqlite``, ``mysqlconnector``, ``pymysql``)
work but require wrapping with ``anyio`` for async ADK runners.
Sync adapters (``psycopg`` sync mode, ``sqlite``, ``mysqlconnector``,
``pymysql``, ``arrow_odbc``) work but require wrapping with ``anyio`` for async
ADK runners.

Each Adapter Provides
=====================
Expand Down
19 changes: 19 additions & 0 deletions docs/extensions/adk/backends.rst
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,11 @@ The table below classifies every backend by its ADK support level.
- Full
- Basic
- Portability layer; native adapters provide optimized search.
* - arrow_odbc
- Supported
- Full
- Basic
- SQL Server through Microsoft ODBC Driver 18.
* - spanner
- Supported
- Full
Expand Down Expand Up @@ -270,6 +275,20 @@ ADBC (Arrow Database Connectivity) provides a driver-agnostic interface:
- Memory search uses the portable baseline path; choose a native adapter for
backend-specific FTS, retention, and storage tuning.

arrow-odbc
----------

``arrow_odbc`` provides SQL Server-backed ADK storage through Microsoft ODBC
Driver 18:

- Session and event storage use SQL Server tables with ``DATETIME2(6)`` and
``NVARCHAR(MAX)`` JSON payload columns.
- ``append_event_and_update_state()`` writes the session update, event row, and
scoped state in one committed session.
- Memory storage uses the portable text-search path with ``LIKE`` matching.
- Row-oriented ``execute_many()`` is not available; use native Arrow ingest for
bulk database writes outside the ADK store.

Spanner
-------

Expand Down
29 changes: 29 additions & 0 deletions docs/reference/adapters/arrow_odbc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,16 @@ Streams ``pyarrow.RecordBatchReader`` results from any ODBC-compliant driver,
making it a good fit for read-heavy analytical transfer between SQL Server,
PostgreSQL, MySQL, or other ODBC sources and the Arrow ecosystem.

SQL Server coverage is exercised in CI against SQL Server 2022 through
``pytest-databases`` and Microsoft ODBC Driver 18. The shared contract matrix
verifies native Arrow reads, Arrow reader/batch output, and Arrow bulk ingest
for this adapter. Row-oriented ``execute_many()`` is intentionally unsupported;
use ``load_from_arrow()`` for bulk writes.

Extension support is SQL Server-backed. The adapter exports a table-backed
events queue store, a Litestar session store, and Google ADK session/event and
memory stores for SQL Server connections through Microsoft ODBC Driver 18.

Configuration
=============

Expand Down Expand Up @@ -36,6 +46,25 @@ Data Dictionary
:members:
:show-inheritance:

Extensions
==========

.. autoclass:: sqlspec.adapters.arrow_odbc.events.ArrowOdbcEventQueueStore
:members:
:show-inheritance:

.. autoclass:: sqlspec.adapters.arrow_odbc.litestar.ArrowOdbcStore
:members:
:show-inheritance:

.. autoclass:: sqlspec.adapters.arrow_odbc.adk.ArrowOdbcADKStore
:members:
:show-inheritance:

.. autoclass:: sqlspec.adapters.arrow_odbc.adk.ArrowOdbcADKMemoryStore
:members:
:show-inheritance:

Schema Discovery
================

Expand Down
4 changes: 4 additions & 0 deletions docs/reference/extensions/events.rst
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,10 @@ Listeners
Event Queue
===========

The durable table queue is available for SQL Server through ``arrow_odbc`` when
configured with Microsoft ODBC Driver 18. It uses SQL Server ``DATETIME2(6)``
timestamps and ``NVARCHAR`` payload columns.

.. autoclass:: sqlspec.extensions.events.AsyncTableEventQueue
:members:
:show-inheritance:
Expand Down
5 changes: 4 additions & 1 deletion docs/usage/drivers_and_querying.rst
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,10 @@ asyncpg and CockroachDB-asyncpg (cursors inside a stream-owned transaction),
pymysql/aiomysql/asyncmy (``SSCursor``), mysql-connector (unbuffered cursors),
sqlite/aiosqlite and oracledb (chunked ``fetchmany``), psqlpy (server-side
cursor with ``array_size``), and BigQuery (page-wise result iteration).
ADBC, DuckDB, mssql-python, arrow-odbc, and Spanner are eager-fallback only.
ADBC, DuckDB, mssql-python, and Spanner are eager-fallback only for row
streaming. ``arrow-odbc`` row streaming also materializes dict rows eagerly, but
``select_to_arrow(..., return_format="reader" | "batches")`` uses the native
``arrow_odbc`` ``RecordBatchReader`` path.

Lifetime and transaction rules:

Expand Down
1 change: 1 addition & 0 deletions docs/usage/frameworks/litestar/session_stores.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ SQLSpec provides stores for async adapters:

- ``AsyncpgStore`` - PostgreSQL via asyncpg
- ``AiosqliteStore`` - SQLite via aiosqlite
- ``ArrowOdbcStore`` - SQL Server via arrow-odbc and Microsoft ODBC Driver 18

Each store automatically creates its session table on first use if it doesn't exist.

Expand Down
5 changes: 5 additions & 0 deletions sqlspec/adapters/arrow_odbc/adk/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
"""arrow-odbc ADK extension exports."""

from sqlspec.adapters.arrow_odbc.adk.store import ArrowOdbcADKConfig, ArrowOdbcADKMemoryStore, ArrowOdbcADKStore

__all__ = ("ArrowOdbcADKConfig", "ArrowOdbcADKMemoryStore", "ArrowOdbcADKStore")
Loading
Loading