Commit 9259644
committed
feat(backend/kernel): route use_sea=True through the Rust kernel
Phase 2 of the PySQL × kernel integration plan
(databricks-sql-kernel/docs/designs/pysql-kernel-integration.md).
Wires `use_sea=True` to a new `backend/kernel/` module that
delegates to the Rust kernel via the `databricks_sql_kernel` PyO3
extension (kernel PR #13).
New module: `src/databricks/sql/backend/kernel/`
- `client.py` — `KernelDatabricksClient(DatabricksClient)`. Lazy-
imports `databricks_sql_kernel` so a connector install without
the kernel wheel doesn't `ImportError` at startup; only
`use_sea=True` surfaces the missing-extra message. Implements
open/close_session, sync + async execute_command (async_op=True
goes through `Statement.submit()` and stashes the handle in a
dict keyed on `CommandId`), cancel/close_command,
get_query_state, get_execution_result, and the metadata calls
(catalogs / schemas / tables / columns) via
`Session.metadata().list_*`. Real server-issued session and
statement IDs flow through (no synthetic UUIDs).
- `auth_bridge.py` — translate the connector's `AuthProvider`
into kernel `Session` kwargs. PAT (including federation-wrapped
PAT — `get_python_sql_connector_auth_provider` always wraps the
base in `TokenFederationProvider`, so a naive isinstance check
never matches) routes through `auth_type="pat"`. Everything
else routes through `auth_type="external"` with a callback that
delegates to `auth_provider.add_headers({})`. (External today
is rejected by the kernel at `build_auth_provider`; the
separate kernel-side enablement PR will flip it on.)
- `result_set.py` — `KernelResultSet(ResultSet)`. Duck-typed over
`databricks_sql_kernel.ExecutedStatement` (sync execute) and
`ResultStream` (metadata + async await_result) since both
expose `arrow_schema()` / `fetch_next_batch()` /
`fetch_all_arrow()` / `close()`. Same FIFO batch buffer the
prior ADBC POC used, so `fetchmany(n)` for n smaller than the
kernel's natural batch size doesn't re-fetch.
- `type_mapping.py` — Arrow → PEP 249 description-string mapper.
Lifted from the prior ADBC POC; centralised here so future
kernel-result wrappers reuse the same mapping.
Kernel errors → PEP 249 exceptions: `KernelError.code` is mapped
in a single table to `ProgrammingError` / `OperationalError` /
`DatabaseError`. The structured fields (`sql_state`,
`error_code`, `query_id`, …) are copied onto the re-raised
exception so callers can branch on them without reaching through
`__cause__`.
Routing: `Session._create_backend` flips the `use_sea=True` branch
to instantiate `KernelDatabricksClient` instead of the native
`SeaDatabricksClient`. The native `backend/sea/` module is left
in place (no users on `use_sea=True` after this PR; its long-
term fate is out of scope here).
Packaging: `[tool.poetry.extras] kernel = ["databricks-sql-kernel"]`.
`pip install 'databricks-sql-connector[kernel]'` pulls in the
kernel wheel; `use_sea=True` without the extra raises a pointed
ImportError telling the user how to install it.
Known gaps (acknowledged, will be follow-ups):
- Parameter binding (`execute_command(parameters=[...])`) raises
NotSupportedError — PyO3 `Statement.bind_param` lands in a
follow-up.
- Statement-level `query_tags` raises NotSupportedError.
- `get_tables(table_types=[...])` returns unfiltered rows (the
native SEA backend's filter is keyed on `SeaResultSet`; needs
a small port to operate on `KernelResultSet`).
- External-auth end-to-end blocked on the kernel-side
`AuthConfig::External` enablement PR.
- Volume PUT/GET (staging operations): kernel has no Volume API.
Test plan:
- Unit: 37 new tests across
`tests/unit/test_kernel_auth_bridge.py` (auth provider →
kwargs mapping, including federation-wrapped PAT and the
External trampoline call-counter check),
`tests/unit/test_kernel_type_mapping.py` (Arrow type mapping
+ description shape), and
`tests/unit/test_kernel_result_set.py` (buffer semantics,
fetchmany across batch boundaries, idempotent close, close()
swallowing handle-close failures). All pass.
- Full unit suite: 600 pre-existing tests still pass; one
pre-existing failure (`test_useragent_header` — agent
detection adds `agent/claude-code` in this env) was already
failing on main, unrelated to this change.
- Live e2e against dogfood with `use_sea=True`: SELECT 1,
`range(10000)`, `fetchmany` pacing, `fetchall_arrow`, all four
metadata calls (returned 75 catalogs / 144 schemas in main /
47 tables in `system.information_schema` / 15 columns),
`session_configuration={'ANSI_MODE': 'false'}` round-trips,
bad SQL surfaces as DatabaseError with `code='SqlError'`
and `sql_state='42P01'` on the exception. All checks pass.
Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>1 parent cbd6a88 commit 9259644
10 files changed
Lines changed: 1321 additions & 8 deletions
File tree
- src/databricks/sql
- backend/kernel
- tests/unit
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
35 | 40 | | |
36 | 41 | | |
37 | 42 | | |
38 | 43 | | |
| 44 | + | |
39 | 45 | | |
40 | 46 | | |
41 | 47 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
0 commit comments