Skip to content

Support query attributes in the Databricks OLAP driver - #9852

Open
mpasa wants to merge 1 commit into
rilldata:mainfrom
mpasa:databricks-query-attributes
Open

Support query attributes in the Databricks OLAP driver#9852
mpasa wants to merge 1 commit into
rilldata:mainfrom
mpasa:databricks-query-attributes

Conversation

@mpasa

@mpasa mpasa commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Databricks is currently the only OLAP driver that ignores a metrics view's query_attributes. #8266 wired them into every driver that existed at the time, but the Databricks OLAP driver landed later in #9255 and was never hooked up, so the attributes are silently dropped today.

This sends them as Databricks query tags, which land in the query_tags column of system.query.history.

Why query tags rather than a SQL comment

Query tags are attached per statement (as a ConfOverlay on the ExecuteStatement request), not per session, so they stay correct even though connections are pooled and shared between users. They're also a real map<string,string> column, so multiple attributes survive without packing them into one string.

A SQL comment prefix was the other option, but it depends on the query text being readable: statement_text can be redacted workspace-wide, in which case the comment carries nothing. On the workspace I tested, 100% of ~41k queries over 24h had statement_text = '<REDACTED>', so that approach would have been a silent no-op there.

Handling workspaces without query tags

Query tags are Public Preview and not available everywhere, and Databricks fails a statement outright when it doesn't recognise a configuration rather than ignoring it:

[CONFIG_NOT_AVAILABLE.WITHOUT_SUGGESTION] Configuration query_tags is not available.  SQLSTATE: 42K0I

Sending tags unconditionally would therefore break every query of any metrics view that sets query_attributes on such a workspace. So the first rejection is detected, the query retried without tags, and the outcome latched on the connection:

  • workspace supports query tags → no extra queries, ever
  • workspace doesn't → exactly one wasted attempt, on the first query only

The retry is safe even though Exec routes through Query: CONFIG_NOT_AVAILABLE is raised while setting up the statement, before it executes, so the first attempt has no effect.

I couldn't find a way to detect support up front — SET -v on my warehouse lists only 8 settable configs and doesn't mention query_tags, yet statement-level tagging works there, so the config listing isn't a usable signal. An up-front probe would also add a round-trip for every workspace, including the supported ones that currently pay nothing.

Dependency bump

databricks-sql-go v1.10.0 → v1.14.0.

Per-statement query tags (driverctx.NewContextWithQueryTags) first shipped in v1.11.0 — note the Databricks docs claim Go connector v1.9.0+, but query_tags.go doesn't exist before v1.11.0. v1.14.0 is the newest release before v1.15.0 began requiring the databricks-sql-kernel-bindings modules, which add ~585MB of prebuilt platform binaries to the module graph (not reachable from any package we build, but still fetched by go mod download/CI). Happy to go to v1.15.x instead if you'd prefer to be on latest — it builds and passes tests, it just also pulls in thrift 0.23 and nine golang.org/x/* upgrades.

Verification

Against a live SQL warehouse, a Query() with QueryAttributes set is recorded in query history as:

[{"key":"rillUserEmail","value":"..."},{"key":"rillProbe","value":"..."}]

Also exercised through a real rill start with query_attributes on a Databricks-backed metrics view. And with the driver locally patched to send a configuration the workspace rejects, queries still succeed via the untagged retry (and fail without it).

Open questions

  • Preference on the dependency version (v1.14.0 vs v1.15.x)?
  • Is the silent retry+latch the behaviour you want, or would you rather surface a reconcile-time warning that query_attributes are unsupported on the workspace?

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

Databricks was the only OLAP driver that ignored a metrics view's
query_attributes. rilldata#8266 wired them into every driver that existed at the time,
but the Databricks OLAP driver landed later in rilldata#9255 and was never hooked up, so
the attributes were silently dropped.

Send them as Databricks query tags, which are recorded in the query_tags column
of system.query.history. The driver attaches tags per statement (as a
ConfOverlay on the ExecuteStatement request) rather than per session, so they
remain correct even though connections are pooled and shared between users.
This also avoids depending on the query text: statement_text can be redacted
workspace-wide, in which case a SQL comment would carry nothing.

Query tags are Public Preview and not available on every workspace, and
Databricks fails a statement outright when it doesn't recognise a configuration
rather than ignoring it:

  [CONFIG_NOT_AVAILABLE.WITHOUT_SUGGESTION] Configuration query_tags is not available.

Sending tags unconditionally would therefore break every query of any metrics
view that sets query_attributes on such a workspace. So the first rejection is
detected, the query retried without tags, and the result latched on the
connection so later queries skip tagging altogether.

Bumps databricks-sql-go to v1.14.0. Per-statement query tags
(driverctx.NewContextWithQueryTags) first shipped in v1.11.0, and v1.14.0 is the
newest release before v1.15.0 started requiring the databricks-sql-kernel-bindings
modules, which add ~585MB of prebuilt platform binaries to the module graph
without being reachable from any package we build.

Verified against a live SQL warehouse: a Query() with QueryAttributes set is
recorded in query history as

  [{"key":"rillUserEmail","value":"..."},{"key":"rillProbe","value":"..."}]

and, with the driver patched to send a configuration the workspace rejects,
queries still succeed via the untagged retry.
@k-anshul

k-anshul commented Sep 2, 2026

Copy link
Copy Markdown
Member

Hey @mpasa

Thanks for the PR.
Regarding open questions:

  1. Yeah it seems better to move to v1.14.0 for now.
  2. A reconcile error should be okay. A user should be expected to remove the query attributes from the connector definition if the service does not support it. We can make a SELECT 1 query in driver.Open method to see if tags are supported.

@nishantmonu51 nishantmonu51 added Type:Feature New feature request Area:Connectors Size:M Medium change: 100-499 lines labels Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area:Connectors Size:M Medium change: 100-499 lines Type:Feature New feature request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants