Support query attributes in the Databricks OLAP driver - #9852
Open
mpasa wants to merge 1 commit into
Open
Conversation
Databricks was the only OLAP driver that ignored a metrics view's query_attributes. rilldata#8266 wired them into every driver that existed at the time, but the Databricks OLAP driver landed later in rilldata#9255 and was never hooked up, so the attributes were silently dropped. Send them as Databricks query tags, which are recorded in the query_tags column of system.query.history. The driver attaches tags per statement (as a ConfOverlay on the ExecuteStatement request) rather than per session, so they remain correct even though connections are pooled and shared between users. This also avoids depending on the query text: statement_text can be redacted workspace-wide, in which case a SQL comment would carry nothing. Query tags are Public Preview and not available on every workspace, and Databricks fails a statement outright when it doesn't recognise a configuration rather than ignoring it: [CONFIG_NOT_AVAILABLE.WITHOUT_SUGGESTION] Configuration query_tags is not available. Sending tags unconditionally would therefore break every query of any metrics view that sets query_attributes on such a workspace. So the first rejection is detected, the query retried without tags, and the result latched on the connection so later queries skip tagging altogether. Bumps databricks-sql-go to v1.14.0. Per-statement query tags (driverctx.NewContextWithQueryTags) first shipped in v1.11.0, and v1.14.0 is the newest release before v1.15.0 started requiring the databricks-sql-kernel-bindings modules, which add ~585MB of prebuilt platform binaries to the module graph without being reachable from any package we build. Verified against a live SQL warehouse: a Query() with QueryAttributes set is recorded in query history as [{"key":"rillUserEmail","value":"..."},{"key":"rillProbe","value":"..."}] and, with the driver patched to send a configuration the workspace rejects, queries still succeed via the untagged retry.
Member
|
Hey @mpasa Thanks for the PR.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Databricks is currently the only OLAP driver that ignores a metrics view's
query_attributes. #8266 wired them into every driver that existed at the time, but the Databricks OLAP driver landed later in #9255 and was never hooked up, so the attributes are silently dropped today.This sends them as Databricks query tags, which land in the
query_tagscolumn ofsystem.query.history.Why query tags rather than a SQL comment
Query tags are attached per statement (as a
ConfOverlayon theExecuteStatementrequest), not per session, so they stay correct even though connections are pooled and shared between users. They're also a realmap<string,string>column, so multiple attributes survive without packing them into one string.A SQL comment prefix was the other option, but it depends on the query text being readable:
statement_textcan be redacted workspace-wide, in which case the comment carries nothing. On the workspace I tested, 100% of ~41k queries over 24h hadstatement_text = '<REDACTED>', so that approach would have been a silent no-op there.Handling workspaces without query tags
Query tags are Public Preview and not available everywhere, and Databricks fails a statement outright when it doesn't recognise a configuration rather than ignoring it:
Sending tags unconditionally would therefore break every query of any metrics view that sets
query_attributeson such a workspace. So the first rejection is detected, the query retried without tags, and the outcome latched on the connection:The retry is safe even though
Execroutes throughQuery:CONFIG_NOT_AVAILABLEis raised while setting up the statement, before it executes, so the first attempt has no effect.I couldn't find a way to detect support up front —
SET -von my warehouse lists only 8 settable configs and doesn't mentionquery_tags, yet statement-level tagging works there, so the config listing isn't a usable signal. An up-front probe would also add a round-trip for every workspace, including the supported ones that currently pay nothing.Dependency bump
databricks-sql-gov1.10.0 → v1.14.0.Per-statement query tags (
driverctx.NewContextWithQueryTags) first shipped in v1.11.0 — note the Databricks docs claim Go connector v1.9.0+, butquery_tags.godoesn't exist before v1.11.0. v1.14.0 is the newest release before v1.15.0 began requiring thedatabricks-sql-kernel-bindingsmodules, which add ~585MB of prebuilt platform binaries to the module graph (not reachable from any package we build, but still fetched bygo mod download/CI). Happy to go to v1.15.x instead if you'd prefer to be on latest — it builds and passes tests, it just also pulls in thrift 0.23 and ninegolang.org/x/*upgrades.Verification
Against a live SQL warehouse, a
Query()withQueryAttributesset is recorded in query history as:Also exercised through a real
rill startwithquery_attributeson a Databricks-backed metrics view. And with the driver locally patched to send a configuration the workspace rejects, queries still succeed via the untagged retry (and fail without it).Open questions
query_attributesare unsupported on the workspace?Checklist: