Fix DatabricksSqlHook sqlalchemy_url missing http_path from connection extra

[Vamsi-klu]

Problem

Databricks connection with http_path in extra JSON (common UI path) works for get_conn() but fails for sqlalchemy_url / get_uri() because latter only reads self._http_path (init arg), ignoring extra_dejson["http_path"] and sql_endpoint_name resolution.

Root cause in providers/databricks/src/airflow/providers/databricks/hooks/databricks_sql.py:

• get_conn correctly resolves explicit arg, sql_endpoint_name API, extra_dejson
• sqlalchemy_url buggy: only uses self._http_path

Existing tests only supply explicit http_path, no extra path test.

What I did

• Added helper _resolve_http_path(allow_endpoint_lookup bool) that:
  • returns cached _http_path if set (explicit wins)
  • checks extra_dejson["http_path"] and caches
  • if allow_endpoint_lookup and sql_endpoint_name, calls API
• Updated get_conn to use helper with allow_endpoint_lookup=True
• Updated sqlalchemy_url to use helper with allow_endpoint_lookup=False (offline-safe, no REST call) per reviewer comment from draft PR Fix DatabricksSqlHook sqlalchemy_url to include http_path from connection extra #69037
• Preserves explicit wins logic, avoids unexpected API call in property

Impact

• Fixes SQLAlchemy usage when connection defined via UI extra (common case)
• get_uri() auto-fixed via sqlalchemy_url
• If only sql_endpoint_name set without get_conn first, property returns None (old behavior, acceptable) - user should call get_conn first or use explicit http_path
• No breaking change, offline-safe

Testing

• uv run --project providers-databricks pytest providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py -k sqlalchemy_url → passed
• Full file → 58 passed
• Manual:
  • Extra http_path now present in sqlalchemy_url (was missing)
  • Explicit wins over extra
  • Endpoint-only stays offline-safe (no API call)

Fixes: #69031

Was generative AI tooling used to co-author this PR?

• Yes — Muse Spark 1.1
  Generated-by: Muse Spark 1.1

[Vamsi-klu]

Reviewers: @potiuk @kaxil (providers) + databricks maintainers. Fixes http_path not propagated to sqlalchemy_url.

Fixes #69031

---

Drafted-by: Muse Spark 1.1; reviewed by @Vamsi-klu before posting

[Vamsi-klu]

Tested this PR branch locally in the Airflow UI with Breeze.

Scenario exercised:

• Created a databricks connection named pr_69747_databricks_extra_http_path from the UI.
• Used host dbc-pr69747.cloud.databricks.com and a dummy token only.
• Put http_path only in Extra JSON: sql/protocolv1/o/1234567890123456/0123-456789-pr69747.
• Triggered a scratch Dag that calls DatabricksSqlHook(databricks_conn_id="pr_69747_databricks_extra_http_path").get_uri().
• No real Databricks credentials or network call were required for this check.

Result:

• Dag run succeeded.
• Task log shows the SQLAlchemy URI includes the encoded http_path from connection extra:

databricks://token:***@dbc-pr69747.cloud.databricks.com?http_path=sql%2Fprotocolv1%2Fo%2F1234567890123456%2F0123-456789-pr69747

Screenshots are hosted on a dedicated assets-only branch in the fork.

Connection Extra JSON configured

Connection saved and filtered in the UI

Scratch Dag run succeeded

Task log showing http_path in the URI

---

Drafted-by: Codex (GPT-5.5)