Skip to content

feat: support env-oidc and file-oidc auth types for workload identity federation - #1666

Open
Gerrit-K wants to merge 3 commits into
databricks:mainfrom
Gerrit-K:env-file-oidc-support
Open

feat: support env-oidc and file-oidc auth types for workload identity federation#1666
Gerrit-K wants to merge 3 commits into
databricks:mainfrom
Gerrit-K:env-file-oidc-support

Conversation

@Gerrit-K

@Gerrit-K Gerrit-K commented Sep 7, 2026

Copy link
Copy Markdown

Description

Adds auth_type: env-oidc and auth_type: file-oidc, so the adapter can authenticate through Databricks workload identity federation instead of a stored secret. Both are existing credential strategies in databricks-sdk; the adapter previously accepted only oauth when no token was set, so a federated identity could not be configured at all.

file-oidc needs the path of the file holding the OIDC ID token. A new oidc_token_filepath profile config supplies it, and it is forwarded to the SDK only when set, so DATABRICKS_OIDC_TOKEN_FILE keeps working when the profile omits it.

client_id is required for both new auth types. Without it the credential manager would hand the SDK its dbt-databricks public client, which carries no federation policy, so dbt parse would pass and the first connection would fail with an opaque 401. validate_creds now rejects that combination up front.

Example profile:

my_project:
  target: prod
  outputs:
    prod:
      type: databricks
      host: my-workspace.cloud.databricks.com
      http_path: /sql/1.0/warehouses/abc123
      catalog: main
      schema: default
      auth_type: env-oidc
      client_id: "{{ env_var('DATABRICKS_CLIENT_ID') }}"

The use_kernel=True connector backend has no OIDC flow, so the two new auth types are rejected there with the existing kernel error. That check reads the raw auth_type and short-circuits before creds_manager.config resolves, so preparing connection arguments never triggers a token exchange.

Both strategies exist in databricks-sdk 0.68.0, the current lower bound, so no dependency change is needed.

Verification

Run in a CI pipeline against a live Unity Catalog workspace with a real GitLab-to-Databricks federation policy, using this branch in place of the released adapter:

  • dbt debug with auth_type: env-oidc and the ID token in DATABRICKS_OIDC_TOKEN: Connection test: OK connection ok.
  • dbt debug with auth_type: file-oidc and oidc_token_filepath pointing at the token file: Connection test: OK connection ok.
  • dbt compile over a ~1000 model project on the same env-oidc target, which runs introspective queries, completed without an auth error.

Environment: databricks-sdk 0.77.0, databricks-sql-connector 4.1.5, dbt 1.11.6. Before this branch, the same pipeline had to pre-mint a PAT with the SDK and pass it to dbt through the token: field, because dbt could not consume the federated identity directly.

Scope note

The accepted set is ("oauth", "env-oidc", "file-oidc"). Other SDK strategies such as github-oidc and azure-devops-oidc are still rejected, and token_audience is not exposed as a profile config. Happy to widen either in this PR if you prefer.

Checklist

  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.
  • [Optional] I have run the dbt-databricks-pr-ready project skill for this PR and addressed its merge-readiness feedback

… federation

Add explicit auth_type dispatch for env-oidc/file-oidc so profiles can
authenticate via workload identity federation instead of pre-minting a PAT.
Also reject both auth types up front in the kernel connection path, since
the SEA kernel backend has no OIDC flow and the existing fallback check
would otherwise trigger a real SDK auth attempt before raising its error.
Without an explicit client_id the credential manager fell back to the
`dbt-databricks` public client, which carries no federation policy, so
`dbt parse` passed and the first connection failed with an opaque 401.
Reject that combination in `validate_creds` instead.

Add an `oidc_token_filepath` profile config so the `file-oidc` token path
can come from `profiles.yml` rather than only `DATABRICKS_OIDC_TOKEN_FILE`.

Fold the kernel's OIDC rejection into the existing OAuth M2M guard, which
keeps the short-circuit that stops `.config` from triggering a token
exchange during argument preparation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant