Skip to content

[CNXT-2446][AIR] support RTX6000 accelerators - #7214

Merged
apeforest merged 6 commits into
mainfrom
apeforest/air-rtx6000-support
Oct 28, 2026
Merged

apeforest merged 6 commits into
mainfrom
apeforest/air-rtx6000-support

Conversation

@apeforest

@apeforest apeforest commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Add GPU_1xRTX6000 to AI Runtime commands with one GPU per node and the RTX6000 display label. Compute help lists RTX6000 only when the workspace API confirms availability; the built-in fallback omits it.

Why

Allow RTX6000 configurations to pass CLI validation and reach the Jobs submission API. Workspace availability remains controlled by the backend.

Tracking: CNXT-2446.

Tests

  • All cmd/air unit tests pass, covering parsing, GPU counts, YAML loading, Jobs payload serialization, display labels, log node counts, and RTX6000 help visibility for successful and failed workspace lookups.
  • air/run-submit and air/config-help acceptance tests pass against the local test server, including an RTX6000 submission with AI Runtime 6.
  • Scoped golangci-lint, formatting, whitespace, and changelog validation pass.

This PR was created with GitHub MCP.

@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 28, 2026
@apeforest apeforest changed the title air: support RTX6000 accelerators air: support RTX6000 accelerators (CNXT-2446) Oct 28, 2026
@apeforest apeforest changed the title air: support RTX6000 accelerators (CNXT-2446) [CNXT-2446][AIR] support RTX6000 accelerators Oct 28, 2026
@apeforest
apeforest marked this pull request as ready for review October 28, 2026 06:14
@apeforest
apeforest requested review from a team as code owners October 28, 2026 06:14
@apeforest apeforest self-assigned this Oct 28, 2026
@apeforest
apeforest requested review from caroline-db, maggiewang-db and rclarey and removed request for rclarey October 28, 2026 06:14
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 862f74283 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

2 tests: regression=1 unreleased=1 coverage=0 failing=0

test branch main (8b4a341) latest (v1.22.1)
TestAccept/air/config-help ✅ ✅ ❌
TestAccept/air/run-submit ✅ ❌ ❌
TestAccept/air/config-help: branch ✅ | main (8b4a341) ✅ | latest (v1.22.1) ❌

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- air/config-help/output.txt
        +++ /tmp/TestAcceptairconfig-help3967154239/002/output.txt
        @@ -74,11 +74,6 @@
         config.compute
           Which accelerators to run on and how many.
         
        -  Supported accelerator types:
        -    GPU_1xH100 - 1 accelerator per node [Private Preview]
        -    GPU_8xH100 - 8 accelerators per node
        -    GPU_1xRTX6000 - 1 accelerator per node
        -
           Fields:
             num_accelerators  Total number of GPUs to allocate.
             accelerator_type  Which accelerator to run on, e.g. GPU_1xA10.
        
FAIL
TestAccept/air/run-submit: branch ✅ | main (8b4a341) ❌ | latest (v1.22.1) ❌

main (8b4a341):

acceptance_test.go:1341: Diff:
        --- air/run-submit/output.txt
        +++ /tmp/TestAcceptairrun-submit2922619039/002/output.txt
        @@ -156,49 +156,9 @@
         
         === submit an RTX6000 workload with AI Runtime 6
         >>> [CLI] air run -f rtx6000.yaml
        -Submitting experiment: rtx6000-submit-smoke
        -Submitted workload with Job Run ID: 555
        -View job run at: [DATABRICKS_URL]/jobs/runs/555
        +Error: compute.accelerator_type: invalid GPU type "GPU_1xRTX6000": valid types are: GPU_1xA10, GPU_1xH100, GPU_8xH100, GPU_8xB300
         
        -Tip: use --watch when submitting a run to stream logs to your terminal.
        -Stream logs after submission using:
        -  databricks air logs 555
        +Tip: use the --debug flag to see more details and a trace of this error:
        +  databricks --debug air run …
         
        -=== the RTX6000 accelerator and runtime reach the submit payload
        ->>> print_requests.py //api/2.2/jobs/runs/submit
        -{
        -  "method": "POST",
        -  "path": "/api/2.2/jobs/runs/submit",
        -  "body": {
        -    "environments": [
        -      {
        -        "environment_key": "default",
        -        "spec": {
        -          "environment_version": "6"
        -        }
        -      }
        -    ],
        -    "idempotency_token": "[UUID]",
        -    "run_name": "rtx6000-submit-smoke",
        -    "tasks": [
        -      {
        -        "ai_runtime_task": {
        -          "deployments": [
        -            {
        -              "command_path": "/Workspace/Users/[USERNAME]/.air/cli_launch/rtx6000-submit-smoke/rtx6000-submit-smoke_[RUN_ID]/command.sh",
        -              "compute": {
        -                "accelerator_count": 1,
        -                "accelerator_type": "GPU_1xRTX6000"
        -              }
        -            }
        -          ],
        -          "experiment": "rtx6000-submit-smoke"
        -        },
        -        "environment_key": "default",
        -        "max_retries": 0,
        -        "run_if": "ALL_SUCCESS",
        -        "task_key": "rtx6000-submit-smoke"
        -      }
        -    ]
        -  }
        -}
        +Exit code: 1
        
    acceptance_test.go:1131: Test produced unexpected files:
        .git/COMMIT_EDITMSG
        .git/HEAD
        .git/config
        .git/description
        .git/hooks/applypatch-msg.sample
        .git/hooks/commit-msg.sample
        .git/hooks/fsmonitor-watchman.sample
        .git/hooks/post-update.sample
        .git/hooks/pre-applypatch.sample
        .git/hooks/pre-commit.sample
        .git/hooks/pre-merge-commit.sample
        .git/hooks/pre-push.sample
        .git/hooks/pre-rebase.sample
        .git/hooks/pre-receive.sample
        .git/hooks/prepare-commit-msg.sample
        .git/hooks/push-to-checkout.sample
        .git/hooks/sendemail-validate.sample
        .git/hooks/update.sample
        .git/index
        .git/info/exclude
        .git/logs/HEAD
        .git/logs/refs/heads/main
        .git/objects/0a/c6cca1474cc854f56a1da35545839207acc21b
        .git/objects/4a/1c612d60070bd110834804bfebc5808dd57818
        .git/objects/62/adcc2dfeb9923aa8cecbcd326f9780e1a68440
        .git/refs/heads/main
FAIL

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- air/run-submit/output.txt
        +++ /tmp/TestAcceptairrun-submit804828126/002/output.txt
        @@ -156,49 +156,9 @@
         
         === submit an RTX6000 workload with AI Runtime 6
         >>> [CLI] air run -f rtx6000.yaml
        -Submitting experiment: rtx6000-submit-smoke
        -Submitted workload with Job Run ID: 555
        -View job run at: [DATABRICKS_URL]/jobs/runs/555
        +Error: compute.accelerator_type: invalid GPU type "GPU_1xRTX6000": valid types are: GPU_1xA10, GPU_1xH100, GPU_8xH100, GPU_8xB300
         
        -Tip: use --watch when submitting a run to stream logs to your terminal.
        -Stream logs after submission using:
        -  databricks air logs 555
        +Tip: use the --debug flag to see more details and a trace of this error:
        +  databricks --debug air run …
         
        -=== the RTX6000 accelerator and runtime reach the submit payload
        ->>> print_requests.py //api/2.2/jobs/runs/submit
        -{
        -  "method": "POST",
        -  "path": "/api/2.2/jobs/runs/submit",
        -  "body": {
        -    "environments": [
        -      {
        -        "environment_key": "default",
        -        "spec": {
        -          "environment_version": "6"
        -        }
        -      }
        -    ],
        -    "idempotency_token": "[UUID]",
        -    "run_name": "rtx6000-submit-smoke",
        -    "tasks": [
        -      {
        -        "ai_runtime_task": {
        -          "deployments": [
        -            {
        -              "command_path": "/Workspace/Users/[USERNAME]/.air/cli_launch/rtx6000-submit-smoke/rtx6000-submit-smoke_[RUN_ID]/command.sh",
        -              "compute": {
        -                "accelerator_count": 1,
        -                "accelerator_type": "GPU_1xRTX6000"
        -              }
        -            }
        -          ],
        -          "experiment": "rtx6000-submit-smoke"
        -        },
        -        "environment_key": "default",
        -        "max_retries": 0,
        -        "run_if": "ALL_SUCCESS",
        -        "task_key": "rtx6000-submit-smoke"
        -      }
        -    ]
        -  }
        -}
        +Exit code: 1
        
    acceptance_test.go:1131: Test produced unexpected files:
        .git/COMMIT_EDITMSG
        .git/HEAD
        .git/config
        .git/description
        .git/hooks/applypatch-msg.sample
        .git/hooks/commit-msg.sample
        .git/hooks/fsmonitor-watchman.sample
        .git/hooks/post-update.sample
        .git/hooks/pre-applypatch.sample
        .git/hooks/pre-commit.sample
        .git/hooks/pre-merge-commit.sample
        .git/hooks/pre-push.sample
        .git/hooks/pre-rebase.sample
        .git/hooks/pre-receive.sample
        .git/hooks/prepare-commit-msg.sample
        .git/hooks/push-to-checkout.sample
        .git/hooks/sendemail-validate.sample
        .git/hooks/update.sample
        .git/index
        .git/info/exclude
        .git/logs/HEAD
        .git/logs/refs/heads/main
        .git/objects/0a/c6cca1474cc854f56a1da35545839207acc21b
        .git/objects/4a/1c612d60070bd110834804bfebc5808dd57818
        .git/objects/62/adcc2dfeb9923aa8cecbcd326f9780e1a68440
        .git/refs/heads/main

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 862f742

Run: 40723052029

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 17 5:18
✅​ aws-windows-latest-4core-16gb 283 15 3:41
✅​ azure linux-2core-8gb 280 17 5:15
✅​ azure-windows-latest-4core-16gb 282 15 5:40
✅​ gcp linux-2core-8gb 281 17 5:32
✅​ gcp-windows-latest-4core-16gb 283 15 4:08
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:03 gcp-windows-latest-4core-16gb TestAccept
3:47 gcp linux-2core-8gb TestAccept
3:45 azure linux-2core-8gb TestAccept
3:39 aws linux-2core-8gb TestAccept
3:35 aws-windows-latest-4core-16gb TestAccept
3:14 azure-windows-latest-4core-16gb TestAccept

@caroline-db caroline-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving with one changelog nit - this is correct with CLI's current accelerator-type structure. note that we plan to make accelerator discovery and validation fully backend defined soon, which will remove the need to per-accelerator updates in the future.

Comment thread .nextchanges/air/rtx6000.md Resolved
apeforest and others added 2 commits October 28, 2026 08:48
@apeforest
apeforest enabled auto-merge October 28, 2026 15:49
@apeforest
apeforest added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit 50f8198 Oct 28, 2026
28 checks passed
@apeforest
apeforest deleted the apeforest/air-rtx6000-support branch October 28, 2026 16:56
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 50f8198

Run: 40730985889

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 1499 1227 167:23
✅​ aws-windows-latest-4core-16gb 1424 1252 173:43
✅​ azure linux-2core-8gb 1346 1279 111:09
🔄​ azure-windows-latest-4core-16gb 3 1268 1304 132:59
✅​ gcp linux-2core-8gb 1333 1284 121:14
✅​ gcp-windows-latest-4core-16gb 1258 1309 138:46
Test Name azure-windows-latest-4core-16gb
🔄​ TestAccept 🔄​f
🔄​ TestAccept/bundle/deploy/immutable/basic 🔄​f
🔄​ TestAccept/bundle/deploy/immutable/basic/DMS= 🔄​f
Top 50 slowest tests (at least 2 minutes):
duration env testname
23:48 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
20:18 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
15:37 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
15:08 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
14:38 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
14:34 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
13:54 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
12:54 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
12:53 aws linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
12:38 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=
12:12 aws linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=
11:57 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
11:45 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=
10:50 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
10:46 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
10:12 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
10:07 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
9:51 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
9:17 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:50 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:30 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=true
8:28 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:22 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
8:17 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
7:58 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:45 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=
7:44 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
7:42 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=true
7:41 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
7:41 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=
7:39 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:39 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:34 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:31 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:28 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
7:26 aws linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
7:26 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:24 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
7:23 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:21 aws linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
7:16 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:11 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=
7:09 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:09 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
7:08 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:04 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
7:03 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:01 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
7:00 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
6:59 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true

deco-sdk-tagging Bot added a commit that referenced this pull request Oct 29, 2026
## Release v1.23.0

### CLI

 * Fix GitHub API rate limit errors in `aitools list` and plugin installs by pinning skills to `0.2.32`. ([#7159](#7159))
 * `databricks ssh connect` now makes `claude` and `codex` available on dedicated clusters. ([#7190](#7190))
 * `databricks ssh connect` on serverless now makes `claude` and `codex` available in IDE sessions (`--ide`) and to non-interactive commands (e.g. `databricks ssh connect -- 'claude -p "..."'`). ([#7123](#7123))

### AI Runtime

 * Add workload support for the GPU_1xRTX6000 accelerator type in enabled workspaces. ([#7214](#7214))
 * Show only the accelerator types available in the current workspace in `databricks air run -h config.compute`. ([#6993](#6993))

### Bundles

 * Fixed `bundle validate` failing with "expected a map to index" when a job `schedule`, `trigger` or `continuous` is a reference to a whole block, such as `${resources.jobs.base.schedule}`. ([#7225](#7225))
 * Added `lifecycle.depends_on` to order resources without referencing their fields. ([#7120](#7120))
 * Immutable-folder deploys now refuse to deploy onto a snapshot that was modified outside the bundle (break glass); `bundle deploy --force` moves the deployment to a fresh snapshot. ([#6774](#6774))
 * Fix `--include-locations` failing when a resource is missing a required field. ([#7185](#7185))
 * Removed spurious "required field is not set" warnings for a field whose value is a reference to a whole block, such as `schedule: ${resources.jobs.base.schedule}`. ([#7226](#7226))
 * Revert bundle file uploads to the legacy `/workspace-files/import-file` endpoint, rolling back [#6149](#6149) and [#7034](#7034) released in [v1.21.0](https://github.com/databricks/cli/releases/tag/v1.21.0). This fixes notebooks being uploaded as plain files and deployments failing (for example SQL notebooks in [#7216](#7216)). A remote file or notebook whose type differs from the local one is now deleted and replaced instead of blocking the deploy. ([#7218](#7218))
 * Support selecting resources for deletion with `bundle deploy --select`. ([#7023](#7023))
 * Add the `workspace.sync_mode` setting with the values `overwrite` (default), `in-place`, and `immutable`. It replaces `presets.source_linked_deployment` (use `in-place`) and `experimental.immutable_folder` (use `immutable`), which are now deprecated and emit a warning. ([#7183](#7183))

### Dependency Updates

 * Bump dependencies with known vulnerabilities. ([#7213](#7213))
 * Update the Go SDK to [v0.190.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.190.0). ([#7155](#7155))
 * Update the Go SDK to [v0.191.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.191.0). ([#7211](#7211))
 * Bump Node.js downloaded by the `ssh` agent shim from v24.21.0 to v24.22.2. ([#7246](#7246))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants