Rebase main changes to air-cli - #6101
Closed
vinchenzo-db wants to merge 111 commits into
Closed
Conversation
…#5818) ## Changes `bundle validate` (and therefore `plan`/`deploy`) now reports two UC fields that the SDK models as optional but the backend treats differently: - **`sql_warehouses.*.name`** — error `sql_warehouse name is required` (backend-confirmed as required; whitespace-only names are also rejected, matching the backend's `name.trim.nonEmpty`). - **`grants[*].principal`** — warning `grant principal is required`, on every securable that supports grants (catalogs, schemas, volumes, external_locations, registered_models, vector_search_indexes). Kept as a warning since the backend contract is unconfirmed. Implemented in the existing `validate:required` mutator (runs in `phases.Initialize()`, covering validate/plan/deploy), following the existing dashboard bespoke-validation precedent. Validation diagnostics are sorted deterministically so multiple messages have stable ordering. ## Why These fields are modeled as loosely-typed/optional (`json:"...,omitempty"`), so `bundle validate` and `bundle plan` pass while the deploy is rejected by the backend with late, low-context 400s. Discovered via fuzz testing; affects both the Terraform and direct engines. Reporting the offending field by name at validate time is much more actionable. ## Tests - New acceptance tests under `acceptance/bundle/validate/`: - `sql_warehouse_required_name/` (both engines; covers a missing name and a whitespace-only name) - `grants_required_principal/` (verifies a missing principal warns while a valid grant passes) - Refreshed the `empty_resources` golden files that surface the `sql_warehouse name is required` error.
## Changes Point the aitools skills version at thedatabricks-agent-skills v0.2.10 release, which graduates databricks-data-discovery from experimental/ to stable skills/. --------- Signed-off-by: Lennart Kats <lennart.kats@databricks.com>
## Changes Run the Changelog Preview workflow on every push to main ## Why To allow preview at any state after we fragmented changelog entries into .nextchanges/
Follow-up refinements from the [#5832](#5832) review (all were left as non-blocking there and deferred with a "will address in a follow-up" reply). The `local-env` command remains hidden until #5835, so there is no user-visible changelog entry. ## Changes - **uv install consent** (`libs/localenv/uv.go`) — `EnsureAvailable` no longer runs the remote uv installer (`curl … | sh` / `irm … | iex`) silently. It now requires consent: a truthy `DATABRICKS_LOCALENV_AUTO_INSTALL_UV` opt-in for non-interactive runs (CI/IDE), or an interactive `y/N` prompt via `cmdio.AskYesOrNo`. A non-interactive run without the opt-in returns an actionable error instead of downloading and executing an installer. (The `--debug` log of the exact installer command from #5832 is kept.) - **serverless job version** (`cmd/localenv/compute.go`) — `GetJobSparkVersion` now reads `Environments[0].Spec.EnvironmentVersion`, so a serverless `--job` resolves to its actual `serverless-vN` instead of always defaulting to v4. Empty still falls back to v4. - **double bundle load** (`cmd/localenv/sync.go`) — skip `bundleTarget` when an explicit `--cluster/--serverless/--job` flag is set. `ResolveTarget` only consults the bundle as a fallback, so the second `TryConfigureBundle` load (and its re-printed load-time diagnostics) was wasted for the explicit-flag case. - **robust `Validate` parse** (`libs/localenv/uv.go`) — the `uv run` probe now prints `PYVER:` / `DBCVER:` sentinels and parses by prefix instead of line position, so a stray stdout line from uv doesn't shift the parse. - **cleanup** (`libs/localenv/uv.go`) — fold single-caller `newUvManager` into `NewUvManager`; collapse the triplicated `resolveIndexURL` + conditional `WithEnv` into one `runUv` helper (the conditional stays so an already-set `UV_INDEX_URL` isn't clobbered). ## Tests Adds unit tests for the install consent gate (`TestConfirmUvInstall`) and the sentinel parse (`TestLineWithPrefix`). Full `libs/localenv` suite passes, including under a CI-like environment with `UV_INDEX_URL`/`PIP_INDEX_URL` set. This pull request and its description were written by Isaac.
## Changes Remove `src` folder from immutable folder paths ## Why API won't unpack the snapshot into `src` folder and instead will unpack in the root of snapshot path. ## Tests Covered by acceptance tests <!-- If your PR needs to be included in the release notes for next release, add a changelog fragment: create .nextchanges/<section>/<name>.md with a one-line description (e.g. .nextchanges/cli/quickstart.md). See .nextchanges/README.md. -->
## Summary - The two subtests (`volume_doesnot_exist`, `volume_not_deployed`) already run locally via their own `Local = true` overrides; only the group's parent `test.toml` still defaulted to `Local = false`. - Flip the parent to `Local = true` so the default matches how the tests actually run and no stale cloud-only default remains. - `RecordRequests = false` stays: it intentionally overrides the `RecordRequests = true` set by the artifacts parent `test.toml`.
## Why - The command's user-visible behavior — text and `--json` output, and every error path — needs end-to-end coverage against the real CLI. - `cmd/localenv/` carries no unit tests by design, so acceptance tests are where that surface is verified (repo convention: user-visible CLI output is covered by acceptance tests). ## What - **`acceptance/localenv/`** — 9 scenarios driven through the (hidden) command against the in-process fake server: `help` (three-level tree), `no-target` (`E_NO_TARGET`), `flag-conflict` (Cobra mutual-exclusion), `manager-unsupported` (conda project → clean P1 exit), `env-unsupported` (404 → `E_ENV_UNSUPPORTED` at fetch), `json-error` (`--output json` error object), `serverless-check` (dry-run plan), `serverless-json` (`--json` plan), and `constraints-only`. - Scripts use `local-env python sync` and the `DATABRICKS_LOCALENV_CONSTRAINT_SOURCE` override; goldens show the `local-env python sync` command field and managed marker. No source changes. ## Testing strategy - Goldens generated with `-update` and verified **stable on a clean re-run** (no `-update`); all 9 subtests pass. - `musterr` guards the five expected-failure scenarios; `trace` shows the three output-producing ones. - Full acceptance suite run to confirm no regressions elsewhere (only pre-existing, environment-specific failures unrelated to this change). - Diff confined to `acceptance/localenv/`. - Independently verified by a review subagent (PASS — goldens, scripts, stubs, stale-refs, hygiene) and by codex (no issues). --- ## About this stack This is one of a series of small, stacked PRs that together add the `databricks local-env python sync` command — it provisions a local Python environment (Python version, `databricks-connect` pin, and dependency constraints) matched to a selected Databricks compute target. The work was split from one large branch into single-concern layers so each is independently reviewable; the command is kept hidden until the final PR so nothing is user-visible mid-stack. **Review bottom-up.** Layers 1–5 have merged (the original layer-5 PR #5828 was split into 5a/5b/5c during review). | # | PR | What | Status | |---|----|------|--------| | 1 | #5823 | foundation: result types + env-key mapping | merged | | 2 | #5824 | compute-target resolution | merged | | 3 | #5826 | constraint fetch + offline cache | merged | | 4 | #5827 | formatting-preserving pyproject.toml merge | merged | | 5a | #5850 | package-manager interface + detection | merged | | 5b | #5851 | six-phase pipeline orchestrator | merged | | 5c | #5854 | --check cache purity, greenfield name, dbc insertion | merged | | 6 | #5832 | uv backend + CLI command (registered hidden) | | | 7 | **#5833 ← you are here** | acceptance tests | | | 8 | #5835 | unveil (unhide + help + changelog) | | This pull request and its description were written by Isaac.
## What `acceptance/bundle/deploy/readplan/grants-remove-principal` was failing intermittently on `main` (seen on the `linux, direct` job of the post-merge `build` run for dce783d). The diff was a pure reordering of the two entries in `remote_state.__embed__`: ``` - "principal": "[USERNAME]", + "principal": "extra@example.test", ... - "principal": "extra@example.test", + "principal": "[USERNAME]", ``` ## Why The testserver deliberately randomizes grant-assignment order — it builds the list by iterating a Go map (`libs/testserver/grants.go`), with a comment noting this is intentional because the Azure backend behaves the same way. The test wrote `remote_state.__embed__` straight into the golden without normalizing that order, so the golden matched only when map iteration happened to emit the principals in the checked-in order. This is a flaky test, not a product regression: `macos, direct` passed on the same commit, and the test passes locally on repeated runs. ## Fix Sort `__embed__` by principal in the `jq` step, matching how the sibling `acceptance/bundle/resources/grants/schemas/out_of_band_principal` test handles the same non-deterministic array. Verified with 40 consecutive local runs of both `EnvMatrix` variants. This pull request and its description were written by Isaac.
) ## Why The code in `libs/sync` conflated **listing** the files selected for sync (git-aware + include/exclude) with **syncing** them (snapshot diff, upload, progress). Callers that only want the list had to reach through the syncer: #5254 faked a partial `Sync` to call the listing method, and `validate:files_to_sync` builds a full `Sync` via `sync.New`, running side effects a validation pass shouldn't (writes `.databricks/.gitignore`, can `MkdirsByPath` the remote path). This extracts the real primitive so listing no longer depends on a syncer. ## What - Add `FileList` (`libs/sync/filelist.go`): owns the filesets and the `(git ∪ include) − exclude` merge. `NewFileList(...)` builds it, `Files(ctx)` lists. - `Sync` composes a `*FileList`; `(*Sync).GetFileList` delegates to it. - Remove the free `GetFileList`; its one caller (snapshot zip builder) builds a `FileList` directly. - Drop the implicit `["."]` default: empty paths now select nothing, matching the default engine. Every real caller already passes paths explicitly. ## Behavior Pure refactor, except an immutable-folder deploy with `sync: {paths: []}` now zips zero files instead of the whole root, aligning it with the default engine. This pull request and its description were written by Isaac.
…5948) ## Why Reading `bundle validate`'s remote-path handling (#5528) surfaced that we have no read-only way to tell a user, early, that they likely can't write to their `workspace.file_path`. Today that only fails at deploy. Reasoning about folder permissions was also split: `folder_permissions.go` owned the ancestor-walk traversal while `bundle/permissions` owned the ACL model. This lands one place for it and the reusable core for a follow-up writability warning. ## What - Consolidate the traversal and ACL model into `bundle/permissions` as `FolderACL` (walks up to the closest existing ancestor for a not-yet-created folder). - `CanWrite(user)`: CAN_EDIT+ directly or via a known group; conservative for admins and unknown group membership. - `CheckWritable(...)`: three-valued (writable / not-writable / unknown), since reading a folder ACL itself requires manage access, so "cannot write" usually surfaces as a 403 on the read. Groundwork only: no validator is wired up yet (the consuming validator is a follow-up, alongside #5528), and existing behavior is unchanged. `ValidateFolderPermissions` now calls `ResolveFolderACL` but produces the same diagnostics. Confined to `bundle/permissions`. This pull request and its description were written by Isaac.
## Changes The fake pipelines handler decoded the create/update body into `pipelines.PipelineSpec`, which has no `parameters` field — it lives on `CreatePipeline`/`EditPipeline` and is echoed back on `GetPipelineResponse.Parameters`. Decode that request-only field separately and mirror it on the stored `GetPipelineResponse`, matching backend behavior. ## Why Because the test server dropped `parameters` on read-back, a re-read never returned it and the direct engine planned a perpetual update for any pipeline that set `parameters`. This gap was found by fuzz testing. ## Tests Adds an acceptance test that deploys a pipeline with `parameters`, asserts `bundle plan` is a no-op, and confirms a redeploy issues no update call. Verified to fail without the fix.
This pull request and its description were written by Isaac.
Prep for new test envs that are all UC-enabled. The UC-gated `bundle/invariant` test was skipped on GCP only because the old GCP env had no metastore; once every env is UC-enabled it would run there and its Lakebase v1 templates fail with "Lakebase is not enabled" (400), since Lakebase v1 isn't on GCP. Exclude `database_instance`/`database_catalog`/`synced_database_table` (Lakebase v1) from all cloud runs via `CONFIG_Cloud=true`, matching the `postgres_*` block (Lakebase v2). Follow-up will add per-cloud (AWS+Azure) enablement across all Lakebase v1/v2 resources. This pull request and its description were written by Isaac.
…TORE_NAME] golden (#5956) Adds a GCP storage-URL normalization repl to `grants/volumes/test.toml` (mirroring the existing Azure one) so the `s3://[METASTORE_NAME]` golden holds on the new GCP UC test env; without it, the raw `gs://` location fails the `Cloud = true` test. No-op for AWS/local/Azure. This pull request and its description were written by Isaac.
## Why Several `bundle/generate` acceptance tests create directories on the **real workspace** (they have `Cloud = true`) and never reliably remove them, so leftovers accumulate on shared test workspaces (e.g. `azure-prod-ucws-is`), bloating `workspace list /`. This was observed contributing to a workspace-listing integration test hanging. Two distinct leaks: 1. **`genie_space`, `dashboard`, `alert`** each ran `workspace mkdirs /Workspace/test-$UNIQUE_NAME` at the **workspace root** and used it as the `parent_path` for the resource they create. Nothing deleted it: no `script.cleanup`, and the `eng-dev-ecosystem` env cleaner only sweeps `/Users`, never the root. The alert test also frequently times out (`TimeoutCloud = "5m"`, #4221). 2. **`auto-bind`** deployed its bundle to `/Workspace/tmp/$UNIQUE_NAME`. Its `bundle destroy` removes that on the happy path, but the inline `trap` only cleaned the `/Users/.../python-*` notebook dirs — so a failure before `destroy` leaked the root into the shared, unswept `/Workspace/tmp` namespace. ## What Every generate test that creates workspace state now cleans up after itself, in the swept user tree: - Relocated the three root dirs to `/Workspace/Users/$CURRENT_USER_NAME/test-$UNIQUE_NAME`, and added a `script.cleanup` to each that recursively deletes it (removing the genie space / dashboard / alert created inside). - Moved `auto-bind`'s bundle root to `~/.bundle/auto-bind-test-$UNIQUE_NAME`, matching the `~/.bundle` convention every other bundle acceptance test uses. Output goldens regenerated with `-update`; the full `bundle/generate` suite passes locally. This pull request and its description were written by Isaac, an AI coding agent.
…thon-files flag (#5892) ## Changes Re-applies #5799 (reverted in #5837) and puts the behaviour behind a new opt-in flag. `bundle generate job` only downloaded notebook tasks; files referenced by `spark_python_task` were left as absolute `/Workspace/...` paths, so the source file was never downloaded and the config wasn't portable. This PR restores the download+rewrite path (reusing the same `markFileForDownload` helper as pipeline libraries) but gates it behind a new `--download-spark-python-files` flag on `bundle generate job`, defaulting to **off**. Git-sourced files (`source: GIT`) and cloud URIs (`dbfs:/`, `s3:/`, `adls:/`, `gcs:/`) are left untouched. The flag is threaded through the `Downloader` as a functional option (`WithSparkPythonFiles`), so the pipeline, app, and import callers are unaffected. ## Why #5799 was reverted in #5837 because a Python file often imports sibling files that the downloader does not capture, so downloading only the entry point can produce a job that fails at runtime with missing imports — whereas leaving the absolute workspace path alone "just works". Making the download opt-in keeps the default safe while letting users who know their `spark_python_task` is self-contained pull the file into their bundle. ## Commits 1. **Re-apply #5799** — a single re-revert of #5837, restoring the original change verbatim (only the `NEXT_CHANGELOG.md` conflict resolved). 2. **Gate behind `--download-spark-python-files`** — the new flag, functional-option plumbing, tests, and regenerated help/changelog. ## Tests - Unit tests in `bundle/generate/downloader_test.go`: the download+rewrite path (with the option), the skipped cases (cloud URI, `source: GIT`), and a new test asserting the default-off behaviour makes no requests. - Acceptance test `acceptance/bundle/generate/spark_python_task_job` exercising the full CLI with the flag: a workspace-file task is downloaded and rewritten, a `dbfs:/` cloud-URI task is preserved. Identical output on both `terraform` and `direct` engines. - Regenerated the `bundle generate job --help` golden file. This pull request and its description were written by Isaac, an AI coding agent.
## Changes - Flip to `Local = true` / `Cloud = false`. The bare `ssh connect` (serverless CPU) now runs against the in-process testserver, whose driver-proxy `/ssh` websocket is backed by a real `sshd` — so the test asserts both the submitted bootstrap job and a full handshake + remote exec over the tunnel. - Linux-only (`[GOOS]`), skipped when `sshd` is absent, matching the sibling SSH tests. - Add a `Repls` for `databricks-cpu-<hash>`: with no `--name`, the connection name is derived from a hash of the workspace host (a random localhost port locally), so it's normalized to keep the submitted job payload deterministic. - Keep the `CLOUD_ENV` branch so the real serverless-CPU path is preserved for when cloud `ssh connect` is re-enabled. ## Why `Cloud = false` reflects that cloud `ssh connect` is currently disabled (#4838); the real coverage now comes from the local run. Consistent with `connect-serverless-gpu` and `connection`.
…te (#5958) On GCS-backed UC Volumes, a directory created implicitly via CreateParentDirectories has no standalone object, so it disappears when its last child file is deleted. `recursiveDelete` deletes files first and directories after, so on GCS the directory-delete returns 404 and fails the whole operation even though its goal (directory and contents gone) is already met. This surfaced as `databricks fs rm -r` failing on GCS volumes. Map that 404 to a not-found error via `apierr.IsMissing` and skip not-found directories in `recursiveDelete`. Correct on every backend: on AWS/Azure the directory-delete still returns 204, so the tolerant branch is never taken. Cross-verified on live test environments: `TestFilerReadWrite/files` and `TestFilerRecursiveDelete/files` now pass on GCP (previously failed) and stay green on AWS. This pull request and its description were written by Isaac. --------- Co-authored-by: Jan N Rose <janniklas.rose@gmail.com>
…correct package (#5959) ## What Renames the local-environment command from `databricks local-env python sync` to **`databricks environments setup-local`** and moves it into the existing `environments` command group, per the updated `[P0] CLI Changes` spec. The `environments` group intentionally spans both server-side environment-resource APIs and local provisioning, so a local-install verb belongs there rather than in a standalone top-level group. ## Why here (package placement) `cmd/workspace/environments/environments.go` is generated (`DO NOT EDIT`), so the command is attached the same way `cmd/apps` extends the generated `apps` group: - **`cmd/environments/`** (hand-written, moved from `cmd/localenv/`) exposes a `Commands()` function returning the `setup-local` verb. - **`cmd/workspace/environments/overrides.go`** (new, non-generated) has an `init()` that appends to the generated group's `cmdOverrides` hook, attaching the command. - The standalone `cli.AddCommand(localenv.New())` is dropped from `cmd/cmd.go`. ## Changes - **Command tree:** `local-env python sync` → `environments setup-local`. The `python` subgroup is removed — P0 is Python-only with no language selector (a language axis like `setup-local python` would be additive later; nothing is reserved now). - **Constants:** `CommandGroup`/`CommandVerb`/`CommandName` updated in `libs/localenv/result.go`; JSON `command` field is now `"environments setup-local"`. - **Managed markers:** the `pyproject.toml` managed-block markers now derive from `CommandName` (they previously hard-coded the old name and are written into user files), so the command name lives in exactly one place. - **Still hidden:** the command remains `Hidden` until the environment constraints repository is public — unchanged behavior from before the rename. - Regenerated acceptance goldens + help output. ## Out of scope (deliberate) - **Flag renames** (`--cluster` → `--cluster-id`, `--serverless` → `--serverless-version`, `--job` → `--job-id`, `--check` → `--dry-run`, `--constraint-source` → `--constraint-source-url`) — the spec renames these too, but they are a separate follow-up to keep this PR to the command rename + package move. - The `libs/localenv` package name and `acceptance/localenv/` directory name are left as-is (internal, not user-visible; renaming is cosmetic churn). ## Interaction with the stack #5835 (`[VPEX][8/8]`, held in draft until the constraints repo is public) unhides and documents this command under its **old** name. Whichever lands second must reconcile: the changelog fragment and the `Hidden` flag should reflect `environments setup-local`. ## Testing - `go build ./...`, lint (0 issues), `deadcode` clean - `libs/localenv` + `cmd/environments` unit tests pass - `acceptance/localenv` + `acceptance/help` regenerated and green - Verified `databricks environments setup-local` is runnable, appears under the `environments` group when unhidden, and stays out of help while `Hidden` This pull request and its description were written by Isaac.
## Changes Add a "Summarize failed tests" step that runs `tools/summarize_failed_tests.py` to print a table of failed tests for easy diagnosis. ## Tests - [x] skipped on success: https://github.com/databricks/cli/actions/runs/29590897873/job/87919543119?pr=5966 - [x] prints step summary on failure: https://github.com/databricks/cli/actions/runs/29590049762/job/87916687105?pr=5966#step:6:1 - [x] prints "the failure may be outside the test run" when the job fails without any failing test (a flaky upload step happened to exercise this path): https://github.com/databricks/cli/actions/runs/29590897873/job/87919543141?pr=5966#step:6:1
## Changes Create a new immutable snapshot when bundle permissions change ## Why If the top-level permissions in the bundle change, we can't change the permission of corresponding snapshot because it's immutable. Hence instead we should create a new snapshot when permissions changed. It's done by storing permissions hash as a metadata file in a snapshot zip. ## Tests Added an acceptance tests <!-- If your PR needs to be included in the release notes for next release, add a changelog fragment: create .nextchanges/<section>/<name>.md with a one-line description (e.g. .nextchanges/cli/quickstart.md). See .nextchanges/README.md. -->
… …) (#5960) Stacked on #5959 (the command rename). ## What Renames the `environments setup-local` flags to match the `[P0] CLI Changes` spec: | Old | New | |-----|-----| | `--cluster` | `--cluster-id` | | `--serverless` | `--serverless-version` | | `--job` | `--job-id` | | `--check` | `--dry-run` | | `--constraint-source` | `--constraint-source-url` (still hidden) | ## Changes - Flag definitions, `GetString`/`GetBool` reads, and the mutually-exclusive group in `cmd/environments/sync.go`. - User-facing flag names in `ValidateTargetFlags`, `noTargetMessage`, the `E_ENV_UNSUPPORTED` hint (`constraints.go`), and the job-ambiguity errors (`compute.go`). - Internal `--check` doc comments updated to `--dry-run` for accuracy. - Acceptance scripts + goldens regenerated. No behavior change beyond the flag spellings; the command stays `Hidden`. ## Testing `go build ./...`, lint (0 issues), deadcode clean, `libs/localenv` + `cmd/environments` unit tests, `acceptance/localenv` + `acceptance/help` regenerated and green. This pull request and its description were written by Isaac.
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.21.0 to 0.22.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/sync/commit/1eb64d4bc0cde6da1bb8ebc7f178bb577508e5d0"><code>1eb64d4</code></a> semaphore: panic on negative weights</li> <li>See full diff in <a href="https://github.com/golang/sync/compare/v0.21.0...v0.22.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…dens (#5976) ## Why The Lakebase (postgres) backend now echoes the submitted `spec` on GET; before it returned only `status`. This broke the nightly cloud run: direct-engine plans for postgres resources started showing spec fields in `remote_state` and dropping the phantom `spec:input_only` skip-changes. The `*Remote` structs already anticipate this, so only the fake server and goldens were stale. Verified on a real workspace: a created project's GET returns `spec` with the fields the user set. Recaptured goldens against a real cloud workspace and the full postgres acceptance suite passes locally and on cloud. ## Notes No `resources.yml` change needed: the `spec:input_only` annotations are now redundant but benign (they only suppress phantom drift when config is unchanged) and self-heal once the OpenAPI schema drops `INPUT_ONLY`. This pull request and its description were written by Isaac.
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.46.0 to 0.47.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/sys/commit/9e7e939dcafac07e8ab4cffa6e5fc74908413f00"><code>9e7e939</code></a> cpu: handle vendor suffixes in parseRelease</li> <li><a href="https://github.com/golang/sys/commit/f6fb8a1e42b1731ca8f87ace29c3307429ecdb5a"><code>f6fb8a1</code></a> unix: use epoll_pwait rather than epoll_wait</li> <li><a href="https://github.com/golang/sys/commit/f3eeabfcab6a9a0585ddee7337d5ccfeebe576ed"><code>f3eeabf</code></a> windows: avoid length overflow in NewNTString</li> <li><a href="https://github.com/golang/sys/commit/3cb66475f895724679601e2580be904c8aaa5f7a"><code>3cb6647</code></a> unix: update glibc to 2.43</li> <li><a href="https://github.com/golang/sys/commit/c507910bb52510a2ae06048a4246ad0fe210a872"><code>c507910</code></a> windows: document safe usage of TrusteeValue</li> <li>See full diff in <a href="https://github.com/golang/sys/compare/v0.46.0...v0.47.0">compare view</a></li> </ul> </details> <br /> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Stacked on #5960 (flag renames), which is stacked on #5959 (command rename). ## What Adds `--cluster-name` as a compute target for `environments setup-local`, per the `[P0] CLI Changes` spec. It resolves the cluster **name** to an **ID** via the Clusters API, then resolves identically to `--cluster-id` (`source=cluster`, env key from the resolved cluster's `spark_version`). ## Changes - `ComputeClient` gains `GetClusterByName`; `sdkCompute` implements it via the SDK's `Clusters.GetByClusterName`, which errors on an **unknown** or **ambiguous** name (two clusters sharing it) — both surfaced as an actionable `E_RESOLVE`. - `--cluster-name` joins the mutually-exclusive target group (`--cluster-id`/`--cluster-name`/`--serverless-version`/`--job-id`) and the bundle-fallback guard. - New `ResolveTarget` precedence branch after `--cluster-id`. ## Testing - Unit tests: success (name→ID→env key), ambiguity → `E_RESOLVE`, and `--cluster-id`/`--cluster-name` mutual exclusivity. - Acceptance: `cluster-name-check` (happy path, stubbed `clusters/list`) and `cluster-name-ambiguous` (`E_RESOLVE`); help golden updated. - `go build ./...`, lint (0 issues), deadcode clean, unit + acceptance green. This pull request and its description were written by Isaac.
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.38.0 to 0.39.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/text/commit/b326f3d3c814ab79b3c516f4ac03c2314d8df65f"><code>b326f3d</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="https://github.com/golang/text/commit/5ae8e578e495731553eddba11b2d0e86c91a00ce"><code>5ae8e57</code></a> unicode/norm: avoid infinite loop on invalid input</li> <li><a href="https://github.com/golang/text/commit/0dc94a239c8bc3b7ef3aa0d059e54af960f459f4"><code>0dc94a2</code></a> all: fix some comments</li> <li>See full diff in <a href="https://github.com/golang/text/compare/v0.38.0...v0.39.0">compare view</a></li> </ul> </details> <br /> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Stacked on #5961 (`--cluster-name`) → #5960 → #5959. ## What Per the `[P0] CLI Changes` spec, the serverless stand-in used when the source does not pin a version is now **`serverless-v5`** (was `v4`). Applies only to the fallback cases: - a serverless `--job-id` whose environment records no version, and - a bundle that records `serverless` without a version. Explicitly passing `--serverless-version <vN>` is unaffected. ## Changes - Introduce `defaultServerlessVersion = "v5"` in `envkey.go`, used at both fallback sites in `target.go` so the default lives in one place. - Update the two unit tests that assert the default. No acceptance goldens change: all serverless acceptance tests pass `--serverless-version v4` explicitly, so none exercise the default path. ## Note VS Code resolves the real serverless version itself and passes `--serverless-version` explicitly (spec §63), so this fallback is only hit when the version is genuinely unknown — but v5 is now the correct stand-in per spec. This pull request and its description were written by Isaac.
## Changes Bump the pinned Databricks Terraform provider from v1.121.0 to v1.122.0. Notable schema changes: - New `databricks_postgres_cdf_config` resource. - `databricks_postgres_endpoint` gains `read_only_pooled_host`, `read_write_pooled_host`, and `last_active_time`. - `databricks_postgres_database` `spec.role` is now required. - `databricks_job` gains a `parent_path` field. ## Tests Acceptance goldens regenerated via `./task test-update`. This pull request and its description were written by Isaac.
While reading `bundle/direct/dresources/cluster.go`, the `Clusters.Delete` calls in the `lifecycle.started=false` paths looked like they might *remove* the cluster rather than just terminate it. The naming is easy to misread. This is a comment-only change. Reworded both `Clusters.Delete` call sites to state that Delete terminates the cluster and that permanent removal is a separate API, so the next reader does not hit the same confusion. This pull request and its description were written by Isaac.
…k-key>) (#6048) ## What Replace the whole-job `--job-id <id>` flag with a task-scoped **`--job-task <job-id>.<task-key>`** (the task key is required). ## Why A single job can bind multiple tasks to **different serverless environment versions** (see the [Workflows notebook-environments PRD](https://docs.google.com/document/d/11b0aweSEoRK9ZoKoVnyPdoyWwxcTmvtFPS4d50jcZwE)). So a bare job ID is ambiguous — it can't resolve to one Python/serverless environment. Targeting a specific task removes the ambiguity: the CLI reads *that task's* environment directly. This implements spec task **D5** (`[P0] CLI Changes.md`, `cli-spec-internal.md` §2.2/§2.3/§4.3). ## Behavior - `--job-task <job-id>.<task-key>` resolves the named task. The value splits on the **first** dot, so task keys may themselves contain dots. - **Serverless task** → version read directly from the environment it binds (`environment_key` → `spec` version). The job path **no longer applies the `serverless-vN` default fallback** — that now belongs solely to the bundle path. A task whose environment records no version is `E_RESOLVE`, not a guess. - **Classic task** → resolved from its own `new_cluster` (or `existing_cluster_id`, via the Clusters API). - **Bare `--job-task <job-id>`** (no task key) → `E_USAGE`, listing the job's task keys. Because listing requires a Jobs API call, this `E_USAGE` is emitted at the **resolve** phase; the error-code annotation in `result.go` is updated to note `E_USAGE` can also come from resolve for this case. - **Unknown task key** → `E_RESOLVE`, also listing the available keys. The `ComputeClient` seam gains `JobTaskEnvironment` (replacing `GetJobSparkVersion`) and a typed `ErrTaskKeyRequired`, so `ResolveTarget` can distinguish the missing-key usage error from a genuine resolve failure. ## Tests - `job-classic-check` / `job-serverless-check` now bind a task and resolve it. - The old whole-job ambiguity cases (both-compute, multi-cluster mismatch, serverless-version mismatch) are obsolete under task scoping and are replaced by `job-task-missing-key` (E_USAGE), `job-task-unknown` (E_RESOLVE), and `job-task-unpinned` (E_RESOLVE, no fallback). - `serverless-default-check` removed: the job path no longer defaults; the bundle default remains covered by `TestResolveBundleServerless`. - Unit tests updated for the new interface + split/enumerate semantics. Build, unit, acceptance (`localenv`/`help`), lint, deadcode all pass. Still `Hidden`; part of the pre-unhide hardening of `environments setup-local`. This pull request and its description were written by Isaac.
### Why
A grant with an empty `privileges` list never converges on the
**direct** engine. The backend drops principals that have no privileges,
so after deploy the securable has no assignment for that principal while
the desired state still keeps `{principal, privileges: []}`. Every
subsequent `bundle plan` reports the grant as a perpetual `update`.
Terraform already rejects this input (`privileges` is required), so the
engines disagreed.
### Changes
Extend the existing grant validation (`errorForMissingGrantPrincipals` →
`errorForInvalidGrants`) to also reject an empty/missing `privileges`
list, alongside the missing-principal check. Both `bundle validate` and
`bundle deploy` now fail early with an actionable, located error before
any securable is created, and both engines behave consistently:
```
Error: grant privileges is required
at resources.catalogs.repro_cat.grants[0]
in databricks.yml:12:11
```
### Tests
- Reworked the `resources/grants/schemas/empty_array` acceptance test:
it previously asserted a successful direct-engine deploy of an
empty-privileges grant; it now asserts the validation error (output is
identical across engines, so the per-engine `out.*` snapshots collapse
into one `output.txt`).
- Confirmed against a live Unity Catalog workspace (direct engine): the
bug reproduces on `main`, and the fixed CLI rejects the grant at both
`validate` and `plan`.
…#5984) `TestCountFuzz/aitools/install` panics with `telemetry logger not found`, but only on machines with a supported coding agent installed. Two causes: 1. `aitools` is hand-written but was being fuzzed. It was missing from the fuzz harness `manualRoots` blocklist added in #5102 (the test targets auto-generated commands, guarding against codegen regressions like #5070). 2. `telemetry.Log` panicked when no logger was on the context, which `aitools install` started triggering in #5862 (telemetry for the command). CI never caught it: `install`'s deferred `logInstallEvent` only runs once an agent is detected, and the panic is unreachable via the real CLI anyway (`cmd/root.Execute` always installs the logger; the fuzz harness bypasses it). This pull request and its description were written by Isaac.
…6068) Bumps [golang.org/x/mod](https://github.com/golang/mod) from 0.37.0 to 0.38.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/mod/commit/792ac169a90372d88fb14e712cb793061ba0c104"><code>792ac16</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="https://github.com/golang/mod/commit/fe2ec04f7c3c3aa6fad97631aafcff6e49e2d1db"><code>fe2ec04</code></a> all: fix some comments to improve readability</li> <li>See full diff in <a href="https://github.com/golang/mod/compare/v0.37.0...v0.38.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…0.28.0 (#6073) Bumps [github.com/hashicorp/terraform-json](https://github.com/hashicorp/terraform-json) from 0.27.2 to 0.28.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/hashicorp/terraform-json/releases">github.com/hashicorp/terraform-json's releases</a>.</em></p> <blockquote> <h2>v0.28.0</h2> <p>ENHANCEMENTS:</p> <ul> <li><code>plan</code>: Added "action_reason" data to resource changes described in a plan by <a href="https://github.com/renescheepers"><code>@renescheepers</code></a> in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/200">hashicorp/terraform-json#200</a></li> <li><code>provider schema</code>: Added ability to parse state store schemas from a provider's schema by <a href="https://github.com/SarahFrench"><code>@SarahFrench</code></a> in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/190">hashicorp/terraform-json#190</a> <ul> <li>This is related to an experimental feature that's still in development in Terraform core.</li> </ul> </li> <li><code>validate</code>: Added <code>Diagnostic.Address</code> data to diagnostics returned when validating configuration by <a href="https://github.com/bbasata"><code>@bbasata</code></a> in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/204">hashicorp/terraform-json#204</a></li> </ul> <p>INTERNAL:</p> <ul> <li>test: Add tests for parsing logs from init commands, tests for parsing of RFC3339 timestamps with different levels of precision by <a href="https://github.com/SarahFrench"><code>@SarahFrench</code></a> in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/180">hashicorp/terraform-json#180</a></li> <li>build(deps): bump actions/checkout from 5.0.0 to 6.0.0 in the github-actions-breaking group by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/182">hashicorp/terraform-json#182</a></li> <li>build(deps): bump actions/setup-go from 6.0.0 to 6.1.0 in the github-actions-backward-compatible group by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/183">hashicorp/terraform-json#183</a></li> <li>build(deps): bump github.com/hashicorp/go-version from 1.7.0 to 1.8.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/184">hashicorp/terraform-json#184</a></li> <li>build(deps): bump actions/checkout from 6.0.0 to 6.0.1 in the github-actions-backward-compatible group by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/186">hashicorp/terraform-json#186</a></li> <li>build(deps): bump the github-actions-backward-compatible group across 1 directory with 2 updates by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/189">hashicorp/terraform-json#189</a></li> <li>Update copyright headers to IBM by <a href="https://github.com/radeksimko"><code>@radeksimko</code></a> in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/192">hashicorp/terraform-json#192</a></li> <li>build(deps): bump actions/setup-go from 6.2.0 to 6.3.0 in the github-actions-backward-compatible group by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/194">hashicorp/terraform-json#194</a></li> <li>build(deps): bump actions/setup-go from 6.3.0 to 6.4.0 in the github-actions-backward-compatible group by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/195">hashicorp/terraform-json#195</a></li> <li>build(deps): bump github.com/hashicorp/go-version from 1.8.0 to 1.9.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/196">hashicorp/terraform-json#196</a></li> <li>build(deps): bump actions/checkout from 6.0.2 to 6.0.3 in the github-actions-backward-compatible group by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/199">hashicorp/terraform-json#199</a></li> <li>build(deps): bump actions/checkout from 6.0.3 to 7.0.0 in the github-actions-breaking group by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/201">hashicorp/terraform-json#201</a></li> <li>build(deps): bump actions/setup-go from 6.4.0 to 6.5.0 in the github-actions-backward-compatible group by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/202">hashicorp/terraform-json#202</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/helenjw"><code>@helenjw</code></a> made their first contribution in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/179">hashicorp/terraform-json#179</a></li> <li><a href="https://github.com/renescheepers"><code>@renescheepers</code></a> made their first contribution in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/200">hashicorp/terraform-json#200</a></li> <li><a href="https://github.com/bbasata"><code>@bbasata</code></a> made their first contribution in <a href="https://redirect.github.com/hashicorp/terraform-json/pull/204">hashicorp/terraform-json#204</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/hashicorp/terraform-json/compare/v0.27.2...v0.28.0">https://github.com/hashicorp/terraform-json/compare/v0.27.2...v0.28.0</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/hashicorp/terraform-json/commit/1e0dcb8caaf6c22e21bc6c042cb85b5467b670cf"><code>1e0dcb8</code></a> validate: add <code>Diagnostic.Address</code> (<a href="https://redirect.github.com/hashicorp/terraform-json/issues/204">#204</a>)</li> <li><a href="https://github.com/hashicorp/terraform-json/commit/34ea8d07cdb50ef7f326f3b448c847d83aa6543d"><code>34ea8d0</code></a> build(deps): bump actions/setup-go from 6.4.0 to 6.5.0 in the github-actions-...</li> <li><a href="https://github.com/hashicorp/terraform-json/commit/ff68feb930e6e451efa6a0d3a12814c6e6914b90"><code>ff68feb</code></a> Merge pull request <a href="https://redirect.github.com/hashicorp/terraform-json/issues/201">#201</a> from hashicorp/dependabot/github_actions/github-actio...</li> <li><a href="https://github.com/hashicorp/terraform-json/commit/f3efae01e492b24275fb33ca93c6130ee479c48f"><code>f3efae0</code></a> build(deps): bump actions/checkout in the github-actions-breaking group</li> <li><a href="https://github.com/hashicorp/terraform-json/commit/09661535096ac52ee85bc4d4968c0a629f1c60e8"><code>0966153</code></a> Merge pull request <a href="https://redirect.github.com/hashicorp/terraform-json/issues/200">#200</a> from renescheepers/rs/add-action-reason</li> <li><a href="https://github.com/hashicorp/terraform-json/commit/b4487cdd6112b7d1d0a4e270ec0f953ac9718d57"><code>b4487cd</code></a> Link ActionReason constants to Terraform's canonical enum</li> <li><a href="https://github.com/hashicorp/terraform-json/commit/eefb8b2ce8d8e1de832e1033408b3ccb122f73b5"><code>eefb8b2</code></a> Add ActionReason to ResourceChange</li> <li><a href="https://github.com/hashicorp/terraform-json/commit/5738e0ad057af2f241a1d79cbaeb9e52c451bfb3"><code>5738e0a</code></a> build(deps): bump actions/checkout from 6.0.2 to 6.0.3 in the github-actions-...</li> <li><a href="https://github.com/hashicorp/terraform-json/commit/e77c39ee5a16d93861c6e3e99fd2aea148399e6d"><code>e77c39e</code></a> Merge pull request <a href="https://redirect.github.com/hashicorp/terraform-json/issues/196">#196</a> from hashicorp/dependabot/go_modules/github.com/hashi...</li> <li><a href="https://github.com/hashicorp/terraform-json/commit/703bd74a48e5461b40eeb5b5663f7f58be3fad52"><code>703bd74</code></a> build(deps): bump github.com/hashicorp/go-version from 1.8.0 to 1.9.0</li> <li>Additional commits viewable in <a href="https://github.com/hashicorp/terraform-json/compare/v0.27.2...v0.28.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.39.0 to 0.40.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/text/commit/724af9c35838492dcaacc1ac51a8a0187c994c54"><code>724af9c</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="https://github.com/golang/text/commit/bf5b9d658a03d36bd3459b3babdb23a9f56038cb"><code>bf5b9d6</code></a> internal/export/idna: always treat Punycode encoding pure ASCII as an error</li> <li>See full diff in <a href="https://github.com/golang/text/compare/v0.39.0...v0.40.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/mod](https://github.com/golang/mod) from 0.37.0 to 0.38.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/mod/commit/792ac169a90372d88fb14e712cb793061ba0c104"><code>792ac16</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="https://github.com/golang/mod/commit/fe2ec04f7c3c3aa6fad97631aafcff6e49e2d1db"><code>fe2ec04</code></a> all: fix some comments to improve readability</li> <li>See full diff in <a href="https://github.com/golang/mod/compare/v0.37.0...v0.38.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.56.0 to 0.57.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/net/commit/b8f09f6f062ceb4531b7af4bd17a5c8fe9c4b2b5"><code>b8f09f6</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="https://github.com/golang/net/commit/f05f21be5927155a88b371674c298ada54b71cf5"><code>f05f21b</code></a> idna: reject all-ASCII xn-- labels on all Go versions</li> <li><a href="https://github.com/golang/net/commit/0f748cfbba95084c87723e3b7eb7dad9241fc2dd"><code>0f748cf</code></a> internal/http3: clean up stream I/O methods usages in tests</li> <li><a href="https://github.com/golang/net/commit/0bb961e45132c680e2a0117e1df951538a19da61"><code>0bb961e</code></a> internal/http3: add net/http.ResponseController support</li> <li><a href="https://github.com/golang/net/commit/0ca694d2a5adcce7f36d4d65e5f8d72390d94051"><code>0ca694d</code></a> webdav: document Dir's lack of defense against filesystem modification</li> <li><a href="https://github.com/golang/net/commit/bd5f1dcf71cf0d6d2424021d0a04f191396a46a7"><code>bd5f1dc</code></a> http2: initialize Transport on NewClientConn</li> <li><a href="https://github.com/golang/net/commit/488ff63197ffddbcef2d01107f8a89906a4f28d1"><code>488ff63</code></a> bpf: add security considerations to package docs</li> <li><a href="https://github.com/golang/net/commit/93d1f25be95961edfdf488b9398e53fb9f05f193"><code>93d1f25</code></a> xsrftoken: avoid token collisions</li> <li><a href="https://github.com/golang/net/commit/5a3baee349e65853c34b4377bebe7676a64dbc87"><code>5a3baee</code></a> internal/http3: prevent panic in QPACK decoder due to overflow</li> <li>See full diff in <a href="https://github.com/golang/net/compare/v0.56.0...v0.57.0">compare view</a></li> </ul> </details> <br /> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.53.0 to 0.54.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/crypto/commit/cdce021fa6c7d9c7eb2743bfbe551f0a98fd5d62"><code>cdce021</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="https://github.com/golang/crypto/commit/d9474cc4853d9ef1a29356975408d4771e5770c6"><code>d9474cc</code></a> openpgp: make the deprecation message more explicit</li> <li><a href="https://github.com/golang/crypto/commit/7626c5025624025bb44739a805f431cf93c06d6e"><code>7626c50</code></a> ssh: verify declared key type matches decoded key in authorized_keys</li> <li><a href="https://github.com/golang/crypto/commit/0471e7969e6740594dfe354646bf03e5e89de52d"><code>0471e79</code></a> ssh/agent: enforce strict limits on DSA key parameters</li> <li><a href="https://github.com/golang/crypto/commit/6435c37ab681759aff37ba751d0f2238b3043767"><code>6435c37</code></a> ssh: sanitize client disconnect messages</li> <li><a href="https://github.com/golang/crypto/commit/7d695da948bfa44ed6eedcebc8f43bcb50e94a57"><code>7d695da</code></a> ssh/agent: drain channel stderr in agent forwarders</li> <li><a href="https://github.com/golang/crypto/commit/5b7f84159940519e89df4d95465538c1797cee8b"><code>5b7f841</code></a> acme/autocert: fix data race in Manager.createCert</li> <li><a href="https://github.com/golang/crypto/commit/0b316e7ee409f8e5789a0535679d34155cecc75e"><code>0b316e7</code></a> argon2: update RFC 9106 parameter recommendations</li> <li><a href="https://github.com/golang/crypto/commit/55aec0a86b4c522b4f7366e69db55349e9f7ff5c"><code>55aec0a</code></a> x509roots/fallback: update bundle</li> <li><a href="https://github.com/golang/crypto/commit/5f2de1a9f1e29059fbb9f3d34321bd0da935556b"><code>5f2de1a</code></a> internal: remove wycheproof tests</li> <li>See full diff in <a href="https://github.com/golang/crypto/compare/v0.53.0...v0.54.0">compare view</a></li> </ul> </details> <br /> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Follow-up to #6045, which consulted the new IsGone hook only at plan time. A saved plan (deploy --plan) records Gone=false if the app was still ACTIVE when planned; if it enters DELETING before apply — a concurrent delete, or an interrupted destroy — apply calls DoDelete on the stale entry and the backend rejects the second delete with 400 BAD_REQUEST. On an otherwise-fatal delete error, re-read the resource and consult IsGone (and IsMissing); treat it as already deleted if so. Applies to both Delete and Recreate. Detection re-reads rather than matching the error because the backend returns a generic 400 with no distinct SDK sentinel.
## Changes When a principal is granted `ALL_PRIVILEGES`, `normalizeAssignments` collapses its privilege list down to just `ALL_PRIVILEGES`, on both the config side and the value read back from the backend, so the two compare equal. ## Why Fixes #6030. On the direct engine `bundle plan` reported `update` on every catalog/schema/volume `.grants` sub-resource that granted `ALL_PRIVILEGES`: the backend reports `ALL_PRIVILEGES` plus the concrete privileges it implies, which never matched the config's lone `ALL_PRIVILEGES`, and the deploy never converged. ## Tests New acceptance test `grants/schemas/all_privileges_coexist` (local + cloud, both engines); verified on aws-prod-ucws.
## Changes Move the shared invariant setup into `acceptance/bundle/invariant/script.prepare` and convert all five targets (`no_drift`, `migrate`, `continue_293`, `delete_idempotent`, `destroy_idempotent`) to it. Each carried its own copy of the config render, the destroy-on-exit trap, the deploy + panic-scan + `INPUT_CONFIG_OK` sequence, and the no-drift check; each script now holds only the invariant it asserts. No behavior change — the helpers run the same sequences as before. The five copies were not identical, so sharing one implementation shifts three log filenames, which are reported rather than compared. Groundwork for #5686.
…#6085) ## Why Baseline for aligning the direct engine with terraform on empty optional strings. Setting a field like `policy_id: ""` is dropped by terraform (via omitempty) but sent as `""` by the direct engine, which the backend rejects (400 INVALID_PARAMETER_VALUE). The test deploys one config through each engine and records which "" fields each drops from the create request, at any nesting depth (job_clusters[].new_cluster.*, cluster_log_conf.s3.*, git_source.*, ...). `out.empty_fields_dropped_by_terraform.txt` is the target; the `_terraform_only` file is the current gap (55 of 59 fields). As direct handling changes, the terraform file stays put and the gap shrinks. `gen_empty_config.py` walks the field inventory (out.fields.txt) and fills every settable string leaf under a deployable base skeleton, so coverage stays broad without hand-maintenance.
## What The `fs` command's UC Volumes filer now runs on the `sdk-go/files/v2` client instead of hand-written Files API calls. The filer is shared, so bundle library uploads to Volumes go through the same path. `files/v2` brings its own multipart/resumable upload engine. `DATABRICKS_EXPERIMENTAL_MULTIPART_UPLOAD` gates whether large Volumes uploads are split into parts; it selects multipart-vs-single-shot within `files/v2`. Default behavior (single-shot PUT) is unchanged. `fs cp` also gains an interactive upload progress bar for a single-file copy to Volumes, and sizes the concurrent part-transfer budget for large (multipart) uploads. The CLI takes on four new direct dependencies: `github.com/databricks/sdk-go/files` (`v0.0.0-dev.1`), `.../auth` and `.../options` (`v0.0.0-dev`), and `.../core` bumped to `v0.0.1-dev`. These are `-dev` pre-release versions; a follow-up should move to stable tags once sdk-go cuts them. ## Notable in the diff - **`libs/filer/files_client.go`** is rewritten onto `*files.Client`. Error mapping keys off the HTTP status (via a `httpStatus` helper) rather than the SDK's canonical `codes.Code`, because the Files API reports "path already exists" as 409, which the SDK maps to `codes.Aborted` (not `codes.AlreadyExists`). - **`libs/filer/files_client_auth.go`** adapts the CLI's resolved `config.Config` into the `auth.Credentials` the client expects, signing a request with `config.Authenticate` and reusing those headers instead of re-reading a profile. Its credentials name is the config's resolved auth type (`pat`, `oauth-m2m`, ...), falling back to `unk` when the config has no resolved auth type. - **`cmd/fs/cp.go`, `cmd/fs/helpers.go`, `cmd/fs/upload_progress.go`:** a single-file `fs cp` to Volumes now renders an upload progress bar, fed through a new `filer.WithUploadProgress` context callback. Large-file (multipart) writes use a shared concurrent part-transfer budget (`multipartUploadConcurrency`), plumbed via a new `filer.WithUploadConcurrency` option; other schemes ignore it. - **`NewFilesClient` gains a `ctx` parameter**, threaded through its callers. - **Shared code:** the Volumes filer is used by both `fs` and `bundle/libraries`, so bundle Volumes uploads are affected. The four new `sdk-go` modules are added to `go.mod` and to the `noticeExclude` allowlist in `internal/build/notice_test.go` (they are Databricks-owned, so they need no `NOTICE` entry). - **`libs/testserver`** now models "409 when creating a directory over an existing file," which the fake previously did not; this closes the gap that let a `Mkdir` error-mapping regression through. ## Out of scope - **Local, DBFS, and workspace filers** are unchanged; `files/v2` only backs the UC Volumes path. This pull request and its description were written by Isaac.
## Changes Add a new flag (`--usage-policy-id`) to specify the serverless usage policy for SSH connections ## Why To allow customers to attribute their SSH usage ## Tests Added unit tests
## Why
An optional (omitempty) resource field set to `""` — often via a
variable that resolved to an empty string, e.g. `policy_id: ${var.x}` —
was force-sent to the backend and rejected with `'' is not a valid
cluster policy ID`. Terraform dropped these via omitempty, but the
direct engine sent them, so users got these errors after migration.
Related: #1255
## Changes
A new `convert.DropEmptyStrings` normalize option removes empty-string
values on omitempty fields, applied by a `DropEmptyStrings` mutator in
the initialize phase after variable resolution. Required fields without
omitempty (e.g. `spark_version`) are kept, and `apps.description` is
exempt to match terraform's existing behavior. The result is
engine-agnostic and reflected in `bundle validate -o json`.
Note, some new terraform resources preserve `""`, for example
database_instance.usage_policy_id. However, this would have the same
issue with variables so there is no exception for these and this empty
string will be dropped as well.
…6022) ## Changes The SDK exposes a dedicated permission level type for vector search endpoints; switch `VectorSearchEndpoint.Permissions` to use it via the `PermissionT[L]` generic, matching the typing already in place for jobs, apps, model serving, etc. Regenerated jsonschema.json and added the annotations placeholder for the new VectorSearchEndpointPermission. ## Why Allows DAB-side validation of types in the schema instead of backend erroring on unsupported permissions
Bump dependencies flagged by `govulncheck -scan module` to their fixed versions. Each CVE links to its Go advisory page. - google.golang.org/grpc → v1.82.1 (fixes [GO-2026-6061]) [GO-2026-6061]: https://pkg.go.dev/vuln/GO-2026-6061 Vulnerabilities in the Go standard library are left to the `Bump Go toolchain` workflow. If a bump promotes a new direct dependency, double-check its license annotation in `go.mod` and `NOTICE`. Co-authored-by: github-merge-queue <118344674+github-merge-queue@users.noreply.github.com>
When the workspace `.bundle` directory is at its child-node limit, writing `deploy.lock` fails with a 403 `MAX_CHILD_NODE_SIZE_EXCEEDED`. `bundle destroy --force-lock` now proceeds without a lock in that case so the deployment can still be torn down, which is what frees up capacity. Deploy and non-forced destroy still fail, and a real permission denial still aborts. The filer previously collapsed any 403-on-mkdir to `permissionError`, discarding the `APIError`; it now wraps it so `lock.Acquire` can inspect the `error_code` while the error still matches `fs.ErrPermission`.
## Changes Add `has_base_environment` to SSH telemetry events ## Why To understand usage ## Tests Automated tests added
## Changes Upgrade TF provider to 1.123.0
The TF provider (v1.122.0, #5977) and SDK (v0.160.0, #5982) bumps dropped `code_source_path` from `ai_runtime_task`, and #5982 disabled its use in the `air` command. This re-enables the `CodeSourcePath` assignment and test assertions in `experimental/air/cmd/`.⚠️ Does not build against SDK v0.160.0 (which lacks the field). Merge after the next SDK bump re-adds it; generated files are left to reappear on codegen. This pull request and its description were written by Isaac. --------- Co-authored-by: Andrew Nester <andrew.nester.dev@gmail.com> Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
## Release v1.10.0 ### CLI * `ssh connect` now supports specifying a serverless usage policy with `--usage-policy-id` ### Bundles * Fixed `bundle deploy`/`bundle destroy` failing when an app enters the transient DELETING state between plan and apply (e.g. with a saved plan); the delete is now treated as complete instead of erroring (direct engine only). * Fixed `bundle deploy`/`bundle destroy` failing when an app is still in the transient DELETING state; the delete is now treated as complete instead of erroring (direct engine only). * `bundle destroy --force-lock` now proceeds without a deployment lock when the workspace directory is at its child-node limit and cannot accept the lock file, so a deployment can still be torn down when the workspace is full. * Empty-string values on optional (omitempty) resource fields are now dropped before deployment instead of being sent to the backend. This fixes deploys failing with errors like `'' is not a valid cluster policy ID` when a field such as `policy_id` was set to `""` (often via a variable that resolved to an empty string). The behavior now matches between the terraform and direct engines and is reflected in `bundle validate -o json`. * `bundle validate` and `bundle deploy` now reject a grant that is missing a `principal` with an error instead of a warning. Previously the deploy would start and, on the direct engine, create the securable before the grants PATCH failed (`400 INVALID_PARAMETER_VALUE`), leaving a partially-applied deployment. * `bundle validate` and `bundle deploy` now reject a grant with an empty `privileges` list with an error. Previously, on the direct engine, such a grant never converged: the backend drops principals with no privileges, so every subsequent `bundle plan` reported the grant as a perpetual update. * Fixes [#6030](#6030): spurious `update` on catalog/schema/volume grants (direct engine); a principal granted `ALL_PRIVILEGES` no longer drifts when the backend also reports the concrete privileges it implies ([#6064](#6064)). * Use vector search endpoint permission types that are supported by the backend ([#6022](#6022)). ### Dependency Updates * Bump `github.com/databricks/databricks-sdk-go` from v0.160.0 to v0.165.0. * Upgrade Terraform provider to 1.123.0
## Changes Warn when `bundle.engine: terraform` is set, pointing to the direct engine docs (https://docs.databricks.com/aws/en/dev-tools/bundles/direct). Add `engine_terraform_config` and `engine_terraform_env` deploy telemetry. ## Why The direct engine is now the default and terraform is on a path to removal, but users who explicitly pin it get no signal, and we have no data on how many still opt in. The warning only fires on the config setting (the durable opt-in that needs migrating); telemetry covers both config and the `DATABRICKS_BUNDLE_ENGINE` env var.
Brings the air-cli feature branch current with main. air-cli had drifted 110 commits behind, spanning Grigory Panov's localenv/environments redesign (cmd/localenv renamed to cmd/environments, new JobTaskEnvironment API, --cluster-name/--job-task flags, uv provisioning) and other infra work. Conflict resolution rule: - Infra (libs/localenv, cmd/environments, libs/filer, bundle/config/validate) and their acceptance goldens: take main. air-cli only carried an older snapshot of this shared code (via the #5968 squash); it had no AIR-specific edits there, so main is authoritative. - AIR (experimental/air/**, acceptance/experimental/air/**): keep air-cli. This is Riddhi's AIR CLI work and the reason the branch exists. The per-dir air test.toml deletions are honored (engine matrix centralized in the shared acceptance/experimental/air/test.toml). - libs/localenv/target_test.go aligned to main to match main's target.go API. Verified: go build ./... clean; 123 packages pass across libs/, cmd/, bundle/config/ (0 failures); localenv acceptance green. Co-authored-by: Isaac
Contributor
Approval status: pending
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Why
Tests