denik · Oct 6, 2026 · Oct 5, 2026 · Oct 5, 2026 · Oct 6, 2026
@@ -66,7 +66,7 @@ The whole `Cloud*` family lives inside an `if isRunningOnCloud` branch in `getSk
 
 Exception: mass string replacement when the change is predictable and much cheaper than re-running the test suite.
 
-**RULE: All `EnvMatrix` variants MUST produce identical output files.** Filenames containing `$DATABRICKS_BUNDLE_ENGINE` (e.g. `output.direct.txt`) are the only per-engine exception.
+**RULE: All `EnvMatrix` variants MUST produce identical output files.** The only exception is a test that sets `EnvVaryOutput`, whose files named after the variant value (e.g. `out.deploy.<value>.txt`) are per-variant.
 
 **RULE: Do not run `-update` while a divergent variant exists.** It is destructive: it overwrites with the last variant and breaks the others. To debug: run a single variant you consider correct with `-update`, then debug the other variant to find why it diverges.
 
@@ -95,13 +95,9 @@ acceptance/cmd/fs/cp/file-to-dir/
   output.txt
 ```
 
-**RULE: When output genuinely diverges between engines (terraform vs direct), split only the diverging file into per-engine variants.** Keep the rest of the output unified. Files named `output.$DATABRICKS_BUNDLE_ENGINE.txt` or `out.requests.$DATABRICKS_BUNDLE_ENGINE.json` are the allowed per-engine form.
-
-If the only reason for divergence is a server-side default that one engine sets and the other doesn't, set the field explicitly in `databricks.yml` so both engines produce identical output. Don't paper over it with per-engine files.
-
 **RULE: On Windows, Git Bash auto-converts a leading-`/` path argument (e.g. `/api/2.0/...`) into a Windows path, so `$CLI` sees the wrong path and the testserver 404s.** Set `Env.MSYS_NO_PATHCONV = "1"` in the test directory's `test.toml`. Quoting the argument in bash does NOT help — the conversion is done by the Windows binary's argument processing. Precedent: `acceptance/cmd/workspace/export-dir-*/test.toml`.
 
-**RULE: `EnvMatrix.<VAR> = []` removes that variable from the inherited matrix** (see `ExpandEnvMatrix` in `acceptance/internal/config.go`). The root `test.toml` matrixes `DATABRICKS_BUNDLE_ENGINE = [terraform, direct]`, so a non-bundle test opts out of both engine runs with `EnvMatrix.DATABRICKS_BUNDLE_ENGINE = []`. The `out.test.toml` snapshot of inherited values is generated and committed by design.
+**RULE: `EnvMatrix.<VAR> = []` removes that variable from the inherited matrix** (see `ExpandEnvMatrix` in `acceptance/internal/config.go`). For example, `acceptance/bundle/test.toml` matrixes `DMS = ["", "true"]`, so a bundle test opts out of the DMS run with `EnvMatrix.DMS = []`. The `out.test.toml` snapshot of inherited values is generated and committed by design.
 
 **RULE: Write every map-valued setting in dotted form at the top of `test.toml`, never under a table header.** This covers `Env`, `EnvMatrix`, `EnvMatrixExclude`, `EnvRepl`, `GOOS` and `CloudEnvs` — e.g. `Env.MSYS_NO_PATHCONV = "1"`, `GOOS.windows = false`. In TOML every key after a `[EnvMatrix]` header belongs to that table until the next header — a blank line does not end it. So a top-level key like `Ignore` placed below `[EnvMatrix]` is silently parsed as `EnvMatrix.Ignore` (a bogus matrix variable) instead of the real top-level field, and the test runs with the field unset. Dotted form keeps each key's table explicit and is immune to ordering.
 
@@ -110,15 +106,15 @@ If the only reason for divergence is a server-side default that one engine sets
 GOOD:
 
 ```toml
-EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["direct"]
+EnvMatrix.READPLAN = ["", "1"]
 Ignore = ["databricks.yml"]
 ```
 
 BAD:
 
 ```toml
 [EnvMatrix]
-DATABRICKS_BUNDLE_ENGINE = ["direct"]
+READPLAN = ["", "1"]
 
 Ignore = ["databricks.yml"]   # parsed as EnvMatrix.Ignore, not top-level Ignore
 ```
@@ -132,7 +128,7 @@ Ignore = ["databricks.yml"]   # parsed as EnvMatrix.Ignore, not top-level Ignore
 - Source files: `test.toml`, `script`, `script.prepare`, `databricks.yml`, etc.
 - Tests are configured via `test.toml`. Config schema and explanation is in `acceptance/internal/config.go`. Certain options are also dumped to `out.test.toml` so that inherited values are visible on PRs.
 - Run a single test: `go test ./acceptance -run TestAccept/bundle/<path>/<to>/<folder>`
-- Run a specific variant by appending `EnvMatrix` values to the test name: `go test ./acceptance -run 'TestAccept/.../DATABRICKS_BUNDLE_ENGINE=direct'`. When there are multiple `EnvMatrix` variables, they appear in alphabetical order.
+- Run a specific variant by appending `EnvMatrix` values to the test name: `go test ./acceptance -run 'TestAccept/.../DMS=true'`. When there are multiple `EnvMatrix` variables, they appear in alphabetical order.
 - Useful flags: `-v` for verbose output, `-tail` to follow test output (requires `-v`), `-logrequests` to log all HTTP requests/responses (requires `-v`).
 - Run tests on cloud: `deco env run -i -n aws-prod-ucws -- <go test command>` (requires `deco` tool and access to test env). This is an *additional* pass over the same test directories, restricted to those with `Cloud = true` set; it does not replace the local run.
 - `script.prepare` files from parent directories are concatenated into the test script. Use them for shared bash helpers.
@@ -183,7 +179,7 @@ Available on `PATH` during test execution (from `acceptance/bin/`):
 - `update_file.py FILENAME OLD NEW`: replace all occurrences of OLD with NEW in FILENAME. Errors if OLD is not found. Cannot be used on `output.txt`.
 - `find.py REGEX [--expect N]`: find files matching regex in current directory. `--expect N` asserts an exact count.
 - `diff.py DIR1 DIR2` or `diff.py FILE1 FILE2`: recursive diff with test replacements applied.
-- `print_state.py [-t TARGET] [--backup]`: print deployment state (terraform or direct).
+- `print_state.py [-t TARGET] [--backup]`: print deployment state (`resources.json`, or a terraform state left by an older CLI).
 - `edit_resource.py TYPE ID < script.py`: fetch resource by ID, execute Python on it (resource in `r`), then update it. TYPE is `jobs` or `pipelines`.
 - `gron.py`: flatten JSON into greppable discrete assignments (simpler than `jq` for searching JSON).
 - `jq` is also available for JSON processing.

@@ -187,7 +187,7 @@ func TestInprocessMode(t *testing.T) {
 	// selftest/server too, which meant a second setup after StartDefaultServer had
 	// pointed HOME at an empty temp dir, so building yamlfmt there re-downloaded the
 	// entire module cache: 44s on Linux CI, 140s on Windows. Tool builds are skipped
-	// for the same reason - selftest/basic uses neither terraform, the wheel, nor yamlfmt.
+	// for the same reason - selftest/basic uses neither the wheel nor yamlfmt.
 	require.Equal(t, 1, testAccept(t, true, []string{"selftest/basic"}, true))
 }
 
@@ -205,29 +205,8 @@ func setReplsForTestEnvVars(t *testing.T, repls *testdiff.ReplacementsContext) {
 	}
 }
 
-// helperScriptUsesEngineCache caches whether a _script helper in a given directory
-// (or any of its ancestors) references $DATABRICKS_BUNDLE_ENGINE.
-// Since _script helpers are shared across many tests, caching avoids redundant reads.
-var helperScriptUsesEngineCache sync.Map
-
-// anyHelperScriptUsesEngine returns true if any _script helper in dir or its ancestors
-// contains $DATABRICKS_BUNDLE_ENGINE.
-func anyHelperScriptUsesEngine(dir string) bool {
-	if dir == "" || dir == "." {
-		return false
-	}
-	if v, ok := helperScriptUsesEngineCache.Load(dir); ok {
-		return v.(bool)
-	}
-	content, err := os.ReadFile(filepath.Join(dir, "_script"))
-	result := (err == nil && strings.Contains(string(content), "$DATABRICKS_BUNDLE_ENGINE")) ||
-		anyHelperScriptUsesEngine(filepath.Dir(dir))
-	helperScriptUsesEngineCache.Store(dir, result)
-	return result
-}
-
 // hasRunFilter returns true if the -run flag contains '=', indicating a specific
-// EnvMatrix variant was requested (e.g. DATABRICKS_BUNDLE_ENGINE=direct).
+// EnvMatrix variant was requested (e.g. DMS=true).
 func hasRunFilter() bool {
 	f := flag.Lookup("test.run")
 	return f != nil && strings.Contains(f.Value.String(), "=")
@@ -264,7 +243,7 @@ func requirePrerequisites(t *testing.T) bool {
 
 // selectedTests, when non-empty, limits the run to those test directories.
 // skipToolBuilds skips building the tools that the selected tests do not use
-// (terraform, the databricks-bundles wheel, yamlfmt); it must stay false for a
+// (the databricks-bundles wheel, yamlfmt); it must stay false for a
 // full run.
 func testAccept(t *testing.T, inprocessMode bool, selectedTests []string, skipToolBuilds bool) int {
 	if testdiff.OverwriteMode && !hasRunFilter() {
@@ -654,10 +633,7 @@ func testAccept(t *testing.T, inprocessMode bool, selectedTests []string, skipTo
 
 			expanded := internal.ExpandEnvMatrix(config.EnvMatrix, config.EnvMatrixExclude, extraVars)
 			if Subset {
-				scriptContent, _ := os.ReadFile(filepath.Join(dir, EntryPointScript))
-				scriptUsesEngine := strings.Contains(string(scriptContent), "$DATABRICKS_BUNDLE_ENGINE") ||
-					anyHelperScriptUsesEngine(dir)
-				expanded = internal.SubsetExpanded(expanded, dir, scriptUsesEngine)
+				expanded = internal.SubsetExpanded(expanded, dir)
 			}
 
 			// If the matrix expands to a single empty envset, run the test directly
@@ -1073,10 +1049,6 @@ func runTest(t *testing.T,
 		// (kill_after.py, callserver.py, …) would route their requests through
 		// the blocking proxy. NO_PROXY exempts them.
 		cmd.Env = append(cmd.Env, "NO_PROXY=127.0.0.1,localhost")
-		// Terraform phones home to checkpoint-api.hashicorp.com on every run to
-		// check for updates. Disable it so these CONNECT requests don't reach the
-		// blocking proxy and fail every terraform-engine test.
-		cmd.Env = append(cmd.Env, "CHECKPOINT_DISABLE=1")
 	}
 	// Run from outputDir so the entry-point script isn't a bundle source; the script
 	// cd's into tmpDir (the bundle dir) as its first line.
@@ -2063,8 +2035,8 @@ func loadScriptReplacements(t *testing.T, repls *testdiff.ReplacementsContext, r
 
 type pathFilter struct {
 	// contains substrings from the variants other than current.
-	// E.g. if EnvVaryOutput is DATABRICKS_BUNDLE_ENGINE and current test running DATABRICKS_BUNDLE_ENGINE="terraform" then
-	// notSelected contains ".direct." meaning if filename contains that (e.g. out.deploy.direct.txt) then we ignore it here.
+	// E.g. if EnvVaryOutput is MODE = ["a", "b"] and current test running MODE="a" then
+	// notSelected contains ".b." meaning if filename contains that (e.g. out.deploy.b.txt) then we ignore it here.
 	notSelected []string
 }
 

@@ -220,10 +220,9 @@ def main():
     for filename in get_state_files(args.target, args.backup):
         if not os.path.exists(filename):
             continue
-        # Recording only applies to the direct engine, so a terraform run prints the file as-is.
+        # Recording only applies to direct-engine state, so a terraform state file prints as-is.
         recording = os.environ.get("DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY") == "true"
-        terraform = os.environ.get("DATABRICKS_BUNDLE_ENGINE") == "terraform"
-        if recording and not terraform and not args.no_dms:
+        if recording and filename.endswith("resources.json") and not args.no_dms:
             print_recorded_state(filename, args.target)
         else:
             print_file(filename)

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 """
-Print selected attributes from terraform state.
+Print selected attributes from the bundle state.
 
 Usage: <group> <name> [attr...]
 """
@@ -13,34 +13,6 @@
 from print_state import get_resources
 
 
-def print_resource_terraform(group, name, *attrs):
-    resource_type = "databricks_" + group[:-1]
-    filename = ".databricks/bundle/default/terraform/terraform.tfstate"
-    # A missing state file (e.g. after destroy removes it) means the resource's
-    # state is not there, same as a missing entry within the file.
-    if not os.path.exists(filename):
-        print(f"State not found for {group}.{name} in {filename}")
-        return
-    raw = open(filename).read()
-    data = json.loads(raw)
-    found = 0
-    for r in data["resources"]:
-        r_type = r["type"]
-        r_name = r["name"]
-        if r_type != resource_type:
-            continue
-        if r_name != name:
-            continue
-        for inst in r["instances"]:
-            attribute_values = inst.get("attributes")
-            if attribute_values:
-                values = [f"{x}={attribute_values.get(x)!r}" for x in attrs]
-                print(group, name, " ".join(values))
-                found += 1
-    if not found:
-        print(f"State not found for {group}.{name} in {filename}")
-
-
 def print_resource_direct(group, name, *attrs):
     filename = ".databricks/bundle/default/resources.json"
     # A missing state file (e.g. after destroy removes it) means the resource's
@@ -79,7 +51,5 @@ def print_resource_recorded(group, name, *attrs):
 
 if os.environ.get("DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY") == "true":
     print_resource_recorded(*sys.argv[1:])
-elif os.environ.get("DATABRICKS_BUNDLE_ENGINE") == "terraform":
-    print_resource_terraform(*sys.argv[1:])
 else:
     print_resource_direct(*sys.argv[1:])
@@ -1,8 +1,4 @@
-if [ "$DATABRICKS_BUNDLE_ENGINE" = "direct" ]; then
-  state_file=".databricks/bundle/default/resources.json"
-else
-  state_file=".databricks/bundle/default/terraform/terraform.tfstate"
-fi
+state_file=".databricks/bundle/default/resources.json"
 
 title "Step 1: Deploy with job_1 only"
 trace $CLI bundle deploy

@@ -4,5 +4,3 @@ Ignore = [
   "databricks_v2.yml",
   "stale_state.json",
 ]
-
-Env.DATABRICKS_BUNDLE_ENGINE = "direct"
@@ -25,6 +25,9 @@ resources:
           kms_key: ''
           region: ''
       cluster_name: ''
+      driver_instance_pool_id: ''
+      driver_node_type_id: ''
+      instance_pool_id: ''
       node_type_id: ''
       num_workers: 1
       policy_id: ''
@@ -45,7 +48,9 @@ resources:
       description: ''
       git_source:
         git_branch: main
+        git_commit: ''
         git_provider: gitHub
+        git_tag: ''
         git_url: https://example.com/repo
       job_clusters:
         - job_cluster_key: main
@@ -63,11 +68,15 @@ resources:
                 kms_key: ''
                 region: ''
             cluster_name: ''
+            driver_instance_pool_id: ''
+            driver_node_type_id: ''
+            instance_pool_id: ''
             node_type_id: ''
             num_workers: 1
             policy_id: ''
             single_user_name: ''
             spark_version: 13.3.x-scala2.12
+          serverless_compute_id: ''
       max_concurrent_runs: 1
       name: ''
       parent_path: ''
@@ -79,6 +88,8 @@ resources:
   pipelines:
     my_pipeline:
       budget_policy_id: ''
+      catalog: ''
+      channel: ''
       clusters:
         - aws_attributes:
             availability: SPOT
@@ -90,14 +101,18 @@ resources:
           label: default
           node_type_id: ''
           policy_id: ''
+      edition: ''
       libraries:
         - file:
             path: pipeline.py
           jar: ''
           whl: ''
       name: ''
       root_path: ''
+      schema: ''
       serverless_compute_id: ''
+      storage: ''
+      target: ''
       usage_policy_id: ''
   registered_models:
     my_registered_model:
@@ -121,6 +136,7 @@ resources:
     my_warehouse:
       auto_stop_mins: 10
       cluster_size: 2X-Small
+      creator_name: ''
       instance_profile_arn: ''
       max_num_clusters: 1
       min_num_clusters: 1

@@ -7,12 +7,11 @@
 settable string LEAF (including nested ones) that the base doesn't already set
 to "". Coverage is therefore scoped to the blocks present in base.yml, which is
 how we avoid mutually-exclusive blocks (aws vs gcp attributes, git_source vs
-none) that would make terraform abort.
+none).
 
 A field is settable/eligible when out.fields.txt types it as `string` (this
 skips enums, which are named types) with flag ALL or INPUT, and it is not
-output_only (<resource_type>.generated.yml), a bundle-framework field, or a
-known terraform-erroring field.
+output_only (<resource_type>.generated.yml) or a bundle-framework field.
 
 Run from the repo root; writes databricks.yml in the test directory:
   acceptance/bundle/empty_string_dropped/gen_empty_config.py
@@ -32,23 +31,6 @@
 # Bundle-framework fields present on every resource; not real API inputs.
 FRAMEWORK_FIELDS = {"id", "url", "modified_status"}
 
-# Leaf field names that make terraform error at plan time (ConflictsWith, enum
-# validation on string-typed fields, computed attributes), so no request is
-# recorded. Applied per resource type. Matched by leaf name at any depth.
-TERRAFORM_ERRORS = {
-    "clusters": {"instance_pool_id", "driver_instance_pool_id", "driver_node_type_id"},
-    "jobs": {
-        "instance_pool_id",
-        "driver_instance_pool_id",
-        "driver_node_type_id",
-        # git_source: branch/commit/tag are mutually exclusive; base sets branch.
-        "git_commit",
-        "git_tag",
-    },
-    "pipelines": {"channel", "edition", "catalog", "storage", "schema", "target"},
-    "sql_warehouses": {"creator_name"},
-}
-
 
 def string_leaf_parents():
     """Map resource type -> {parent_schema_path: {leaf field names}}.
@@ -115,7 +97,7 @@ def main():
     base = yaml.safe_load(BASE.read_text())
 
     for rtype, entries in sorted(base.get("resources", {}).items()):
-        excluded = output_only.get(rtype, set()) | FRAMEWORK_FIELDS | TERRAFORM_ERRORS.get(rtype, set())
+        excluded = output_only.get(rtype, set()) | FRAMEWORK_FIELDS
         for cfg in entries.values():
             fill(cfg, "", parents.get(rtype, {}), excluded)
 

@@ -35,7 +35,6 @@
   "method": "POST",
   "path": "/api/2.0/pipelines",
   "body": {
-    "channel": "CURRENT",
     "clusters": [
       {
         "aws_attributes": {
@@ -48,7 +47,6 @@
       "kind": "BUNDLE",
       "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/direct/state/metadata.json"
     },
-    "edition": "ADVANCED",
     "libraries": [
       {
         "file": {

@@ -75,7 +75,6 @@
   },
   "pipelines": {
     "my_pipeline": {
-      "channel": "CURRENT",
       "clusters": [
         {
           "aws_attributes": {
@@ -88,7 +87,6 @@
         "kind": "BUNDLE",
         "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/direct/state/metadata.json"
       },
-      "edition": "ADVANCED",
       "libraries": [
         {
           "file": {

@@ -1,5 +1,5 @@
 
->>> DATABRICKS_BUNDLE_ENGINE=direct [CLI] bundle deploy -t direct
+>>> [CLI] bundle deploy -t direct
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/direct/files...
 Created apps.my_app
 Created clusters.my_cluster

@@ -1,13 +1,12 @@
 # databricks.yml is committed, generated by gen_empty_config.py with every settable
-# string field set to "". Deploy on the direct engine and record the create requests to
-# show which "" fields it sends. (The terraform engine, which dropped "" via omitempty,
-# was removed in v1.20.0, so there is no longer a cross-engine comparison.)
+# string field set to "". Deploy and record the create requests to show which "" fields
+# are sent.
 
 # Resolved config the engine sees. Today every "" field is still present here; a
 # fix in the initialize phase would drop them, and this golden would show that.
 $CLI bundle validate -o json -t direct | nostamp | jq .resources > out.validate.json
 
 # Exclude non-create traffic: workspace file ops, telemetry (nondeterministic), and the
 # deployment history calls the DMS run adds.
-trace DATABRICKS_BUNDLE_ENGINE=direct $CLI bundle deploy -t direct
-print_requests.py --nostamp ^//api/2.0/workspace ^//telemetry ^//api/2.0/bundle --sort > out.requests.direct.json
+trace $CLI bundle deploy -t direct
+print_requests.py --nostamp ^//api/2.0/workspace ^//telemetry ^//api/2.0/bundle --sort > out.requests.json