bundle: route config sync edits to the block that defines them

.nextchanges/bundles/configsync-split-list-routing.md

@@ -0,0 +1 @@
+`bundle config-remote-sync` now writes each change back to the YAML block that actually defines it. When a resource's list field (a job's `tasks` or `job_clusters`, a pipeline's `clusters`) is defined both at the top level and in a `targets.<target>` override, loading merges the two blocks into one sorted list. The write-back used the merged position against a single file, so an edit to a task defined in the target block could be written into an unrelated top-level task, an edit to a task in an included file could be written to `databricks.yml`, and removing one task per block could fail or delete the wrong element. An element defined in more than one block has a part in each, so its removal is now written to every block that defines it.

acceptance/bundle/config-remote-sync/README.md

@@ -1,30 +1,25 @@
 # Split-list write-back tests
 
-The `split_*` directories cover `config-remote-sync` write-back
-when a resource's list field is defined in more than one physical YAML region: a top-level
-`resources.<type>.<name>.<list>` block and a `targets.<target>.resources…<list>` override,
-possibly across several included files.
+The `split_*` directories cover `config-remote-sync` write-back when a resource's
+list field is defined in more than one physical YAML region: a top-level
+`resources.<type>.<name>.<list>` block and a `targets.<target>.resources…<list>`
+override, possibly across several included files.
 
 Loading merges those regions into one in-memory list (keyed lists are also sorted by key),
 but on disk they remain separate regions that have to be edited independently. A remote
 change is expressed against the merged view, so write-back has to map it back to the right
 `(file, block, position)`.
 
-**The goldens in these directories record current behavior, which is wrong.** They are
-committed as-is so that the fixes which follow show up as a golden diff: corruption on one
-side, correct output on the other. Each `script` names the scenario it covers next to the
-step that exercises it.
+Each `script` names the scenario it covers next to the step that exercises it.
 
-What the goldens currently show:
-
-| Directory | Current behavior |
+| Directory | What it pins down |
 | --- | --- |
-| `split_keyed_edit` | Editing a task defined only in the target block appends a keyless `- timeout_seconds: 111` element to the top-level block instead of updating the task. The follow-up sync then reports `tasks[task_key='']: remove` for the element it just created. |
-| `split_keyed_twoblock` | Editing a target-block task whose sibling is defined in both blocks appends another keyless element. Routing one field per block on the two-block element already works and is covered here too. |
-| `split_keyed_remove` | Removing one task per block fails with `remove index key out of bounds (idx 2, len 2)`: a merged index is used against a block that has fewer elements. Removing a task defined in both blocks deletes it from the top-level block, leaving the target block's half orphaned. |
-| `split_multifile` | An edit to a task defined in an included override file is written to the top-level `databricks.yml` instead, so the wrong file changes and the intended one does not. |
-| `split_positional` | A field edit on a positionally-diffed pipeline cluster in the target block fails with `parent path /resources/pipelines/split_pipeline/clusters/1 does not exist`. A length change rewrites the whole list into one block. |
-| `split_isolation` | A rename of a two-block task is applied by guessing: the task key, the merged `timeout_seconds`, and quoted scalars are all written into the top-level block, and the blank line between resources is consumed. |
+| `split_keyed_edit` | A field edit on a task defined only in the target block, and on a top-level task whose merged position differs from its position in the block. Each is written to its own block. |
+| `split_keyed_twoblock` | A task defined in both blocks. Editing a plain single-block sibling leaves it untouched; each of its own fields is written to the block that defines that field, and a brand-new field goes to the block declaring the resource. |
+| `split_keyed_remove` | Removing one task per block in a single run removes exactly those two. A task defined in both blocks has a part in each, so its removal is written to both. |
+| `split_multifile` | One target's override spread across two included files: the edit lands in the file that defines the element, and the other two files are untouched. |
+| `split_positional` | Pipeline clusters, which diff by position rather than by key. Length-preserving edits route per block; a length change is applied or left unapplied without corrupting the surviving cluster. |
+| `split_isolation` | A two-block rename and an unrelated scalar edit in the same run. The unrelated edit is applied regardless of how the rename is placed, and the renamed task is never duplicated. |
 
 All of these run locally against the in-process test server (no `Cloud = true`), under both
 deployment engines, so they execute in normal CI.

acceptance/bundle/config-remote-sync/split_isolation/output.txt

@@ -38,6 +38,13 @@ Resource: resources.jobs.job_b
 +      max_concurrent_runs: 6
        tasks:
          - task_key: simple
+@@ -30,5 +30,3 @@
+       jobs:
+         job_a:
+-          tasks:
+-            - task_key: shared
+-              timeout_seconds: 45
++          tasks: []
 
 >>> grep -c max_concurrent_runs: 6 databricks.yml
 1

acceptance/bundle/config-remote-sync/split_keyed_edit/output.txt

@@ -17,20 +17,16 @@ Resource: resources.jobs.split_job
 >>> diff.py databricks.yml.backup databricks.yml
 --- databricks.yml.backup
 +++ databricks.yml
-@@ -18,4 +18,5 @@
-           notebook_task:
-             notebook_path: /Users/{{workspace_user_name}}/mu
-+        - timeout_seconds: 111
-
- targets:
+@@ -29,2 +29,3 @@
+               notebook_task:
+                 notebook_path: /Users/{{workspace_user_name}}/alpha
++              timeout_seconds: 111
 
 === Edit mu, a top-level task that is not first after the merge sort
 === Sync
 Detected changes in 1 resource(s):
 
 Resource: resources.jobs.split_job
-  tasks[task_key='']: remove
-  tasks[task_key='alpha'].timeout_seconds: add
   tasks[task_key='mu'].timeout_seconds: add
 
 
@@ -44,8 +40,8 @@ Resource: resources.jobs.split_job
            notebook_task:
              notebook_path: /Users/{{workspace_user_name}}/mu
 +          timeout_seconds: 222
-         - timeout_seconds: 111
 
+ targets:
 
 >>> [CLI] bundle destroy --auto-approve -t dev
 The following resources will be deleted:

acceptance/bundle/config-remote-sync/split_keyed_remove/output.txt

@@ -5,13 +5,37 @@ Deployment complete!
 
 === Remove gamma from the top-level block and alpha from the target block
 === Sync
-Error: failed to generate YAML files: failed to apply change to file [TEST_TMP_DIR]/databricks.yml for a field resources.jobs.remove_job.tasks[2]: failed to apply change: op remove /resources/jobs/remove_job/tasks/2: remove index key out of bounds (idx 2, len 2)
+Detected changes in 1 resource(s):
+
+Resource: resources.jobs.remove_job
+  tasks[task_key='alpha']: remove
+  tasks[task_key='gamma']: remove
+
 
-Exit code: 1
 
 === Exactly gamma and alpha are gone; beta survives
 
 >>> diff.py databricks.yml.backup databricks.yml
+--- databricks.yml.backup
++++ databricks.yml
+@@ -13,7 +13,4 @@
+     remove_job:
+       tasks:
+-        - task_key: gamma
+-          notebook_task:
+-            notebook_path: /Users/{{workspace_user_name}}/gamma
+         - task_key: beta
+           notebook_task:
+@@ -36,8 +33,5 @@
+       jobs:
+         remove_job:
+-          tasks:
+-            - task_key: alpha
+-              notebook_task:
+-                notebook_path: /Users/{{workspace_user_name}}/alpha
++          tasks: []
+         twoblock_remove_job:
+           tasks:
 
 === Remove the task defined in BOTH blocks
 === Sync
@@ -27,7 +51,7 @@ Resource: resources.jobs.twoblock_remove_job
 >>> diff.py databricks.yml.second-backup databricks.yml
 --- databricks.yml.second-backup
 +++ databricks.yml
-@@ -22,8 +22,4 @@
+@@ -19,8 +19,4 @@
      twoblock_remove_job:
        tasks:
 -        - task_key: both
@@ -36,9 +60,18 @@ Resource: resources.jobs.twoblock_remove_job
 -            notebook_path: /Users/{{workspace_user_name}}/both
          - task_key: keep
            notebook_task:
+@@ -35,5 +31,3 @@
+           tasks: []
+         twoblock_remove_job:
+-          tasks:
+-            - task_key: both
+-              timeout_seconds: 30
++          tasks: []
 
 >>> grep -c task_key: both databricks.yml
-1
+0
+
+Exit code: 1
 
 >>> grep -c task_key: keep databricks.yml
 1

acceptance/bundle/config-remote-sync/split_keyed_twoblock/output.txt

@@ -17,20 +17,16 @@ Resource: resources.jobs.twoblock_job
 >>> diff.py databricks.yml.backup databricks.yml
 --- databricks.yml.backup
 +++ databricks.yml
-@@ -21,4 +21,5 @@
-           notebook_task:
-             notebook_path: /Users/{{workspace_user_name}}/nu
-+        - timeout_seconds: 303
-
- targets:
+@@ -34,2 +34,3 @@
+               notebook_task:
+                 notebook_path: /Users/{{workspace_user_name}}/omega
++              timeout_seconds: 303
 
 === Edit one field per block on the two-block element, plus add a new field
 === Sync
 Detected changes in 1 resource(s):
 
 Resource: resources.jobs.twoblock_job
-  tasks[task_key='']: remove
-  tasks[task_key='omega'].timeout_seconds: add
   tasks[task_key='shared'].max_retries: replace
   tasks[task_key='shared'].min_retry_interval_millis: add
   tasks[task_key='shared'].timeout_seconds: replace
@@ -52,7 +48,7 @@ Resource: resources.jobs.twoblock_job
 +          min_retry_interval_millis: 5000
          - task_key: nu
            notebook_task:
-@@ -31,5 +32,5 @@
+@@ -30,5 +31,5 @@
            tasks:
              - task_key: shared
 -              timeout_seconds: 60

acceptance/bundle/config-remote-sync/split_multifile/output.txt

@@ -15,12 +15,6 @@ Resource: resources.jobs.multifile_job
 === databricks.yml is untouched
 
 >>> diff.py databricks.yml.backup databricks.yml
---- databricks.yml.backup
-+++ databricks.yml
-@@ -16,2 +16,3 @@
-           notebook_task:
-             notebook_path: /Users/{{workspace_user_name}}/base
-+          timeout_seconds: 777
 
 === overrides/10-first.yml is untouched
 
@@ -29,6 +23,12 @@ Resource: resources.jobs.multifile_job
 === overrides/20-second.yml gains timeout_seconds: 777
 
 >>> diff.py overrides/20-second.yml.backup overrides/20-second.yml
+--- overrides/20-second.yml.backup
++++ overrides/20-second.yml
+@@ -8,2 +8,3 @@
+               notebook_task:
+                 notebook_path: /Users/{{workspace_user_name}}/second
++              timeout_seconds: 777
 
 >>> [CLI] bundle destroy --auto-approve -t dev
 The following resources will be deleted:

acceptance/bundle/config-remote-sync/split_positional/output.txt

@@ -5,20 +5,39 @@ Deployment complete!
 
 === Edit num_workers on the top-level cluster and on the target cluster
 === Sync
-Error: failed to generate YAML files: failed to apply change to file [TEST_TMP_DIR]/databricks.yml for a field resources.pipelines.split_pipeline.clusters[1].num_workers: field not found in YAML configuration: op replace /resources/pipelines/split_pipeline/clusters/1/num_workers: parent path /resources/pipelines/split_pipeline/clusters/1 does not exist
+Detected changes in 1 resource(s):
+
+Resource: resources.pipelines.split_pipeline
+  clusters[0].num_workers: replace
+  clusters[1].num_workers: replace
+
 
-Exit code: 1
 
 === default updates top-level, maintenance updates the target block
 
 >>> diff.py databricks.yml.backup databricks.yml
+--- databricks.yml.backup
++++ databricks.yml
+@@ -15,5 +15,5 @@
+       clusters:
+         - label: default
+-          num_workers: 2
++          num_workers: 5
+       libraries:
+         - notebook:
+@@ -28,3 +28,3 @@
+           clusters:
+             - label: maintenance
+-              num_workers: 1
++              num_workers: 3
 
 === Remove the target-block cluster (length change)
 === Sync
 Detected changes in 1 resource(s):
 
 Resource: resources.pipelines.split_pipeline
   clusters: replace
+  clusters[1].num_workers: remove
 
 
 
@@ -27,13 +46,10 @@ Resource: resources.pipelines.split_pipeline
 >>> diff.py databricks.yml.backup databricks.yml
 --- databricks.yml.backup
 +++ databricks.yml
-@@ -15,5 +15,5 @@
-       clusters:
-         - label: default
--          num_workers: 2
-+          num_workers: 5
-       libraries:
-         - notebook:
+@@ -28,3 +28,2 @@
+           clusters:
+             - label: maintenance
+-              num_workers: 3
 
 >>> grep -c label: default databricks.yml
 1