Skip to content

bundle: route config sync edits to the block that defines them - #6117

Open
ilyakuz-db wants to merge 1 commit into
configsync/acc-split-list-baselinefrom
configsync/block-provenance
Open

bundle: route config sync edits to the block that defines them#6117
ilyakuz-db wants to merge 1 commit into
configsync/acc-split-list-baselinefrom
configsync/block-provenance

Conversation

@ilyakuz-db

@ilyakuz-db ilyakuz-db commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Changes

Routes each config-remote-sync change back to the YAML block that actually defines it, instead of using a merged-view position against a single file.

New bundle/configsync/blockindex.go reloads the bundle through the standard loader mutators without selecting a target, so the top-level and targets.<target> regions are still separate and every value carries the location of the one place it was written. That gives a location → block mapping. resolveSelectors now records the sequence elements a change path traverses, and each merged index is rewritten into the index inside the chosen block. yamlFileIndex is deleted — it counted elements sharing a file, which cannot distinguish a top-level block from a target override in that same file.

How a destination gets picked:

  • element defined in exactly one block → that block;
  • element assembled from several blocks → the leaf field decides, since merging records each field's location separately (so max_retries and timeout_seconds on the same task go to different blocks);
  • removing an element defined in several blocks → one deletion per block, because the element has a part in each and the merged element only disappears once both are gone;
  • brand-new element or field → the block declaring the resource, keeping it out of a target-specific scope the user did not ask for.

Index bookkeeping (indicesToReplaceMap, indexOperations) is now scoped per block, so a removal in one block no longer shifts indices in another.

Golden diffs in this PR, against the corruption recorded by the parent:

# top-level: [zeta, mu]   targets.dev: [alpha]   merged+sorted: [alpha, mu, zeta]
# remote: edit alpha.timeout_seconds
-        - timeout_seconds: 111        # appended to the top-level block, keyless
+              timeout_seconds: 111    # on alpha, in the target block

split_keyed_remove and split_positional previously failed outright (remove index key out of bounds, parent path ... does not exist) and now apply. Removing a task defined in both blocks goes from task_key: both count 1 — half-deleted, the other half orphaned — to 0.

Why

The merged list and the physical blocks disagree on both order and length, so a merged index addresses the wrong element — or no element — in any single block. FieldCandidates made this worse by guessing: it emitted [path, targets.<t>.path] and patch.go tried the top-level pointer first, falling back only on error, so a top-level block that happened to have an element at that index silently accepted the patch. Routing is now exact, and the candidate list holds the single resolved path.

The sync is unattended, so leaving a change for the next run is acceptable; writing it to the wrong place is not.

Tests

The six split goldens flip from corrupted to correct. All 21 pre-existing directories are byte-for-byte unchanged — including job_multiple_tasks, which covers edit/add/remove/rename on a single-block job and is the evidence that the common path is unaffected, and multiple_files, which covers renames with depends_on references across included files. Unit tests in blockindex_test.go cover per-field routing on a two-block element, removal spanning every defining block, and placement into the right included file.

@github-actions

Copy link
Copy Markdown
Contributor

Approval status: pending

/acceptance/bundle/ - needs approval

7 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @lennartkats-db, @shreyas-goenka, @andrewnester, @anton-107

/bundle/ - needs approval

4 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @lennartkats-db, @shreyas-goenka, @andrewnester, @anton-107

General files (require maintainer)

Files: .nextchanges/bundles/configsync-split-list-routing.md
Based on git history:

  • @denik -- recent work in bundle/configsync/, .nextchanges/bundles/

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db) can approve all areas.
See OWNERS for ownership rules.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: f957e06

Run: 30658499958

Env 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 3 1 4 322 1072 7:14
🟨​ aws windows 3 2 1 4 322 1070 6:40
🟨​ azure linux 3 1 4 322 1071 6:54
🟨​ azure windows 3 1 4 324 1069 8:53
💚​ gcp linux 1 5 321 1073 4:53
💚​ gcp windows 1 5 323 1071 4:14
10 interesting tests: 4 SKIP, 3 KNOWN, 2 flaky, 1 RECOVERED
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestFsCpFileToFile ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
🔄​ TestFsCpFileToFile/uc-volumes_to_dbfs ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
🟨​ TestFetchRepositoryInfoAPI_FromRepo 🟨​K 🟨​K 🟨​K 🟨​K 🙈​S 🙈​S
🟨​ TestFetchRepositoryInfoAPI_FromRepo/root 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestFetchRepositoryInfoAPI_FromRepo/subdir 🟨​K 🟨​K 🟨​K 🟨​K
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:55 azure windows TestAccept
3:15 aws windows TestAccept
3:02 gcp windows TestAccept

@ilyakuz-db
ilyakuz-db force-pushed the configsync/acc-split-list-baseline branch from c6a5c96 to 711e1c7 Compare July 31, 2026 13:33
@ilyakuz-db
ilyakuz-db force-pushed the configsync/block-provenance branch from 12b54e7 to ac0c041 Compare July 31, 2026 13:33
@ilyakuz-db
ilyakuz-db force-pushed the configsync/acc-split-list-baseline branch from 711e1c7 to 00635ac Compare July 31, 2026 13:41
@ilyakuz-db
ilyakuz-db force-pushed the configsync/block-provenance branch from ac0c041 to 388627d Compare July 31, 2026 13:41
@ilyakuz-db
ilyakuz-db force-pushed the configsync/acc-split-list-baseline branch from 00635ac to 0d5de02 Compare July 31, 2026 17:55
@ilyakuz-db
ilyakuz-db force-pushed the configsync/block-provenance branch from 388627d to 0dec516 Compare July 31, 2026 17:55
@ilyakuz-db
ilyakuz-db force-pushed the configsync/acc-split-list-baseline branch from 0d5de02 to 3854cff Compare July 31, 2026 19:16
@ilyakuz-db
ilyakuz-db force-pushed the configsync/block-provenance branch from 0dec516 to f957e06 Compare July 31, 2026 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants