feat: embed distro-specific rendered script hotfixes in ANC

[Devinwong]

What this changes

This adjusts #9101's ANC script-hotfix design to embed fully rendered, distro-specific nodecustomdata YAML files in the ANC package instead of a JSON manifest plus separate payload files.

The hotfix generator now selects changed write_files blocks from the traditional Linux nodecustomdata.yml path and renders them through AgentBaker's canonical Go-template functions. ANC selects the YAML matching the running distro and transactionally replaces only files already present on the VHD. The embedded path reuses ANC's existing nodecustomdata model, decoder, and file-application engine rather than maintaining a second applier.

Architecture

%%{init: {"themeVariables": {"fontSize": "18px"}}}%%
flowchart TB
    subgraph Legend[Legend]
        direction LR
        L1[New]:::new
        L2[Updated]:::updated
        L3[Existing]:::existing
    end

    A["Changed provisioning scripts"]:::existing --> B["hotfix_generate.py"]:::updated
    B --> C["Render distro-specific<br/>nodecustomdata YAML"]:::new
    C --> D["Publish ANC package to PMC"]:::existing

    E["LPS base-version map<br/>default selector"]:::existing --> G["Install selected ANC package"]:::existing
    F["ABsvc version pointer<br/>cold-start fallback"]:::existing -.-> G
    D --> G

    G --> H["ANC applies embedded YAML<br/>when active"]:::new
    H --> I["Normal CSE provisioning"]:::existing
    H -. fail open .-> J["Keep VHD-baked scripts"]:::existing
    J --> I

    classDef new fill:#dafbe1,stroke:#1a7f37,color:#0b1f0f,stroke-width:2px,font-size:18px
    classDef updated fill:#fff8c5,stroke:#9a6700,color:#24292f,stroke-width:2px,font-size:18px
    classDef existing fill:#ddf4ff,stroke:#0969da,color:#0b1f0f,stroke-width:2px,font-size:18px

Loading

Generated payload example

#cloud-config
write_files:
- path: /opt/azure/containers/provision_configs.sh
  permissions: "0744"
  encoding: gzip
  owner: root
  content: !!binary |
    H4sIAAAAAAAA/...

Each generated file is complete, template-free cloud-config. Conditional distro blocks are resolved by the same getBakerFuncMap and node-custom-data variables used by production rendering rather than reimplemented in Python.

Package selection and fallback

The LPS base-version map is the primary hotfix selector. ABsvc seeds the on-node configuration with a small single-version pointer so first-node provisioning still has a fallback when LPS is not ready. ABsvc never carries the rendered script contents; those are stored in the corresponding PMC package, avoiding CustomData size growth.

The package version identifies the exact hotfix content for a VHD version base, so this path does not generate or depend on scripts_version. Package installation and payload application remain fail-open to the original VHD scripts.

Application guarantees

• Strict YAML decoding rejects unknown fields, unsupported encodings/owners, invalid modes, duplicate paths, unsafe destinations, and empty decoded content.
• Payloads are replace-only. Missing destinations are skipped, preserving runtime gates such as custom-image exclusions and preventing hotfixes from creating unexpected files.
• All replacements are staged and synced before commit. A later commit failure rolls back earlier replacements and preserves a backup if rollback itself fails.
• Reapplying identical content and mode is idempotent.
• The embedded active marker avoids distro detection and file access when a package contains no script hotfix.
• Ubuntu, Mariner/Azure Linux, ACL, OS Guard, and Flatcar render artifacts are generated. Current package-manager delivery covers Ubuntu and Azure Linux-family images; Flatcar remains lower priority and retains the node-image-upgrade fallback.

Workflow behavior

The hotfix-generate workflow now tracks the generated ANC YAML directory and hotfix version JSON. It no longer edits parts/linux/cloud-init/nodecustomdata.yml. Script changes make the embedded ANC package a production change, causing the normal base-version patch bump and PMC package publication path. When a PR has no new script or ANC hotfix, the existing generated payload and version pointer are left unchanged; retirement remains explicit.

Validation

• go test ./pkg/agent ./hotfix/render-nodecustomdata
• Focused aks-node-controller nodecustomdata and embedded payload tests
• Focused ANC provisioning integration tests for embedded apply, fail-open, and dry-run behavior
• Focused E2E fixture tests
• python3 -m unittest hotfix.hotfix_generate_test
• go vet for the changed root, ANC, and E2E packages

The broader ANC suite was also attempted locally; unrelated Windows-only failures remain where existing tests require /bin/bash or hold temporary watcher files during cleanup.

[github-actions]

Windows Unit Test Results

  3 files   13 suites   50s ⏱️
409 tests 409 ✅ 0 💤 0 ❌
412 runs  412 ✅ 0 💤 0 ❌

Results for commit 96e31d6.

♻️ This comment has been updated with latest results.

[Copilot]

🟡 Changes recommended

Critical stale-binary fallback and unsupported-script issues, plus payload completeness and durability concerns, remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Embeds distro-specific rendered Linux script hotfix payloads in ANC and applies them during provisioning.

Changes:

• Adds canonical AgentBaker rendering and hotfix generation.
• Adds transactional ANC payload validation and application.
• Adds unit, integration, E2E, documentation, and workflow coverage.

Required fixes:

• Critical — aks-node-controller/app.go:691: A failed package update can leave a previously staged hotfix binary selected, reapplying stale payloads instead of falling back to VHD scripts. Remove or validate stale binaries before launcher selection.
• Moderate — aks-node-controller/scripthotfix/applier.go:351: Apply file permissions before syncing, or sync again afterward, so executable modes are durable before commit.
• Moderate — hotfix/hotfix_generate.py:384: Track every requested key and fail generation when any mapped artifact lacks a selected write_files block.
• Critical — hotfix/hotfix_generate.py:83: initAKSCustomCloud lacks the canonical variable and write_files block required to generate its payload. Remove the mapping or add canonical support.

File summaries

File	Description
pkg/agent/nodecustomdata_render_test.go	Tests distro-aware rendering.
pkg/agent/baker.go	Exposes canonical rendering.
hotfix/render-nodecustomdata/main.go	Renders platform payloads.
hotfix/hotfix_generate.py	Generates ANC hotfix assets.
hotfix/hotfix_generate_test.py	Tests generation behavior.
e2e/vmss.go	Builds fixture-enabled ANC binaries.
e2e/vmss_test.go	Tests fixture generation.
e2e/types.go	Defines hotfix fixtures.
e2e/scenario_test.go	Adds Ubuntu hotfix E2E coverage.
aks-node-controller/scripthotfix/generated/rendered_nodecustomdata_ubuntu.yml	Adds the Ubuntu baseline payload.
aks-node-controller/scripthotfix/generated/rendered_nodecustomdata_mariner.yml	Adds the Mariner baseline payload.
aks-node-controller/scripthotfix/generated/rendered_nodecustomdata_flatcar.yml	Adds the Flatcar baseline payload.
aks-node-controller/scripthotfix/generated/rendered_nodecustomdata_azlosguard.yml	Adds the OS Guard baseline payload.
aks-node-controller/scripthotfix/generated/rendered_nodecustomdata_acl.yml	Adds the ACL baseline payload.
aks-node-controller/scripthotfix/generated/active	Marks the baseline payload inactive.
aks-node-controller/scripthotfix/applier.go	Validates and transactionally applies payloads.
aks-node-controller/scripthotfix/applier_test.go	Tests validation, application, and rollback.
aks-node-controller/README.md	Documents hotfix delivery.
aks-node-controller/app.go	Applies embedded payloads before provisioning.
aks-node-controller/app_test.go	Tests application and fail-open behavior.
.github/workflows/hotfix-generate.yml	Generates and commits hotfix artifacts.

Review details

• Files reviewed: 15/21 changed files
• Comments generated: 4
• Review effort level: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Three unresolved issues affect rollback safety, ACL package delivery, and custom-image protections.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

aks-node-controller/app.go:699

• 🔴 High Risk — 🏗️ Architecture: Not every application error is safely fail-open. commitStagedNodeCustomDataWithRename can return an error containing rollback failed after earlier destinations were already replaced; continuing here then runs a mixed old/hotfixed script set rather than the VHD-baked fallback. Distinguish an uncertain/partial-commit error and stop provisioning (or synchronously restore the preserved backups); continue only after validation/staging failures or a successful rollback.

	if err != nil {
		// Hotfixes are fail-open: the VHD-baked scripts remain available, so an
		// embedded payload must not block provisioning.
		slog.Warn("failed to apply embedded hotfix payload; continuing with existing scripts",
			"error", err)

• Files reviewed: 17/23 changed files
• Comments generated: 2
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Hotfix generation is non-cumulative, and failed rollbacks can leave provisioning with inconsistent scripts.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

aks-node-controller/app.go:699

• 🔴 High Risk — 🔄 Backward Compatibility: this fail-open branch also catches the rollback failed error from commitStagedNodeCustomData. In that case at least one destination can still contain the new payload while later entries remain old, so continuing into CSE runs an inconsistent script set rather than the promised VHD-baked fallback. Distinguish incomplete rollback from pre-commit/fully-rolled-back failures and abort provisioning (or finish restoring the preserved backups) for that case.

	if err != nil {
		// Hotfixes are fail-open: the VHD-baked scripts remain available, so an
		// embedded payload must not block provisioning.
		slog.Warn("failed to apply embedded hotfix payload; continuing with existing scripts",
			"error", err)

aks-node-controller/nodecustomdata.go:305

• 🟡 Medium Risk — the file is synced before its executable mode is applied, so the staged artifact is not fully durable when it is committed. After a crash or abrupt reboot, a hotfixed script can retain CreateTemp's non-executable mode even though its content was flushed. Apply the mode before Sync so both data and metadata are persisted before rename.

	if err := temp.Sync(); err != nil {

• Files reviewed: 17/23 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Critical custom-image compatibility issues can overwrite a script that production rendering intentionally preserves.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 17/23 changed files
• Comments generated: 2
• Review effort level: Balanced