fix: align containerd v2 configs with 2.3 schema - #8948
fix: align containerd v2 configs with 2.3 schema#8948Sylvain Boily (djsly) wants to merge 56 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Aligns AgentBaker-managed containerd v2 (Ubuntu 24.04) configuration with the containerd 2.3 config schema v4 so NVIDIA GPU Operator–generated v4 drop-ins are accepted, and updates runtime/plugin namespaces accordingly.
Changes:
- Updated Ubuntu 24.04 containerd v2 root config templates in
pkg/agentto declareversion = 4. - Migrated Kata runtime blocks in the v2 templates to the split
io.containerd.cri.v1.runtimeplugin namespace. - Updated the Ubuntu 24.04 GB static NVIDIA containerd config to schema v4 and the split
images/runtimeplugin namespaces.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| pkg/agent/baker.go | Updates Ubuntu 24.04 containerd v2 template schema to v4 and moves Kata runtime blocks to the split CRI runtime plugin namespace. |
| parts/linux/cloud-init/artifacts/ubuntu/gb/containerd-nvidia.toml | Updates the Ubuntu 24.04 GB NVIDIA containerd config to schema v4 and split CRI plugin namespaces. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
aks-node-controller/parser/templates/containerd.toml.gtpl:27
- When rendering config schema v4 (containerd >= 2.3),
snapshotter/disable_snapshot_annotationsshould be configured under the images plugin (io.containerd.cri.v1.images) rather than under the runtime plugin’s.containerdsection. Keeping these fields under[plugins."{{$runtimePlugin}}".containerd]means overlaybd artifact streaming (and potentially Kata’s snapshot-annotation behavior) may be ignored on v4 configs.
disable_snapshot_annotations = false
{{- end}}
{{- if .GetEnableArtifactStreaming }}
snapshotter = "overlaybd"
disable_snapshot_annotations = false
aks-node-controller/parser/templates/containerd_no_GPU.toml.gtpl:23
- For config schema v4 (containerd >= 2.3),
snapshotter/disable_snapshot_annotationsneed to be set on the images plugin (io.containerd.cri.v1.images), not under the runtime plugin’s.containerdtable. As-is, the v4 path will still emit these fields under[plugins."{{$runtimePlugin}}".containerd], which containerd v2.3’s split plugins are unlikely to consume.
{{- if .GetIsKata }}
disable_snapshot_annotations = false
{{- end}}
{{- if .GetEnableArtifactStreaming }}
snapshotter = "overlaybd"
76a6b7a to
7c46353
Compare
Windows Unit Test Results 3 files 13 suites 52s ⏱️ Results for commit 09b4986. ♻️ This comment has been updated with latest results. |
…ion on 24.04/26.04" This reverts commit 61c896e.
…rovision-config vs nbc-cmd parity check
The parity check only logged the differing env-var name ("differs: CONTAINERD_CONFIG_CONTENT"),
which isn't actionable — the scriptful (baker.go) and scriptless (aks-node-controller) containerd
configs render byte-identically for matching input, so a failure means the two config INPUTS
disagree on some field. Emit the canonical TOML entries that are only-in-provision-config vs
only-in-nbc-cmd so the exact diverging field is named in the failure log.
The nbc-cmd env value is parsed from a shell assignment and keeps its surrounding double quotes (CONTAINERD_CONFIG_CONTENT="<base64>"), so base64-decoding it directly failed and the entry diff silently returned empty. Strip quotes first (as envValsEqual does) so the exact differing TOML entries are actually logged.
Normalize (strip quotes + whitespace) before base64-decoding, and on any decode failure or entry-set-equal-but-content-differs case, return a descriptive string instead of collapsing to a bare 'differs: KEY'. This ensures the parity failure log names the exact differing TOML entries (or explains why it can't) in one run.
…heck (Copilot review) - containerdConfigContentEqual now normalizes (strip quotes + whitespace) before base64-decoding, matching containerdConfigEntryDiff, so it no longer silently falls back to a raw string compare and report a false diff for configs that are byte-identical modulo shell quoting/formatting. - Extract the shared normalizeContainerdConfigContent helper (DRY). - Correct the misleading comment: parseEnvValue already strips shell quoting, so normalization is defensive rather than the common case. - Add TestEnvValsEqualForKey_ContainerdContentNormalizesQuotesAndWhitespace covering a quoted + embedded-whitespace nbc value.
…nfigEntryDiff doc (Copilot review) - parser_test.go: assert the explicit `version = 4` header (the split-plugin namespace alone also matches containerd 2.0-2.2 schema v2, so it wasn't a real v4 regression test). - app.go: correct containerdConfigEntryDiff's doc comment — it returns a descriptive "decode-failed ..."/"entry-sets-equal ..." detail on undecodable/edge cases, not "".
…phase-2.5 path The e2e injects the freshly-compiled aks-node-controller by replacing `#hotfix-marker` in the boothook customData with a curl download (CustomDataWithNBCCmdHack); the on-node launcher then prefers `/opt/azure/containers/aks-node-controller-hotfix` when present. Only `boothookTemplate` (EnableScriptlessCSECmd) carried the marker — `cseBootHookTemplate`, used by the scriptless NBC-cmd / "phase 2.5" path (EnableScriptlessNBCCSECmd) that the provision-config vs nbc-cmd parity check runs on, did not. So that path ran the VHD-baked (stale) aks-node-controller while the nbc-cmd was rendered by the PR's baker, making the parity check compare a stale baked ANC against the PR's baker — a false "differs" for any PR that changes containerd config rendering. Add `#hotfix-marker` to `cseBootHookTemplate` (a harmless bash comment in production) so e2e runs the compiled binary on this path too, and the parity check compares PR-ANC against PR-baker. Add a regression test asserting both scriptless boothook templates carry the marker.
The main merge (#9129) added isContainerdV2, but this PR's routing uses isContainerdV2OrLater; after the merge isContainerdV2 had no callers and tripped the unused linter. Remove it.
…on GPU scenario Test_Ubuntu2404_DraDriverNvidiaGpuRunning_AKSNodeController is Scriptless:true (so it runs ValidateScriptlessPhase3, the provision-config vs nbc-cmd env-var parity check) but also provisions via the published managed AKSNode VM extension. That extension runs its own VHD-baked aks-node-controller, which e2e cannot override with the PR-compiled binary the way the boothook #hotfix-marker injection does on the pure-scriptless path (see 577b5ba). So the parity check compares the published (baked) ANC's containerd rendering against the PR's baker-rendered nbc-cmd and reports a false 'differs: CONTAINERD_CONFIG_CONTENT; differs: CONTAINERD_CONFIG_NO_GPU_CONTENT' (with no entry detail, the tell-tale of the stale binary running the old compareEnvs) for any PR that changes containerd rendering. Verified offline that the parser (ANC) and baker render byte-identical containerd v4 config for this exact GPU/2404/containerd-2.3.3 scenario, so there is no real divergence to mask. Add Config.SkipScriptlessEnvParity, gate ValidateScriptlessPhase3 on it, and set it for this scenario.
…on 24.04
shouldUseContainerdV4Config fell back to the containerd 2.3+ (v4) schema whenever
ContainerdVersion was empty and the distro was 24.04 or 26.04. Older 24.04 VHDs still
ship containerd 2.0-2.2 (24.04 only moved to 2.3.1 on 2026-06-11), which reject
'version = 4' ("expected containerd config version equal to or less than 3") and
crash-loop, failing CSE with exit 84. The scriptful CSE writes CONTAINERD_CONFIG_CONTENT
verbatim with no on-node version detection, so a new-CSE-on-old-VHD render bricks the node.
Only emit v4 when the version is positively >= 2.3. Empty/unknown versions on 24.04 now
fall through to shouldUseContainerdV2Config -> before-2.3 split-plugin v2 schema, which
loads on all containerd 2.x. 26.04 keeps the v4 assumption since it only ever shipped
containerd >= 2.3. (Refined re-apply of reverted 01ce829, which over-corrected 26.04 too.)
The scriptless aks-node-controller path is already safe: it detects the on-node containerd
version at runtime. Unit tests assert empty+24.04->v2, empty+26.04->v4.
…ersion path setExpectedContainerdVersionForE2E backfills nbc.ContainerdVersion from HEAD's components.json whenever it is empty, so the parity check and scriptful CSE always run with a concrete (HEAD) containerd version. That masks the realistic backward-compat path where the RP omits ContainerdVersion and baker selects the containerd config schema from the distro fallback. E2E_OMIT_CONTAINERD_VERSION=true keeps the version empty so a backward-compat run (older VHD + HEAD CSE) exercises that fallback instead of pairing an old VHD with HEAD's latest containerd version.
…split config (AKS #5909) The before-2.3 containerd templates declared 'version = 2' while using the split [plugins."io.containerd.cri.v1.images"] / [...cri.v1.runtime] plugin paths (and pinned_images.sandbox). Under version = 2, containerd 2.x interprets config with the legacy io.containerd.grpc.v1.cri layout and silently ignores the cri.v1.* tables - including the registry config - which breaks ACR authentication (401) on Azure Linux 3 (containerd 2.2.4). Reported as Azure/AKS#5909; also drops the split kata runtime handlers. Set the split-plugin schema to 'version = 3', which is the correct schema for the cri.v1.* paths and loads on all containerd 2.0-2.2 (they accept up to v3; only v4 is rejected before 2.3). containerd 1.x keeps version = 2 + grpc.v1.cri; containerd 2.3+ keeps version = 4 + CDI. This also makes the empty-version 24.04 fallback render v3 (valid on 2.0-2.2 and 2.3), fully retiring the backward-compat brick. Supersedes the standalone fix in #9308 (team agreed to carry it here). Covers baker containerdV2BeforeV23{,NoGPU}ConfigTemplate + parser containerd_v2{,_no_GPU}.toml.gtpl, with unit tests updated to assert version = 3 for the 2.0-2.2 split path.
…nerd-v2 distros 3-agent review of the PR flagged a scriptful<->scriptless divergence: when the RP omits ContainerdVersion for a Kata containerd-2.x distro (e.g. AzureLinuxV3 Kata), baker fell through to the legacy v1 (grpc.v1.cri, version = 2) template while the scriptless ANC renders the split v3/v4 schema from the detected on-node version. IsContainerdV2Distro excluded Kata as a workaround for AKS#5909 (when the before-2.3 template wrongly declared version = 2 with the split cri.v1.* paths, dropping the Kata/registry config). Now that that template renders version = 3, the exclusion is obsolete and harmful: drop it so empty-version Kata containerd-v2 distros render the split v3 schema (kata handlers under cri.v1.runtime are parsed) and baker stays aligned with the ANC. Adds a baker test. Also from review: correct a stale 'schema v2' comment in parser_test.go (2.0-2.2 is v3), and document the >= 2.3 version coupling of the static GB containerd-nvidia.toml (v4).
…ime in empty-version v3 path
…y empty-24.04 fallback trade-off Final pre-review pass (3 agents + codex) flagged stale 'v2' comments where the 2.0-2.2 path now renders v3, and that the empty-version 24.04 fallback comment read as a present-tense 'older 24.04 VHDs still ship 2.0-2.2' (current r2404 is 2.3.3). Reword baker.go / baker_test.go / parser helper.go comments to say v3, and document the deliberate trade-off: empty-version 24.04 renders v3 (loads on all 2.x, no brick) even though a current 2.3.3 node would ideally get v4+CDI -- the scriptless ANC self-heals to v4 via on-node detection; the scriptful path stays conservative. Comment-only; no behavior change.
…s parity honestly The DRA managed-AKSNode scenario now enforces the provision-config vs nbc-cmd env-var parity check like every other scriptless scenario. The earlier skip was only needed because standalone e2e boots the published VHD's baked aks-node-controller, so its rendering was compared against the PR's baker (version skew, not a real diff). Proven by GPU E2E build 179341193 run from a VHD built off this PR (VHD_BUILD_ID=179244573): the node ran the PR's own baked ANC (launcher logged "Using VHD-baked binary") and Test_Ubuntu2404_DraDriverNvidiaGpuRunning_AKSNodeController passed with the skip removed.
Remove the newly-added TestScriptlessBoothookTemplatesHaveHotfixMarker guard. The functional fix (#hotfix-marker present in both scriptless boothook templates) is retained in baker.go; only the string-presence guard test is dropped to keep this PR scoped to the containerd config schema change.
The #hotfix-marker appears in both the cloud-boothook and the phase-2.5
boothook (two separate set -euo pipefail scripts in one customData), so
CustomDataWithNBCCmdHack's replace-all injected the curl download twice. A
transient failure on the second run could abort provisioning even after the
first succeeded. Guard with [ -x <binary> ] || { curl ... } so the second
occurrence is a no-op once the binary is present. Production is unaffected
(#hotfix-marker stays a no-op comment there). Addresses Copilot review.
Re-add the caller-side description of the special-key comparison logic (dropped during the if->switch refactor), covering SYSCTL_CONTENT and the two CONTAINERD_CONFIG* keys plus the literal-comparison fallback.
Fold TestContainerdConfigEntryDiff_NamesDifferingEntry into app_test.go next to the sibling TestEnvValsEqualForKey_ContainerdContent* tests and drop the stray single-test entrydiff_test.go. No coverage change.
We validate this PR's scriptless containerd rendering by running e2e against a VHD built from this PR (baked ANC == PR ANC), so the boothook binary-injection override is unnecessary for these scenarios. Revert the phase-2.5 #hotfix-marker added in 65db296 and the double-download guard, restoring e2e/vmss.go and cseBootHookTemplate to their main state (single pre-existing boothook marker). Consequence: scriptless env-parity scenarios (incl. Test_Ubuntu2404_DraDriver NvidiaGpuRunning_AKSNodeController) are validated on a PR-built VHD, not the published-VHD gate.
GB (Ubuntu 24.04 GPU) VHDs are a MAI-owned, self-managed path: the static config is baked verbatim and CSE skips regenerating it, so baker's schema selection never touches it at runtime and nothing in the tree depends on its schema. Keep this PR scoped to general-fleet containerd schema alignment and avoid coupling GB to 'baked containerd >= 2.3' (v4). main's version = 2 + grpc.v1.cri config is internally consistent and works on 2.x; MAI can migrate GB's schema on their own timeline.
There was a problem hiding this comment.
🔵 Needs a closer look
It changes containerd schema selection and config rendering across multiple provisioning paths (CSE + aks-node-controller) with direct impact on node bootstrap behavior, warranting final human validation despite the added tests.
Review details
- Files reviewed: 16/16 changed files
- Comments generated: 0 new
- Review effort level: Lite
isContainerdVersionGe was calling IsKubernetesVersionGe to compare containerd versions, which read oddly. Extract the underlying semver >= comparison into a scheme-agnostic semverGe helper; IsKubernetesVersionGe now delegates to it, and isContainerdVersionGe calls semverGe. No behavior change (containerd versions are still normalized by containerdSemverCore first).
Mirror the scriptful-side change on the scriptless (aks-node-controller/parser) path: extract the semver >= comparison into semverGe, have IsKubernetesVersionGe delegate to it, and route isContainerdVersionGe through semverGe. Genuine Kubernetes version comparisons (parser.go, isKubernetesVersionGe template func) keep the k8s-named function. No behavior change.
There was a problem hiding this comment.
🔵 Needs a closer look
It changes critical containerd provisioning/config rendering across multiple modes (scriptful/scriptless) and versions, so it warrants final human review despite solid test updates.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
aks-node-controller/app.go:502
- The comment says normalizeContainerdConfigContent "strips surrounding quotes", but the implementation removes all double quotes via stripDoubleQuotes (strings.ReplaceAll). Updating the comment to match the actual behavior will avoid confusion if this helper is reused or modified later.
- Files reviewed: 17/17 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core containerd configuration generation across multiple provisioning modes/distros and warrants final human review despite strong unit/E2E coverage.
Review details
- Files reviewed: 17/17 changed files
- Comments generated: 0 new
- Review effort level: Lite
isContainerdV2OrLater checked the containerd *runtime* major (>=2), but its 'V2' read as the config-schema vocabulary (v2/v3/v4) used elsewhere in this PR -- and runtime 2.x actually renders schema v3/v4, never v2. Replace its single call site with the existing isContainerdVersionGe(cfg, "2.0.0") (same semverGe-backed compare) and delete the function. Behavior-identical (Major()>=2 == >=2.0.0).
There was a problem hiding this comment.
🔵 Needs a closer look
It changes production-critical containerd config rendering across multiple execution paths (scriptful/scriptless) and OS/distro variants, so it warrants final human review despite strong unit/E2E validation.
Review details
- Files reviewed: 17/17 changed files
- Comments generated: 0 new
- Review effort level: Lite
Summary
Fix the containerd 2.3.x / NVIDIA GPU Operator config-version mismatch by making AgentBaker-owned containerd 2.3 root configs use containerd config schema v4. This prevents NVIDIA-generated v4 drop-ins from being rejected under a lower-version root config while keeping containerd 2.0-2.2 on the split-plugin schema v3 (fixing AKS #5909, where declaring version 2 with the split cri.v1.* paths made containerd ignore the ACR/registry config).
Changes
>= 2.3.0and split-plugin schema v3 for containerd 2.0-2.2 (version 2 with the splitcri.v1.*paths is invalid and silently drops the registry/kata config — AKS test: add taints e2e test for scriptless. #5909). containerd 1.x keeps version 2 +grpc.v1.cri.ContainerdVersiontaking precedence and local detection as fallback.ContainerdVersionstill renders the correct split-plugin config.Testing
GOPROXY=https://proxy.golang.org,direct go test ./pkg/agent/...cd aks-node-controller && GOPROXY=https://proxy.golang.org,direct go test ./parsercd aks-node-controller && GOPROXY=https://proxy.golang.org,direct go test ./...No generated files are affected (the containerd templates are
go:embed/inline, not generated; no containerd golden testdata pins the 2.0-2.2 path); CI generate/version-consistency checks pass. See the E2E verification below.Backward compatibility (new CSE against older VHDs)
Because CSE is rendered by the latest AgentBaker but can run against older VHDs (PUTAgentPool / NAP / NPS use AgentBaker as a library), the schema selection must not brick a node whose installed containerd predates 2.3, nor silently mis-declare the schema for containerd 2.0-2.2.
version = 4is emitted only on a positive explicitContainerdVersion >= 2.3.0. When the version is empty/unknown, 24.04 (and other containerd-v2 distros, incl. Kata) fall back to the before-2.3 split-plugin v3 schema (loads on all containerd 2.0-2.2 and 2.3); 26.04 keeps v4 (it only ever shipped containerd >= 2.3). The scriptless (aks-node-controller) path additionally detects the on-node containerd version at runtime. Unit tests assertempty+24.04 -> v3,empty+26.04 -> v4, andempty+Kata-v2 -> v3.version = 2with the splitio.containerd.cri.v1.*paths makes containerd read the legacygrpc.v1.crilayout and silently drop the registry/ACR + Kata config ([BUG] Azure Linux 3 Node Image containerd config uses deprecated v1 API path causing ACR authentication failures AKS#5909, ACR 401).version = 3is the correct schema for those paths and loads on all containerd 2.x.3", CSE exit 84) only affects containerd 2.0-2.1, which the v3 fallback above avoids.E2E verification
New (current published) VHDs — containerd 2.3.x, v4 path — GREEN: Agentbaker E2E
179020674and Agentbaker GPU E2E179020678both pass on the latest commit. (Agentbaker Windows E2E is red on a pre-existing, unrelated VHD-caching harness flake — also red ~4/6 recent runs onmain, node itself healthy; tracked/fixed separately in fix(windows): retry kubelet start check after NodeResetScriptTask #9344.)Old VHD (Ubuntu 24.04 + containerd 2.2.4, build
178126164) — backward compat — provisions cleanly on both suites: scriptful (DISABLE_SCRIPTLESS=true) + omitted version (E2E_OMIT_CONTAINERD_VERSION=true) so baker takes the distro fallback -> v3, written verbatim to the 2.2.4 nodes with no on-node detection.178998863: 2.2.4 nodes came up with no containerd schema errors, no CSE exit-84, no NotReady. (2 CSE failures are network-isolated ORAS artifact pulls, an old-VHD artifact — that scenario passes on the current VHD.)179015136: containerd v3 also loads clean on the GPU nodes (no schema errors); the red is the old VHD's baked GPU driver / managed-NVIDIA stack (CSE exit 227ERR_MANAGED_NVIDIA_EXP_INSTALL_FAIL), not containerd.Old VHD (Ubuntu 24.04 + containerd
2.1.7, VHD build179936512) — the 2.0-2.1 range that actually rejects v4 — node-level proof: re-ran on the PR branch (Agentbaker E2E179941072, Agentbaker GPU E2E179941075) withVHD_BUILD_ID=179936512+E2E_OMIT_CONTAINERD_VERSION=true, so baker takes the empty-version distro fallback → v3. On a 2.1.7 node (Test_Ubuntu2404_SecondaryNIC) the node's own containerd log confirms it accepted the v3 split schema and booted cleanly:Had the fallback emitted v4, containerd 2.1.7 would have rejected it (containerd fatal / CSE exit 84); instead it loaded the config and the node provisioned. The aggregate run is red only from running HEAD's full e2e against a 2-week-old VHD — ~81 scenarios fail a validator that expects the new aks-node-controller's
"Using NBC command for scriptless phase 2"log line (the old VHD's baked ANC predates it; the nodes booted fine — the validator SSH'd in to check), plus infra (the old branch didn't build the 2604 SKU, SIG replication timeouts, GPU capacity). No containerd/provisioning failures.The 2.0-2.1 brick is now directly disproven on a live containerd 2.1.7 node (above), in addition to the unit tests and the v3 fallback path.
Related
🤖 Generated by GitHub Copilot