feat(vhdbuilder): configure the Trident ACL Agent on ACL images - #9347
Draft
Paco Huelsz (frhuelsz) wants to merge 2 commits into
Draft
feat(vhdbuilder): configure the Trident ACL Agent on ACL images#9347Paco Huelsz (frhuelsz) wants to merge 2 commits into
Paco Huelsz (frhuelsz) wants to merge 2 commits into
Conversation
The Trident ACL Agent drives A/B OS updates on Azure Container Linux nodes, replacing the OS image in place rather than reprovisioning the node. The agent and its unit ship in the ACL base image, but the packaged unit carries no configuration and is not enabled, so a deployment supplies both. This adds the AKS-specific half. Configuration is delivered as a systemd drop-in. Only the settings whose defaults are wrong for AKS are set: the node annotation prefix, and the file the agent reads to learn the node's current image version. Everything else, including the kubeconfig path, node name, API server, timeouts and heartbeat interval, is already correct and is deliberately left alone. The agent compares the node's running version against the target version in an update request, and reads it from an os-release-formatted file. The VERSION_ID in /etc/os-release cannot serve, because it carries the OS release rather than the node image version that update requests specify, so the two are not comparable. The node image version is already recorded in image-bom.json, but that is JSON and the agent parses KEY=VALUE, so the same value is now also written to /opt/azure/containers/image-version. The build fails if the version is unavailable, rather than falling back to a date-derived value that would be silently wrong. Enablement uses a path unit rather than enabling the service. The agent authenticates to the API server with kubelet's kubeconfig and exits if that file is absent, which on a new node it is until kubelet finishes TLS bootstrapping. Enabling the service directly would start it on every boot before kubelet and leave it restarting until the file appeared. The path unit waits for the file instead, which is the actual precondition, and systemd activates the service immediately on later boots where the file already exists. This follows the existing update_certs.path pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The drop-in previously set only the settings whose defaults were wrong for AKS, leaving the rest to the agent. That makes node behaviour depend on the agent's defaults at the time the image is built, so an upstream default change would silently alter how AKS nodes update on the next rebuild. Set every setting the agent reads from the environment, including those whose current default already matches, so this file rather than the agent's defaults defines the behaviour. All pinned values were verified against the agent's current defaults; only the annotation prefix deliberately differs. The settings that cannot be pinned are now listed explicitly with the reason: node name and API server are per-node and per-cluster, the Nebraska settings are unused in annotation mode, and the Kubernetes watch poll interval has no environment variable at all. Records the trade this makes: a deliberate upstream change, such as a timeout that proves too short, no longer reaches AKS until this file is updated, so these values need reviewing when the Trident version in the base image moves. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Configures the Trident ACL Agent on Azure Container Linux node images. The agent drives A/B OS updates, replacing the OS image in place rather than reprovisioning the node.
The agent binary and its systemd unit ship in the ACL base image, but the packaged unit deliberately carries no configuration and is not enabled, so a deployment supplies both. This adds the AKS-specific half.
What this changes
Configuration, as a systemd drop-in at
/etc/systemd/system/trident-acl-agent.service.d/10-aks.conf.Every setting the agent reads from the environment is set explicitly, including those whose current default already matches. Relying on the agent's defaults would make node behaviour depend on whatever those defaults were when the image was built, so an upstream change could silently alter how AKS nodes update on the next rebuild. Pinning them means this file, rather than the agent's defaults, defines the behaviour.
The trade is recorded in the file: a deliberate upstream change, such as a timeout that proves too short, no longer reaches AKS until the file is updated, so these values need reviewing when the Trident version in the base image moves. All pinned values were verified against the agent's current defaults; only the annotation prefix deliberately differs.
Four settings are deliberately left unset, listed in the file with reasons: node name and API server are per-node and per-cluster, the Nebraska settings are unused in annotation mode, and the Kubernetes watch poll interval has no environment variable at all.
..._KUBERNETES_ANNOTATION_PREFIXacl.microsoft.com; the AKS annotation contract usesacl.azure.com. A mismatch is silent: the resource provider writes a key the agent never watches, and no update runs...._CURRENT_VERSION_PATH/_KEY..._CURRENT_VERSION_FALLBACK=error0.0.0, which parses cleanly but is always wrong.StateDirectory=/var/lib/trident-acl-agentbefore the agent starts.A node image version stamp at
/opt/azure/containers/image-version.The agent compares the node's running version against the target version in an update request, reading an os-release-formatted
KEY=VALUEfile.VERSION_IDin/etc/os-releasecannot serve: it carries the OS release (3.0.YYYYMMDD) rather than the node image version (YYYYMM.DD.PATCH) that update requests specify, so the two are not comparable.The correct value is already recorded in
image-bom.json, but that is JSON and the agent parsesKEY=VALUE, so the same value is now also written in the format the agent can read. The build fails if the version is unavailable rather than inheriting the date-derived fallback used elsewhere, since a plausible-but-wrong version would not be caught byFALLBACK=error.Enablement, via a path unit rather than by enabling the service.
The agent authenticates to the API server with kubelet's kubeconfig and exits if that file is absent, which on a new node it is until kubelet finishes TLS bootstrapping. Enabling the service directly would start it on every boot before kubelet and leave it restarting until the file appeared.
The path unit waits for the file itself, which is the actual precondition rather than a proxy for it. On later boots the file already exists and systemd activates the service immediately when the path unit starts, so first boot and subsequent boots take the same code path. This follows the existing
update_certs.pathpattern.Dependency, and why this is a draft
This configures a package that is not yet in the ACL base image. It should not merge until:
trident-acl-agentpackage. Note that installingtridentalone is not sufficient:trident-acl-agentis a separate subpackage whoseRequirespoints attrident, not the reverse.Merging earlier would break ACL VHD builds, because
systemctlEnableAndStart trident-acl-agent.pathwould reference a service unit that does not exist.Scope
isACLor lives in the ACL packer template.vhd-image-builder-acl-arm64.jsonis not updated. If the agent is in scope for arm64, that template needs the same two provisioner entries.Testing
bash -nclean on all modified shell scripts; the packer template parses as JSON.