Repository navigation
fix(arc): run the role controller's probe only on voters and eligible hosts - #200
Merged
Merged
Conversation
The psi-reporter DaemonSet mounted the host's /proc/pressure as a hostPath volume. On a kernel without pressure stall information that directory does not exist, the container runtime tries to create it under /proc, and the pod stays in CreateContainerError. /proc/pressure/memory, io and cpu are system-wide files, served by any procfs as /proc/meminfo is, so the reporter reads its own and nothing is mounted. A kernel without pressure stall information now gives a report with no pressure figures, which the autoscaler already reads as no pressure.
… hosts The probe DaemonSet ran on every Linux node with a kubelet, although the controller reads only a voter or a host the owner marks eligible and the probe removed its own entry anywhere else. A node that is neither (an Unraid host, an agent flashed from an image) only gave the pod a way to fail. A node affinity now schedules a pod only where the node carries k3s's etcd label or the eligible label the controller reads, so no new list is kept. A pod stopped on a node that has lost both labels, or is gone, removes the node's entry as it ends, which the pod's own removal on the next sample no longer can do. The probe also mounted the host's /proc/pressure, which a kernel without pressure stall information lacks, so the container runtime failed to create the mount and the pod never started on an eligible host of that kind. It now reads /proc/pressure/memory of its own procfs, and a node without it records the MemAvailable and MemTotal figures and no pressure figure, which the controller counts as not over the memory limit. check memory prints a null pressure there instead of failing.
Mearman
marked this pull request as ready for review
November 3, 2026 17:31
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
|
🎉 This PR is included in version 2.28.2 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The role controller's probe DaemonSet ran on every Linux node with a kubelet and mounted the host's
/proc/pressureas ahostPath. On a kernel without pressure stall information (an Unraid host) that directory does not exist, the container runtime tries to create it under/proc, and the pod sits inCreateContainerError. A node the controller never reads (not a voter, not eligible) still got a pod.Two changes, one per cause:
TERMhandler that re-reads the node first, so a rollout on a node that is still measured keeps its entry)./proc/pressure/memory, which is the node's as/proc/meminfois, so no volume is mounted. A node without pressure stall information recordsMemAvailableandMemTotaland no pressure figure, which the controller already counts as not over the limit;check memoryprints a null pressure there instead of failing.The
psi-reporterDaemonSet had the same mount and gets the same fix in its own commit.Tests: the manifest's affinity terms for a voter, an eligible host, both, neither and a changed label name; no pressure volume; the probe with and without a pressure file (the sample and
check memory); the termination handler for a node that lost both labels, a gone node, a node still eligible, a voter and a refused removal. Putting the volume back, removing the affinity or its eligible term, dropping the handler or its removal, and unguarding either read of the pressure file each fail the tests written for them.tests/fleet_role/image-smoke.shnow also checks, in a real container on CI, that the container sees/proc/pressure/memoryexactly when its host does.