fix(windows): wait for kubelet healthz after node reset - #9356
Closed
r2k1 wants to merge 1 commit into
Closed
Conversation
NSSM Running/Paused does not mean kubelet initialized. Poll local /healthz after k8s-restart-job so CSE tolerates the containerd pipe race and fails before RP node readiness when kubelet never comes up. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
Windows Unit Test Results 3 files 13 suites 54s ⏱️ Results for commit 07b5ffc. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds reliable Windows kubelet health validation after node reset.
Changes:
- Polls kubelet
/healthzwith bounded retries. - Makes kubelet startup failures terminating while restoring HNS remediation.
- Adds Pester coverage for success, retry, exhaustion, and task failure.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
staging/cse/windows/provisioningscripts/windowsnodereset.ps1 |
Improves kubelet startup handling and cleanup. |
parts/windows/windowscsehelper.ps1 |
Replaces service-state validation with health polling. |
parts/windows/windowscsehelper.tests.ps1 |
Tests the new health-check behavior. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Windows CSE cannot use NSSM service state to decide whether kubelet is healthy. NSSM reports
Runningafter it launches the process andPausedduring restart delay. A healthy recovery after the containerd named-pipe race can fail a one-time service check, while a kubelet that is about to crash can pass it.RP node readiness eventually replaces the node, but only after extra time and API calls.
This PR:
k8s-restart-jobto succeed, then polls kubelet's localhttp://127.0.0.1:10248/healthzWINDOWS_CSE_ERROR_START_NODE_RESET_SCRIPT_TASKif kubelet never becomes healthyStart-Service kubeletterminating and logs direct SCM failuresfinally, including when kubelet startup failsThe health check runs from current CustomData and works with old VHDs. The stricter
Start-Servicehandling is VHD-baked and applies to updated VHDs./healthzproves local kubelet health. It does not prove node registration or KubernetesReady. RP still owns that check.Validated that AKS does not override kubelet
--healthz-port/--healthz-bind-addressfor Windows:defaults_kubelet.goleaves the kubelet defaults (127.0.0.1:10248)CustomKubeletConfigdoes not expose healthz settings/healthzby default through current supported versionsRelated: #9354, #8970, #9344
Which issue(s) this PR fixes:
N/A
Testing:
windowscsehelper.ps1,windowscsehelper.tests.ps1, andwindowsnodereset.ps1/healthzsuccess, retry through transient failures, exhausted retries, and skipping the health check when the reset task itself fails