fix: reject empty resolver state during LocalDNS startup by saewoni · Pull Request #9361 · Azure/AgentBaker

Saewon Kwak (saewoni) · 2026-09-01T16:26:09Z

Summary

Avoid treating an empty resolver configuration as successful DNS recovery.
Preserve the existing networkctl reload behavior without reconfiguring the primary interface.
Add coverage for empty and missing resolver state.

Problem

When LocalDNS exits, cleanup removes 70-localdns.conf and runs networkctl reload. The active resolver state can temporarily contain no nameservers, or can still contain 169.254.10.10 while the LocalDNS listener is unavailable.

The startup path waits for the LocalDNS address to disappear before reading upstream DNS servers. Previously, an empty /run/systemd/resolve/resolv.conf satisfied that check because it did not contain 169.254.10.10. LocalDNS then proceeded, failed to find upstream DNS servers, and exited with code 216.

Change

wait_for_localdns_removed_from_resolv_conf now requires both:

At least one nameserver is present.
The LocalDNS listener address is absent.

The change does not call networkctl reconfigure. Live testing showed that reconfiguring the default-route link can clear resolv.conf instead of restoring upstream DNS and can reproduce the same exit-216 failure one step later. The existing asynchronous networkctl reload behavior is retained.

Validation

bash -n parts/linux/cloud-init/artifacts/localdns.sh
bash -n spec/parts/linux/cloud-init/artifacts/localdns_spec.sh
git diff --check
Direct production-function check: a non-empty upstream resolver passes; an empty resolver fails.
ShellSpec coverage now verifies empty, comment-only, and missing resolver files do not report successful recovery.

Live AKS validation

A live Ubuntu 24.04 baseline was captured on the disposable AKS node:

Subscription: 359833f5-8592-40b6-8175-edc664e2196a
Resource group: sakwa-localdns-repro-scu-0710
Cluster: sakwa-localdns-repro-0710
Node: aks-sysnp-14424852-vmss000000
ActiveState=active
SubState=running
Result=success
NRestarts=0
ControlGroup=/localdns.slice/localdns.service
nameserver 169.254.10.10
CoreDNS listening on 169.254.10.10 and 169.254.10.11

The bounded fault-injection command was not run because the current identity lacks:

Microsoft.Compute/virtualMachineScaleSets/virtualMachines/runCommand/action

on the node VMSS instance. No live process was killed and no node state was changed.

A reviewer-provided E2E comparison showed that the earlier networkctl reconfigure implementation caused an empty-resolver failure on Ubuntu 24.04, Azure Linux V3, and ACL scenarios, while the known-good builds passed. The reconfigure implementation was removed from this PR.

Source PR

The reconfigure implementation was developed in the earlier PR #9360 branch but is intentionally not included here. This PR is based on main and contains only the resolver-wait correction.

github-actions · 2026-09-01T16:28:29Z

Windows Unit Test Results

3 files 13 suites 49s ⏱️
404 tests 404 ✅ 0 💤 0 ❌
407 runs 407 ✅ 0 💤 0 ❌

Results for commit c3c511b.

♻️ This comment has been updated with latest results.

Copilot

Pull request overview

Restores upstream DNS after LocalDNS cleanup by reapplying network configuration to the live interface.

Changes:

Reconfigures the default-route interface after network reload.
Adds success and failure ShellSpec coverage.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
`parts/linux/cloud-init/artifacts/localdns.sh`	Reapplies DNS configuration during cleanup.
`spec/parts/linux/cloud-init/artifacts/localdns_spec.sh`	Tests reconfiguration success and failure.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Ye Wang [msft] (yewmsft)

don't merge yet — the e2e gate is red because of this pr, not flake. networkctl reconfigure on the default-route link wipes resolv.conf instead of reapplying upstream dns, so localdns hits the same exit 216 one step later. details inline on localdns.sh:720.

scenario	this pr	9362	9328
localdns hosts plugin ubuntu 2404	❌ 216	✅	✅
localdns hosts plugin azure linux v3	❌ 216	✅	✅
localdns hosts plugin ACL	❌ 216	✅	✅
localdns hosts plugin ubuntu 2204 / 2604 minimal	✅	✅	✅

Test_DCGM_Exporter_Compatibility/Ubuntu2404 shows the same empty-dns trace, so this hits any scenario with localdns on, not just the localdns tests. 2204/2604 only pass because they win the race.

the wait is also wrong, and this pr is what exposes it. wait_for_localdns_removed_from_resolv_conf (localdns.sh:607) only checks that 169.254.10.10 is absent. an empty nameserver list satisfies that, so it returns success and logs "DNS configuration refreshed successfully" on an empty resolver, then replace_azurednsip_in_corefile finds nothing and exits 216. it should require a non-empty upstream list that does not contain the listener ip. worth fixing regardless of what happens to this pr.

the repro in the description is circular — the mock was wired to write the corrected resolver file, then the test observed the corrected resolver file. the real reconfigure does the opposite.

suggested direction: if networkctl reload really isn't converging on some nodes, fix the wait (require a non-empty upstream list, extend the 5s budget) instead of reconfiguring the default-route link. please get a live ubuntu 24.04 repro first to confirm reload-only convergence is actually the problem.

nit: commit body says (cherry picked from abcca13) — link the source pr in the description.

Ye Wang [msft] (yewmsft) · 2026-09-01T20:59:54Z

    echo "Reloading network configuration succeeded."

+    echo "Reconfiguring network interface ${DEFAULT_ROUTE_INTERFACE}."
+    eval "$NETWORKCTL_RECONFIGURE_CMD \"${DEFAULT_ROUTE_INTERFACE}\""


this is what's breaking e2e. networkctl reconfigure does not reapply the upstream dns — it tears the link's dns state down and resolv.conf comes back empty before dhcp re-supplies it:

Reconfiguring network interface eth0. Network interface reconfiguration succeeded. Waiting for localdns (169.254.10.10) to be removed from resolv.conf... DNS configuration refreshed successfully. Current DNS: No Upstream VNET DNS servers found in /run/systemd/resolve/resolv.conf. localdns.service: Main process exited, code=exited, status=216/GROUP

same 216 you are fixing, one step later. build 179175113, Test_LocalDNSHostsPlugin/Ubuntu2404 + /AzureLinuxV3 + /ACL. all three pass on 9362 (179183947) and 9328 (179179245).

Addressed by removing networkctl reconfigure. The live E2E evidence showed it clears resolv.conf rather than restoring upstream DNS, causing the same exit-216 failure. This PR now retains networkctl reload only and fixes the resolver wait to require a non-empty upstream nameserver list.

Ye Wang [msft] (yewmsft) · 2026-09-01T20:59:54Z

    fi
    echo "Reloading network configuration succeeded."

+    echo "Reconfiguring network interface ${DEFAULT_ROUTE_INTERFACE}."


this is not only the cleanup path. cleanup_iptables_and_dns is called at localdns.sh:1027 during startup, right before wait_for_localdns_removed_from_resolv_conf. that's the call that breaks. please fix the title and description too — "after cleanup" hides where this actually runs.

Addressed. cleanup_iptables_and_dns runs during startup as well as shutdown, so the PR description now calls this out explicitly. The reconfigure change was removed because live E2E showed it causes empty DNS state during startup.

Ye Wang [msft] (yewmsft) · 2026-09-01T20:59:54Z

+    eval "$NETWORKCTL_RECONFIGURE_CMD \"${DEFAULT_ROUTE_INTERFACE}\""
+    if [ "$?" -ne 0 ]; then
+        echo "Failed to reconfigure network interface ${DEFAULT_ROUTE_INTERFACE}."
+        return 1


this aborts the rest of shutdown. cleanup_localdns_configs does cleanup_iptables_and_dns || return 1 (line 742), so a failed reconfigure skips the SIGINT to coredns and skips ip link del localdns — after the iptables rules are already removed. node ends up in a worse state than the bug. on the cleanup path, log a warning and continue, don't return 1.

Addressed by removing the reconfigure call entirely. Cleanup now retains the prior reload-only behavior, so a failed reconfigure cannot interrupt CoreDNS shutdown or dummy-interface removal.

Ye Wang [msft] (yewmsft) · 2026-09-01T20:59:54Z

+# Reapply the loaded network configuration to the default-route link after removing
+# the LocalDNS drop-in. Reload only refreshes networkd's configuration; reconfigure
+# causes the live link state, including DNS settings, to be reapplied.
+NETWORKCTL_RECONFIGURE_CMD="networkctl reconfigure"


why do we need this on the shutdown path at all? the comment at localdns.sh:1032 says shutdown doesn't read resolv.conf afterward. bouncing the primary nic on every localdns stop and every watchdog restart is a lot of blast radius for a dns daemon. what happens to the dhcp lease and to in-flight kubelet/containerd connections?

Addressed. The primary interface is no longer reconfigured on shutdown or watchdog restart. The PR now keeps the existing reload-only behavior to avoid DHCP lease and in-flight connection blast radius.

Ye Wang [msft] (yewmsft) · 2026-09-01T20:59:54Z

+            NETWORK_DROPIN_FILE="/tmp/test-network-dropin.conf"
+            touch "$NETWORK_DROPIN_FILE"
+            NETWORKCTL_RELOAD_CMD="true"
+            NETWORKCTL_RECONFIGURE_CMD="false"


these tests can't catch this class of bug — reconfigure is mocked as a no-op true/false, the real one wipes resolv.conf. a green shellspec is not evidence the fix works; only a live node is.

also missing:

empty DEFAULT_ROUTE_INTERFACE — cleanup_iptables_and_dns only calls initialize_network_variables when the DROPIN vars are empty (localdns.sh:665), so add DEFAULT_ROUTE_INTERFACE to that condition or you can end up running networkctl reconfigure "".

assert reload runs before reconfigure.

Addressed. Removed the mocked reconfigure tests with the unsafe implementation, added empty/comment-only/missing resolver coverage, and added DEFAULT_ROUTE_INTERFACE to the initialization guard. The PR description now documents that the earlier simulation was not valid and that live E2E showed reconfigure clearing DNS.

Copilot

🟢 Approval recommended

The startup guard correctly rejects unusable resolver states and is covered by focused regression tests.

Review details

Files reviewed: 2/2 changed files
Comments generated: 0 new
Review effort level: Balanced

Copilot AI balanced review requested due to automatic review settings September 1, 2026 16:26

Saewon Kwak (saewoni) requested a review from Ye Wang [msft] (yewmsft) as a code owner September 1, 2026 16:26

Copilot started reviewing on behalf of Saewon Kwak (saewoni) September 1, 2026 16:32 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Saewon Kwak (saewoni) requested a balanced review from Copilot September 1, 2026 17:19

Copilot started reviewing on behalf of Saewon Kwak (saewoni) September 1, 2026 17:20 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Ye Wang [msft] (yewmsft) reviewed Sep 1, 2026

View reviewed changes

fix: reject empty resolver state during LocalDNS startup

c3c511b

Copilot AI review requested due to automatic review settings September 1, 2026 22:31

Saewon Kwak (saewoni) force-pushed the fix/localdns-dns-reconfigure branch from dad5b69 to c3c511b Compare September 1, 2026 22:31

Saewon Kwak (saewoni) changed the title ~~fix: restore upstream DNS after LocalDNS cleanup~~ fix: reject empty resolver state during LocalDNS startup Sep 1, 2026

Copilot started reviewing on behalf of Saewon Kwak (saewoni) September 1, 2026 22:31 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Uh oh!

Conversation

Saewon Kwak (saewoni) commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Change

Validation

Live AKS validation

Source PR

Uh oh!

github-actions Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Windows Unit Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Ye Wang [msft] (yewmsft) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Saewon Kwak (saewoni) commented Sep 1, 2026 •

edited

Loading

github-actions Bot commented Sep 1, 2026 •

edited

Loading