fix(e2e): gate phase 3 VHD-sourced env tolerance on VHD provenance

[Ganeshkumar Ashokavardhanan (ganeshkumarashok)]

Stacked on #9244 — base is ganesh/fix-e2e-phase3-gpu-vhd-skew, so this diff shows only the gate. GitHub will retarget to main when #9244 merges.

Addresses Lily's feedback that the fix should key off VHD provenance and not be GPU-specific.

Problem with the tolerance as merged in #9244

#9244 tolerates phase 3 diffs confined to GPU_DRIVER_VERSION / GPU_IMAGE_SHA, because the two sides have different sources of truth:

path	source
provision-config	/opt/azure/components.json, baked into the VHD (aks-node-controller/app.go:246 → getCSEEnv(gpuConfig) → parser/helper.go:564,574)
nbc-cmd	cse_cmd.sh:159 {{GPUDriverVersion}} → pkg/agent/baker.go:1485 — compiled from the source under test

That tolerance is only legitimate when the two sides genuinely come from different commits. It is applied unconditionally today, including in the VHD builder pipelines — where the VHD was built from the source under test and the two sides are expected to agree exactly. There, the tolerance would silently mask a real GPU env bug.

Fix

Gate the tolerance on a signal that already exists, rather than adding new plumbing.

.pipelines/scripts/e2e_run.sh:83-86 exports SIG_VERSION_TAG_NAME=buildId only when VHD_BUILD_ID is set, and the VHD builder pipelines set VHD_BUILD_ID: $(Build.BuildId) (.vsts-vhd-builder.yaml:351, .vsts-vhd-builder-release.yaml:941, .build-and-test-windows-vhd-template.yaml:110). So the VHD selected by that tag is the one built earlier in the same run, from the same commit.

New config.Configuration.VHDBuiltFromSourceUnderTest() captures exactly that, and unexpectedEnvCompareDiffVars now takes it as a parameter:

e2e run	VHD source	GPU env skew
standalone PR check	built from main	tolerated — driver bumps stay mergeable
VHD-build-triggered	built from this commit	fails — both sides must agree

Every non-VHD-sourced var still fails in both modes.

Why gate the tolerance rather than skip the validator

The alternative discussed was skipping ValidateScriptlessPhase3 entirely on the standalone check. That drops all phase 3 coverage from every PR, since the standalone check is the one that runs on each push. Gating only the tolerance keeps the whole comparison enforced on every PR and restores strictness where it actually matters, so it is strictly stronger.

Note on generality

vhdSourcedEnvVars is a deliberately narrow allowlist rather than a blanket rule. getCSEEnv takes exactly two inputs that are not derived from the source under test — gpuConfig (these two vars) and detectContainerdVersion (parser/parser.go:47) — and the GPU pair is the only one sourced from a repo-tracked manifest baked at VHD build time, which is what makes a version bump deadlock. If another such var appears, the failure names it explicitly, which is the discoverable outcome.

isExpectedDiffCSEVar (aks-node-controller/app.go:29) remains untouched on purpose: adding these vars there would suppress the production Kusto signal, whereas the artificial condition is E2E's alone.

Tests

• TestVHDBuiltFromSourceUnderTest — asserts the provenance signal matches what e2e_run.sh exports (buildId / branch / empty)
• TestUnexpectedEnvCompareDiffVars — now table-driven over both provenance modes, including GPU vars are NOT tolerated when the VHD was built from this source
• TestEnvCompareGPUBumpIsTolerated — the real build 176997071 log line is tolerated on the standalone check and still fails when the VHD is same-source

Both directions of the gate were verified load-bearing by mutation:

mutation	test that caught it
VHDBuiltFromSourceUnderTest always false	TestVHDBuiltFromSourceUnderTest/buildId_tag...
tolerance ignores provenance	TestUnexpectedEnvCompareDiffVars/GPU_vars_are_NOT_tolerated..., TestEnvCompareGPUBumpIsTolerated

go vet ./... clean in e2e/; tree restored and byte-identical after mutation runs.

[github-actions]

Windows Unit Test Results

  3 files   13 suites   43s ⏱️
409 tests 409 ✅ 0 💤 0 ❌
412 runs  412 ✅ 0 💤 0 ❌

Results for commit 1622881.