fix: stop SIG image-version GC from deleting every untagged version by ganeshkumarashok · Pull Request #9397 · Azure/AgentBaker

Ganeshkumar Ashokavardhanan (ganeshkumarashok) · 2026-09-04T02:23:29Z

ADO build 179633029 deleted 39 SIG image versions — including 2404gen2arm64gbcontainerd 1.2.103–1.2.106 (the GB200/GB300 images), 30 Flatcar versions, and five Windows definitions — from a failed AzureLinuxV3gen1 build that never produced an image.

Root cause

vhdbuilder/packer/cleanup.sh ages off SIG image versions with:

az sig image-version list ... | jq --arg dl $deadline ".[] | select(.tags.now < \$dl).name"

SIG image versions never carry a now tag. Packer applies azure_tags (which is where now comes from) to the managed image; the shared_image_gallery_destination block has no tags at all — see vhd-image-builder-arm64-gb.json and every other template.

So .tags.now is null for every version, and in jq null sorts below every string:

$ jq -n "(null < \"1787873664\")"
true

The filter is therefore unconditionally true. The "delete versions older than 168 hours" pass actually deletes the first 15 versions of every image definition in the gallery, on every linuxVhdMode build, regardless of age — including versions created minutes earlier. Only the resource-lock check limited the blast radius.

Reproduced against a fixture (deadline 2026-08-27):

version	published	before	after
`1.2.103`	2026-05-28	DELETE	DELETE
`1.2.999`	2026-09-03 (today)	DELETE ← bug	keep
`1.1755629795.32733`	2025-08-19	DELETE	DELETE
`260819.045326.24524`	2026-08-19	DELETE	DELETE

Fix

Age SIG versions off publishingProfile.publishedDate, which ARM always populates, with a null guard so anything lacking it is skipped rather than swept.

Two existing precedents in this repo confirm the shape of the fix:

the adjacent managed image loop already guards with select(.tags.now != null) — and managed images genuinely do carry the tag (packer azure_tags), so that loop is correct as-is and is left alone;
.pipelines/scripts/windows-sub-cleanup.sh:105 already ages SIG versions off publishingProfile.publishedDate.

This aligns cleanup.sh with both.

Second defect: the GB exemption was dead

if [ "${image_definition,,}" = "2404gen2arm64gbcontainerd" ] && [[ "$image_version" == "1.1."* ]]; then

The exemption also required the version to start with 1.1.. GB builds moved to 1.2.x, so it silently stopped protecting anything, which is why 1.2.103–1.2.106 were collected despite the guard being present. It now matches on the image definition, per the existing TODO (protect GB until it is released to official production galleries).

version	before	after
`1.1.99`	PROTECTED	PROTECTED
`1.2.103`–`1.2.107`	DELETED	PROTECTED

Testing

bash -n clean.
shellcheck run exactly as verify_shell.sh invokes it (bash mode, repo ignore list): exit 0 before and after; no new findings. SC3014 count actually drops 5 → 4 since a [[ ]] is removed.
jq filter behaviour verified against a fixture mirroring az sig image-version list output (table above).
date -u -d "@1787873664" +%Y-%m-%dT%H:%M:%SZ → 2026-08-27T23:34:24Z, verified with GNU date, matching the deadline in the failing run. ISO-8601 UTC compares correctly against ARM's ...+00:00 form.

Note on scope

This does not change the behaviour of the managed-image loop, the resource-group loops, or the storage-account loop; those operate on resources that really are tagged with now.

vhdbuilder/packer/cleanup.sh ages off SIG image versions with jq --arg dl $deadline '.[] | select(.tags.now < $dl).name' but SIG image versions never carry a "now" tag. Packer applies azure_tags to the managed image only; the shared_image_gallery_destination block has no tags. So .tags.now is null for every version, and in jq null compares less than any string, making the filter unconditionally true. The result is that the "delete versions older than 168 hours" pass really deletes the first 15 versions of every image definition in the gallery on every linuxVhdMode build, regardless of age, including versions created minutes earlier. Only the resource-lock check limited the damage. The adjacent managed-image loop already guards with select(.tags.now != null) -- and managed images genuinely do carry the tag, so that loop is correct. windows-sub-cleanup.sh already ages SIG versions off publishingProfile.publishedDate. This aligns cleanup.sh with both. Also fix the 2404gen2arm64gbcontainerd exemption, which additionally required the version to start with "1.1.". GB builds moved to 1.2.x, so the exemption silently stopped protecting anything and versions 1.2.103-1.2.106 were collected. It now matches the image definition. Observed in ADO build 179633029, where a failed AzureLinuxV3gen1 build ran cleanup and deleted 39 SIG versions across 2404gen2arm64gbcontainerd, flatcargen2, flatcargen2arm64 and five Windows definitions.

github-actions · 2026-09-04T02:25:55Z

Windows Unit Test Results

3 files 13 suites 51s ⏱️
409 tests 409 ✅ 0 💤 0 ❌
412 runs 412 ✅ 0 💤 0 ❌

Results for commit 3781e5e.

Copilot

🟡 Changes recommended

The destructive cleanup behavior and expanded exemption need automated ShellSpec regression coverage.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes SIG image-version garbage collection to prevent deletion of recent or protected images.

Changes:

Uses publishedDate with null handling for age checks.
Protects all GB image versions regardless of version prefix.

File summaries

File	Description
`vhdbuilder/packer/cleanup.sh`	Corrects SIG age filtering and GB exemptions.

Review details

Suppressed comments (1)

vhdbuilder/packer/cleanup.sh:183

🟡 Medium Risk — 🧪 Test Coverage: Removing the version condition materially broadens a deletion exemption, but no automated case locks in that both legacy 1.1.x and current 1.2.x GB versions remain protected. Please cover those versions (and a non-GB definition) in ShellSpec so the exemption cannot silently become dead or over-broad again.

      if [ "${image_definition,,}" = "2404gen2arm64gbcontainerd" ]; then

Files reviewed: 1/1 changed files
Comments generated: 1
Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

  set +x
  for image_definition in $(az sig image-definition list -g ${AZURE_RESOURCE_GROUP_NAME} -r ${SIG_GALLERY_NAME} | jq '.[].name' | tr -d '\"' || ""); do
-    for image_version in $(az sig image-version list -g ${AZURE_RESOURCE_GROUP_NAME} -r ${SIG_GALLERY_NAME} -i ${image_definition} | jq --arg dl $deadline '.[] | select(.tags.now < $dl).name' | head -n 15 | tr -d '\"' || ""); do
+    for image_version in $(az sig image-version list -g ${AZURE_RESOURCE_GROUP_NAME} -r ${SIG_GALLERY_NAME} -i ${image_definition} | jq --arg dl "$publishedDateDeadline" '.[] | select(.publishingProfile.publishedDate != null) | select(.publishingProfile.publishedDate < $dl).name' | head -n 15 | tr -d '\"' || ""); do


Sylvain Boily (djsly) · 2026-09-04T02:28:04Z

+      # NOTE: this deliberately matches the whole image definition. It previously also required the
+      # version to start with "1.1.", which silently stopped protecting anything once GB builds moved
+      # to 1.2.x, and 1.2.103-1.2.106 were garbage collected as a result.
+      if [ "${image_definition,,}" = "2404gen2arm64gbcontainerd" ]; then


This will never cleanup anything filling up the gallery which is already full of use 1.X at a minimum and add a logic that forces 1.X for the version. Blocks someone from creating 2.x

Copilot AI balanced review requested due to automatic review settings September 4, 2026 02:23

Copilot started reviewing on behalf of Ganeshkumar Ashokavardhanan (ganeshkumarashok) September 4, 2026 02:24 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Sylvain Boily (djsly) requested changes Sep 4, 2026

View reviewed changes

Ganeshkumar Ashokavardhanan (ganeshkumarashok) mentioned this pull request Sep 4, 2026

fix: stop SIG image-version GC from deleting every untagged version (backport of #9397) #9399

Open

Navigation Menu

Uh oh!

fix: stop SIG image-version GC from deleting every untagged version - #9397

fix: stop SIG image-version GC from deleting every untagged version#9397
Ganeshkumar Ashokavardhanan (ganeshkumarashok) wants to merge 1 commit into
mainfrom
ganesh/fix-sig-gc-published-date

Ganeshkumar Ashokavardhanan (ganeshkumarashok) commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Sylvain Boily (djsly) Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Ganeshkumar Ashokavardhanan (ganeshkumarashok) commented Sep 4, 2026

Root cause

Fix

Second defect: the GB exemption was dead

Testing

Note on scope

Uh oh!

github-actions Bot commented Sep 4, 2026

Windows Unit Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Sylvain Boily (djsly) Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants