Replace InternalMirrorRegistry and PublicMirrorRegistry with a single MirrorRegistry#2120
Draft
lbussell wants to merge 10 commits into
Draft
Replace InternalMirrorRegistry and PublicMirrorRegistry with a single MirrorRegistry#2120lbussell wants to merge 10 commits into
InternalMirrorRegistry and PublicMirrorRegistry with a single MirrorRegistry#2120lbussell wants to merge 10 commits into
Conversation
copyBaseImages already imports every base image into the internal
staging ACR ('mirror/' prefix) immediately before getStaleImages runs,
but getStaleImages was still resolving FROM tags against docker.io.
That path is unreachable from the internal 1ES pool, so the command
hangs for 30s on each Docker Hub base image and the job fails.
Rewrite any non-MCR / non-*.azurecr.io FROM reference to the staging
mirror via --base-override-regex/--base-override-sub. The job already
authenticates to InternalMirrorRegistry via reference-service-connections,
so no credential changes are needed. Also drops the buildtools-only
override (which never matched the actual library/<distro> FROM lines).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous --base-override-regex/sub approach rewrote external FROM tags to point at the staging mirror, but the rewritten repo prefix also leaked into the digest comparison string. image-info.json stores the digest against the canonical (public) repo, so every rewritten image compared unequal and was reported stale on every run. Switch getStaleImages to the same mechanism the build/matrix flow already uses: - Add --registry-override and --source-repo-prefix options (mirroring what ManifestOptions exposes and what copyBaseImages consumes). - Construct ImageNameResolverForMatrix per subscription manifest. GetFromImagePullTag returns the staging mirror location for fetching the digest; GetFromImagePublicTag returns the canonical reference used to build the digest comparison string. The pipeline yml now passes --registry-override / --source-repo-prefix in place of the regex pair, matching how the copyBaseImages step in the same job is invoked. --base-override-regex/sub remains supported for genuine one-off overrides. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two different FROM spellings can normalize to the same pull tag (e.g. 'almalinux:8' and 'library/almalinux:8' both pull from '<staging>/mirror/library/almalinux:8') but produce different public tags. The previous code cached the full '<repo>@<sha>' comparison string by pull tag, which meant the second lookup could reuse the first FROM's public repo prefix and falsely mark the image as stale. Cache only the raw SHA so the comparison string is always built from the current platform's own public tag. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ride-sub The same redirect-to-mirror behavior is expressible via the existing --registry-override + --source-repo-prefix pair, which is type-checked rather than an opaque regex/replacement and is already used by every non-trivial caller (build, copyBaseImages, generateBuildMatrix matrix flow). The regex form survived as a parallel mechanism but is no longer needed. Changes: - Delete BaseImageOverrideOptions and its two ApplyBaseImageOverride call sites in ImageNameResolver (GetFromImagePublicTag and GetFromImageTag). - Drop BaseImageOverrideOptions from BuildOptions, BuildCommand, GenerateBuildMatrixOptions, GenerateBuildMatrixCommand, CopyBaseImagesOptions, CopyBaseImagesCommand, GetStaleImagesOptions, GetStaleImagesCommand. - Update init-common.yml public-build branch to use '--source-repo-prefix "" --registry-override <public-mirror>' instead of the regex pair. - Delete the three tests that exercised the regex form (BuildCommand_MirroredImages_BaseImageTagOverride, CopyBaseImagesCommand.OverridenBaseTag, GetStaleImagesCommand_BaseImageTagOverride). Their scenarios have no semantic analogue under the registry/prefix system. - Document the breaking change in eng/docker-tools/CHANGELOG.md, including migration notes for downstream repos and a note about the inert custom override file in dotnet-buildtools-prereqs-docker. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the conflated --registry-override/--source-repo-prefix pair
(and the regex-based --base-override-regex/--base-override-sub form
removed in earlier commits) with a single MirrorRegistry field on
PublishConfiguration. The pipeline templates choose the appropriate
mirror (internal staging vs. public mirror) at template-compile time
based on the AzDO team project, so the C# app sees one registry to
redirect external base-image lookups to without any runtime
conditionals or per-command CLI plumbing.
Key change in ImageNameResolver: external FROM tags are now rewritten
using the mirror server (not Manifest.Registry), so source redirection
no longer leaks into the destination tag of built images. This fixes
the previous bug where setting --registry-override on public builds
would also rewrite the push destination.
Surface changes:
- New RegistryEndpoint.RepoPrefix field; new PublishConfiguration.MirrorRegistry.
- Removed PublishConfiguration.InternalMirrorRegistry/PublicMirrorRegistry
(the YAML side keeps them for project-specific dispatch; the C# binder
ignores unknown JSON keys).
- Removed --source-repo-prefix CLI option from build/buildMatrix/getStaleImages.
- Removed --registry-override CLI option from getStaleImages (was added
earlier in this branch and is no longer needed for stale detection).
- publish-config-prod.yml + publish-config-nonprod.yml: emit MirrorRegistry
conditionally based on ${{ variables['System.TeamProject'] }}.
- init-common.yml: drop public-build override branch (now handled in publish
config); internal branch keeps --registry-override only.
- check-base-image-updates.yml: drop --registry-override/--source-repo-prefix
from the getStaleImages invocation.
- BuildCommand, GenerateBuildMatrixCommand, GetStaleImagesCommand take
IOptions<PublishConfiguration> via DI.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is based on #2119. It follows the changes in that PR to their natural conclusion by first removing the unused/old base image regex override CLI options in favor of
PublishConfiguration, and then unifiesPublicMirrorRegistryandInternalMirrorRegistryinto a singleMirrorRegistryproperty.In effect, this means that mirror registry info is always read from
PublishConfiguration, and ImageBuilder never has to know about the differences between internal vs. private configuration (that would be an anti-pattern).