Optimize sync for repositories with specific tag lists - #2475
Open
parmstro wants to merge 1 commit into
Open
Conversation
parmstro
force-pushed
the
upstream-pr-bypass-optimization
branch
4 times, most recently
from
August 21, 2026 19:00
5837e3f to
5889d3e
Compare
Add bypass optimization to skip expensive /tags/list pagination when syncing container repositories with specific (non-wildcard) tag lists. When include_tags contains only specific references without wildcards, and exclude_tags is empty or contains only harmless patterns that won't match the includes (e.g., *-source), the sync bypasses /tags/list enumeration entirely and fetches manifests directly. Cosign companion tags (signatures, attestations, SBOMs) are discovered via concurrent HEAD request probing instead of full tag enumeration, maintaining security artifact discovery while avoiding expensive pagination through deep tag lists. Performance impact: Reduces sync time from 3-8 minutes to ~20 seconds for repositories with 50,000+ tags when syncing specific digest references. Changes: - Add auto_discover_cosign field to ContainerRemote model (default True) - Implement _can_bypass_taglist() detection in ContainerFirstStage - Add _discover_cosign_companions_without_taglist() for HEAD probing - Add _tag_exists() helper for tag existence validation - Pass mirror parameter through synchronize() to ContainerFirstStage - Add database migration for auto_discover_cosign field - Add comprehensive unit test coverage closes pulp#2474
parmstro
force-pushed
the
upstream-pr-bypass-optimization
branch
from
August 21, 2026 19:04
5889d3e to
92978f0
Compare
Contributor
|
Thanks for the contribution, but we won't accept this as is. We do not allow adding migrations in older branches. It prevents upgrading to a newer version. From what I read it seems the slow part is the tag pagination fetching, did you try increasing the number of results for each tag request with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add bypass optimization to skip expensive /tags/list pagination when syncing container repositories with specific (non-wildcard) tag lists.
When include_tags contains only specific references without wildcards, and exclude_tags is empty or contains only harmless patterns that won't match the includes (e.g., *-source), the sync bypasses /tags/list enumeration entirely and fetches manifests directly.
Cosign companion tags (signatures, attestations, SBOMs) are discovered via concurrent HEAD request probing instead of full tag enumeration, maintaining security artifact discovery while avoiding expensive pagination through deep tag lists.
Performance impact: Reduces sync time from 3-8 minutes to ~20 seconds for repositories with 50,000+ tags when syncing specific digest references.
Changes:
closes #2474
Assisted by: Claude Sonnet 4.5 (model ID: claude-sonnet-4-5@20250929)