[andrewpark-scaleai]

Why

The goal is a shorter Local backends run on the same 12 slow-tier runners. The run finishes when the slowest slow shard does, and that shard is usually rebuilding computer images another shard already built.

• Each shard saved its whole local registry to the Actions cache, keyed computers-<shard>-<week>-<tags>.
• In main run 42701339058 the tests used 15 distinct computer images, and those images were stored 25 times across 10 shards. One image sat on shards 3, 6 and 7. Four sat on both 8 and 12.
• Entries were 1.6 to 6.3GB each, against the repository's 10GB cache. They evicted each other, so only two shards' entries survived at a time, and every Monday the week in the key reset them all.
• In PR runs 42699224484 and 42699237630, shard 8 found no cache both times. It ran 1065s and 1126s of pytest, against the 596s the shard balancer planned. Every other shard ran within 0.94x to 1.22x of its plan.

What changes

The composer copies an image it would build from a mirror.

• _stored looks for a key's image in the env's image store, as before.
• When AGENT_ENV_COMPUTER_IMAGE_MIRROR is set and the store lacks the image, the composer reads the image anonymously from the mirror and copies it into the store. It then reuses the image as if it had been built earlier.
• The mirror tag is <composer digest>-<key>. The composer digest is composition(), which already covers the composer, the builder, the computers' build files and the protocol's computer package. So a change to the code that builds images misses the mirror and builds them again, and the build path stays tested.
• A mirror that lacks the key, refuses, or fails mid-copy leaves the image to be built, with a warning. The setting is empty by default, so nothing changes outside CI.
• The key is a digest of everything the image is built from, so a mirror image under a key is the image a build of that key makes.

registry_api.copy_image copies an image between registries by its digest.

• It sends only the blobs the target lacks, then each manifest byte for byte, so the copy keeps the source's digest. An index's platform manifests go first.
• A held token that the registry refuses is asked for once more. GHCR scopes its token to the first challenge, which is a pull's, so a push needs a second one.
• Blobs stream through a temporary file with a Content-Length. They are never held in memory.

The slow tier shares images through ghcr.io/scaleapi/agentenv-ci-computers.

• Every slow shard reads the mirror.
• Once a shard's tests pass on main, it shares each image it built or copied, under its mirror tag and <tag>-seen-<YYYYMMDD>. share_computer_images.py does this as GITHUB_TOKEN. An image already in the mirror is only tagged again. If anonymous reads fail, it warns that the package needs to be public.
• The job's checkout no longer persists its token, so only the share step, which runs on main, gets one. A failed share doesn't fail the job.
• The actions/cache restore, the "keep only the computer images" step, and the save of the registry are gone.
• The test packages folder from ci(slow tier): keep the computer tests' resolved packages between modules and runs #591 keeps its own per-shard weekly cache, saved only on main and named by the locks it holds. prune_registry.py and its test, which only trimmed the registry cache, are removed.

A daily prune deletes images unused for a week.

• computer-image-mirror.yml runs prune_computer_images.py daily at 06:17 UTC, or on demand with a dry run.
• It keeps every version whose newest seen- tag, or its upload date when it has none, falls within 7 days. It also keeps the platform manifests a kept index lists, because GHCR lists those as versions of their own. Every other version is deleted.

Before merge

• The package must stay public. Shards read the mirror anonymously. The first run created agentenv-ci-computers, and it's already readable without credentials.
• Delete permission. The prune deletes versions with the workflow's GITHUB_TOKEN, which needs the repository to hold admin access to the package. A package admin has been asked to grant it.
• Main-only sharing. This PR's runs pushed to the mirror once so a rerun could show hits. Sharing is now main-only again.

Measured on this PR

Run 42716509774 was measured twice. Attempt 1 started with an empty mirror and built everything. Attempt 2 found every image in the mirror.

	Attempt 1, empty mirror	Attempt 2, every image in the mirror
Image builds the composer logged, across the 12 shards	72	0
Copies from the mirror it logged	0	50
Slowest shard, job time	1238s (shard 8)	941s (shard 8)
Sum of the 12 shards' pytest time	8882s	7773s

Main's slowest slow shard has a median of 940s over its last 46 runs (800s to 1198s). Those runs mixed cache hits and misses from shard to shard. With the mirror, every shard reuses every image whose sources are unchanged, so a run stops depending on which cache entries survived. Shard 8 is still the slowest at 865s of pytest, against the 596s the durations table plans for it. Refreshing the table from a run with every image in the mirror is the next step.

Verification

• Unit tests.
  • The composer copies a missing key from the mirror and reuses it rather than building it.
  • A mirror that refuses leaves the image to be built.
  • copy_image sends only missing blobs and keeps each manifest's bytes.
  • The prune keeps used images and the platform manifests of kept indexes.
  • A change to the composer's code misses the mirror and builds.
  • A copy to GHCR asks again for a push token after its pull token is refused.
  • The share step tags each key, skips index tags, and warns when the mirror can't be read anonymously.
• Mutation checks. Each of these fails its test:
  • having the composer ignore the mirror;
  • the old retry rule, which kept a pull-only token for a push to GHCR;
  • having copy_image resend blobs the target has;
  • having the prune drop a kept index's platform manifests.
• Real registries.
  • copy_image copied the published 2-platform agentenv-db-web index from GHCR into a local registry:2 with the same digest, in 10.6s. A second copy, with the blobs already there, took 0.6s.
  • share_computer_images.py between two local registries pushed the key and its seen- tag, and skipped index-* tags.
• This PR's CI. The first run builds and shares the images. A rerun should copy them. That rerun is the measurement of the speedup.

Confidence Score: 5/5

The PR appears safe to merge, with no new blocking issues found.

Summary

The slow test runners now share computer images through GHCR instead of caching each runner’s registry.

• Computer builds reuse matching images from a shared mirror.
• Passing main runs share their computer images with every slow-tier shard.
• The mirror drops computer images unused for a week.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Composer needs an image] --> B{Local store has it?}
  B -->|Yes| F[Reuse and probe]
  B -->|No| C{Mirror provides it?}
  C -->|Yes| D[Copy into local store]
  D --> F
  C -->|No or copy fails| E[Build image]
  E --> F
  F --> G{Main tests pass?}
  G -->|Yes| H[Share image and add dated tag]
  H --> I[GHCR mirror]
  J[Daily prune] --> K[Keep recent images and their platform manifests]
  K --> I

Loading

_{Reviews (9) · Last reviewed commit: "fix(registry): refuse an upload a regist..." · Reviewed by Greptile}