test: consolidate integration runtime fixtures under Devsy GHCR #1449

Current status — October 10, 2026

Phase A is complete: PR #1450 merged as signed squash 511d21603e602d3081a7df8a19521b481b45adeb, and focused issue #1451 is closed. Both merge-commit workflows passed: primary CI and Desktop. The actual main Compose suite executed 55 cases successfully, including both shutdown cases and missing-app recovery. Final-head CodeRabbit and Greptile reviews completed before merge.

This broader tracker remains open for Phases B/C and the inventory guard. Their destination availability/digests and publication prerequisites below remain unverified; no registry images were published as part of Phase A. The October 9 local-failure evidence below is historical. Phase A was accepted on its reviewed, scoped CI-first evidence; this does not claim a full macOS suite pass or resolve the separate UID test correction #1452.

Problem and evidence

Integration tests still pull incidental runtime fixtures from Docker Hub. Consolidating those fixtures under ghcr.io/devsy-org/test-images will reduce upstream pull-rate exposure and give Devsy control over the tested environments, while preserving existing regression coverage.

Source baseline: main 56485803a98ccbf0505b1b42b8ecc12629ccfaf9. Manifest observations below were made on 2026-10-09. Existing issues and open PRs were checked for a matching consolidation effort before creating this tracker.

Phase A: existing-image Compose shutdown sidecars

Focused child issue: #1451, implemented by merged #1450. PR #1450 closes the focused issue; this broader tracker remains open.

• Validate the existing public image ghcr.io/devsy-org/test-images/base:alpine@sha256:fbd195076b0c5d37580debed78de5e4dd238436cd5aa348339a4d48463ff71ae for the sleep-only sidecars in e2e/tests/up-docker-compose/testdata/docker-compose-shutdown-action/docker-compose.yaml and docker-compose-shutdown-action-container/docker-compose.yaml.
• Compare anonymous index/child/config evidence, default user, entrypoint and working directory, then exercise the original and candidate sleep infinity containers with Compose.
• Change only the two sidecar image values after those checks pass. Retain the app images, commands, mounts, provider code, test bodies and timeouts.
• Run the actual stopCompose and stopContainer regression cases, the reused missing-app secret fixture case and the full up-docker-compose label. Record passed, failed and blocked gates separately.

This candidate is a richer Alpine 3.23 Dev Container base, not a byte-identical bare Alpine mirror. Reuse is conditional on the role-specific checks above; a semantic discrepancy requires reverting these two substitutions. No new image publication or inventory guard is a prerequisite for this bounded slice. This tracker stays open for the remaining phases; the Phase A PR closes #1451 without claiming to complete all consolidation.

Phase B: bare runtime mirrors and inventory guard

Use byte-preserving mirrors of these four source snapshots. Destination names are proposed publication targets, not verified available images. Anonymous GHCR token requests for each target returned HTTP 403; that establishes inaccessible/unverified status, not absence. Do not substitute until exact destination manifests and digests are recorded.

All four source indexes contain Linux amd64 and arm64 manifests. Copy all source platforms, preserve manifest/config/layer bytes, and retain upstream source/license/provenance records. No custom Dockerfile or rebuild is needed for these mirrors. Resolve future updates once to immutable source digests; do not copy a mutable tag after recording a different snapshot.

Existing reusable families

Public manifest reads verified the following exact indexes and Linux amd64/arm64 coverage. Retain current consumers; this first phase does not globally repin them.

base:ubuntu is a Ubuntu 24.04 Dev Container image; base:alpine is an Alpine 3.23 Dev Container image. They carry common-utils/git metadata and a vscode remote user, and must not replace bare fixtures without a separate semantic decision. Bare Devsy ubuntu:latest reports Ubuntu 24.04 whereas current upstream ubuntu:latest reports 26.04. Similarly named tags are not equivalent artifacts.

Existing docker:dind returned HTTP 401; digest/platform availability is unknown. No permission or credential change is part of this issue. No test-image publisher was found in current Devsy source or accessible organization code search; an external publisher may exist. Existing base/Go/Python attestations identify upstream build materials, but their builder identity is blank and no Devsy publishing workflow was verified. Do not present those observations as verified publisher provenance.

Phase B publication prerequisites

The authorized publisher should first establish whether each proposed target already exists using current permitted access. A 401/403, timeout, or transport failure must remain distinct from a missing-manifest response.

1. Mirror each immutable source above, preserving all platform manifests and original configurations/layers. With an already available Skopeo, the command form is skopeo copy --all --preserve-digests docker://SOURCE@sha256:DIGEST docker://DESTINATION:TAG; this issue does not request automated publication or tool installation.
2. Verify destination by tag and digest, compare index/media type and per-platform child digests, and record the observed destination digest. A registry transformation requires investigation rather than assumed equality.
3. Verify anonymous manifest and blob pull access for Linux amd64 and arm64. Record upstream project/license links and SBOM/provenance/referrer availability; a basic image copy does not establish that every referrer was preserved.
4. Supply the observed destination index and platform digests before changing consumers to tag@sha256:destination_digest. Preserve old artifacts/tags for rollback.
5. Run source/destination fixture smoke checks before substitution.

Phase C: service-specific mirrors

PostgreSQL and nginx Compose services require their exact service-image snapshots, configuration and startup behavior. These proposed targets remain inaccessible/unverified, not confirmed missing.

All paths above are under e2e/tests/up-docker-compose/testdata/. Apply the same all-platform, byte-preserving publication and verification prerequisites as Phase B. Generic base images are not service replacements. Infrastructure and build parents remain separately scoped.

Preservation boundaries and later scope

• Keep the snapshot destination host.docker.internal:15500, insecure-local behavior, delete-enabled environment, port binding, readiness/retry/cleanup behavior and snapshot assertions unchanged. Only its SERVER image is eligible for migration.
• Keep Ubuntu 22.04 strings in extends/feature/outdated/upgrade metadata, synthetic Docker Hub error URLs, intentional auth/registry fixtures, external feature/template artifacts and locally built image names unchanged.
• Keep upstream build parents separate: Kubernetes multi-stage FROM ubuntu, UID-mapping FROM ubuntu:latest, and Kubernetes SSH helper FROM alpine:3.22. The UID fixture creates vscode; the richer existing Ubuntu image would change that setup.
• The delivery E2E also constructs LocalDockerDelivery without its existing HelperImage override, so internal populate/version-check containers still use the production busybox:latest default. Future migration must set that test-level field to the same verified mirror; changing only the two visible literals is insufficient. Keep the production default unchanged. Additional integration pulls in pkg/agent/delivery/delivery_integration_test.go remain separately inventoried. The alpine:latest constant in pkg/driver/apple/lifecycle_test.go feeds mocks and is retained as synthetic unit-test input, not counted as a runtime pull.
• Later runtime candidates include PostgreSQL/nginx Compose services, unqualified Docker machine/tunnel helpers, Alpine 3.22 Kubernetes route probes, and provider-specific MicroSandbox images. Preserve each service's actual behavior; do not treat generic base images as service replacements.
• Keep Kind kindest/node:v1.36.4@sha256:099e049362a1526b2db71494e1947aae99bd16290d7c895f2b7ea312e3cbfaed separate as infrastructure. Windows jobs use Linux containers; a Windows-native image build is not required.
• Exclude all six files owned by fix(dockerless): preserve image environment after credential cleanup #1447, all MicroSandbox/CI changes owned by test(runtime): cover MicroSandbox image backend parity #1448, unrelated driver/runtime/provider work, pkg/devcontainer/single.go, and .github/workflows/pr-ci.yml. Recheck fresh main and PR file overlap before future implementation. Do not add a competing workflow change.

Acceptance and delivery

• Four Phase B targets have immutable source/destination evidence, verified publisher ownership/source mapping, matching index/child/config/layer digests for all source platforms, and anonymous amd64/arm64 manifest/blob reads.
• Make only the Phase B fixture edits above (the two shutdown sidecars are needed only if Phase A cannot safely reuse the existing image), preserving commands, users, mounts, cleanup and regression assertions, including the existing test-level delivery HelperImage override described above.
• Record pending versus verified images in a versioned lock; add an offline inventory/drift guard with exact path/context exceptions. New upstream runtime references and migrated references without verified digests must fail. No global registry or all-FROM exemption.
• Guard tests cover JSON/JSONC, YAML service images, Dockerfile stages/scratch, Go constants, exact metadata/error exceptions, missing consumers/stale exceptions, malformed digests and platform/digest drift. Offline checks perform no network or container operations.
• Smoke-check Alpine shell/sleep/temp-file behavior, Alpine 3.20 package/SSH behavior, BusyBox mounted executable delivery, and exact registry startup/snapshot push/restore/cleanup.
• Existing config-apply, delivery, ssh-agent-forward, snapshot, and up-docker-compose cases pass with existing timeouts and oracles. Run applicable unit/framework tests, lint/hooks/analyzer, CodeRabbit and independent review. Report blocked/never-run gates accurately.
• Future code delivery uses an isolated branch, existing personal GPG/SSH signed commits verified locally and by GitHub, and an initially Draft PR referencing this issue. No merge before final-head review/CI.
• CI guard wiring waits for the active workflow owner; final-head hosted CI/review and protected merge remain separate delivery gates.

Historical Phase A evidence (2026-10-09; superseded status above)

Anonymous Docker pulls and disposable Compose checks passed for the immutable bare-Alpine source and the existing pinned candidate on Linux arm64 and amd64. Both use root at runtime, working directory /, no entrypoint/stop-signal override, and PID 1 sleep under the unchanged sleep infinity command. Mounted executable/service behavior is not inferred from this narrow smoke test. Stop-app preserved the running sidecar; stop-all stopped both. Candidate child digests: amd64 sha256:ef7596ce0829a63c77e5408468b632e4d7628d8d144fd6e2f6bd3e03f27b46aa; arm64 sha256:7c450a774d8d1346a42ac19152d006b0368f05c36c80184d71acffaf407d296e.

Draft PR #1450 implements exactly the two sidecar pins in commit 81653971450e7021b1fc3c1469dcc950b70e5fc6. The existing personal GPG signature verifies locally and on GitHub (verified: true, reason: valid). It closes focused child #1451 and leaves this broader tracker open.

The unchanged Devsy stopCompose, stopContainer and missing-app secret recovery cases passed in both the focused run (3/3) and full label. A read-only observer confirmed both pinned shutdown sidecars running before the stop assertions. Relevant framework/Compose unit tests, applicable pre-commit/commit-message/pre-push hooks and task cli:lint:ci passed (0 lint issues). Independent local review found no defect. CodeRabbit reviewed both changed files after its free OSS capacity reset and completed with 0 findings. CodeScene does not support YAML; no score is claimed.

The complete up-docker-compose label ran 55 cases in 798.527 seconds: 49 passed, 6 failed. The unresolved failures are:

• Two unchanged UID-mapping checks: www-data remains UID 33 and vscode UID 1001 where the tests expect host UID 501. Both reproduced separately with test code/fixtures verified identical to main.
• Three unchanged Feature-copy cases: features, features with build target, and does not retag shared image when applying features to image backed services fail on macOS with lchown: operation not permitted. A separate fresh-state selection reproduced all three failures (4 selected cases: 1 passed, 3 failed).
• should start a new workspace with multistage build failed after its unchanged Debian HTTP package-download attempts exhausted the configured retries; apt-get update && apt-get install -y vim exited 100 after connection timeouts. No retry or timeout was weakened.

None of those failing fixtures consumes the changed shutdown sidecars. All six failure identities also reproduce against unchanged current main 56485803a98ccbf0505b1b42b8ecc12629ccfaf9. The package case fails on its single bounded rerun with existing sources and configured retries unchanged. No regression attributable to the two sidecar substitutions was identified. The broad full-suite acceptance gate remains unsatisfied; these baseline failures are not waived. Hosted final-head review/CI and merge remain pending. Phase B/C publication prerequisites and the later inventory guard remain open; no registry image has been published.