[earakely-scale]

What and why

integration-local-slow fails now and then at _seed_bootstrap_envs() with a build error:

#7 naming to docker.io/library/gateway-default:latest done
#7 ERROR: image "docker.io/library/gateway-default:latest": already exists
ERROR: failed to build: failed to solve: image "docker.io/library/gateway-default:latest": already exists

It hit main in shard 10 (run 42921640696, db-mcp-default-db), and #598 twice in a row in the same shard (db-mcp-default-db, then gateway-default).

_seed_bootstrap_envs() builds four images under fixed tags: service-db-default-db, db-web-default-db, db-mcp-default-db and gateway-default. About a dozen integration test files call it, and since #600 the slow tier runs them two at a time on each shard, on workers that share one Docker daemon. In #598's failed shard, gw0 started local_parallel_deploy_test and gw1 started test_advance_to_next_event_hook_local about 20 s later. Both built the same four images in the same order.

When a tag doesn't exist yet, as on a fresh runner, BuildKit names the image by creating the tag. Two first builds of one tag at the same moment both try to create it, and the one that comes second fails with "already exists".

What changes

_seed_bootstrap_envs() holds an exclusive fcntl.flock on one lock file in the temp directory while it builds each image, so workers take turns. The tests already use this pattern for their shared wheel cache (tst/installer/conftest.py). The tags stay fixed, so no extra tags pile up on a developer's machine. Once one worker has built an image, the next worker's build comes from the layer cache, so waiting costs a few seconds.

Tests only, so no release.

How it was tested

On an Ubuntu 24.04 VM in agent-environments-admin with Docker 29.1.3 on the containerd image store, as on CI:

• The race: 6 processes built the gateway image under one new tag at the same moment, in 5 rounds, with the tag removed before each round.
  • Without the lock, 9 of 30 builds failed with image "docker.io/library/race-repro-gw:latest": already exists.
  • With the lock, 0 of 30 failed.
  • When the tag already existed before a round, no build failed either way, which is why the race shows up only on a fresh runner.
• The callers: four of them ran on 4 workers (-n 4): the two that collided in feat(multi): a multi env names its website browser (browser_envs), a browser computer on a sandbox of its own #598's shard, local_parallel_deploy_test.py and test_advance_to_next_event_hook_local.py, plus multi_env_local_test.py and build_context_local_test.py.
  • Main's version of _seed_bootstrap_envs, starting with none of the four tags: one of 2 runs failed as on CI, image "docker.io/library/db-mcp-default-db:latest": already exists, 1 failed and 5 passed.
  • With the lock, starting with none of the four tags: 3 of 3 runs passed, 6 of 6 tests each.
  • With the lock, starting with the tags already built: 2 of 2 runs passed.
• ruff check passes on the file.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge.

Summary

The PR makes bootstrap image builds take turns across test workers that share a Docker daemon.

• Bootstrap images build one at a time across test workers.

_{Reviews (4) · Last reviewed commit: "test: the bootstrap images are built one..." · Reviewed by Greptile}