Skip to content

Commit a08f00a

Browse files
ci: Run the smoke test VM with gcloud instead of Vagrant
smoke-test-vm.sh creates the VM with gcloud, copies the repository to it over SSH, runs test/smoke-test.sh and test/executor-smoke-test.sh, and asks GCP to delete the VM without waiting. GCP deletes each VM an hour after it starts, and each job deletes the VMs and disks older jobs left behind. The boot disk is 20 GB. If the VM stops answering SSH while the repository is copied to it, the script retries the copy on the same VM. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5 Co-authored-by: Amp <amp@ampcode.com>
1 parent 4706f1b commit a08f00a

7 files changed

Lines changed: 130 additions & 137 deletions

File tree

‎.buildkite/pipeline.yaml‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
11
$schema: "http://json.schemastore.org/buildkite"
22
env:
3-
# GCP VM image for the Vagrant smoke test. Build it in sourcegraph/infrastructure by running `make e2e` in `gcp/images`
3+
# GCP VM image for the smoke test. Build it in sourcegraph/infrastructure by running `make e2e` in `gcp/images`
44
IMAGE: "e2e-db8a40720-2024-08-26t08-44-10z"
5-
VAGRANT_RUN_ENV: "CI"
65
steps:
76
- label: ":docker:"
87
command: .buildkite/validate-docker-compose.sh
@@ -20,9 +19,11 @@ steps:
2019
# so it finds no changes in a commit which is already on main, unless it is the newest
2120
- &smoke_test
2221
label: ":rice: docker-compose-test"
23-
command: .buildkite/vagrant-run.sh docker-test
22+
command: .buildkite/smoke-test-vm.sh
2423
artifact_paths: ./*.log
2524
agents: { queue: "vagrant" }
25+
# smoke-test-vm.sh deletes VMs and disks older than an hour, so no job may run that long
26+
timeout_in_minutes: 30
2627
if: build.branch == "main"
2728

2829
# Builds on other branches skip the smoke test if they only change files it does not read.

‎.buildkite/smoke-test-vm.sh‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
#!/usr/bin/env bash
2+
3+
# Runs test/smoke-test.sh and test/executor-smoke-test.sh on a new GCP VM booted from $IMAGE,
4+
# then deletes the VM
5+
6+
cd "$(dirname "${BASH_SOURCE[0]}")/.."
7+
set -euo pipefail
8+
9+
project="sourcegraph-ci"
10+
zone="us-central1-f"
11+
12+
# GCP creates a disk, then a VM, both with the same name. Naming them after the job
13+
# lets the cleanup find them if the job is cancelled before the VM exists
14+
instance_name_prefix="docker-test-job-"
15+
instance_name="${instance_name_prefix}${BUILDKITE_JOB_ID}"
16+
17+
# The builder account key, and an SSH key for this job only
18+
temporary_directory="$(mktemp -d)"
19+
20+
# Ask GCP to delete the VMs matching a filter, and the matching disks which no VM uses,
21+
# without waiting the 1-2 minutes a deletion takes. GCP deletes a VM's disk with the VM
22+
request_deletion() {
23+
{
24+
gcloud compute instances list --project="$project" --filter="$1" --format="value(selfLink)" &&
25+
gcloud compute disks list --project="$project" --filter="($1) AND -users:*" --format="value(selfLink)"
26+
} | tee /dev/stderr |
27+
xargs --no-run-if-empty --max-args=1 curl --fail --silent --show-error --output /dev/null \
28+
--request DELETE --header "Authorization: Bearer $(gcloud auth print-access-token)"
29+
}
30+
31+
cleanup() {
32+
request_deletion "name=${instance_name}" ||
33+
echo "Could not delete this job's VM. GCP deletes it an hour after it started"
34+
rm -rf "$temporary_directory"
35+
}
36+
trap cleanup EXIT
37+
38+
# The VM's guest agent authorizes this key for the buildkite user.
39+
# The keepalive ends the connection to a VM which stopped answering, after a minute
40+
ssh-keygen -q -t ed25519 -N "" -C buildkite -f "$temporary_directory/ssh-key"
41+
ssh_options=(-i "$temporary_directory/ssh-key" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR
42+
-o ConnectTimeout=5 -o ServerAliveInterval=15 -o ServerAliveCountMax=4)
43+
44+
echo --- ":lock: builder account key"
45+
gcloud secrets versions access latest --secret=e2e-builder-sa-key --quiet --project="$project" >"$temporary_directory/builder-key.json"
46+
DOCKER_PASSWORD="$(gcloud secrets versions access latest --secret=DOCKER_PASSWORD_BUILDKITE_AGENT --quiet --project="$project")"
47+
export CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE="$temporary_directory/builder-key.json"
48+
49+
echo --- ":wastebasket: deleting VMs and disks left behind by jobs which died before their cleanup"
50+
# The smoke test step times out after 30 minutes, so a VM or disk older than an hour has no job.
51+
# GCP deletes the VMs itself after an hour, but not a disk whose VM was never created
52+
request_deletion "name~^${instance_name_prefix} AND creationTimestamp<-PT1H" || echo "Could not delete stale VMs and disks"
53+
54+
echo "--- :gcloud: starting VM $instance_name"
55+
# `beta`, because --max-run-duration is only there in the agents' gcloud 417. It is in every track from gcloud 483
56+
ip_address="$(gcloud beta compute instances create "$instance_name" \
57+
--project="$project" --zone="$zone" \
58+
--image="$IMAGE" --image-project="$project" \
59+
--machine-type=custom-16-20480 \
60+
--boot-disk-size=20GB --boot-disk-type=pd-standard \
61+
--max-run-duration=1h --instance-termination-action=DELETE \
62+
--network=default --no-address \
63+
--service-account="e2e-builder@${project}.iam.gserviceaccount.com" --scopes=compute-rw,storage-ro \
64+
--labels=team=dev-infra,type=smoke-test \
65+
--metadata="ssh-keys=buildkite:$(cat "$temporary_directory/ssh-key.pub")" \
66+
--format="value(networkInterfaces[0].networkIP)")"
67+
remote="buildkite@${ip_address}"
68+
69+
echo --- ":ssh: waiting for the VM to accept SSH"
70+
deadline=$((SECONDS + 300))
71+
until ssh "${ssh_options[@]}" "$remote" true; do
72+
if ((SECONDS > deadline)); then
73+
echo "The VM did not accept SSH within 5 minutes"
74+
exit 1
75+
fi
76+
sleep 2
77+
done
78+
79+
copy_repository() {
80+
tar --create --gzip --exclude=.git --exclude=node_modules . |
81+
ssh "${ssh_options[@]}" "$remote" "sudo mkdir -p /deploy-sourcegraph-docker && sudo tar --extract --gzip --directory=/deploy-sourcegraph-docker"
82+
}
83+
84+
echo --- ":package: copying the repository to the VM"
85+
# A VM can accept SSH, then stop answering for a while. ssh exits with 255 when the connection fails,
86+
# so retry the copy on the same VM. The tests have not started yet, so nothing runs twice
87+
exit_code=0
88+
copy_repository || exit_code=$?
89+
for retry in {1..10}; do
90+
if [ "$exit_code" != 255 ]; then
91+
break
92+
fi
93+
echo "The VM does not answer SSH, retry $retry of 10 in 15 seconds"
94+
sleep 15
95+
exit_code=0
96+
copy_repository || exit_code=$?
97+
done
98+
99+
if [ "$exit_code" = 0 ]; then
100+
echo --- ":docker: running the smoke tests"
101+
# Runs as root on the VM. The Docker Hub password comes over stdin, so it does not show in the VM's process list
102+
remote_script='
103+
set -euo pipefail
104+
read -r DOCKER_PASSWORD
105+
export DOCKER_USERNAME=sourcegraph DOCKER_PASSWORD
106+
/deploy-sourcegraph-docker/test/smoke-test.sh
107+
# The Ubuntu 20.04 VM image has no jq. Skip /etc/apt/sources.list.d: that image lacks the key for its Chrome repository, so a full update fails
108+
command -v jq >/dev/null || { apt-get update --quiet --option Dir::Etc::sourceparts=- && apt-get install --quiet --yes jq; }
109+
/deploy-sourcegraph-docker/test/executor-smoke-test.sh
110+
'
111+
# shellcheck disable=SC2029 # remote_script is quoted here, to run on the VM
112+
ssh "${ssh_options[@]}" "$remote" "sudo bash -c $(printf %q "$remote_script")" <<<"$DOCKER_PASSWORD" || exit_code=$?
113+
114+
if [ "$exit_code" != 0 ]; then
115+
# Copy back the container logs the smoke test wrote, for the step's artifacts
116+
ssh "${ssh_options[@]}" "$remote" "cd /deploy-sourcegraph-docker && sudo tar --create --gzip -- *.log" | tar --extract --gzip || true
117+
fi
118+
fi
119+
120+
# Not a plain `exit`, after which shellcheck reports that cleanup is never invoked (SC2329)
121+
[ "$exit_code" = 0 ] || exit "$exit_code"

‎.buildkite/vagrant-run.sh‎

Lines changed: 0 additions & 39 deletions
This file was deleted.

‎.gitignore‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
.env
22
.envrc
3-
.vagrant
43
*.log
54
node_modules/

‎AGENTS.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,20 +17,22 @@ CI runs these on every branch. `.tool-versions` pins Node, the GitHub CLI, and s
1717
- **Compose files:** `.buildkite/validate-docker-compose.sh` runs `docker compose config` on every Compose file, in each combination the docs say to run. Add a line to it for every new Compose file or combination.
1818
- **Formatting:** `npm ci && npm run prettier-check`; `npm run prettier` rewrites the files. Prettier formats the JavaScript and JSON files; the repository has no other file type its glob matches. `.prettierignore` excludes Markdown and `.github/`, and no formatter runs on YAML, so match the surrounding YAML style by hand.
1919
- **Shell scripts:** `git ls-files -z '*.sh' .buildkite/hooks | xargs -0 shellcheck`.
20-
- **Smoke test:** the `docker-compose-test` step runs `.buildkite/vagrant-run.sh docker-test`. See below.
20+
- **Smoke test:** the `docker-compose-test` step runs `.buildkite/smoke-test-vm.sh`. See below.
2121

2222
Use `docker compose` (v2) in scripts and docs, not `docker-compose`.
2323

2424
## Smoke test
2525

26-
`.buildkite/vagrant-run.sh` uses Vagrant to boot a GCP VM from the image named in `IMAGE` in `.buildkite/pipeline.yaml`, runs the two tests below on it, and deletes it. It reads CI secrets and creates a billed VM, so take its result from the pull request's Buildkite build; run the script yourself only when a person asks.
26+
`.buildkite/smoke-test-vm.sh` boots a GCP VM from the image named in `IMAGE` in `.buildkite/pipeline.yaml`, runs the two tests below on it, and deletes it. It reads CI secrets and creates a billed VM, so take its result from the pull request's Buildkite build; run the script yourself only when a person asks.
2727

2828
- `test/smoke-test.sh` starts the base file, `jaeger/`, `executors/`, and `test/docker-compose.override.yaml` with `docker compose up --wait`, which fails if any container reports `unhealthy`. It then checks that every service has a running container, no container restarted, and the frontend and Jaeger respond. It counts the services itself, so a new long-running service in those files needs no change to the test. A new service which exits when it is done, like `migrator`, must be left out of the count. Add a new optional Compose file to its `compose_files`.
2929
- `test/executor-smoke-test.sh` checks that the executor registers with the frontend, and serves exactly the queues which `test/docker-compose.override.yaml` gives it. The script names them again, so change both files together.
3030
- A service which is not ready until another service is needs a `start_period` on its health check, or `--wait` fails.
31-
- A test failure is not retried, and the step uploads every container's logs as artifacts. Only the image pull is retried: `test/smoke-test.sh` tries `docker compose pull` three times.
31+
- A test failure is not retried, and the step uploads every container's logs as artifacts. Only connections are retried: the script waits up to 5 minutes for the VM to accept SSH, retries the SSH connection which copies the repository to the VM, and `test/smoke-test.sh` tries `docker compose pull` three times.
3232
- A branch which only changes files listed in the step's `if_changed` `exclude` skips the step. Builds on `main` always run it. When the smoke test starts reading a file on that list, remove it from the list.
33+
- GCP deletes each VM an hour after it starts, and the script deletes the VMs and disks which older jobs left behind. Keep the step's `timeout_in_minutes` under an hour.
3334
- The VM image is built from `gcp/images/e2e` in sourcegraph/infrastructure.
35+
- The CI agents' queue is named `vagrant`. Nothing on `main` uses Vagrant.
3436

3537
## Conventions
3638

‎test/Vagrantfile‎

Lines changed: 0 additions & 74 deletions
This file was deleted.

‎test/servers.yaml‎

Lines changed: 0 additions & 17 deletions
This file was deleted.

0 commit comments

Comments
 (0)