Skip to content

ci: every workflow step is one task, and Task from go.mod #214

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aledbf merged 3 commits into from
Oct 24, 2026
Merged

ci: every workflow step is one task, and Task from go.mod #214

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
db226ae
ci: every workflow step is one task, and Task from go.mod
aledbf Oct 24, 2026
eb2241f
go-tools at main (dfcf2c1): sfw and taskgate as merged
aledbf Oct 24, 2026
44d02ee
go-tools at main (5824d9a): refs runs the Task go.mod pins as go tool…
aledbf Oct 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 8 additions & 42 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,39 +53,19 @@ jobs:
go-version-file: go.mod
cache: true

- name: Set up Task
uses: go-task/setup-task@a00fbb05ce67b35648be3c78cbc9fd85354c757e # v2.2.0

# `task lint` and `task test`, and not a list of steps: a second definition of what
# the gate is, maintained in a workflow file, goes out of step with the first exactly
# when a check is added.
- name: Lint
run: task lint
run: go tool task lint

- name: Test the machine definition
run: task test
run: go tool task test

# always(): a failed suite is when the per-function numbers are wanted.
- name: Coverage summary
if: always()
run: |
if [ ! -f coverage.out ]; then
echo "coverage.out was not produced; the tests may have failed before coverage ran." >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
{
echo "## Go coverage"
echo
echo "**$(go tool cover -func=coverage.out | awk '/^total:/ {print $3}')** total"
echo
echo '<details><summary>Function detail</summary>'
echo
echo '```'
go tool cover -func=coverage.out
echo '```'
echo '</details>'
} >> "$GITHUB_STEP_SUMMARY"
go tool cover -html=coverage.out -o coverage.html
run: go tool task coverage

- name: Upload Go coverage HTML
if: always()
Expand All @@ -100,23 +80,13 @@ jobs:
# is against what it merges into, on a push against what was there before.
- name: What this change broke, and whether a test noticed
env:
EVENT: ${{ github.event_name }}
BASE_REF: ${{ github.base_ref }}
BEFORE: ${{ github.event.before }}
run: |
set -euo pipefail
if [ "$EVENT" = "pull_request" ]; then
base="origin/$BASE_REF"
else
base="$BEFORE"
fi
git rev-parse --verify "$base^{commit}" >/dev/null 2>&1 || base=HEAD~1
task mutate BASE="$base"
run: go tool task mutate

# The steps below run spin-machine, so it has to exist. `task test` exercises the
# machine package; only a build catches what the command around it does with it.
- name: Build the tools
run: task tools
run: go tool task tools

# `task test` proves the machine package agrees with itself. The steps below ask the
# QEMU binary, which is the half that can disagree: the device set is decided in
Expand Down Expand Up @@ -148,7 +118,7 @@ jobs:
- name: Fetch the pinned QEMU
id: qemu
continue-on-error: true
run: task qemu:fetch
run: go tool task qemu:fetch

# vhost-vsock is a kernel module, and the machine's vsock device cannot be created
# without /dev/vhost-vsock. The runner has the node and gives the user no access to
Expand All @@ -158,12 +128,8 @@ jobs:
# told what went unchecked instead of being shown a pass.
- name: Load vhost_vsock
if: steps.qemu.outcome == 'success'
run: |
# The device first: a kernel that builds it in - Spin OS's, which has no modules - has
# it from boot, and modprobe there has nothing to find.
{ [ -c /dev/vhost-vsock ] || sudo modprobe vhost_vsock; } && sudo chmod 0666 /dev/vhost-vsock ||
echo "no vhost_vsock here; the check will name vhost-vsock-pci as unchecked"
run: go tool task vsock

- name: Check QEMU accepts the machine's arguments
if: steps.qemu.outcome == 'success'
run: task verify:args
run: go tool task verify:args
30 changes: 7 additions & 23 deletions .github/workflows/compat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,37 +63,21 @@ jobs:
go-version-file: go.mod
cache: false

- name: Set up Task
uses: go-task/setup-task@a00fbb05ce67b35648be3c78cbc9fd85354c757e # v2.2.0

# A builder that can export its cache to the directory above; the runner keeps it.
- name: A BuildKit that keeps its cache
run: docker buildx inspect compat > /dev/null 2>&1 && docker buildx use compat || docker buildx create --use --name compat --driver docker-container
run: go tool task boot:buildkit

- name: Build this change's QEMU and kernel, and the tools
run: task qemu:build kernel:build tools
run: go tool task qemu:build kernel:build tools

# The newest release, checked against its sha256 and its signature, as the machine before;
# this change's machine is the same tree with what was just built in place of its own.
# this change's machine is the same tree with what was just built in place of its own. The
# steps after boot that tree: it is their SPIN_MACHINE_OUTPUT.
- name: The machine before, and this one
run: |
set -euo pipefail
task boot:previous
previous=$(cat _output/compat/previous)
here=$PWD/_output/compat/here
rm -rf "$here" && cp -as "$previous" "$here"
rm -rf "$here/bin" "$here/qemu" "$here/kernel"
mkdir -p "$here/bin" "$here/kernel"
cp _output/bin/qemu-system-x86_64 _output/bin/qemu-img _output/bin/spin-machine "$here/bin/"
cp -r _output/qemu "$here/qemu"
cp _output/kernel/vmlinux "$here/kernel/vmlinux"
{
echo "SPIN_COMPAT_PREVIOUS=$previous"
echo "SPIN_MACHINE_OUTPUT=$here"
} >> "$GITHUB_ENV"
run: go tool task boot:here

- name: Every ROM the machine maps is the fingerprint's
run: SPIN_ROM_TEST=1 go test ./boot/ -run '^TestEveryROMIsTheFingerprints$' -count=1 -v -timeout 3m
run: go tool task boot:roms

- name: A checkpoint of the release before resumes here
run: go test ./boot/ -run '^TestACheckpointOfTheReleaseBeforeResumesHere$' -count=1 -v -timeout 10m
run: go tool task boot:compat
24 changes: 10 additions & 14 deletions .github/workflows/image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,7 @@ jobs:
id: touches
env:
BASE: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
build=$(hack/touches "$BASE" alpine e2fsprogs ubuntu mkosi task git-lfs buildkit -- \
image e2fsprogs go.mod .github/workflows/image.yml)
echo "build=${build}" | tee -a "$GITHUB_OUTPUT"
run: go tool task touches -- alpine e2fsprogs ubuntu mkosi task git-lfs buildkit -- image e2fsprogs go.mod .github/workflows/image.yml

build:
needs: changed
Expand All @@ -81,9 +78,6 @@ jobs:
go-version-file: go.mod
cache: true

- name: Set up Task
uses: go-task/setup-task@a00fbb05ce67b35648be3c78cbc9fd85354c757e # v2.2.0

# One sticky disk per thing built, and no silent fallback; see the note in qemu.yml. This
# job builds e2fsprogs and the image builder, which is one job and so one disk.
- name: Setup Blacksmith Builder
Expand All @@ -100,8 +94,10 @@ jobs:
# The image is made and checked with the static e2fsprogs this repository builds, not
# the build container's, and image/build.sh stops when it is not there.
- name: Build e2fsprogs
run: |
CACHE_EXPORT=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} CACHE_BACKEND=gha task e2fsprogs:build
env:
CACHE_BACKEND: gha
CACHE_EXPORT: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
run: go tool task e2fsprogs:build

# image/build.sh asserts what came out on the finished filesystem rather than
# trusting the build that made it: that there is an /sbin/init and a /bin/sh, that
Expand All @@ -114,13 +110,13 @@ jobs:
# it would buy minutes of cache traffic in exchange for minutes of archive traffic,
# and a cache entry large enough to push others out of the repository's 10 GB.
- name: Build the base image
run: |
CACHE_EXPORT=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} CACHE_BACKEND=gha task image:build
env:
CACHE_BACKEND: gha
CACHE_EXPORT: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
run: go tool task image:build

- name: What this image is
run: |
ls -la _output/image/rootfs.qcow2
echo "sha256: $(sha256sum _output/image/rootfs.qcow2 | cut -d' ' -f1)"
run: go tool task image:summary

# Not uploaded as an artifact: it is ~860 MB and the release workflow rebuilds it
# from the same inputs. What this job is for is finding out that it still builds and
Expand Down
21 changes: 6 additions & 15 deletions .github/workflows/kernel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,7 @@ jobs:
id: touches
env:
BASE: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
build=$(hack/touches "$BASE" debian debian-snapshot kernel moby-check-config -- \
kernel go.mod .github/workflows/kernel.yml)
echo "build=${build}" | tee -a "$GITHUB_OUTPUT"
run: go tool task touches -- debian debian-snapshot kernel moby-check-config -- kernel go.mod .github/workflows/kernel.yml

build:
needs: changed
Expand All @@ -78,9 +75,6 @@ jobs:
go-version-file: go.mod
cache: true

- name: Set up Task
uses: go-task/setup-task@a00fbb05ce67b35648be3c78cbc9fd85354c757e # v2.2.0

# One sticky disk per thing built, and no silent fallback; see the note in qemu.yml.
- name: Setup Blacksmith Builder
uses: useblacksmith/setup-docker-builder@19215110ab936351210feebdfa5b440b4493e184 # v2.2.0
Expand All @@ -97,20 +91,17 @@ jobs:
# survived the strip. Without them QEMU has no entry point into this kernel and the
# failure is a VM that does not start, in whichever lane boots one next.
- name: Build the kernel
run: |
CACHE_EXPORT=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} CACHE_BACKEND=gha task kernel:build KERNEL_NPROC=4
env:
CACHE_BACKEND: gha
CACHE_EXPORT: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
run: go tool task kernel:build KERNEL_NPROC=4

# What it produced, on the run's summary: a pull request that changes the config, or
# bumps the Debian toolchain the kernel is compiled with, changes every guest's next
# boot, and the change that does it need not look like much. Checkpoints are not
# affected: the kernel is not in the fingerprint (machine.Spec.Fingerprint).
- name: What this kernel is
run: |
set -euo pipefail
{
echo "kernel sha256: $(sha256sum _output/kernel/vmlinux | cut -d' ' -f1)"
echo "config sha256: $(sha256sum _output/kernel/kernel-config | cut -d' ' -f1)"
} | tee -a "$GITHUB_STEP_SUMMARY"
run: go tool task kernel:summary

- name: Upload the kernel
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
Expand Down
113 changes: 10 additions & 103 deletions .github/workflows/lab.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,9 @@ jobs:
KERNEL_RUN_C: ${{ inputs.kernel_run_c }}
QEMU_RUN: ${{ inputs.qemu_run }}
VARS: ${{ inputs.vars }}
# GitHub reads a workflow command (::error::) only at the start of a line, where Task's
# prefixed output puts the task's name.
TASK_OUTPUT: interleaved
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand All @@ -101,39 +104,14 @@ jobs:
go-version-file: go.mod
cache: false

- name: Set up Task
uses: go-task/setup-task@a00fbb05ce67b35648be3c78cbc9fd85354c757e # v2.2.0

- name: Unpack the release
run: |
set -euo pipefail
if [ -z "$RELEASE" ]; then
RELEASE=$(gh release list -R "$GITHUB_REPOSITORY" --exclude-drafts --exclude-pre-releases --limit 1 --json tagName -q '.[0].tagName')
fi
rm -rf _output/lab
mkdir -p _output/lab/tmp
gh release download "$RELEASE" -R "$GITHUB_REPOSITORY" -p '*.tar.gz' -D _output/lab
tar -xzf _output/lab/*.tar.gz -C _output/lab
{
echo "RELEASE=$RELEASE"
echo "SPIN_MACHINE_OUTPUT=$PWD/_output/lab/spin-machine-${RELEASE}-linux-x86_64/usr/share/spin-stack"
# On disk: the overlays and memory files go here, gigabytes a tmpfs /tmp holds in RAM.
echo "TMPDIR=$PWD/_output/lab/tmp"
} >> "$GITHUB_ENV"
run: go tool task lab:unpack

# Another build to compare: a Kernel run's vmlinux (B, and C beside it) or a QEMU run's
# QEMU and firmware. qemu.yml and kernel.yml build any branch on dispatch.
- name: Fetch the builds to compare
if: inputs.kernel_run != '' || inputs.kernel_run_c != '' || inputs.qemu_run != ''
run: |
set -euo pipefail
fetch() { # run, artifact pattern, directory
[[ "$1" =~ ^[0-9]+$ ]] || { echo "::error::a run id is a number, got: $1"; exit 1; }
gh run download "$1" -R "$GITHUB_REPOSITORY" -p "$2" -D "$3"
}
if [ -n "$KERNEL_RUN" ]; then fetch "$KERNEL_RUN" 'vmlinux-*' _output/lab/kernel-b; fi
if [ -n "$KERNEL_RUN_C" ]; then fetch "$KERNEL_RUN_C" 'vmlinux-*' _output/lab/kernel-c; fi
if [ -n "$QEMU_RUN" ]; then fetch "$QEMU_RUN" 'qemu-*-linux-amd64' _output/lab/qemu-b; fi
run: go tool task lab:fetch

# beside: the build is booted interleaved with the release's own - SPIN_KERNEL_B and
# SPIN_KERNEL_C for a kernel, and for QEMU a second release tree, SPIN_MACHINE_OUTPUT_B,
Expand All @@ -145,94 +123,23 @@ jobs:
if: inputs.kernel_run != '' || inputs.kernel_run_c != '' || inputs.qemu_run != ''
env:
VARIANT: ${{ inputs.variant }}
run: |
set -euo pipefail
one() { # the one file under $1 whose path matches $2
f=$(find "$1" -path "$2" -type f | head -1)
[ -n "$f" ] || { echo "::error::$1 has no $2"; exit 1; }
echo "$PWD/$f"
}
b=$PWD/_output/lab/release-b
tree_b() { [ -d "$b" ] || cp -as "$SPIN_MACHINE_OUTPUT" "$b"; }
if [ -n "$KERNEL_RUN" ]; then
k=$(one _output/lab/kernel-b '*/vmlinux')
if [ "$VARIANT" = instead ]; then
tree_b; rm -f "$b/kernel/vmlinux"; install -m 0644 "$k" "$b/kernel/vmlinux"
else
echo "SPIN_KERNEL_B=$k" >> "$GITHUB_ENV"
fi
fi
if [ -n "$KERNEL_RUN_C" ]; then
[ "$VARIANT" = beside ] || { echo "::error::kernel_run_c is a third kernel beside the release, not one to boot instead"; exit 1; }
echo "SPIN_KERNEL_C=$(one _output/lab/kernel-c '*/vmlinux')" >> "$GITHUB_ENV"
fi
if [ -n "$QEMU_RUN" ]; then
q=$(one _output/lab/qemu-b '*/bin/qemu-system-x86_64')
tree_b; rm -rf "$b/bin/qemu-system-x86_64" "$b/qemu"
install -m 0755 "$q" "$b/bin/qemu-system-x86_64"
cp -r "$(dirname "$(dirname "$q")")/qemu" "$b/qemu"
[ "$VARIANT" = instead ] || echo "SPIN_MACHINE_OUTPUT_B=$b" >> "$GITHUB_ENV"
fi
if [ "$VARIANT" = instead ]; then echo "SPIN_MACHINE_OUTPUT=$b" >> "$GITHUB_ENV"; fi
run: go tool task lab:place

- name: Take the variables
run: |
set -euo pipefail
IFS=';' read -ra pairs <<< "$VARS"
for pair in "${pairs[@]}"; do
pair="${pair#"${pair%%[![:space:]]*}"}"
pair="${pair%"${pair##*[![:space:]]}"}"
[ -z "$pair" ] && continue
key="${pair%%=*}"
[[ "$pair" == *=* && "$key" =~ ^(REPS|TOP|GAP|ONLY|FLAGS|CPUS|MEMORY_MIB|DEBUG|FILE_MB|CACHE_TIMEOUT_S|GRAPH_[A-Z_]+|SHARE_[A-Z_]+|SPIN_[A-Z0-9_]+)$ ]] ||
{ echo "::error::not a variable this workflow passes on: $pair"; exit 1; }
case "$key" in
SPIN_MACHINE_OUTPUT|SPIN_MACHINE_OUTPUT_B|SPIN_KERNEL_B|SPIN_KERNEL_C) echo "::error::$key is set by this workflow"; exit 1 ;;
esac
echo "$pair" >> "$GITHUB_ENV"
done
run: go tool task lab:vars

- name: What this run is
env:
VARIANT: ${{ inputs.variant }}
run: |
{
echo "### $EXPERIMENT on $RUNNER_NAME"
echo
echo "- release: $RELEASE"
echo "- kernel B: ${KERNEL_RUN:+run }${KERNEL_RUN:-none}"
echo "- kernel C: ${KERNEL_RUN_C:+run }${KERNEL_RUN_C:-none}"
echo "- QEMU B: ${QEMU_RUN:+run }${QEMU_RUN:-none}"
if [ -n "$KERNEL_RUN$KERNEL_RUN_C$QEMU_RUN" ]; then echo "- booted: $VARIANT the release"; fi
echo "- vars: ${VARS:-none}"
echo "- CPU: $(sed -n 's/^model name\s*: //p' /proc/cpuinfo | head -1), $(nproc) CPUs"
echo "- load: $(cut -d' ' -f1-3 /proc/loadavg)"
if sudo -n true 2>/dev/null; then echo "- sudo -n: yes"; else echo "- sudo -n: **no** (targets that edit an overlay skip)"; fi
} | tee -a "$GITHUB_STEP_SUMMARY"
[ -r /dev/kvm ] && [ -w /dev/kvm ] || { echo "::error::no usable /dev/kvm"; exit 1; }
run: go tool task lab:about

# report writes its JSON where OUT says; the others print their tables through t.Log.
# A skipped test exits 0, and a green run that measured nothing reads as a result, so a
# skip fails the run with the test's reason.
- name: Run the experiment
run: |
set -o pipefail
task "$EXPERIMENT" OUTPUT_DIR="$SPIN_MACHINE_OUTPUT" OUT="$PWD/_output/lab/report.json" 2>&1 | tee _output/lab/lab.log
if grep -q -- '--- SKIP' _output/lab/lab.log; then
echo "::error::the experiment skipped: $(grep -B1 -- '--- SKIP' _output/lab/lab.log | head -1 | sed 's/^.*_test.go:[0-9]*: //')"
exit 1
fi
run: go tool task lab:run

- name: What it found
if: always()
run: |
[ -f _output/lab/lab.log ] || exit 0
{
echo
echo '```'
grep -E '^\s{4}\S' _output/lab/lab.log | sed 's/^ //' | tail -300
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
run: go tool task lab:found

- name: Keep the log
if: always()
Expand Down
Loading