From 0d3ad84e941722d0d0310eb4ba81466798196bdb Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Tue, 3 Feb 2026 21:54:50 -0500 Subject: [PATCH 1/4] Refine Docker workflows and reproducibility tooling - Add shared Docker helpers and wrapper-specific help while keeping full env reference in common.sh - Introduce digest tooling (get_digest, pin-and-run, fetch_nix_installer) and pin DOCKER_REPRO_DIGEST - Rework reproducibility checks (config digest comparison, remote resolution) and improve docker load/Xauthority handling - Update README and QEMU docs; tighten initrd kexec-seal-key parsing Signed-off-by: Thierry Laurion --- README.md | 505 ++++++++++++- docker/DOCKER_REPRO_DIGEST | 12 + docker/check_reproducibility.sh | 53 ++ docker/common.sh | 1248 +++++++++++++++++++++++++++++++ docker/fetch_nix_installer.sh | 109 +++ docker/get_digest.sh | 285 +++++++ docker/pin-and-run.sh | 132 ++++ docker_latest.sh | 97 ++- docker_local_dev.sh | 136 ++-- docker_repro.sh | 116 +-- initrd/bin/kexec-seal-key | 4 +- targets/qemu.md | 192 +++-- 12 files changed, 2624 insertions(+), 265 deletions(-) create mode 100644 docker/DOCKER_REPRO_DIGEST create mode 100755 docker/check_reproducibility.sh create mode 100755 docker/common.sh create mode 100755 docker/fetch_nix_installer.sh create mode 100755 docker/get_digest.sh create mode 100755 docker/pin-and-run.sh diff --git a/README.md b/README.md index 1dece18fa..92b542b4b 100644 --- a/README.md +++ b/README.md @@ -36,10 +36,139 @@ Building heads with prebuilt and versioned docker images Heads now builds with Nix built docker images since https://github.com/linuxboot/heads/pull/1661. The short path to build Heads is to do what CircleCI would do (./docker_repro.sh under heads git cloned directory): -- Install _docker-ce_ for your OS of choice (refer to their documentation) +- Install Docker (docker-ce) for your OS by following Docker's official installation instructions: https://docs.docker.com/engine/install/ - run `./docker_repro.sh make BOARD=XYZ` -Using Nix local dev environement / building docker images with Nix +Note: `./docker_repro.sh` is the canonical, reproducible way to build and test Heads. The `docker_local_dev.sh` helper is intended for developers who need to modify the local image built from `flake.nix`/`flake.lock` and is not recommended for general testing. + +Important: the supported and tested workflow uses the provided Docker +wrappers (`./docker_repro.sh`, `./docker_local_dev.sh`, or +`./docker_latest.sh`). Host-side installation of QEMU, `swtpm`, or other +QEMU-related tooling is unnecessary for the standard workflow and is not +part of the tested configuration. Only advanced or edge-case workflows +may require installing those tools on the host (see `targets/qemu.md` +for guidance). + +The Docker images produced by our Nix build include QEMU +(`qemu-system-x86_64`), `swtpm` / `libtpms`, `canokey-qemu` (a virtual +OpenPGP smartcard), and other userspace tooling required to build and +test QEMU boards. If you use `./docker_repro.sh` you only need Docker on +the host (for example, `docker-ce`). For KVM acceleration the host +must expose `/dev/kvm` (load `kvm_intel` / `kvm_amd` as appropriate); +our wrapper scripts mount `/dev/kvm` automatically when it exists. + +If you plan to manage disk images or use `qemu-img` snapshots on the +host (outside containers), install the `qemu-utils` package locally +(which provides `qemu-img`). + +Inspecting and cleaning local Docker images +--- + +```bash +# List local images +docker images + +# Inspect a specific image (IDs, digests, repo tags) +docker image inspect + +# Remove a specific image +docker rmi + +# Remove all local images (destructive) +docker rmi -f $(docker images -aq) + +# Remove unused images/containers/networks/build cache (destructive) +docker system prune -a --volumes +``` + +Note: you may need to prefix commands with `sudo` depending on your Docker setup. + +QEMU disk snapshots with `qemu-img` +--- + +If you manage qcow2 disk images on the host, `qemu-img` can create, list, +restore, and delete snapshots. These examples assume a qcow2 disk image: + +```bash +# Create a snapshot +qemu-img snapshot -c clean root.qcow2 + +# List snapshots +qemu-img snapshot -l root.qcow2 + +# Restore (apply) a snapshot +qemu-img snapshot -a clean root.qcow2 + +# Delete a snapshot +qemu-img snapshot -d clean root.qcow2 + +# Optional: create an overlay backed by a base image +qemu-img create -f qcow2 -b base.qcow2 overlay.qcow2 +``` + +If you prefer to run these inside the container, prefix with +`./docker_repro.sh` (for example, `./docker_repro.sh qemu-img snapshot -l root.qcow2`). + +If you do not specify `USB_TOKEN` when running QEMU targets, the container will use the included `canokey-qemu` virtual token by default; set `USB_TOKEN` (or use `hostbus`/`hostport`/`vendorid,productid`) to forward a hardware token instead. + +Docker wrapper helper reference +--- + +Each wrapper now shows its own focused help (only the variables it actually uses). For the complete environment reference, run `docker/common.sh` directly: + +```bash +# Wrapper-specific help +./docker_repro.sh --help +./docker_latest.sh --help +./docker_local_dev.sh --help + +# Full environment variable reference (shared helper) +./docker/common.sh +``` + +The shared helper documents all supported environment variables (opt-ins and opt-outs) and defaults. Wrapper help is intentionally narrower so it only lists variables relevant to that wrapper. + +Wrapper options & environment variables +--- + +**All wrapper scripts** (`./docker_repro.sh`, `./docker_latest.sh`, `./docker_local_dev.sh`): +- `HEADS_MAINTAINER_DOCKER_IMAGE` — override the canonical maintainer's Docker image repository (default: `tlaurion/heads-dev-env`). Use this for local testing or if you maintain a fork. Example: `export HEADS_MAINTAINER_DOCKER_IMAGE="myuser/heads-dev-env"`. This affects reproducibility checks and default image references across all Docker wrapper scripts. + +- `HEADS_CHECK_REPRODUCIBILITY_REMOTE` — specify which remote image to compare against when verifying reproducibility (default: `${HEADS_MAINTAINER_DOCKER_IMAGE}:latest`). Use this to test against a specific tagged version instead of `:latest`. + ```bash + # Compare against a specific version + export HEADS_CHECK_REPRODUCIBILITY_REMOTE="tlaurion/heads-dev-env:v0.2.7" + HEADS_CHECK_REPRODUCIBILITY=1 ./docker_local_dev.sh + ``` +- `HEADS_DISABLE_USB=1` — disable automatic USB passthrough and the + automatic USB cleanup (default: `0`). +- `HEADS_X11_XAUTH=1` — force mounting your `${HOME}/.Xauthority` into the container for X11 authentication. When set the helper will bypass programmatic Xauthority generation and mount your `${HOME}/.Xauthority` (if present); if the file is missing the helper will warn and will not attempt automatic cookie creation (GUI may fail). + +`./docker_local_dev.sh`: +- `HEADS_SKIP_DOCKER_REBUILD=1` — skip automatically rebuilding the local image when `flake.nix`/`flake.lock` are dirty +- `HEADS_CHECK_REPRODUCIBILITY=1` — **recommended for verifying reproducible builds**. After building/loading the local image, automatically compares its digest with the published maintainer image to verify reproducibility. Requires network access. By default compares against `${HEADS_MAINTAINER_DOCKER_IMAGE}:latest`. Use `HEADS_CHECK_REPRODUCIBILITY_REMOTE` to specify a different tag (e.g., `v0.2.7`). See the "Verifying reproducibility" section below for detailed examples and expected outputs. +- `HEADS_AUTO_INSTALL_NIX=1` — automatically attempt to download the Nix single-user installer when `nix` is missing (interactive prompt suppressed). + For supply-chain safety the helper will download the installer to a temporary file and print its SHA256; it will NOT execute the installer automatically unless the downloaded installer matches a pinned hash. The helper will also attempt to detect the installer version heuristically (when possible) and suggest the canonical releases URL (for example `https://releases.nixos.org/nix/nix-2.33.2/install.sha256`) so you can fetch the published sha and compare. To verify: + + - Preferred: pin a release version (recommended): set `HEADS_NIX_INSTALLER_VERSION` to a release (for example `nix-2.33.2`). The helper will fetch `https://releases.nixos.org/nix/${HEADS_NIX_INSTALLER_VERSION}/install` and `install.sha256` and show both checksums for you to compare. To auto-run in trusted automation, set `HEADS_NIX_INSTALLER_SHA256` to the expected sha256 as well. + + - Or compute-and-pin locally: run `./docker/fetch_nix_installer.sh --version nix-2.33.2` (or `--url`) to download the installer and print its sha256, then set `HEADS_NIX_INSTALLER_SHA256` to that value for automation. + + Otherwise verify the downloaded installer manually and run it yourself: `sh /path/to/installer --no-daemon`. +- `HEADS_AUTO_ENABLE_FLAKES=1` — automatically enable flakes by writing `experimental-features = nix-command flakes` to `$HOME/.config/nix/nix.conf` (interactive prompt suppressed) +- `HEADS_MIN_DISK_GB` — minimum free disk space in GB required on `/nix` (or `/` if `/nix` missing) for building (default: `50`) +- `HEADS_SKIP_DISK_CHECK=1` — skip the preflight disk-space check +- `HEADS_ALLOW_UNPINNED_LATEST=1` — when set, bypass the interactive warning that using `:latest` in `./docker_latest.sh` is a supply-chain risk (otherwise `:latest` requires confirmation unless `DOCKER_LATEST_DIGEST` is set or the wrapper can fall back to `DOCKER_REPRO_DIGEST` for the maintainer image) +- `DOCKER_REPRO_DIGEST` — pin the image used by `./docker_repro.sh` to an immutable digest: `tlaurion/heads-dev-env@` (recommended for reproducible and secure builds). Note: `DOCKER_REPRO_DIGEST` is *consumed by* `./docker_repro.sh` (via `resolve_docker_image` in `docker/common.sh`) and is the canonical way to pin the repro image for reproducible builds. + +For details about selecting or forwarding a physical USB token to QEMU +(handled by the `USB_TOKEN` make variable), see `targets/qemu.md`. + +Note: when USB passthrough is active the wrappers will detect processes that may be holding a USB token (for example `scdaemon` or `pcscd`). The wrapper will warn and, on interactive shells, give a 3s abort window before attempting to kill those processes to free the token. Set `HEADS_DISABLE_USB=1` to opt out of this automatic cleanup. + +Example: `HEADS_DISABLE_USB=1 ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run` + +Using Nix local dev environment / building docker images with Nix == Under QubesOS? @@ -61,13 +190,28 @@ Build docker from nix develop layer locally * `mkdir -p ~/.config/nix` * `echo 'experimental-features = nix-command flakes' >>~/.config/nix/nix.conf` +Notes on automation and requirements: + +- The `./docker_local_dev.sh` helper will attempt to ensure Nix and flakes are available when you run it interactively. If Nix is missing it can optionally install it for you and prompt to enable flakes; set `HEADS_AUTO_INSTALL_NIX=1` / `HEADS_AUTO_ENABLE_FLAKES=1` to suppress prompts. +- Building the Docker image and populating `/nix` can require significant disk space — we recommend at least **50 GB** free on `/nix` (or `/` if `/nix` is not present). Adjust via `HEADS_MIN_DISK_GB` or skip the check with `HEADS_SKIP_DISK_CHECK=1`. +- The Nix installer requires a downloader; either `curl` or `wget` must be available on the host. The helper will guide you to install one if neither is present. +- For reproducible builds prefer `./docker_repro.sh`; `./docker_local_dev.sh` is intended for development and will rebuild the local image when `flake.nix`/`flake.lock` are dirty (unless `HEADS_SKIP_DOCKER_REBUILD=1`). #### Build image * Have docker and Nix installed * Build nix developer local environment with flakes locked to specified versions - * `./docker_local_dev.sh` + * Manual: `nix --print-build-logs --verbose build .#dockerImage && docker load < result` + * Helper: `./docker_local_dev.sh` will perform a conditional rebuild when `flake.nix`/`flake.lock` are dirty (unless `HEADS_SKIP_DOCKER_REBUILD=1`). + +Using `./docker_local_dev.sh` + +* `./docker_local_dev.sh` is a developer helper that ensures a local Nix-based Docker image (`linuxboot/heads:dev-env`) is available for interactive development. It performs a few preflight checks and interactive prompts to make the process easier: + - Ensures `nix` is installed and **flakes** are enabled; if missing it will prompt to install Nix and enable flakes. Set `HEADS_AUTO_INSTALL_NIX=1` and/or `HEADS_AUTO_ENABLE_FLAKES=1` to suppress prompts and proceed automatically. + - Requires either `curl` or `wget` to fetch the Nix installer; if neither is present the script will print how to install one and abort. + - Checks disk space on `/nix` (or `/` if `/nix` is absent); default minimum is **50 GB** (`HEADS_MIN_DISK_GB=50`) — override or skip the check with `HEADS_SKIP_DISK_CHECK=1`. + - If `flake.nix` or `flake.lock` are dirty (uncommitted changes), the helper will rebuild the local Docker image. To intentionally trigger a rebuild, make and keep changes to `flake.nix` (for example update an input or a harmless comment) or update `flake.lock`, then run `./docker_local_dev.sh`; the helper detects the dirty flake files and will rebuild automatically. To avoid an automatic rebuild, commit or stash your changes or set `HEADS_SKIP_DOCKER_REBUILD=1` to disable the check. On some hardened OSes, you may encounter problems with ptrace. ``` @@ -88,17 +232,37 @@ Your local docker image "linuxboot/heads:dev-env" is ready to use, reproducible Jump into nix develop created docker image for interactive workflow ==== -There is 3 helpers: -- `./docker_local_dev.sh`: for developers wanting to customize docker image built from flake.nix(nix devenv creation) and flake.lock (pinned versions used by flake.nix) -- `./docker_latest.sh`: for Heads developers, wanting to use latest published docker images to develop Heads -- `./docker_repro.sh`: versioned docker image used under CircleCI to produce reproducivle builds, both locally and under CircleCI. **Use this one if in doubt** +There are three helpers designed for different use cases: -ie: `./docker_repro.sh` will jump into CircleCI used versioned docker image for that Heads commit id to build images reproducibly if git repo is clean (not dirty). +| Script | Use Case | Reproducibility | When to Use | +|--------|----------|------------------|------------| +| `./docker_repro.sh` | **Canonical reproducible builds** | Pinned to immutable digest | **All users & maintainers**: Standard way to build Heads; matches CircleCI exactly; use for releases and critical builds | +| `./docker_local_dev.sh` | **Developer customization** | Local build may differ if flake changes | **Developers only**: Rebuilds from local `flake.nix`/`flake.lock` when dirty; useful for testing flake changes; use `HEADS_CHECK_REPRODUCIBILITY=1` to verify against published version | +| `./docker_latest.sh` | **Convenience** | Defaults to reproducible digest; may be unpinned if no digest is available | **Testing/convenience**: Uses latest published image; by default falls back to the reproducible digest (`DOCKER_REPRO_DIGEST`) when available (no confirmation needed). Runs unpinned only when no digest is configured, in which case it requires confirmation unless `HEADS_ALLOW_UNPINNED_LATEST=1` or `DOCKER_LATEST_DIGEST` is set. | -From there you can use the docker image interactively. +**Recommendation by role**: +- **End users & QA**: Use `./docker_repro.sh` for all builds (ensures reproducibility and security) +- **Developers**: Use `./docker_local_dev.sh` when iterating on the build system or Nix flake, but verify reproducibility with `HEADS_CHECK_REPRODUCIBILITY=1` before committing +- **Maintainers**: Use `./docker_repro.sh` for official releases; use the maintenance workflow in [Maintenance notes on docker image](#maintenance-notes-on-docker-image) when updating the Docker image base version -`make BOARD=board_name` where board_name is the name of the board directory under `./boards` directory. +**Examples**: +Use `./docker_repro.sh` for canonical, reproducible builds: +```bash +./docker_repro.sh make BOARD=x230-hotp-maximized +./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run +``` + +Use `./docker_local_dev.sh` when developing with the Nix flake (verify reproducibility before committing): +```bash +# Modify flake.nix for testing +./docker_local_dev.sh make BOARD=nitropad-nv41 + +# Before committing, verify the build is reproducible +HEADS_CHECK_REPRODUCIBILITY=1 ./docker_local_dev.sh make BOARD=nitropad-nv41 +``` + +If you are already inside the container interactively, run `make BOARD=board_name` as usual. One such useful example is to build and test qemu board roms and test them through qemu/kvm/swtpm provided in the docker image. Please refer to [qemu documentation](targets/qemu.md) for more information. @@ -116,51 +280,318 @@ Eg: `./docker_local_dev.sh make BOARD=nitropad-nv41` -Pull docker hub image to prepare reproducible ROMs as CircleCI in one call +Building with the published Docker image (recommended for reproducible builds) ==== + +The canonical, reproducible way to build Heads is to use `./docker_repro.sh`, which automatically pulls the pinned Docker image digest from `docker/DOCKER_REPRO_DIGEST` and ensures your builds match the CI environment exactly. + +**For users**: +```bash +./docker_repro.sh make BOARD=x230-hotp-maximized +./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run ``` + +This will: +1. Resolve the canonical image digest from `docker/DOCKER_REPRO_DIGEST` (immutable, pinned to a specific version) +2. Pull the image if not present locally +3. Execute your build inside that exact Docker environment +4. Guarantee reproducibility: your ROM output will match official CircleCI builds for that commit + +**About the published image**: +- **Repository**: `tlaurion/heads-dev-env` on Docker Hub is the maintainer's canonical image (configurable via `HEADS_MAINTAINER_DOCKER_IMAGE`) +- **Versioning**: Tagged with version numbers (e.g., `v0.2.7`) for stability; `:latest` is mutable and not recommended +- **Pinning**: The repository file `docker/DOCKER_REPRO_DIGEST` pins an immutable digest (`tlaurion/heads-dev-env@sha256:...`) to ensure reproducibility +- **Trust**: As long as flake.nix and flake.lock are not modified locally, your build will produce identical digests, confirming integrity +- **Fork/Override**: To use a different image repository (e.g., for testing or forks), set `HEADS_MAINTAINER_DOCKER_IMAGE="youruser/your-image"` before running any Docker wrapper script + +Pinning the reproducible image +--- + +- `DOCKER_REPRO_DIGEST` — pin the image used by `./docker_repro.sh` to an immutable digest: `tlaurion/heads-dev-env@`. This environment variable (or the repository file `docker/DOCKER_REPRO_DIGEST`) is *consumed by* `./docker_repro.sh` via `resolve_docker_image()`; pinning ensures reproducible builds and mitigates supply-chain risk from mutable `:latest` tags. + +```bash ./docker_repro.sh make BOARD=x230-hotp-maximized ./docker_repro.sh make BOARD=nitropad-nv41 ``` +Verifying reproducibility of locally-built Docker images +--- + +**Best practice**: Verify that your locally-built Docker image is reproducible by comparing its digest with the published maintainer image. + +The Heads project maintains the canonical `tlaurion/heads-dev-env` Docker image on Docker Hub (configurable via `HEADS_MAINTAINER_DOCKER_IMAGE` environment variable for forks or testing). As long as you do not modify `flake.nix` or `flake.lock`, your locally-built image **should produce an identical digest** to the published image, demonstrating that your build is fully reproducible. + +#### Quick reference + +| Scenario | Command | +|----------|---------| +| **Check against latest maintainer image** | `HEADS_CHECK_REPRODUCIBILITY=1 ./docker_local_dev.sh` | +| **Check against specific version tag** | `HEADS_CHECK_REPRODUCIBILITY=1 HEADS_CHECK_REPRODUCIBILITY_REMOTE="tlaurion/heads-dev-env:v0.2.7" ./docker_local_dev.sh` | +| **Check fork maintainer's image** | `HEADS_MAINTAINER_DOCKER_IMAGE="youruser/heads-dev-env" HEADS_CHECK_REPRODUCIBILITY=1 ./docker_local_dev.sh` | +| **Standalone check (any time)** | `./docker/check_reproducibility.sh linuxboot/heads:dev-env tlaurion/heads-dev-env:v0.2.7` | + +#### Prerequisites + +You have either: +- Built a local Docker image with `./docker_local_dev.sh` (produces `linuxboot/heads:dev-env`), or +- Built from `nix build .#dockerImage` (results in `result` symlink loadable via `docker load`) + +#### Check reproducibility + +**Method 1: Automated check during build (recommended)** + +Enable reproducibility verification automatically during your build with `HEADS_CHECK_REPRODUCIBILITY=1`: + +```bash +# Verify against the default (maintainer's :latest image) +HEADS_CHECK_REPRODUCIBILITY=1 ./docker_local_dev.sh + +# Example output when digests MATCH (reproducible build): +# === Reproducibility Check === +# Local image (linuxboot/heads:dev-env): sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 +# Remote image (tlaurion/heads-dev-env:latest): sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 +# ✓ MATCH: Local build is reproducible! +``` + +To test against a **specific version tag** instead of `:latest`: + +```bash +HEADS_CHECK_REPRODUCIBILITY=1 \ + HEADS_CHECK_REPRODUCIBILITY_REMOTE="tlaurion/heads-dev-env:v0.2.7" \ + ./docker_local_dev.sh + +# Example output when digests DIFFER (expected for different versions): +# === Reproducibility Check === +# Local image (linuxboot/heads:dev-env): sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 +# Remote image (tlaurion/heads-dev-env:v0.2.6): sha256:75af4c816a4a92ebdd0030c2e56ebf23c066858e08145ec1cc64a9e750a0031d +# ✗ MISMATCH: Local build differs from remote +# (This is expected if Nix/flake.lock versions differ or if uncommitted changes exist) +``` + +Note: Docker images can have two different identifiers: a local image ID and a registry manifest digest. If a local image has no `RepoDigests` entry, the reproducibility check will compare image IDs (and may pull the remote tag) instead of manifest digests to avoid false mismatches. This can happen for locally built images that have not been pulled from a registry. + +**Method 2: Standalone reproducibility check** + +Use the provided reproducibility checker script to compare hashes at any time: + +```bash +# Compare your local dev image with a published version +./docker/check_reproducibility.sh linuxboot/heads:dev-env tlaurion/heads-dev-env:v0.2.7 + +# Output (example of a match): +# ✓ SUCCESS: Digests match! +# Your local build is reproducible and identical to tlaurion/heads-dev-env:v0.2.7 +``` + +**Method 3: Manual digest inspection** + +Manually inspect the digest: + +```bash +# Get the digest of your local image (after docker load) +docker inspect --format='{{.Id}}' linuxboot/heads:dev-env +# Output: sha256:8ae7744cc8b4ff0e959aa6dfeeb40dbd40d20ac6fa1f7071dd21ec0c2d0f9f41 + +# Compare with the published image (will pull if needed) +docker pull tlaurion/heads-dev-env:v0.2.7 +docker inspect --format='{{.Id}}' tlaurion/heads-dev-env:v0.2.7 +# Output: sha256:8ae7744cc8b4ff0e959aa6dfeeb40dbd40d20ac6fa1f7071dd21ec0c2d0f9f41 +``` + +#### When digests should match + +✓ **Digests match** → Your build is **reproducible and trustworthy**; matches the maintainer's published image for that Nix snapshot. + +Your locally-built image **will** produce an identical digest to the published image when: +- `flake.nix` and `flake.lock` are **not modified** (i.e., repository is clean relative to these files) +- The same Nix version and dependencies are used +- Build runs on the same Nix store state + +✗ **Digests differ** → Expected in these cases: + +- You have uncommitted changes in `flake.nix` or `flake.lock` +- Different Nix version or Nix dependencies resolved differently on your system +- Using a different `nixpkgs` version than the locked one in `flake.lock` + +#### Trust model + +The `tlaurion/heads-dev-env` image on Docker Hub is the **maintainer's canonical build** and serves as the source of truth for reproducibility. By verifying that your locally-built image produces the same digest as the published `v0.2.7` (or current version), you confirm: + +1. **No tampering**: Your build environment has not been compromised +2. **Reproducibility**: The Heads build system is deterministic for your specific Nix snapshot +3. **Auditability**: You can map your build back to a specific published, reviewed version + +**Recommendation**: Always pin to a specific version tag (e.g., `tlaurion/heads-dev-env:v0.2.7`) rather than `:latest`, and verify the digest matches the published value before using it for critical builds. + Maintenance notes on docker image === -Redo the steps above in case the flake.nix or nix.lock changes. Commit changes. Then publish on docker hub: - -``` -#put relevant things in variables: -docker_version="vx.y.z" && docker_hub_repo="tlaurion/heads-dev-env" -#update pinned packages to latest available ones if needed, modify flake.nix derivatives if needed: -nix flakes update -#modify CircleCI image to use newly pushed docker image -sed "s@\(image: \)\(.*\):\(v[0-9]*\.[0-9]*\.[0-9]*\)@\1\2:$docker_version@" -i .circleci/config.yml -# commit changes -git commit --signoff -m "Bump nix develop based docker image to $docker_hub_repo:$docker_version" -#use commited flake.nix and flake.lock in nix develop -nix --print-build-logs --verbose develop --ignore-environment --command true -#build new docker image from nix develop environement -nix --print-build-logs --verbose build .#dockerImage && docker load < result -#tag produced docker image with new version + +To update the Docker image to a new version (e.g., vx.y.z), follow these steps. This ensures reproducible builds with immutable digests. + +``` +# Set variables +docker_version="vx.y.z" +docker_hub_repo="tlaurion/heads-dev-env" + +# Update pinned packages to latest if needed, modify flake.nix as required +nix flake update + +# Commit flake changes +git add flake.nix flake.lock +git commit --signoff -m "Bump nix develop based docker image to $docker_version" + +# Verify reproducibility: ensure the local build matches (no further changes to flake files) +nix develop --ignore-environment --command true + +# Build the new Docker image +nix build .#dockerImage +docker load < result + +# Verify you can extract the digest (for fully reproducible builds, flake.nix/flake.lock must be committed) +docker inspect --format='{{.Id}}' linuxboot/heads:dev-env + +# Tag the image with the new version docker tag linuxboot/heads:dev-env "$docker_hub_repo:$docker_version" -#push newly created docker image to docker hub + +# Push the new version to Docker Hub (requires push access) docker push "$docker_hub_repo:$docker_version" -#test with CircleCI in PR. Merge. -git push ... -#make last tested docker image version the latest -docker tag "$docker_hub_repo:$docker_version" "$docker_hub_repo:latest" -docker push "$docker_hub_repo:latest" + +# Capture the digest of the pushed image (use --yes to auto-pull) +new_digest=$(./docker/get_digest.sh -y "$docker_hub_repo:$docker_version" | tail -n1) +prev_digest=$(grep '^[^#]' docker/DOCKER_REPRO_DIGEST | head -n1) + +# Update the digest in the repository file +sed -i "s|$prev_digest|$new_digest|" docker/DOCKER_REPRO_DIGEST + +# Update the version comment in the repository file +sed -i "s|# Version: .*|# Version: $docker_version|" docker/DOCKER_REPRO_DIGEST + +# Update .circleci/config.yml to use the new digest and add version comments +# The first -e removes existing "# Docker image" comment lines. The second -e inserts a +# fresh "# Docker image: $docker_hub_repo:$docker_version" comment immediately above the +# matching "- image: $docker_hub_repo@" line while preserving indentation. +sed -i -e "/^[[:space:]]*# Docker image: /d" -e "/^[[:space:]]*- image: ${docker_hub_repo//\//\\/}@/ s|^\([[:space:]]*\)\(- image: ${docker_hub_repo//\//\\/}@\)|\\1# Docker image: $docker_hub_repo:$docker_version\n\\1\\2|" .circleci/config.yml + +# Commit the digest and config changes +git add docker/DOCKER_REPRO_DIGEST .circleci/config.yml +git commit --signoff -m "Pin docker image to digest for $docker_version" + +# Push the branch and create a PR for testing with CircleCI +git push origin docker/squash-docker-changes + +# After PR is merged and tested: +# Tag the tested version as latest (optional; use with caution, prefer explicit versioning) +# docker tag "$docker_hub_repo:$docker_version" "$docker_hub_repo:latest" +# docker push "$docker_hub_repo:latest" +``` + +**Maintainer checklist**: +1. **Reproducibility**: Before pushing, verify `nix build .#dockerImage` produces a deterministic result (flake.nix and flake.lock must be committed and clean). +2. **Digest verification**: After pushing, use `./docker/check_reproducibility.sh` to verify local and remote digests match, confirming the build is reproducible. +3. **Supply chain**: Pin the digest in `docker/DOCKER_REPRO_DIGEST` and `.circleci/config.yml` to ensure all builds reference an immutable, auditable image. +4. **Documentation**: Update the version comment in `docker/DOCKER_REPRO_DIGEST` so users know which image version is pinned. +5. **User migration**: When releasing a new version, communicate the new digest and version to users via release notes. + +**For forks and alternate maintainers**: +If you maintain a fork or want to test with a different Docker image repository, set `HEADS_MAINTAINER_DOCKER_IMAGE` before running any wrapper script: +```bash +# Example: use your own Docker image repository +export HEADS_MAINTAINER_DOCKER_IMAGE="youruser/heads-dev-env" + +# Now all scripts will reference your repository +./docker_local_dev.sh make BOARD=x230 +HEADS_CHECK_REPRODUCIBILITY=1 ./docker_local_dev.sh + +# Reproducibility check will compare against youruser/heads-dev-env:latest +# resolve_docker_image will use youruser/heads-dev-env as the base image ``` -This can be put in reproducible oneliners to ease maintainership. +Maintenance tip: The repository file `docker/DOCKER_REPRO_DIGEST` pins the canonical reproducible image used by `./docker_repro.sh`, ensuring immutable, secure builds. + +Acceptable formats include `sha256:<64-hex>`, `sha256-<64-hex>` (normalized to `sha256:`), or just `<64-hex>` (normalized to `sha256:`). The helper will normalize these formats and produce an image reference like `tlaurion/heads-dev-env@sha256:`. + +If you need to pin the convenience `./docker_latest.sh` wrapper, set the `DOCKER_LATEST_DIGEST` environment variable locally; we do not maintain a `docker/DOCKER_LATEST_DIGEST` file in the repository because 'latest' is a user-level convenience and should be explicitly chosen. When `DOCKER_LATEST_DIGEST` is unset, `./docker_latest.sh` may fall back to `DOCKER_REPRO_DIGEST` only when the base image matches the maintainer repo; otherwise it will prompt before using an unpinned `:latest` unless `HEADS_ALLOW_UNPINNED_LATEST=1` is set in the environment. + +Example: obtain the immutable digest for a published image and use it to force `docker_latest.sh` to use an immutable image: + +```bash +# 1) Obtain the digest for a published image (exact repo:name:tag form is required) +# +# Tip: inspect tags on Docker Hub: https://hub.docker.com/layers/tlaurion/heads-dev-env/ +# Click a tag to see details (Content type, Digest (sha256:...), Size, Last updated). +# Use the shown tag name with docker pull, e.g.: +# docker pull tlaurion/heads-dev-env:v0.2.7 +# +# Example: pull the image and then obtain its digest locally +# docker pull tlaurion/heads-dev-env:v0.2.7 +# ./docker/get_digest.sh tlaurion/heads-dev-env:v0.2.7 +# +# Or: query the registry for the digest and optionally pull it when prompted +# ./docker/get_digest.sh tlaurion/heads-dev-env:v0.2.7 +# (the script will show the remote digest and ask if you want to pull the image to create a local repo@digest) +# +# Use -y to auto-pull and return the digest in one go: +# ./docker/get_digest.sh -y tlaurion/heads-dev-env:v0.2.7 + +./docker/get_digest.sh tlaurion/heads-dev-env:v0.2.7 +# Output (example): tlaurion/heads-dev-env@sha256:50a9110c...\nsha256:50a9110c... + +# 2) If the image is not present locally, the helper will offer to pull it so a local repo@digest is available. +# Use '-y' / '--yes' to skip the interactive prompt and pull automatically. +./docker/get_digest.sh -y tlaurion/heads-dev-env:latest + +# 3) Export the raw digest into the env var expected by the wrapper +export DOCKER_LATEST_DIGEST=$(./docker/get_digest.sh tlaurion/heads-dev-env:latest | tail -n1) + +# 4) Run the convenience wrapper using the pinned digest +DOCKER_LATEST_DIGEST=$DOCKER_LATEST_DIGEST ./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 + +Note: when a digest is discovered, helpers print a concise summary to help auditing, for example: + + Image: tlaurion/heads-dev-env@sha256:50a9... + Digest: sha256:50a9... + Resolved from: local|registry API|env|file + Tip: export DOCKER_LATEST_DIGEST=sha256:50a9... + +This makes it easy to copy/pin digests or verify provenance. + +If you want to change what `./docker_latest.sh` uses as the "latest" image: +- For a temporary override: run `./docker/pin-and-run.sh -- ./docker_latest.sh ` to run the wrapper pinned to a specific digest. +- To set a local convenience env: `export DOCKER_LATEST_DIGEST=$(./docker/get_digest.sh tlaurion/heads-dev-env:vX.Y.Z | tail -n1)`. +- To change the canonical fallback used by the project: edit `docker/DOCKER_REPRO_DIGEST` with the desired digest and commit the change. + +# Convenience: helper to obtain a digest and run a wrapper pinned to that digest +# Example: obtains digest and runs the 'latest' wrapper pinned to that digest (explicit wrapper is recommended) +./docker/pin-and-run.sh tlaurion/heads-dev-env:v0.2.7 -- ./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 +# Auto-pull and run (non-interactive) +./docker/pin-and-run.sh -y tlaurion/heads-dev-env:v0.2.7 -- ./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 + +# Shortcut: omit the wrapper and just provide the command — the helper will use the default './docker_latest.sh' +./docker/pin-and-run.sh tlaurion/heads-dev-env:v0.2.7 -- make BOARD=qemu-coreboot-fbwhiptail-tpm2 + +# Explicit wrapper flag: use -w/--wrapper to avoid ambiguity +./docker/pin-and-run.sh -w ./docker_repro.sh tlaurion/heads-dev-env:v0.2.7 -- make BOARD=qemu-coreboot-fbwhiptail-tpm2 + -Test image in dirty mode: ``` -docker_version="vx.y.z" && docker_hub_repo="tlaurion/heads-dev-env" && sed "s@\(image: \)\(.*\):\(v[0-9]*\.[0-9]*\.[0-9]*\)@\1\2:$docker_version@" -i .circleci/config.yml && nix --print-build-logs --verbose develop --ignore-environment --command true && nix --print-build-logs --verbose build .#dockerImage && docker load < result && docker tag linuxboot/heads:dev-env "$docker_hub_repo:$docker_version" && docker push "$docker_hub_repo:$docker_version" + +Alternative (manual) commands without the helper script: + +```bash +docker pull tlaurion/heads-dev-env:latest +# prints full repo@digest (if available) +docker inspect --format='{{index .RepoDigests 0}}' tlaurion/heads-dev-env:latest +# to get only the digest portion: +docker inspect --format='{{index .RepoDigests 0}}' tlaurion/heads-dev-env:latest | cut -d'@' -f2 ``` +Notes: some registries or Docker versions may require `docker manifest inspect` or `skopeo inspect` to obtain an authoritative digest; the helper script tries `docker inspect` first, then `docker manifest inspect` when available. + +Update the appropriate file after publishing a new image to keep the repo in sync. + Notes: - Local builds can use ":latest" tag, which will use latest tested successful CircleCI run -- To reproduce CirlceCI results, make sure to use the same versioned tag declared under .circleci/config.yml's "image:" +- To reproduce CircleCI results, make sure to use the same versioned tag declared under .circleci/config.yml's "image:" diff --git a/docker/DOCKER_REPRO_DIGEST b/docker/DOCKER_REPRO_DIGEST new file mode 100644 index 000000000..7cb961358 --- /dev/null +++ b/docker/DOCKER_REPRO_DIGEST @@ -0,0 +1,12 @@ +# Optional: pin the Docker image used by ./docker_repro.sh to an immutable digest +# This file is read by docker_repro.sh if DOCKER_REPRO_DIGEST is not set in the +# environment. The first non-empty, non-comment line is used as the digest value. +# Acceptable formats are: +# - sha256:<64-hex> +# - sha256-<64-hex> (will be normalized to sha256:) +# - <64-hex> (will be normalized to sha256:) +# Example: +# sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + +# Place the digest on the first non-comment line below (remove the leading '#') +sha256-50a9110cdfc6a74a383169d7c624139c3b3e05567b87203498118a8a33dd79f1 diff --git a/docker/check_reproducibility.sh b/docker/check_reproducibility.sh new file mode 100755 index 000000000..731151737 --- /dev/null +++ b/docker/check_reproducibility.sh @@ -0,0 +1,53 @@ +#!/bin/bash +# Helper to compare local Docker image digest with remote docker.io +# Usage: ./docker/check_reproducibility.sh [local_image] [remote_image] +# Example: +# ./docker/check_reproducibility.sh linuxboot/heads:dev-env tlaurion/heads-dev-env:latest + +set -euo pipefail + +usage() { + cat <<'USAGE' >&2 +Usage: $0 [local_image] [remote_image] + +Compare a local Docker image digest with a remote docker.io image. + +Arguments: + local_image Local image to check (default: linuxboot/heads:dev-env) + remote_image Remote docker.io image to compare against (default: ${HEADS_MAINTAINER_DOCKER_IMAGE}:latest, where HEADS_MAINTAINER_DOCKER_IMAGE defaults to tlaurion/heads-dev-env) + +Environment: + HEADS_MAINTAINER_DOCKER_IMAGE Override the canonical maintainer's image repository (default: tlaurion/heads-dev-env) + +Examples: + ./docker/check_reproducibility.sh + ./docker/check_reproducibility.sh linuxboot/heads:dev-env tlaurion/heads-dev-env:latest + ./docker/check_reproducibility.sh linuxboot/heads:dev-env tlaurion/heads-dev-env:v0.2.7 + HEADS_MAINTAINER_DOCKER_IMAGE="myuser/heads-dev-env" ./docker/check_reproducibility.sh + +Requirements: + - docker CLI (required; to inspect local images and perform pulls) + - Recommended (optional): `skopeo` (preferred for manifest inspection without pulling), `jq` + `curl` (fallback to query Docker Hub API). If these are missing the script will fall back to `docker pull` which may download large image layers. + - Network access (to pull remote images or query registries) + +USAGE +} + +if [ "${1:-}" = "-h" ] || [ "${1:-}" = "--help" ]; then + usage + exit 0 +fi + +echo "=== Docker Image Reproducibility Check ===" >&2 +# Source shared helpers and delegate to centralized reproducibility checker +# shellcheck source=docker/common.sh +source "$(dirname "$0")/common.sh" +# Ensure docker is available +require_docker || exit $? +# Resolve local and remote images (remote uses shared defaulting logic) +local_image="${1:-linuxboot/heads:dev-env}" +remote_image=$(resolve_repro_remote_image "${2:-}") +# Delegate to the refactored checker which prefers image ID / config digest comparison +compare_image_reproducibility "${local_image}" "${remote_image}" +exit $? + diff --git a/docker/common.sh b/docker/common.sh new file mode 100755 index 000000000..50a812d9a --- /dev/null +++ b/docker/common.sh @@ -0,0 +1,1248 @@ +#!/bin/bash + +# Shared common Docker helpers for Heads dev scripts +# Meant to be sourced from docker_latest.sh / docker_local_dev.sh / docker_repro.sh +# +# This module provides: +# - ensure_nix_and_flakes() : Infrastructure setup and validation +# - resolve_docker_image() : Image reference resolution with digest pinning +# - maybe_rebuild_local_image() : Conditional Docker image rebuilding from flake +# - kill_usb_processes() : USB device cleanup for token passthrough +# - build_docker_opts() : Docker runtime options construction +# - run_docker() : Container execution wrapper +# - print_digest_info() : User-friendly digest output +# +# Environment variables and configuration are documented in the usage() function below. + +__HEADS_RESTORE_SHELL_OPTS=0 +if [ "${BASH_SOURCE[0]}" != "${0}" ]; then + __HEADS_SHELL_OPTS=$(set +o) + __HEADS_RESTORE_SHELL_OPTS=1 +fi +set -euo pipefail + +# Color support: enable when stderr is a TTY and not explicitly disabled +if [ -t 2 ] && [ -z "${HEADS_NO_COLOR:-}" ]; then + RED="$(printf '\033[31m')" + GREEN="$(printf '\033[32m')" + YELLOW="$(printf '\033[33m')" + BOLD="$(printf '\033[1m')" + RESET="$(printf '\033[0m')" + # Reference optional colors to avoid unused-variable warnings from shellcheck + : "${YELLOW}" "${BOLD}" +else + RED=""; GREEN=""; YELLOW=""; BOLD=""; RESET="" +fi + +# Simple print-once helper to avoid repeated messages during a run +# Usage: print_once +# Note: uses an associative array, requires bash +if [ -z "${__heads_printed_initialized:-}" ]; then + declare -A __heads_printed + __heads_printed_initialized=1 +fi +print_once() { + local key="$1"; shift + if [ -z "${__heads_printed[$key]:-}" ]; then + __heads_printed[$key]=1 + printf "%s\n" "$*" >&2 + fi +} + +# Ensure docker is available in PATH. +require_docker() { + if ! command -v docker >/dev/null 2>&1; then + echo "Error: docker not found in PATH" >&2 + return 127 + fi + return 0 +} + +# Interactive prompt helper to confirm pulls. Returns 0 to proceed, 1 to abort. +prompt_for_pull() { + local remote_image="$1" + # Respect explicit no-pull or auto-pull flags + if [ "${HEADS_CHECK_REPRODUCIBILITY_NO_PULL:-0}" = "1" ]; then + echo "Auto-pull suppressed by HEADS_CHECK_REPRODUCIBILITY_NO_PULL=1; aborting reproducibility check." >&2 + return 1 + fi + if [ "${HEADS_CHECK_REPRODUCIBILITY_AUTO_PULL:-0}" = "1" ]; then + return 0 + fi + # Interactive prompt + if [ -t 0 ]; then + printf "${BOLD}Pulling the remote image will download potentially large layers and may still result in a mismatch.${RESET} Continue and pull %s? [y/N] " "$remote_image" >&2 + read -r _ans + case "${_ans:-N}" in [Yy]*) return 0 ;; *) echo "Skipping pull; aborting reproducibility check." >&2; return 1 ;; esac + else + echo "Non-interactive session; set HEADS_CHECK_REPRODUCIBILITY_AUTO_PULL=1 to auto-pull or HEADS_CHECK_REPRODUCIBILITY_NO_PULL=1 to abort without pulling." >&2 + return 1 + fi +} + +# ================================================================ +# Configuration: Maintainer Docker image +# ================================================================ +# This is the canonical maintainer's Docker image repository. +# Override by setting HEADS_MAINTAINER_DOCKER_IMAGE in your environment +# for local testing or if you maintain a fork. +# Example: export HEADS_MAINTAINER_DOCKER_IMAGE="myuser/heads-dev-env" +HEADS_MAINTAINER_DOCKER_IMAGE="${HEADS_MAINTAINER_DOCKER_IMAGE:-tlaurion/heads-dev-env}" + +# For reproducibility checks, this specifies the remote image to compare against. +# If not set, defaults to ${HEADS_MAINTAINER_DOCKER_IMAGE}:latest +# Example: export HEADS_CHECK_REPRODUCIBILITY_REMOTE="tlaurion/heads-dev-env:v0.2.7" +HEADS_CHECK_REPRODUCIBILITY_REMOTE="${HEADS_CHECK_REPRODUCIBILITY_REMOTE:-}" + +# Resolve the default reproducibility remote image. +# Usage: resolve_repro_remote_image [override_image] +resolve_repro_remote_image() { + local override_image="${1:-}" + if [ -n "${override_image}" ]; then + echo "${override_image}" + return 0 + fi + if [ -n "${HEADS_CHECK_REPRODUCIBILITY_REMOTE:-}" ]; then + echo "${HEADS_CHECK_REPRODUCIBILITY_REMOTE}" + return 0 + fi + local img base + img="${HEADS_MAINTAINER_DOCKER_IMAGE:-tlaurion/heads-dev-env}" + base="${img##*/}" + if [[ "${base}" == *":"* || "${base}" == *"@"* ]]; then + echo "${img}" + else + echo "${img}:latest" + fi +} + +# Track whether we supply Xauthority into the container +DOCKER_XAUTH_USED=0 +# Track temporary Xauthority file for cleanup +DOCKER_XAUTH_FILE="" +DOCKER_XAUTH_TEMP=0 + +# ================================================================ +# Usage and informational functions +# ================================================================ + +usage() { + cat <<'USAGE' +Usage: $0 [OPTIONS] -- [COMMAND] +Options: +Environment variables (opt-ins / opt-outs): + HEADS_MAINTAINER_DOCKER_IMAGE Override the canonical maintainer's Docker image repository (default: tlaurion/heads-dev-env). Use for forks or local testing. + HEADS_CHECK_REPRODUCIBILITY_REMOTE Override the remote image for reproducibility checks (default: ${HEADS_MAINTAINER_DOCKER_IMAGE}:latest). Example: tlaurion/heads-dev-env:v0.2.7 + HEADS_DISABLE_USB=1 Disable automatic USB passthrough (default: enabled when /dev/bus/usb exists) + HEADS_X11_XAUTH=1 Explicitly mount $HOME/.Xauthority into the container for X11 auth + HEADS_SKIP_DOCKER_REBUILD=1 Skip automatic rebuild of the local Docker image when flake.nix/flake.lock are uncommitted + HEADS_CHECK_REPRODUCIBILITY=1 Verify reproducibility by comparing local image digest with remote (uses skopeo or curl/jq and network access) + HEADS_AUTO_INSTALL_NIX=1 Automatically install Nix (single-user) if it's missing (interactive prompt suppressed). For supply-chain safety, this helper will not auto-execute a downloaded installer unless + HEADS_NIX_INSTALLER_SHA256 is set to the expected sha256 of the installer. + HEADS_AUTO_ENABLE_FLAKES=1 Automatically enable flakes by writing to $HOME/.config/nix/nix.conf (if needed) + HEADS_SKIP_DISK_CHECK=1 Skip disk-space preflight check (default: perform check and warn) + HEADS_MIN_DISK_GB=50 Minimum disk free (GB) required on '/' or '/nix' (default: 50) +Command: + The command to run inside the Docker container, e.g., make BOARD=BOARD_NAME +USAGE +} + +# ================================================================ +# Infrastructure and setup functions +# ================================================================ + +# Build Nix Docker image from flake.nix/flake.lock with proper error handling. +# Verify Nix environment, build image, load it into Docker. +# Returns: 0 on success, 1 on failure +_build_nix_docker_image() { + # Ensure Nix and flakes are available; prompt if needed + ensure_nix_and_flakes || return 1 + + # Verify the Nix environment works with a simple develop test + echo "Verifying Nix environment..." >&2 + if ! nix develop --ignore-environment --command true; then + echo "Error: nix develop failed; see above output for diagnostics." >&2 + echo "Suggestion: ensure Nix is installed and flakes are enabled (see README.md)." >&2 + return 1 + fi + + # Build the Docker image from flake + echo "Building Docker image from flake.nix..." >&2 + if ! nix build .#dockerImage; then + echo "Error: nix build .#dockerImage failed; see above output for diagnostics." >&2 + return 1 + fi + + # Load the image into Docker + echo "Loading Docker image..." >&2 + if ! docker load < result; then + echo "Error: docker load failed." >&2 + return 1 + fi + + return 0 +} + +ensure_nix_and_flakes() { + # Check available disk space (on /nix if present, otherwise on /). Warn if < HEADS_MIN_DISK_GB (default 50GB). + if [ "${HEADS_SKIP_DISK_CHECK:-0}" != "1" ]; then + local min_gb=${HEADS_MIN_DISK_GB:-50} + local target="/" + if [ -d /nix ]; then target="/nix"; fi + # df -Pk reports 1K-blocks, available in $4 + local avail_kb + avail_kb=$(df -Pk "$target" | awk 'NR==2{print $4}') || avail_kb=0 + local required_kb=$((min_gb * 1024 * 1024)) + if [ "$avail_kb" -lt "$required_kb" ]; then + echo "Warning: building the docker image and populating /nix may require ${min_gb}GB+ free on ${target}." >&2 + echo "Detected available: $(df -h "$target" | awk 'NR==2{print $4}')" >&2 + if [ -t 0 ]; then + printf "Continue despite low disk space? [y/N] " >&2 + read -r _ans + case "${_ans:-N}" in + [Yy]* ) echo "Continuing despite low disk space." >&2 ;; + * ) echo "Aborting due to insufficient disk space." >&2; return 1 ;; + esac + else + echo "Non-interactive shell and insufficient disk space; aborting." >&2 + return 1 + fi + fi + fi + + # Ensure a downloader (curl or wget) is available for the Nix install script. + local downloader_cmd="" + if command -v curl >/dev/null 2>&1; then + downloader_cmd="curl -L" + elif command -v wget >/dev/null 2>&1; then + downloader_cmd="wget -qO-" + else + echo "Error: neither 'curl' nor 'wget' is available; one is required to fetch the Nix installer." >&2 + if [ -t 1 ]; then + echo "Please install 'curl' (recommended) or 'wget' and re-run this script." >&2 + echo "Examples (Debian/Ubuntu): sudo apt-get update && sudo apt-get install -y curl" >&2 + echo "(Fedora): sudo dnf install -y curl; (Arch): sudo pacman -Syu curl" >&2 + fi + return 1 + fi + + if ! command -v nix >/dev/null 2>&1; then + echo "Error: 'nix' not found on PATH." >&2 + echo "You can install Nix (single-user) with:" >&2 + echo " [ -d /nix ] || ${downloader_cmd} https://nixos.org/nix/install | sh -s -- --no-daemon" >&2 + + # Allow non-interactive automation when explicitly requested; checksum pinning still required. + if [ "${HEADS_AUTO_INSTALL_NIX:-0}" = "1" ]; then + echo "HEADS_AUTO_INSTALL_NIX=1: attempting automatic Nix install..." >&2 + local installer_url="https://nixos.org/nix/install" + local tmpf + tmpf=$(mktemp) || { echo "Failed to create temporary file for installer." >&2; return 1; } + if [ "$downloader_cmd" = "curl -L" ]; then + if ! curl -fsSL "$installer_url" -o "$tmpf"; then + echo "Failed to download Nix installer." >&2; rm -f "$tmpf"; return 1 + fi + else + if ! wget -qO "$tmpf" "$installer_url"; then + echo "Failed to download Nix installer." >&2; rm -f "$tmpf"; return 1 + fi + fi + local inst_sha + if command -v sha256sum >/dev/null 2>&1; then + inst_sha=$(sha256sum "$tmpf" | awk '{print $1}') || inst_sha="" + elif command -v shasum >/dev/null 2>&1; then + inst_sha=$(shasum -a 256 "$tmpf" | awk '{print $1}') || inst_sha="" + else + inst_sha="" + fi + if [ -n "$inst_sha" ]; then + echo "Downloaded Nix installer to: $tmpf" >&2 + echo "Installer sha256: $inst_sha" >&2 + else + echo "Downloaded Nix installer to: $tmpf (sha256 unavailable)" >&2 + fi + + # For supply-chain safety, always verify against published checksum when available. + # First attempt to fetch the published checksum + local published_sha="" + local sha_url="" + if [ -n "${HEADS_NIX_INSTALLER_VERSION:-}" ]; then + sha_url="https://releases.nixos.org/nix/${HEADS_NIX_INSTALLER_VERSION}/install.sha256" + fi + + if [ -n "${sha_url}" ]; then + if command -v curl >/dev/null 2>&1; then + published_sha=$(curl -fsSL "${sha_url}" 2>/dev/null | tr -d '[:space:]' || true) + elif command -v wget >/dev/null 2>&1; then + published_sha=$(wget -qO- "${sha_url}" 2>/dev/null | tr -d '[:space:]' || true) + fi + fi + + # If we have both published and downloaded checksums, validate they match + if [ -n "${inst_sha:-}" ] && [ -n "${published_sha}" ]; then + if [ "${inst_sha}" = "${published_sha}" ]; then + echo "✓ Downloaded installer sha256 validated against published checksum." >&2 + + # If HEADS_NIX_INSTALLER_SHA256 is already set, proceed with auto-install + if [ -n "${HEADS_NIX_INSTALLER_SHA256:-}" ] && [ "${HEADS_NIX_INSTALLER_SHA256}" = "${inst_sha}" ]; then + echo "Installer checksum matches HEADS_NIX_INSTALLER_SHA256; running installer..." >&2 + if ! sh "$tmpf" --no-daemon; then echo "Nix install failed." >&2; rm -f "$tmpf"; return 1; fi + rm -f "$tmpf" + export PATH="$HOME/.nix-profile/bin:$PATH" || true + hash -r 2>/dev/null || true + else + # HEADS_NIX_INSTALLER_SHA256 not set, but we've validated the installer. Suggest setting it and re-running. + echo "" >&2 + echo "Installer validated. To enable automatic installation, re-run:" >&2 + _suggest_nix_installer_rerun "${inst_sha}" + echo "" >&2 + echo "Or verify manually and run: sh $tmpf --no-daemon" >&2 + rm -f "$tmpf" + return 1 + fi + else + echo "Error: Downloaded installer checksum does not match published checksum!" >&2 + echo "Downloaded: ${inst_sha}" >&2 + echo "Published: ${published_sha}" >&2 + echo "URL: ${sha_url}" >&2 + rm -f "$tmpf" + return 1 + fi + elif [ -n "${inst_sha:-}" ] && [ -n "${HEADS_NIX_INSTALLER_SHA256:-}" ]; then + # We have HEADS_NIX_INSTALLER_SHA256 set but no published checksum to validate against + if [ "${HEADS_NIX_INSTALLER_SHA256}" = "${inst_sha}" ]; then + echo "Installer checksum matches HEADS_NIX_INSTALLER_SHA256; running installer..." >&2 + if ! sh "$tmpf" --no-daemon; then echo "Nix install failed." >&2; rm -f "$tmpf"; return 1; fi + rm -f "$tmpf" + export PATH="$HOME/.nix-profile/bin:$PATH" || true + hash -r 2>/dev/null || true + else + echo "Error: Downloaded installer checksum does not match HEADS_NIX_INSTALLER_SHA256." >&2 + echo "Downloaded: ${inst_sha}" >&2 + echo "Expected: ${HEADS_NIX_INSTALLER_SHA256}" >&2 + rm -f "$tmpf" + return 1 + fi + else + # Unable to validate; ask user to verify manually + echo "For supply-chain safety, this helper will not execute the installer without verification." >&2 + echo "" >&2 + if [ -n "${inst_sha:-}" ]; then + echo "Downloaded installer sha256: ${inst_sha}" >&2 + fi + if [ -n "${sha_url}" ]; then + echo "You can verify it at: ${sha_url}" >&2 + echo "" >&2 + if [ -n "${inst_sha:-}" ]; then + echo "Verification passed? Re-run with:" >&2 + _suggest_nix_installer_rerun "${inst_sha}" + fi + else + echo "Published checksum unavailable; verify the installer before running it." >&2 + fi + echo "" >&2 + echo "Or run manually when ready: sh $tmpf --no-daemon" >&2 + rm -f "$tmpf" + return 1 + fi + elif [ -t 0 ]; then + echo "Note: building the Docker image and populating /nix may require ${HEADS_MIN_DISK_GB:-50}GB+ free on '/' or '/nix'." >&2 + printf "Install Nix now and enable flakes (required) [Y/n]? " >&2 + read -r ans + case "${ans:-Y}" in + [Yy]* ) + # Determine installer URL. If HEADS_NIX_INSTALLER_VERSION is set, use a pinned release URL + # (e.g. https://releases.nixos.org/nix/nix-2.33.2/install and its .sha256). Otherwise fall back to the + # canonical script at https://nixos.org/nix/install. Users may also set HEADS_NIX_INSTALLER_URL to override. + local installer_url + local sha_url + if [ -n "${HEADS_NIX_INSTALLER_VERSION:-}" ]; then + installer_url="https://releases.nixos.org/nix/${HEADS_NIX_INSTALLER_VERSION}/install" + sha_url="${installer_url}.sha256" + elif [ -n "${HEADS_NIX_INSTALLER_URL:-}" ]; then + installer_url="${HEADS_NIX_INSTALLER_URL}" + sha_url="" + else + installer_url="https://nixos.org/nix/install" + sha_url="" + fi + + local tmpf + tmpf=$(mktemp) || { echo "Failed to create temporary file for installer." >&2; return 1; } + if [ "$downloader_cmd" = "curl -L" ]; then + if ! curl -fsSL "$installer_url" -o "$tmpf"; then + echo "Failed to download Nix installer from $installer_url." >&2; rm -f "$tmpf"; return 1 + fi + else + if ! wget -qO "$tmpf" "$installer_url"; then + echo "Failed to download Nix installer from $installer_url." >&2; rm -f "$tmpf"; return 1 + fi + fi + local inst_sha + if command -v sha256sum >/dev/null 2>&1; then + inst_sha=$(sha256sum "$tmpf" | awk '{print $1}') || inst_sha="" + elif command -v shasum >/dev/null 2>&1; then + inst_sha=$(shasum -a 256 "$tmpf" | awk '{print $1}') || inst_sha="" + else + inst_sha="" + fi + if [ -n "$inst_sha" ]; then + echo "Downloaded Nix installer to: $tmpf" >&2 + echo "Installer sha256: $inst_sha" >&2 + else + echo "Downloaded Nix installer to: $tmpf (sha256 unavailable)" >&2 + fi + + # Show the installer URL and attempt to detect a version string from the installer contents. + echo "Installer URL: ${installer_url}" >&2 + installer_detected_version=$(sed -n '1,200p' "$tmpf" | tr -d '\r' | grep -oE 'nix-[0-9]+(\.[0-9]+)*' | head -n1 || true) + if [ -n "${installer_detected_version}" ]; then + echo "Detected installer version (heuristic): ${installer_detected_version}" >&2 + fi + + # If we can derive a .sha256 URL (releases.nixos.org), try to fetch it and show it to the user so they + # can verify the downloaded installer. Do not treat failure to fetch the .sha256 as fatal; it's advisory. + remote_inst_sha="" + # Prefer explicit sha_url (set via HEADS_NIX_INSTALLER_VERSION or HEADS_NIX_INSTALLER_URL override) + candidate_sha_urls=() + if [ -n "${sha_url:-}" ]; then + candidate_sha_urls+=("${sha_url}") + fi + # If we heuristically detected a version, suggest the canonical releases URL + if [ -n "${installer_detected_version}" ]; then + candidate_sha_urls+=("https://releases.nixos.org/nix/${installer_detected_version}/install.sha256") + fi + + for candidate in "${candidate_sha_urls[@]:-}"; do + echo "Attempting to fetch published sha256 from: ${candidate}" >&2 + if command -v curl >/dev/null 2>&1; then + remote_inst_sha=$(curl -fsSL "${candidate}" 2>/dev/null | tr -d '[:space:]' || true) + elif command -v wget >/dev/null 2>&1; then + remote_inst_sha=$(wget -qO- "${candidate}" 2>/dev/null | tr -d '[:space:]' || true) + else + remote_inst_sha="" + fi + if [ -n "${remote_inst_sha:-}" ]; then + echo "Published sha256 (from ${candidate}): ${remote_inst_sha}" >&2 + if [ -n "$inst_sha" ] && [ "$inst_sha" = "$remote_inst_sha" ]; then + echo "Published sha256 matches downloaded installer." >&2 + else + echo "Warning: published sha256 does NOT match downloaded installer; do not run automatically." >&2 + fi + break + fi + done + + if [ -z "${remote_inst_sha:-}" ] && [ ${#candidate_sha_urls[@]} -gt 0 ]; then + echo "Note: could not fetch published sha256 from any of the suggested locations." >&2 + fi + # For supply-chain safety, require a pinned installer hash to auto-execute; otherwise instruct user to run manually. + if [ -n "${inst_sha:-}" ] && [ -n "${HEADS_NIX_INSTALLER_SHA256:-}" ]; then + # Check if HEADS_NIX_INSTALLER_SHA256 matches the published checksum (if available) + local checksum_valid=false + if [ -n "${remote_inst_sha:-}" ] && [ "${HEADS_NIX_INSTALLER_SHA256}" = "${remote_inst_sha}" ] && [ "${HEADS_NIX_INSTALLER_SHA256}" = "${inst_sha}" ]; then + checksum_valid=true + echo "Installer checksum matches HEADS_NIX_INSTALLER_SHA256 and published checksum; running installer..." >&2 + elif [ -n "${remote_inst_sha:-}" ] && [ "${HEADS_NIX_INSTALLER_SHA256}" != "${remote_inst_sha}" ]; then + echo "Error: HEADS_NIX_INSTALLER_SHA256 does not match published checksum" >&2 + echo "Published checksum: ${remote_inst_sha}" >&2 + echo "HEADS_NIX_INSTALLER_SHA256: ${HEADS_NIX_INSTALLER_SHA256}" >&2 + else + # Require published checksum verification for security - no fallback allowed + echo "Error: Cannot verify installer against published checksum for automatic execution." >&2 + echo "Published checksum could not be fetched or does not match HEADS_NIX_INSTALLER_SHA256." >&2 + if [ -z "${remote_inst_sha:-}" ]; then + echo "No published checksum available from any source." >&2 + fi + fi + + if [ "$checksum_valid" = true ]; then + if ! sh "$tmpf" --no-daemon; then echo "Nix install failed." >&2; rm -f "$tmpf"; return 1; fi + rm -f "$tmpf" + export PATH="$HOME/.nix-profile/bin:$PATH" || true + hash -r 2>/dev/null || true + else + echo "For supply-chain safety this helper will not execute a downloaded installer automatically." >&2 + echo "Installer saved to: $tmpf" >&2 + echo "Installer sha256: ${inst_sha}" >&2 + echo "" >&2 + echo "To complete Nix installation, verify the installer and re-run:" >&2 + _suggest_nix_installer_rerun "${inst_sha}" + echo "" >&2 + echo "Or run manually when ready: sh $tmpf --no-daemon" >&2 + rm -f "$tmpf" + return 1 + fi + else + echo "For supply-chain safety this helper will not execute a downloaded installer automatically." >&2 + echo "Installer saved to: $tmpf" >&2 + if [ -n "${inst_sha:-}" ]; then + echo "Installer sha256: ${inst_sha}" >&2 + echo "" >&2 + echo "To complete Nix installation, verify the installer and re-run:" >&2 + _suggest_nix_installer_rerun "${inst_sha}" + else + echo "sha256 unavailable; verify the downloaded installer before running it." >&2 + fi + echo "" >&2 + echo "Or run manually when ready: sh $tmpf --no-daemon" >&2 + rm -f "$tmpf" + return 1 + fi + ;; + * ) echo "Flakes are required; aborting." >&2; return 1 ;; + esac + else + echo "Non-interactive shell: cannot install Nix automatically. Please install Nix and enable flakes (see README.md)." >&2 + return 1 + fi + fi + + mkdir -p "$HOME/.config/nix" + if ! grep -q "nix-command" "$HOME/.config/nix/nix.conf" 2>/dev/null && ! grep -q "nix-command" /etc/nix/nix.conf 2>/dev/null; then + if [ "${HEADS_AUTO_ENABLE_FLAKES:-0}" = "1" ]; then + echo "Enabling flakes by writing 'experimental-features = nix-command flakes' to $HOME/.config/nix/nix.conf" >&2 + echo "experimental-features = nix-command flakes" >> "$HOME/.config/nix/nix.conf" || true + elif [ -t 0 ]; then + printf "Flakes are required but not enabled. Add 'experimental-features = nix-command flakes' to %s now [Y/n]? " "$HOME/.config/nix/nix.conf" >&2 + read -r ans2 + case "${ans2:-Y}" in + [Yy]* ) echo "experimental-features = nix-command flakes" >> "$HOME/.config/nix/nix.conf" || true; echo "Wrote experimental features to $HOME/.config/nix/nix.conf" >&2 ;; + * ) echo "Flakes are required; aborting. Please enable flakes manually and rerun the script." >&2; return 1 ;; + esac + else + echo "Flakes are required but not enabled in non-interactive shell. Please enable them and rerun the script (see README.md)." >&2 + return 1 + fi + fi +} + +# Build and suggest a re-run command with pinned installer hash and preserved environment variables +# Usage: _suggest_nix_installer_rerun +_suggest_nix_installer_rerun() { + local inst_sha="$1" + local rerun_cmd="HEADS_NIX_INSTALLER_SHA256=${inst_sha} HEADS_AUTO_INSTALL_NIX=1" + + if [ -n "${HEADS_MAINTAINER_DOCKER_IMAGE:-}" ]; then + rerun_cmd="$rerun_cmd HEADS_MAINTAINER_DOCKER_IMAGE='${HEADS_MAINTAINER_DOCKER_IMAGE}'" + fi + if [ -n "${HEADS_CHECK_REPRODUCIBILITY_REMOTE:-}" ]; then + rerun_cmd="$rerun_cmd HEADS_CHECK_REPRODUCIBILITY_REMOTE='${HEADS_CHECK_REPRODUCIBILITY_REMOTE}'" + fi + if [ "${HEADS_CHECK_REPRODUCIBILITY:-0}" = "1" ]; then + rerun_cmd="$rerun_cmd HEADS_CHECK_REPRODUCIBILITY=1" + fi + rerun_cmd="$rerun_cmd $0" + + echo " $rerun_cmd" >&2 +} + +# ================================================================ +# USB device management functions +# ================================================================ + +# Kill scdaemon/pcscd when USB passthrough is present (minimal, automatic). Only targets processes that are actually using USB device nodes. +kill_usb_processes() { + [ -d /dev/bus/usb ] || return 0 + [ "${HEADS_DISABLE_USB:-0}" = "1" ] && { echo "HEADS_DISABLE_USB=1: skipping USB cleanup" >&2; return 0; } + + # Use lsof to find processes holding /dev/bus/usb nodes, then filter for scdaemon/pcscd + local pids + + # Choose how to run lsof: prefer direct invocation as root, else use sudo if available without prompting. + local lsof_cmd="" + if [ "$(id -u)" = "0" ]; then + lsof_cmd="lsof" + elif command -v sudo >/dev/null 2>&1 && sudo -n true 2>/dev/null; then + lsof_cmd="sudo lsof" + elif command -v sudo >/dev/null 2>&1 && [ -t 1 ]; then + # Interactive shell with sudo available: attempt it (will prompt for password) + echo "Attempting to check USB device usage; sudo access required:" >&2 + lsof_cmd="sudo lsof" + elif command -v sudo >/dev/null 2>&1; then + # Non-interactive shell and sudo would prompt: skip cleanup + echo "sudo requires a password; skipping automatic USB cleanup in this context" >&2 + return 0 + elif command -v lsof >/dev/null 2>&1; then + # No sudo, but lsof present; attempt to run it (may fail if insufficient permissions) + lsof_cmd="lsof" + else + echo "lsof not available; cannot detect processes holding USB devices; skipping cleanup" >&2 + return 0 + fi + + # Match all bus/device nodes to avoid missing higher-numbered buses (no assumption about leading zeros). + # Use lsof -t to obtain PIDs only, then filter those PIDs for the commands we care about so we + # only attempt to kill numeric PIDs (avoid passing ps headers or other text to kill). + local raw_pids + raw_pids=$($lsof_cmd -t /dev/bus/usb/*/* 2>/dev/null || true) + if [ -z "${raw_pids}" ]; then + [ "${HEADS_USB_VERBOSE:-0}" = "1" ] && echo "No processes holding /dev/bus/usb nodes." >&2 + return 0 + fi + + local -a matched_pids=() + for _pid in ${raw_pids}; do + # Ensure _pid is numeric + case "${_pid}" in + ''|*[!0-9]* ) continue ;; + *) + # Get command name and match exactly 'scdaemon' or 'pcscd' + cmd=$(ps -p "${_pid}" -o comm= 2>/dev/null || true) + if printf '%s' "${cmd}" | grep -qE '^scdaemon$|^pcscd$'; then + matched_pids+=("${_pid}") + fi + ;; + esac + done + + if [ ${#matched_pids[@]} -eq 0 ]; then + [ "${HEADS_USB_VERBOSE:-0}" = "1" ] && echo "No scdaemon/pcscd processes using USB devices." >&2 + return 0 + fi + + # Join the PIDs into a space-separated string for messaging and kill commands + pids="${matched_pids[*]}" + echo "Detected scdaemon/pcscd processes using USB devices: ${pids}" >&2 + echo "WARNING: About to kill the above processes to free USB devices for passthrough. To skip this automatic action set HEADS_DISABLE_USB=1 in your environment." >&2 + if [ -t 1 ]; then + echo "Press Ctrl-C to abort within 3 seconds if you do NOT want these processes killed." >&2 + sleep 3 + fi + + # Try to kill: prefer running as root, else try sudo without prompting in non-interactive shells + # Convert the whitespace-separated PID list into an array for safe expansion + read -r -a pids_array <<< "${pids}" + + if [ "$(id -u)" = "0" ]; then + if kill -9 "${pids_array[@]}" 2>/dev/null; then + echo "Killed PIDs: ${pids}" >&2 + else + echo "Failed to kill some PIDs: ${pids}" >&2 + fi + elif command -v sudo >/dev/null 2>&1 && sudo -n true 2>/dev/null; then + if sudo kill -9 "${pids_array[@]}" 2>/dev/null; then + echo "Killed PIDs: ${pids}" >&2 + else + echo "Failed to kill some PIDs: ${pids}" >&2 + fi + elif [ -t 1 ]; then + # Interactive and sudo present but may prompt for password; attempt it so user can enter password. + if command -v sudo >/dev/null 2>&1; then + echo "Attempting to free USB devices for Docker passthrough; sudo access required:" >&2 + if sudo kill -9 "${pids_array[@]}"; then + echo "Killed PIDs: ${pids}" >&2 + else + echo "Failed to kill some PIDs: ${pids}" >&2 + fi + else + echo "Interactive shell but sudo not available; please run: kill -9 ${pids}" >&2 + fi + else + echo "Non-interactive: unable to kill PIDs (sudo not available or would prompt); please run: sudo kill -9 ${pids}" >&2 + fi +} + +# Rebuild local Docker image when flake.nix or flake.lock are modified and repo is dirty. +# Opt-out by setting HEADS_SKIP_DOCKER_REBUILD=1 in the environment. +maybe_rebuild_local_image() { + local image="$1" + + if [ "${HEADS_SKIP_DOCKER_REBUILD:-0}" = "1" ]; then + echo "HEADS_SKIP_DOCKER_REBUILD=1: skipping Docker rebuild" >&2 + return 0 + fi + + # Check if flake.nix or flake.lock have uncommitted changes + if git rev-parse --is-inside-work-tree >/dev/null 2>&1 && [ -n "$(git status --porcelain | grep -E 'flake\.nix|flake\.lock' || true)" ]; then + # There are uncommitted changes in flake files + echo "**Warning: Uncommitted changes detected in flake.nix or flake.lock. The Docker image will be rebuilt!**" >&2 + echo "If this was not intended, please CTRL-C now, commit your changes and rerun the script." >&2 + else + # No changes in flake files; check if image exists locally + local image_name + # Extract repository name without tag/digest, preserving registry host:port + # Strip @digest if present + image_name="${image%@*}" + # Now strip :tag from the last path component only (preserves host:port) + if [[ "$image_name" == */* ]]; then + # Has path components; extract prefix and suffix around last / + local prefix="${image_name%/*}" + local suffix="${image_name##*/}" + # Strip :tag from suffix only + suffix="${suffix%:*}" + image_name="${prefix}/${suffix}" + else + # No path; strip :tag from entire string + image_name="${image_name%:*}" + fi + + if docker images --format '{{.Repository}}:{{.Tag}}' | grep -q "^${image_name}:"; then + echo "Git repository is clean. Using existing Docker image." >&2 + return 0 + fi + + # Image doesn't exist; try to load from build result + if [ -L "result" ] && [ -e "result" ]; then + # Show where the 'result' symlink points and its size to give the user clear feedback + local result_target + result_target=$(readlink -f result 2>/dev/null || echo result) + local result_size="" + if [ -f "${result_target}" ]; then + result_size=$(stat -c '%s' "${result_target}" 2>/dev/null || echo "") + fi + echo "Git repository is clean but Docker image not found locally; loading existing build result..." >&2 + printf " Loading from: %s%s\n" "${result_target}" "${result_size:+ (size: ${result_size} bytes)}" >&2 + echo " This may take a few minutes depending on image size and disk I/O. Showing 'docker load' output below:" >&2 + + # If 'result' is a symlink, mention it explicitly (show this before running docker load) + if [ -L result ]; then + printf " Note: 'result' is a symlink to: %s\n" "${result_target}" >&2 + fi + + echo " Running: docker load < ${result_target}" >&2 + # Run 'docker load' directly so its output is printed live to the console in both + # interactive and non-interactive contexts (no piping/redirection of docker output). + if docker load < "${result_target}"; then + echo " docker load completed successfully" >&2 + else + echo " docker load failed (see output above)" >&2 + fi + + # Attempt to show the loaded image summary (best-effort for the requested image name) + local found + found=$(docker images --format '{{.Repository}}:{{.Tag}}\t{{.ID}}\t{{.Size}}' | grep -E "^${image%%:*}" | head -n1 || true) + if [ -n "${found}" ]; then + printf " Found image: %s\n" "${found}" >&2 + else + echo " Note: could not find a matching repo tag in 'docker images'; run 'docker images' to inspect available images." >&2 + fi + + return 0 + fi + + # No image and no build result; need to build + echo "Git repository is clean but Docker image '${image}' not found locally. Building from flake.nix..." >&2 + fi + + # Build the Docker image using the helper function + _build_nix_docker_image || return 1 + + return 0 +} + +# ================================================================ +# Image resolution and validation functions +# ================================================================ + +# Resolve Docker image preferring a pinned digest from the environment or a repository file. +# Usage: resolve_docker_image [prompt_on_latest] +# - : e.g. ${HEADS_MAINTAINER_DOCKER_IMAGE}:vX.Y.Z or ${HEADS_MAINTAINER_DOCKER_IMAGE}:latest +# - : name of env var to consult (e.g. DOCKER_REPRO_DIGEST) +# - : filename under the repo's docker/ directory to read if env var is unset +# - [prompt_on_latest]: if '1', prompt interactively before using an unpinned ':latest' when no digest is found +resolve_docker_image() { + local fallback_image="$1" + local digest_env_varname="$2" + local digest_filename="$3" + local prompt_on_latest="${4:-1}" + + # If the caller already supplied a digest (image@sha256:...), return as-is + if [[ "${fallback_image}" == *@* ]]; then + echo "${fallback_image}" + return 0 + fi + + # Check environment variable first + local digest_value="" + digest_value="${!digest_env_varname:-}" + local digest_source="" + if [ -n "${digest_value}" ]; then + digest_source="env ${digest_env_varname}" + fi + + # If not present in env, look for a repository file under docker/ + if [ -z "${digest_value}" ]; then + local repo_dir + repo_dir=$(cd "$(dirname "$0")" && pwd) + local digest_file="$repo_dir/docker/${digest_filename}" + if [ -f "${digest_file}" ]; then + digest_value=$(sed -n 's/#.*//; /^[[:space:]]*$/d; p' "${digest_file}" | head -n1 || true) + digest_source="file ${repo_dir}/docker/${digest_filename}" + fi + + # Special-case: if we're resolving the LATEST digest and none is provided, fall + # back to the REPRO digest (env var first, then repository file) since the + # latest convenience image normally mirrors the repro image in practice. + if [ -z "${digest_value}" ] && [ "${digest_env_varname}" = "DOCKER_LATEST_DIGEST" ]; then + local allow_latest_fallback=0 + local fallback_repo + fallback_repo="${fallback_image%%@*}" + local _fallback_last="${fallback_repo##*/}" + if [[ "${_fallback_last}" == *:* ]]; then + fallback_repo="${fallback_repo%:*}" + fi + + if [ "${fallback_repo}" = "${HEADS_MAINTAINER_DOCKER_IMAGE}" ]; then + allow_latest_fallback=1 + fi + + if [ -n "${DOCKER_REPRO_DIGEST:-}" ] && [[ "${DOCKER_REPRO_DIGEST}" == *@* ]]; then + local repro_repo="${DOCKER_REPRO_DIGEST%@*}" + if [ "${repro_repo}" != "${fallback_repo}" ]; then + allow_latest_fallback=0 + echo "Note: DOCKER_REPRO_DIGEST points to '${repro_repo}', not '${fallback_repo}'; not using it for latest image." >&2 + fi + fi + + if [ "${allow_latest_fallback}" -eq 1 ]; then + if [ -n "${DOCKER_REPRO_DIGEST:-}" ]; then + digest_value="${DOCKER_REPRO_DIGEST}" + digest_source="env DOCKER_REPRO_DIGEST" + else + local repro_file="$repo_dir/docker/DOCKER_REPRO_DIGEST" + if [ -f "${repro_file}" ]; then + digest_value=$(sed -n 's/#.*//; /^[[:space:]]*$/d; p' "${repro_file}" | head -n1 || true) + digest_source="file ${repo_dir}/docker/DOCKER_REPRO_DIGEST" + fi + fi + if [ -n "${digest_value}" ]; then + echo "Note: no DOCKER_LATEST_DIGEST set; using DOCKER_REPRO_DIGEST as fallback for latest image." >&2 + echo "To change which image 'latest' points to, either:" >&2 + echo " - Export a digest for convenience: export DOCKER_LATEST_DIGEST=sha256:" >&2 + echo " (get a digest with: ./docker/get_digest.sh tlaurion/heads-dev-env:vX.Y.Z | tail -n1)" >&2 + echo " - Or update the canonical file: edit 'docker/DOCKER_REPRO_DIGEST' in this repo to a preferred digest and commit it." >&2 + echo " - For one-off runs use the pin-and-run helper: ./docker/pin-and-run.sh -- ./docker_latest.sh " >&2 + fi + fi + fi + fi + + if [ -n "${digest_value}" ]; then + # Allow digest_value to be either a full 'repo@digest' or just the digest itself. + # Trim whitespace/newlines + digest_value=$(printf '%s' "${digest_value}" | tr -d '[:space:]') + + # If the value already contains an '@', treat it as a full image reference and normalize digest form below. + if [[ "${digest_value}" == *@* ]]; then + # Normalize possible 'sha256-' -> 'sha256:' or raw hex -> 'sha256:' inside the trailing part + local prefix=${digest_value%@*} + local trailing=${digest_value#*@} + if [[ "$trailing" =~ ^sha256-[0-9a-fA-F]{64}$ ]]; then + trailing="${trailing/-/:}" + elif [[ "$trailing" =~ ^[0-9a-fA-F]{64}$ ]]; then + trailing="sha256:${trailing}" + fi + # Final validation: ensure trailing digest is exactly in the expected format after normalization + if [[ ! "$trailing" =~ ^sha256:[0-9a-fA-F]{64}$ ]]; then + echo "Error: Invalid digest format '${trailing}' in '${digest_value}'; expected sha256:<64 hex characters>" >&2 + return 1 + fi + local image_ref="${prefix}@${trailing}" + print_digest_info "${image_ref}" "${trailing}" "${digest_source}" "${digest_env_varname}" + echo "${image_ref}" + return 0 + fi + + # Normalize forms: accept 'sha256-' or raw 64-hex by converting them to 'sha256:' + if [[ "${digest_value}" =~ ^sha256-[0-9a-fA-F]{64}$ ]]; then + digest_value="${digest_value/-/:}" + elif [[ "${digest_value}" =~ ^[0-9a-fA-F]{64}$ ]]; then + digest_value="sha256:${digest_value}" + fi + + # Final validation: ensure digest_value is exactly in the expected format after normalization + if [[ ! "${digest_value}" =~ ^sha256:[0-9a-fA-F]{64}$ ]]; then + echo "Error: Invalid digest format '${digest_value}'; expected sha256:<64 hex characters>" >&2 + return 1 + fi + + # Strip any existing digest and, if present, a tag after the last '/' from fallback_image + # to get the repository name. This preserves registry prefixes like 'registry.example.com:5000/' + local image_repo + # First, drop any '@digest' suffix from the fallback image + image_repo="${fallback_image%%@*}" + # Then, if the last path component contains a ':', treat that as a tag and strip it + local _last_component="${image_repo##*/}" + if [[ "${_last_component}" == *:* ]]; then + image_repo="${image_repo%:*}" + fi + print_digest_info "${image_repo}@${digest_value}" "${digest_value}" "${digest_source}" "${digest_env_varname}" + echo "${image_repo}@${digest_value}" + return 0 + fi + + # No digest available; handle prompts for unpinned :latest if requested + if [[ "${fallback_image}" == *":latest" && "${HEADS_ALLOW_UNPINNED_LATEST:-0}" != "1" && "${prompt_on_latest}" = "1" ]]; then + if [ -t 0 ]; then + printf "The configured image '%s' is unpinned (':latest'). Proceed despite supply-chain risk? [y/N] " "${fallback_image}" >&2 + read -r _ans + case "${_ans:-N}" in + [Yy]* ) echo "Proceeding with unpinned image." >&2 ;; + * ) printf "Aborting: set %s to pin an immutable image or set HEADS_ALLOW_UNPINNED_LATEST=1 to bypass this prompt.\n" "${digest_env_varname}" >&2; return 1 ;; + esac + else + echo "Refusing to use unpinned ':latest' in non-interactive mode without HEADS_ALLOW_UNPINNED_LATEST=1; aborting." >&2 + return 1 + fi + fi + + # No digest and no prompting required; return the fallback image as-is + echo "${fallback_image}" +} + + +# ================================================================ +# Utility functions +# ================================================================ + +# Print concise, consistent digest information for users and scripts. +# Usage: print_digest_info [] [] +print_digest_info() { + local image_ref="${1:-}" + local digest="${2:-}" + local source="${3:-}" + local envvar="${4:-}" + + # Keep output explicit and easy to copy into an export command + echo "Image: ${image_ref}" >&2 + echo "Digest: ${digest}" >&2 + if [ -n "${source}" ]; then + echo "Resolved from: ${source}" >&2 + fi + if [ -n "${envvar}" ]; then + echo "Tip: To force this image in future: export ${envvar}=${digest}" >&2 + else + echo 'Tip: To force a wrapper to use this image next time, export the digest, e.g.:' >&2 + printf " export DOCKER_LATEST_DIGEST=%s\n" "${digest}" >&2 + fi +} + + +# ================================================================ +# Docker execution and configuration functions +# ================================================================ + +# Build docker options (returns single string on stdout) +build_docker_opts() { + local opts=( -e "DISPLAY=${DISPLAY:-}" --network host --rm -ti ) + + # USB passthrough + if [ -d "/dev/bus/usb" ] && [ "${HEADS_DISABLE_USB:-0}" != "1" ]; then + opts+=( --device=/dev/bus/usb:/dev/bus/usb ) + echo "--->USB passthrough enabled; to disable set HEADS_DISABLE_USB=1" >&2 + elif [ -d "/dev/bus/usb" ]; then + echo "--->Host USB present; USB passthrough disabled by HEADS_DISABLE_USB=1" >&2 + fi + + # KVM passthrough + if [ -e /dev/kvm ]; then + opts+=( --device=/dev/kvm:/dev/kvm ) + echo "--->Host KVM device found; enabling /dev/kvm passthrough" >&2 + elif [ -e /proc/kvm ]; then + echo "--->Host reports KVM available but /dev/kvm is missing; load kvm module" >&2 + fi + + # X11 forwarding: mount socket and try programmatic Xauthority when possible + if [ -d "/tmp/.X11-unix" ]; then + opts+=( -v /tmp/.X11-unix:/tmp/.X11-unix ) + + # If the user explicitly requests to use their $HOME/.Xauthority, honor that and bypass programmatic cookie logic. + if [ "${HEADS_X11_XAUTH:-0}" != "0" ]; then + if [ -f "${HOME}/.Xauthority" ]; then + DOCKER_XAUTH_USED=1 + opts+=( -v "${HOME}/.Xauthority:/root/.Xauthority:ro" -e "XAUTHORITY=/root/.Xauthority" ) + echo "--->HEADS_X11_XAUTH set: mounting ${HOME}/.Xauthority into container and bypassing programmatic Xauthority" >&2 + else + echo "--->HEADS_X11_XAUTH set but ${HOME}/.Xauthority not found; not attempting programmatic Xauthority; GUI may fail" >&2 + fi + elif command -v xauth >/dev/null 2>&1; then + local XAUTH_HOST + XAUTH_HOST="" + if command -v mktemp >/dev/null 2>&1; then + XAUTH_HOST=$(mktemp -t heads-docker-xauth-XXXXXX 2>/dev/null || true) + fi + if [ -z "${XAUTH_HOST}" ]; then + XAUTH_HOST="/tmp/.docker.xauth-$(id -u)" + DOCKER_XAUTH_TEMP=0 + DOCKER_XAUTH_FILE="" + else + DOCKER_XAUTH_TEMP=1 + DOCKER_XAUTH_FILE="$XAUTH_HOST" + fi + # Create Xauthority file securely (restrict permissions) to avoid leaking the X11 cookie. + # Use a restrictive umask so the file is created with 0600, and ensure chmod enforces it. + local old_umask + old_umask=$(umask) + umask 077 + : >"$XAUTH_HOST" 2>/dev/null || true + umask "$old_umask" + chmod 600 "$XAUTH_HOST" 2>/dev/null || true + xauth nlist "${DISPLAY}" 2>/dev/null | sed -e 's/^..../ffff/' | xauth -f "$XAUTH_HOST" nmerge - 2>/dev/null || true + if [ -s "$XAUTH_HOST" ]; then + DOCKER_XAUTH_USED=1 + opts+=( -v "$XAUTH_HOST:$XAUTH_HOST:ro" -e "XAUTHORITY=$XAUTH_HOST" ) + echo "--->Using programmatic Xauthority $XAUTH_HOST for X11 auth" >&2 + elif [ -f "${HOME}/.Xauthority" ]; then + DOCKER_XAUTH_USED=1 + opts+=( -v "${HOME}/.Xauthority:/root/.Xauthority:ro" -e "XAUTHORITY=/root/.Xauthority" ) + echo "--->Falling back to mounting ${HOME}/.Xauthority into container" >&2 + else + echo "--->X11 socket present but no Xauthority found; GUI may fail" >&2 + fi + else + if [ -f "${HOME}/.Xauthority" ]; then + opts+=( -v "${HOME}/.Xauthority:/root/.Xauthority:ro" -e "XAUTHORITY=/root/.Xauthority" ) + echo "--->Mounting ${HOME}/.Xauthority into container for X11 auth (xauth missing)" >&2 + fi + fi + elif [ "${HEADS_X11_XAUTH:-0}" != "0" ] && [ -f "${HOME}/.Xauthority" ]; then + opts+=( -v "${HOME}/.Xauthority:/root/.Xauthority:ro" -e "XAUTHORITY=/root/.Xauthority" ) + echo "--->HEADS_X11_XAUTH=1: mounting ${HOME}/.Xauthority into container" >&2 + fi + + # If host xhost does not list LOCAL, warn the user about enabling access only when + # we did NOT supply an Xauthority cookie. We do NOT modify xhost automatically (security). + if [ "${DOCKER_XAUTH_USED:-0}" = "0" ] && command -v xhost >/dev/null 2>&1 && ! xhost | grep -q "LOCAL:"; then + echo "--->X11 auth may be strict; no automatic 'xhost' changes are performed. Provide Xauthority (install xauth) or run 'xhost +SI:localuser:root' manually if you accept the security risk." >&2 + fi + + # Output each option on its own line so callers can safely populate an array + for o in "${opts[@]}"; do + printf '%s\n' "$o" + done +} + +# Compare local image digest with remote (docker.io) digest +# Usage: compare_image_reproducibility [remote_image_ref] +# Prints comparison info and returns 0 if digests match, 1 if different +# Default remote image uses HEADS_CHECK_REPRODUCIBILITY_REMOTE or ${HEADS_MAINTAINER_DOCKER_IMAGE}:latest +# Helper: fetch remote image's config digest (which corresponds to image ID) without pulling. +# Tries 'skopeo' first, then a lightweight Docker Registry v2 manifest fetch using curl and token auth. +# Output: single line as "\t". +# Returns 0 on success, else non-zero. +get_remote_config_digest() { + local remote_image="$1" + local digest="" + local method="unknown" + + # 1) Prefer skopeo (simplest, handles auth automatically) + if command -v skopeo >/dev/null 2>&1; then + local skopeo_output + # Run skopeo without suppressing errors—this helps debug why it might fail + if skopeo_output=$(skopeo inspect "docker://${remote_image}" 2>&1); then + # skopeo succeeded; extract config digest + if command -v jq >/dev/null 2>&1; then + digest=$(printf '%s' "${skopeo_output}" | jq -r '.config.digest // empty' 2>/dev/null || true) + if [ -n "${digest}" ]; then + method="skopeo+jq" + [ -t 2 ] && printf " Using skopeo + jq to get config digest\n" >&2 + fi + else + digest=$(printf '%s' "${skopeo_output}" | tr -d '\n' | sed -nE 's/.*"config"[^}]*"digest"\s*:\s*"([^" ]+)".*/\1/p' || true) + if [ -n "${digest}" ]; then + method="skopeo+sed" + [ -t 2 ] && printf " Using skopeo + sed to get config digest (jq not available)\n" >&2 + fi + fi + else + # skopeo failed; error message is in skopeo_output, show it if interactive + [ -t 2 ] && printf " Note: skopeo inspect failed: %s\n" "${skopeo_output}" >&2 + fi + if [ -n "${digest}" ]; then + printf '%s\t%s\n' "${digest}" "${method}" + return 0 + fi + fi + + # 2) Lightweight registry API fetch (best-effort, avoids jq dependency) + # Skip this method if remote_image is a digest reference (contains @) + if [[ "${remote_image}" == *"@"* ]]; then + return 1 + fi + # Parse out registry (host), repo and tag + local host repo tag repo_with_tag first + repo_with_tag="${remote_image}" + tag="${repo_with_tag##*:}" + repo="${repo_with_tag%:*}" + + first="${repo%%/*}" + if echo "${first}" | grep -qE '\\.|:'; then + host="${first}" + repo="${repo#*/}" + else + # Default to Docker Hub registry + host="registry-1.docker.io" + fi + + # For docker hub official images, prefix 'library/' when missing namespace + if [ "${host}" = "registry-1.docker.io" ] && ! echo "${repo}" | grep -q '/'; then + repo="library/${repo}" + fi + + # Get auth token (Docker Hub auth endpoint). Ignore failures silently. + local auth_url token manifest + auth_url="https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" + if command -v jq >/dev/null 2>&1; then + token=$(curl -fsSL "${auth_url}" 2>/dev/null | jq -r '.token // empty' 2>/dev/null || true) + else + token=$(curl -fsSL "${auth_url}" 2>/dev/null | tr -d '\n' | sed -nE 's/.*"token"\s*:\s*"([^\"]+)".*/\1/p' || true) + fi + if [ -n "${token}" ]; then + if command -v jq >/dev/null 2>&1; then + manifest=$(curl -fsSL -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer ${token}" "https://${host}/v2/${repo}/manifests/${tag}" 2>/dev/null || true) + digest=$(printf '%s' "${manifest}" | jq -r '.config.digest // empty' 2>/dev/null || true) + method="registry+jq" + [ -t 2 ] && printf " Using registry API + jq to get config digest\n" >&2 + else + manifest=$(curl -fsSL -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer ${token}" "https://${host}/v2/${repo}/manifests/${tag}" 2>/dev/null | tr -d '\n' || true) + digest=$(printf '%s' "${manifest}" | sed -nE 's/.*"config"[^}]*"digest"\s*:\s*"([^" ]+)".*/\1/p' || true) + method="registry+sed" + [ -t 2 ] && printf " Using registry API + sed to get config digest (jq not available)\n" >&2 + fi + if [ -n "${digest}" ]; then + printf '%s\t%s\n' "${digest}" "${method}" + return 0 + fi + fi + + return 1 +} + +# Simple helper: get local image ID (docker .Id) +get_local_image_id() { + docker inspect --format='{{.Id}}' "$1" 2>/dev/null || return 1 +} + +# Compare local image digest with remote (docker.io) digest +# Usage: compare_image_reproducibility [remote_image_ref] +# Prefer comparing image-config digest (image ID) fetched from registry when possible. +compare_image_reproducibility() { + local local_image="$1" + local remote_image + remote_image=$(resolve_repro_remote_image "${2:-}") + + echo "" >&2 + echo "=== Reproducibility Check (image ID / config digest) ===" >&2 + # Tools summary (TTY only): show optional helpers availability so user knows which path will be tried + if [ -t 2 ]; then + printf " Tools available: skopeo=%s jq=%s\n" "$(command -v skopeo >/dev/null 2>&1 && echo yes || echo no)" "$(command -v jq >/dev/null 2>&1 && echo yes || echo no)" >&2 + fi + + local local_id + local_id=$(get_local_image_id "${local_image}") || { echo "Error: local image not found: ${local_image}" >&2; return 1; } + printf "%-48s\t%s\n" "Local image (${local_image}):" "${local_id}" >&2 + + # Try to obtain remote config digest (no pull) + local remote_config remote_method + if IFS=$'\t' read -r remote_config remote_method < <(get_remote_config_digest "${remote_image}"); then + : + fi + + if [ -n "${remote_config}" ]; then + printf "%-48s\t%s\n" "Remote image (${remote_image}):" "${remote_config}" >&2 + # Print method used for clarity + printf "%-48s\t%s\n" "Method used:" "${remote_method:-registry+sed}" >&2 + if [ "${local_id##*:}" = "${remote_config##*:}" ]; then + printf "%s\n" "${GREEN}✓ MATCH:${RESET} Image IDs match (image config digest identical)." >&2 + printf "%s\n" "${GREEN}Reproducibility: ✓ MATCH — image IDs identical${RESET}" >&2 + echo "=== End Reproducibility Check ===" >&2 + echo "" >&2 + return 0 + else + printf "%s\n" "${RED}✗ MISMATCH:${RESET} Local image ID differs from remote image config digest." >&2 + echo " Local image ID: ${local_id}" >&2 + echo " Remote config ID: ${remote_config}" >&2 + echo " Method used: ${remote_method:-registry+sed}" >&2 + echo "=== End Reproducibility Check ===" >&2 + echo "" >&2 + return 1 + fi + fi + + # If we couldn't get a remote config digest, fall back to optional pull + image-ID compare + print_once "pull_notice" " Could not fetch remote image config digest without pulling; falling back to 'docker pull' to compare image IDs (progress will be shown)." + if [ "${HEADS_CHECK_REPRODUCIBILITY_NO_PULL:-0}" = "1" ]; then + echo "Auto-pull suppressed by HEADS_CHECK_REPRODUCIBILITY_NO_PULL=1; aborting reproducibility check." >&2 + return 1 + fi + if ! prompt_for_pull "${remote_image}"; then + return 1 + fi + if ! docker pull "${remote_image}" >/dev/null 2>&1; then + echo "Error: failed to pull remote image ${remote_image}" >&2 + return 1 + fi + # Record method used: pulled image (not via registry fetch) + local pulled_method="pulled" + local remote_id + remote_id=$(get_local_image_id "${remote_image}" 2>/dev/null || true) + printf "%-48s\t%s\n" "Remote image (pulled ${remote_image}):" "${remote_id:-}" >&2 + printf "%-48s\t%s\n" "Method used:" "${pulled_method}" >&2 + if [ "${local_id}" = "${remote_id}" ]; then + printf "%s\n" "${GREEN}✓ MATCH:${RESET} Image IDs identical after pull." >&2 + printf "%s\n" "${GREEN}Reproducibility: ✓ MATCH — image IDs identical${RESET}" >&2 + echo "=== End Reproducibility Check ===" >&2 + echo "" >&2 + return 0 + else + printf "%s\n" "${RED}✗ MISMATCH:${RESET} Image IDs differ after pull." >&2 + echo " Local: ${local_id}" >&2 + echo " Remote: ${remote_id}" >&2 + echo "=== End Reproducibility Check ===" >&2 + echo "" >&2 + return 1 + fi +} + +# Common run helper +run_docker() { + local image="$1"; shift + local opts host_workdir container_workdir DOCKER_OPTS_ARRAY + # Read docker options (one-per-line) into an array, preserving spaces within options + mapfile -t DOCKER_OPTS_ARRAY < <(build_docker_opts) + # Also create a single-string representation for legacy substring checks + opts=$(printf '%s\n' "${DOCKER_OPTS_ARRAY[@]}") + host_workdir="$(pwd)" + container_workdir="${host_workdir}" + + local -a parts=() + case "${opts}" in *"/dev/kvm"*) parts+=(KVM=on) ;; *) parts+=(KVM=off) ;; esac + case "${opts}" in *"/dev/bus/usb"*) parts+=(USB=on) ;; *) parts+=(USB=off) ;; esac + case "${opts}" in *"/tmp/.X11-unix"*) parts+=(X11=on) ;; *) parts+=(X11=off) ;; esac + + echo "---> Running container with: ${parts[*]} ; mount ${host_workdir} -> ${container_workdir}" >&2 + + # If no command was provided by the caller, start an interactive shell inside the container. + # We prefer bash when available, and fall back to sh; the sh -c wrapper ensures the + # container will get a usable shell on minimal images. + if [ $# -eq 0 ]; then + echo "---> No command provided: launching interactive shell inside container (bash if available, otherwise sh)" >&2 + set -- sh -c 'exec bash || exec sh' + fi + + echo "---> Full docker command: docker run ${DOCKER_OPTS_ARRAY[*]} -v ${host_workdir}:${container_workdir} -w ${container_workdir} ${image} -- $*" >&2 + + docker run "${DOCKER_OPTS_ARRAY[@]}" -v "${host_workdir}:${container_workdir}" -w "${container_workdir}" "${image}" -- "$@" + local status=$? + if [ "${DOCKER_XAUTH_TEMP:-0}" = "1" ] && [ -n "${DOCKER_XAUTH_FILE}" ]; then + rm -f "${DOCKER_XAUTH_FILE}" || true + fi + return $status +} + +# ================================================================ +# Script initialization and setup +# ================================================================ + +# Detect if script is being sourced or executed directly +# When sourced: BASH_SOURCE[0] != $0 +# When executed: BASH_SOURCE[0] == $0 +if [ "${BASH_SOURCE[0]}" = "${0}" ]; then + # Script is being executed directly: show the full environment usage. + usage + exit 0 +fi + +if [ "${__HEADS_RESTORE_SHELL_OPTS}" = "1" ]; then + eval "${__HEADS_SHELL_OPTS}" +fi diff --git a/docker/fetch_nix_installer.sh b/docker/fetch_nix_installer.sh new file mode 100755 index 000000000..9e2e20215 --- /dev/null +++ b/docker/fetch_nix_installer.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat <<'USAGE' >&2 +Usage: $0 [--version VERSION] [--url URL] + +Download the Nix installer (no execution) and print its sha256. If a release +version is specified (e.g. 'nix-2.33.2') the script will also try to fetch + + https://releases.nixos.org/nix/${VERSION}/install.sha256 + +so you can compare the published checksum against the downloaded installer. + +Examples: + ./docker/fetch_nix_installer.sh --version nix-2.33.2 + ./docker/fetch_nix_installer.sh --url https://nixos.org/nix/install +USAGE +} + +if [ $# -eq 0 ]; then + usage + exit 2 +fi + +installer_url="" +sha_url="" +while [ $# -gt 0 ]; do + case "$1" in + --version) + if [ $# -lt 2 ]; then echo "Missing argument for --version" >&2; usage; exit 2; fi + installer_url="https://releases.nixos.org/nix/$2/install" + sha_url="$installer_url.sha256" + shift 2 ;; + --url) + if [ $# -lt 2 ]; then echo "Missing argument for --url" >&2; usage; exit 2; fi + installer_url="$2" + sha_url="" + shift 2 ;; + -h|--help) + usage; exit 0 ;; + *) + echo "Unknown arg: $1" >&2; usage; exit 2 ;; + esac +done + +if [ -z "$installer_url" ]; then + echo "No installer URL determined; provide --version or --url" >&2 + usage + exit 2 +fi + +# choose downloader +if command -v curl >/dev/null 2>&1; then + downloader=curl +elif command -v wget >/dev/null 2>&1; then + downloader=wget +else + echo "Error: neither curl nor wget available to fetch installer" >&2 + exit 1 +fi + +tmpf=$(mktemp) || { echo "Failed to create temporary file" >&2; exit 1; } +trap 'rm -f "$tmpf"' EXIT + +if [ "$downloader" = "curl" ]; then + curl -fsSL "$installer_url" -o "$tmpf" || { echo "Failed to download $installer_url" >&2; exit 1; } +else + wget -qO "$tmpf" "$installer_url" || { echo "Failed to download $installer_url" >&2; exit 1; } +fi + +# compute sha +inst_sha="" +if command -v sha256sum >/dev/null 2>&1; then + inst_sha=$(sha256sum "$tmpf" | awk '{print $1}') || inst_sha="" +elif command -v shasum >/dev/null 2>&1; then + inst_sha=$(shasum -a 256 "$tmpf" | awk '{print $1}') || inst_sha="" +else + inst_sha="" +fi + +echo "Downloaded installer: $installer_url" +if [ -n "$inst_sha" ]; then + echo "Installer sha256: $inst_sha" +else + echo "sha256 unavailable (no sha256sum/shasum)" +fi + +if [ -n "$sha_url" ]; then + pub_sha="" + if [ "$downloader" = "curl" ]; then + pub_sha=$(curl -fsSL "$sha_url" 2>/dev/null | tr -d '[:space:]' || true) + else + pub_sha=$(wget -qO- "$sha_url" 2>/dev/null | tr -d '[:space:]' || true) + fi + if [ -n "$pub_sha" ]; then + echo "Published sha at: $sha_url" + echo "Published sha256: $pub_sha" + if [ -n "$inst_sha" ] && [ "$inst_sha" = "$pub_sha" ]; then + echo "OK: published sha matches downloaded installer" + else + echo "WARNING: published sha does NOT match downloaded installer" + fi + else + echo "Note: could not fetch published sha from: $sha_url" + fi +fi + +exit 0 diff --git a/docker/get_digest.sh b/docker/get_digest.sh new file mode 100755 index 000000000..87ddb35f2 --- /dev/null +++ b/docker/get_digest.sh @@ -0,0 +1,285 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat <<'USAGE' >&2 +Usage: $0 [--yes|-y] IMAGE[:TAG|@DIGEST] + +Helper to print the full 'repo@digest' and the raw digest for a docker image. +Behavior: + - The script treats the provided image reference literally. Provide exact `repo/name:tag` or `repo@digest` (e.g. `tlaurion/heads-dev-env:v0.2.6`). + - If the image exists locally, the script prints the first RepoDigest (repo@digest) and the raw digest. + - If the image is not present locally, the script will offer to pull the exact provided reference to obtain a local RepoDigest (interactive or `-y`). + - The script prefers to operate on local image state (e.g., Docker local RepoDigests). If a local digest is not available it may query the Docker Hub v2 HTTP API (docker.io) via `curl` to obtain an authoritative manifest digest for docker.io images; this requires network access and appropriate registry connectivity. For other registries or Docker versions you may still need to use `docker manifest inspect` or `skopeo inspect` manually if `RepoDigests` is not populated. + +Options: + -y, --yes Automatically pull the image if it is not present locally (non-interactive) + -h, --help Show this help message + +Examples: + ./docker/get_digest.sh tlaurion/heads-dev-env:v0.2.6 + ./docker/get_digest.sh tlaurion/heads-dev-env:latest + ./docker/get_digest.sh -y tlaurion/heads-dev-env:v0.2.6 + # Note: provide the exact repo:name:tag you intend; the script treats the reference literally. +USAGE +} + +if [ $# -lt 1 ]; then + usage + exit 2 +fi + +auto_yes=0 +if [ "${1:-}" = "-y" ] || [ "${1:-}" = "--yes" ]; then + auto_yes=1 + shift +fi +if [ "${1:-}" = "-h" ] || [ "${1:-}" = "--help" ]; then + usage + exit 0 +fi + +if [ $# -ne 1 ]; then + usage + exit 2 +fi + +image="$1" + +# Treat the provided image reference literally and do not try to append ':latest'. +# The caller should provide the exact reference they intend (e.g. 'tlaurion/heads-dev-env:v0.2.6'), +# and the script will inspect that exact reference and prompt to pull it if missing. +image_provided="${image}" +image="${image_provided}" + +# Reject refs without a tag (unless a digest was provided). +if [[ "${image}" != *@* ]]; then + _last_component="${image##*/}" + if [[ "${_last_component}" != *:* ]]; then + echo "Error: image reference '${image}' has no tag; please specify :tag or @digest." >&2 + exit 2 + fi +fi + +# Ensure docker is available +if ! command -v docker >/dev/null 2>&1; then + echo "Error: docker not found in PATH" >&2 + exit 1 +fi + +# Source shared helpers so we can print digest info consistently +# shellcheck source=docker/common.sh +. "$(dirname "$0")/common.sh" + +# If the image already includes a digest (repo@sha256:...), return it +if [[ "${image}" == *@* ]]; then + echo "${image}" + echo "${image#*@}" + exit 0 +fi + +# Use the provided image reference exactly; do not attempt alternate forms. +local_repo_digest="" +manifest_digest="" + +# Check local RepoDigest for the exact provided image reference +local_repo_digest=$(docker inspect --format='{{index .RepoDigests 0}}' "${image}" 2>/dev/null || true) +if [ -n "${local_repo_digest}" ]; then + echo "${local_repo_digest}" + echo "${local_repo_digest#*@}" + exit 0 +fi + +# We prefer to operate on local image state (RepoDigests). If there's no local RepoDigest we may query the +# Docker Hub v2 API (docker.io) to obtain a manifest digest for docker.io images as a best-effort. This requires +# network access and a working curl; for non-docker.io registries or if the Hub API cannot be used, the user may +# need to pull the image or use tools like `docker manifest inspect`/`skopeo inspect` manually. +manifest_digest="" + +# If we couldn't get a manifest digest locally, try the Docker Hub registry API as a fallback +if [ -z "${manifest_digest}" ]; then + # Only attempt the Docker Hub v2 API for docker.io-style images + # Parse repo and tag + repo="${image%:*}" + tag="${image##*:}" + + # Normalize repo for Docker Hub API: strip docker.io/ or registry-1.docker.io/ prefixes and + # ensure 'library/' prefix for official images (e.g., 'ubuntu' -> 'library/ubuntu'). + repo_for_api="${repo#docker.io/}" + repo_for_api="${repo_for_api#registry-1.docker.io/}" + if ! printf '%s' "${repo_for_api}" | grep -q '/'; then + repo_for_api="library/${repo_for_api}" + fi + + # If repo contains a registry hostname (e.g., myregistry.example.com/...), skip hub API + if ! printf '%s' "${repo}" | grep -q '/'; then + # no slash in repo -- unlikely, but skip + : + fi + + # Determine if it's a docker.io (default) reference, explicitly docker.io, or a non-Hub registry. + # We only attempt the Docker Hub API when: + # - the first path component is explicitly 'docker.io' or 'registry-1.docker.io', or + # - there is no explicit registry-like first component (no '.' or ':' and not 'localhost'). + # This avoids misclassifying host:port registries (e.g. localhost:5000/repo:tag) as docker.io. + first_component="${repo%%/*}" + is_docker_hub_ref=0 + if [ "${first_component}" = "docker.io" ] || [ "${first_component}" = "registry-1.docker.io" ]; then + # Explicit Docker Hub hostname + is_docker_hub_ref=1 + elif printf '%s' "${first_component}" | grep -q '\.'; then + # Has a dot: looks like a custom registry hostname (e.g., myregistry.example.com) + is_docker_hub_ref=0 + elif printf '%s' "${first_component}" | grep -q ':'; then + # Has a colon: likely host:port (e.g., localhost:5000), treat as non-Hub + is_docker_hub_ref=0 + elif [ "${first_component}" = "localhost" ]; then + # localhost without an explicit port: also treat as non-Hub + is_docker_hub_ref=0 + else + # No dot, no colon, not localhost: treat as implicit Docker Hub (e.g., 'library/ubuntu', 'user/repo') + is_docker_hub_ref=1 + fi + if [ "${is_docker_hub_ref}" -eq 1 ]; then + registry_api="https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo_for_api}:pull" + + # Prefer curl but fall back to wget; if neither is present skip the Hub API gracefully. + downloader="" + if command -v curl >/dev/null 2>&1; then + downloader="curl" + elif command -v wget >/dev/null 2>&1; then + downloader="wget" + else + downloader="" + fi + + if [ -z "${downloader}" ]; then + echo "Note: neither 'curl' nor 'wget' is available; skipping Docker Hub API fallback." >&2 + elif ! command -v jq >/dev/null 2>&1; then + echo "Note: 'jq' is not available; skipping Docker Hub API fallback (jq required for secure JSON parsing)." >&2 + echo "Install jq to enable registry API queries without pulling the image." >&2 + else + if [ "${downloader}" = "curl" ]; then + # Use jq for robust JSON parsing + token=$(curl -fsSL "${registry_api}" | jq -r '.token // empty' 2>/dev/null || true) + if [ -n "${token}" ]; then + header=$(curl -fsSI -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer ${token}" "https://registry-1.docker.io/v2/${repo_for_api}/manifests/${tag}" 2>/dev/null || true) + manifest_digest=$(printf '%s\n' "$header" | sed -n 's/Docker-Content-Digest:[[:space:]]*//Ip' | tr -d '\r' | head -n1 || true) + fi + else + # wget path: fetch token body, then request manifest and parse headers from stderr + # Use jq for robust JSON parsing + token=$(wget -qO- "${registry_api}" | jq -r '.token // empty' 2>/dev/null || true) + if [ -n "${token}" ]; then + header=$(wget --server-response --header="Accept: application/vnd.docker.distribution.manifest.v2+json" --header="Authorization: Bearer ${token}" "https://registry-1.docker.io/v2/${repo_for_api}/manifests/${tag}" -O - 2>&1 || true) + manifest_digest=$(printf '%s\n' "$header" | sed -n 's/Docker-Content-Digest:[[:space:]]*//Ip' | tr -d '\r' | head -n1 || true) + fi + fi + fi + fi + + if [ -n "${manifest_digest}" ]; then + print_digest_info "${image%@*}@${manifest_digest}" "${manifest_digest}" "registry API" "" + echo "${image%@*}@${manifest_digest}" + echo "${manifest_digest}" + + # Offer to pull the exact image so the local Docker daemon has a repo@digest entry. + if [ "${auto_yes}" = 1 ]; then + echo "Auto-pull enabled: pulling ${image} (progress will be shown)..." >&2 + if ! docker pull "${image}" 2>&1 | sed -u 's/^/ /'; then + exit 1 + fi + local_repo_digest=$(docker inspect --format='{{index .RepoDigests 0}}' "${image}" 2>/dev/null || true) + if [ -n "${local_repo_digest}" ]; then + echo "${local_repo_digest}" + echo "${local_repo_digest#*@}" + exit 0 + fi + # else fall through and print the manifest digest as best-effort + exit 0 + fi + + if [ -t 0 ]; then + printf "Image '%s' is not present locally. Pull it now to obtain a local repo@digest? [y/N] " "${image}" >&2 + read -r ans + case "${ans:-N}" in + [Yy]* ) + if ! docker pull "${image}"; then + echo "Failed to pull ${image}; aborting." >&2 + exit 1 + fi + local_repo_digest=$(docker inspect --format='{{index .RepoDigests 0}}' "${image}" 2>/dev/null || true) + if [ -n "${local_repo_digest}" ]; then + echo "${local_repo_digest}" + echo "${local_repo_digest#*@}" + exit 0 + fi + # If still no RepoDigests, print manifest digest + echo "${image%@*}@${manifest_digest}" + echo "${manifest_digest}" + exit 0 + ;; + * ) + echo "Aborting without pulling; remote digest was: ${manifest_digest}" >&2 + echo "${image%@*}@${manifest_digest}" + echo "${manifest_digest}" + exit 0 + ;; + esac + else + echo "Non-interactive shell: image not present locally and --yes not supplied; remote digest: ${manifest_digest}" >&2 + echo "${image%@*}@${manifest_digest}" + echo "${manifest_digest}" + exit 0 + fi + fi +fi + +# If we're here, we could not determine a digest from local RepoDigests or the registry. +# Offer to pull the image interactively (or non-interactively with --yes) to obtain a local RepoDigest. +if [ -t 0 ] || [ "${auto_yes}" = 1 ]; then + if [ "${auto_yes}" = 1 ]; then + pull_yes=1 + else + echo "Note: the script treats the provided image reference literally. If you intended the tag 'v0.2.6' of repo 'tlaurion/heads', pass 'tlaurion/heads:v0.2.6'." >&2 + printf "Image '%s' is not present locally. Pull it now to try to obtain a local repo@digest? [y/N] " "${image}" >&2 + read -r _ans + case "${_ans:-N}" in + [Yy]* ) pull_yes=1 ;; + * ) pull_yes=0 ;; + esac + fi + + if [ "${pull_yes:-0}" = 1 ]; then + echo "Pulling ${image}..." >&2 + if ! docker pull "${image}"; then + echo "Failed to pull ${image}; check network/credentials, ensure the reference is correct (e.g. 'tlaurion/heads:v0.2.6' if v0.2.6 is a tag), and run 'docker login' if needed." >&2 + exit 1 + fi + # After pull, prefer repo@digest from local RepoDigests if available + local_repo_digest=$(docker inspect --format='{{index .RepoDigests 0}}' "${image}" 2>/dev/null || true) + if [ -n "${local_repo_digest}" ]; then + print_digest_info "${local_repo_digest}" "${local_repo_digest#*@}" "local" "DOCKER_LATEST_DIGEST" + echo "${local_repo_digest}" + echo "${local_repo_digest#*@}" + exit 0 + fi + + # After pulling, check local RepoDigest again. If still missing, fail with a clear message. + local_repo_digest=$(docker inspect --format='{{index .RepoDigests 0}}' "${image}" 2>/dev/null || true) + if [ -n "${local_repo_digest}" ]; then + print_digest_info "${local_repo_digest}" "${local_repo_digest#*@}" "local" "DOCKER_LATEST_DIGEST" + echo "${local_repo_digest}" + echo "${local_repo_digest#*@}" + exit 0 + fi + + echo "Pull completed but still did not produce a repo@digest for ${image}." >&2 + echo "You may need to inspect the image manually with 'docker inspect' or consult the registry for this specific ref." >&2 + exit 1 + fi +fi + +# Nothing else we can do +echo "Failed to obtain digest for ${image}. Try pulling the image or use 'docker inspect'/'docker manifest inspect' manually." >&2 +exit 1 diff --git a/docker/pin-and-run.sh b/docker/pin-and-run.sh new file mode 100755 index 000000000..a4c34b9ee --- /dev/null +++ b/docker/pin-and-run.sh @@ -0,0 +1,132 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat <<'USAGE' >&2 +Usage: $0 [-y|--yes] [-w|--wrapper WRAPPER] IMAGE [-- [WRAPPER [WRAPPER_ARGS...]]] + +Helper: obtain an image digest and run a docker wrapper pinned to that digest. +- IMAGE: an exact docker image ref (e.g. tlaurion/heads-dev-env:v0.2.6) +- If the image is not present locally, the helper will probe the registry and + offer to pull it (use -y/--yes to auto-pull). +- WRAPPER: the docker wrapper to execute (e.g. ./docker_latest.sh or ./docker_repro.sh). + If omitted the helper will use ./docker_latest.sh by default when the first + argument after '--' does not look like a wrapper or when none is supplied. + +Options: + -y, --yes Automatically pull the image if it is not present locally (non-interactive) + -w, --wrapper Specify the wrapper to run (explicitly); useful when default detection is ambiguous + -h, --help Show this help message + +Examples: + # Interactive: obtain digest and run the 'latest' wrapper pinned to that digest (explicit wrapper recommended) + ./docker/pin-and-run.sh tlaurion/heads-dev-env:v0.2.6 -- ./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 + + # Auto-pull and run (auto-pull the ref to obtain a local digest then run wrapper) + ./docker/pin-and-run.sh -y tlaurion/heads-dev-env:v0.2.6 -- ./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 + + # Shortcut: omit the wrapper and just provide the command — the helper will use the default './docker_latest.sh' + ./docker/pin-and-run.sh tlaurion/heads-dev-env:v0.2.6 -- make BOARD=qemu-coreboot-fbwhiptail-tpm2 + + # Use a different wrapper explicitly (e.g. repro): + ./docker/pin-and-run.sh -w ./docker_repro.sh tlaurion/heads-dev-env:v0.2.6 -- make BOARD=qemu-coreboot-fbwhiptail-tpm2 +USAGE +} + +auto_yes=0 +wrapper_override=0 +wrapper="" + +# Parse options (allow -y/--yes, -w/--wrapper WRAPPER, -h/--help) +while [ $# -gt 0 ]; do + case "$1" in + -y|--yes) + auto_yes=1; shift ;; + -w|--wrapper) + if [ $# -lt 2 ]; then + echo "Missing argument for --wrapper" >&2; usage; exit 2 + fi + wrapper="$2"; wrapper_override=1; shift 2 ;; + -h|--help) + usage; exit 0 ;; + --) + shift; break ;; + *) + break ;; + esac +done + +if [ $# -lt 1 ]; then + usage + exit 2 +fi + +image="$1"; shift + +# Default wrapper if none supplied via -w +default_wrapper="$(dirname "$0")/docker_latest.sh" +wrapper_args=() + +if [ $wrapper_override -eq 0 ]; then + # No explicit wrapper - try heuristic or use default + wrapper="$default_wrapper" + if [ $# -gt 0 ]; then + # Allow the caller to separate with '--' or just provide the wrapper/args directly + if [ "$1" = "--" ]; then + shift + fi + if [ $# -gt 0 ]; then + # Heuristic: if the first argument looks like a wrapper (existing file/executable, ends with .sh, or starts with 'docker_'), + # treat it as the wrapper. Otherwise use the default wrapper and treat all args as wrapper_args. + first_arg="$1" + if [ -f "$first_arg" ] || [ -x "$first_arg" ] || [[ "$first_arg" == *.sh ]] || [[ "$first_arg" == docker_* ]]; then + wrapper="$1"; shift || true + wrapper_args=("$@") + else + wrapper_args=("$@") + fi + fi + fi +else + # Explicit wrapper provided with -w: consume optional leading '--' and treat remaining args as wrapper_args + if [ $# -gt 0 ] && [ "$1" = "--" ]; then + shift + fi + wrapper_args=("$@") +fi + +# Source common helpers so the output is consistent +# shellcheck source=docker/common.sh +. "$(dirname "$0")/common.sh" + +# Obtain the raw digest (second line of output). Use script-relative path so this works regardless of $PWD +if [ "$auto_yes" = 1 ]; then + digest="$("$(dirname "$0")/get_digest.sh" -y "$image" | tail -n1)" +else + digest="$("$(dirname "$0")/get_digest.sh" "$image" | tail -n1)" +fi + +if [ -z "${digest:-}" ]; then + echo "Failed to obtain a digest for ${image}; aborting." >&2 + exit 1 +fi + +# Decide which env var to set based on wrapper name +case "$(basename "$wrapper")" in + *repro*) envvar=DOCKER_REPRO_DIGEST ;; + *) envvar=DOCKER_LATEST_DIGEST ;; +esac + +print_digest_info "${image%@*}@${digest}" "${digest}" "user" "${envvar}" +echo "Running ${wrapper} pinned to ${digest} (exporting ${envvar})" >&2 + +# Validate that the wrapper exists and is executable before exec'ing it +if [ -z "${wrapper:-}" ] || [ ! -x "$wrapper" ]; then + echo "Error: wrapper '${wrapper:-}' not found or not executable." >&2 + usage + exit 1 +fi + +# Exec the wrapper with the pinned digest in the environment +# Note: use env to avoid exporting the var in caller environment +env "${envvar}=${digest}" "$wrapper" "${wrapper_args[@]:-}" \ No newline at end of file diff --git a/docker_latest.sh b/docker_latest.sh index f073e5a79..3b13da595 100755 --- a/docker_latest.sh +++ b/docker_latest.sh @@ -1,58 +1,55 @@ #!/bin/bash -# Inform the user that the latest published Docker image is being used -echo "Using the latest Docker image: tlaurion/heads-dev-env:latest" -DOCKER_IMAGE="tlaurion/heads-dev-env:latest" +# Source shared docker helper functions +# shellcheck source=docker/common.sh +source "$(dirname "$0")/docker/common.sh" + +# Determine an initial Docker image (allow override via DOCKER_LATEST_IMAGE) +DOCKER_IMAGE="${DOCKER_LATEST_IMAGE:-tlaurion/heads-dev-env:latest}" -# Function to display usage information usage() { - echo "Usage: $0 [OPTIONS] -- [COMMAND]" - echo "Options:" - echo " CPUS=N Set the number of CPUs" - echo " V=1 Enable verbose mode" - echo "Command:" - echo " The command to run inside the Docker container, e.g., make BOARD=BOARD_NAME" + cat <<'USAGE' +Usage: ./docker_latest.sh [COMMAND...] + +Run the maintainer "latest" image (or a pinned digest if configured). + +Environment: + DOCKER_LATEST_IMAGE=... Override the image/tag to run + DOCKER_LATEST_DIGEST=... Pin to a specific digest (sha256:...) + HEADS_ALLOW_UNPINNED_LATEST=1 Allow unpinned :latest without prompting + HEADS_DISABLE_USB=1 Disable USB passthrough + HEADS_X11_XAUTH=1 Force mounting ~/.Xauthority + +Examples: + ./docker_latest.sh + DOCKER_LATEST_DIGEST=sha256:... ./docker_latest.sh +USAGE } -# Function to kill GPG toolstack related processes using USB devices -kill_usb_processes() { - # check if scdaemon or pcscd processes are using USB devices - if [ -d /dev/bus/usb ]; then - if sudo lsof /dev/bus/usb/00*/0* 2>/dev/null | awk 'NR>1 {print $2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' >/dev/null; then - echo "Killing GPG toolstack related processes using USB devices..." - sudo lsof /dev/bus/usb/00*/0* 2>/dev/null | awk 'NR>1 {print $2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' | awk '{print $1}' | xargs -r sudo kill -9 - fi - fi -} +if [ "${1:-}" = "-h" ] || [ "${1:-}" = "--help" ]; then + usage + exit 0 +fi -# Handle Ctrl-C (SIGINT) to exit gracefully -trap "echo 'Script interrupted. Exiting...'; exit 1" SIGINT - -# Check if --help or -h is provided -for arg in "$@"; do - if [[ "$arg" == "--help" || "$arg" == "-h" ]]; then - usage - exit 0 - fi -done - -# Kill processes using USB devices -kill_usb_processes - -# Inform the user about entering the Docker container -echo "----" -echo "Usage reminder: The minimal command is 'make BOARD=XYZ', where additional options, including 'V=1' or 'CPUS=N' are optional." -echo "For more advanced QEMU testing options, refer to targets/qemu.md and boards/qemu-*/*.config." -echo -echo "Type exit within docker image to get back to host if launched interactively!" -echo "----" -echo - -# Execute the docker run command with the provided parameters -if [ -d "/dev/bus/usb" ]; then - echo "--->Launching container with access to host's USB buses (some USB devices were connected to host)..." - docker run --device=/dev/bus/usb:/dev/bus/usb -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) $DOCKER_IMAGE -- "$@" -else - echo "--->Launching container without access to host's USB buses (no USB devices was connected to host)..." - docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) $DOCKER_IMAGE -- "$@" +trap 'echo "Script interrupted. Exiting..."; exit 1' SIGINT + +# Resolve pinned digest (env var preferred, repository file fallback), and prompt if using unpinned :latest +DOCKER_IMAGE="$(resolve_docker_image "$DOCKER_IMAGE" "DOCKER_LATEST_DIGEST" "DOCKER_LATEST_DIGEST" "1")" +# If resolve_docker_image returned empty for any reason, abort +if [ -z "${DOCKER_IMAGE}" ]; then + echo "Error: failed to resolve Docker image; aborting." >&2 + exit 1 +fi +echo "Using latest image: $DOCKER_IMAGE" >&2 +echo "" >&2 + +# Only perform host-side side-effects when executed directly (not when sourced) +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + require_docker || exit $? + # Clean up host processes holding USB devices first (if applicable) + kill_usb_processes + + # Execute the docker run command with the provided parameters + # Delegate to shared run_docker so all docker_* scripts share identical device/X11/KVM handling + run_docker "$DOCKER_IMAGE" "$@" fi diff --git a/docker_local_dev.sh b/docker_local_dev.sh index 43b8022bb..2afc60e43 100755 --- a/docker_local_dev.sh +++ b/docker_local_dev.sh @@ -1,93 +1,77 @@ #!/bin/bash +# Source shared docker helper functions +# shellcheck source=docker/common.sh +source "$(dirname "$0")/docker/common.sh" + #locally build docker name is linuxboot/heads:dev-env DOCKER_IMAGE="linuxboot/heads:dev-env" -# Check if Nix is installed -if ! command -v nix &>/dev/null; then - echo "Nix is not installed or not in the PATH. Please install Nix before running this script." - echo "Refer to the README.md at the root of the repository for installation instructions." - exit 1 -fi +usage() { + cat <<'USAGE' +Usage: ./docker_local_dev.sh [COMMAND...] -# Check if Docker is installed -if ! command -v docker &>/dev/null; then - echo "Docker is not installed or not in the PATH. Please install Docker before running this script." - echo "Refer to the README.md at the root of the repository for installation instructions." - exit 1 -fi +Run the local dev image (linuxboot/heads:dev-env). If flake.nix/flake.lock are dirty, +rebuilds the image first. -# Inform the user about the Docker image being used -echo "!!! This ./docker_local_dev.sh script is for developers usage only. !!!" -echo "" -echo "Using the last locally built Docker image when flake.nix/flake.lock was modified and repo was dirty: linuxboot/heads:dev-env" -echo "!!! Warning: Using anything other than the published Docker image might lead to non-reproducible builds. !!!" -echo "" -echo "For using the latest published Docker image, refer to ./docker_latest.sh." -echo "For producing reproducible builds as CircleCI, refer to ./docker_repro.sh." -echo "" +Environment: + HEADS_SKIP_DOCKER_REBUILD=1 Skip rebuild even if flake files changed + HEADS_CHECK_REPRODUCIBILITY=1 Compare local image ID to maintainer image + HEADS_CHECK_REPRODUCIBILITY_REMOTE=... Override remote image for the check + HEADS_DISABLE_USB=1 Disable USB passthrough + HEADS_X11_XAUTH=1 Force mounting ~/.Xauthority -# Function to display usage information -usage() { - echo "Usage: $0 [OPTIONS] -- [COMMAND]" - echo "Options:" - echo " CPUS=N Set the number of CPUs" - echo " V=1 Enable verbose mode" - echo "Command:" - echo " The command to run inside the Docker container, e.g., make BOARD=BOARD_NAME" -} +Nix (only when rebuild is required): + HEADS_AUTO_INSTALL_NIX=1 Auto-install Nix (requires HEADS_NIX_INSTALLER_SHA256) + HEADS_NIX_INSTALLER_SHA256=... Expected sha256 for the installer + HEADS_NIX_INSTALLER_VERSION=... Use a pinned Nix installer version + HEADS_NIX_INSTALLER_URL=... Override installer URL + HEADS_AUTO_ENABLE_FLAKES=1 Auto-enable flakes in nix.conf + HEADS_SKIP_DISK_CHECK=1 Skip disk preflight check + HEADS_MIN_DISK_GB=... Override disk free threshold (GB) -# Function to kill GPG toolstack related processes using USB devices -kill_usb_processes() { - # check if scdaemon or pcscd processes are using USB devices - if [ -d /dev/bus/usb ]; then - if sudo lsof /dev/bus/usb/00*/0* 2>/dev/null | awk 'NR>1 {print $2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' >/dev/null; then - echo "Killing GPG toolstack related processes using USB devices..." - sudo lsof /dev/bus/usb/00*/0* 2>/dev/null | awk 'NR>1 {print $2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' | awk '{print $1}' | xargs -r sudo kill -9 - fi - fi +Examples: + ./docker_local_dev.sh + HEADS_CHECK_REPRODUCIBILITY=1 ./docker_local_dev.sh +USAGE } -# Handle Ctrl-C (SIGINT) to exit gracefully -trap "echo 'Script interrupted. Exiting...'; exit 1" SIGINT +if [ "${1:-}" = "-h" ] || [ "${1:-}" = "--help" ]; then + usage + exit 0 +fi -# Check if --help or -h is provided -for arg in "$@"; do - if [[ "$arg" == "--help" || "$arg" == "-h" ]]; then - usage - exit 0 - fi -done +trap 'echo "Script interrupted. Exiting..."; exit 1' SIGINT -# Check if the git repository is dirty and if flake.nix or flake.lock are part of the uncommitted changes -if [ -n "$(git status --porcelain | grep -E 'flake\.nix|flake\.lock')" ]; then - echo "**Warning: Uncommitted changes detected in flake.nix or flake.lock. The Docker image will be rebuilt!**" - echo "If this was not intended, please CTRL-C now, commit your changes and rerun the script." - echo "Building the Docker image from flake.nix..." - nix --print-build-logs --verbose develop --ignore-environment --command true - nix --print-build-logs --verbose build .#dockerImage && docker load &2 + +# Optional: verify reproducibility against docker.io latest +# Requires HEADS_CHECK_REPRODUCIBILITY=1 and either skopeo or curl installed +if [ "${HEADS_CHECK_REPRODUCIBILITY:-0}" = "1" ]; then + compare_image_reproducibility "$DOCKER_IMAGE" || { + echo "Note: Reproducibility check failed (expected if Nix versions or flake.lock differs from maintainer build)" >&2 + } +fi -# Inform the user about entering the Docker container -echo "----" -echo "Usage reminder: The minimal command is 'make BOARD=XYZ', where additional options, including 'V=1' or 'CPUS=N' are optional." -echo "For more advanced QEMU testing options, refer to targets/qemu.md and boards/qemu-*/*.config." -echo -echo "Type exit within docker image to get back to host if launched interactively!" -echo "----" -echo +# Only perform host-side side-effects when executed directly (not when sourced) +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + # If USB passthrough is possible, clean up host processes that may hold tokens (interactive abort allowed). + kill_usb_processes -# Execute the docker run command with the provided parameters -if [ -d "/dev/bus/usb" ]; then - echo "--->Launching container with access to host's USB buses (some USB devices were connected to host)..." - docker run --device=/dev/bus/usb:/dev/bus/usb -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) $DOCKER_IMAGE -- "$@" -else - echo "--->Launching container without access to host's USB buses (no USB devices was connected to host)..." - docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) $DOCKER_IMAGE -- "$@" + # Execute the docker run command with the provided parameters + # Delegate to shared run_docker so all docker_* scripts share identical device/X11/KVM handling + run_docker "$DOCKER_IMAGE" "$@" fi diff --git a/docker_repro.sh b/docker_repro.sh index 0dbeb2f6b..dccf4154a 100755 --- a/docker_repro.sh +++ b/docker_repro.sh @@ -1,66 +1,78 @@ #!/bin/bash -# Extract the Docker image version from the CircleCI config file -DOCKER_IMAGE=$(grep -oP '^\s*-?\s*image:\s*\K(tlaurion/heads-dev-env:[^\s]+)' .circleci/config.yml | head -n 1) +# Source shared docker helper functions (use the docker/ path where common.sh lives) +# shellcheck source=docker/common.sh +source "$(dirname "$0")/docker/common.sh" -# Check if the Docker image was found -if [ -z "$DOCKER_IMAGE" ]; then - echo "Error: Docker image not found in .circleci/config.yml" - exit 1 -fi +usage() { + cat <<'USAGE' +Usage: ./docker_repro.sh [COMMAND...] -# Inform the user about the versioned CircleCI Docker image being used -echo "Using CircleCI Docker image: $DOCKER_IMAGE" +Run the reproducible (pinned digest) image. -# Function to display usage information -usage() { - echo "Usage: $0 [OPTIONS] -- [COMMAND]" - echo "Options:" - echo " CPUS=N Set the number of CPUs" - echo " V=1 Enable verbose mode" - echo "Command:" - echo " The command to run inside the Docker container, e.g., make BOARD=BOARD_NAME" -} +Environment: + HEADS_MAINTAINER_DOCKER_IMAGE=... Override base repository + DOCKER_REPRO_DIGEST=... Pin to a specific digest (sha256:...) + HEADS_DISABLE_USB=1 Disable USB passthrough + HEADS_X11_XAUTH=1 Force mounting ~/.Xauthority -# Function to kill GPG toolstack related processes using USB devices -kill_usb_processes() { - # check if scdaemon or pcscd processes are using USB devices - if [ -d /dev/bus/usb ]; then - if sudo lsof /dev/bus/usb/00*/0* 2>/dev/null | awk 'NR>1 {print $2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' >/dev/null; then - echo "Killing GPG toolstack related processes using USB devices..." - sudo lsof /dev/bus/usb/00*/0* 2>/dev/null | awk 'NR>1 {print $2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' | awk '{print $1}' | xargs -r sudo kill -9 - fi - fi +Examples: + ./docker_repro.sh + DOCKER_REPRO_DIGEST=sha256:... ./docker_repro.sh +USAGE } -# Handle Ctrl-C (SIGINT) to exit gracefully -trap "echo 'Script interrupted. Exiting...'; exit 1" SIGINT +if [ "${1:-}" = "-h" ] || [ "${1:-}" = "--help" ]; then + usage + exit 0 +fi -# Check if --help or -h is provided -for arg in "$@"; do - if [[ "$arg" == "--help" || "$arg" == "-h" ]]; then - usage - exit 0 - fi -done +trap 'echo "Script interrupted. Exiting..."; exit 1' SIGINT -# Kill processes using USB devices -kill_usb_processes +# Use the pinned digest from the repository file for reproducible builds +DOCKER_IMAGE="${HEADS_MAINTAINER_DOCKER_IMAGE:-tlaurion/heads-dev-env}" -# Inform the user about entering the Docker container -echo "----" -echo "Usage reminder: The minimal command is 'make BOARD=XYZ', where additional options, including 'V=1' or 'CPUS=N' are optional." -echo "For more advanced QEMU testing options, refer to targets/qemu.md and boards/qemu-*/*.config." -echo -echo "Type exit within docker image to get back to host if launched interactively!" -echo "----" -echo +# Resolve pinned digest (env var preferred, repository file fallback), and prompt if using unpinned :latest +DOCKER_IMAGE="$(resolve_docker_image "$DOCKER_IMAGE" "DOCKER_REPRO_DIGEST" "DOCKER_REPRO_DIGEST" "1")" +# If resolve_docker_image returned empty for any reason, abort +if [ -z "${DOCKER_IMAGE}" ]; then + echo "Error: failed to resolve Docker image; aborting." >&2 + exit 1 +fi -# Execute the docker run command with the provided parameters -if [ -d "/dev/bus/usb" ]; then - echo "--->Launching container with access to host's USB buses (some USB devices were connected to host)..." - docker run --device=/dev/bus/usb:/dev/bus/usb -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) $DOCKER_IMAGE -- "$@" +# Validate that image is pinned to a digest (not an unpinned tag) +if [[ ! "${DOCKER_IMAGE}" =~ @sha256:[0-9a-f]{64} ]]; then + echo "Error: Reproducible builds require pinned digest (@sha256:...), but got: $DOCKER_IMAGE" >&2 + exit 1 +fi + +# Extract digest for CircleCI validation +DIGEST="${DOCKER_IMAGE#*@}" +VERSION=$(grep '^# Version:' "$(dirname "$0")/docker/DOCKER_REPRO_DIGEST" 2>/dev/null | sed 's/# Version: //' | head -n1) +if [ -z "$VERSION" ]; then VERSION="unknown"; fi + +# Cross-validate with .circleci/config.yml (use POSIX grep, not -P) +if [ "${DOCKER_IMAGE%%@*}" = "tlaurion/heads-dev-env" ]; then + CIRCLECI_DIGEST=$(sed -n 's/.*tlaurion\/heads-dev-env@\([^ ]*\).*/\1/p' "$(dirname "$0")/.circleci/config.yml" | head -n1) + if [ -z "$CIRCLECI_DIGEST" ]; then + echo "Warning: Could not find repro image digest in .circleci/config.yml" >&2 + elif [ "$DIGEST" != "$CIRCLECI_DIGEST" ]; then + echo "Error: Digest in resolved image ($DIGEST) does not match the digest used in .circleci/config.yml ($CIRCLECI_DIGEST)" >&2 + exit 1 + fi + echo "Reproducible build (matched .circleci/config.yml): $DOCKER_IMAGE" >&2 + echo "" >&2 else - echo "--->Launching container without access to host's USB buses (no USB devices was connected to host)..." - docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) $DOCKER_IMAGE -- "$@" + echo "Note: Skipping CircleCI digest check for non-canonical image: ${DOCKER_IMAGE%%@*}" >&2 + echo "" >&2 +fi + + +# Only perform host-side side-effects when executed directly (not when sourced) +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + require_docker || exit $? + + # Clean up host processes holding USB devices first (if applicable) + kill_usb_processes + run_docker "$DOCKER_IMAGE" "$@" fi diff --git a/initrd/bin/kexec-seal-key b/initrd/bin/kexec-seal-key index 5cda58bbf..39b8c9e85 100755 --- a/initrd/bin/kexec-seal-key +++ b/initrd/bin/kexec-seal-key @@ -151,8 +151,8 @@ for dev in $key_devices; do if [ "$luks_version" == "2" ]; then # LUKSv2 last key slot is 31 duk_keyslot=31 - regex="^\s+([0-9]+):\s*luks2" - sed_command="s/^\s\+\([0-9]\+\):\s*luks2/\1/g" + regex="^[[:space:]]+([0-9]+):[[:space:]]*luks2" + sed_command="s/^[[:space:]]\+\([0-9]\+\):[[:space:]]*luks2/\1/g" previous_luks_header_version=2 DEBUG "$dev: LUKSv2 device detected" elif [ "$luks_version" == "1" ]; then diff --git a/targets/qemu.md b/targets/qemu.md index 1587fa882..0a7767f17 100644 --- a/targets/qemu.md +++ b/targets/qemu.md @@ -15,20 +15,40 @@ The TPM and disks for this configuration are persisted in the build/qemu-coreboo Bootstrapping a working system === -1. Install QEMU and swtpm. (Optionally, KVM.) - * Many distributions already package swtpm, but Debian Bullseye does not. (Bookworm does.) On Bullseye you will have to build and install libtpms and swtpm from source, see below for detailed instructions. - * https://github.com/stefanberger/libtpms - * https://github.com/stefanberger/swtpm +Important: The supported and tested workflow uses the provided Docker wrappers (`./docker_repro.sh`, `./docker_local_dev.sh`, or `./docker_latest.sh`). Host-side installation of QEMU, `swtpm`, or other QEMU-related tooling is unnecessary and is not part of the standard, supported workflow; only advanced or edge-case scenarios should install those tools on the host (see 'Troubleshooting' below for guidance). + +1. Install Docker + * Install Docker (docker-ce) for your OS by following Docker's official installation guide: https://docs.docker.com/engine/install/ + +Note: the Nix-built Docker images used by `./docker_repro.sh` include +QEMU (`qemu-system-x86_64`), `swtpm` / `libtpms`, `canokey-qemu` (a +virtual OpenPGP smartcard), and other userspace tooling required to +build and test QEMU targets. These images are intended to be +self-contained for QEMU testing; host-focused build instructions +(e.g., building `swtpm` on the host) were removed to avoid +divergence—use the Docker wrappers for the tested workflow. + +If you do not specify `USB_TOKEN` when running QEMU targets, the +container will use the included `canokey-qemu` virtual token by +default. To forward a hardware token from the host, set `USB_TOKEN` or +pass `hostbus`/`hostport`/`vendorid,productid` to the make invocation. + +If you plan to manage disk images or use `qemu-img` snapshots on the +host (outside the container), install the `qemu-utils` package locally +(which provides `qemu-img`). + + 2. Build Heads - * `make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp` + * `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp` 3. Install OS - * `make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp INSTALL_IMG= run` + * `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp INSTALL_IMG=<~/heads/path_to_iso.iso> run` * Lightweight desktops (XFCE, LXDE, etc.) are recommended, especially if KVM acceleration is not available (such nested in Qubes OS) * When running nested in a qube, disable memory ballooning for the qube, or performance will be very poor. * Include `QEMU_MEMORY_SIZE=6G` to set the guest's memory (`6G`, `8G`, etc.). The default is 4G to be conservative, but more may be needed depending on the guest OS. * Include `QEMU_DISK_SIZE=30G` to set the guest's disk size, the default is `20G`. 4. Shut down and boot Heads with the USB token attached, proceed with OEM reset - * `make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp USB_TOKEN= run` + * `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp USB_TOKEN= run` + * If you do not set `USB_TOKEN`, the included `canokey-qemu` virtual token will be used by default. * For ``, use one of: * `NitrokeyPro` - a Nitrokey Pro by VID/PID * `NitrokeyStorage` - a Nitrokey Storage by VID/PID @@ -41,60 +61,136 @@ Bootstrapping a working system * Then Heads will indicate that there is no TOTP code yet, at this point shut down (Continue to main menu -> Power off) 5. Get the public key that was saved to the virtual USB flash drive * `sudo mkdir /media/fd_heads_gpg` - * `sudo losetup --find --partscan ./build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/usb_fd.raw` - * `sudo mount /dev/loop0p2 /media/fd_heads_gpg` to mount the second partition (public) or if only one partition, /dev/loop0p1 + * Attach the image and print the loop device in one step: + + sudo losetup --find --show --partscan ./build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/usb_fd.raw + + The command prints the loop device used (for example `/dev/loop0`) and the kernel will create partition nodes such as `/dev/loop0p1` and `/dev/loop0p2` when supported. + + Then mount the appropriate partition (usually the second/public partition): + + sudo mount /dev/loop0p2 /media/fd_heads_gpg # adjust based on the loop device reported above + * Look in `/media/fd_heads_gpg` and copy the most recent public key * `sudo umount /media/fd_heads_gpg` * `sudo losetup --detach /dev/loop0` 6. Inject the GPG key into the Heads image and run again - * `make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp PUBKEY_ASC= inject_gpg` - * `make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp USB_TOKEN=LibremKey PUBKEY_ASC= run` + * `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp PUBKEY_ASC= inject_gpg` + * `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp USB_TOKEN=LibremKey PUBKEY_ASC= run` 7. Initialize the TPM - select "Reset the TPM" at the TOTP error prompt and follow prompts 8. Select "Default boot" and follow prompts to sign /boot for the first time and set a default boot option You can reuse an already created ROOT_DISK_IMG by passing its path at runtime. -Ex: `make BOARD=qemu-coreboot-fbwhiptail-tpm1 PUBKEY_ASC=~/pub_key_counterpart_of_usb_dongle.asc USB_TOKEN=NitrokeyStorage ROOT_DISK_IMG=~/heads/build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/root.qcow2 run` +Ex: `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1 PUBKEY_ASC=~/pub_key_counterpart_of_usb_dongle.asc USB_TOKEN=NitrokeyStorage ROOT_DISK_IMG=~/heads/build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/root.qcow2 run` + +Note: hardlinks are your friend. You can (should?) have qemu disk images kept somewhere (cp/mv) ~/qemu_img/test.qcow2 and do: + * `cp -alf ~/qemu_img/test.qcow2 ~/heads/build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/root.qcow2` + +This way, if you accidentally wipe ~/heads/build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/root.qcow2, the original is kept intact. +Also note that hardlinks share the same underlying data; modifications to one linked copy affect them all, and the filesystem maintains a link count to track how many references exist. + +`cp -alf` is basically creating a hardlink to destination overwriting it, and doesn't cost additional disk space. On a daily development cycle, usage looks like: -1. `make BOARD=qemu-coreboot-fbwhiptail-tpm1 PUBKEY_ASC=~/pub_key_counterpart_of_usb_dongle.asc USB_TOKEN=NitrokeyStorage ROOT_DISK_IMG=~/heads/build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/root.qcow2 inject_gpg` -2. `make BOARD=qemu-coreboot-fbwhiptail-tpm1 PUBKEY_ASC=~/pub_key_counterpart_of_usb_dongle.asc USB_TOKEN=NitrokeyStorage ROOT_DISK_IMG=~/heads/build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/root.qcow2 run` +1. `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1 PUBKEY_ASC=~/pub_key_counterpart_of_usb_dongle.asc USB_TOKEN=NitrokeyStorage ROOT_DISK_IMG=~/heads/build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/root.qcow2 inject_gpg` +2. `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1 PUBKEY_ASC=~/pub_key_counterpart_of_usb_dongle.asc USB_TOKEN=NitrokeyStorage ROOT_DISK_IMG=~/heads/build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/root.qcow2 run` -The first command builds latest uncommited/unsigned changes and injects the public key inside of the rom to be ran by the second command. +The first command builds the latest uncommitted/unsigned changes and injects the public key inside the ROM to be run by the second command. To test across all qemu variants, one only has to change BOARD name and run the two previous commands, adapting `QEMU_MEMORY_SIZE=1G` or modifying the file directly under build dir to adapt to host resources. -swtpm on Debian Bullseye -=== -libtpms and swtpm must be built and installed from source on Debian Bullseye. Upstream provides tooling to build these as Debian packages, which allows things to work seamlessly with default AppArmor configs, etc. - -1. Install dependencies - * `sudo apt install automake autoconf libtool make gcc libc-dev libssl-dev dh-autoreconf libssl-dev libtasn1-6-dev pkg-config net-tools iproute2 libjson-glib-dev libgnutls28-dev expect gawk socat gnutls-bin libseccomp-dev libfuse-dev python3-twisted selinux-policy-dev trousers devscripts equivs` -2. Build libtpms - * `git clone https://github.com/stefanberger/libtpms` - * `cd libtpms; git checkout v0.9.4` (latest release as of this writing) - * `sudo mk-build-deps --install ./debian/control` - * `debuild -us -uc` - * `sudo apt install ../libtpms*.deb` -3. Build swtpm - * `git clone https://github.com/stefanberger/swtpm` - * `cd swtpm; git checkout v0.7.3` (latest release as of this writing) - * `echo "libtpms0 libtpms" > ./debian/shlibs.local` - * `sudo mk-build-deps --install ./debian/control` - * `debuild -us -uc` - * `sudo apt install ../swtpm*.deb` - -swtpm on Debian bookworm +Running via Docker wrappers === -1. Install dependencies - * `sudo apt install swtpm swtpm-tools` +We provide convenient wrapper scripts at the repository root that encapsulate Docker invocation and automatically handle common host integrations needed for QEMU runs. -swtpm on nix docker image -=== -Nothing to do. Everything needed is in the docker image. - -Just make sure to pass DISPLAY environement variable on your docker command line. eg: -* Remotely downloaded docker image (doing make command only inside of docker example): - * `docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=qemu-coreboot-whiptail-tpm2` - * `docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=qemu-coreboot-whiptail-tpm2 run` -* Locally created docker image from nix develop environment (jumping into docker image variation of the above, where developer does what he wants within): - * `docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) linuxboot/heads:dev-env` +Wrapper comparison +--- + +| Script | Image | Use | +|---|---:|---| +| `docker_latest.sh` | Defaults to pinned digest when available | Convenience: run the latest published image | +| `docker_local_dev.sh` | `linuxboot/heads:dev-env` | Development: use local image built from the flake (rebuilds when flake files are dirty) | +| `docker_repro.sh` | Image pinned from `.circleci/config.yml` | Reproducible builds that match CircleCI | + +What the wrappers handle +--- + +Wrapper options: some runtime behavior is controlled via environment +variables documented in the repository README (see 'Wrapper options & +environment variables'). Wrapper scripts now have focused `--help` output +for their own variables, and `./docker/common.sh` prints the full +environment reference. Important ones are `HEADS_DISABLE_USB` +(set to `1` to disable automatic USB passthrough and cleanup) and +`HEADS_X11_XAUTH` (force mounting your `$HOME/.Xauthority`). + +Make variables such as `USB_TOKEN`, `PUBKEY_ASC`, `INSTALL_IMG`, +`QEMU_MEMORY_SIZE`, `QEMU_DISK_SIZE`, `ROOT_DISK_IMG`, `CPUS` and `V` +are forwarded to the `make` invocation and affect how +`targets/qemu.mk` runs QEMU. See `targets/qemu.mk` for token formats +and examples. + +Note: when USB passthrough is active the wrapper will warn and, on +interactive shells, give a 3s abort window before attempting to kill +processes that hold the token (e.g., `scdaemon`/`pcscd`) to free the +device; set `HEADS_DISABLE_USB=1` to opt out. + +- **KVM passthrough**: when `/dev/kvm` exists on the host the container is run with `/dev/kvm` mounted into the container, enabling KVM-accelerated QEMU. +- **X11 GUI support**: the wrappers mount the X11 socket and programmatically create a temporary Xauthority file (via `mktemp -t heads-docker-xauth-XXXXXX`, or `/tmp/.docker.xauth-` as fallback when mktemp is unavailable) when `xauth` is available; they fall back to mounting `${HOME}/.Xauthority` when needed and set `XAUTHORITY` inside the container so GTK/SDL QEMU windows work. The temp file is cleaned up automatically after `docker run` completes. + - To force mounting your `${HOME}/.Xauthority` regardless of socket detection, set `HEADS_X11_XAUTH=1`. +- **USB passthrough**: when host USB buses exist `/dev/bus/usb` is mounted into the container so VMs can access hardware tokens. To explicitly disable automatic USB passthrough set `HEADS_DISABLE_USB=1`. +- **USB token cleanup**: the wrappers attempt to detect and stop local GPG/toolstack processes (e.g., `scdaemon`, `pcscd`) which might hold USB tokens. Behavior notes: + - If `sudo` can be run without a password the cleanup runs silently. + - The cleanup avoids prompting for a password in non-interactive shells; it will prompt only when running interactively (attached to a TTY). To skip the cleanup entirely set `HEADS_DISABLE_USB=1`. +- **Convenience variables accepted by the wrappers**: `V=1` for verbose make output, `CPUS=N` to set parallelism for builds, and any `make` variables may be passed through to the container command. +- **Argument forwarding**: arguments given to the wrapper are forwarded directly to the container command (no special separator needed). For example: `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run`. + +Environment variables reference +--- + +| Variable | Default | Effect | +|---|---:|---| +| `HEADS_DISABLE_USB` | `0` | When `1`, disable automatic USB passthrough and USB cleanup | +| `HEADS_X11_XAUTH` | `0` | When `1`, mount `${HOME}/.Xauthority` into the container (force usage even when a programmatic Xauthority would otherwise be created) | +| `HEADS_SKIP_DOCKER_REBUILD` | `0` | When `1`, skip rebuilding the local Docker image when `flake.nix`/`flake.lock` are dirty | +| `HEADS_AUTO_INSTALL_NIX` | `0` | When `1`, automatically attempt single-user Nix install if `nix` is missing (suppresses prompt) | +| `HEADS_AUTO_ENABLE_FLAKES` | `0` | When `1`, automatically enable flakes by writing to `$HOME/.config/nix/nix.conf` (suppresses prompt) | +| `HEADS_MIN_DISK_GB` | `50` | Minimum free disk in GB required on `/nix` or `/` before attempting rebuild | +| `HEADS_SKIP_DISK_CHECK` | `0` | When `1`, skip the disk-space preflight check | + +Examples +--- + +- Reproducible (uses image version from CircleCI config): + - `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run` + - `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 PUBKEY_ASC=pubkey.asc USB_TOKEN=Nitrokey3NFC inject_gpg` + - `HEADS_DISABLE_USB=1 ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 PUBKEY_ASC=pubkey.asc run` + - `HEADS_X11_XAUTH=1 ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run` + +- Local development image (uses locally built `linuxboot/heads:dev-env`): + - `./docker_local_dev.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2` + +- Published latest image (convenience): + - `./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run` + +How I tested these wrappers (smoke checks) +--- + +- Minimal: `source docker/common.sh && build_docker_opts` — should print a short description and show flags such as `--device=/dev/kvm` when KVM is available and `-v /tmp/heads-docker-xauth-XXXXXX:...` (or `-v /tmp/.docker.xauth-:...` as fallback) when Xauthority was created. +- Functional (examples tested by PR author): see the tests in the PR body (Ubuntu, Debian, Fedora installer flows). Consider testing `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run` locally to verify KVM+GTK behavior. + +Troubleshooting +--- + +- Quick checks: + - `echo $DISPLAY` — ensure `DISPLAY` is set on the host. + - `command -v xauth` — preferred for programmatic Xauthority cookies. + - `ls -l /dev/kvm` — verify `/dev/kvm` exists and is accessible. + - `groups | grep -q kvm` — confirm your user is in a group with access to KVM (or run with appropriate privileges). + - `source docker/common.sh && build_docker_opts` — inspect the options the wrapper will use without launching Docker. +- GUI issues: prefer installing `xauth` on the host so the wrappers can create a safe programmatic Xauthority file. As a last resort you can run `xhost +SI:localuser:root` (less secure). +- USB/GPG cleanup: if the cleanup is refusing to run due to non-interactive sudo, run the kill steps manually or set `HEADS_DISABLE_USB=1` to skip automatic cleanup. + +Notes +--- +- Ensure you have an X server available on the host; the wrappers forward `DISPLAY` automatically. +- If KVM is available but `/dev/kvm` is missing, load kernel modules (e.g., `kvm`, `kvm_intel`, `kvm_amd`) so `/dev/kvm` appears. From 882370f61180e69ed43788fb6af1e4519162d7a3 Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Wed, 4 Feb 2026 17:25:05 -0500 Subject: [PATCH 2/4] Bump nix develop based docker image to tlaurion/heads-dev-env:v0.2.7 Upgrade the Docker development environment image from v0.2.5 to v0.2.7. This version adds coreboot-utils as a dependency, which is required by diffoscope for ROM analysis and component extraction (ifdtool, cbfsutils). The image has been verified to be reproducible (unchanged content after push to docker.io). Changes: .circleci/config.yml: - Pin all Docker image references from :v0.2.5 to @sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 (v0.2.7) - Add version comment for human readability (# Docker image: tlaurion/heads-dev-env:v0.2.7) - All four job definitions (prep_env, build_and_persist, build, save_cache) updated - Ensures reproducible CI builds matching published image exactly docker/DOCKER_REPRO_DIGEST: - Update pinned digest from v0.2.5 to v0.2.7 - sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 - Add version comment for human readability flake.nix: - Add coreboot-utils dependency - Comment: "consumed by diffoscope for ifdtool cbfsutils etc" - Required for ROM analysis and investigation workflows docker/get_digest.sh, docker/pin-and-run.sh: - Update example references from v0.2.6 to v0.2.7 - Improve documentation consistency Testing: - Reproducibility checks against tlaurion/heads-dev-env:v0.2.7 - CircleCI workflow validated with pinned digest - All existing functionality confirmed working Summary of changes: - 1 configuration update: .circleci/config.yml (docker image pinning) - 1 digest file update: docker/DOCKER_REPRO_DIGEST - 1 build dependency: flake.nix (coreboot-utils) - 2 documentation updates: docker/get_digest.sh, docker/pin-and-run.sh example references Signed-off-by: Thierry Laurion --- .circleci/config.yml | 12 ++++++++---- docker/DOCKER_REPRO_DIGEST | 3 ++- docker/get_digest.sh | 8 ++++---- docker/pin-and-run.sh | 10 +++++----- flake.nix | 1 + 5 files changed, 20 insertions(+), 14 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index c168524dc..4f3f8256b 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -48,7 +48,8 @@ commands: jobs: prep_env: docker: - - image: tlaurion/heads-dev-env:v0.2.5 + # Docker image: tlaurion/heads-dev-env:v0.2.7 + - image: tlaurion/heads-dev-env@sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 resource_class: large working_directory: ~/heads steps: @@ -123,7 +124,8 @@ jobs: build_and_persist: docker: - - image: tlaurion/heads-dev-env:v0.2.5 + # Docker image: tlaurion/heads-dev-env:v0.2.7 + - image: tlaurion/heads-dev-env@sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 resource_class: large working_directory: ~/heads parameters: @@ -151,7 +153,8 @@ jobs: build: docker: - - image: tlaurion/heads-dev-env:v0.2.5 + # Docker image: tlaurion/heads-dev-env:v0.2.7 + - image: tlaurion/heads-dev-env@sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 resource_class: large working_directory: ~/heads parameters: @@ -172,7 +175,8 @@ jobs: save_cache: docker: - - image: tlaurion/heads-dev-env:v0.2.5 + # Docker image: tlaurion/heads-dev-env:v0.2.7 + - image: tlaurion/heads-dev-env@sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 resource_class: large working_directory: ~/heads steps: diff --git a/docker/DOCKER_REPRO_DIGEST b/docker/DOCKER_REPRO_DIGEST index 7cb961358..7e1624946 100644 --- a/docker/DOCKER_REPRO_DIGEST +++ b/docker/DOCKER_REPRO_DIGEST @@ -9,4 +9,5 @@ # sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa # Place the digest on the first non-comment line below (remove the leading '#') -sha256-50a9110cdfc6a74a383169d7c624139c3b3e05567b87203498118a8a33dd79f1 +# Version: v0.2.7 +sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 diff --git a/docker/get_digest.sh b/docker/get_digest.sh index 87ddb35f2..cf7683ada 100755 --- a/docker/get_digest.sh +++ b/docker/get_digest.sh @@ -7,7 +7,7 @@ Usage: $0 [--yes|-y] IMAGE[:TAG|@DIGEST] Helper to print the full 'repo@digest' and the raw digest for a docker image. Behavior: - - The script treats the provided image reference literally. Provide exact `repo/name:tag` or `repo@digest` (e.g. `tlaurion/heads-dev-env:v0.2.6`). + - The script treats the provided image reference literally. Provide exact `repo/name:tag` or `repo@digest` (e.g. `tlaurion/heads-dev-env:v0.2.7`). - If the image exists locally, the script prints the first RepoDigest (repo@digest) and the raw digest. - If the image is not present locally, the script will offer to pull the exact provided reference to obtain a local RepoDigest (interactive or `-y`). - The script prefers to operate on local image state (e.g., Docker local RepoDigests). If a local digest is not available it may query the Docker Hub v2 HTTP API (docker.io) via `curl` to obtain an authoritative manifest digest for docker.io images; this requires network access and appropriate registry connectivity. For other registries or Docker versions you may still need to use `docker manifest inspect` or `skopeo inspect` manually if `RepoDigests` is not populated. @@ -17,9 +17,9 @@ Options: -h, --help Show this help message Examples: - ./docker/get_digest.sh tlaurion/heads-dev-env:v0.2.6 + ./docker/get_digest.sh tlaurion/heads-dev-env:v0.2.7 ./docker/get_digest.sh tlaurion/heads-dev-env:latest - ./docker/get_digest.sh -y tlaurion/heads-dev-env:v0.2.6 + ./docker/get_digest.sh -y tlaurion/heads-dev-env:v0.2.7 # Note: provide the exact repo:name:tag you intend; the script treats the reference literally. USAGE } @@ -47,7 +47,7 @@ fi image="$1" # Treat the provided image reference literally and do not try to append ':latest'. -# The caller should provide the exact reference they intend (e.g. 'tlaurion/heads-dev-env:v0.2.6'), +# The caller should provide the exact reference they intend (e.g. 'tlaurion/heads-dev-env:v0.2.7'), # and the script will inspect that exact reference and prompt to pull it if missing. image_provided="${image}" image="${image_provided}" diff --git a/docker/pin-and-run.sh b/docker/pin-and-run.sh index a4c34b9ee..33f9ffd4e 100755 --- a/docker/pin-and-run.sh +++ b/docker/pin-and-run.sh @@ -6,7 +6,7 @@ usage() { Usage: $0 [-y|--yes] [-w|--wrapper WRAPPER] IMAGE [-- [WRAPPER [WRAPPER_ARGS...]]] Helper: obtain an image digest and run a docker wrapper pinned to that digest. -- IMAGE: an exact docker image ref (e.g. tlaurion/heads-dev-env:v0.2.6) +- IMAGE: an exact docker image ref (e.g. tlaurion/heads-dev-env:v0.2.7) - If the image is not present locally, the helper will probe the registry and offer to pull it (use -y/--yes to auto-pull). - WRAPPER: the docker wrapper to execute (e.g. ./docker_latest.sh or ./docker_repro.sh). @@ -20,16 +20,16 @@ Options: Examples: # Interactive: obtain digest and run the 'latest' wrapper pinned to that digest (explicit wrapper recommended) - ./docker/pin-and-run.sh tlaurion/heads-dev-env:v0.2.6 -- ./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 + ./docker/pin-and-run.sh tlaurion/heads-dev-env:v0.2.7 -- ./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 # Auto-pull and run (auto-pull the ref to obtain a local digest then run wrapper) - ./docker/pin-and-run.sh -y tlaurion/heads-dev-env:v0.2.6 -- ./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 + ./docker/pin-and-run.sh -y tlaurion/heads-dev-env:v0.2.7 -- ./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 # Shortcut: omit the wrapper and just provide the command — the helper will use the default './docker_latest.sh' - ./docker/pin-and-run.sh tlaurion/heads-dev-env:v0.2.6 -- make BOARD=qemu-coreboot-fbwhiptail-tpm2 + ./docker/pin-and-run.sh tlaurion/heads-dev-env:v0.2.7 -- make BOARD=qemu-coreboot-fbwhiptail-tpm2 # Use a different wrapper explicitly (e.g. repro): - ./docker/pin-and-run.sh -w ./docker_repro.sh tlaurion/heads-dev-env:v0.2.6 -- make BOARD=qemu-coreboot-fbwhiptail-tpm2 + ./docker/pin-and-run.sh -w ./docker_repro.sh tlaurion/heads-dev-env:v0.2.7 -- make BOARD=qemu-coreboot-fbwhiptail-tpm2 USAGE } diff --git a/flake.nix b/flake.nix index ba0afa9cd..0b985424a 100644 --- a/flake.nix +++ b/flake.nix @@ -30,6 +30,7 @@ bzip2 cacert ccache + coreboot-utils #consumed by diffoscope for ifdtool cbfsutils etc cmake cpio curl From de7e630f63b70c3d045fd379f1dacd9696288e31 Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Sun, 8 Feb 2026 17:50:42 -0500 Subject: [PATCH 3/4] etc/functions: add wait_for_usb_devices helper to fix gpg --card-status failing because race condition without sleep Signed-off-by: Thierry Laurion --- initrd/etc/functions | 93 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 90 insertions(+), 3 deletions(-) diff --git a/initrd/etc/functions b/initrd/etc/functions index c27e69b72..e1b5acfdf 100644 --- a/initrd/etc/functions +++ b/initrd/etc/functions @@ -195,13 +195,16 @@ confirm_gpg_card() { # setup the USB so we can reach the USB Security dongle's OpenPGP smartcard enable_usb + # Wait for USB enumeration before accessing GPG card to avoid race condition + wait_for_usb_devices echo -e "\nVerifying presence of GPG card...\n" # ensure we don't exit without retrying errexit=$(set -o | grep errexit | awk '{print $2}') set +e - gpg_output=$(gpg --card-status 2>&1) - if [ $? -ne 0 ]; then + DEBUG "Attempting gpg card detection (bounded wait)" + if ! wait_for_gpg_card; then + DEBUG "GPG card access failed with output: $gpg_output" # prompt for reinsertion and try a second time read -n1 -r -p \ "Can't access GPG key; remove and reinsert, then press Enter to retry. " \ @@ -211,8 +214,10 @@ confirm_gpg_card() { set -e fi # retry card status - gpg_output=$(gpg --card-status 2>&1) || + DEBUG "Retrying gpg --card-status after reinsertion (bounded wait)" + wait_for_gpg_card || die "gpg card read failed" + DEBUG "Retry succeeded" fi # Extract and display GPG PIN retry counters @@ -419,7 +424,89 @@ enable_usb() { insmod /lib/modules/ehci-pci.ko || die "ehci_pci: module load failed" insmod /lib/modules/xhci-hcd.ko || die "xhci_hcd: module load failed" insmod /lib/modules/xhci-pci.ko || die "xhci_pci: module load failed" +} + +# Wait for USB bus enumeration to complete after enable_usb() loads modules. +# Uses time-bounded polling (max 2s) to avoid race conditions where device +# nodes haven't been created yet. No hardcoded sleep - checks actual readiness. +# Waits for actual USB peripheral devices (e.g., 1-1, 5-3), not just hubs/controllers. +wait_for_usb_devices() { + TRACE_FUNC + if [ ! -d /sys/bus/usb/devices ] || [ ! -r /proc/uptime ]; then + DEBUG "USB sysfs or uptime not available, skipping wait" + return + fi + + local start now elapsed + start=$(awk '{print $1}' /proc/uptime) + DEBUG "Waiting for USB peripheral devices (not just hubs) - max 2s timeout" + + local iteration=0 + while :; do + iteration=$((iteration + 1)) + + # Check for actual USB peripheral devices (format: bus-port like 1-1, 5-3) + # Root hubs are named usb1, usb2, etc. - we want devices downstream from them + # Pattern: /sys/bus/usb/devices/[0-9]*-[0-9]*/idVendor (e.g., 1-1, 5-3.2) + local peripheral_count=0 + if [ -d /sys/bus/usb/devices ]; then + # Count devices matching bus-port pattern (not usb* root hubs) + for dev in /sys/bus/usb/devices/*-*/idVendor; do + if [ -r "$dev" ]; then + peripheral_count=$((peripheral_count + 1)) + fi + done + fi + + now=$(awk '{print $1}' /proc/uptime) + elapsed=$(awk -v s="$start" -v n="$now" 'BEGIN{printf "%.3f", n - s}') + + if [ $peripheral_count -gt 0 ]; then + DEBUG "USB peripheral devices ready after ${elapsed}s (iteration $iteration): found $peripheral_count device(s)" + return + fi + + # Timeout after 2 seconds + if awk -v s="$start" -v n="$now" 'BEGIN{exit (n - s > 2.0) ? 0 : 1}'; then + DEBUG "USB wait timeout at ${elapsed}s (iter $iteration): only found $peripheral_count peripheral device(s)" + return + fi + done +} +# Wait for gpg --card-status to succeed (bounded, no sleep). +# Sets global gpg_output with the last command output. +wait_for_gpg_card() { + TRACE_FUNC + if [ ! -r /proc/uptime ]; then + gpg_output=$(gpg --card-status 2>&1) + return $? + fi + + local start now elapsed + start=$(awk '{print $1}' /proc/uptime) + local attempt=0 + while :; do + attempt=$((attempt + 1)) + gpg_output=$(gpg --card-status 2>&1) + if [ $? -eq 0 ]; then + now=$(awk '{print $1}' /proc/uptime) + elapsed=$(awk -v s="$start" -v n="$now" 'BEGIN{printf "%.3f", n - s}') + DEBUG "gpg --card-status succeeded after ${elapsed}s (attempt $attempt)" + return 0 + fi + + now=$(awk '{print $1}' /proc/uptime) + elapsed=$(awk -v s="$start" -v n="$now" 'BEGIN{printf "%.3f", n - s}') + if awk -v s="$start" -v n="$now" 'BEGIN{exit (n - s > 2.0) ? 0 : 1}'; then + DEBUG "gpg --card-status timeout at ${elapsed}s (attempt $attempt)" + return 1 + fi + done +} + +enable_usb_keyboard() { + TRACE_FUNC # For resiliency, test CONFIG_USB_KEYBOARD_REQUIRED explicitly rather # than having it imply CONFIG_USER_USB_KEYBOARD at build time. # Otherwise, if a user got CONFIG_USER_USB_KEYBOARD=n in their From 699a1a7ed7efab9b91f4d3a70d1182fa0cf8c2cb Mon Sep 17 00:00:00 2001 From: Thierry Laurion Date: Sun, 8 Feb 2026 18:16:33 -0500 Subject: [PATCH 4/4] Remove DO_WITH_DEBUG from increment_tpm_counter calls DO_WITH_DEBUG redirects stdout/stderr through tee for logging, which breaks interactive password prompts by interfering with TTY access. Both increment_tpm_counter calls already redirect output to /dev/null, so DO_WITH_DEBUG provided no logging benefit while breaking prompts. This allows TPM owner password prompts to display correctly on console when TPM counters need to be created or incremented. Signed-off-by: Thierry Laurion --- initrd/bin/gui-init | 2 +- initrd/bin/kexec-sign-config | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/initrd/bin/gui-init b/initrd/bin/gui-init index 7eedc4757..ed32a6143 100755 --- a/initrd/bin/gui-init +++ b/initrd/bin/gui-init @@ -566,7 +566,7 @@ reset_tpm() { DEBUG "TPM_COUNTER: $TPM_COUNTER" #TPM_COUNTER can be empty - DO_WITH_DEBUG increment_tpm_counter $TPM_COUNTER>/dev/null 2>&1 || + increment_tpm_counter $TPM_COUNTER>/dev/null 2>&1 || die "Unable to increment tpm counter" DO_WITH_DEBUG sha256sum /tmp/counter-$TPM_COUNTER >/boot/kexec_rollback.txt || diff --git a/initrd/bin/kexec-sign-config b/initrd/bin/kexec-sign-config index a3f1a7c32..b994a8b51 100755 --- a/initrd/bin/kexec-sign-config +++ b/initrd/bin/kexec-sign-config @@ -94,7 +94,7 @@ if [ "$rollback" = "y" ]; then # Increment the TPM counter DEBUG "rollback=y: Incrementing counter $TPM_COUNTER." - DO_WITH_DEBUG increment_tpm_counter $TPM_COUNTER >/dev/null 2>&1 || + increment_tpm_counter $TPM_COUNTER >/dev/null 2>&1 || die "$paramsdir: Unable to increment tpm counter" # Ensure the incremented counter file exists