From ee2200de668cc6d9cf51037433afe4a577e251e1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Phorcys=20=F0=9F=90=BE?= Date: Mon, 24 Aug 2026 22:45:47 +0000 Subject: [PATCH] refactor(installer): use a path: flake so the host stays gitignored MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The generated per-host dir (hosts//) is gitignored because it holds secrets (install-answers.json, local.nix). A git flake only sees tracked files, so install.sh force-added the host with `git add --intent-to-add -f` just to make Nix see it — which staged those secrets and left the tree permanently dirty. Reference the repo as a `path:` flake instead, which copies the directory verbatim (gitignored files included), and drop the intent-add entirely. The host is now truly untracked: never staged, never in `git status`, secrets never enter the index. All install-time and on-box flake commands (nix eval/build, disko, nixos-install, coder-reset's nixos-rebuild) use path:/… refs. Also make the installed /etc/nixos-repo usable as a normal git checkout: chown root:wheel + group-writable so wheel users `git pull` / edit without sudo, and check out a `main` branch tracking origin (a baked image .git is a detached, branchless CI checkout, so `git pull` had no branch to pull). Docs (README, agents.md, hosts/incus-vm/README) updated for the path: rebuild command and the writable, branch-tracking repo; removed the stale intent-to-add / dirty-tree guidance. --- README.md | 31 ++++++++++++++------------- agents.md | 30 ++++++++++++++------------ configuration.nix | 11 +++++----- hosts/incus-vm/README.md | 16 ++++++-------- install.sh | 46 +++++++++++++++++++++++----------------- 5 files changed, 73 insertions(+), 61 deletions(-) diff --git a/README.md b/README.md index e2fd6e5..5d954a8 100644 --- a/README.md +++ b/README.md @@ -20,9 +20,12 @@ NixOS configuration for Coder demo and workshop boxes. This repo is a Nix flake. `flake.nix` auto-discovers every subdirectory of `./hosts/` that contains a `default.nix` and exposes it as `nixosConfigurations.`. For normal install hosts the folder name -is also the hostname, so `nixos-rebuild switch --flake .` auto-selects the -right config on the running box. Adding a new host means creating a host -folder, no flake.nix edit. The installer does this for you. +is also the hostname, so `nixos-rebuild switch --flake path:/etc/nixos-repo` +auto-selects the right config on the running box. Adding a new host means +creating a host folder, no flake.nix edit. The installer does this for you. +(The per-host dir is gitignored, so the box uses a `path:` flake ref — which +copies the tree verbatim, gitignored files included — rather than a git flake, +which would only see tracked files. See [Applying changes](#applying-changes).) Hosts whose folder name starts with an underscore (`_appliance-iso`, `_appliance-disk`, `_installer-iso`) are image builds, not per-machine installs: they @@ -61,7 +64,7 @@ sudo ./install.sh \ `./install.sh --help` lists everything. `--coder-admin-password-file PATH` and `--nixos-password-file PATH` read passwords from a file so they don't end up in shell history. `--no-reboot` skips the automatic reboot at the end. -The installer generates `hosts//{default.nix,local.nix,install-answers.json,facter.json}`, copies the repo into `/etc/nixos-repo` on the target, and symlinks `/etc/nixos/flake.nix`. After reboot, `nixos-rebuild switch` Just Works. Continue with [After install](#after-install). +The installer generates `hosts//{default.nix,local.nix,install-answers.json,facter.json}`, copies the repo into `/etc/nixos-repo` on the target (owned `root:wheel` and checked out on `main`, so wheel users can `git pull` / edit it without sudo), and symlinks `/etc/nixos/flake.nix`. Rebuild with `sudo nixos-rebuild switch --flake path:/etc/nixos-repo`. Continue with [After install](#after-install). > **Different partition layout?** Don't import `installer/bootstrap/disko-standard.nix`; drop your own disko config into the host folder instead. See [disko examples](https://github.com/nix-community/disko/tree/master/example). @@ -196,26 +199,24 @@ boot via `coder-init-admin.service`. After the reboot: defaults. Subsequent edits to `coderd/` templates go out via `coder-template-sync` -on every `sudo nixos-rebuild switch`. +on every `sudo nixos-rebuild switch --flake path:/etc/nixos-repo`. ## Applying changes +The per-host dir (`hosts//`) is gitignored and stays untracked, so the +box uses a `path:` flake ref (a plain git flake would not see it). The repo is +owned `root:wheel`, so wheel users edit it and `git pull` without sudo. + ```sh -sudo nixos-rebuild switch # most changes -sudo nixos-rebuild boot && sudo reboot # changes that touch the desktop stack - -# Edited hosts//local.nix, install-answers.json, or facter.json? Re-mark intent-to-add: -sudo git -C /etc/nixos-repo add --intent-to-add -f \ - hosts//local.nix \ - hosts//install-answers.json \ - hosts//facter.json +sudo nixos-rebuild switch --flake path:/etc/nixos-repo # most changes +sudo nixos-rebuild boot --flake path:/etc/nixos-repo && sudo reboot # desktop-stack changes ``` ## Updating nixpkgs / disko / facter ```sh sudo nix flake update --flake /etc/nixos-repo -sudo nixos-rebuild switch +sudo nixos-rebuild switch --flake path:/etc/nixos-repo ``` This bumps `flake.lock` to the latest of each input. @@ -274,7 +275,7 @@ Fully automated, no follow-up steps needed. The service: ### Changing the admin password 1. Edit `hosts//install-answers.json`, update `initialUser.password` (or override `services.coder-nixos.initialUser.password` in `local.nix`). -2. Run `sudo nixos-rebuild switch` to bake the new password into the service. +2. Run `sudo nixos-rebuild switch --flake path:/etc/nixos-repo` to bake the new password into the service. 3. Run `sudo systemctl start coder-reset` to wipe and re-bootstrap with the new password. > If you need to change the password on a **live** deployment without a full wipe: diff --git a/agents.md b/agents.md index 9e31472..878af90 100644 --- a/agents.md +++ b/agents.md @@ -8,7 +8,7 @@ Reference for AI coding agents and humans working on this repo. |---|---| | SSH | `ssh -i ~/.ssh/id_ed25519 coderbox@` | | Repo path | `/etc/nixos-repo/` (a Nix flake; `nixosConfigurations.`) | -| Git ops | `sudo git -C /etc/nixos-repo ` | +| Git ops | `git -C /etc/nixos-repo ` (repo is `root:wheel`, wheel-writable) | | Coder URL | `http://coder-thinkcentre.local:3000` | | Coder token | stored in `/etc/coder/session-token` | | Coder binary | `coder` (in PATH via NixOS; resolves from nix store) | @@ -27,16 +27,19 @@ sudo k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml ... ```sh # Service, package, or config changes — safe, non-destructive: -cd /etc/nixos-repo && sudo nixos-rebuild switch --flake /etc/nixos-repo +cd /etc/nixos-repo && sudo nixos-rebuild switch --flake path:/etc/nixos-repo # Desktop stack (GNOME, GDM, Wayland) — must reboot: -cd /etc/nixos-repo && sudo nixos-rebuild boot --flake /etc/nixos-repo && sudo reboot +cd /etc/nixos-repo && sudo nixos-rebuild boot --flake path:/etc/nixos-repo && sudo reboot ``` The repo is baked onto the box at **`/etc/nixos-repo`** (the canonical flake; `nixosConfigurations.`, auto-selected by the running hostname). Edit -files there, then rebuild. Always pass `--flake /etc/nixos-repo` (or `cd` into -it and use `--flake .`) — see the `/etc/nixos` pitfall below. +files there, then rebuild. Always pass `--flake path:/etc/nixos-repo`. The +per-host dir (`hosts//`) is gitignored and untracked, so a `path:` flake +ref is required — a plain git flake (`--flake /etc/nixos-repo` or `--flake .`) +only sees tracked files and would not find the host. See the `/etc/nixos` +pitfall below. `nixos-rebuild switch` triggers the `coder-template-sync` activation script, which runs `terraform apply` in `coderd/` and pushes any template changes to Coder. The `/etc/coder/session-token` it needs is populated automatically by `coder-init-admin.service` on first boot, so this just works post-install. @@ -73,13 +76,14 @@ if something is off. Run `make fmt` to fix formatting locally. ## Git Workflow -All files in `/etc/nixos-repo/` are root-owned. Use `sudo git`: +All files in `/etc/nixos-repo/` are owned `root:wheel` and group-writable, so +wheel users run git without sudo: ```sh cd /etc/nixos-repo -sudo git status -sudo git add -p -sudo git commit -m "feat: describe your change" +git status +git add -p +git commit -m "feat: describe your change" # Don't push unless explicitly asked ``` @@ -122,7 +126,7 @@ sudo terraform apply \ -var="coder_url=http://localhost:3000" \ -var="coder_session_token=$(sudo cat /etc/coder/session-token)" \ -var="hostname=coder-thinkcentre" \ - -var="version_name=$(sudo git -C /etc/nixos-repo rev-parse --short HEAD)" + -var="version_name=$(git -C /etc/nixos-repo rev-parse --short HEAD)" ``` ## nook-android Template @@ -197,8 +201,8 @@ sudo k3s kubectl describe pod -n coder-workspaces - **Tailscale auth doesn't re-run** — `tailscale-autoauth` has `RemainAfterExit = true`. If you change auth key config, run `sudo systemctl restart tailscale-autoauth`. - **Template sync skips**, if `/etc/coder/session-token` is empty, the activation script exits cleanly. The token is auto-populated by `coder-init-admin.service`; if it's missing, check `journalctl -u coder-init-admin`. - **`coder` binary path** — the binary is in PATH via NixOS environment; don't hardcode nix store paths in scripts (they change with every package update). -- **`--flake /etc/nixos` fails** — `/etc/nixos` is a plain dir holding only a `flake.nix` *symlink* into `/etc/nixos-repo`. Nix follows the symlink into the store but can't find the sibling files (configuration.nix, hosts/, nixos/), dying with `path '/nix/store/...-source/etc/nixos-repo/flake.nix' does not exist`. Always rebuild against the real tree: `--flake /etc/nixos-repo` (or `cd /etc/nixos-repo && nixos-rebuild switch --flake .`). -- **`Git tree '/etc/nixos-repo' is dirty` warning** — harmless. `hosts//{local.nix,install-answers.json,facter.json}` are gitignored and intent-to-added by the installer, so the tree always reads "dirty". After editing them, re-mark intent-to-add so the flake sees them: `sudo git -C /etc/nixos-repo add --intent-to-add -f hosts//local.nix hosts//install-answers.json hosts//facter.json`. +- **`--flake /etc/nixos` fails** — `/etc/nixos` is a plain dir holding only a `flake.nix` *symlink* into `/etc/nixos-repo`. Nix follows the symlink into the store but can't find the sibling files (configuration.nix, hosts/, nixos/), dying with `path '/nix/store/...-source/etc/nixos-repo/flake.nix' does not exist`. Always rebuild against the real tree with a `path:` ref: `--flake path:/etc/nixos-repo`. +- **Host not found / `does not provide attribute 'nixosConfigurations.'`** — the per-host dir is gitignored and untracked, so a git flake (`--flake /etc/nixos-repo` or `--flake .`) can't see it. Use `--flake path:/etc/nixos-repo`, which copies the tree verbatim (gitignored files included). This is also why nothing intent-adds the host and `git status` stays clean. - **ScreenConnect blank screen** — the box now runs GNOME on Wayland (GDM), and GNOME 49 dropped the Xorg session, so there is no X11 desktop to fall back to. ScreenConnect reaches `DISPLAY=:0` through XWayland (see `nixos/screenconnect.nix`) but **cannot screen-capture the Wayland compositor** through it, so the remote view may be black/blank. Capturing the GNOME session needs a Wayland-aware path (PipeWire/portal, e.g. `gnome-remote-desktop`); the X11 agent will connect but not mirror the desktop. ## Wildcard App Access (TODO) @@ -212,7 +216,7 @@ sudo k3s kubectl describe pod -n coder-workspaces ## File Layout (agent-relevant paths) ``` -/etc/nixos-repo/ # repo root (a Nix flake; sudo git required) +/etc/nixos-repo/ # repo root (a Nix flake; root:wheel, wheel-writable) flake.nix # entry point: nixosConfigurations. per machine flake.lock # pinned nixpkgs / disko configuration.nix # shared NixOS config (edit here for services/packages) diff --git a/configuration.nix b/configuration.nix index 1e2f246..8cf2e27 100644 --- a/configuration.nix +++ b/configuration.nix @@ -2,13 +2,14 @@ # Per-host modules (hardware-configuration.nix or facter.json, optional disko, # local.nix) live under ./hosts//; everything else lives here. # -# Apply: sudo nixos-rebuild switch (packages/services only) -# sudo nixos-rebuild boot + sudo reboot (anything touching desktop/display stack) +# Apply: sudo nixos-rebuild switch --flake path:/etc/nixos-repo (packages/services only) +# sudo nixos-rebuild boot --flake path:/etc/nixos-repo + sudo reboot (desktop/display stack) # # First-time setup and the live-USB install walkthrough are in ./README.md. # This file expects a flake (./flake.nix) to assemble the configuration via -# nixosConfigurations., so `nixos-rebuild switch` should resolve -# through /etc/nixos/flake.nix (symlinked to /etc/nixos-repo/flake.nix). +# nixosConfigurations.. Rebuilds use a `path:` flake ref +# (path:/etc/nixos-repo) rather than the plain git flake so the gitignored +# per-host dir (hosts//, holding secrets) is visible to Nix. # # Per-host local.nix lives at hosts//local.nix and is gitignored. @@ -797,7 +798,7 @@ in # 10. Re-run nixos-rebuild switch to push templates via coder-template-sync echo "--- running nixos-rebuild switch (template sync)" /run/current-system/sw/bin/nixos-rebuild switch \ - --flake /etc/nixos-repo 2>&1 \ + --flake path:/etc/nixos-repo 2>&1 \ | ${pkgs.gnused}/bin/sed 's/^/[coder-reset] /' echo "" diff --git a/hosts/incus-vm/README.md b/hosts/incus-vm/README.md index 4411307..3e054e1 100644 --- a/hosts/incus-vm/README.md +++ b/hosts/incus-vm/README.md @@ -92,11 +92,12 @@ cp /etc/nixos-repo/hosts/incus-vm/incus-vm.nix \ /etc/nixos-repo/hosts/$HOSTNAME/incus-vm.nix # For bare-metal — write your own default.nix or copy from another host. - -# Stage the files — the flake's builtins.readDir only sees tracked files. -git -C /etc/nixos-repo add hosts/$HOSTNAME/ ``` +The host dir is gitignored (it holds per-host secrets) and stays untracked. +Rebuilds below use a `path:` flake ref, which sees gitignored files, so there's +nothing to `git add`. + > **`/etc/nixos/coder.nix`:** The copied `default.nix` does **not** import this > file. It only exists on VMs that are *also* running as a coder-agent workspace > (i.e. the `incus-nixos` template writes it). On a pure box host it won't be @@ -138,9 +139,6 @@ gitignored and must be created manually: ```sh cp /etc/nixos-repo/installer/bootstrap/local.nix.example \ /etc/nixos-repo/hosts/$HOSTNAME/local.nix - -# Mark it so the flake's builtins.readDir can see it without committing it. -git -C /etc/nixos-repo add --intent-to-add -f hosts/$HOSTNAME/local.nix ``` Edit `hosts/$HOSTNAME/local.nix` and at minimum set: @@ -178,7 +176,7 @@ EOF ### 6. Apply ```sh -nixos-rebuild switch --flake /etc/nixos-repo#$(hostname -s) --impure +nixos-rebuild switch --flake path:/etc/nixos-repo#$(hostname -s) --impure ``` `--impure` is required because `/etc/nixos/incus.nix` lives outside the flake @@ -214,5 +212,5 @@ Open that URL in a browser, create the admin user, then log in with the CLI: CODER_URL=http://localhost:3000 coder login http://localhost:3000 ``` -Once logged in, run `sudo nixos-rebuild switch` again to push templates via -`template-sync`. +Once logged in, run `sudo nixos-rebuild switch --flake path:/etc/nixos-repo#$(hostname -s) --impure` +again to push templates via `template-sync`. diff --git a/install.sh b/install.sh index 62f3052..a7f687e 100755 --- a/install.sh +++ b/install.sh @@ -695,23 +695,17 @@ if [[ ! -f "$HOST_DIR/facter.json" ]]; then echo " wrote hosts/$HOSTNAME_ARG/facter.json" fi -# A git path flake ignores untracked files, so the freshly written host files -# must be intent-to-added for the flake to see them (local.nix and -# install-answers.json are gitignored, so force-add them). Only meaningful when -# REPO_DIR is a git repo; the ISO writable copy may have no .git (a non-git path -# flake already sees every file), so skip. -if git -C "$REPO_DIR" rev-parse --is-inside-work-tree >/dev/null 2>&1; then - git -C "$REPO_DIR" add --intent-to-add -f \ - "hosts/$HOSTNAME_ARG/default.nix" \ - "hosts/$HOSTNAME_ARG/facter.json" \ - "hosts/$HOSTNAME_ARG/install-answers.json" \ - "hosts/$HOSTNAME_ARG/local.nix" >/dev/null -fi +# The host dir (hosts/$HOSTNAME_ARG/) is gitignored and stays untracked — it +# holds per-box secrets (install-answers.json, local.nix). Every flake command +# below references the repo as a `path:` flake, which copies the directory +# verbatim (gitignored files included) instead of using the git tree, so the +# host is visible without intent-to-add / force-staging and its secrets never +# enter the git index. # ── Validate ─────────────────────────────────────────────────────────────── echo " validating flake ..." nix --extra-experimental-features 'nix-command flakes' \ - eval "$REPO_DIR#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel.drvPath" \ + eval "path:$REPO_DIR#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel.drvPath" \ >/dev/null # ── Partition + format + mount ───────────────────────────────────────────── @@ -720,7 +714,7 @@ echo "=== Partitioning $DISK_ARG via disko ===" # Use the flake's pinned disko (one nixpkgs source for the whole install). nix --extra-experimental-features 'nix-command flakes' \ run "$REPO_DIR#disko" -- \ - --mode disko --flake "$REPO_DIR#${HOSTNAME_ARG}" + --mode disko --flake "path:$REPO_DIR#${HOSTNAME_ARG}" mountpoint -q /mnt || { echo "disko did not mount /mnt" >&2 @@ -745,8 +739,22 @@ echo "=== Copying repo into /mnt/etc/nixos-repo ===" mkdir -p /mnt/etc/nixos-repo cp -a "$REPO_DIR/." /mnt/etc/nixos-repo/ -# Symlink /etc/nixos/flake.nix so plain `nixos-rebuild switch` finds the -# config after reboot. +# Make the repo writable by the login user (wheel group) so they can `git pull` +# and edit config in place without sudo. Also ensure a `main` branch is checked +# out and tracking origin, so `git pull` works out of the box: a baked image's +# .git is a detached, branchless CI checkout, and a fresh live-USB clone is +# already on main (this is a harmless no-op there). +chown -R root:wheel /mnt/etc/nixos-repo +chmod -R g+w /mnt/etc/nixos-repo +if [[ -d /mnt/etc/nixos-repo/.git ]]; then + git -C /mnt/etc/nixos-repo checkout -B main >/dev/null 2>&1 || true + git -C /mnt/etc/nixos-repo config branch.main.remote origin + git -C /mnt/etc/nixos-repo config branch.main.merge refs/heads/main +fi + +# Symlink /etc/nixos/flake.nix so tooling that looks in /etc/nixos finds the +# flake. Rebuilds must use `--flake path:/etc/nixos-repo` so the gitignored +# per-host dir is visible (a plain git flake would not see it). mkdir -p /mnt/etc/nixos ln -sf /etc/nixos-repo/flake.nix /mnt/etc/nixos/flake.nix @@ -782,7 +790,7 @@ if [[ ${CODER_BOX_FROM_IMAGE:-0} == "1" ]]; then SYSTEM_TOPLEVEL=$(nix --extra-experimental-features 'nix-command flakes' \ build --no-link --print-out-paths \ --option download-buffer-size 268435456 \ - "/mnt/etc/nixos-repo#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel") + "path:/mnt/etc/nixos-repo#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel") [[ -n $SYSTEM_TOPLEVEL ]] || { echo "failed to build system closure" >&2 exit 1 @@ -802,7 +810,7 @@ else echo "=== Running nixos-install ===" echo " (closure builds into /mnt/nix/store; no tmpfs OOM risk)" nixos-install \ - --flake "/mnt/etc/nixos-repo#${HOSTNAME_ARG}" \ + --flake "path:/mnt/etc/nixos-repo#${HOSTNAME_ARG}" \ --no-channel-copy \ --no-root-passwd \ --option download-buffer-size 268435456 @@ -820,7 +828,7 @@ echo " http://${HOSTNAME_ARG}.local:3000 (direct LAN access)" echo " the *.try.coder.app URL itself is written to /etc/motd on first boot once coder.service is up" echo echo "Optional after first login:" -echo " - Update the box: cd /etc/nixos-repo && sudo git pull && sudo nixos-rebuild switch" +echo " - Update the box: cd /etc/nixos-repo && git pull && sudo nixos-rebuild switch --flake path:/etc/nixos-repo" echo if [[ $NO_REBOOT -eq 0 ]]; then