Skip to content

refactor(installer): path: flake so the installed host stays gitignored + writable repo #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
phorcys420 wants to merge 1 commit into
base: main
Choose a base branch
from
+73 −61
Draft

refactor(installer): path: flake so the installed host stays gitignored + writable repo #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 16 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,12 @@ NixOS configuration for Coder demo and workshop boxes.
This repo is a Nix flake. `flake.nix` auto-discovers every subdirectory of
`./hosts/` that contains a `default.nix` and exposes it as
`nixosConfigurations.<folder-name>`. For normal install hosts the folder name
is also the hostname, so `nixos-rebuild switch --flake .` auto-selects the
right config on the running box. Adding a new host means creating a host
folder, no flake.nix edit. The installer does this for you.
is also the hostname, so `nixos-rebuild switch --flake path:/etc/nixos-repo`
auto-selects the right config on the running box. Adding a new host means
creating a host folder, no flake.nix edit. The installer does this for you.
(The per-host dir is gitignored, so the box uses a `path:` flake ref — which
copies the tree verbatim, gitignored files included — rather than a git flake,
which would only see tracked files. See [Applying changes](#applying-changes).)

Hosts whose folder name starts with an underscore (`_appliance-iso`,
`_appliance-disk`, `_installer-iso`) are image builds, not per-machine installs: they
Expand Down Expand Up @@ -61,7 +64,7 @@ sudo ./install.sh \

`./install.sh --help` lists everything. `--coder-admin-password-file PATH` and `--nixos-password-file PATH` read passwords from a file so they don't end up in shell history. `--no-reboot` skips the automatic reboot at the end.

The installer generates `hosts/<hostname>/{default.nix,local.nix,install-answers.json,facter.json}`, copies the repo into `/etc/nixos-repo` on the target, and symlinks `/etc/nixos/flake.nix`. After reboot, `nixos-rebuild switch` Just Works. Continue with [After install](#after-install).
The installer generates `hosts/<hostname>/{default.nix,local.nix,install-answers.json,facter.json}`, copies the repo into `/etc/nixos-repo` on the target (owned `root:wheel` and checked out on `main`, so wheel users can `git pull` / edit it without sudo), and symlinks `/etc/nixos/flake.nix`. Rebuild with `sudo nixos-rebuild switch --flake path:/etc/nixos-repo`. Continue with [After install](#after-install).

> **Different partition layout?** Don't import `installer/bootstrap/disko-standard.nix`; drop your own disko config into the host folder instead. See [disko examples](https://github.com/nix-community/disko/tree/master/example).

Expand Down Expand Up @@ -196,26 +199,24 @@ boot via `coder-init-admin.service`. After the reboot:
defaults.

Subsequent edits to `coderd/` templates go out via `coder-template-sync`
on every `sudo nixos-rebuild switch`.
on every `sudo nixos-rebuild switch --flake path:/etc/nixos-repo`.

## Applying changes

The per-host dir (`hosts/<host>/`) is gitignored and stays untracked, so the
box uses a `path:` flake ref (a plain git flake would not see it). The repo is
owned `root:wheel`, so wheel users edit it and `git pull` without sudo.

```sh
sudo nixos-rebuild switch # most changes
sudo nixos-rebuild boot && sudo reboot # changes that touch the desktop stack

# Edited hosts/<host>/local.nix, install-answers.json, or facter.json? Re-mark intent-to-add:
sudo git -C /etc/nixos-repo add --intent-to-add -f \
hosts/<host>/local.nix \
hosts/<host>/install-answers.json \
hosts/<host>/facter.json
sudo nixos-rebuild switch --flake path:/etc/nixos-repo # most changes
sudo nixos-rebuild boot --flake path:/etc/nixos-repo && sudo reboot # desktop-stack changes
```

## Updating nixpkgs / disko / facter

```sh
sudo nix flake update --flake /etc/nixos-repo
sudo nixos-rebuild switch
sudo nixos-rebuild switch --flake path:/etc/nixos-repo
```

This bumps `flake.lock` to the latest of each input.
Expand Down Expand Up @@ -274,7 +275,7 @@ Fully automated, no follow-up steps needed. The service:
### Changing the admin password

1. Edit `hosts/<host>/install-answers.json`, update `initialUser.password` (or override `services.coder-nixos.initialUser.password` in `local.nix`).
2. Run `sudo nixos-rebuild switch` to bake the new password into the service.
2. Run `sudo nixos-rebuild switch --flake path:/etc/nixos-repo` to bake the new password into the service.
3. Run `sudo systemctl start coder-reset` to wipe and re-bootstrap with the new password.

> If you need to change the password on a **live** deployment without a full wipe:
Expand Down
30 changes: 17 additions & 13 deletions agents.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Reference for AI coding agents and humans working on this repo.
|---|---|
| SSH | `ssh -i ~/.ssh/id_ed25519 coderbox@<TAILSCALE_IP>` |
| Repo path | `/etc/nixos-repo/` (a Nix flake; `nixosConfigurations.<hostname>`) |
| Git ops | `sudo git -C /etc/nixos-repo <command>` |
| Git ops | `git -C /etc/nixos-repo <command>` (repo is `root:wheel`, wheel-writable) |
| Coder URL | `http://coder-thinkcentre.local:3000` |
| Coder token | stored in `/etc/coder/session-token` |
| Coder binary | `coder` (in PATH via NixOS; resolves from nix store) |
Expand All @@ -27,16 +27,19 @@ sudo k3s kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml ...

```sh
# Service, package, or config changes — safe, non-destructive:
cd /etc/nixos-repo && sudo nixos-rebuild switch --flake /etc/nixos-repo
cd /etc/nixos-repo && sudo nixos-rebuild switch --flake path:/etc/nixos-repo

# Desktop stack (GNOME, GDM, Wayland) — must reboot:
cd /etc/nixos-repo && sudo nixos-rebuild boot --flake /etc/nixos-repo && sudo reboot
cd /etc/nixos-repo && sudo nixos-rebuild boot --flake path:/etc/nixos-repo && sudo reboot
```

The repo is baked onto the box at **`/etc/nixos-repo`** (the canonical flake;
`nixosConfigurations.<hostname>`, auto-selected by the running hostname). Edit
files there, then rebuild. Always pass `--flake /etc/nixos-repo` (or `cd` into
it and use `--flake .`) — see the `/etc/nixos` pitfall below.
files there, then rebuild. Always pass `--flake path:/etc/nixos-repo`. The
per-host dir (`hosts/<host>/`) is gitignored and untracked, so a `path:` flake
ref is required — a plain git flake (`--flake /etc/nixos-repo` or `--flake .`)
only sees tracked files and would not find the host. See the `/etc/nixos`
pitfall below.

`nixos-rebuild switch` triggers the `coder-template-sync` activation script, which runs `terraform apply` in `coderd/` and pushes any template changes to Coder. The `/etc/coder/session-token` it needs is populated automatically by `coder-init-admin.service` on first boot, so this just works post-install.

Expand Down Expand Up @@ -73,13 +76,14 @@ if something is off. Run `make fmt` to fix formatting locally.

## Git Workflow

All files in `/etc/nixos-repo/` are root-owned. Use `sudo git`:
All files in `/etc/nixos-repo/` are owned `root:wheel` and group-writable, so
wheel users run git without sudo:

```sh
cd /etc/nixos-repo
sudo git status
sudo git add -p
sudo git commit -m "feat: describe your change"
git status
git add -p
git commit -m "feat: describe your change"
# Don't push unless explicitly asked
```

Expand Down Expand Up @@ -122,7 +126,7 @@ sudo terraform apply \
-var="coder_url=http://localhost:3000" \
-var="coder_session_token=$(sudo cat /etc/coder/session-token)" \
-var="hostname=coder-thinkcentre" \
-var="version_name=$(sudo git -C /etc/nixos-repo rev-parse --short HEAD)"
-var="version_name=$(git -C /etc/nixos-repo rev-parse --short HEAD)"
```

## nook-android Template
Expand Down Expand Up @@ -197,8 +201,8 @@ sudo k3s kubectl describe pod -n coder-workspaces <pod-name>
- **Tailscale auth doesn't re-run** — `tailscale-autoauth` has `RemainAfterExit = true`. If you change auth key config, run `sudo systemctl restart tailscale-autoauth`.
- **Template sync skips**, if `/etc/coder/session-token` is empty, the activation script exits cleanly. The token is auto-populated by `coder-init-admin.service`; if it's missing, check `journalctl -u coder-init-admin`.
- **`coder` binary path** — the binary is in PATH via NixOS environment; don't hardcode nix store paths in scripts (they change with every package update).
- **`--flake /etc/nixos` fails** — `/etc/nixos` is a plain dir holding only a `flake.nix` *symlink* into `/etc/nixos-repo`. Nix follows the symlink into the store but can't find the sibling files (configuration.nix, hosts/, nixos/), dying with `path '/nix/store/...-source/etc/nixos-repo/flake.nix' does not exist`. Always rebuild against the real tree: `--flake /etc/nixos-repo` (or `cd /etc/nixos-repo && nixos-rebuild switch --flake .`).
- **`Git tree '/etc/nixos-repo' is dirty` warning** — harmless. `hosts/<host>/{local.nix,install-answers.json,facter.json}` are gitignored and intent-to-added by the installer, so the tree always reads "dirty". After editing them, re-mark intent-to-add so the flake sees them: `sudo git -C /etc/nixos-repo add --intent-to-add -f hosts/<host>/local.nix hosts/<host>/install-answers.json hosts/<host>/facter.json`.
- **`--flake /etc/nixos` fails** — `/etc/nixos` is a plain dir holding only a `flake.nix` *symlink* into `/etc/nixos-repo`. Nix follows the symlink into the store but can't find the sibling files (configuration.nix, hosts/, nixos/), dying with `path '/nix/store/...-source/etc/nixos-repo/flake.nix' does not exist`. Always rebuild against the real tree with a `path:` ref: `--flake path:/etc/nixos-repo`.
- **Host not found / `does not provide attribute 'nixosConfigurations.<host>'`** — the per-host dir is gitignored and untracked, so a git flake (`--flake /etc/nixos-repo` or `--flake .`) can't see it. Use `--flake path:/etc/nixos-repo`, which copies the tree verbatim (gitignored files included). This is also why nothing intent-adds the host and `git status` stays clean.
- **ScreenConnect blank screen** — the box now runs GNOME on Wayland (GDM), and GNOME 49 dropped the Xorg session, so there is no X11 desktop to fall back to. ScreenConnect reaches `DISPLAY=:0` through XWayland (see `nixos/screenconnect.nix`) but **cannot screen-capture the Wayland compositor** through it, so the remote view may be black/blank. Capturing the GNOME session needs a Wayland-aware path (PipeWire/portal, e.g. `gnome-remote-desktop`); the X11 agent will connect but not mirror the desktop.

## Wildcard App Access (TODO)
Expand All @@ -212,7 +216,7 @@ sudo k3s kubectl describe pod -n coder-workspaces <pod-name>
## File Layout (agent-relevant paths)

```
/etc/nixos-repo/ # repo root (a Nix flake; sudo git required)
/etc/nixos-repo/ # repo root (a Nix flake; root:wheel, wheel-writable)
flake.nix # entry point: nixosConfigurations.<host> per machine
flake.lock # pinned nixpkgs / disko
configuration.nix # shared NixOS config (edit here for services/packages)
Expand Down
11 changes: 6 additions & 5 deletions configuration.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,14 @@
# Per-host modules (hardware-configuration.nix or facter.json, optional disko,
# local.nix) live under ./hosts/<host>/; everything else lives here.
#
# Apply: sudo nixos-rebuild switch (packages/services only)
# sudo nixos-rebuild boot + sudo reboot (anything touching desktop/display stack)
# Apply: sudo nixos-rebuild switch --flake path:/etc/nixos-repo (packages/services only)
# sudo nixos-rebuild boot --flake path:/etc/nixos-repo + sudo reboot (desktop/display stack)
#
# First-time setup and the live-USB install walkthrough are in ./README.md.
# This file expects a flake (./flake.nix) to assemble the configuration via
# nixosConfigurations.<hostname>, so `nixos-rebuild switch` should resolve
# through /etc/nixos/flake.nix (symlinked to /etc/nixos-repo/flake.nix).
# nixosConfigurations.<hostname>. Rebuilds use a `path:` flake ref
# (path:/etc/nixos-repo) rather than the plain git flake so the gitignored
# per-host dir (hosts/<host>/, holding secrets) is visible to Nix.
#
# Per-host local.nix lives at hosts/<host>/local.nix and is gitignored.

Expand Down Expand Up @@ -797,7 +798,7 @@ in
# 10. Re-run nixos-rebuild switch to push templates via coder-template-sync
echo "--- running nixos-rebuild switch (template sync)"
/run/current-system/sw/bin/nixos-rebuild switch \
--flake /etc/nixos-repo 2>&1 \
--flake path:/etc/nixos-repo 2>&1 \
| ${pkgs.gnused}/bin/sed 's/^/[coder-reset] /'

echo ""
Expand Down
16 changes: 7 additions & 9 deletions hosts/incus-vm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,11 +92,12 @@ cp /etc/nixos-repo/hosts/incus-vm/incus-vm.nix \
/etc/nixos-repo/hosts/$HOSTNAME/incus-vm.nix

# For bare-metal — write your own default.nix or copy from another host.

# Stage the files — the flake's builtins.readDir only sees tracked files.
git -C /etc/nixos-repo add hosts/$HOSTNAME/
```

The host dir is gitignored (it holds per-host secrets) and stays untracked.
Rebuilds below use a `path:` flake ref, which sees gitignored files, so there's
nothing to `git add`.

> **`/etc/nixos/coder.nix`:** The copied `default.nix` does **not** import this
> file. It only exists on VMs that are *also* running as a coder-agent workspace
> (i.e. the `incus-nixos` template writes it). On a pure box host it won't be
Expand Down Expand Up @@ -138,9 +139,6 @@ gitignored and must be created manually:
```sh
cp /etc/nixos-repo/installer/bootstrap/local.nix.example \
/etc/nixos-repo/hosts/$HOSTNAME/local.nix

# Mark it so the flake's builtins.readDir can see it without committing it.
git -C /etc/nixos-repo add --intent-to-add -f hosts/$HOSTNAME/local.nix
```

Edit `hosts/$HOSTNAME/local.nix` and at minimum set:
Expand Down Expand Up @@ -178,7 +176,7 @@ EOF
### 6. Apply

```sh
nixos-rebuild switch --flake /etc/nixos-repo#$(hostname -s) --impure
nixos-rebuild switch --flake path:/etc/nixos-repo#$(hostname -s) --impure
```

`--impure` is required because `/etc/nixos/incus.nix` lives outside the flake
Expand Down Expand Up @@ -214,5 +212,5 @@ Open that URL in a browser, create the admin user, then log in with the CLI:
CODER_URL=http://localhost:3000 coder login http://localhost:3000
```

Once logged in, run `sudo nixos-rebuild switch` again to push templates via
`template-sync`.
Once logged in, run `sudo nixos-rebuild switch --flake path:/etc/nixos-repo#$(hostname -s) --impure`
again to push templates via `template-sync`.
46 changes: 27 additions & 19 deletions install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -695,23 +695,17 @@ if [[ ! -f "$HOST_DIR/facter.json" ]]; then
echo " wrote hosts/$HOSTNAME_ARG/facter.json"
fi

# A git path flake ignores untracked files, so the freshly written host files
# must be intent-to-added for the flake to see them (local.nix and
# install-answers.json are gitignored, so force-add them). Only meaningful when
# REPO_DIR is a git repo; the ISO writable copy may have no .git (a non-git path
# flake already sees every file), so skip.
if git -C "$REPO_DIR" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
git -C "$REPO_DIR" add --intent-to-add -f \
"hosts/$HOSTNAME_ARG/default.nix" \
"hosts/$HOSTNAME_ARG/facter.json" \
"hosts/$HOSTNAME_ARG/install-answers.json" \
"hosts/$HOSTNAME_ARG/local.nix" >/dev/null
fi
# The host dir (hosts/$HOSTNAME_ARG/) is gitignored and stays untracked — it
# holds per-box secrets (install-answers.json, local.nix). Every flake command
# below references the repo as a `path:` flake, which copies the directory
# verbatim (gitignored files included) instead of using the git tree, so the
# host is visible without intent-to-add / force-staging and its secrets never
# enter the git index.

# ── Validate ───────────────────────────────────────────────────────────────
echo " validating flake ..."
nix --extra-experimental-features 'nix-command flakes' \
eval "$REPO_DIR#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel.drvPath" \
eval "path:$REPO_DIR#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel.drvPath" \
>/dev/null

# ── Partition + format + mount ─────────────────────────────────────────────
Expand All @@ -720,7 +714,7 @@ echo "=== Partitioning $DISK_ARG via disko ==="
# Use the flake's pinned disko (one nixpkgs source for the whole install).
nix --extra-experimental-features 'nix-command flakes' \
run "$REPO_DIR#disko" -- \
--mode disko --flake "$REPO_DIR#${HOSTNAME_ARG}"
--mode disko --flake "path:$REPO_DIR#${HOSTNAME_ARG}"

mountpoint -q /mnt || {
echo "disko did not mount /mnt" >&2
Expand All @@ -745,8 +739,22 @@ echo "=== Copying repo into /mnt/etc/nixos-repo ==="
mkdir -p /mnt/etc/nixos-repo
cp -a "$REPO_DIR/." /mnt/etc/nixos-repo/

# Symlink /etc/nixos/flake.nix so plain `nixos-rebuild switch` finds the
# config after reboot.
# Make the repo writable by the login user (wheel group) so they can `git pull`
# and edit config in place without sudo. Also ensure a `main` branch is checked
# out and tracking origin, so `git pull` works out of the box: a baked image's
# .git is a detached, branchless CI checkout, and a fresh live-USB clone is
# already on main (this is a harmless no-op there).
chown -R root:wheel /mnt/etc/nixos-repo
chmod -R g+w /mnt/etc/nixos-repo
if [[ -d /mnt/etc/nixos-repo/.git ]]; then
git -C /mnt/etc/nixos-repo checkout -B main >/dev/null 2>&1 || true
git -C /mnt/etc/nixos-repo config branch.main.remote origin
git -C /mnt/etc/nixos-repo config branch.main.merge refs/heads/main
fi

# Symlink /etc/nixos/flake.nix so tooling that looks in /etc/nixos finds the
# flake. Rebuilds must use `--flake path:/etc/nixos-repo` so the gitignored
# per-host dir is visible (a plain git flake would not see it).
mkdir -p /mnt/etc/nixos
ln -sf /etc/nixos-repo/flake.nix /mnt/etc/nixos/flake.nix

Expand Down Expand Up @@ -782,7 +790,7 @@ if [[ ${CODER_BOX_FROM_IMAGE:-0} == "1" ]]; then
SYSTEM_TOPLEVEL=$(nix --extra-experimental-features 'nix-command flakes' \
build --no-link --print-out-paths \
--option download-buffer-size 268435456 \
"/mnt/etc/nixos-repo#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel")
"path:/mnt/etc/nixos-repo#nixosConfigurations.${HOSTNAME_ARG}.config.system.build.toplevel")
[[ -n $SYSTEM_TOPLEVEL ]] || {
echo "failed to build system closure" >&2
exit 1
Expand All @@ -802,7 +810,7 @@ else
echo "=== Running nixos-install ==="
echo " (closure builds into /mnt/nix/store; no tmpfs OOM risk)"
nixos-install \
--flake "/mnt/etc/nixos-repo#${HOSTNAME_ARG}" \
--flake "path:/mnt/etc/nixos-repo#${HOSTNAME_ARG}" \
--no-channel-copy \
--no-root-passwd \
--option download-buffer-size 268435456
Expand All @@ -820,7 +828,7 @@ echo " http://${HOSTNAME_ARG}.local:3000 (direct LAN access)"
echo " the *.try.coder.app URL itself is written to /etc/motd on first boot once coder.service is up"
echo
echo "Optional after first login:"
echo " - Update the box: cd /etc/nixos-repo && sudo git pull && sudo nixos-rebuild switch"
echo " - Update the box: cd /etc/nixos-repo && git pull && sudo nixos-rebuild switch --flake path:/etc/nixos-repo"
echo

if [[ $NO_REBOOT -eq 0 ]]; then
Expand Down
Loading