diff --git a/plugins/cozystack/skills/cluster-upgrade/SKILL.md b/plugins/cozystack/skills/cluster-upgrade/SKILL.md index 863f48c..aca9209 100644 --- a/plugins/cozystack/skills/cluster-upgrade/SKILL.md +++ b/plugins/cozystack/skills/cluster-upgrade/SKILL.md @@ -140,6 +140,12 @@ Run both general **and** targeted-per-change checks from Step 1. Show user: result (success/partial/failed), before→after version, HR/Package totals, any warnings, one-line outcome per targeted check. +## Node / Talos OS upgrades + +The steps above cover only the `cozy-installer` Helm upgrade. When a release also moves the **Talos OS** or **Kubernetes** version, that is a separate node-by-node procedure — and on DRBD/LINSTOR-backed clusters the sequencing (one node at a time, wait for each node's DRBD resources to return `UpToDate` before the next) is what keeps VMs alive. Order the layers Talos → Kubernetes → Cozystack, since some fixes ship only in a newer Talos extension bundle (e.g. a DRBD kernel-module fix) and must land first. + +**Full node-upgrade runbook:** read `references/talos-node-upgrade.md`. + ## Rollback `helm --kube-context $CTX rollback cozystack --namespace cozy-system` is possible but has caveats (data migrations don't reverse). Before rolling back, snapshot: `kubectl --context $CTX get packages.cozystack.io -A -o yaml > pre-rollback.yaml`. @@ -162,6 +168,7 @@ High-blast-radius stuck states — stuck helm `uninstalling`, Kamaji datastore c | HR `UninstallFailed, failed to delete release` | Stuck helm history (known-failures #1) | | TCP `INSTALLED VERSION` diverges from `VERSION` | Kamaji upgrade stuck (known-failures #4) | | `cozy-system` namespace gone | Missing `helm.sh/resource-policy=keep` (known-failures #7); restore from backup | +| VM `PausedIOError` loop after a node reboot / `auto-diskful` | DRBD block-size mismatch on 9.2.16, Talos ≤ 1.12 (known-failures #8) | ## Common mistakes @@ -174,7 +181,7 @@ High-blast-radius stuck states — stuck helm `uninstalling`, Kamaji datastore c ## References -- Skill files: `references/release-notes-analysis.md`, `references/preflight-checks.md`, `references/post-upgrade-checks.md`, `references/rollback.md`, `references/known-failures.md` +- Skill files: `references/release-notes-analysis.md`, `references/preflight-checks.md`, `references/post-upgrade-checks.md`, `references/rollback.md`, `references/known-failures.md`, `references/talos-node-upgrade.md` - Upstream (pick the version matching your target from the docs site version selector): `https://cozystack.io/docs//operations/cluster/upgrade/` - Troubleshooting checklist: `https://cozystack.io/docs//operations/troubleshooting/` - Releases: `https://github.com/cozystack/cozystack/releases` diff --git a/plugins/cozystack/skills/cluster-upgrade/references/known-failures.md b/plugins/cozystack/skills/cluster-upgrade/references/known-failures.md index 675412a..52bc739 100644 --- a/plugins/cozystack/skills/cluster-upgrade/references/known-failures.md +++ b/plugins/cozystack/skills/cluster-upgrade/references/known-failures.md @@ -253,6 +253,44 @@ Restore from backup. There is no clean in-cluster recovery for a deleted `cozy-s 3. Re-apply the Platform Package from rescue.yaml (manual review required; CRD schemas may have moved). 4. Expect tenant disruption; communicate to users. +## 8. VM `PausedIOError` loop after LINSTOR `auto-diskful` (DRBD block-size mismatch) + +### Symptom + +A running KubeVirt VM on a DRBD volume flips into `PausedIOError` and loops pause↔unpause every ~3 min. No data loss — all replicas stay `UpToDate`. dmesg on the node shows: + +```text +drbd : setting new queue limits failed +``` + +and the VM's qemu log shows an EINVAL on the disk: + +```text +IO error device='ua-disk-boot' ... reason='Invalid argument' +``` + +Trigger: the VM was running on a node **without** a local replica (DRBD *diskless* + Primary), and LINSTOR `auto-diskful` converted it to *diskful* online (`toggle-disk`) while the VM held the device open. + +### Root cause + +LINSTOR `Linstor/Drbd/auto-block-size` (default-on since LINSTOR 1.33.2) raises the DRBD logical block size to 4096 on the **diskful** side only; the **diskless** side stays 512. qemu opens the device diskless (512) and caches 512-byte O_DIRECT alignment. The `auto-diskful` toggle then changes the device geometry 512→4096 under the open qemu → qemu keeps submitting 512-aligned O_DIRECT (e.g. MariaDB/InnoDB) → the now-4096 device returns EINVAL → `errorPolicy=stop` → pause. + +Underneath, on **DRBD 9.2.16 + kernel 6.x** `queue_limits_commit_update()` fails (the `setting new queue limits failed` line); the freshly-computed limits are discarded and the queue keeps stale values. A diskless device has no backing device whose limits can be stacked in, so the computed 4096 never commits and it stays 512. + +- Affected: Talos ≤ 1.12.x (DRBD 9.2.16), kernel 6.x, LINSTOR ≥ 1.33.2 with `auto-block-size` on. +- Fixed upstream in **DRBD ≥ 9.2.17** ("Correctly request BLK_FEAT_STABLE_WRITES from the kernel" + "always two-phase-commit for attach"). Talos `release-1.13` ships DRBD 9.3.2. The changelog does not name this diskless-block-size path explicitly — validate in a lab before banking on it: a diskless resource with `block-size=4096` should report `logical_block_size=4096` with no `setting new queue limits failed`. + +### Recovery / mitigation + +```bash +# Immediate: reopen the device cleanly (recomputes limits from the current geometry) +virtctl restart -n +``` + +- **Reduce exposure** without an OS upgrade: keep VMs on nodes that hold a replica (volume affinity), or raise the replica count so most nodes have a local copy — fewer diskless opens toggled under a live VM. +- **New volumes only:** set `Linstor/Drbd/auto-block-size=False` before create → diskful and diskless both stay 512 (aligned), no mismatch. Existing volumes already carry the 4096 diskful geometry, so this does **not** fix them. +- **Permanent:** upgrade Talos to 1.13.x (DRBD 9.3.2) — a node-by-node OS upgrade, see `talos-node-upgrade.md`. + ## Diagnostic quick reference | Question | Command | diff --git a/plugins/cozystack/skills/cluster-upgrade/references/talos-node-upgrade.md b/plugins/cozystack/skills/cluster-upgrade/references/talos-node-upgrade.md new file mode 100644 index 0000000..894f181 --- /dev/null +++ b/plugins/cozystack/skills/cluster-upgrade/references/talos-node-upgrade.md @@ -0,0 +1,80 @@ +# Talos / node OS upgrade (DRBD-aware rolling) + +Read this when a Cozystack upgrade also moves the underlying **Talos OS** or **Kubernetes** version — the main `cluster-upgrade` flow only covers the `cozy-installer` Helm upgrade. Node upgrades are a **separate, node-by-node procedure**, and on DRBD/LINSTOR-backed clusters the sequencing is what keeps VMs alive. + +## Where it fits in the upgrade order + +When a release bumps Talos, treat each layer as its own window and run them in this order: + +1. **Talos OS** — one node at a time (this file). +2. **Kubernetes** — per the release notes (control-plane first). +3. **Cozystack platform** — the Helm flow in the main skill. + +Rationale: some fixes ship only in the newer Talos extension bundle (e.g. a DRBD kernel-module fix), so the OS bump has to land first. Talos pins every extension to its release branch — you **cannot** ship a newer DRBD without moving the Talos minor. Check the DRBD version a Talos release carries before planning: `release-1.12` ships DRBD 9.2.16, `release-1.13` ships 9.3.2. + +## The one rule that prevents outages + +**One node at a time — and never start the next node until the just-upgraded node's DRBD resources are back `UpToDate`.** + +DRBD replicates each volume across a small number of nodes (often 2–3). Rebooting a node takes its replicas offline; the volume keeps quorum from the survivors. Reboot a **second** replica-holding node before the first finished resyncing and the volume loses quorum → I/O suspends → every VM on it stalls. The resync wait is not optional. + +## Pre-flight (storage must be clean) + +```bash +alias linstor='kubectl --context $CTX exec -n cozy-linstor deploy/linstor-controller -ti -- linstor' +linstor node list # every node Online +linstor resource list --faulty # empty +linstor storage-pool list # no Err +# no lost-quorum taints stuck on any node: +kubectl --context $CTX get nodes -o json \ + | jq -r '.items[] | select(.spec.taints[]?.key=="drbd.linbit.com/lost-quorum") | .metadata.name' +``` + +Any faulty resource, pending resync, or lost-quorum taint blocks the upgrade — fix it first (the `linstor:recover` skill covers the recovery paths). + +## Per-node loop + +Do control-plane nodes first (one at a time, so etcd quorum holds), then storage/worker nodes one at a time. + +```bash +NODE= + +# 1. Cordon +kubectl --context $CTX cordon "$NODE" + +# 2. Drain — live-migrate VMs off, respect PDBs. +# KubeVirt VMs with evictionStrategy=LiveMigrate migrate on drain; ones without +# it are stopped instead, so check before draining a node that hosts VMs. +kubectl --context $CTX get vmi -A --field-selector status.nodeName="$NODE" +kubectl --context $CTX drain "$NODE" --ignore-daemonsets --delete-emptydir-data + +# 3. Upgrade Talos. talm resolves the installer image from values.yaml::image — +# bump that to the target release first, then: +talm upgrade -f nodes/"$NODE".yaml +# talm's post-upgrade verify gate confirms the node booted the new version +# (catches the silent A/B rollback case). Do NOT pass --skip-post-upgrade-verify. + +# 4. Wait for the node to come back Ready +kubectl --context $CTX wait --for=condition=Ready node/"$NODE" --timeout=15m + +# 5. WAIT FOR DRBD RESYNC on this node before touching the next one. +# All resources back UpToDate, nothing SyncTarget / Inconsistent / Connecting: +linstor resource list --faulty # must be empty again +linstor exec -- drbdsetup status | grep -E 'SyncTarget|Inconsistent|Connecting' || echo clean + +# 6. Uncordon +kubectl --context $CTX uncordon "$NODE" +``` + +Only after step 5 reports clean do you move to the next node. Large volumes can take a while to resync — that wait is the whole point. + +## After a node reboots: watch for the auto-diskful pause + +A VM that restarts onto a node **without a local replica** opens its disk *diskless*, and LINSTOR `auto-diskful` may then convert it to *diskful* under the running VM. On DRBD 9.2.16 (Talos ≤ 1.12) that online toggle can leave the VM in a `PausedIOError` loop — see `known-failures.md` #8. Two ways to shrink the exposure during a node-upgrade window: + +- Prefer scheduling VMs onto nodes that already hold a replica (volume affinity), or run enough replicas that most nodes have a local copy — fewer diskless opens, fewer live toggles. +- If a VM does get stuck, `virtctl restart ` reopens the device cleanly. + +## Rollback + +Talos upgrades are A/B: a failed boot rolls back to the previous partition automatically, and talm's post-upgrade verify catches the silent-rollback case. There is no "roll the whole cluster back" step — you fix the one node and re-run. Never proceed to the next node while any node is on the wrong version or any resource is still resyncing.