fix(cilium): keep spire-server off the Flux-controller node (soft anti-affinity)#1660
Closed
devantler wants to merge 1 commit into
Closed
fix(cilium): keep spire-server off the Flux-controller node (soft anti-affinity)#1660devantler wants to merge 1 commit into
devantler wants to merge 1 commit into
Conversation
…i-affinity) spire-server is a single replica and the cluster's identity root: if its node fails, every spire-agent loses its upstream (spire-server ClusterIP -> i/o timeout) and Cilium mutual auth degrades cluster-wide. On 2026-05-28 spire-server shared prod-worker-2 with kustomize-controller; when that node's Cilium ClusterIP datapath degraded after an OOMKill, workload identity AND GitOps reconciliation went down together — and reconciliation was exactly what was needed to apply the fix, so the cluster could not self-heal. Add a soft (preferred) podAntiAffinity so spire-server prefers a worker without app.kubernetes.io/part-of=flux pods, decorrelating the identity SPOF from the GitOps controllers. Soft so the single replica always schedules even when every node hosts a Flux pod. Verified the Cilium 1.19.4 chart renders authentication.mutual.spire.install.server.affinity into the StatefulSet. SPIRE is disabled in the Docker overlay, so this is prod-only and inert for local/CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a soft podAntiAffinity to the SPIRE server (deployed via Cilium's mutual auth chart values) so it prefers nodes not running Flux controllers, decorrelating the workload-identity SPOF from the GitOps control plane.
Changes:
- Inject
affinity.podAntiAffinity(preferred, weight 100) onauthentication.mutual.spire.install.serverkeyed onapp.kubernetes.io/part-of=flux.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
spire-serveris a single replica and the cluster's identity root. If its node fails, everyspire-agentloses its upstream:…all six
spire-agentpods crash-loop and Cilium mutual auth degrades cluster-wide.During the 2026-05-28/29 incident,
spire-serversharedprod-worker-2withkustomize-controller. When that node's Cilium ClusterIP datapath degraded after an OOMKill, workload identity and GitOps reconciliation went down together — and reconciliation was exactly what was needed to apply the fix (#1649). One node loss took out two critical subsystems at once, which is what made recovery a deadlock.Fix
Add a soft (
preferredDuringSchedulingIgnoredDuringExecution, weight 100)podAntiAffinitysospire-serverprefers a worker that is not running the Flux controllers (app.kubernetes.io/part-of=flux), decorrelating the identity SPOF from the GitOps control plane.Soft, so the single replica always schedules even when every node hosts a Flux pod — never risks leaving the cluster with no identity. Pairs with #1659 (which spreads the Flux controllers): together they push identity and reconciliation onto different workers.
Validation
authentication.mutual.spire.install.server.affinityinto the spire-server StatefulSet (templates/spire/server/statefulset.yamlline 114), so the value takes effect (not a silent no-op).kubectl kustomizeof the base cilium dir renders thepodAntiAffinity; the docker controllers overlay still builds withspire.enabled: false(prod-only, inert locally — no merge conflict).k8s/clusters/local/andk8s/clusters/prod/build.Scope
Preventative (decorrelation / blast-radius reduction), placed in the base alongside the SPIRE server config added in #1649. Does not resolve the active outage on its own — that needs
prod-worker-2's datapath rebuilt so reconciliation recovers.