fix(ci): raise local infrastructure Flux timeout to 20m for vault-config bootstrap

[devantler]

🤖 Generated by the Daily AI Assistant

Problem

The base infrastructure Flux Kustomization at k8s/bases/cluster/infrastructure-flux-kustomization.yaml has timeout: 3m, wait: true. One of its child resources is Job/openbao/vault-config (defined at k8s/bases/infrastructure/vault-config/job.yaml) whose own settings include backoffLimit: 30 and activeDeadlineSeconds: 3600 with the explicit comment that "openbao bootstrap can take 20-40 min on cold clusters".

On cold Docker CI runners the System Test sporadically fails with:

Kustomization/flux-system/infrastructure — health check failed after 3m0.432401941s:
  timeout waiting for: [Job/openbao/vault-config status: 'InProgress'] (HealthCheckFailed)

(observed on PR #1636 system-test run 26603473269; that PR only modified homepage/headlamp/actual-budget HelmReleases — unrelated to the failure.)

Dependency model — who waits on vault-config?

The vault-config Job bootstraps OpenBao: enables the KV v2 engine, configures Kubernetes auth, writes policies, and creates the external-secrets and fleetdm-mysql-rotated auth roles. Everything that consumes the openbao ClusterSecretStore (k8s/bases/infrastructure/cluster-secret-stores/cluster-secret-store.yaml) depends on that role existing — concretely the vault-seed PushSecrets (k8s/bases/infrastructure/vault-seed/) and every OpenBao-backed ExternalSecret under k8s/bases/infrastructure/external-secrets/ and k8s/bases/infrastructure/vault-config/external-secret.yaml. All of these live in the same infrastructure Flux Kustomization, so they share one health-check budget.

Options considered

1. Raise infrastructure Kustomization timeout to accommodate the Job's worst-case cold start.
2. Split vault-config into its own Flux Kustomization with a longer timeout, leaving the rest of infrastructure on the tighter 3m budget.
3. Reduce vault-config bootstrap time — explicitly rejected: activeDeadlineSeconds: 3600 was set generously on purpose.

Chose option 1, applied as a per-cluster patch. It follows the established overlay-patch convention in this repo:

• Prod (k8s/clusters/prod/kustomization.yaml:32-39) already patches infrastructure→20m with the same rationale ("3m base timeout" insufficient for heavier bootstrap components).
• Local (k8s/clusters/local/kustomization.yaml) already patches apps→20m and infrastructure-controllers→12m for the same cold-Docker-CI reason; this PR adds the missing peer patch for infrastructure.

Option 2 would mean per-provider vault-config overlays (Docker + Hetzner), a new Flux Kustomization, and rewired dependsOn edges — significant overlay surface for one Job, when a single timeout knob expresses the same intent and prod has been running with that value reliably.

Change

A patch on the local cluster overlay setting infrastructure timeout to 20m, with a comment linking the patch to the Job's activeDeadlineSeconds and the observed CI failure.

retryInterval: 2m is unchanged — a genuinely broken resource still surfaces on the next retry, only the initial wait window expands. The Job is idempotent (the kustomize.toolkit.fluxcd.io/force: enabled annotation triggers Flux to delete-and-recreate on spec changes; the script checks state before every write), so longer waits are safe.

Validation

Static checks only — does not require a cluster (per AGENTS.md):

$ ksail workload validate
✔ 256 files validated

$ ksail --config ksail.prod.yaml workload validate
✔ 256 files validated

Rendered Flux Kustomization timeouts after the change:

Layer	local (before)	local (after)	prod
variables	3m	3m	3m
infrastructure-controllers	12m	12m	25m
infrastructure	3m	20m	20m
apps	20m	20m	20m

The full system-test will run in CI on this PR.

Root cause vs symptom

This patch addresses the symptom (health-check timeout too tight). The underlying root cause — cold-start latency for vault-config — is intrinsic to bootstrapping OpenBao from zero (waiting for the OpenBao Pod to schedule + image to pull + server to listen, then a fast idempotent script). The Job is already designed around that with activeDeadlineSeconds: 3600 and backoffLimit: 30; the Flux Kustomization just needs to match the realistic wait window.

🤖 Generated with Claude Code

[devantler]

Closing while investigating whether the underlying slowness is fixable instead of worked around.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.