crashloop-operator

A Kubernetes Operator that watches pods for terminal failure states and scales down (or suspends) the owning workload after configurable thresholds are exceeded. Prevents resource waste and alert fatigue from permanently broken deployments.

• 🔄 CrashLoopBackOff Detection - Catches pods stuck in CrashLoopBackOff, ImagePullBackOff, ErrImagePull, and other terminal states
• 📉 Workload Scale-Down - Scales Deployments and StatefulSets to zero, suspends CronJobs
• ⏱️ Configurable Thresholds - Restart count and duration thresholds before action is taken
• 🔍 All-Replicas Check - Only acts when all replicas of a workload are failing (configurable)
• 🛡️ Namespace Filtering - Label-based namespace selector and explicit exclusion list
• 🚫 Workload Exclusion - Exclude workloads via label selector
• 🧪 Dry Run Mode - Log what would happen without actually scaling down
• 📢 Kubernetes Events - Emits events explaining why a workload was scaled down
• 🏷️ Annotations - Records reason, timestamp, and previous replica count on scaled-down workloads

Status: Early Development - This project is experimental and under active development. CRDs, APIs, and behavior may change at any time. Feedback is welcome via issues.

Why?

Kubernetes has no built-in mechanism to stop retrying workloads that are permanently broken. Pods stuck in CrashLoopBackOff, ImagePullBackOff, or similar terminal states keep consuming resources and generating noise indefinitely.

The crashloop-operator watches for these failure patterns and scales down the owning workload after configurable thresholds, preserving the previous replica count in an annotation for easy recovery.

Architecture

graph TD
    Policy["CrashLoopPolicy"]
    Policy -->|watches| Pods["Pods (all namespaces)"]
    Pods -->|failing pod detected| Check["Threshold Check<br/>restarts >= N OR duration >= T"]
    Check -->|resolve owner| Owner["Owner Resolution<br/>Pod -> RS/Job -> Deploy/STS/CronJob"]
    Owner -->|all replicas failing?| Scale["Scale Down / Suspend"]
    Scale --> Deploy["Deployment<br/>replicas: 0"]
    Scale --> STS["StatefulSet<br/>replicas: 0"]
    Scale --> CJ["CronJob<br/>suspend: true"]

Loading

The controller reconciles on a configurable interval (default: 60s) and on pod events. For each failing pod, it:

1. Checks if the failure reason matches the watch list
2. Verifies restart count or duration exceeds the threshold
3. Resolves the owner chain (Pod -> ReplicaSet/Job -> Deployment/StatefulSet/CronJob)
4. Optionally checks if ALL replicas are failing
5. Scales down the workload and annotates it with the reason

CRD

The operator introduces a single CRD: CrashLoopPolicy (crashloop-operator.lauger.de/v1alpha1).

CrashLoopPolicy is cluster-scoped. A policy evaluates workloads across the whole cluster, so it is an administrative resource: grant write access to it the same way you would grant any other cluster-wide permission. Use namespaceSelector and excludeNamespaces to limit which namespaces a policy acts on, and the chart's scope.mode to confine the operator itself to a single namespace.

Short name: clp (kubectl get clp).

Spec

Field	Default	Description
watchReasons	[CrashLoopBackOff, ImagePullBackOff, ErrImagePull, CreateContainerConfigError, InvalidImageName, RunContainerError]	Container waiting reasons to watch. Termination reasons such as OOMKilled never appear here; use watchTerminationReasons for those
watchTerminationReasons	[]	Container termination reasons to act on, for containers that restart repeatedly without settling into a watched waiting state. Off by default
restartWindow	1h	How recently the last termination must have happened for watchTerminationReasons to match
restartThreshold	10	Number of container restarts before action
durationThreshold	30m	How long a pod must have been continuously not ready before action. Go duration format, rejected by the API server if malformed
allReplicasFailing	true	Require all replicas to be failing
failureCorrelation	Pod	How replicas are compared when allReplicasFailing is true. Container additionally requires the same container name to be the failing one everywhere
targets	[Deployment, StatefulSet, CronJob]	Workload types to act on. Only these three values are accepted
namespaceSelector	nil	Label selector for namespaces to watch (nil = all)
excludeNamespaces	[kube-system, kube-public, kube-node-lease]	Namespaces to ignore (applied after namespaceSelector)
excludeWorkloadSelector	nil	Label selector to exclude matching workloads from scale-down
reconcileInterval	60s	Maximum time between policy evaluations. A pod nearing durationThreshold is evaluated when that threshold expires. Same duration format as durationThreshold
dryRun	false	Log actions without executing them

Status

Field	Description
phase	Pending before the first evaluation, Active afterwards
observedGeneration	Generation of the spec that was last evaluated
conditions	Ready reports whether the last evaluation succeeded. Degraded is true while any workload is held scaled down
scaledDownWorkloads	Lifetime counter of scale-down actions performed by this policy
activeScaledDown	Workloads currently held at zero replicas (or suspended), as kind/namespace/name entries. Capped at 1000
activeScaledDownTruncated	How many entries were omitted from activeScaledDown because of that cap
lastEvaluationTime	When an evaluation last changed the status. Unchanged evaluations do not write status, to avoid churning the object every interval

Quick Start

1. Install the Operator

helm install crashloop-operator \
  oci://ghcr.io/slauger/charts/crashloop-operator \
  --namespace crashloop-system \
  --create-namespace

2. Create a Policy

apiVersion: crashloop-operator.lauger.de/v1alpha1
kind: CrashLoopPolicy
metadata:
  name: default
spec:
  restartThreshold: 10
  durationThreshold: "30m"
  allReplicasFailing: true
  # namespaceSelector:
  #   matchLabels:
  #     env: production
  excludeNamespaces:
    - kube-system
    - kube-public
    - kube-node-lease

3. Recover a Scaled-Down Workload

When the operator scales down a workload, it stores the previous replica count in an annotation:

# Check what happened
kubectl get deployment my-app -o jsonpath='{.metadata.annotations}'

# Restore the original replica count
kubectl scale deployment my-app --replicas=$(kubectl get deployment my-app -o jsonpath='{.metadata.annotations.crashloop-operator\.lauger\.de/previous-replicas}')

Excluding Workloads

To exclude workloads from being scaled down, use excludeWorkloadSelector to match workload labels. For example, to exclude all workloads managed by ArgoCD:

spec:
  excludeWorkloadSelector:
    matchLabels:
      argocd.argoproj.io/instance: my-app

Any Deployment, StatefulSet, or CronJob whose labels match the selector will be skipped.

Annotations

When a workload is scaled down, the operator adds these annotations:

Annotation	Description
crashloop-operator.lauger.de/scaled-down-reason	Human-readable reason for the scale-down
crashloop-operator.lauger.de/scaled-down-at	RFC3339 timestamp of when the scale-down occurred
crashloop-operator.lauger.de/previous-replicas	Previous replica count (Deployments and StatefulSets only)
crashloop-operator.lauger.de/scaled-down-by	Name of the policy that performed the scale-down

Multiple Policies

Policies are cluster-scoped and every policy sees every workload, so more than one can match the same workload. When that happens the most restrictive policy wins: it performs the scale-down, records itself in the scaled-down-by annotation, and counts the action in its own status. Every other matching policy leaves the workload alone, so a workload is never scaled down twice or attributed to two policies.

Only policies that would actually act on the workload right now take part in that comparison. A policy whose watchReasons do not cover the observed failure, or whose thresholds are not yet exceeded, cannot win and therefore cannot block a policy that would act.

Restrictiveness is compared in this order, and the first difference decides:

1. Lower restartThreshold
2. Shorter durationThreshold
3. allReplicasFailing: false before true, since it also acts on partial failure
4. failureCorrelation: Pod before Container, since it acts in more situations
5. dryRun: false before true, so a real action outranks a simulated one
6. Name in ascending order, purely to break a remaining tie

Because the order is total, the winner does not depend on the order in which policies are evaluated.

Metrics

The operator serves Prometheus metrics on port 8080. The chart does not expose them by default:

helm upgrade --install crashloop-operator oci://ghcr.io/slauger/charts/crashloop-operator \
  --set metrics.service.enabled=true \
  --set metrics.serviceMonitor.enabled=true

The serviceMonitor option needs the Prometheus Operator CRDs. The endpoint is unauthenticated, so restrict access if you expose it; see SECURITY.md.

Metric	Type	Labels	Description
crashloop_scaled_down_total	Counter	policy, namespace, kind, reason, dry_run	Actions taken, including dry-run ones
crashloop_workloads_scaled_down	Gauge	policy	Workloads the policy is currently holding at zero
crashloop_policy_evaluation_errors_total	Counter	policy	Per-workload errors that did not fail the reconcile
crashloop_policy_ready	Gauge	policy	1 when the last evaluation completed without errors

Alongside these, controller-runtime exports the usual controller_runtime_reconcile_* and workqueue_* metrics.

No metric carries a workload name. Namespaces are bounded by the cluster, workload names are not, and a series for a workload that is later deleted would never go away. Use status.activeScaledDown or the scaled-down-by annotation to identify individual workloads.

Two alerts worth having:

# A policy has been failing on individual workloads.
- alert: CrashLoopPolicyDegraded
  expr: crashloop_policy_ready == 0
  for: 15m

# Something has been held at zero replicas for a day and nobody noticed.
- alert: WorkloadScaledDownTooLong
  expr: crashloop_workloads_scaled_down > 0
  for: 24h

Note that crashloop_scaled_down_total counts dry-run actions too. Filter with dry_run="false" when alerting on real ones.

Slow restart loops

watchReasons only sees containers that are waiting. Kubelet resets its restart backoff once a container has stayed up longer than roughly twice the maximum backoff, so a container that survives beyond that between deaths restarts immediately every time and never enters CrashLoopBackOff. A memory leak has exactly this shape: run, grow, get OOM-killed, restart at once, repeat. Such a workload can reach hundreds of restarts unnoticed.

watchTerminationReasons covers that case by matching on why the container last exited rather than on what it is waiting for:

spec:
  watchTerminationReasons:
    - OOMKilled
  restartThreshold: 10
  restartWindow: 1h

The workload is acted on when a container has reached restartThreshold restarts and its most recent exit carries a listed reason and that exit happened within restartWindow. The window matters: the restart count is cumulative for the pod's whole life and never decays, so without it a workload that misbehaved last month would still be scaled down.

OOMKilled is the safe value to start with. Error also works but is broad, covering ordinary crashes, liveness kills and SIGKILL after the grace period alike.

Not counted: pods being deleted, pods that have completed, containers still inside their startup probe, and classic init containers, which run once and cannot loop. Init containers declared with restartPolicy: Always are sidecars and do count.

Troubleshooting

The policy exists but nothing is scaled down

Check the policy first:

kubectl get clp
kubectl describe clp default

Phase: Pending means the policy has not been evaluated yet. Phase: Active with Degraded: False means it ran and found nothing to act on. In that case work through the conditions the operator applies, in the order it applies them:

• dryRun is true. The operator logs and emits WorkloadScaleDownDryRun events for everything it would have acted on, and changes nothing. This is the intended starting point for a new policy.
• Neither threshold is exceeded. A workload is only acted on once restartThreshold restarts are reached or the pod has been failing for durationThreshold. A pod that never starts has no restarts, so it is the duration that applies. The duration is measured from the moment the pod stopped being ready, not from the last container exit, so a pod restarting in a loop accumulates it correctly rather than having the clock reset on every restart. A container that recovers and later fails again starts the clock over.
• failureCorrelation: Container and the replicas fail in different containers. The stricter mode holds off when the failures do not share a container name, on the grounds that they look coincidental rather than systematic. Switch back to Pod if you want any broken replica to count.
• allReplicasFailing is true and some replica is healthy. This defaults to true, so a Deployment with one broken and one running pod is left alone by design. Set it to false if you want partial failure to count.
• The namespace is excluded. excludeNamespaces defaults to kube-system, kube-public and kube-node-lease. Setting the field replaces that default rather than adding to it.
• namespaceSelector or excludeWorkloadSelector filters it out.
• The failure reason is not watched. watchReasons matches the container's waiting reason exactly. Note that termination reasons such as OOMKilled never appear as a waiting reason, so putting one in watchReasons matches nothing; see Slow restart loops. Check it with kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].state.waiting.reason}'.

Ready is False

Reason: ReconcilePartiallyFailed

The evaluation completed but hit errors on individual workloads, for example a workload it could not read. The operator does not abort the rest of the run. The operator logs name the workload:

kubectl -n crashloop-system logs -l app.kubernetes.io/name=crashloop-operator

A workload was scaled down and I want to know why

Every action is recorded on the workload itself:

kubectl get deployment my-app -o jsonpath='{.metadata.annotations}' | jq

scaled-down-reason carries the failure reason and the policy name, scaled-down-at the timestamp, scaled-down-by the policy that acted, and previous-replicas the replica count to restore. The operator also emits a WorkloadScaledDown event on the policy:

kubectl get events --field-selector involvedObject.name=default

Several policies match the same workload

Only one acts: the most restrictive one, as described under Multiple Policies. scaled-down-by names it. The other policies leave the workload out of their activeScaledDown list and do not count it, which is why a policy you expected to act may show a counter of zero.

Restoring a scaled-down workload

The operator never scales anything back up, so that recovery stays with whatever manages your workloads. To do it by hand:

kubectl scale deployment my-app \
  --replicas="$(kubectl get deployment my-app \
    -o jsonpath='{.metadata.annotations.crashloop-operator\.lauger\.de/previous-replicas}')"

For a CronJob, unset spec.suspend. Note that the operator will act again on the next evaluation if the underlying failure persists.

The CRD is out of date after a chart upgrade

Helm installs CRDs from the chart's crds/ directory but never upgrades or deletes them. After an upgrade that changes the CRD, apply it yourself:

kubectl apply -f https://raw.githubusercontent.com/slauger/crashloop-operator/develop/config/crd/bases/crashloop-operator.lauger.de_crashlooppolicies.yaml

Local Development

make generate manifests   # Regenerate deepcopy, CRD YAML, RBAC and chart CRDs
make build                # Build operator binary
make test                 # Run unit and envtest tests
make ci                   # Everything CI runs
make e2e-cluster && make e2e   # End-to-end tests on kind
make docker-build         # Build container image

make help lists every target. See CONTRIBUTING.md for the branching model, commit conventions and the drift checks that catch stale generated files.

Supply Chain Security

Released container images and Helm charts are:

• Signed with cosign keyless signing (Sigstore OIDC), so there are no private keys to manage or rotate
• Attested with SLSA provenance, both the buildkit attestation on each architecture and a registry-attached attestation on the multi-arch index
• Accompanied by an SBOM generated during the build

Signatures are made against the image digest, so the :latest tag and the matching version tag are covered by the same signature.

Verify the image:

cosign verify ghcr.io/slauger/crashloop-operator:latest \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp 'github\.com/slauger/crashloop-operator'

Verify the Helm chart:

cosign verify ghcr.io/slauger/charts/crashloop-operator:<version> \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp 'github\.com/slauger/crashloop-operator'

Inspect the provenance attestation:

cosign verify-attestation ghcr.io/slauger/crashloop-operator:latest \
  --type slsaprovenance \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp 'github\.com/slauger/crashloop-operator'

Released images are additionally validated against a Conforma policy during the release, which checks the signature, the provenance attestation and the base images against the rules in .conforma/policy.yaml. To run the same check yourself:

ec validate image \
  --image ghcr.io/slauger/crashloop-operator:<version> \
  --policy .conforma/policy.yaml \
  --certificate-identity-regexp 'https://github.com/slauger/crashloop-operator/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --output yaml

List everything attached to an image:

cosign tree ghcr.io/slauger/crashloop-operator:latest

Security

Please report vulnerabilities privately to simon@lauger.de rather than in a public issue. See SECURITY.md, which also describes the operator's blast radius and the mechanisms that limit it.

License

Apache License 2.0