Skip to content

Suspend the managed workloads of a component when there is no input to build it from #207

Description

@sourcehawk

Problem

Builder.Suspend(bool) and concepts.Suspendable suspend a built resource: the suspension intent is stored on the wrapper and applied in a later Mutate(). That requires a rendered desired state, which requires resolved input. The case where suspension matters most is exactly the case where there is no input: a pre-check fails because a referenced config object was deleted, the controller stages a failure on Ready and returns, and the workloads keep running on their last applied configuration. For a workload that writes to a backend the owner no longer resolves, that is not a cosmetic gap. It keeps writing.

konsole-is/camunda-operator hit this twice and hand-rolled the same fix both times (PR konsole-is/camunda-operator#319, which also carries the earlier #320): list the live workloads by label, patch spec.replicas to zero directly, and hand-fake the ocf suspension condition vocabulary (Suspending while draining, Suspended at zero) so the status still reads like a component. See internal/controller/camundacluster/suspend.go and internal/controller/camundaoptimize/suspend.go on that PR. Three more controllers in the same repository have the identical latent gap and no fix yet (elasticsearchcluster, databaseserver, camundamanagementcluster), tracked there as a separate bug.

The two copies differ only in the label set, the workload-to-condition map, and the ordering rule (the Optimize copy scales its writer first). All three are parameters.

Approach

A detached suspension entry point that needs no built component, for example component.SuspendManaged(ctx, rec, SuspendSpec{Selector labels.Selector, Namespace string, ConditionByComponentLabel map[string]ConditionType, Order []string, EventReason, EventAction string}) (found bool, err error).

It reads the live workloads matching the selector with an uncached reader, checks controller ownership, patches replicas to zero with a merge patch so the field stays outside server-side apply ownership (the normal component apply then takes the field back with force once the input resolves again, so recovery needs no code), tries every workload and joins the errors instead of stopping at the first, and stages the same Suspending/Suspended conditions the built path stages so the status vocabulary is one vocabulary.

This is a hole in an existing concept, not a new concept: the built path and the detached path should share the condition and event code.

Verification

  • The two hand-rolled copies on PR fix(camundacluster): keep two clusters from ever writing one storage backend konsole-is/camunda-operator#319 can be replaced by calls to the primitive plus a SuspendSpec each.
  • A test covers: workload missing (found=false, no error), workload not owned (skipped), workload already at zero (condition staged, no patch), patch conflict on one of two workloads (the other is still scaled, errors joined).
  • Recovery is free: after the input resolves, the ordinary component apply restores replicas without any resume code path.
  • The staged conditions are byte-identical to what Builder.Suspend(true) produces for the same workload.

Context

Found while auditing konsole-is/camunda-operator for repeated hand-rolled patterns. Related to the pre-check failure reporting gap filed separately: the two compose (fail the pre-check, then suspend detached).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

featureUser-observable capability or surface

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions