Provide a single-holder claim primitive for Lease and annotation based mutual exclusion

[sourcehawk]

Problem

Operators built on ocf keep needing the same guarantee: one holder per key, where the key is a storage contract, a logical database, or a Keycloak realm. The framework has no claim concept, so every consumer writes its own. konsole-is/camunda-operator now carries six implementations, and three of them are the same Lease algorithm typed three times: identical holder namespace/name/UID/key annotation quadruple, identical 128 character holder identity bound, identical hashed name bounding, identical stale-holder takeover with UID and resourceVersion preconditions, identical label-selector sweep that releases every claim the holder no longer names, and the same "newest sibling claimant yields" fairness pre-filter.

The algorithm is subtle: Create-wins against a stale cache needs an uncached reader, takeover must fence on UID plus resourceVersion, and a crash between claim and release must not leak the key. Each new copy re-derives these decisions and re-tests them.

Call sites in konsole-is/camunda-operator:

• Lease per CamundaCluster: pkg/clusterclaim/claim.go (Claim, Holds, HolderActive, Release)
• Lease per logical database: internal/controller/database/claim.go (claim, takeClaim, createClaim, holderKeeps, releaseHeldClaims, dropClaim)
• Lease per Keycloak realm: internal/controller/camundamanagementcluster/realmclaim.go on PR feat(management): claim the Keycloak realm so one realm answers to one plane konsole-is/camunda-operator#321
• Annotation with optimistic lock on the claimed CR: pkg/wrappers/secondarystorageconfig/claim.go, driven from internal/controller/camundacluster/secondarystorage.go (PR fix(camundacluster): keep two clusters from ever writing one storage backend konsole-is/camunda-operator#319)
• Annotation per claimed CamundaCluster: internal/controller/camundamanagementcluster/attachment.go
• Holder read on the target CR: internal/controller/camundaoptimize/attachment.go

Approach

A claim package with a constructor such as claim.New(reader client.Reader, writer client.Client, Config{Namespace, KeyFunc, Prefix, Labels}) and the operations Acquire(ctx, key, self Holder) (Holder, Outcome, error), Release(ctx, self Holder, keep ...string) error, and Holds(ctx, key, self) (bool, error).

Two injection points cover everything the copies differ in: a pluggable Liveness(ctx, Holder) (bool, error) that decides when a stale holder may be taken over, and a pluggable backend. The LeaseBackend implements Create-wins on coordinationv1.Lease; the AnnotationBackend implements the optimistic-lock patch on a claimed object the operator owns. Outcome carries a parked reason the consumer folds into its Ready condition.

Two optional sub-features belong to the same primitive rather than to separate issues: a HandoverGate hook that confirms the stale holder's workloads are gone before takeover (the consumer repo has four drain-wait variants feeding a WaitingForHandover reason), and release-on-delete so a finalizer can call one sweep instead of hand-writing the release ordering.

Verification

• The three Lease call sites above can be ported to the primitive with only a Liveness function and message wording as consumer code.
• A takeover test covers: stale holder, UID changed, resourceVersion changed, and a holder that re-appears between the liveness check and the write.
• A crash between Acquire and Release leaks nothing: the release sweep by label finds and frees the orphan.
• An Acquire against a stale cache does not double-grant: the uncached read path is exercised in a test with a deliberately lagging cache.

Context

Found while auditing konsole-is/camunda-operator for repeated hand-rolled patterns. The realm claim (PR konsole-is/camunda-operator#321) was written from scratch this week while two sibling copies already existed in the same repository, which is the clearest signal the framework should own this.