Add opt-in cleanup policy for related-resource copies#181
Conversation
|
Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kcp-ci-bot
added
kind/feature
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kcp-ci-bot
added
needs-ok-to-test
|
Hi @wojciech12. Thanks for your PR. I'm waiting for a kcp-dev member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kcp-ci-bot
added
the
size/XXL
For `origin: service` related resources the agent never prunes a copy when its
origin object is deleted mid-life; the copy is orphaned permanently (no
per-object finalizer, no set-diff). Even on primary teardown, copies whose
origin already vanished are not reclaimed.
Add a `cleanupPolicy` field to RelatedResourceSpec with three values:
- Orphan (default): copies are never deleted (== legacy cleanup:false).
- OnPrimaryDeletion: copies deleted only on primary deletion (== legacy cleanup:true).
- MatchOrigin: a copy is pruned as soon as its origin object is gone, and all
copies are deleted on primary teardown (also reclaiming mid-life orphans).
The prune is List-and-diff driven by the primary reconcile, never a finalizer on
related objects (issues kcp-dev#172/kcp-dev#116). Copies are labelled with owning-primary and
identifier provenance so only agent-created copies are ever deleted, scoped to
the destination client so the origin object is never touched. The List is
cluster-wide (scoped by the provenance label selector) so copies mapped into a
different namespace via spec.object.namespace rewrites are pruned too. The
deprecated `cleanup` bool is still honoured via EffectiveCleanupPolicy(). CEL
rules require `watch` for MatchOrigin+origin:service and reject cleanup:true+Orphan.
Extends the opt-in cleanup work from kcp-dev#114 (PR kcp-dev#164).
Signed-off-by: wojciech12 <wojciech.barczynski@kubermatic.com>
bb0462a to
62c8745
Compare
kcp-ci-bot
added
dco-signoff: yes
xmudrii
left a comment
xmudrii
left a comment
There was a problem hiding this comment.
Reviewed the cleanup-policy change end-to-end (API types, CRD, syncer_related.go, object_syncer.go, tests). The core design — List-and-diff pruning driven by the primary reconcile with provenance labels, instead of per-object finalizers — is a sound way to sidestep the finalizer problems from #172/#116, and the EffectiveCleanupPolicy() back-compat mapping plus the CEL guards read well.
One item I'd treat as blocking, two worth addressing, and a couple of minor notes — details inline:
- Blocker:
Identifieris now used verbatim as a label value but has no length/charset validation; identifiers the CRD accepts today produce labels the API server rejects, silently breaking sync. - Provenance labels can collide across PublishedResources whose primaries share name+namespace and reuse an identifier.
- Provenance labels aren't re-stamped on the client-side-apply update path (the default), so copies predating the feature are never pruned.
rememberRelatedObjectsnow runs on an empty resolved set → annotation churn on the primary.- The prune has no fake-client unit test, though the package tests deletion that way elsewhere.
| func relatedCopyLabels(primary ctrlruntimeclient.Object, identifier, agentName string) map[string]string { | ||
| set := map[string]string{ | ||
| relatedPrimaryNameHashLabel: crypto.Hash(primary.GetName()), | ||
| relatedIdentifierLabel: identifier, |
There was a problem hiding this comment.
Identifier is used verbatim as a label value here, and the comment above says it's "validated to be alphanumeric by the API" — but there's no such validation: the CRD schema for identifier is bare type: string (no maxLength/pattern) and the Go field carries no kubebuilder markers.
I ran this branch's relatedCopyLabels() output through the apiserver's own metav1validation.ValidateLabels:
- 63-char identifier → valid; 64-char → rejected (
must be no more than 63 bytes) connection/details,connection details,-creds→ rejected by the label-value regex
So an identifier the CRD accepts can make the copy's labels invalid, and the apply/create then fails — sync silently breaks for that related resource. (New failure surface specifically for origin: kcp; origin: service was already constrained via the related-resources.syncagent.kcp.io/<identifier>.N annotation key.) Suggest hashing the identifier like name/namespace, or adding +kubebuilder:validation:MaxLength=63 + an alphanumeric Pattern to Identifier — and fixing the comment.
| // namespaces are hashed because they can exceed the label value limit or contain invalid | ||
| // characters; the identifier is validated to be alphanumeric by the API, so it is used verbatim. | ||
| // The agent name (when set) is included so that each agent only ever prunes its own copies. | ||
| func relatedCopyLabels(primary ctrlruntimeclient.Object, identifier, agentName string) map[string]string { |
There was a problem hiding this comment.
The label set is {primary-name-hash, primary-namespace-hash, identifier, agent}, and the prune List is typed only by projected Kind — nothing encodes the primary's GVK or the owning PublishedResource. Two PublishedResources that project copies to the same Kind, reuse the same identifier, and have primaries sharing name+namespace produce byte-identical labels, so one primary's prune selector matches the other's copies (verified: identical hashes, selector matches). Consider adding the PublishedResource name and/or primary GVK to the provenance labels.
| // ensureDestMetadata stamps the syncer's additional destination labels/annotations onto the given | ||
| // object. It is additive (it never removes labels the object already carries) and is used to | ||
| // attach related-resource provenance to destination copies. | ||
| func (s *objectSyncer) ensureDestMetadata(obj *unstructured.Unstructured) { |
There was a problem hiding this comment.
ensureDestMetadata runs from applyServerSide, ensureDestinationObject (create), and adoptExistingDestinationObject, but not from syncObjectSpec — the client-side-merge update path, which is the default (--enable-server-side-apply defaults to false). New copies get stamped at birth, so this is fine going forward; but copies that already exist when a user enables MatchOrigin/OnPrimaryDeletion are updated in place and never receive the provenance labels, so they're never enumerated for pruning — exactly the mid-life-orphan reclaim this PR targets. Worth re-stamping in the update path or documenting the limitation.
| // Remember the related objects on the primary object for the end-user. This is rebuilt from the | ||
| // freshly-resolved set so stale entries for pruned objects do not linger. | ||
| if relRes.Origin == syncagentv1alpha1.RelatedResourceOriginService { | ||
| annRequeue, err := s.rememberRelatedObjects(ctx, log, remote, relRes.Identifier, resolvedObjects) |
There was a problem hiding this comment.
With the early len(resolvedObjects) == 0 return removed, rememberRelatedObjects now runs even when zero objects resolve (for origin: service). On an empty resolution it wipes all related-resources.syncagent.kcp.io/<identifier>.* annotations and patches the primary, re-adding them next cycle — extra writes/requeues. Minor, but it's a behavior change for all policies, not just the pruning ones.
| // (mid-life prune). It only ever operates on the destination client, so origin objects are never | ||
| // touched, and it only ever sees objects that carry our provenance labels, so hand-created objects | ||
| // are never in scope. | ||
| func (s *ResourceSyncer) pruneRelatedCopies(ctx context.Context, log *zap.SugaredLogger, dest syncSide, primary *unstructured.Unstructured, projectedGVK schema.GroupVersionKind, selector labels.Selector, keep sets.Set[string], deleteAll bool) (requeue bool, err error) { |
There was a problem hiding this comment.
The package already unit-tests deletion/cleanup by driving ResourceSyncer.Process() against fakectrlruntimeclient (e.g. the DeletionTimestamp cases in syncer_test.go), and the fake client supports label-selector List. The new prune (keep-set filtering, deleteAll teardown, already-deleting requeue, empty-set) is currently covered only by e2e plus the two pure-function unit tests. A fake-client unit test of pruneRelatedCopies would match the package convention and guard the destructive path directly.
relatedCopyLabels uses the RelatedResource identifier verbatim as the value of the syncagent.kcp.io/related-identifier label, and the comment claimed it was "validated to be alphanumeric by the API". No such validation existed: the Go field carried no kubebuilder markers and the CRD schema was a bare `type: string`. An identifier the CRD accepted (>63 chars, or containing '/', spaces, leading/trailing '-') therefore produced an invalid label value, so the apply/create of the copy was rejected and sync silently broke for that related resource. This is a new failure surface for origin: kcp (origin: service was already constrained via the related-resources annotation key). Constrain Identifier at the API level to a lowercase RFC 1123 label of at most 63 characters (MaxLength=63 + Pattern), which is always a valid label value, and regenerate the CRD. Fix the misleading comment in meta.go and add a regression test asserting relatedCopyLabels output passes the API server's own ValidateLabels for CRD-accepted identifiers. Signed-off-by: wojciech12 <wojciech.barczynski@kubermatic.com>
The prune of related-resource copies (cleanupPolicy: MatchOrigin / primary
teardown) is driven by a cluster-wide List on the destination, filtered only by
the provenance labels relatedCopyLabels stamps on each copy. Those labels were
{primary-name-hash, primary-namespace-hash, identifier, agent}, which does not
uniquely identify the owning primary, so one primary's prune selector could
match — and delete — another's copies along two axes:
- Across PublishedResources: two PublishedResources projecting to the same
Kind, reusing an identifier, with primaries sharing name+namespace produce
byte-identical labels.
- Across kcp workspaces: the destination is shared by all workspaces, but the
labels carried no logical-cluster dimension, so primaries with identical
name+namespace in different workspaces collided too (the main synced object
already guards this via the remote-object-cluster label; related copies did
not).
Add two provenance labels so the set uniquely identifies the owner: the primary's
logical cluster (verbatim, like the main object's remote-object-cluster label,
since a kcp cluster name is a valid label value) and the owning PublishedResource
name (hashed, as it can exceed the label length limit). The identifier stays
unique within a PublishedResource, so these together fully disambiguate; the
primary GVK the review suggested as an alternative is implied by the
PublishedResource and thus redundant. Extend the round-trip test with the new
cluster and PublishedResource collision cases.
Signed-off-by: wojciech12 <wojciech.barczynski@kubermatic.com>
Related-resource copies are looked up on the destination by name, so an existing copy is never nil and its reconcile goes through the client-side-merge update path (syncObjectSpec). Unlike the create, adopt and Server-Side Apply paths, that path never calls ensureDestMetadata, so a copy that already existed when the user opts into a pruning cleanupPolicy (MatchOrigin / OnPrimaryDeletion) would never receive the provenance labels the prune enumerates by — the exact mid-life-orphan reclaim the feature targets would silently skip it. Since SSA is off by default, this is the default code path. Add ensureDestMetadataPersisted and call it from Sync() on the existing-object, non-SSA branch: it diffs the desired destLabels/destAnnotations against the live object and patches only when something is missing, so it backfills legacy copies once and is a no-op afterwards. It self-scopes to related copies (main-object syncers set no destLabels) and requeues after a restamp so the content sync runs on the next pass. Add a fake-client unit test covering the backfill, its idempotency, and the no-metadata no-op. Signed-off-by: wojciech12 <wojciech.barczynski@kubermatic.com>
This PR removed the early `len(resolvedObjects) == 0` return so the prune can run when nothing resolves, but that also let rememberRelatedObjects run on an empty set. On an empty resolution it wiped all related-resources.syncagent.kcp.io/<identifier>.* annotations off the primary and patched it (with a requeue), and would re-add them the next time objects resolved — extra writes and requeues, and a behavior change affecting every cleanup policy, not just the pruning ones. Return early from rememberRelatedObjects when the resolved set is empty. These annotations are purely informational and the prune is the authoritative cleanup for the copies, so leaving the last-known annotations untouched on a fully-empty resolution is preferable to churning the primary. Partial resolutions still rebuild the set and drop entries for objects that no longer resolve. Add a unit test asserting an empty resolution neither patches the primary nor requeues. Signed-off-by: wojciech12 <wojciech.barczynski@kubermatic.com>
The destructive prune path (keep-set filtering, deleteAll teardown, already-deleting requeue, empty-set, selector scoping) was only covered by e2e and two pure-function unit tests. Drive pruneRelatedCopies against fakectrlruntimeclient — matching how the package already tests deletion — with subtests for: a mid-life prune that keeps the keep-set and deletes the rest (including a copy in another namespace, proving the cluster-wide List), a teardown that deletes every matching copy, an empty match set that is a no-op, and a copy that is already being deleted (finalizer-held) requeueing instead of erroring. Each case also asserts that a copy carrying a different identifier's provenance labels is never in scope, guarding the selector scoping. Signed-off-by: wojciech12 <wojciech.barczynski@kubermatic.com>
…opies The MatchOrigin mid-life prune keeps the destination set equal to the origin set by deleting every labelled copy whose origin object was not in the freshly resolved set. When the resolution is fully empty the keep set is empty, so the prune deletes ALL copies at once. That resolution comes from a cache-backed read (localManager.GetClient() / the per-cluster GetClient()), and a relisting or otherwise stale informer -- notably a kcp virtual-workspace cache -- can momentarily return nothing while the origin objects still exist. The prune would then delete and immediately recreate every copy, disrupting consumers such as Secrets mounted by workloads. The non-destructive annotation bookkeeping already refuses to act on an empty resolution (rememberRelatedObjects) for exactly this reason; the destructive prune must be at least as careful. A blanket "skip prune when empty" is not an option: it would break the feature's contract that deleting the only origin object mid-life prunes its copy (and for origin:service a watch is mandatory precisely to drive that), leaving the last copy orphaned until teardown. Before deleting the whole set, re-run the origin resolution against an uncached reader and only prune if the live read also finds nothing; otherwise requeue and let the cache converge. A genuinely empty origin still prunes, so the single-object mid-life case keeps working. Partial under-resolution is left unguarded on purpose: it prunes at most a subset and self-heals on the next pass, matching the cache semantics the rest of the sync relies on. Plumb uncached readers via a new WithLiveReaders option (localManager.GetAPIReader() and the per-cluster GetAPIReader()); a small liveReadClient serves reads from the uncached reader while borrowing the RESTMapper from the cached client, so resolveRelatedResourceObjects is reused verbatim. When no live reader is configured the destructive full-set prune is skipped rather than run on unverified data. Add a unit test covering the not-checked, still-populated and genuinely-empty cases. Signed-off-by: wojciech12 <wojciech.barczynski@kubermatic.com>
The related-resource identifier gained a MaxLength/Pattern validation (a lowercase RFC 1123 label), but the TestValidateRelatedResourceSpec cases build a RelatedResourceSpec without an identifier. An empty identifier now fails the pattern before the rule actually under test (the GVK, syncStatus and cleanupPolicy CEL rules) is ever reached, so every valid:true case was rejected for the wrong reason and failed. Inject a valid identifier in the test loop when a case does not set its own, so the identifier is never the confound; each case still exercises exactly the rule it targets, and the valid:false cases still fail for their real reason. Signed-off-by: wojciech12 <wojciech.barczynski@kubermatic.com>
The agent now stamps provenance metadata on every destination copy of a related resource: a syncagent.kcp.io/agent-name label, the related-resource ownership labels (cluster/name/namespace/published-resource hashes and identifier) and the plaintext primary name/namespace annotations, all used to enumerate copies for pruning. compareUnstructured only stripped the internal kcp claim labels and the cluster annotation, so the related-resource sync tests saw these extra syncagent.kcp.io/* keys on the actual object and failed the diff against their expected object (which carries none). The hashed values also embed the per-run logical cluster id, so they cannot simply be hard-coded into the expectation. Extend compareUnstructured to drop any syncagent.kcp.io/* label and annotation, matching its existing intent of ignoring sync-related metadata. Fixes TestSyncRelatedObjects, TestSyncRelatedMultiObjects, TestSyncNonStandardRelatedResources and TestSyncNonStandardRelatedResourcesMultipleAPIExports. Signed-off-by: wojciech12 <wojciech.barczynski@kubermatic.com>
The prune of related-resource copies only ever selects by the full owner tuple — relatedCopySelector always uses the complete relatedCopyLabels set and never queries a partial key — so the four separate provenance labels (related-primary-cluster, related-published-resource-hash, related-primary-name-hash, related-primary-namespace-hash) can be a single related-owner label holding sha1(cluster, pubResName, namespace, name). related-identifier and agent-name stay as their own labels (both are valid label values, human-readable, and agent-name is used elsewhere via OwnedBy). This drops each copy from six provenance labels to three and removes the per-dimension uniqueness reasoning entirely: uniqueness is now inherent in the hash input, and a NUL separator keeps the dimensions unambiguous. A cluster-scoped primary's empty namespace folds into the hash cleanly and cannot collide with a namespaced primary. The plaintext primary name/namespace annotations are unchanged. Update the round-trip and metadata unit tests; e2e comparisons already ignore all syncagent.kcp.io/* metadata, so no e2e change is needed. Verified with the sync unit suite and the MatchOrigin / cleanup / related-sync e2e tests against a live kcp. Signed-off-by: wojciech12 <wojciech.barczynski@kubermatic.com>
548377d to
2d582ec
Compare
3 participants
For
origin: servicerelated resources the agent never prunes a copy when its origin object is deleted mid-life; the copy is orphaned permanently (no per-object finalizer, no set-diff). Even on primary teardown, copies whose origin already vanished are not reclaimed.Add a
cleanupPolicyfield to RelatedResourceSpec with three values:The prune is List-and-diff driven by the primary reconcile, never a finalizer on related objects (issues #172/#116). Copies are labelled with owning-primary and identifier provenance so only agent-created copies are ever deleted, scoped to the destination client so the origin object is never touched. The List is cluster-wide (scoped by the provenance label selector) so copies mapped into a different namespace via spec.object.namespace rewrites are pruned too. The deprecated
cleanupbool is still honoured via EffectiveCleanupPolicy(). CEL rules requirewatchfor MatchOrigin+origin:service and reject cleanup:true+Orphan.Extends the opt-in cleanup work from #114 (PR #164).
Summary
What Type of PR Is This?
/kind feature
Related Issue(s)
Fixes #
Release Notes
Add a
cleanupPolicyfield to RelatedResourceSpec with three values: