fix: preserve immutable DaemonSet selector on FluentBit label changes - #2017
Open
pujitha24 wants to merge 3 commits into
Open
fix: preserve immutable DaemonSet selector on FluentBit label changes#2017pujitha24 wants to merge 3 commits into
pujitha24 wants to merge 3 commits into
Conversation
### What this PR does / why we need it:
Motivation: When `spec.labels` on a FluentBit CR is changed after its
DaemonSet already exists, the reconciler fails on every attempt with:
DaemonSet.apps "fluent-bit" is invalid: spec.selector: Invalid value:
...: field is immutable
`MakeDaemonSet()` derives `spec.selector.matchLabels` directly from
`fb.Spec.Labels`, and the controller's `mutate()` function unconditionally
overwrote the existing DaemonSet's `Spec` (including the selector) with the
newly computed one on every reconcile. Kubernetes rejects any change to a
DaemonSet's `spec.selector` after creation, so this update is always
rejected, and the reconciler keeps retrying and failing forever. The issue
also reports that other legitimate DaemonSet updates get blocked by the
same rejected patch while this selector mismatch persists.
Approach: In the DaemonSet branch of `mutate()`, capture the existing
`Spec.Selector` before overwriting `Spec`, then restore it afterwards
instead of applying the newly computed one. Since a DaemonSet's pod
template labels must always be a superset of its selector's matchLabels,
any selector labels that would otherwise have been dropped from the new
`fb.Spec.Labels` are merged back into `Spec.Template.Labels` via a freshly
built map (not written in place, since `MakeDaemonSet` reuses
`fb.Spec.Labels` as the same map object for the DaemonSet's own labels and
selector — mutating it in place would leak back into the FluentBit CR's
own spec in memory).
This does not make the DaemonSet's selector track later label changes
(that would require deleting and recreating the DaemonSet, which is a
separate, more invasive change already under discussion in the issue
thread and is out of scope here). User-visible behavior for the selector
itself is unchanged from before label changes were introduced: it simply
stays pinned to whatever it was at creation time. The concrete, narrower
benefit is that changing `spec.labels` (or removing RBAC/labels-triggered
churn) no longer wedges the controller in a permanent error loop, and
other pending DaemonSet updates are no longer blocked by it.
Validation: `go build ./...` passes. Added
`TestFluentBitMutateDaemonSetPreservesImmutableSelector` (label added) and
`TestFluentBitMutateDaemonSetRemovedLabelDoesNotMutateSpec` (label removed,
also asserts `fb.Spec.Labels` is not mutated as a side effect) to
`controllers/fluentbit_controller_test.go`. Verified both new tests fail
against the pre-fix code and pass against the fix. Ran:
go test ./controllers/... -v
All tests pass, including the pre-existing (empty) Ginkgo suite.
### Which issue(s) this PR fixes:
Fixes fluent#1944
### Does this PR introduced a user-facing change?
```release-note
Fixed a bug where changing `spec.labels` on a FluentBit resource after its DaemonSet was created caused the operator to repeatedly fail reconciliation trying to update the DaemonSet's immutable `spec.selector` field.
```
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses a reconciliation failure in the FluentBit controller where changing spec.labels after a DaemonSet is created causes repeated reconcile errors due to the DaemonSet spec.selector being immutable. The fix preserves the existing DaemonSet selector during mutation and adjusts pod template labels to remain compatible with that selector, and adds unit tests covering the label-add and label-remove cases (Issue #1944).
Changes:
- Preserve the existing DaemonSet
Spec.SelectorduringCreateOrPatchmutation to avoid immutable-field update failures. - Merge selector labels back into the DaemonSet pod template labels to keep the template compatible with the preserved selector.
- Add controller unit tests reproducing the selector-immutability scenario for label add/remove.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| controllers/fluentbit_controller.go | Preserves existing DaemonSet selector during mutation and rebuilds template labels to satisfy the preserved selector. |
| controllers/fluentbit_controller_test.go | Adds unit tests validating selector preservation and avoiding unintended mutation of fb.Spec.Labels. |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Josh Baird <jbaird@galileo.io>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
controllers/fluentbit_controller_test.go:121
- This assertion can pass even if the selector’s other key/value changes unexpectedly (it only checks
teampluslen==2). To make the regression test stricter, also assert the expected value for the other selector label (e.g.,got[\"app\"] == \"fluentbit\").
if got := existing.Spec.Selector.MatchLabels; len(got) != 2 || got["team"] != "logging" {
t.Fatalf("expected the immutable selector to stay unchanged, got %v", got)
}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Motivation: When
spec.labelson a FluentBit CR is changed after itsDaemonSet already exists, the reconciler fails on every attempt with:
MakeDaemonSet()derivesspec.selector.matchLabelsdirectly fromfb.Spec.Labels, and the controller'smutate()function unconditionallyoverwrote the existing DaemonSet's
Spec(including the selector) with thenewly computed one on every reconcile. Kubernetes rejects any change to a
DaemonSet's
spec.selectorafter creation, so this update is alwaysrejected, and the reconciler keeps retrying and failing forever. The issue
also reports that other legitimate DaemonSet updates get blocked by the
same rejected patch while this selector mismatch persists.
Approach: In the DaemonSet branch of
mutate(), capture the existingSpec.Selectorbefore overwritingSpec, then restore it afterwardsinstead of applying the newly computed one. Since a DaemonSet's pod
template labels must always be a superset of its selector's matchLabels,
any selector labels that would otherwise have been dropped from the new
fb.Spec.Labelsare merged back intoSpec.Template.Labelsvia a freshlybuilt map (not written in place, since
MakeDaemonSetreusesfb.Spec.Labelsas the same map object for the DaemonSet's own labels andselector — mutating it in place would leak back into the FluentBit CR's
own spec in memory).
This does not make the DaemonSet's selector track later label changes
(that would require deleting and recreating the DaemonSet, which is a
separate, more invasive change already under discussion in the issue
thread and is out of scope here). User-visible behavior for the selector
itself is unchanged from before label changes were introduced: it simply
stays pinned to whatever it was at creation time. The concrete, narrower
benefit is that changing
spec.labels(or removing RBAC/labels-triggeredchurn) no longer wedges the controller in a permanent error loop, and
other pending DaemonSet updates are no longer blocked by it.
Validation:
go build ./...passes. AddedTestFluentBitMutateDaemonSetPreservesImmutableSelector(label added) andTestFluentBitMutateDaemonSetRemovedLabelDoesNotMutateSpec(label removed,also asserts
fb.Spec.Labelsis not mutated as a side effect) tocontrollers/fluentbit_controller_test.go. Verified both new tests failagainst the pre-fix code and pass against the fix. Ran:
All tests pass, including the pre-existing (empty) Ginkgo suite.
Which issue(s) this PR fixes:
Fixes #1944
Does this PR introduced a user-facing change?
Signed-off-by: Pujitha Paladugu 10557236+pujitha24@users.noreply.github.com