Skip to content

fix: preserve immutable DaemonSet selector on FluentBit label changes - #2017

Open
pujitha24 wants to merge 3 commits into
fluent:masterfrom
pujitha24:auto/issue-1944
Open

fix: preserve immutable DaemonSet selector on FluentBit label changes#2017
pujitha24 wants to merge 3 commits into
fluent:masterfrom
pujitha24:auto/issue-1944

Conversation

@pujitha24

Copy link
Copy Markdown

What this PR does / why we need it:

Motivation: When spec.labels on a FluentBit CR is changed after its
DaemonSet already exists, the reconciler fails on every attempt with:

DaemonSet.apps "fluent-bit" is invalid: spec.selector: Invalid value:
...: field is immutable

MakeDaemonSet() derives spec.selector.matchLabels directly from
fb.Spec.Labels, and the controller's mutate() function unconditionally
overwrote the existing DaemonSet's Spec (including the selector) with the
newly computed one on every reconcile. Kubernetes rejects any change to a
DaemonSet's spec.selector after creation, so this update is always
rejected, and the reconciler keeps retrying and failing forever. The issue
also reports that other legitimate DaemonSet updates get blocked by the
same rejected patch while this selector mismatch persists.

Approach: In the DaemonSet branch of mutate(), capture the existing
Spec.Selector before overwriting Spec, then restore it afterwards
instead of applying the newly computed one. Since a DaemonSet's pod
template labels must always be a superset of its selector's matchLabels,
any selector labels that would otherwise have been dropped from the new
fb.Spec.Labels are merged back into Spec.Template.Labels via a freshly
built map (not written in place, since MakeDaemonSet reuses
fb.Spec.Labels as the same map object for the DaemonSet's own labels and
selector — mutating it in place would leak back into the FluentBit CR's
own spec in memory).

This does not make the DaemonSet's selector track later label changes
(that would require deleting and recreating the DaemonSet, which is a
separate, more invasive change already under discussion in the issue
thread and is out of scope here). User-visible behavior for the selector
itself is unchanged from before label changes were introduced: it simply
stays pinned to whatever it was at creation time. The concrete, narrower
benefit is that changing spec.labels (or removing RBAC/labels-triggered
churn) no longer wedges the controller in a permanent error loop, and
other pending DaemonSet updates are no longer blocked by it.

Validation: go build ./... passes. Added
TestFluentBitMutateDaemonSetPreservesImmutableSelector (label added) and
TestFluentBitMutateDaemonSetRemovedLabelDoesNotMutateSpec (label removed,
also asserts fb.Spec.Labels is not mutated as a side effect) to
controllers/fluentbit_controller_test.go. Verified both new tests fail
against the pre-fix code and pass against the fix. Ran:

go test ./controllers/... -v

All tests pass, including the pre-existing (empty) Ginkgo suite.

Which issue(s) this PR fixes:

Fixes #1944

Does this PR introduced a user-facing change?

Fixed a bug where changing `spec.labels` on a FluentBit resource after its DaemonSet was created caused the operator to repeatedly fail reconciliation trying to update the DaemonSet's immutable `spec.selector` field.

Signed-off-by: Pujitha Paladugu 10557236+pujitha24@users.noreply.github.com

### What this PR does / why we need it:

Motivation: When `spec.labels` on a FluentBit CR is changed after its
DaemonSet already exists, the reconciler fails on every attempt with:

    DaemonSet.apps "fluent-bit" is invalid: spec.selector: Invalid value:
    ...: field is immutable

`MakeDaemonSet()` derives `spec.selector.matchLabels` directly from
`fb.Spec.Labels`, and the controller's `mutate()` function unconditionally
overwrote the existing DaemonSet's `Spec` (including the selector) with the
newly computed one on every reconcile. Kubernetes rejects any change to a
DaemonSet's `spec.selector` after creation, so this update is always
rejected, and the reconciler keeps retrying and failing forever. The issue
also reports that other legitimate DaemonSet updates get blocked by the
same rejected patch while this selector mismatch persists.

Approach: In the DaemonSet branch of `mutate()`, capture the existing
`Spec.Selector` before overwriting `Spec`, then restore it afterwards
instead of applying the newly computed one. Since a DaemonSet's pod
template labels must always be a superset of its selector's matchLabels,
any selector labels that would otherwise have been dropped from the new
`fb.Spec.Labels` are merged back into `Spec.Template.Labels` via a freshly
built map (not written in place, since `MakeDaemonSet` reuses
`fb.Spec.Labels` as the same map object for the DaemonSet's own labels and
selector — mutating it in place would leak back into the FluentBit CR's
own spec in memory).

This does not make the DaemonSet's selector track later label changes
(that would require deleting and recreating the DaemonSet, which is a
separate, more invasive change already under discussion in the issue
thread and is out of scope here). User-visible behavior for the selector
itself is unchanged from before label changes were introduced: it simply
stays pinned to whatever it was at creation time. The concrete, narrower
benefit is that changing `spec.labels` (or removing RBAC/labels-triggered
churn) no longer wedges the controller in a permanent error loop, and
other pending DaemonSet updates are no longer blocked by it.

Validation: `go build ./...` passes. Added
`TestFluentBitMutateDaemonSetPreservesImmutableSelector` (label added) and
`TestFluentBitMutateDaemonSetRemovedLabelDoesNotMutateSpec` (label removed,
also asserts `fb.Spec.Labels` is not mutated as a side effect) to
`controllers/fluentbit_controller_test.go`. Verified both new tests fail
against the pre-fix code and pass against the fix. Ran:

    go test ./controllers/... -v

All tests pass, including the pre-existing (empty) Ginkgo suite.

### Which issue(s) this PR fixes:

Fixes fluent#1944

### Does this PR introduced a user-facing change?

```release-note
Fixed a bug where changing `spec.labels` on a FluentBit resource after its DaemonSet was created caused the operator to repeatedly fail reconciliation trying to update the DaemonSet's immutable `spec.selector` field.
```

Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 26, 2026 22:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a reconciliation failure in the FluentBit controller where changing spec.labels after a DaemonSet is created causes repeated reconcile errors due to the DaemonSet spec.selector being immutable. The fix preserves the existing DaemonSet selector during mutation and adjusts pod template labels to remain compatible with that selector, and adds unit tests covering the label-add and label-remove cases (Issue #1944).

Changes:

  • Preserve the existing DaemonSet Spec.Selector during CreateOrPatch mutation to avoid immutable-field update failures.
  • Merge selector labels back into the DaemonSet pod template labels to keep the template compatible with the preserved selector.
  • Add controller unit tests reproducing the selector-immutability scenario for label add/remove.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
controllers/fluentbit_controller.go Preserves existing DaemonSet selector during mutation and rebuilds template labels to satisfy the preserved selector.
controllers/fluentbit_controller_test.go Adds unit tests validating selector preservation and avoiding unintended mutation of fb.Spec.Labels.

Comment thread controllers/fluentbit_controller.go
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Josh Baird <jbaird@galileo.io>
Copilot AI review requested due to automatic review settings July 28, 2026 14:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

controllers/fluentbit_controller_test.go:121

  • This assertion can pass even if the selector’s other key/value changes unexpectedly (it only checks team plus len==2). To make the regression test stricter, also assert the expected value for the other selector label (e.g., got[\"app\"] == \"fluentbit\").
	if got := existing.Spec.Selector.MatchLabels; len(got) != 2 || got["team"] != "logging" {
		t.Fatalf("expected the immutable selector to stay unchanged, got %v", got)
	}

Comment thread controllers/fluentbit_controller.go
Copilot AI review requested due to automatic review settings July 28, 2026 14:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Changing FluentBit labels causes immutable selector error

3 participants