Changing `spec.solrAddressability.podPort` deletes every pod without migrating replicas

[autokctrl-team]

Changing spec.solrAddressability.podPort deletes every pod without migrating replicas

Environment

• solr-operator built from ed5c5c7d28a4c1189d19f581259e05385c0d4b20
• Solr 9.7.0
• Kubernetes: kind
• SolrCloud: 2 pods, dataStorage: ephemeral, updateStrategy.method: Managed

What happened

I edited one field on a healthy SolrCloud:

spec:
  solrAddressability:
    podPort: 8983   # -> 8984

That is a pod template change, so the operator correctly started a managed rolling update. But it deleted the pods without migrating any replicas first.

The operator log shows the delete happening five times:

ManagedUpdateSelector ... Deleting solr pod for update ... "pod": "t4-solrcloud-1"

and zero migration activity for those pods — no Migrating all replicas off of pod before deletion, no ReplicaMigrationStarted event, no move-replicas-* async request. Nothing was logged as an error. As far as the operator was concerned the rolling update succeeded.

Where the source code is wrong

A running pod registers its Solr node name in ZooKeeper using the podPort value from the pod template at creation time. After the CR is updated, the pod remains registered as ...:8983_solr.

On each reconcile, the operator constructs the lookup key from the current CR using SolrNodeName. After the update, this function returns ...:8984_solr. The registered node name and the operator's lookup key therefore no longer match.

So the lookup in PodContents misses every running pod:

// controllers/util/solr_update_util.go:112-115
contents, isInClusterState = state.NodeContents[SolrNodeName(cloud, podName)]

That failed lookup is immediately converted to podHasReplicas=false:

// controllers/util/solr_update_util.go:118-121
func (state NodeReplicaState) PodHasReplicas(cloud *solr.SolrCloud, podName string) bool {
  contents, isInClusterState := state.PodContents(cloud, podName)
  return isInClusterState && contents.replicas > 0
}

EvictReplicasForPodIfNecessary treats this false as proof that there are no replicas. It skips REPLACENODE and allows the pod to be deleted:

// controllers/util/solr_update_util.go:569-591
if podHasReplicas {
  // Submit new Replace Node request
  ...
} else {
  canDeletePod = true
}

DeletePodForUpdate then deletes the pod without rechecking the cluster state:

// controllers/solr_pod_lifecycle_util.go:90-106
if podHasReplicas {
  requeueAfterDuration = time.Millisecond * 10
} else {
  deletePod = true
}
...
if deletePod {
  err = r.Delete(ctx, pod, client.Preconditions{UID: &pod.UID})
}

The bug is that "node name not found" is treated as "pod has no replicas." After podPort changes, the lookup key is wrong, but the operator still skips migration and deletes the pod.

Because the same edit is a pod template change, this repeats for every pod in the StatefulSet.

Expected behavior

A missed lookup should not be treated as proof that the pod has no replicas. The operator should distinguish between these cases:

• The pod is present in the cluster state and has no replicas: it is safe to delete.
• The pod is not found in the cluster state: do not delete it until its replica state can be determined.