CKS: scale requests without an offering (autoscaler, size, enabling autoscaling) resize every node to an outdated cluster-wide offering #14552

problem

What happens, in plain words

CloudStack Kubernetes Service (CKS) never resizes a single node. It resizes whole node groups: all control nodes together, all worker nodes together (and all dedicated etcd nodes, if the cluster has any). Since 4.21 the scale API can give each group its own offering (the nodeofferings option), so a cluster stores its compute offering in two places:

• one cluster-wide offering (serviceofferingid). It dates from before 4.21, when every node used one offering. Today it is the fallback for any group without its own offering, and it is what "scale everything" (serviceofferingid=X without nodeofferings) sets; and
• separate offerings for control nodes and for worker nodes (controlofferingid, workerofferingid).

When an operator resizes only the workers or only the control nodes, CloudStack updates that group's offering but never updates the cluster-wide one. From then on the cluster-wide offering is out of date: it still shows whatever was set when the cluster was created or last resized as a whole, and no longer matches anything the nodes actually run.

Every scaleKubernetesCluster request that does not name an offering is then treated as "move all nodes to the cluster-wide offering". That includes:

• turning on autoscaling (autoscalingenabled=true minsize=… maxsize=…),
• adding workers by hand (size=N),
• every scale-up done by the cluster-autoscaler itself. Its CloudStack provider adds workers by calling exactly scaleKubernetesCluster id=… size=N (and removes them with nodeids=…).

So on a cluster where the workers or the control nodes were ever resized on their own, CloudStack first resizes all existing nodes, control nodes included, to the old offering and records it as the worker offering, and only then adds the new worker, at the old size. The operator (or the autoscaler, unattended) asked for one more worker and got a resize of the whole cluster. If the old offering is smaller than what the nodes run now, the request is most likely refused instead, so the cluster cannot be scaled out at all.

What should happen: a request without an offering changes no node's size, and new workers use the cluster's current worker offering, i.e. the size the existing workers were last scaled to. If the cluster-wide offering happens to be up to date (for example right after a cluster-wide resize), this is exactly what happens today.

A second, smaller problem shows up when enabling autoscaling: the setting is not saved correctly. The cluster-autoscaler is deployed and running, but CloudStack still records autoscaling as off, so a later autoscalingenabled=false does nothing and the autoscaler has to be removed by hand.

The CloudStack UI is not affected: once a cluster has per-group offerings, its Scale dialog pre-fills the worker and control offering fields with their current values and sends them as nodeofferings, so the out-of-date cluster-wide offering is never used. The problem hits requests that name no offering: API and cmk calls, automation, and — most importantly — the cluster-autoscaler, which scales up with size=N only.

Steps to reproduce

1. Create a CloudManaged CKS cluster on offering A (1 control, 2 workers) and wait for Running.
2. Resize per node group, for example workers to B and control to C (from the UI or the API):
   scaleKubernetesCluster id=<id> nodeofferings[0].node=worker nodeofferings[0].offering=<B> nodeofferings[1].node=control nodeofferings[1].offering=<C>
   → listKubernetesClusters still shows serviceofferingid = A.
   (Variant: do a cluster-wide scale to X first, then per-group scales; the cluster-wide field stays X.)
3. Through the API (not the UI), either
   a) enable autoscaling: scaleKubernetesCluster id=<id> autoscalingenabled=true minsize=2 maxsize=3, or
   b) add a worker: scaleKubernetesCluster id=<id> size=3.
4. Observe:
   • existing control and worker VMs are moved to offering A (on a running cluster live, one by one; a VM may be migrated for capacity), and workerofferingid becomes A;
   • in b), the new worker is deployed with A;
   • in a), the cluster-autoscaler is deployed but autoscalingenabled is not set, and autoscalingenabled=false afterwards leaves the autoscaler running.

Observed for 3a) during QA of #14223; 3b) follows the same code path (traced below). Manual scale-out with size=N was also tested in that QA run and behaved correctly, but only while the cluster-wide offering was up to date.

Expected

• A request that names no offering (size, nodeids, autoscaling parameters) changes no node offering and resizes no existing VM, neither control nor worker nodes.
• Workers added by size=N, whether sent by an operator or by the cluster-autoscaler, use the current workerofferingid.
• autoscalingenabled, minsize and maxsize are saved, and disabling autoscaling removes the cluster-autoscaler.

Technical details

• Where the outdated offering is injected: KubernetesClusterManagerImpl#scaleKubernetesCluster always calls createNodeTypeToServiceOfferingMap(cmd.getServiceOfferingNodeTypeMap(), cmd.getServiceOfferingId(), cluster). When the request has no nodeofferings map, that method always adds a DEFAULT entry: the request's serviceofferingid if given, otherwise cluster.getServiceOfferingId(). A request without any offering therefore reaches the scale worker as "scale everything to the cluster-wide offering".
• Where the resize happens: KubernetesClusterScaleWorker#scaleCluster sets scaleClusterDefaultOffering = true when DEFAULT is present and loops over CONTROL, ETCD, WORKER. Every node group whose current offering differs from DEFAULT gets scaleKubernetesClusterOffering, which resizes its VMs and commits the offering via updateKubernetesClusterEntryForNodeType. CONTROL and ETCD are handled before WORKER. For WORKER with a larger size, the offering is changed first and then scaleKubernetesClusterSize adds VMs; in the autoscaling branch, autoScaled && isNodeOfferingScalingNeeded also resizes the workers.
• Which offering new VMs get: KubernetesClusterResourceModifierActionWorker#createKubernetesNode uses getServiceOfferingForNodeTypeOnCluster(WORKER, cluster) (worker offering, else cluster-wide). That part is correct; the problem is that the step above has already overwritten the worker offering.
• How the cluster-autoscaler calls CloudStack: kubernetes/autoscaler, cluster-autoscaler/cloudprovider/cloudstack/service/cks.go: scale-up scaleKubernetesCluster {id, size}, scale-down scaleKubernetesCluster {id, nodeids}. Neither passes an offering.
• Why the UI is not affected: ui/src/views/compute/ScaleKubernetesCluster.vue shows the cluster-wide serviceofferingid field only while the cluster has no workerofferingid / controlofferingid / etcdofferingid, and pre-fills every offering field with the current value; handleSubmit sends them as serviceofferingid or nodeofferings[], so a UI request always names an offering.
• Why the cluster-wide offering goes stale: per-group scales update control_service_offering_id / worker_service_offering_id only; service_offering_id changes only on a cluster-wide scale. The UI "Compute Offering" field shows this value, so it is out of date as well.
• Likely cause of the lost autoscaling flag: autoscaleCluster saves the autoscaling fields, then scaleKubernetesClusterOffering → updateKubernetesClusterEntryForNodeType saves the worker's cluster object, which was loaded earlier and still holds the old autoscaling values.
• Suggested fix:
  1. Do not add a DEFAULT entry when the request contains neither serviceofferingid nor nodeofferings; size, nodeids and autoscaling-only requests then plan no offering change, and new workers keep using workerofferingid.
  2. Reload the cluster entry after autoscaleCluster (or update only the offering, count and capacity columns in updateKubernetesClusterEntryForNodeType) so later saves cannot revert the autoscaling fields.
  3. Optional: keep service_offering_id consistent after per-group scales, or stop using it as a fallback once per-group offerings exist.

Environment

Found during functional testing of #14223 (CKS live-resize capacity reconciliation) on a 2 management server + 2 KVM host lab, Kubernetes 1.33.1. The code paths above are unchanged on main (checked at af9764b); #14223 does not touch them.