Skip to content

CKS: a single Kubernetes node can be scaled from the Instances view, leaving the cluster inconsistent #14553

Open

Description

@andrijapanicsb

What happens, in plain words

Every node of a CloudStack Kubernetes Service (CKS) cluster is an ordinary CloudStack instance, so it also appears in the Instances list, for the admin and for the user who owns the cluster. CKS is designed to resize whole node groups only (all workers together, all control nodes together), through the cluster's own Scale action.

Nothing stops someone from opening one worker in the Instances view and using the instance Scale button (or calling scaleVirtualMachine directly). The resize goes through, live on a running KVM node, and the cluster is left in a state CKS never expects:

  • one worker is bigger (or smaller) than the others;
  • the cluster's recorded offering and its CPU/RAM totals no longer match the nodes;
  • Kubernetes keeps reporting the node's old CPU and memory, because the work that makes Kubernetes see added resources only runs when the resize goes through the cluster scale (see CKS: reconcile node capacity after live resize #14223);
  • later cluster operations act on that inconsistent picture. For example, a cluster resize to an offering smaller than the oversized node is refused, and new workers are created with the recorded worker offering, not the size of the node that was changed by hand.

CloudStack already protects CKS nodes against other per-instance actions. Deleting a node, changing its affinity groups, restoring it and unmanaging it are all refused with a message pointing to the cluster's own actions. Scaling is missing from that list.

Steps to reproduce

  1. Create a CloudManaged CKS cluster (1 control, 2 workers) on offering A, on a dynamically scalable template, and wait for Running.

  2. In the UI go to Compute → Instances, open one worker node and click Scale, or call
    scaleVirtualMachine id=<worker VM> serviceofferingid=<B> with a bigger offering B.

  3. Observe (confirmed by manual test):

    • the request succeeds without any check: in the test, worker cksbug-cks-node-1a199027059 went live (state stayed Running) from cksbug-small (2 CPU / 2048 MB) to cksbug-large (3 CPU / 3072 MB); the other worker stayed on cksbug-small;
    • the Kubernetes clusters list, the cluster details page and listKubernetesClusters all still show the compute offering A and the old totals. In the test: offering cksbug-small, 6 CPU cores / 6144 MB, while the nodes actually had 7 CPU / 7168 MB (2/2048 + 2/2048 + 3/3072).

    Expected but not yet checked: kubectl get node <worker> -o jsonpath='{.status.capacity}' still shows the old CPU and memory.

Expected

  • For a node of a CloudManaged CKS cluster, scaleVirtualMachine (and any other per-instance offering change) is refused with a clear message, for example: "Instance is a part of a Kubernetes cluster: . Use the Scale action of the Kubernetes cluster to change the offering of its nodes."
  • The Scale button is hidden (or disabled with that explanation) for CKS nodes in the Instances view.
  • Nodes of ExternalManaged clusters keep working as today, the same as the existing delete and affinity-group checks.
  • CKS's own cluster scale keeps working; it resizes the node VMs through the same internal method.

Technical details

  • API, no guard: ScaleVMCmd → UserVmManagerImpl#upgradeVirtualMachine(ScaleVMCmd) → upgradeVirtualMachine(Long, Long, Map) has no check for CKS nodes.
  • Existing guards to reuse: KubernetesServiceHelper#checkVmCanBeDestroyed and #checkVmAffinityGroupsCanBeUpdated (in KubernetesServiceHelperImpl) refuse the action when userVm.getUserVmType() is CKS_NODE and the VM is mapped to a cluster that is not ExternalManaged. UserVmManagerImpl#isVMPartOfAnyCKSCluster is used to block restoreVirtualMachine and, in UnmanagedVMsManagerImpl, unmanaging. A checkVmCanBeScaled along the same lines is the natural fix.
  • Where to put the check: in the API-facing path (upgradeVirtualMachine(ScaleVMCmd), and also the stopped-VM offering change behind changeServiceForVirtualMachine / UpgradeVMCmd, which should be checked too), not in the internal upgradeVirtualMachine(Long, Long, Map[, Runnable]). CKS's cluster scale (KubernetesClusterScaleWorker and, with CKS: reconcile node capacity after live resize #14223, KubernetesClusterNodeOfferingResizer) calls the internal method for every node and must keep working.
  • UI: in ui/src/config/section/compute.js the scaleVirtualMachine action's show condition excludes External hypervisors, LXC while running and sharedfsvm, but not cksnode. Another action in the same file already excludes record.vmtype !== 'cksnode'; the same condition can be added to scaleVirtualMachine (and to the stopped-VM change-offering action if it is not excluded already).

Environment

Confirmed by manual test on a 2 management server + 2 KVM host lab (Kubernetes 1.33.1, build of #14223 at a8a500f): a live Scale of one worker node from the Instances view succeeded without any check, and the cluster's recorded offering and totals stayed unchanged. The code paths involved are unchanged by #14223.

Image Image Image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions