Repository navigation
CKS: reconcile node capacity after live resize - #14223
andrijapanicsb wants to merge 15 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14223 +/- ##
============================================
+ Coverage 20.45% 20.87% +0.42%
- Complexity 21415 22137 +722
============================================
Files 6487 6503 +16
Lines 589303 592255 +2952
Branches 72245 72678 +433
============================================
+ Hits 120525 123638 +3113
+ Misses 455643 455178 -465
- Partials 13135 13439 +304
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
andrijapanicsb
changed the title
|
@ACSHomeBot package G |
Packaging results:
Test packages are available at: |
🔴 Test Coverage Grade:
|
| Metric | Value |
|---|---|
| Line coverage | 26.00% |
| Branch coverage | 20.40% |
Grade Scale
| Grade | Line Coverage | Meaning |
|---|---|---|
| 🟢 A | ≥ 80% | Excellent - this code sleeps well at night 😴 |
| 🟡 B | 60-79% | Good - almost there, don't stop now 😉 |
| 🟠 C | 40-59% | Acceptable - your code is wearing a seatbelt, but no airbags 😬 |
| 🔴 D | 20-39% | Marginal - boldly shipping where no test has gone before 🖖 |
| ⛔ F | < 20% | Failing - tests? what tests? 🔥 |
Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run
40b50b0 to
7e5a95a
Compare
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Queued for worker G. Jobs ahead: 0. |
|
Package building started on worker G. |
Packaging results:
Test packages are available at: |
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Queued for worker G. Jobs ahead: 0. |
|
Package building started on worker G. |
Packaging results:
Test packages are available at: |
andrijapanicsb
changed the title
|
@blueorangutan package |
|
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
@DaanHoogland, unrelated to all this work - share with @rajujith some of the PR fixes that I've raised against virt-v2v that are critical for VMware migrations, I'm sure Jithin will appreciate those - https://github.com/libguestfs/virt-v2v/pulls?q=is%3Apr+author%3Aandrijapanicsb - all merged, and he can just take the source DEB/RPM (whatever your worker uses) and backport those, together with the stupid FSTRIM that virt-v2v does - I had a 30TB migration over weekened, and FSTRIM added 2h of duration inside virt-v2v on top of 200 seconds that it really took to convert a WIndows VM with 12 disks/30TB of space. Perhaps he wants to also backport the --no-fstrim option from upstream. @rajujith ping me if you want to test openvddk.com - it's NBD/NBDSSL and HotAdd - that is a 1:1 binary compatible replacement for VMware's VDDK - completely free. I'm sure you might want to test it in your in-house product. |
@andrijapanicsb , looks like a compatitor of the one mentioned in https://www.networkworld.com/article/4241265/fluidcloud-replaces-vddk-with-its-own-disk-copy-engine-for-vmware-exits.html needs a comparison? (in a discussion item, probably not here) |
Packaging results:
Test packages are available at: |
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Completed. Packaging results. |
|
code looks very clean now @andrija, thnks. |
@DaanHoogland you know me well enough - don't compare apples and oranges... ;) "The FluidCloud engine exports the VM disk using Network File Copy (NFC) through standard application programming interfaces (API) on the VMware ESXi hypervisor. It then streams the virtual machine disk (VMDK) file over the network to the destination. Omar said the export captures everything, with no functionality loss. The engine does not do delta sync, so every export is a full export of the VM." Their stuff is a workaorund but uses NFC, My artefact is the libvixdisk.so.8.0.3 that you replace and off you go (HotAdd, NBD/NBDSSL (via NFC) - 100% compatible with VDDK = so all tools previously designed to work with VMware VDDK now works with OpenVDDK. Check the sites, numbers, compatibility, etc. I will share this publicly soon. Free and open source. |
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Completed. Packaging results. |
Packaging results:
Test packages are available at: |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19802 |
Packaging results:
Test packages are available at: |
|
@blueorangutan test |
|
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-17329)
|
Defer node preparation until a live CPU or memory change can be dispatched. On a retry of the committed offering, reconcile owned pending work without changing individual VM offerings or operator cordons.
|
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Completed. Packaging results. |
Packaging results:
Test packages are available at: |
Functional and regression testing on the final commit
|
| Area | Tests | Result |
|---|---|---|
| Live increases: worker CPU-only, RAM-only, combined; control CPU-only (API) and RAM-only (UI); both node types in one call; cluster-wide offering; scale-out + offering change in one call | 13 | PASS |
| Jobs executed on either management server (node access and reconciliation from MGMT_B) | – | PASS |
| Same committed offering again, same-capacity offering change, stopped-cluster offering change | – | PASS: no node reads beyond one get node, no markers, no kubelet restart |
| Node preparation only after placement: control VM live-migrated for capacity, then node read, then the single preparation patch | – | PASS |
| Failure paths: control plane unreachable, node SSH unreachable, placement/capacity failure (no pending marker left), placement failure with earlier pending work (kept byte-identical), guest already above target, kubelet failure after resize (pending + CKS cordon kept, second worker untouched, retry finishes), scale timeout, operator cordon with failure, stale ownership marker, foreign marker, committed-offering retry with a mixed-offering node, VM stopped outside CKS, concurrent scale, parameter validation | 15 | PASS |
Regression: scale-out, scale-in (size and nodeids), migrate after resize, stop/start, deleted offering in totals, plain (non-CKS) VM live/stopped scale, UI totals with mixed offerings |
7 | PASS |
| Autoscaling toggle | 1 | Pre-existing issue, same code on main → #14552 |
In every live resize: no VM reboot, no pod restart or UID change, DNS/HTTP service checks passed, the API server answered every readiness poll during control-node resizes, cluster metadata was committed only after all nodes succeeded.
Not tested: VPC tier / Netris and shared (direct-access) networks (node access changed in 306cc2e), multi-control HA, dedicated etcd, hypervisor rejection after dispatch (covered by unit tests), Kubernetes version upgrade after resizes.
Found during testing, pre-existing and independent of this PR:
- CKS: scale requests without an offering (autoscaler, size, enabling autoscaling) resize every node to an outdated cluster-wide offering #14552: scale requests without an offering (cluster-autoscaler,
size, enabling autoscaling) resize every node to an outdated cluster-wide offering - CKS: a single Kubernetes node can be scaled from the Instances view, leaving the cluster inconsistent #14553: a single CKS node can be scaled from the Instances view, leaving the cluster inconsistent
The Trillian smoke run on a8a500f is still to be done.
|
@blueorangutan package |
|
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✖️ suse15. SL-JID 19872 |
Context
#13226 added live compute-offering changes for CKS nodes, building on #12975. The VM could gain CPU/RAM while Kubernetes still reported the old capacity. Operators had to activate guest resources and restart
kubeletmanually. #14213 tracks that gap.Changes
CKS now activates and verifies the added guest CPU/RAM, restarts
kubeletwhen needed, and waits for the Node to report the new capacity and allocatable resources before completing the scale operation.Nodes are handled one at a time without draining resident Pods. While a node's resize is in progress, the node carries a
cloudstack.apache.org/cks-live-resize-pendingannotation and, unless an operator had already cordoned it, a CKS cordon. Both are set and removed with single conditional Node updates. If a scale fails, the annotation stays, and the CKS cordon stays too if the VM already reached the new offering. A retry of the same scale finishes that node before it changes any other. An existing operator cordon is preserved.QA also exposed a pre-existing cluster CPU/RAM accounting bug, not a regression from this PR. Totals are now calculated from the configured per-node-type offerings and counts, so the API and UI reflect mixed control/worker offerings correctly.
Functional and regression testing
Test environment: Cloud-managed KVM, isolated network, published packages from
f92e52d736b9, before the review changes.The tests will be repeated on the final commit - see comment #14223 (comment)
VPC/direct-access and multi-control HA were not tested. Dedicated etcd VMs are resized the same way as on main; they are not Kubernetes nodes, so nothing is reconciled on them (not live-tested). The separate existing
nodeids-only scale-in issue found during QA is tracked in #14458, with its fix in #14459.UI evidence
Successful scale task and correct aggregate capacity:
Scale request and result after Refresh
Control RAM: 2048 → 3072 MiB; worker unchanged at 3 CPU / 4096 MiB.
After Refresh: 6 CPU / 7168 MiB.