fix domain cleanup when a removed account owns templates

[Damans227]

Fixes #13612

Deleting a domain with cleanup fails when an account in it owns a template. During cleanup the account is removed first, then its templates are deleted, and the template's resource-count recalculation used findById (which skips removed rows) and threw "Unable to find account". This makes the recalculation include removed accounts, so the domain deletes in one pass. The caller privilege check is unchanged.

Test:

Create a domain, an account under it, register a template under that account, then delete the domain with cleanup.

Before: fails.

After: deletes in one pass. Log from the delete job:

Removed account u1d1 (id=5)
Delete template: u1d1-template (id=202)
Updating resource Type = template count for Account with id = 5 Operation = decreasing Amount = 1
Complete async job-37, jobStatus: SUCCEEDED, result: {"success":"true"}

[Damans227]

@blueorangutan package

[blueorangutan]

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

❌ Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 19.65%. Comparing base (17e5947) to head (9107b89).
⚠️ Report is 279 commits behind head on main.

Files with missing lines	Patch %	Lines
.../cloud/resourcelimit/ResourceLimitManagerImpl.java	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13630      +/-   ##
============================================
+ Coverage     17.69%   19.65%   +1.96%     
- Complexity    15829    19806    +3977     
============================================
  Files          5925     6368     +443     
  Lines        533534   575107   +41573     
  Branches      65267    70376    +5109     
============================================
+ Hits          94412   113043   +18631     
- Misses       428444   449770   +21326     
- Partials      10678    12294    +1616     

Flag	Coverage Δ
uitests	3.41% <ø> (-0.28%)	⬇️
unittests	20.93% <83.33%> (+2.15%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18595

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[DaanHoogland]

clgtm

[kiranchavala]

@Damans227

The global setting "account.cleanup.interval" is not being considered

Cloudstack is directly setting the removed date for the account when the domain is deleted.

Ideally, the feild "cleanup_needed" is set to 1 whenever the domain is deleted

Before deleting the domain

*************************** 6. row ***************************
             id: 6
   account_name: u1d1
           uuid: f979d2d1-c0f6-4e40-8bef-2619f59352bd
           type: 0
        role_id: 4
      domain_id: 3
          state: enabled
        created: 2026-07-21 04:00:54
        removed: NULL
 cleanup_needed: 0
 network_domain: NULL
default_zone_id: NULL
        default: 0
 api_key_access: NULL

After domain is deleted

*************************** 6. row ***************************
             id: 6
   account_name: u1d1
           uuid: f979d2d1-c0f6-4e40-8bef-2619f59352bd
           type: 0
        role_id: 4
      domain_id: 3
          state: removed
        created: 2026-07-21 04:00:54
        removed: 2026-07-21 04:13:56
 cleanup_needed: 0
 network_domain: NULL
default_zone_id: NULL
        default: 0
 api_key_access: NULL
6 rows in set (0.00 sec)

logs

[root@pr13630-t16583-kvm-ol8-mgmt1 ~]# cat  /var/log/cloudstack/management/management-server.log |grep -i "logid:e0453c6e"
2026-07-21 04:13:56,827 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-1:[ctx-de93b47d, job-40]) (logid:e0453c6e) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd","cmdInfo":"{\"cleanup\":\"true\",\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"EFRx1uNQjKqEw_wAtjzf6gPW3PA\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"1556\",\"id\":\"1e603f92-ff76-45f0-bf69-28dac9fe0206\",\"ctxDetails\":\"{\\\"interface com.cloud.domain.Domain\\\":\\\"1e603f92-ff76-45f0-bf69-28dac9fe0206\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"1e603f92-ff76-45f0-bf69-28dac9fe0206\",\"cmdEventType\":\"DOMAIN.DELETE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":40,"initMsid":32989241147872,"instanceId":3,"instanceType":"Domain","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"e0453c6e-77ea-4b67-b0c6-d11b6d0181e1"}
2026-07-21 04:13:56,850 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Access granted to Account [{"accountName":"admin","id":2,"uuid":"3cfba268-8128-11f1-ba5a-1e00e80001e0"}] to Domain {"id":3,"name":"d1","path":"\/d1\/","uuid":"1e603f92-ff76-45f0-bf69-28dac9fe0206"} by AffinityGroupAccessChecker
2026-07-21 04:13:56,854 DEBUG [c.c.u.DomainManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Marking domain Domain {"id":3,"name":"d1","path":"\/d1\/","uuid":"1e603f92-ff76-45f0-bf69-28dac9fe0206"} as Inactive before actually deleting it
2026-07-21 04:13:56,868 DEBUG [c.c.u.DomainManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Cleaning up domain Domain {"id":3,"name":"d1","path":"\/d1\/","uuid":"1e603f92-ff76-45f0-bf69-28dac9fe0206"}
2026-07-21 04:13:56,873 DEBUG [c.c.u.DomainManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Deleting account Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}] as a part of domain Domain {"id":3,"name":"d1","path":"\/d1\/","uuid":"1e603f92-ff76-45f0-bf69-28dac9fe0206"} cleanup
2026-07-21 04:13:56,887 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Removed account Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}]
2026-07-21 04:13:56,903 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Successfully deleted snapshots directories for all volumes under account Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}] across all zones
2026-07-21 04:13:56,918 INFO  [c.c.t.HypervisorTemplateAdapter] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Delete template: Template {"format":"QCOW2","id":203,"name":"testcleanup-template","uniqueName":"203-6-2a16990e-f687-3b7d-862b-7320b9d73881","uuid":"2400ae94-8dfa-4e21-b30a-1736e414079a"} from image store: ImageStore {"id":1,"name":"pr13630-t16583-kvm-ol8-sec1","uuid":"ec2bb673-ef0e-43f6-8dd0-8f8a7f090176"}
2026-07-21 04:13:56,921 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) template Template {"format":"QCOW2","id":203,"name":"testcleanup-template","uniqueName":"203-6-2a16990e-f687-3b7d-862b-7320b9d73881","uuid":"2400ae94-8dfa-4e21-b30a-1736e414079a"} with id 203 is already in store:ImageStore {"id":1,"name":"pr13630-t16583-kvm-ol8-sec1","uuid":"ec2bb673-ef0e-43f6-8dd0-8f8a7f090176"}, type: Image
2026-07-21 04:13:56,946 DEBUG [c.c.h.XenServerGuru] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) We are returning the default host to execute commands because the command is not of Copy type.
2026-07-21 04:13:56,946 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Wait time setting on org.apache.cloudstack.storage.command.DeleteCommand is 1800 seconds
2026-07-21 04:13:56,947 DEBUG [c.c.a.m.ClusteredAgentAttache] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Seq 1-5467932897581203469: Routed from 32989241147872
2026-07-21 04:13:56,950 DEBUG [c.c.a.t.Request] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Seq 3-5467932897581203469: Sending  { Cmd , MgmtId: 32989241147872, via: 3(s-2-VM), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/6/203/04a57c57-5592-33d9-9086-54514c37f1f5.qcow2","origUrl":"http://10.0.3.130/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2","uuid":"2400ae94-8dfa-4e21-b30a-1736e414079a","id":"203","format":"QCOW2","accountId":"6","checksum":"{SHA-512}f78abd3c8a3e92b43ab8ba7916bb1635303adfbc8f73f0729cd4f162f7a314bea59a8d96ecb617226e4d7e73be73f205b9c2e301c1cf6a9ee3e2f3385bb085be","hvm":"true","displayText":"testcleanup-template","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"NFS://10.0.32.4/acs/secondary/pr13630-t16583-kvm-ol8/pr13630-t16583-kvm-ol8-sec1","_role":"Image"}},"name":"203-6-2a16990e-f687-3b7d-862b-7320b9d73881","size":"(3.50 GB) 3758096384","hypervisorType":"KVM","bootable":"false","uniqueName":"203-6-2a16990e-f687-3b7d-862b-7320b9d73881","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"deleteChain":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-07-21 04:13:57,035 DEBUG [c.c.a.t.Request] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Seq 3-5467932897581203469: Received:  { Ans: , MgmtId: 32989241147872, via: 3(s-2-VM), Ver: v1, Flags: 10, { Answer } }
2026-07-21 04:13:57,064 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Updating resource Type = template count for Account with id = 6 Operation = decreasing Amount = 1
2026-07-21 04:13:57,073 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Access granted to Account [{"accountName":"admin","id":2,"uuid":"3cfba268-8128-11f1-ba5a-1e00e80001e0"}] to Domain {"id":3,"name":"d1","path":"\/d1\/","uuid":"1e603f92-ff76-45f0-bf69-28dac9fe0206"} by AffinityGroupAccessChecker
2026-07-21 04:13:57,074 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Verifying whether the caller has the correct privileges based on the user's role type and API permissions: Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}]
2026-07-21 04:13:57,075 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Clearing tagged resource limits and counts which do not match host tags: [], storage tags: [] for null
2026-07-21 04:13:57,107 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Expunging # of Instances (Account=Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}]): 0
2026-07-21 04:13:57,119 INFO  [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) deleteAccount: Deleted 1 network groups for account Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}]
2026-07-21 04:13:57,120 INFO  [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) deleteAccount: Deleted 0 affinity groups for account Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}]
2026-07-21 04:13:57,121 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Deleting networks for account Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}]
2026-07-21 04:13:57,123 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Deleting vpcs for account Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}]
2026-07-21 04:13:57,129 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Deleting site-to-site VPN customer gateways for account Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}]
2026-07-21 04:13:57,140 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Account specific Virtual IP ranges are successfully released as a part of account Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}] cleanup.
2026-07-21 04:13:57,149 INFO  [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) deleteAccount: Released 0 dedicated guest vlan ranges from account Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}]
2026-07-21 04:13:57,167 DEBUG [o.a.c.k.KMSManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) No KMS keys found for account 6
2026-07-21 04:13:57,167 INFO  [c.c.u.AccountManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Cleanup for account Account [{"accountName":"u1d1","id":6,"uuid":"f979d2d1-c0f6-4e40-8bef-2619f59352bd"}] is not needed.
2026-07-21 04:13:57,169 DEBUG [c.c.u.DomainManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Deleting networks for domain Domain {"id":3,"name":"d1","path":"\/d1\/","uuid":"1e603f92-ff76-45f0-bf69-28dac9fe0206"}
2026-07-21 04:13:57,194 DEBUG [c.c.u.DomainManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Domain specific Virtual IP ranges  are successfully released as a part of domain Domain {"id":3,"name":"d1","path":"\/d1\/","uuid":"1e603f92-ff76-45f0-bf69-28dac9fe0206"} cleanup.
2026-07-21 04:13:57,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Complete async job-40, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.SuccessResponse/null/{"success":"true"}
2026-07-21 04:13:57,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Publish async job-40 complete on message bus
2026-07-21 04:13:57,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Wake up jobs related to job-40
2026-07-21 04:13:57,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Update db status for job-40
2026-07-21 04:13:57,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:[ctx-de93b47d, job-40, ctx-63abd430]) (logid:e0453c6e) Wake up jobs joined with job-40 and disjoin all subjobs created from job- 40
2026-07-21 04:13:57,244 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-1:[ctx-de93b47d, job-40]) (logid:e0453c6e) Done executing org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd for job-40
2026-07-21 04:13:57,245 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-1:[ctx-de93b47d, job-40]) (logid:e0453c6e) Remove job-40 from job monitoring