High load on management server due to stale KVM agent

[rajujith]

ISSUE TYPE

• Bug Report
• Improvement Request
• Other

COMPONENT NAME

KVM, Management Server

CLOUDSTACK VERSION

4.18.1, Any

CONFIGURATION

KVM

OS / ENVIRONMENT

Rocky 8 KVM, Management server

SUMMARY

Removed KVM host with agent management server IP configured pointing to a new management server leads to high CPU load on the management server.

We can see the below logs in the management server:

2024-01-16 10:14:00,089 WARN  [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-381:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.1.35.66:46493. Please investigate this connection.
2024-01-16 10:14:00,099 WARN  [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-29:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.1.35.66:46475. Please investigate this connection.

STEPS TO REPRODUCE

1. Remove a KVM host. Restart the agent. 
2. Reinstall the management server or run cloudstack-setup-database
3. Watch the Management server CPU load. 

OR

1. Update the agent.properties of a KVM host to point it to an existing management server where the host is not added.
2. Restart the cloudstack agent.
3. Wait for a few minutes until the management server CPU load goes very high. 

[DaanHoogland]

this is a bad bug, @rajujith . I know some people are explicitely using this scenario in operations, so it should be supported. Can you confirm the the host/agent is not added to the cloud-installation?

[weizhouapache]

this is a bad bug, @rajujith . I know some people are explicitely using this scenario in operations, so it should be supported. Can you confirm the the host/agent is not added to the cloud-installation?

@DaanHoogland
the issue is easy to reproduce: reinstall cloudstack database on management server
the running cloudstack-agent will try to connect the management server with old configuration, again and again.

the workaround is also very simple: stop cloudstack-agent on kvm hosts, and add hosts on cloudstack UI if needed.

[rajujith]

this is a bad bug, @rajujith . I know some people are explicitely using this scenario in operations, so it should be supported. Can you confirm the the host/agent is not added to the cloud-installation?

@DaanHoogland @weizhouapache Agent is not added to cloudstack and the workaround is to stop the agent. However it would be nice to have it fixed. There could be scenarios in operations where the host is not added instantly after starting the agent. The load increase can bee seen within a few minutes with one agent. With many agents I suspect this could be much higher.

[DaanHoogland]

should I up the level from major to critical on this, @rajujith @weizhouapache ?

[weizhouapache]

should I up the level from major to critical on this, @rajujith @weizhouapache ?

@DaanHoogland
I do not think it is a major/critical issue.
when users reinstall an environment, they should better stop all running cloudstack services (management, usage, agent).

[DaanHoogland]

right, there is an issue when the management is reachable to rogue agents, but that is an operational issue. I was thinking we must guard against that.

[rajujith]

Rogue agents shoudn't have the ability to create high load on the management server and crash it.