Describe the bug
When we extract a large number of hosts or vulnerabilities, we see a large memory consumption.
Expected behavior
The memory consumption must be constant and predictable during the data recovery.
Current behavior
Since we have a huge and increasing number of instances, the memory consumption increases to the point of reaching more than 10Go of RAM memory.
Type of error we have from AWS step function :
{
"errorType": "Runtime.ExitError",
"errorMessage": "RequestId: XXXX Error: Runtime exited with error: signal: killed"
}Possible solution
During our investigation, we have concluded the function execute is the root cause of the consumption. One way to solve this is to create a generator instead of a list that increases in size at each loop.
Steps to reproduce
- Have more than 6000 hosts onboarded on primsa with vulnerabilities.
- Run a script on an environment with a memory size limited to 4 Go.
- Constat the crash at one point, with a MemoryError or an ExitError
Context
In our enterprise context, we want to create a useful dashboard to animate our team around vulnerabilities and patch these vulnerabilities.
Your Environment
- Version used: The latest eu SaaS version
- Environment name and version: Container image on python:3.11.6-slim-bullseye with prismacloud-api==5.2.8 and aws-lambda-powertools==2.26.0
Describe the bug
When we extract a large number of hosts or vulnerabilities, we see a large memory consumption.
Expected behavior
The memory consumption must be constant and predictable during the data recovery.
Current behavior
Since we have a huge and increasing number of instances, the memory consumption increases to the point of reaching more than 10Go of RAM memory.
Type of error we have from AWS step function :
{ "errorType": "Runtime.ExitError", "errorMessage": "RequestId: XXXX Error: Runtime exited with error: signal: killed" }Possible solution
During our investigation, we have concluded the function execute is the root cause of the consumption. One way to solve this is to create a generator instead of a list that increases in size at each loop.
Steps to reproduce
Context
In our enterprise context, we want to create a useful dashboard to animate our team around vulnerabilities and patch these vulnerabilities.
Your Environment