Implement a `max_events` policy for the `EventsCBGExecutor`

[skyegalaxy]

Description

Implement a max_events option, which will (at the minimum) warn the user if the number of ready events in the queue exceeds this amount and optionally prevent the queue from growing unbounded.

Motivation

Currently, using the EventsCBGExecutor can result in unbounded memory growth if the system is slowed down and ready, un-serviced events stack up in the queue.

Design / Implementation Considerations

What is the desired behavior if we reach max_events? Would we want the policy to be user-configurable (don't accept new events, remove the oldest event, do nothing but warn)?

Additional Information

No response

[fujitatomoya]

Would we want the policy to be user-configurable (don't accept new events, remove the oldest event, do nothing but warn)?

i say user-configurable would be best provided by system. it will be hard to guess what user wants to behave. but in default, i think it should be defensive and safe, that user must be able to see this needs to be configurable by their requirement. that said, don't accept new events with waring would be the default behavior? 🤔

[jmachowinski]

Dropping events is problematic as it results in other memory leaks.
What about two thresholds a low one for warnings and a ridiculous high one for termination ?

[AdityaJindal07]

Hi, I had a question about the intended behavior of max_events.

Since ready events are currently queued in FirstInFirstOutCallbackGroupHandle::ready_entities, should max_events be enforced per callback group queue, or as a global limit across all ready events in the executor?

Also, when the limit is reached, should the implementation only emit a warning, or should it prevent additional events from being queued?

Thanks.