Enable System Parameters Globally

[popescu-v]

Problem Statement

The max_cores Core API functions parameter allows to restrict the number of CPU cores per Core API function call (as per issue #585) and the n_cores does the same per Sklearn estimator instance (as per issue #605).

The same is true for the memory_limit_mb and temp_dir, at the Core API function level.

However, this constraint needs to be enforced at each function call or estimator instantiation and there is no officially-supported mechanism for enforcing it at a global level.

Details

The goal of this issue is to provide a global mechanism for setting the maximum CPU core number, the maximum memory used, and the temporary directory. These settings must:

• take effect for the remainder of the current Python process;
• until they are globally disabled;
• unless they are overridden via the local settings (see issues Use KHIOPS_PROC_NUMBER when max_cores is set #585 and, for the CPU core number only, Issue with core limit option in Khiops Python library not working with scikit-learn interface or environment variable #605).

Specifications

A set of functions are added to the Core API, for:

• updating the KHIOPS_PROC_NUMBER, KHIOPS_MEMORY_LIMIT and KHIOPS_TMP_DIR environment variables:
  • the values of these variables in the environment the current Python process runs in are memorized;
  • the values of these variables are updated, if set to non-None values;
  • the values of these variables are reset to the values they had in the environment the current Python process runs in, at the time the latter was launched, if they are set to None (this includes the case the functions are called with no parameters; see below for the concrete signatures of these functions);
• getting the current values of these environment variables.

Function signatures and docstrings:

# The functions are defined in the khiops.core.api module
def set_default_max_cores(max_cores=None):
    """Sets the maximum number of cores used by Khiops Core API function calls

    The maximum number of cores used by Khiops is set globally, in that all subsequent Khiops
    Core API calls use `max_cores` CPU cores at most, unless their own `max_cores`
    argument is passed, in which case it overrides the global setting.

    This function modifies the environment Khiops runs in, so that it uses at most `max_cores`
    CPU cores.

    Parameters
    ------------
    max_cores : int, optional
        Maximum number of CPU cores to be used by subsequent Khiops Core API function calls.
        If not set, then the function sets the maximum number of cores to the value it had prior to
        launching the Khiops Python process.
    """

def get_default_max_cores():
    """Gets the maximum number of cores used by Khiops Core API function calls"""

def set_default_memory_limit_mb(memory_limit_mb=None):
    """Sets the maximum amount of memory (megabytes) used by Khiops Core API function calls

    The maximum amount of memory used by Khiops is set globally, in that all subsequent Khiops
    Core API calls use `max_memory_mb` megabytes of memory at most, unless their own
    `max_max_memory_mb` argument is passed, in which case it overrides the global setting.

    This function modifies the environment Khiops runs in, so that it uses at most `max_memory_mb`
    megabytes of memory.

    Parameters
    ------------
    memory_limit_mb : int, optional
        Maximum amount of memory (megabytes) to be used by subsequent Khiops Core API
        function calls. If not set, then the function sets the maximum amount of memory
        (megabytes) to the value it had prior to launching the Khiops Python process.
    """

def get_default_memory_limit_mb():
    """Gets the maximum amount of memory (megabytes) used by Khiops Core API function calls"""

def set_default_temp_dir(temp_dir=None):
    """Sets the path to the temporary directory used by Khiops Core API function calls

    The temporary directory path used by Khiops is set globally, in that all subsequent Khiops
    Core API calls write Khipops temporary files in the `temp_dir` directory, unless their own
    `temp_dir` argument is passed, in which case it overrides the global setting.

    This function modifies the environment Khiops runs in, so that it writes its temporary files
    in the `temp_dir` directory.

    Parameters
    ------------
    temp_dir : str, optional
        Path to the temporary directory to be used by subsequent Khiops Core API function calls.
        If not set, then the function sets the temporary directory path to the value it had prior to
        launching the Khiops Python process.
    """

def get_default_temp_dir():
    """Gets the path to the temporary directory used by Khiops Core API function calls"""

Implementation

The priority order of these parameters is (highest to lowest):

1. values set in the Core API calls (e.g. train_predictor(..., max_cores=10, ...));
2. default values set via the khiops.core.api.set_default_{max_cores,memory_limit_mb,temp_dir};
3. values given by the KHIOPS_{PROC_NUMBER,MEMORY_LIMIT,TMP_DIR} environment variables, if set.

This policy can be implemented as follows:

1. Declare a current_environ variable of type dictionary and set it to a copy of os.environ;
2. if a khiops.core.api.set_default_* Core API function is called with an argument that is not None, then:
   • set the relevant KHIOPS_* environment variable key in the current_environ dictionary;
   • create new KhiopsLocalRunner instance in the environment specified by this new dictionary (copy of os.environ, initialized in the previous steps) and call khiops.core.internals.runner.set_runner() on it; details:
     • protect this call with a mutex;
     • add optional env parameter to KhiopsLocalRunner (default: None) so that the runner can be initialized (and khiops_env --env ran into) this specific environment (KhiopsLocalRunner(env=current_environ)); os.environ is used (as it is the case currently) if env is not given or is set to None;
3. if a khiops.core.api.set_default_* Core API function is called without an argument (hence the default value None is used), then:
   • check the relevant KHIOPS_* environment variable in os.environ:
     • if set, then restore the original variable value: current_environ["KHIOPS_*"] = os.environ["KHIOPS_*"];
     • else, delete the variable: del current_environ["KHIOPS_*"]
   • create new KhiopsLocalRunner instance in the environment specified by this new dictionary (copy of os.environ, initialized in the previous steps) and call khiops.core.internals.runner.set_runner() on it; details:
     • protect this call with a mutex;
     • add optional env parameter to KhiopsLocalRunner (default: None) so that the runner can be initialized (and khiops_env --env ran into) this specific environment.

Notes

• This implementation changes the initialization of the "internal" KhiopsLocalRunner class, by adding an optional env parameter, which is used to launch the khiops_env --env subprocess within the environment given by env.
• This implementation does not change the original, global environment in os.environ; neither does it change the khiops_env script.
• For memory_limit_mb and temp_dir, no further modifications are needed, as they can continue to be settable, on a per-Core API call basis (local) at the scenario level.
• For max_cores, the local per-Core API call setting should still override the default setting, but this is done via the same mechanism used for overriding the global setting from os.environ; hence, no further change is need here either, with respect to issue Use KHIOPS_PROC_NUMBER when max_cores is set #585 .
• The only input parameter checks done in the khiops.core.api.set_default_* functions pertain to:
  • their types: int, str
  • their values (> 0, non-empty)
    No upper-bound checks are done on max_cores and memory_limit_mb; this should be handled by khiops_env in the same way as when users set KHIOPS_PROC_NUMBER and KHIOPS_MEMORY_LIMIT in the environment before launching Khiops.

[popescu-v]

See also #585 (comment).