Context:
Nvidia driver daemonset deployed by GPU Operator (for non-pre-compiled driver mode) has a step to pull some Linux OS distro images at run-time (during driver pod bringup) from a Linux distro packages repository (e.g. an apt packages repository for Unbutu), location of this repository is over the network (e.g. by default, archive.ubuntu.com over the Internet, or in case of air-gap, a local mirror setup by the user, till over the network).
(The packages being fetched over the network and installed inside the driver pod, e.g. linux-kernel headers etc., are required to compile the driver from sources - in order to match the exact running kernel on the kubernetes cluster node/"host").
Problem:
Runtime issues:
Fetching packages over the network often creates a failure point any time the k8s cluster node is being "prepared" (before GPU using workload can run). Some failure scenarios (not exhaustive):
- Due to "Internet weather", archive.ubuntu.com is inaccessible
- Canonical's policy removes some older package versions (e.g. kernel headers package for an "older" version) from main repo (the default one) into snapshot
etc.
Operational issues:
This need for an Linux distro packages repo also complicates setting up air-gap environment - an additional mirror that the end-user needs to setup (in addition to mirroring the Nvidia container images).
Proposed Solution:
If the k8s cluster node/"host" provides the additional Linux distro OS packages on the node's file system itself (e.g. also bundle the matching kernel header package & other packages, for the kernel version being run by the node), then instruct GPU operator to fetch the needed packages from the node itself - a "node-local" I/O instead of network I/O.
This eliminates a class of problems describes above.
Implementation problem:
While GPU Operator today allows overriding the location of the Linux distro packages repo, e.g. say the GPU Operator chart's driver.repConfig.configMapName points to a configmap that has .data.value set as:
Types: deb
URIs: file:///opt/local-packages
Suites: ./
Signed-By: /opt/local-packages/keyring.gpg
currently GPU operator implicitly assumes that repo is accessed over the network, the driver daemonset pod/container ( that runs the driver install script which in turn triggers the fetching of packages) does not have access to the node/host's file system (the init container has it, but not the main container)
The template deployed by GPU Operator would need to allow "host path volume mount" in a specific manner (to avoid increasing attack surface), e.g. instead of arbitrary host mount config, have a specific config that mounts a read-only volume inside the pod/container.
Also this feature needs to be opt-in, if the provider of node image, of the k8s cluster that is going to run GPU Operator, is supplying this "node-local packages repo", then the user should be able to configure access to it through GPU Operator config API.
Context:
Nvidia driver daemonset deployed by GPU Operator (for non-pre-compiled driver mode) has a step to pull some Linux OS distro images at run-time (during driver pod bringup) from a Linux distro packages repository (e.g. an apt packages repository for Unbutu), location of this repository is over the network (e.g. by default, archive.ubuntu.com over the Internet, or in case of air-gap, a local mirror setup by the user, till over the network).
(The packages being fetched over the network and installed inside the driver pod, e.g. linux-kernel headers etc., are required to compile the driver from sources - in order to match the exact running kernel on the kubernetes cluster node/"host").
Problem:
Runtime issues:
Fetching packages over the network often creates a failure point any time the k8s cluster node is being "prepared" (before GPU using workload can run). Some failure scenarios (not exhaustive):
etc.
Operational issues:
This need for an Linux distro packages repo also complicates setting up air-gap environment - an additional mirror that the end-user needs to setup (in addition to mirroring the Nvidia container images).
Proposed Solution:
If the k8s cluster node/"host" provides the additional Linux distro OS packages on the node's file system itself (e.g. also bundle the matching kernel header package & other packages, for the kernel version being run by the node), then instruct GPU operator to fetch the needed packages from the node itself - a "node-local" I/O instead of network I/O.
This eliminates a class of problems describes above.
Implementation problem:
While GPU Operator today allows overriding the location of the Linux distro packages repo, e.g. say the GPU Operator chart's
driver.repConfig.configMapNamepoints to a configmap that has.data.valueset as:currently GPU operator implicitly assumes that repo is accessed over the network, the driver daemonset pod/container ( that runs the driver install script which in turn triggers the fetching of packages) does not have access to the node/host's file system (the init container has it, but not the main container)
The template deployed by GPU Operator would need to allow "host path volume mount" in a specific manner (to avoid increasing attack surface), e.g. instead of arbitrary host mount config, have a specific config that mounts a read-only volume inside the pod/container.
Also this feature needs to be opt-in, if the provider of node image, of the k8s cluster that is going to run GPU Operator, is supplying this "node-local packages repo", then the user should be able to configure access to it through GPU Operator config API.