Environment
Cloud: Microsoft Azure
Platform: Azure Red Hat OpenShift (ARO)
OpenShift: 4.20.x
GPU: NVIDIA Tesla T4
Azure VM SKU: Standard_NC16as_T4_v3
GPU memory: ~16 GB
GPU Operator: 26.x
NVIDIA Driver: 595.91.07
CUDA: 13.2
Container runtime: CRI-O
Workload: IBM Maximo Visual Inspection (MVI) / Vision Edge
Issue description
We are running IBM Maximo Visual Inspection Vision Edge on an OpenShift cluster with an NVIDIA Tesla T4 GPU managed by NVIDIA GPU Operator.
When an MVI model is deployed and subsequently completed/removed, the Vision Edge DLE and its python-inference-process child processes can continue to retain significant GPU memory on the physical GPU.
When another model deployment is attempted, a new python-inference-process is created while the previous process(es) are still holding GPU memory. Over time, multiple inference processes accumulate on the same physical GPU and eventually consume almost all available GPU memory.
The result is a CUDA out-of-memory error during a subsequent model deployment.
Observed GPU memory usage
For example, nvidia-smi showed the following processes consuming memory on a single Tesla T4:
PID GPU Memory
3794364 5144MiB
3794783 2070 MiB
3798331 1562 MiB
3800664 866 MiB
3800979 5154 MiB
The GPU has approximately 14.56 GiB usable memory, and the accumulated processes eventually leave almost no free memory.
The CUDA error reported by the application was:
torch.OutOfMemoryError: CUDA out of memory.
Tried to allocate 12.00 MiB.
GPU 0 has a total capacity of 14.56 GiB
of which 1.75 MiB is free.
Reproduction / observed behavior
The issue can be reproduced approximately as follows:
Deploy an MVI model using Vision Edge in GPU mode.
Allow the model deployment/inference operation to complete.
Remove/complete the model operation.
Observe the GPU using nvidia-smi.
The corresponding python-inference-process may continue to exist and retain GPU memory.
Deploy another model.
A new inference process is created and allocates additional GPU memory.
Repeating this process eventually consumes most/all available GPU memory.
A subsequent model deployment fails with CUDA OOM.
The issue temporarily recovers when the stale inference processes are terminated. After terminating the processes, the corresponding GPU memory is released and the next model deployment can proceed successfully.
This indicates that the GPU memory is being held by active OS/CUDA processes rather than simply being unused memory cached by the PyTorch allocator.
Could you please check and advise on this issue.
Environment
Cloud: Microsoft Azure
Platform: Azure Red Hat OpenShift (ARO)
OpenShift: 4.20.x
GPU: NVIDIA Tesla T4
Azure VM SKU: Standard_NC16as_T4_v3
GPU memory: ~16 GB
GPU Operator: 26.x
NVIDIA Driver: 595.91.07
CUDA: 13.2
Container runtime: CRI-O
Workload: IBM Maximo Visual Inspection (MVI) / Vision Edge
Issue description
We are running IBM Maximo Visual Inspection Vision Edge on an OpenShift cluster with an NVIDIA Tesla T4 GPU managed by NVIDIA GPU Operator.
When an MVI model is deployed and subsequently completed/removed, the Vision Edge DLE and its python-inference-process child processes can continue to retain significant GPU memory on the physical GPU.
When another model deployment is attempted, a new python-inference-process is created while the previous process(es) are still holding GPU memory. Over time, multiple inference processes accumulate on the same physical GPU and eventually consume almost all available GPU memory.
The result is a CUDA out-of-memory error during a subsequent model deployment.
Observed GPU memory usage
For example, nvidia-smi showed the following processes consuming memory on a single Tesla T4:
PID GPU Memory
3794364 5144MiB
3794783 2070 MiB
3798331 1562 MiB
3800664 866 MiB
3800979 5154 MiB
The GPU has approximately 14.56 GiB usable memory, and the accumulated processes eventually leave almost no free memory.
The CUDA error reported by the application was:
torch.OutOfMemoryError: CUDA out of memory.
Tried to allocate 12.00 MiB.
GPU 0 has a total capacity of 14.56 GiB
of which 1.75 MiB is free.
Reproduction / observed behavior
The issue can be reproduced approximately as follows:
Deploy an MVI model using Vision Edge in GPU mode.
Allow the model deployment/inference operation to complete.
Remove/complete the model operation.
Observe the GPU using nvidia-smi.
The corresponding python-inference-process may continue to exist and retain GPU memory.
Deploy another model.
A new inference process is created and allocates additional GPU memory.
Repeating this process eventually consumes most/all available GPU memory.
A subsequent model deployment fails with CUDA OOM.
The issue temporarily recovers when the stale inference processes are terminated. After terminating the processes, the corresponding GPU memory is released and the next model deployment can proceed successfully.
This indicates that the GPU memory is being held by active OS/CUDA processes rather than simply being unused memory cached by the PyTorch allocator.
Could you please check and advise on this issue.