PyTorch cannot detect NVIDIA drivers causing run_server.sh to fail

[palpen]

Running bash setup_server.sh appears to succeed but PyTorch actually fails to detect the NVIDIA drivers. This then causes bash run_server.sh to fail because the triton inference server couldn't load the min_dalle model due to the apparently missing NVIDIA drivers.

Here's the output when I run setup_server.sh where it warns me that the NVIDIA driver was not detected:

=============
== PyTorch ==
=============

NVIDIA Release 22.05 (build 37432893)
PyTorch Version 1.12.0a0+8a1a93a

Container image Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copyright (c) 2014-2022 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

I do, however, actually have the NVIDIA drivers installed on my system:

Poking around the internet also led me to installing the NVIDIA Container Toolkit but it still did not resolve the issue.

Any suggestions on what could be causing this problem?

[FamousDirector]

@palpen Yeah installing the Nvidia container toolkit should be all you need.

[palpen]

Thanks for the suggestion. I have tried that. It seems to be having trouble loading the asr_infer model and min_dalle

For what it's worth, I added this to the docker-compose.yaml file to enable GPUs, which made the "No GPU detected" error go away.

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

[palpen]

Turns out that the torch library installed in the triton-inference-server Docker images uses CUDA 10.3, which is incompatible with my GPU (an RTX 3090). I fixed this by updating the Dockerfile.triton file to uninstall torch and reinstall it using a version that uses CUDA 11.3 using this command:

pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113

Now, I just need to figure out why asr_infer is unable to find libtriton_onnxruntime.so 😕