[BUG] ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory

[vikashg]

Describe the bug
When I try to build the monai-deploy docker image, I get the following error

Traceback (most recent call last):
  File "/home/gupta/miniconda3/envs/monai-deploy-test/bin/monai-deploy", line 5, in <module>
    from holoscan.cli.__main__ import main
  File "/home/gupta/miniconda3/envs/monai-deploy-test/lib/python3.8/site-packages/holoscan/__init__.py", line 18, in <module>
    from . import core, gxf
  File "/home/gupta/miniconda3/envs/monai-deploy-test/lib/python3.8/site-packages/holoscan/core/__init__.py", line 65, in <module>
    from ..graphs._graphs import FragmentGraph, OperatorGraph
  File "/home/gupta/miniconda3/envs/monai-deploy-test/lib/python3.8/site-packages/holoscan/graphs/__init__.py", line 24, in <module>
    from ._graphs import FragmentFlowGraph, OperatorFlowGraph
ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory

I have the following pytorch version '2.0.1+cu117'. and cuda version 12.0. Maybe the version mismatch is causing this problem?

Steps/Code to reproduce bug
With the above version, try running the simple_imaging_app in the notebook tutorials

Environment details (please complete the following information)

• OS/Platform: Ubuntu 20.04
• Python Version: 3.8.17
• Method of MONAI Deploy App SDK install: [pip, conda, Docker, or from source] from source
• pytorch version: '2.0.1+cu117'
• Cuda version: 12.0

[MMelQin]

@vikashg Thanks for the issue.

Yes, it is a compatibility issue between the underlying Holoscan SDK, CUDA, and Torch, and I posted Discussion 460 a couple of days ago. Please see the combo that I have tested.

[MMelQin]

@vikashg here is the CUDA Toolkit, GPU Driver version, and torch in my environment. With this combo, the apps work and torch can access the GPU.

$ nvidia-smi
Mon Oct  9 13:46:27 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.12             Driver Version: 535.104.12   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro GV100                   On  | 00000000:65:00.0 Off |                  Off |
| 29%   40C    P2              26W / 250W |     89MiB / 32768MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1193      G   /usr/lib/xorg/Xorg                           74MiB |
|    0   N/A  N/A      1338      G   /usr/bin/gnome-shell                         12MiB |
+---------------------------------------------------------------------------------------+

$ pip list | grep torch
torch                         2.0.1

python --version
Python 3.8.10

Can you please ensure torch is pinned at ~=2.0.1 and try again?

[MMelQin]

@vikashg So, I have reproduced the Packager error with Torch 2.0.1 and CUDA 12.02, though I had been using Torch 2.0.1 and CUDA 11.7 without issues with the app itself or packager. The underlying Holoscan SDK v0.6 supports CUDA 11.8, so I will reset the CUDA on my desktop to it and test.

[MMelQin]

@vikashg Here is the confirmed workaround/fix, confirmed by @mocsharp and me.
sudo apt-get install cuda-runtime-11.7

I tried to remove CUDA 12.2 and install CUDA 11.8, but after multiple attempts, I still had CUDA 12.2 per report by nvidia-smi. The same ImportError: libcudart.so.11.0 persisted.

@mocsharp had the similar error, though on libcudart.so.12.2, and apt-get install cuda-runtime fixed it. I can confirm sudo apt-get install cuda-runtime-11.8 does the trick with the CUDA 12.2 in my dev env. In your case with CUDA 11.7, the cuda-runtime-11.7 can be used.

Please confirm the results once you can give it a shot.

[vikashg]

Hey @MMelQin Thanks for looking into it . I will test again today and post my outcome. I was bit occupied with couple of other things.
Thanks

[mpsampat]

@MMelQin , just a quick report on the same issue in case it provides a clue.
In the past I was able to build with monai-deploy on CPU only machine. (tested 0.5.0);
we put monai-deploy in our CI/CD (very happy about that :))

https://gitlab.com/flywheel-io/scientific-solutions/gears/breast-density-classifier/-/blob/main/.gitlab-ci.yml?ref_type=heads#L34

Now with 0.6.0 I am still having this error that @vikashg reported (shown below). I am on CPU only machine;
I did "conda install -c "nvidia/label/cuda-11.8.0" cuda-runtime' But no luck resolving it. I am on Ubuntu 18.04.
I also try "conda list | grep nvidia" and i dont have any 12.x.x (see below);
Will try on a VM with GPU on vertex AI tomorrow.
Mehul

the error is:

"monai-deploy exec ct-seg-monai-swin-unetr/ -i ~/monai-bundles/1_axial_chest_ct.nii.gz -o output. -m ct-seg-monai-swin-unetr/best_metric_model.ts
Traceback (most recent call last):
File "/home/msampat/miniconda3/envs/py38b/bin/monai-deploy", line 5, in
from holoscan.cli.main import main
File "/home/msampat/miniconda3/envs/py38b/lib/python3.8/site-packages/holoscan/init.py", line 17, in
from . import cli, core, gxf
File "/home/msampat/miniconda3/envs/py38b/lib/python3.8/site-packages/holoscan/core/init.py", line 65, in
from ..graphs._graphs import FragmentGraph, OperatorGraph
File "/home/msampat/miniconda3/envs/py38b/lib/python3.8/site-packages/holoscan/graphs/init.py", line 24, in
from ._graphs import FragmentFlowGraph, OperatorFlowGraph
ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory"

(py38b) msampat@scien-dev1:~/gears$ conda list | grep nvidia
cuda-cudart 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-cupti 11.8.87 0 nvidia
cuda-libraries 11.8.0 0 nvidia/label/cuda-11.8.0
cuda-nvrtc 11.8.89 0 nvidia/label/cuda-11.8.0
cuda-nvtx 11.8.86 0 nvidia
cuda-runtime 11.8.0 0 nvidia/label/cuda-11.8.0
libcublas 11.11.3.6 0 nvidia/label/cuda-11.8.0
libcufft 10.9.0.58 0 nvidia/label/cuda-11.8.0
libcufile 1.4.0.31 0 nvidia/label/cuda-11.8.0
libcurand 10.3.0.86 0 nvidia/label/cuda-11.8.0
libcusolver 11.4.1.48 0 nvidia/label/cuda-11.8.0
libcusparse 11.7.5.86 0 nvidia/label/cuda-11.8.0
libnpp 11.8.0.86 0 nvidia/label/cuda-11.8.0
libnvjpeg 11.9.0.86 0 nvidia/label/cuda-11.8.0
nvidia-cublas-cu11 11.10.3.66 pypi_0 pypi
nvidia-cuda-cupti-cu11 11.7.101 pypi_0 pypi
nvidia-cuda-nvrtc-cu11 11.7.99 pypi_0 pypi
nvidia-cuda-runtime-cu11 11.7.99 pypi_0 pypi
nvidia-cudnn-cu11 8.5.0.96 pypi_0 pypi
nvidia-cufft-cu11 10.9.0.58 pypi_0 pypi
nvidia-curand-cu11 10.2.10.91 pypi_0 pypi
nvidia-cusolver-cu11 11.4.0.1 pypi_0 pypi
nvidia-cusparse-cu11 11.7.4.91 pypi_0 pypi
nvidia-nccl-cu11 2.14.3 pypi_0 pypi
nvidia-nvtx-cu11 11.7.91 pypi_0 pypi

[SameerShanbhogue]

Hi

Is this issue related to PATH ?

After export the PATH the issue gets resolved temporarily.

Also during monai-deploy-sdk install, there are warnings related to holoscan and monai-deploy not in PATH

[MMelQin]

@mpsampat I noticed another issue in the log, the sub-command, monai-deploy exec, is not supported in v0.6 CLI (this command behaves similarly to directly running the app with Python interpreter but with the required env vars set based on the command line options)

For the real issue with not finding CUDA 11 runtime, I cannot say for sure, as I have not tested with conda and Ubuntu 18.04. MD App SDK (and the underlying Holoscan SDK needs Ubuntu 20.04).

[mpsampat]

@MMelQin thank you very much for your super quick response.

I see both your points. Tomorrow I will try on a VM with Ubuntu 20.04.

Also will try the path issue Sameer mentioned in this thread. Maybe some environment variable is not setup correctly on my side.

Will test tomorrow and report back here.

[vikashg]

@vikashg Here is the confirmed workaround/fix, confirmed by @mocsharp and me. sudo apt-get install cuda-runtime-11.7

Hi @MMelQin, So I am still getting the error. I am installing cuda-runtime-11.7 from here as sudo apt-get is not working for me. I will try again after the installation is complete.
Thanks

I tried to remove CUDA 12.2 and install CUDA 11.8, but after multiple attempts, I still had CUDA 12.2 per report by nvidia-smi. The same ImportError: libcudart.so.11.0 persisted.

@mocsharp had the similar error, though on libcudart.so.12.2, and apt-get install cuda-runtime fixed it. I can confirm sudo apt-get install cuda-runtime-11.8 does the trick with the CUDA 12.2 in my dev env. In your case with CUDA 11.7, the cuda-runtime-11.7 can be used.

Please confirm the results once you can give it a shot.

I have CUDA 12.0. Waiting for the above trial with cuda-runtime-11.7 before trying to installing CUDA 12.2.

[vikashg]

Hi @MMelQin After installing Cuda 11.7 it is working but docker packaging is not working. I am not able to copy the text but here is a screenshot

Initially, I thought it is because of the holoscan base image needs nvcr.io login. So I downloaded the holoscan image manually. But I still get this error.

[mpsampat]

@vikashg could you tell me which version of torch and python you are using.
We use monai-deploy in our CI/CD and I now seeing this error with 0.5.1
https://gitlab.com/flywheel-io/scientific-solutions/gears/breast-density-classifier/-/jobs/5365017305#L82
"OSError: libcufft.so.11: cannot open shared object file: No such file or directory" from this line:
https://gitlab.com/flywheel-io/scientific-solutions/gears/breast-density-classifier/-/jobs/5365017305#L90

@MMelQin My pyproject.toml file has this:

[tool.poetry.dependencies]
python = "^3.10"
flywheel-gear-toolkit = "^0.2"
monai = "1.1.0"
monai-deploy-app-sdk = "0.5.1"
pydicom = "2.3.1"
highdicom = "0.20.0"
torch = ">=2.0.0, !=2.0.1"

So it picked python 3.11. @MMelQin do you have a preference for python version ? I can pin that and try with a recommended python version

[MMelQin]

Hi @MMelQin After installing Cuda 11.7 it is working but docker packaging is not working. I am not able to copy the text but here is a screenshot

Initially, I thought it is because of the holoscan base image needs nvcr.io login. So I downloaded the holoscan image manually. But I still get this error.

@vikashg Sorry about this error, which already logged as an issue. Workaround is to download the Wheel file from pypi.org, and then use the commanline option to pass the Wheel file to the Packager.

It will be fixed in the next public release of Holoscan SDK, and hence in MD App SDK. In the meantime, you can also patch the HS SDK code once it is installed in the env, as documented in the issue.