Description
Discussed in #358
Originally posted by Leengit September 28, 2022
I successfully build a docker image with monai-deploy package that runs on the computer on which I built it. However when I try to run the same docker image on a computer with a significantly newer / more powerful GPU, it fails. It appears that the underlying docker image nvcr.io/nvidia/pytorch:21.07-py3 uses a version of CUDA 11.3 and torch that do not support sm_86. Upgrading to torch==1.12.1 within the docker image that I create (and committing the change to a new image that I then use) does not help. Despite my attempts with apt-get, I have been unable to install a newer version of CUDA within the created docker image.
Your help with getting support for an NVIDIA RTX A5000 would be much appreciated! The error from running the docker image that I created with monai deploy includes
NVIDIA RTX A5000 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA RTX A5000 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
...
File "~/venv/lung/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 453, in _conv_forward
weight, bias, self.stride,
_pair(0), self.dilation, self.groups)
return F.conv2d(input, weight, bias, self.stride,
~~~~~~~~ <--- HERE
self.padding, self.dilation, self.groups)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```</div>