cannot run AutoSeg3D with GPU

[dlabella29]

Hello,
I am trying to get AutoSeg3D to run using Windows 11 in PyCharm.
I have been able to get the swin2021 tutorial to train multiclass-multilabel MRI brain models.

For some reason the python interpreter switches to my system interpreter and cannot access my GPU/CUDA for training:
The python interpreter works with the virtual environment when running the code from the main script.

Please advise how I can get it to run with my GPU. I've been stuck for a while.

Thanks in advance!

main AutoSeg3D script:

if name == 'main':
from monai.apps.auto3dseg import AutoRunner
import os
import sys
import os
import torch
import sys
print("auto executable")
print(sys.executable)
def main():

   input_config = {
      "modality": "MRI",
      "dataroot": r"C:\Users\***\PycharmProjects\AutoSegBrats\data\brats23",
      "datalist": "brats23_short.json",
      "class_names": ["ncr","ed","et"],
   }
   # print(os.__file__)
   # runner = AutoRunner(input=input_config, algos = "segresnet", work_dir= "./brats23_workdir2")
   os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
   device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
   # device = torch.device("cpu")
   print("cuda yes or no in main script:")
   print(torch.cuda.is_available())
   runner = AutoRunner(input=input_config, algos = "segresnet", work_dir= "./")
   print("executable in main after runner called")
   print(sys.executable)
   runner.run()

main()

---

Train.py script from within segresnet_0:

print("cuda yes or no in train script:")
print(torch.cuda.is_available())
print("train executable:")
print(sys.executable)

if package in (None, ""):
from segmenter import run_segmenter
else:
from .segmenter import run_segmenter

def run(config_file: Optional[Union[str, Sequence[str]]] = None, **override):
run_segmenter(config_file=config_file, **override)

if name == "main":
fire.Fire()

---

output after running AutoSeg3D

C:\Users*\PycharmProjects\AutoSegBrats\venv\Scripts\python.exe C:\Users*\PycharmProjects\AutoSegBrats\AutoSegSynapseMutuallyExclusive.py
auto executable
C:\Users*\PycharmProjects\AutoSegBrats\venv\Scripts\python.exe
cuda yes or no in main script:
True
2023-08-04 12:54:46,103 - INFO - AutoRunner using work directory ./
2023-08-04 12:54:46,105 - INFO - Setting num_fold 5 based on the input datalist C:\Users*\PycharmProjects\AutoSegBrats\brats23_short.json.
executable in main after runner called
C:\Users*\PycharmProjects\AutoSegBrats\venv\Scripts\python.exe
2023-08-04 12:54:46,132 - INFO - Skipping data analysis...
2023-08-04 12:54:46,132 - INFO - Skipping algorithm generation...
2023-08-04 12:54:46,150 - INFO - Launching: python C:\Users\dlabe\PycharmProjects\AutoSegBrats\segresnet_0\scripts\train.py run --config_file='C:/Users//PycharmProjects/AutoSegBrats/segresnet_0/configs/hyper_parameters.yaml'
cuda yes or no in train script:
False
train executable:
C:\Users**\AppData\Local\Programs\Python\Python311\python.exe

[KumoLiu]

Hi @dlabella29, since torch.cuda.is_available() returns False, you may:

• Missing or have incompatible GPU driver
• Have incompatible CUDA version
• PyTorch installation without CUDA

Could you please check the env outside your virtual env?
A reference link https://saturncloud.io/blog/why-torchcudaisavailable-returns-false-even-after-installing-pytorch-with-cuda/ which may help you.
Thanks!

[dlabella29]

Thanks for the response.

The GPU driver is installed with proper CUDA version and PyTorch installation.
I can run other MONAI 3D segmentation tutorials on my GPU without issues.

The AutoSeg3D script recognizes my GPU/CUDA when it processes the main AutoSeg3D script which recognizes my virtual environment.
At this time, it correctly uses the venv configured with GPU/CUDA as shown below:
C:\Users*\PycharmProjects\AutoSegBrats\venv\Scripts\python.exe

However, when it tries to run the train.py subscript within segresnet_0, it does not recognize GPU/CUDA.
At this time it switched to the system python interpreter as shown below:
C:\Users**\AppData\Local\Programs\Python\Python311\python.exe

Is there anyway to not have the python interpreter switch when the train.py subscript is called?

Thanks!

[KumoLiu]

Hi @dlabella29, it's wired that the Python interpreter switched.

@mingxin-zheng, have you ever encountered something similar before? Could you please share some comments here?

Thanks in advance!

[mingxin-zheng]

Hi @dlabella29 , I am not sure about the cause. But the Auto3DSeg classes/methods support an environment variable setting CMD_PREFIX as doc here

You can try running the command with the python you want

# assume the correct python is located in /venv/bin/python
(venv)$ export CMD_PREFIX=/venv/bin/python
(venv)$ python <your_auto3dseg_script>.py

[dlabella29]

I tried listing the CMD_PREFIX "venv/bin/python" value with various paths with and without .exe onto the python and it still does not use the venv with CUDA/GPU capabilities.

Any other thoughts would be appreciated.
Thanks again.

if name == 'main':

from monai.apps.auto3dseg import AutoRunner
import os
import sys
import torch
print(sys.path)
print(os.__file__)
print("auto executable")
print(sys.executable)
def main():
   input_config = {
      "modality": "MRI",
      "dataroot": r"C:\Users\dlabe\PycharmProjects\AutoSegBrats\data\brats23",
      "datalist": "brats23_short.json",
      "class_names": ["ncr","ed","et"],
       "CMD_PREFIX": "/venv/Scripts/python"
   }
   # print(os.__file__)
   # runner = AutoRunner(input=input_config, algos = "segresnet", work_dir= "./brats23_workdir2")
   os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
   device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
   # device = torch.device("cpu")
   print("cuda yes or no in main script:")
   print(torch.cuda.is_available())
   runner = AutoRunner(input=input_config, algos = "segresnet", work_dir= "./")
   print("executable in main after runner called")
   print(sys.executable)
   runner.run()

main()

[dlabella29]

And here is what my configured environment looks like that I am attempting to use. This environment successfully runs other MONAI 3D segmentation tutorials using my GPU/CUDA.