Force to remap to CPU for TorchScript model if GPUdoesn't exist

[gigony]

The TorchScript model file is loaded to the device where the model is trained (torch.jit.load).

For that reason, if the model is trained/saved by GPU and loaded by the system without GPU, it causes errors because the model cannot be loaded.

monai-deploy-app-sdk/monai/deploy/core/models/torch_model.py

Line 50 in fa4df1d

self._predictor = torch.jit.load(self.path).eval()

  File "/Users/Erik/opt/anaconda3/lib/python3.8/site-packages/monai/deploy/core/models/model.py", line 220, in __call__
    if self.predictor:
  File "/Users/Erik/opt/anaconda3/lib/python3.8/site-packages/monai/deploy/core/models/torch_model.py", line 50, in predictor
    self._predictor = torch.jit.load(self.path).eval()
  File "/Users/Erik/opt/anaconda3/lib/python3.8/site-packages/torch/jit/_serialization.py", line 161, in load
    cpp_module = torch._C.import_ir_module(cu, str(f), map_location, _extra_files)
NotImplementedError: Could not run 'aten::empty_strided' with arguments from the 'CUDA' backend. 
This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). 
If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 
'aten::empty_strided' is only available for these backends: [CPU, Meta, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].

This patch resolves the issue by remapping the storage to CPU if GPU is not available.

Resolve #166 that is from #156.

A test package is available by

pip install -i https://test.pypi.org/simple/ monai-deploy-app-sdk==0.0.167

Verified locally by disabling GPU

export CUDA_VISIBLE_DEVICES=""
python examples/apps/mednist_classifier_monaideploy/mednist_classifier_monaideploy.py -i input/AbdomenCT_007000.jpeg -o output -m classifier.zip

python examples/apps/ai_spleen_seg_app/app.py -i dcm/ -o output -m model.pt

[MMelQin]

This PR address the issue of jit.load should target available device (GPU preferred but not forced in not present). It, however, does not, or have not proved to, address the issue of force-load model traced with GPU onto CPU.

[MMelQin]

Wonder why not using torch device, to be explicit here and for clarity (https://pytorch.org/docs/master/generated/torch.jit.load.html#torch.jit.load)?

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

[gigony]

Thanks for the suggestion!

I am afraid the behavior is different (map_location needs to be None, not torch.device("cuda")).

We wanted to remap to 'cpu' only if cuda is not available. If cuda is available, we wouldn't want to remap location, and use the device specified in the model.

device = torch.device("cuda" if torch.cuda.is_available() else "CPU")
self._predictor = torch.jit.load(self.path, map_location=device).eval()

vs

map_location = None if torch.cuda.is_available() else "cpu"
self._predictor = torch.jit.load(self.path, map_location=map_location).eval()

However, on second thought, it would be better to force remap, depending on cuda availability (as operator/application's code also usually use the condition to determine which device to use.

So, will fix as suggested

[MMelQin]

I'd change this to use the device which is instantiated in this block of code. Also, this else block is a fallback when the SDK model loader fails to load the model, the original line #172 below.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

Also force a model traced on GPU to CPU will most likely fail, so, this is only shifting the failure points.

[gigony]

Thanks Ming!
Updated to use device variable

[MMelQin]

This conditional logic is repeated multiple times. Wonder why not using instantiating a device, the tensor.to(device)?

[gigony]

Updated existing code/tutorial.
Thank you for the suggestion!