[BUG] monai.deploy.exceptions.ItemNotExistsError: A predictor of the model is not set

[Suchi97]

Describe the bug
We are working on deployment of spleen segmentation using monai deploy, we are using pretrained models from ngc for spleen segmentation from (https://catalog.ngc.nvidia.com/orgs/nvidia/teams/med/models/clara_pt_spleen_ct_segmentation) link.

Similarly we are using the app.py from examples in deploy-app-sdk(https://github.com/Project-MONAI/monai-deploy-app-sdk/tree/main/examples/apps/ai_spleen_seg_app).
when we run python app.py -i ./dcm -o ./output -m ./model.pt .

We get the following error

raise ItemNotExistsError("A predictor of the model is not set.")
monai.deploy.exceptions.ItemNotExistsError: A predictor of the model is not set.

We tried our own trained model and still got the same error.

Steps/Code to reproduce bug

1. Used app.py from examples in monai-deploy-app-sdk(https://github.com/Project-MONAI/monai-deploy-app-sdk/tree/main/examples/apps/ai_spleen_seg_app)
2. Run the python app.py -i ./dcm -o ./output -m ./model.pt
   1. Input images from ai_spleen_seg_data which is provided in the MONAI Deploy Segmentation App documentation
   2. Used pretrained model from ngc for spleen segmentation from (https://catalog.ngc.nvidia.com/orgs/nvidia/teams/med/models/clara_pt_spleen_ct_segmentation) link.

Expected behavior
We expect the pretrained model from the link , also results in the output without any issues.

Environment details (please complete the following information)

• OS/Platform: linux 4.15.0-136-generic
• Python Version: Python 3.8.10 | packaged by conda-forge
• Method of MONAI Deploy App SDK install: [pip, conda, Docker, or from source] : pip install monai-deploy-app-sdk
• SDK Version: 0.2.0

Additional context
This works fine for the spleen model downloaded from google drive link (https://drive.google.com/uc?id=1uTQsm8omwimBcp_kRXlduWBP2M6cspr1) which was in the tutorial MONAI Deploy Segmentation App documentation

Is there any specific thing we need to do while saving and using the model with deploy app sdk.

Any suggestion and help is appreciated. Thank you

[vikashg]

Hello @Suchi97, I think the models in the NGC cloud and the one shared at google drive might not be the same. We have opened an issue where we want to sync the models at NGC.
On the bright side, if your notebooks are working that means the monai-deploy-app-sdk is properly set up. If possible can you print the model and see if their input/output size is the same.

[MMelQin]

The Clara Train (PyTorch) MMAR on NGC contains both PyTorch and TorchScript model files, namely model.pt and model.ts. <MMAR>/models/model.ts is the TorchScript to use. From looking at the command in your initial message, the model.pt downloaded from NGC was used instead.

In the Spleen example dataset that is downloaded from Google Drive and used in the Jupyter notebook, the model file, model.pt, is used, however this file was renamed from the <PT Spleen MMAR>/models/model.ts, so it is actually the TorchScript file. Apologies for the confusion; the renaming was done for another use case (Triton inference server which by default expects the TorchScript model named model.pt), but was not reverted when I packaged the dataset.

[MMelQin]

An updated version of the dataset zip file has now been uploaded to Google Drive, and Pull Request #222 created to update the Jupyter notebooks to use the this updated zip file, as well as the model file with .ts extension to avoid confusion.

[Suchi97]

@vikashg and @MMelQin, Thanks for your reply.
@MMelQin, I converted our trained model model.pt to TorchScript file.
Now its working fine.

[MMelQin]

@Suchi97 Thanks for the confirmation. The aforementioned PR #222 has already been reviewed and test, with the improvement avoiding confusions of what model to use.
I will therefore close this issues as fixed.

[Suchi97]

The Clara Train (PyTorch) MMAR on NGC contains both PyTorch and TorchScript model files, namely model.pt and model.ts. <MMAR>/models/model.ts is the TorchScript to use. From looking at the command in your initial message, the model.pt downloaded from NGC was used instead.

In the Spleen example dataset that is downloaded from Google Drive and used in the Jupyter notebook, the model file, model.pt, is used, however this file was renamed from the <PT Spleen MMAR>/models/model.ts, so it is actually the TorchScript file. Apologies for the confusion; the renaming was done for another use case (Triton inference server which by default expects the TorchScript model named model.pt), but was not reverted when I packaged the dataset.

When I tried <MMAR>/models/model.ts in the spleen segmentation app.py (https://github.com/Project-MONAI/monai-deploy-app-sdk/tree/main/examples/apps/ai_spleen_seg_app).

[So the command looks like below
python app.py -i ./dcm -o ./output -m ./model.ts]

I got the following error

raise ItemNotExistsError("A predictor of the model is not set.")
monai.deploy.exceptions.ItemNotExistsError: A predictor of the model is not set.

@MMelQin, Everytime , do we need to convert the model.pt to torchscript file?

Any suggestion and help is appreciated. Thank you

[MMelQin]

@Suchi97 Sorry the example application folder itself does not contain the model.ts file, nor the DICOM instance files. If it is available, then a TorchScript model can be directly used.

I'll show below two ways of getting the model files, though the test DICOM files are only available from the Google drive download.

I updated the testdata (dcm folder and the model.ts) on Google Drive, and updated the download URL in the Jupyter notebook. You can directly use the gdown command. The following is what I have just tested,

mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated$ ls
mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated$ gdown https://drive.google.com/uc?id=1GC_N8YQk_mOWN02oOzAU_2YDmNRWk--n
Downloading...
From: https://drive.google.com/uc?id=1GC_N8YQk_mOWN02oOzAU_2YDmNRWk--n
To: /home/mqin/src/monai-app-sdk/ai_spleen_seg_data_updated/ai_spleen_seg_data_updated_1203.zip
100%|█████████████████████████████████████████████████████████████████████████████████████████| 104M/104M [00:01<00:00, 102MB/s]
mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated$ unzip -o "ai_spleen_seg_data_updated_1203.zip"
mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated$ ls
ai_spleen_seg_data_updated_1203.zip  dcm  model.ts
mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated$ python3 /home/mqin/src/monai-app-sdk/examples/apps/ai_spleen_seg_app/app.py -i ./dcm -o ./output -m ./model.ts

There is a Spleen MMAR on NGC, and I have downloaded the zip and inflated it,

mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated/ngc$ ls
'files (1).zip'
mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated/ngc$ unzip -o 'files (1).zip'
Archive:  files (1).zip
  inflating: commands/export.sh      
  inflating: commands/finetune.sh    
  inflating: commands/finetune_multi_gpu.sh  
  inflating: commands/infer.sh       
  inflating: commands/infer_multi_gpu.sh  
  inflating: commands/set_env.sh     
  inflating: commands/train.sh       
  inflating: commands/train_multi_gpu.sh  
  inflating: commands/validate.sh    
  inflating: commands/validate_ckpt.sh  
  inflating: commands/validate_multi_gpu.sh  
  inflating: commands/validate_multi_gpu_ckpt.sh  
  inflating: config/config_aiaa.json  
  inflating: config/config_inference.json  
  inflating: config/config_train.json  
  inflating: config/config_validation.json  
  inflating: config/dataset_0.json   
  inflating: config/environment.json  
  inflating: docs/README.md          
  inflating: docs/license.txt        
  inflating: models/model.pt         
  inflating: models/model.ts         
  inflating: resources/log.config    
mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated/ngc$ ls
 commands   config   docs  'files (1).zip'   models   resources
mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated/ngc$ ls models
model.pt  model.ts

Both model.pt and model.ts are there in the models folder. So, back in the previous working dir, ran the app successfully with the correct path as shown below.

mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated$ ls ngc/models/model.pt  model.ts
mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated$ python3 /home/mqin/src/monai-app-sdk/examples/apps/ai_spleen_seg_app/app.py -i dcm -o output -m ngc/models/model.ts
DEBUG:__main__.AISpleenSegApp:Begin compose
DEBUG:__main__.AISpleenSegApp:End compose
DEBUG:__main__.AISpleenSegApp:Begin run
Going to initiate execution of operator DICOMDataLoaderOperator
Executing operator DICOMDataLoaderOperator (Process ID: 16559, Operator ID: e90f9d03-6430-45b4-b766-1ccd74502e10)
Done performing execution of operator DICOMDataLoaderOperator

Going to initiate execution of operator DICOMSeriesSelectorOperator
Executing operator DICOMSeriesSelectorOperator (Process ID: 16559, Operator ID: e47c487a-d728-4fb6-be89-36dee15c0a86)
[2021-12-13 00:22:32,267] [INFO] (root) - Finding series for Selection named: CT Series
[2021-12-13 00:22:32,268] [INFO] (root) - Searching study, : 1.2.826.0.1.3680043.2.1125.1.67295333199898911264201812221946213
  # of series: 1
[2021-12-13 00:22:32,268] [INFO] (root) - Working on series, instance UID: 1.2.826.0.1.3680043.2.1125.1.68102559796966796813942775094416763
[2021-12-13 00:22:32,268] [INFO] (root) - On attribute: 'StudyDescription' to match value: '(.*?)'
[2021-12-13 00:22:32,268] [INFO] (root) -     Series attribute value: spleen
[2021-12-13 00:22:32,268] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2021-12-13 00:22:32,268] [INFO] (root) - On attribute: 'Modality' to match value: '(?i)CT'
[2021-12-13 00:22:32,268] [INFO] (root) -     Series attribute value: CT
[2021-12-13 00:22:32,268] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2021-12-13 00:22:32,268] [INFO] (root) - On attribute: 'SeriesDescription' to match value: '(.*?)'
[2021-12-13 00:22:32,268] [INFO] (root) -     Series attribute value: No series description
[2021-12-13 00:22:32,268] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2021-12-13 00:22:32,268] [INFO] (root) - Selected Series, UID: 1.2.826.0.1.3680043.2.1125.1.68102559796966796813942775094416763
Done performing execution of operator DICOMSeriesSelectorOperator

Going to initiate execution of operator DICOMSeriesToVolumeOperator
Executing operator DICOMSeriesToVolumeOperator (Process ID: 16559, Operator ID: e2321772-4e38-4f00-89fc-14d4a51d0633)
Done performing execution of operator DICOMSeriesToVolumeOperator

Going to initiate execution of operator SpleenSegOperator
Executing operator SpleenSegOperator (Process ID: 16559, Operator ID: 94a76b19-77d9-4e69-bae2-9b4479f3f078)
Converted Image object metadata:
SeriesInstanceUID: 1.2.826.0.1.3680043.2.1125.1.68102559796966796813942775094416763, type <class 'str'>
Modality: CT, type <class 'str'>
SeriesDescription: No series description, type <class 'str'>
PatientPosition: HFS, type <class 'str'>
SeriesNumber: 1, type <class 'int'>
row_pixel_spacing: 1.0, type <class 'float'>
col_pixel_spacing: 1.0, type <class 'float'>
depth_pixel_spacing: 1.0, type <class 'float'>
row_direction_cosine: [-1.0, 0.0, 0.0], type <class 'list'>
col_direction_cosine: [0.0, -1.0, 0.0], type <class 'list'>
depth_direction_cosine: [0.0, 0.0, 1.0], type <class 'list'>
dicom_affine_transform: [[-1.  0.  0.  0.]
 [ 0. -1.  0.  0.]
 [ 0.  0.  1.  0.]
 [ 0.  0.  0.  1.]], type <class 'numpy.ndarray'>
nifti_affine_transform: [[ 1. -0. -0. -0.]
 [-0.  1. -0. -0.]
 [ 0.  0.  1.  0.]
 [ 0.  0.  0.  1.]], type <class 'numpy.ndarray'>
StudyInstanceUID: 1.2.826.0.1.3680043.2.1125.1.67295333199898911264201812221946213, type <class 'str'>
StudyID: SLICER10001, type <class 'str'>
StudyDate: 2019-09-16, type <class 'str'>
StudyTime: 010100.000000, type <class 'str'>
StudyDescription: spleen, type <class 'str'>
AccessionNumber: 1, type <class 'str'>
selection_name: CT Series, type <class 'str'>
file written: /home/mqin/src/monai-app-sdk/ai_spleen_seg_data_updated/output/prediction_output/1.2.826.0.1.3680043.2.1125.1/1.2.826.0.1.3680043.2.1125.1_seg.nii.gz.
Output Seg image numpy array shaped: (515, 440, 440)
Output Seg image pixel max value: 1
Done performing execution of operator SpleenSegOperator

Going to initiate execution of operator DICOMSegmentationWriterOperator
Executing operator DICOMSegmentationWriterOperator (Process ID: 16559, Operator ID: 62fa369f-c3d9-4fc5-8007-4323eabb966f)
[2021-12-13 00:22:55,584] [INFO] (monai.deploy.operators.dicom_seg_writer_operator.DICOMSegWriter) - Number of DICOM instance datasets in the list: 515
[2021-12-13 00:22:55,584] [INFO] (monai.deploy.operators.dicom_seg_writer_operator.DICOMSegWriter) - Number of slices in the numpy image: 515
[2021-12-13 00:22:55,584] [INFO] (monai.deploy.operators.dicom_seg_writer_operator.DICOMSegWriter) - Labels of the segments: ['Spleen']
[2021-12-13 00:22:57,503] [INFO] (monai.deploy.operators.dicom_seg_writer_operator.DICOMSegWriter) - Unique values in seg image: [0 1]
[2021-12-13 00:22:59,511] [INFO] (monai.deploy.operators.dicom_seg_writer_operator.DICOMSegWriter) - Saving output file /home/mqin/src/monai-app-sdk/ai_spleen_seg_data_updated/output/dicom_seg-DICOMSEG.dcm
[2021-12-13 00:22:59,642] [INFO] (monai.deploy.operators.dicom_seg_writer_operator.DICOMSegWriter) - File saved.
Done performing execution of operator DICOMSegmentationWriterOperator

mqin@mingq-dt:~/src/monai-app-sdk/ai_spleen_seg_data_updated$ 

Hope this explains it, and resolves your issue.

[Suchi97]

Thanks @MMelQin, for your detailed description.
I tried the updated .ts files. And its works fine.

So, this resolves the issue.

[ArpanGyawali]

@vikashg and @MMelQin, Thanks for your reply. @MMelQin, I converted our trained model model.pt to TorchScript file. Now its working fine.

@Suchi97 Can you share how you converted model.pt into TorchScript file?
It would be very appreciable.