multiple input, multiple AI models and multiple out app

[TmnGitHub]

Hi everyone, could you please suggest or refer me on how to deploy multiple AI models in a single app? It is similar to 07_multi_model_app.ipynb tutorials on deployment. In this tutorial, only one image is given as input, and multiple models are used for predictions (monai bundle concepts). However, I would like to give multiple images as input that will be used by multiple AI models and outputs multiple images as results in a single app. For example, loading various DICOM/NifTI of various body parts (eg. brain and lungs), inferencing with corresponding trained models, and writing corresponding multiple outputs (brain and lungs) in a single app. @dbericat @monai-deploy-app-sdk

I appreciate your help.

Thanks,
Taman

[MMelQin]

Hi @TmnGitHub

I have responded to your inquiry on MONAI Slack Channel with a brief description of the solution, but will provide it here, too.

• How do you determine what input is for which model? For DICOM input, metadata on modality and body parts may be sufficient, depending on the body part specific models, but if you have a CT spleen as well as a liver seg and tumor detection, relying on body part abdomen to distinguish is not at the right granularity. For NIfTI input(s), it is even worse, as the clinical information is absent. You'd have to first come up with and implement this business logic,
• The App SDK provides a model loader factory, behind the scene used by the executor and not directly accessed by the user app code. It loads all models in the subfolders of the well-known models folder and the folder structure is as explained in the aforementioned example. So, loading multiple named model is taken care off.
• The built-in DCOM data loader operator in the App SDK can load and organize any number of DICOM Studies/Series into in memory representation of the standards defined hierarchy, so there will be a list of Studies containing N number of Series containing N number of Instances. For NIfTI files, the built-in nii loader support single file, but can be used as an example to extend to support multiple input files and output image. BUT, distinguishing what file is for what purpose is really the application logic and has to be implemented, as mentioned above.
• The App SDK does provide a simple series selection operator, mostly just matching on string type DICOM metadata. This can be used to filter the parsed DICOM study/series to filer out the specific series for the intended model.
• Now, how do you put all together.
  • Option 1, is to use the same DCIOM loader instance to get all studies/series, then linked to multiple DICOM series selector instances, each will have a specific rule, e.g. BodyPart head, or abdomen, and you can have your custom rule and processing. After each series selector, then link the series to volume, model specific inference operator, seg writer etc, essentially making this a whole branch off of the DICOM data loader. The downside of this approach is that all branches will get invoke, and the some will have not filtered data, which may cause downstream operators to fail. In the next release, I am hoping to support conditional running of a branch.
  • Option 2, create an Application which is model specific, e,g like the standalone liver tumor seg app and the spleen seg app. Then in the main function or a main script, inspect the DICOM or nii input data with your own logic, and then determine which model/Application to run. It is then a matter of instantiating the specific Application and call its run method.

Give these options a try. In the MONAI Deploy realm, we have the Workflow Manager which supports rule based routing of input DICOM study to a model/organ specific app, so the series selection functionality in the app itself is more for data validation.