Question about MAISI data preprocessing

[johnson111788]

Hi,

I'm preparing the data for finetuning MAISI's ControlNet. I have some questions about the 3D VISTA segmentation.

1. How does a VISTA output pseudo label merge with the original label? Are the existing classes in the original label used directly? If it overlaps with other classes, do the existing classes apply?
2. Should I simply remove the prediction mask of dummy classes in label_dict? For example, the kidney is set to dummy1.

Thank you in advance for your help. It would be great if you could provide the official implementation of the merging original and pseudo labels.

Best,
Yu-Cheng

[Can-Zhao]

Thank you for reaching out! @guopengf Pengfei, would you like to help answer this question? Thanks.

[guopengf]

Hi @johnson111788 ,
For Q1: MAISI and VISTA 3D share the label_dict. You don't need to modify the label index of pseudo-labels generated by VISTA3D. If your data contains some ground truth for certain labels in generated pseudo-labels, you can replace those pseudo-labels with the GT. For new labels not existing in pseudo-labels, you can map it to any 8 dummy labels in default label_dict.json for finetuning.

For Q2: Please refer to the "Execute batch inference for segmenting everything" section of the VISTA3D bundle in MONAI Model Zoo https://github.com/Project-MONAI/model-zoo/tree/dev/models/vista3d#execute-batch-inference-for-segmenting-everything.
You may notice the segmenting everything prompt in VISTA3D already removed those dummy labels.

[johnson111788]

Hi Pengfei,

Thanks for your answer. I've tried using python -m monai.bundle run --config_file="['configs/inference.json', 'configs/batch_inference.json']" --input_dir="MY_DATA_PATH" --output_dir="./output" to inference. However, I encounter the following error:

Traceback (most recent call last):
File "/data/yucheng/anaconda/envs/vista/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
File "/data/yucheng/anaconda/envs/vista/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
File "/mnt/ccvl15/yucheng/DiffAd/Code/model-zoo/models/vista3d/scripts/monai_trans_utils.py", line 178, in call
pred[pred == frac] = data["label_prompt"][i - 1].to(pred.dtype)
IndexError: index 1 is out of bounds for dimension 0 with size 1

vista-errorlog_ipdb.txt

Meaning data["label_prompt"] is changed to 1 (Not sure why) and object_num does not match the label_prompt (object_num could be larger than 117, which I'm not sure how could this happen).

batch_inference.json seems to set the label_prompt as @everything_labels, but the results seem not matching the setting.

It would be great if you could tell me how and where the label_prompt is set and how should I avoid this problem.

Also, this is an error that cannot be reproduced at the same data every time. Here's another two examples (vista-errorlog_1.txt, vista-errorlog_2.txt) that I encountered this error. It will appear as RuntimeError when applying transform, making it hard to locate the error in the first place.

[Can-Zhao]

@heyufan1995 It seems this question is related to Vista3D. Do you have any idea? Thanks.

[heyufan1995]

@johnson111788 Hi thanks for the finding. There is a racing condition when InvertD is used along with ThreadDataLoader. The error can be reproduced when the preprocesssing/inversing workload is large and thread dataloader is used. We have an internal fix but has not been updated to model zoo yet. It's a short fix you can do it manually.