Add generative model metrics notebook

[yiheng-wang-nv]

Fixes #1770 .

Description

A few sentences describing the changes proposed in this pull request.

Checks

• Avoid including large-size files in the PR.
• Clean up long text outputs from code cells in the notebook.
• For security purposes, please check the contents and remove any sensitive info such as user names and private key.
• Ensure (1) hyperlinks and markdown anchors are working (2) use relative paths for tutorial repo files (3) put figure and graphs in the ./figure folder
• Notebook runs automatically ./runner.sh -t <path to .ipynb file>

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[yiheng-wang-nv]

Hi @ericspod @KumoLiu
the model weights used in this tutorial is removed, and the url mentioned in the original repo does not exist:
https://github.com/marksgraham/pretrained_generative_models/blob/main/models/ddpm_2d.py#L34

Do you have any suggestions? Can we use other weights to replace it?
The cell shows:

if use_pre_trained:
    unet = torch.hub.load("marksgraham/pretrained_generative_models:v0.2", model="ddpm_2d", verbose=True)
    unet = unet.to(device)
else:
    model_path = Path.cwd() / Path("tutorials/generative/2d_ldm/best_aeutoencoderkl.pth")
    autoencoderkl.load_state_dict(torch.load(str(model_path)))
    model_path = Path.cwd() / Path("tutorials/generative/2d_ldm/best_unet.pth")
    unet.load_state_dict(torch.load(str(model_path)))

it seems the 2d_ldm weights can be used, can we prepare a copy for users to download?

[ericspod]

@marksgraham would you have the file that was stored in your google drive? We can rehost it somewhere else perhaps and get this added.

[marksgraham]

Sorry I deleted them in a bulk drive clearout, without realising they were still being used here.

It is easy to regenerate the checkpoint by running the tutorial with use_pretrained=False. Then it can be stored somewhere more permanent..

[yiheng-wang-nv]

Sorry I deleted them in a bulk drive clearout, without realising they were still being used here.

It is easy to regenerate the checkpoint by running the tutorial with use_pretrained=False. Then it can be stored somewhere more permanent..

Hi @marksgraham , thanks for the info. Can I confirm that we at least need to use trained 2d_ldm weights?

[marksgraham]

Sorry I deleted them in a bulk drive clearout, without realising they were still being used here.
It is easy to regenerate the checkpoint by running the tutorial with use_pretrained=False. Then it can be stored somewhere more permanent..

Hi @marksgraham , thanks for the info. Can I confirm that we at least need to use trained 2d_ldm weights?

Hi @yiheng-wang-nv yes they will need to be trained too. You should be able to get trained versions for both of the sets of weights using this tutorial

[yiheng-wang-nv]

Thanks @marksgraham , I will run 2d_ldm tutorial, and save weights into https://github.com/Project-MONAI/MONAI-extra-test-data, then we can use that links in the metric tutorial. cc @ericspod @KumoLiu

[yiheng-wang-nv]

Hi @ericspod , do you have any experiences on this error: https://github.com/Project-MONAI/tutorials/actions/runs/10400120252/job/28800115713?pr=1771#step:5:710
Run the notebook locally does not bring me same error.

[yiheng-wang-nv]

Hi @ericspod , do you have any experiences on this error: https://github.com/Project-MONAI/tutorials/actions/runs/10400120252/job/28800115713?pr=1771#step:5:710 Run the notebook locally does not bring me same error.

I guess the CI machine may have some issues when load radimagenet weights, I changed to use torchvision's resnet50 as the feature extractor, and now tests pass.

[KumoLiu]

I guess the CI machine may have some issues when load radimagenet weights, I changed to use torchvision's resnet50 as the feature extractor, and now tests pass.

I think the update here should be fine since the notebook is mainly talk about the metric used in the generative model. What do you think? @ericspod

[ericspod]

Hi @ericspod , do you have any experiences on this error: https://github.com/Project-MONAI/tutorials/actions/runs/10400120252/job/28800115713?pr=1771#step:5:710
Run the notebook locally does not bring me same error.

If you can get it to work with another network let's go with that. For the future this is related to numerical error with scipy.linalg.sqrtm in the FID calculation that's somewhat documented in other implementations, such as here. It appears to be sensitive to the version of scipy being used or other libraries it relies on.