cron test error 3d_segmentation/unetr_btcv_segmentation_3d.ipynb

[wyli]

Describe the bug
log: https://github.com/Project-MONAI/MONAI/runs/3073470810?check_suite_focus=true

Running ./3d_segmentation/unetr_btcv_segmentation_3d.ipynb
Checking PEP8 compliance...
stdin:384:2: N816 variable 'split_JSON' in global scope should not be mixedCase
split_JSON = "dataset_0.json"
 ^
stdin:464:17: C408 Unnecessary list call - rewrite as a literal.
    dice_vals = list()
                ^
stdin:466:13: B007 Loop control variable 'step' not used within the loop body. If this is intended, start the name with an underscore.
        for step, batch in enumerate(epoch_iterator_val):
            ^
Error: Try running with autofixes: --autofix.

Check failed!
Running notebook...

Executing:   0%|          | 0/23 [00:00<?, ?cell/s]
Executing:   4%|▍         | 1/23 [00:00<00:18,  1.19cell/s]
Executing:  13%|█▎        | 3/23 [00:06<00:27,  1.37s/cell]
Executing:  17%|█▋        | 4/23 [00:08<00:32,  1.73s/cell]
Executing:  43%|████▎     | 10/23 [00:08<00:15,  1.23s/cell]
Executing:  43%|████▎     | 10/23 [00:10<00:13,  1.02s/cell]
Traceback (most recent call last):
  File "/opt/conda/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/papermill/cli.py", line 250, in papermill
    execute_notebook(
  File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 122, in execute_notebook
    raise_for_execution_errors(nb, output_path)
  File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 234, in raise_for_execution_errors
    raise error
papermill.exceptions.PapermillExecutionError: 
---------------------------------------------------------------------------
Exception encountered at "In [5]":
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-73b1e433c495> in <module>
      2 split_JSON = "dataset_0.json"
      3 datasets = data_dir + split_JSON
----> 4 datalist = load_decathlon_datalist(datasets, True, "training")
      5 val_files = load_decathlon_datalist(datasets, True, "validation")
      6 train_ds = CacheDataset(

/opt/conda/lib/python3.8/site-packages/monai/data/decathlon_datalist.py in load_decathlon_datalist(data_list_file_path, is_segmentation, data_list_key, base_dir)
    114     """
    115     if not os.path.isfile(data_list_file_path):
--> 116         raise ValueError(f"Data list file {data_list_file_path} does not exist.")
    117     with open(data_list_file_path) as json_file:
    118         json_data = json.load(json_file)

ValueError: Data list file /dataset/dataset_0.json does not exist.

[wyli]

this is from our nightly CI tests, could you please help look into the testing error @ahatamiz? I think you can replicate it with this command:

./runner.sh -p "-and \( -name '*unetr*' \)"

[ahatamiz]

this is from our nightly CI tests, could you please help look into the testing error @ahatamiz? I think you can replicate it with this command:

./runner.sh -p "-and \( -name '*unetr*' \)"

Thanks @wyli. The testing error is due to the fact that dataset_0.json is not placed in the /dataset/ folder. As we explained in the Download dataset and format in the folder cell, the user needs to download dataset and put json file on that folder. Since the dataset requires additional registration step, we cannot link to it and download it.

One change I can think of is to host the json file on the cloud, download it and place it in the right folder. But user still needs to download the dataset itself.

[wyli]

sure, we have a share drive for demo/testing data (https://drive.google.com/drive/u/2/folders/1ZOqwF44newvq-jOSgnbXRDWRD9DRq8hH), is it possible to put a subset there?

otherwise I can think of two approaches:

• Try...Except the runtime error in the notebook, so that it stops quietly if there is no data
• modify the default testing filename pattern to exclude this file from auto tests...
  

  tutorials/runner.sh

  Line 57 in 5824072

  pattern="-and -name '*' -and ! -wholename '*federated_learning*'"

I'm fine with all these approaches... cc @rijobro @ericspod @Nic-Ma comments?

[ahatamiz]

Hi @wyli I checked again and unfortunately it is not possible to publicly release portions of dataset ( needs user registration on their website). I can certainly create a new PR that reflects your solutions, if this is what we agree on.

Thanks

[wyli]

Hi @wyli I checked again and unfortunately it is not possible to publicly release portions of dataset ( needs user registration on their website). I can certainly create a new PR that reflects your solutions, if this is what we agree on.

Thanks

Sounds good, we deploy the tests mainly to detect basic errors due to API changes, we also manually check these notebooks before major releases. So some gentle tests or skip the tests are ok...

[wyli]

i'm submitting a PR (#292) to skip the test for unetr.