Closed
Description
[2023-09-25T15:14:51.787Z] Running ./modules/inverse_transforms_and_test_time_augmentations.ipynb
[2023-09-25T15:14:51.787Z] Checking PEP8 compliance...
[2023-09-25T15:14:52.715Z] Running notebook...
[2023-09-25T15:14:52.715Z] Before:
[2023-09-25T15:14:52.715Z] " max_epochs = 20\n",
[2023-09-25T15:14:52.715Z] After:
[2023-09-25T15:14:52.715Z] " max_epochs = 1\n",
[2023-09-25T15:14:52.715Z] Before:
[2023-09-25T15:14:52.715Z] " val_interval = 1\n",
[2023-09-25T15:14:52.970Z] After:
[2023-09-25T15:14:52.970Z] " val_interval = 1\n",
[2023-09-25T15:14:59.502Z] MONAI version: 1.3.0rc1+4.g18a671a7
[2023-09-25T15:14:59.503Z] Numpy version: 1.26.0
[2023-09-25T15:14:59.503Z] Pytorch version: 2.1.0a0+29c30b1
[2023-09-25T15:14:59.503Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2023-09-25T15:14:59.503Z] MONAI rev id: 18a671a7f12389dd8263a4c2232196dcca814bb9
[2023-09-25T15:14:59.503Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2023-09-25T15:14:59.503Z]
[2023-09-25T15:14:59.503Z] Optional dependencies:
[2023-09-25T15:14:59.503Z] Pytorch Ignite version: 0.4.11
[2023-09-25T15:14:59.503Z] ITK version: 5.3.0
[2023-09-25T15:14:59.503Z] Nibabel version: 5.1.0
[2023-09-25T15:14:59.503Z] scikit-image version: 0.21.0
[2023-09-25T15:14:59.503Z] scipy version: 1.11.1
[2023-09-25T15:14:59.503Z] Pillow version: 9.2.0
[2023-09-25T15:14:59.503Z] Tensorboard version: 2.9.0
[2023-09-25T15:14:59.503Z] gdown version: 4.7.1
[2023-09-25T15:14:59.503Z] TorchVision version: 0.16.0a0
[2023-09-25T15:14:59.503Z] tqdm version: 4.65.0
[2023-09-25T15:14:59.503Z] lmdb version: 1.4.1
[2023-09-25T15:14:59.503Z] psutil version: 5.9.4
[2023-09-25T15:14:59.503Z] pandas version: 2.1.1
[2023-09-25T15:14:59.503Z] einops version: 0.6.1
[2023-09-25T15:14:59.503Z] transformers version: 4.21.3
[2023-09-25T15:14:59.503Z] mlflow version: 2.7.1
[2023-09-25T15:14:59.503Z] pynrrd version: 1.0.0
[2023-09-25T15:14:59.503Z] clearml version: 1.13.0
[2023-09-25T15:14:59.503Z]
[2023-09-25T15:14:59.503Z] For details about installing the optional dependencies, please visit:
[2023-09-25T15:14:59.503Z] https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2023-09-25T15:14:59.503Z]
[2023-09-25T15:15:02.018Z] papermill --progress-bar -k python3
[2023-09-25T15:15:02.579Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-09-25T15:15:02.579Z] warnings.warn(
[2023-09-25T15:50:44.295Z]
Executing: 0%| | 0/29 [00:00<?, ?cell/s]
Executing: 3%|▎ | 1/29 [00:01<00:41, 1.49s/cell]
Executing: 14%|█▍ | 4/29 [00:09<01:02, 2.51s/cell]
Executing: 21%|██ | 6/29 [00:15<00:59, 2.58s/cell]
Executing: 34%|███▍ | 10/29 [35:36<1:33:29, 295.23s/cell]
Executing: 38%|███▊ | 11/29 [35:39<1:14:04, 246.92s/cell]
Executing: 38%|███▊ | 11/29 [35:41<58:24, 194.72s/cell]
[2023-09-25T15:50:45.223Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-09-25T15:50:45.223Z] warnings.warn(
[2023-09-25T15:50:45.479Z] Traceback (most recent call last):
[2023-09-25T15:50:45.479Z] File "/usr/local/bin/papermill", line 8, in <module>
[2023-09-25T15:50:45.479Z] sys.exit(papermill())
[2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
[2023-09-25T15:50:45.479Z] return self.main(*args, **kwargs)
[2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
[2023-09-25T15:50:45.479Z] rv = self.invoke(ctx)
[2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
[2023-09-25T15:50:45.479Z] return ctx.invoke(self.callback, **ctx.params)
[2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
[2023-09-25T15:50:45.479Z] return __callback(*args, **kwargs)
[2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
[2023-09-25T15:50:45.479Z] return f(get_current_context(), *args, **kwargs)
[2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 250, in papermill
[2023-09-25T15:50:45.479Z] execute_notebook(
[2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 128, in execute_notebook
[2023-09-25T15:50:45.479Z] raise_for_execution_errors(nb, output_path)
[2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2023-09-25T15:50:45.479Z] raise error
[2023-09-25T15:50:45.479Z] papermill.exceptions.PapermillExecutionError:
[2023-09-25T15:50:45.479Z] ---------------------------------------------------------------------------
[2023-09-25T15:50:45.479Z] Exception encountered at "In [5]":
[2023-09-25T15:50:45.479Z] ---------------------------------------------------------------------------
[2023-09-25T15:50:45.479Z] RuntimeError Traceback (most recent call last)
[2023-09-25T15:50:45.479Z] Cell In[5], line 46
[2023-09-25T15:50:45.479Z] 44 ds_2d = Dataset(data_dicts, transform_2d_slice)
[2023-09-25T15:50:45.479Z] 45 dl_2d = DataLoader(ds_2d, batch_size=1, num_workers=10)
[2023-09-25T15:50:45.479Z] ---> 46 data_dicts_single_slice = list(tqdm(dl_2d))
[2023-09-25T15:50:45.479Z]
[2023-09-25T15:50:45.479Z] File /usr/local/lib/python3.10/dist-packages/tqdm/std.py:1178, in tqdm.__iter__(self)
[2023-09-25T15:50:45.479Z] 1175 time = self._time
[2023-09-25T15:50:45.479Z] 1177 try:
[2023-09-25T15:50:45.479Z] -> 1178 for obj in iterable:
[2023-09-25T15:50:45.479Z] 1179 yield obj
[2023-09-25T15:50:45.479Z] 1180 # Update and possibly print the progressbar.
[2023-09-25T15:50:45.479Z] 1181 # Note: does not call self.update(1) for speed optimisation.
[2023-09-25T15:50:45.479Z]
[2023-09-25T15:50:45.479Z] File /usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:633, in _BaseDataLoaderIter.__next__(self)
[2023-09-25T15:50:45.479Z] 630 if self._sampler_iter is None:
[2023-09-25T15:50:45.479Z] 631 # TODO(https://github.com/pytorch/pytorch/issues/76750)
[2023-09-25T15:50:45.479Z] 632 self._reset() # type: ignore[call-arg]
[2023-09-25T15:50:45.479Z] --> 633 data = self._next_data()
[2023-09-25T15:50:45.479Z] 634 self._num_yielded += 1
[2023-09-25T15:50:45.479Z] 635 if self._dataset_kind == _DatasetKind.Iterable and \
[2023-09-25T15:50:45.479Z] 636 self._IterableDataset_len_called is not None and \
[2023-09-25T15:50:45.479Z] 637 self._num_yielded > self._IterableDataset_len_called:
[2023-09-25T15:50:45.479Z]
[2023-09-25T15:50:45.479Z] File /usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:1348, in _MultiProcessingDataLoaderIter._next_data(self)
[2023-09-25T15:50:45.479Z] 1346 else:
[2023-09-25T15:50:45.479Z] 1347 del self._task_info[idx]
[2023-09-25T15:50:45.479Z] -> 1348 return self._process_data(data)
[2023-09-25T15:50:45.479Z]
[2023-09-25T15:50:45.479Z] File /usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:1374, in _MultiProcessingDataLoaderIter._process_data(self, data)
[2023-09-25T15:50:45.479Z] 1372 self._try_put_index()
[2023-09-25T15:50:45.479Z] 1373 if isinstance(data, ExceptionWrapper):
[2023-09-25T15:50:45.479Z] -> 1374 data.reraise()
[2023-09-25T15:50:45.479Z] 1375 return data
[2023-09-25T15:50:45.479Z]
[2023-09-25T15:50:45.479Z] File /usr/local/lib/python3.10/dist-packages/torch/_utils.py:697, in ExceptionWrapper.reraise(self)
[2023-09-25T15:50:45.479Z] 693 except TypeError:
[2023-09-25T15:50:45.479Z] 694 # If the exception takes multiple arguments, don't try to
[2023-09-25T15:50:45.479Z] 695 # instantiate since we don't know how to
[2023-09-25T15:50:45.479Z] 696 raise RuntimeError(msg) from None
[2023-09-25T15:50:45.479Z] --> 697 raise exception
[2023-09-25T15:50:45.479Z]
[2023-09-25T15:50:45.479Z] RuntimeError: Caught RuntimeError in DataLoader worker process 0.
[2023-09-25T15:50:45.479Z] Original Traceback (most recent call last):
[2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 141, in apply_transform
[2023-09-25T15:50:45.479Z] return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
[2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 98, in _apply_transform
[2023-09-25T15:50:45.479Z] return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
[2023-09-25T15:50:45.479Z] File "/tmp/ipykernel_21374/3811870245.py", line 24, in __call__
[2023-09-25T15:50:45.479Z] fname = os.path.basename(data[key + "_meta_dict"]["filename_or_obj"])
[2023-09-25T15:50:45.479Z] KeyError: 'image_meta_dict'
[2023-09-25T15:50:45.479Z]
[2023-09-25T15:50:45.479Z] The above exception was the direct cause of the following exception:
[2023-09-25T15:50:45.479Z]
[2023-09-25T15:50:45.479Z] Traceback (most recent call last):
[2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 141, in apply_transform
[2023-09-25T15:50:45.479Z] return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
[2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 98, in _apply_transform
[2023-09-25T15:50:45.479Z] return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
[2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/compose.py", line 335, in __call__
[2023-09-25T15:50:45.479Z] result = execute_compose(
[2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/compose.py", line 111, in execute_compose
[2023-09-25T15:50:45.479Z] data = apply_transform(
[2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 171, in apply_transform
[2023-09-25T15:50:45.480Z] raise RuntimeError(f"applying transform {transform}") from e
[2023-09-25T15:50:45.480Z] RuntimeError: applying transform <__main__.SaveSliced object at 0x7f566c30a260>
[2023-09-25T15:50:45.480Z]
[2023-09-25T15:50:45.480Z] The above exception was the direct cause of the following exception:
[2023-09-25T15:50:45.480Z]
[2023-09-25T15:50:45.480Z] Traceback (most recent call last):
[2023-09-25T15:50:45.480Z] File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
[2023-09-25T15:50:45.480Z] data = fetcher.fetch(index)
[2023-09-25T15:50:45.480Z] File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
[2023-09-25T15:50:45.480Z] data = [self.dataset[idx] for idx in possibly_batched_index]
[2023-09-25T15:50:45.480Z] File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
[2023-09-25T15:50:45.480Z] data = [self.dataset[idx] for idx in possibly_batched_index]
[2023-09-25T15:50:45.480Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/data/dataset.py", line 112, in __getitem__
[2023-09-25T15:50:45.480Z] return self._transform(index)
[2023-09-25T15:50:45.480Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/data/dataset.py", line 98, in _transform
[2023-09-25T15:50:45.480Z] return apply_transform(self.transform, data_i) if self.transform is not None else data_i
[2023-09-25T15:50:45.480Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 171, in apply_transform
[2023-09-25T15:50:45.480Z] raise RuntimeError(f"applying transform {transform}") from e
[2023-09-25T15:50:45.480Z] RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f566c30a2c0>