validation dice dropped to 0 immediately after some epochs, using 2d segmentation with dicom

[esther1262]

Hi, I want to train a 2d segmentation model on dicom images (dim 256x256). I followed the tutorial unet_training_dict.py. Since I imported the files as dicom, the tensor size is 256x256x1. When using the tutorial codes, I modified the part batch_data["img"].to(device) to batch_data["img"].squeeze(dim=4).to(device) for all training/validation images/labels.
I tried in_channel=1 and out_channel=1, with sigmoid. I also tried in_channel=1 and out_channel=2, with softmax, onehotcoding, argmax (like the one in spleen_segmentation_3d.ipynb). But the training loss and validation dice acted weird after very certain epochs every time (can be 8, 20, 32, no more than 40). All of a sudden, the training loss increased to 0.9 and then validation dice dropped to 0.
Training set = 1740 images, validation = 436.

Does it imply the model not learning after certain epochs?

Anyone knows how to fix it? Thanks!

[KumoLiu]

Hi @esther1262, I would suggest you reduce your learning rate and try to retrain the experiments.
Hope it helps, thanks!

[esther1262]

unfortunately that didnt solve the issue. it just took more epochs for validation dice to 0.