low dice segmentation result for monai0.9.0

[jjhhan]

When I updated to version 0.9, some errors occurred in the transform (0.8 and 0.7 are OK). The dataloader exceeded the RAM memory and prompted a pid (a random number) error. My task is 3D multi organization segmentation. I have simplified the transform according to the tutorial. The model(swin_UNETR and UNETR) can be retrained, but the results are very low. I can post some screenshots of previous training records and current code.
now training process:

old log

the transform code i used now is like this:

train_transforms = [
        LoadImaged(keys=['image', 'label']),
        EnsureChannelFirstd(keys=['image', 'label']),
        RandCropByLabelClassesd(
            keys=["image", "label"],
            label_key="label",
            image_key="image",
            spatial_size=(96, 96, 96),
            num_samples=1,
            num_classes=6,
        ),
    RandFlipd(keys=['image', 'label'], prob=0.15, spatial_axis=1),
    RandFlipd(keys=['image', 'label'], prob=0.15, spatial_axis=0),
    RandFlipd(keys=['image', 'label'], prob=0.15, spatial_axis=2),
    RandAffined(keys=['image', 'label'], mode=('bilinear', 'nearest'), prob=0.1,
                rotate_range=(np.pi / 36, np.pi / 36, np.pi * 2), padding_mode="zeros"),
    RandAffined(keys=['image', 'label'], mode=('bilinear', 'nearest'), prob=0.1,
                rotate_range=(np.pi / 36, np.pi / 2, np.pi / 36), padding_mode="zeros"),
    RandAffined(keys=['image', 'label'], mode=('bilinear', 'nearest'), prob=0.1,
                rotate_range=(np.pi / 2, np.pi / 36, np.pi / 36), padding_mode="zeros"),
    RandGaussianSmoothd(keys=["image"], sigma_x=(0.5, 1.15), sigma_y=(0.5, 1.15), sigma_z=(0.5, 1.15), prob=0.1, ),
    NormalizeIntensityd(
        keys="image", nonzero=True, channel_wise=True),
    ]

val_transforms = [
        LoadImaged(keys=['image', 'label']),
        EnsureChannelFirstd(keys=['image', 'label']),
        NormalizeIntensityd(
        keys="image", nonzero=True, channel_wise=True),
    ]

train_transforms = Compose(train_transforms)
val_transforms = Compose(val_transforms)

train_ds = CacheDataset(    
    data=train_dicts, transform=train_transforms, cache_num=12, copy_cache=True,
    cache_rate=1.0, num_workers=4)
train_loader = DataLoader(train_ds, batch_size=2, shuffle=False, collate_fn=list_data_collate, num_workers=4, pin_memory=True)

val_ds = CacheDataset(
    data=val_dicts, transform=val_transforms, cache_num=6, cache_rate=1.0, num_workers=4)
val_loader = DataLoader(val_ds, batch_size=1, num_workers=4, collate_fn=list_data_collate, pin_memory=True)
net = SwinUNETR(
img_size=(96,96,96),
in_channels=1,
out_channels=6,
feature_size=48,
drop_rate=0.0,
attn_drop_rate=0.0,
dropout_path_rate=0.0,
use_checkpoint=True,
).to(device)

I want to reproduce the results and do not know how to do it

[KumoLiu]

Hi @jjhhan, did you mean when you update to 0.9.0 and use the same code which would cause the ram memory error? Did you check if the memory is occupied by other processes by using htop?

[jjhhan]

guess so, because I saved the training code two months ago and the training log at that time. Recently, when I wanted to reproduce it, I found that there were always errors in the transform and dataloader. My device is 3090 and 64g RAM, and only 11g gpu and 10g RAM are needed for training.
There are two specific cases of errors: 1. epoch1 just started to read 6 data, and then suddenly RAM soared to the upper limit. 2. When I make correct changes, the Train Epoch process is rather slow. I observe nvidia-smi and find that a small amount of data is read each time. Then the GPU is occupied but the utilization rate is very low. The training process of 200-300s is now 2000s and the results are very poor. I can upload a txt of the past and a screenshot of the current operation。

[jjhhan]

cpu and ram

gpu


0818-swin3090.txt

[jjhhan]

i have gone back to version 0.8.1, and the unetr training is just like running on the cpu and
the GPU is occupied but the utilization rate is very low

the record txt i saved 3 months ago.
0513_unetr_task55.txt
then dataloader error
File "/home/dluser/anaconda3/envs/monai/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 999, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 2943326) exited unexpectedly

[KumoLiu]

Hi @jjhhan, if you go back to 0.8.1 and are still stuck in the error, it seems it's not a monai issue, you may need to check your code and see if you have changed something or removed something. When you stopped your training, also check whether the gpu and mem are still occupied. RuntimError can be caused by a variety of reasons and you may need to look a little more closely for the source of the error. BTW, you can set the num_workers as 0 to debug your process. Hope it can help you, Thanks!

[jjhhan]

i update to version 1.0, the training is ok. And the code i used 2 months ago is fine. Now the only question is the dice stay still low , dice is 0.7-0.8 ,the original is 0.9+(you can see that in txt file i uploaded yesterday)。
Thank you for your patient reply

[KumoLiu]

The result of 0.9+ is trained on which version of monai?

[jjhhan]

the version 0.8.1. I remember Monai didn't integrate Swin_UNetR, so I made a version update.