The dice of each samples in validation set

[Yangtting]

Hi, I'm following the code spleen_segmentation_3d.ipynb for my multi-label (two targets and background) segmentation task. When I checked the dice metrics of each samples in the validation set with the best saved model after training and validation, the average dice is inconsistent with the highest dice saved in the validation process. It is so strange that the highest dice which was saved in the training and validation process is same as as dice of the last sample in validation set when I checked each dices of samples in the validation set. I guess there must be something wrong in my code, could you please help me, thanks a lot.
The validation process in the training process:

if (epoch + 1) % val_interval == 0:
    model.eval()
    val_loss = 0
    num=0
    with torch.no_grad():
        for val_data in val_loader:
            num+=1
            val_inputs, val_labels = (
                val_data["image"].to(device),
                val_data["label"].to(device),
            )
            roi_size = (160, 160, 160)
            sw_batch_size = 4
            val_outputs = sliding_window_inference(
                val_inputs, roi_size, sw_batch_size, model)
            # compute metric for current iteration
            val_loss1=loss_function(val_outputs,val_labels)
            val_loss += val_loss1.item()
            val_outputs = [post_pred(i) for i in decollate_batch(val_outputs)]
            val_labels = [post_label(i) for i in decollate_batch(val_labels)]
            # compute metric for current iteration
            dice_metric(y_pred=val_outputs, y=val_labels)
        # aggregate the final mean dice result
        # reset the status for next validation round
        metric = dice_metric.aggregate().item()
        dice_metric.reset() 
        metric_values.append(metric)       
        val_loss /= num
        val_loss_values.append(val_loss)     
        if metric > best_metric:
            best_metric = metric
            best_metric_epoch = epoch + 1
            torch.save(model.state_dict(), os.path.join(
                save_dir, "best_metric_model.pth"))
            print("saved new best metric model")
        print(
            f"current epoch: {epoch + 1} current mean dice: {metric:.4f}"
            f"\nbest mean dice: {best_metric:.4f} "
            f"at epoch: {best_metric_epoch}"
        )

Check the dice of each samples in the validation set

model.load_state_dict(torch.load(
    os.path.join(save_dir, "best_metric_model.pth")))
model.eval()
os.chdir(r"C:\Auto segmentation\cal_ar\differentdataset") 
post_pred = Compose([EnsureType(), AsDiscrete(argmax=True, to_onehot=3)])
post_label = Compose([EnsureType(), AsDiscrete(to_onehot=3)])
with torch.no_grad():
    i=1
    dice_vals = list()
    for val_data in val_loader:
        val_inputs, val_labels = (
            val_data["image"].to(device),
            val_data["label"].to(device),
        )
        roi_size = (160, 160, 160)
        sw_batch_size = 4
        val_outputs = sliding_window_inference(
            val_inputs, roi_size, sw_batch_size, model)
        val_outputs_convert = [post_pred(i) for i in decollate_batch(val_outputs)]
        val_labels_convert = [post_label(i) for i in decollate_batch(val_labels)]
        dice_metric(y_pred=val_outputs_convert, y=val_labels_convert)
        dice = dice_metric.aggregate().item()   
        print(dice)
        dice_vals.append(dice)
mean_dice_val= np.mean(dice_vals)
print(mean_dice_val)

[wyli]

by default dice_metric.aggregate() will compute an average over all samples. so you can follow this change for the same validation loop:

        dice_metric(y_pred=val_outputs_convert, y=val_labels_convert)
-        dice = dice_metric.aggregate().item()   
-        print(dice)
-        dice_vals.append(dice)
- mean_dice_val= np.mean(dice_vals)
+ mean_dice_val= dice_metric.aggregate().item()
+ dice_metric.reset()  # clear the buffer
print(mean_dice_val)

[Yangtting]

Thanks for your reply, the mean dice of validation samples by your code matched with the highest dice saved during training and validation. However, how to check out the dice of each sample in the validation set, thanks again.

[wyli]

yes, dice_metric.get_buffer() will return the sample scores

[Yangtting]

Thanks for your help, have a nice day!