Dice Metric and tutorial notebooks are confusing

[cosminbvb]

Hi, I followed the semantic segmentation tutorial notebooks and I have some questions regarding them:

1. In spleen_segmentation_3d.ipynb, why does the approach use 2 output channels and softmax instead of 1 output channel and sigmoid?
2. Same question for /3d_segmentation/challenge_baseline/run_net.py.

Also, I can't seem to understand the documentation for DiceMetric:

1. It states that it computes the Dice Loss. Shouldn't it be the Dice Coefficient instead?
2. What is the correct way of using it? The docs says "y_preds is expected to have binarized predictions and y should be in one-hot format. ". By that I understand that the ground truth should be one hot formatted while the prediction should't. If that's the case, why are the prediction and gt both not in one hot encoding in 3d_segmentation/torch/unet_training_dict.py for example.
3. I tried testing the DiceMetric with the following code snippet:

a = torch.zeros(1, 5, 5, 5)
a[0, 2:4, 2:4, 2:4] = 1
b = torch.zeros(1, 5, 5, 5)
# b[0, 2:4, 2:4, 2:4] = 1
dice = DiceMetric(include_background=True)
dice(a, b)
coeff = dice.aggregate().item()
print(coeff)  # => 0.0

post_trans = Compose([EnsureType(), AsDiscrete(to_onehot=2)])
dice2 = DiceMetric(include_background=True)
dice2(post_trans(a), post_trans(b))
coeff = dice2.aggregate().item()
print(coeff)  # => 0.9652174115180969

# The following produces the error: y_pred and y should have same shapes.
# dice3 = DiceMetric(include_background=True)
# dice3(y_pred=a, y=post_trans(b))
# coeff = dice3.aggregate().item()
# print(coeff)

In my understanding, the first approach does not follow the documentation. The second one, similar to the way it is used in spleen_segmentation_3d.ipynb gives wrong results. The third one, which comes closer to the documentation, produces an error. Could anyone explain what's the correct way of using it?

I had 3 approaches for solving a binary segmentation on CTs and obtained some very unexpected results, with the same model and the same hyperparameters. The only thing that differ are: last activation function and the post transformations applied before computing dice metric.

1. I trained a 3d UNet with 1 output channel (and sigmoid activation prior to the loss function). At evaluation, after sliding window inference, I do the following post transformations and plug them into dice metric:

post_trans_output = Compose([EnsureType(), Activations(sigmoid=True), AsDiscrete(threshold=0.5, to_onehot=2)])
post_trans_gt = Compose([EnsureType(), AsDiscrete(to_onehot=2)])
outputs = [post_trans_output(x) for x in decollate_batch(outputs)]
gt = [post_trans_gt(x) for x in decollate_batch(gt)]

I achieve a best dice coefficient of 75%.

2. I trained a 3d UNet with 1 output channel (and sigmoid activation prior to the loss function). At evaluation, after sliding window inference, I do the following post transformations and plug them into dice metric:

post_trans = Compose([EnsureType(), Activations(sigmoid=True), AsDiscrete(threshold=0.5)])
outputs = [post_trans(x) for x in decollate_batch(outputs)]

I achieve a best dice coefficient of 60% (with the exact same code except for the two lines above).
Can anybody explain why is there such a difference?

3. I trained another 3d UNet but this time with 2 output channels (and softmax activation prior to the loss function). At evaluation, after sliding window inference, I do the following post transformations and plug them into dice metric:

post_trans_output = Compose([EnsureType(), AsDiscrete(argmax=True, to_onehot=2)])
post_trans_gt = Compose([EnsureType(), AsDiscrete(to_onehot=2)])
outputs = [post_trans_output(x) for x in decollate_batch(outputs)]
gt = [post_trans_gt(x) for x in decollate_batch(gt)]

With this I achieve a best dice coefficient of 78%.

If there is anyone that could take the time to answer the question above and / or explain why are there such differences between the approaches, I would very much appreciate it! Thank you!

[Nic-Ma]

Hi @cosminbvb ,

For the usage example of DiceMetric, you can refer to:
https://github.com/Project-MONAI/tutorials/blob/main/3d_segmentation/torch/unet_evaluation_dict.py#L95
Just call the metric in every iteration, then finally call aggregate() and reset().
If there is some misleading doc-string, @yiheng-wang-nv can help update it.
And CC @ericspod @wyli for some dice metric theory questions.

Thanks.

[bibhabasumohapatra]

It would be better to get the formulaes with bit theory in docs

[Nic-Ma]

Hi @bibhabasumohapatra ,

Thanks for your feedback, duplicate to the ticket: #4540 (reply in thread).
I will update the docs soon.

Thanks.

[cosminbvb]

@ericspod @wyli, could you please take a moment to clarify what is the correct way of using DiceMetric (input shape/format/encoding) and clear the waters on the discrepancies detailed above?
Thank you!

[cosminbvb]

Thanks for your reply!

First of all, the doc states that "y_preds and y can be a list of channel-first Tensor (CHW[D]) or a batch-first Tensor (BCHW[D])".
Secondly, shouldn't the second approach (where a and b are one hot encoded) also give a score of 0.8?

[wyli]

so by default the reduction method is "mean", which is the averaged score over the channels, you can use the following function to get the class-wise results:

def dice_coef_monai(y_pred, y_true):
-    dice = DiceMetric(include_background=True)
+    dice = DiceMetric(include_background=True, reduction="none")
-    dice(y_true, y_pred)
+    dice([y_true], [y_pred])
-    coeff = dice.aggregate().item()
+    coeff = dice.aggregate()
    print(coeff)

tensor([[1.]])
tensor([[1., 1.]])
tensor([[0.8000]])
tensor([[0.7500, 0.8000]])

the last case (0.7749999761581421) is an average of background Dice (0.75) and foreground (0.8)...

[cosminbvb]

Just to make sure I understood correctly:
When using one hot encoding, the coefficient is computed channel wise and then default 'mean' reduction is applied. And for a binary segmentation problem where both y and y_pred are in one hot format, I should either set reduction to 'none' and get the second channel coefficient (the one for the foreground) or set include_background=False instead. Is that right?

[wyli]

yes if the interest is in the foreground score for a binary problem.

[cosminbvb]

Thank you for your help, it is much clearer now!

[ericspod]

I've tried to capture everything about the dice loss and metrics in regards to binary and multiclass formulations here: https://gist.github.com/ericspod/529e19038c403ea382cfd79e6012ca49 If I've missed anything or something doesn't make sense here let me know, we can add this eventually to the documentation and amend the docstrings again to clarify behaviour.

[cosminbvb]

Thank you for taking the time to write an in-depth explanation. It is perfectly clear now. I think it would be very useful for newcomers to include it in the documentation.

[ericspod]

Glad to hear it helped. I think we'll do that in some form, I might put the notebook in tutorials as well as write up documentation pages.

[royak]

I've tried to capture everything about the dice loss and metrics in regards to binary and multiclass formulations here: https://gist.github.com/ericspod/529e19038c403ea382cfd79e6012ca49 If I've missed anything or something doesn't make sense here let me know, we can add this eventually to the documentation and amend the docstrings again to clarify behaviour.

Hi
This GitHub link does not exist. Is there a tutorial about this?

[wyli]

yes https://github.com/Project-MONAI/tutorials/blob/main/modules/dice_loss_metric_notes.ipynb

[royak]

Thanks, Wengi.