You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Auto3DSeg notebook and tutorial documentation improvements (Project-MONAI#1065)
Fixes # 1047.
### Description
- Fix HPO overriding parameters doesn't work for DiNTS in NNI
- Fix documentation issues, such as typo, bad sentences,
- Auto3DSeg notebook running failures on 8-GPU machine
- Update the notebook writing style to match other MONAI tutorials
### Checks
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [ ] Notebook runs automatically `./runner [-p <regex_pattern>]`
Signed-off-by: Mingxin Zheng <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The module of algorithm generation is to create self-contained algorithm folders for further model training, inference, and validation with various neural netwrok architecture and training recipes. This module takes input configuration ".yaml" files (shown below), dataset summaries (e.g. "datastats.yaml") from our provided data analysis tools, and algorithm templates. And it outputs different algorithm folders under cross-validation. In the default design, the generated algorithm folders follow the designs of [MONAI bundle](https://docs.monai.io/en/latest/mb_specification.html). User can run model training, inference, and validation inside those self-contained folders.
7
+
The module of algorithm generation is to create self-contained algorithm folders for further model training, inference, and validation with various neural network architectures and training recipes. This module takes input configuration ".yaml" files (shown below), dataset summaries (e.g. "data_stats.yaml") from our provided data analysis tools, and algorithm templates. And it outputs different algorithm folders under cross-validation. In the default design, the generated algorithm folders follow the designs of [MONAI bundle](https://docs.monai.io/en/latest/mb_specification.html). Users can run model training, inference, and validation inside those self-contained folders.
8
8
9
9
```
10
10
modality: CT
@@ -21,21 +21,22 @@ The default algorithms are based on three different networks, [DiNTS](https://op
|**Network**| Densely-connected lattice-based network | U-shape network architecture with 2D residual blocks | U-shape network architecture with 3D residual blocks | U-shape network architecture with Swin-Transformer based encoder |
24
-
|**Training Recipes**| Model Input: <br /> - 96 x 96 x 96 for traing <br /> - 96 x 96 x 96 for inference <br /> AMP: True <br /> Optimizer: SGD <br /> Initial learning Rate: 0.2 <br /> Loss: DiceFocalLoss | Model Input: <br /> - 320 x 320 for traing <br /> - 320 x 320 for inference <br /> AMP: True <br /> Optimizer: SGD <br /> Initial learning Rate: 0.2 <br /> Loss: DiceFocalLoss | Model Input: <br /> - 224 x 224 x 144 for traing <br /> - 224 x 224 x 144 for inference <br /> AMP: True <br /> Optimizer: AdamW <br /> Initial learning Rate: 0.0002 <br /> Loss: DiceCELoss | Model Input: <br /> - 96 x 96 x 96 for traing <br /> - 96 x 96 x 96 for inference <br /> AMP: True <br /> Optimizer: AdamW <br /> Initial learning Rate: 0.0001 <br /> Loss: DiceCELoss |
24
+
|**Training Recipes**| Model Input: <br /> - 96 x 96 x 96 for training <br /> - 96 x 96 x 96 for inference <br /> AMP: True <br /> Optimizer: SGD <br /> Initial learning Rate: 0.2 <br /> Loss: DiceFocalLoss | Model Input: <br /> - 320 x 320 for training <br /> - 320 x 320 for inference <br /> AMP: True <br /> Optimizer: SGD <br /> Initial learning Rate: 0.2 <br /> Loss: DiceFocalLoss | Model Input: <br /> - 224 x 224 x 144 for training <br /> - 224 x 224 x 144 for inference <br /> AMP: True <br /> Optimizer: AdamW <br /> Initial learning Rate: 0.0002 <br /> Loss: DiceCELoss | Model Input: <br /> - 96 x 96 x 96 for training <br /> - 96 x 96 x 96 for inference <br /> AMP: True <br /> Optimizer: AdamW <br /> Initial learning Rate: 0.0001 <br /> Loss: DiceCELoss |
25
25
|**Transforms**| - Intensity Normalization <br /> - Random ROI cropping <br /> - Random rotation <br /> - Random zoom <br /> - Random Gaussian smoothing <br /> - Random intensity scaling <br /> - Random intensity shifting <br /> - Random Gaussian noising <br /> - Random flipping | - Intensity Normalization <br /> - Random ROI cropping <br /> - Random rotation <br /> - Random zoom <br /> - Random Gaussian smoothing <br /> - Random intensity scaling <br /> - Random intensity shifting <br /> - Random Gaussian noising <br /> - Random flipping | - Intensity Normalization <br /> - Random ROI cropping <br /> - Random affine transformation <br /> - Random Gaussian smoothing <br /> - Random intensity scaling <br /> - Random intensity shifting <br /> - Random Gaussian noising <br /> - Random flipping | - Intensity Normalization <br /> - Random ROI cropping <br /> - Random rotation <br /> - Random intensity shifting <br /> - Random flipping |
26
26
27
-
For model inference, we use a sliding-window scheme to generate probability maps for output classes/channels through a softmax/sigmoid layer. The overlap for sliding window inference more than 25\% of the window size. The probability map is re-sampled back to its original spacing through each class channel. Next, a segmentation mask is generated using the argmax or thresholding operation on the channel dimension (with or without model ensemble) and saved with the original affine matrix.
27
+
For model inference, we use a sliding-window scheme to generate probability maps for output classes/channels through a softmax/sigmoid layer. The overlap for sliding window inference is more than 25\% of the window size. The probability map is re-sampled back to its original spacing through each class channel. Next, a segmentation mask is generated using the `argmax` or thresholding operation on the channel dimension (with or without model ensemble) and saved with the original affine matrix.
28
28
29
29
### Python Command
30
30
31
31
The follow Python script shows how to generate algorithm bundles using Python class **BundleGen**.
@@ -64,7 +65,7 @@ The code block would generate multiple algorithm bundles as follows. The folder
64
65
65
66
### Algorithm Templates
66
67
67
-
The Python class **BundleGen**utilize[the default algorithm templates](https://github.com/Project-MONAI/research-contributions/tree/main/auto3dseg) implicitly. The default algorithms are based on four established works (DiNTS, SegResNet, SegResNet2D, SwinUNETR). They supports both 3D CT and MR image segmentation. In the template, some items are empty or null, and they will be filled together with dataset information. The part of configuration file "hyper_parameters.yaml" is shown below. In the configuration, the items (like "bundle_root", "data_file_base_dir", "patch_size") will be filled up automatically with any user interaction.
68
+
The Python class **BundleGen**utilizes[the default algorithm templates](https://github.com/Project-MONAI/research-contributions/tree/main/auto3dseg) implicitly. The default algorithms are based on four established works (DiNTS, SegResNet, SegResNet2D, and SwinUNETR). They support both 3D CT and MR image segmentation. In the template, some items are empty or null, and they will be filled together with dataset information. The part of the configuration file "hyper_parameters.yaml" is shown below. In the configuration, the items (like "bundle_root", "data_file_base_dir", and "patch_size") will be filled up automatically with any user interaction.
68
69
69
70
```
70
71
bundle_root: null
@@ -96,7 +97,7 @@ training:
96
97
...
97
98
```
98
99
99
-
The actual template filling is done using the "fill_template_config" function in the "Algo" class of the script "scripts/algo.py". The "algo.py" of different algorithms are located inside their bundle templates.
100
+
The actual template filling is done using the "fill_template_config" function in the "Algo" class of the script "scripts/algo.py". The "algo.py" of different algorithms is located inside their bundle templates.
Copy file name to clipboardExpand all lines: auto3dseg/docs/bundle.md
+5-5Lines changed: 5 additions & 5 deletions
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
## Algorithm Bundle
2
2
3
-
We provide four default algorithms (DiNTS, 2D/3D SegResNet, SwinUNETR) as baseline algorithms of **Auto3DSeg**. Each algorithm includes adavanced network architectures and well-tuned model training recipes. And the algorithms are formulated in the MONAI package format, with an independent folder structure, providing the necessary functions for model training, verification, and inference. The concept of MONAI bundle can be found in the following [link](https://docs.monai.io/en/latest/mb_specification.html).
3
+
We provide four default algorithms (DiNTS, 2D/3D SegResNet, SwinUNETR) as baseline algorithms of **Auto3DSeg**. Each algorithm includes advanced network architectures and well-tuned model training recipes. And the algorithms are formulated in the MONAI package format, with an independent folder structure, providing the necessary functions for model training, verification, and inference. The concept of the MONAI bundle can be found in the following [link](https://docs.monai.io/en/latest/mb_specification.html).
4
4
5
5
### Bundle Structure
6
6
@@ -27,7 +27,7 @@ algorithm_x
27
27
└── validate.py
28
28
```
29
29
30
-
In order to run model training of several bundle algorithms in parallel, users can manually utilize the following bash commands inside bundle folder to launch model traning within different computing environments (via [Python Fire library](https://github.com/google/python-fire)). After model training is accomplished, the model ensemble can be further conducted with existing bundle algorithms. All commands (including model training, inference, validation) are decribed in **README.md** file of each bundle.
30
+
In order to run model training of several bundle algorithms in parallel, users can manually utilize the following bash commands inside bundle folder to launch model training within different computing environments (via [Python Fire library](https://github.com/google/python-fire)). After model training is accomplished, the model ensemble can be further conducted with existing bundle algorithms. All commands (including model training, inference, and validation) are described in **README.md** file of each bundle.
31
31
32
32
```bash
33
33
## single-gpu training
@@ -37,11 +37,11 @@ python -m scripts.train run --config_file "['configs/hyper_parameters.json','con
37
37
torchrun --nnodes=1 --nproc_per_node=8 -m scripts.train run --config_file "['configs/hyper_parameters.json','configs/network.yaml','configs/transforms_train.json','configs/transforms_validate.json']"
38
38
```
39
39
40
-
After model is trained, all related output files are saved in the **models** folder, including model checkpoint **best_metric_model.pt** (with best validation accuracy), and training history files. Among all training history files, **Events** folders contains event files for learning curve visualization via [TensorBoard](https://www.tensorflow.org/tensorboard). The file **accuracy_history.csv** lists details about the model training progress, including training loss, validation accuracy, number of training steps, and training time cost. It can be used as an alternative of TensorBoard events. The file **progress.yaml** records the best validation accuracy and when it was reached.
40
+
After the model is trained, all related output files are saved in the **models** folder, including model checkpoint **best_metric_model.pt** (with best validation accuracy), and training history files. Among all training history files, **Events** folders contain event files for learning curve visualization via [TensorBoard](https://www.tensorflow.org/tensorboard). The file **accuracy_history.csv** lists details about the model training progress, including training loss, validation accuracy, number of training steps, and training time cost. It can be used as an alternative of TensorBoard events. The file **progress.yaml** records the best validation accuracy and when it was reached.
41
41
42
-
### An Quick Way to Try Different Hyper-Parameters
42
+
### A Quick Way to Try Different Hyper-Parameters
43
43
44
-
In each bundle algorithm, we provide an easy API for users to quickly update hyper-parameters in model training, inference, and validation. User can update any parameter at different levels in the configuration ".yaml" file by appending strings to the bash command starting with "--". The values after "--" would override the default values in the configurations. The following command shows a multi-gpu training example. During the actual model training, learning rate becomes 0.001, **num_images_per_batch** is increased to 6, and the momentum in optimizers is updated to 0.99.
44
+
In each bundle algorithm, we provide an easy API for users to quickly update hyper-parameters in model training, inference, and validation. Users can update any parameter at different levels in the configuration ".yaml" file by appending strings to the bash command starting with "--". The values after "--" would override the default values in the configurations. The following command shows a multi-GPU training example. During the actual model training, the learning rate becomes 0.001, **num_images_per_batch** is increased to 6, and the momentum in optimizers is updated to 0.99.
To achieve robust predictions for unseen data, the **auto3dseg**provide a model ensemble module to summarize predictions from various trained models. The module firstly ranks checkpoints of different algorithms based on validation accuracy in each fold of ```N```-fold cross-valition, picks the top-```M``` algorithms from each fold, and creates ensemble predictions using ```MN``` checkpoints. The default ensemble algorithm averages the probability maps (from softmax/sigmoid activations) of candidate predictions.
5
+
To achieve robust predictions for unseen data, the **Auto3dSeg**provides a model ensemble module to summarize predictions from various trained models. The module firstly ranks checkpoints of different algorithms based on validation accuracy in each fold of ```N```-fold cross-validation, picks the top-```M``` algorithms from each fold, and creates ensemble predictions using ```MN``` checkpoints. The default ensemble algorithm averages the probability maps (from softmax/sigmoid activations) of candidate predictions and generates the final outputs.
6
6
7
7
### How to Run Model Ensemble Independently
8
8
9
-
The follow Python script shows how to ensemble predictions from various algorithms using Python class **AlgoEnsembleBuilder**.
9
+
The following Python script shows how to ensemble predictions from various algorithms using the Python class **AlgoEnsembleBuilder**.
10
10
11
11
```python
12
+
import os
12
13
from monai.apps.auto3dseg import (
13
-
BundleGen,
14
14
AlgoEnsembleBestN,
15
15
AlgoEnsembleBuilder,
16
+
import_bundle_algo_history,
16
17
)
17
-
from monai.apps.auto3dseg.ensemble_builderimport AlgoEnsembleKeys
@@ -47,7 +42,7 @@ for algo in ensemble.get_algo_ensemble():
47
42
48
43
### Essential Component for General Algorithm/Mdoel Ensemble
49
44
50
-
The essential component for model ensemble is the "infer()" function in the "InferClass" class of the script "scripts/infer.py". After class initializaiton of the "InferClass", "infer()" takes image file names as input, and outputs multi-channel probablity maps. And the "infer.py" of different algorithms are located inside their bundle templates. In general, the ensemble module would work for any algorithms as long as the "infer()" function is provided with proper setup.
45
+
The essential component for the model ensemble is the "infer()" function in the "InferClass" class of the script "scripts/infer.py". After class initialization of the "InferClass", "infer()" takes image file names as input, and outputs multi-channel probability maps. And the "infer.py" of different algorithms is located inside their bundle templates. In general, the ensemble module would work for any algorithm as long as the "infer()" function is provided with the proper setup.
![pre-commit-ci[bot]](https://avatars.githubusercontent.com/in/68672?v=4&size=40){kind=avatar}
0 commit comments