Auto3DSeg notebook and tutorial documentation improvements

auto3dseg/docs/algorithm_generation.md

@@ -4,7 +4,7 @@
 
 <div align="center"> <img src="../figures/algorithm_generation.png" width="600"/> </div>
 
-The module of algorithm generation is to create self-contained algorithm folders for further model training, inference, and validation with various neural netwrok architecture and training recipes. This module takes input configuration ".yaml" files (shown below), dataset summaries (e.g. "datastats.yaml") from our provided data analysis tools, and algorithm templates. And it outputs different algorithm folders under cross-validation. In the default design, the generated algorithm folders follow the designs of [MONAI bundle](https://docs.monai.io/en/latest/mb_specification.html). User can run model training, inference, and validation inside those self-contained folders.
+The module of algorithm generation is to create self-contained algorithm folders for further model training, inference, and validation with various neural network architectures and training recipes. This module takes input configuration ".yaml" files (shown below), dataset summaries (e.g. "data_stats.yaml") from our provided data analysis tools, and algorithm templates. And it outputs different algorithm folders under cross-validation. In the default design, the generated algorithm folders follow the designs of [MONAI bundle](https://docs.monai.io/en/latest/mb_specification.html). Users can run model training, inference, and validation inside those self-contained folders.
 
 ```
 modality: CT
@@ -21,21 +21,22 @@ The default algorithms are based on three different networks, [DiNTS](https://op
 | **Algorithm** | **DiNTS**  | **2D SegResNet**  | **SegResNet**  | **SwinUNETR**  |
 | ------------ | ------------ | ------------ | ------------ | ------------ |
 | **Network** | Densely-connected lattice-based network | U-shape network architecture with 2D residual blocks | U-shape network architecture with 3D residual blocks | U-shape network architecture with Swin-Transformer based encoder |
-| **Training Recipes** | Model Input: <br /> - 96 x 96 x 96 for traing <br /> - 96 x 96 x 96 for inference <br /> AMP: True <br /> Optimizer: SGD <br /> Initial learning Rate: 0.2 <br /> Loss: DiceFocalLoss | Model Input: <br /> - 320 x 320 for traing <br /> - 320 x 320 for inference <br /> AMP: True <br /> Optimizer: SGD <br /> Initial learning Rate: 0.2 <br /> Loss: DiceFocalLoss | Model Input: <br /> - 224 x 224 x 144 for traing <br /> - 224 x 224 x 144 for inference <br /> AMP: True <br /> Optimizer: AdamW <br /> Initial learning Rate: 0.0002 <br /> Loss: DiceCELoss | Model Input: <br /> - 96 x 96 x 96 for traing <br /> - 96 x 96 x 96 for inference <br /> AMP: True <br /> Optimizer: AdamW <br /> Initial learning Rate: 0.0001 <br /> Loss: DiceCELoss |
+| **Training Recipes** | Model Input: <br /> - 96 x 96 x 96 for training <br /> - 96 x 96 x 96 for inference <br /> AMP: True <br /> Optimizer: SGD <br /> Initial learning Rate: 0.2 <br /> Loss: DiceFocalLoss | Model Input: <br /> - 320 x 320 for training <br /> - 320 x 320 for inference <br /> AMP: True <br /> Optimizer: SGD <br /> Initial learning Rate: 0.2 <br /> Loss: DiceFocalLoss | Model Input: <br /> - 224 x 224 x 144 for training <br /> - 224 x 224 x 144 for inference <br /> AMP: True <br /> Optimizer: AdamW <br /> Initial learning Rate: 0.0002 <br /> Loss: DiceCELoss | Model Input: <br /> - 96 x 96 x 96 for training <br /> - 96 x 96 x 96 for inference <br /> AMP: True <br /> Optimizer: AdamW <br /> Initial learning Rate: 0.0001 <br /> Loss: DiceCELoss |
 | **Transforms**  |  - Intensity Normalization <br /> - Random ROI cropping <br /> - Random rotation <br /> - Random zoom <br /> - Random Gaussian smoothing <br /> - Random intensity scaling <br /> - Random intensity shifting <br /> - Random Gaussian noising <br /> - Random flipping | - Intensity Normalization <br /> - Random ROI cropping <br /> - Random rotation <br /> - Random zoom <br /> - Random Gaussian smoothing <br /> - Random intensity scaling <br /> - Random intensity shifting <br /> - Random Gaussian noising <br /> - Random flipping | - Intensity Normalization <br /> - Random ROI cropping <br /> - Random affine transformation <br /> - Random Gaussian smoothing <br /> - Random intensity scaling <br /> - Random intensity shifting <br /> - Random Gaussian noising <br /> - Random flipping | - Intensity Normalization <br /> - Random ROI cropping <br /> - Random rotation <br /> - Random intensity shifting <br /> - Random flipping |
 
-For model inference, we use a sliding-window scheme to generate probability maps for output classes/channels through a softmax/sigmoid layer. The overlap for sliding window inference more than 25\% of the window size. The probability map is re-sampled back to its original spacing through each class channel. Next, a segmentation mask is generated using the argmax or thresholding operation on the channel dimension (with or without model ensemble) and saved with the original affine matrix.
+For model inference, we use a sliding-window scheme to generate probability maps for output classes/channels through a softmax/sigmoid layer. The overlap for sliding window inference is more than 25\% of the window size. The probability map is re-sampled back to its original spacing through each class channel. Next, a segmentation mask is generated using the `argmax` or thresholding operation on the channel dimension (with or without model ensemble) and saved with the original affine matrix.
 
 ### Python Command
 
 The follow Python script shows how to generate algorithm bundles using Python class **BundleGen**.
 
 ```python
 ## algorithm generation
+import os
 from monai.apps.auto3dseg import BundleGen
 
 work_dir = "./work_dir"
-data_output_yaml = os.path.join(work_dir, "datastats.yaml")
+data_output_yaml = os.path.join(work_dir, "data_stats.yaml")
 data_src_cfg = "./task.yaml"
 
 bundle_generator = BundleGen(
@@ -64,7 +65,7 @@ The code block would generate multiple algorithm bundles as follows. The folder
 
 ### Algorithm Templates
 
-The Python class **BundleGen** utilize [the default algorithm templates](https://github.com/Project-MONAI/research-contributions/tree/main/auto3dseg) implicitly. The default algorithms are based on four established works (DiNTS, SegResNet, SegResNet2D, SwinUNETR). They supports both 3D CT and MR image segmentation. In the template, some items are empty or null, and they will be filled together with dataset information. The part of configuration file "hyper_parameters.yaml" is shown below. In the configuration, the items (like "bundle_root", "data_file_base_dir", "patch_size") will be filled up automatically with any user interaction.
+The Python class **BundleGen** utilizes [the default algorithm templates](https://github.com/Project-MONAI/research-contributions/tree/main/auto3dseg) implicitly. The default algorithms are based on four established works (DiNTS, SegResNet, SegResNet2D, and SwinUNETR). They support both 3D CT and MR image segmentation. In the template, some items are empty or null, and they will be filled together with dataset information. The part of the configuration file "hyper_parameters.yaml" is shown below. In the configuration, the items (like "bundle_root", "data_file_base_dir", and "patch_size") will be filled up automatically with any user interaction.
 
 ```
 bundle_root: null
@@ -96,7 +97,7 @@ training:
 ...
 ```
 
-The actual template filling is done using the "fill_template_config" function in the "Algo" class of the script "scripts/algo.py". The "algo.py" of different algorithms are located inside their bundle templates.
+The actual template filling is done using the "fill_template_config" function in the "Algo" class of the script "scripts/algo.py". The "algo.py" of different algorithms is located inside their bundle templates.
 
 ```python
 class DintsAlgo(BundleAlgo):

auto3dseg/docs/bundle.md

@@ -1,6 +1,6 @@
 ## Algorithm Bundle
 
-We provide four default algorithms (DiNTS, 2D/3D SegResNet, SwinUNETR) as baseline algorithms of **Auto3DSeg**. Each algorithm includes adavanced network architectures and well-tuned model training recipes. And the algorithms are formulated in the MONAI package format, with an independent folder structure, providing the necessary functions for model training, verification, and inference. The concept of MONAI bundle can be found in the following [link](https://docs.monai.io/en/latest/mb_specification.html).
+We provide four default algorithms (DiNTS, 2D/3D SegResNet, SwinUNETR) as baseline algorithms of **Auto3DSeg**. Each algorithm includes advanced network architectures and well-tuned model training recipes. And the algorithms are formulated in the MONAI package format, with an independent folder structure, providing the necessary functions for model training, verification, and inference. The concept of the MONAI bundle can be found in the following [link](https://docs.monai.io/en/latest/mb_specification.html).
 
 ### Bundle Structure
 
@@ -27,7 +27,7 @@ algorithm_x
         └── validate.py
 ```
 
-In order to run model training of several bundle algorithms in parallel, users can manually utilize the following bash commands inside bundle folder to launch model traning within different computing environments (via [Python Fire library](https://github.com/google/python-fire)). After model training is accomplished, the model ensemble can be further conducted with existing bundle algorithms. All commands (including model training, inference, validation) are decribed in **README.md** file of each bundle.
+In order to run model training of several bundle algorithms in parallel, users can manually utilize the following bash commands inside bundle folder to launch model training within different computing environments (via [Python Fire library](https://github.com/google/python-fire)). After model training is accomplished, the model ensemble can be further conducted with existing bundle algorithms. All commands (including model training, inference, and validation) are described in **README.md** file of each bundle.
 
 ```bash
 ## single-gpu training
@@ -37,11 +37,11 @@ python -m scripts.train run --config_file "['configs/hyper_parameters.json','con
 torchrun --nnodes=1 --nproc_per_node=8 -m scripts.train run --config_file "['configs/hyper_parameters.json','configs/network.yaml','configs/transforms_train.json','configs/transforms_validate.json']"
 ```
 
-After model is trained, all related output files are saved in the **models** folder, including model checkpoint **best_metric_model.pt** (with best validation accuracy), and training history files. Among all training history files, **Events** folders contains event files for learning curve visualization via [TensorBoard](https://www.tensorflow.org/tensorboard). The file **accuracy_history.csv** lists details about the model training progress, including training loss, validation accuracy, number of training steps, and training time cost. It can be used as an alternative of TensorBoard events. The file **progress.yaml** records the best validation accuracy and when it was reached.
+After the model is trained, all related output files are saved in the **models** folder, including model checkpoint **best_metric_model.pt** (with best validation accuracy), and training history files. Among all training history files, **Events** folders contain event files for learning curve visualization via [TensorBoard](https://www.tensorflow.org/tensorboard). The file **accuracy_history.csv** lists details about the model training progress, including training loss, validation accuracy, number of training steps, and training time cost. It can be used as an alternative of TensorBoard events. The file **progress.yaml** records the best validation accuracy and when it was reached.
 
-### An Quick Way to Try Different Hyper-Parameters
+### A Quick Way to Try Different Hyper-Parameters
 
-In each bundle algorithm, we provide an easy API for users to quickly update hyper-parameters in model training, inference, and validation. User can update any parameter at different levels in the configuration ".yaml" file by appending strings to the bash command starting with "--". The values after "--" would override the default values in the configurations. The following command shows a multi-gpu training example. During the actual model training, learning rate becomes 0.001, **num_images_per_batch** is increased to 6, and the momentum in optimizers  is updated to 0.99.
+In each bundle algorithm, we provide an easy API for users to quickly update hyper-parameters in model training, inference, and validation. Users can update any parameter at different levels in the configuration ".yaml" file by appending strings to the bash command starting with "--". The values after "--" would override the default values in the configurations. The following command shows a multi-GPU training example. During the actual model training, the learning rate becomes 0.001, **num_images_per_batch** is increased to 6, and the momentum in optimizers is updated to 0.99.
 
 ```bash
 torchrun --nnodes=1 --nproc_per_node=8 -m scripts.train run --config_file "['configs/hyper_parameters.json','configs/network.yaml','configs/transforms_train.json','configs/transforms_validate.json']  --learning_rate 0.001 --num_images_per_batch 6 --optimizer#momentum 0.99"

auto3dseg/docs/data_analyzer.md

@@ -4,7 +4,7 @@ In **Auto3DSeg**, the data analyzer automatically analyzes given medical image d
 
 ### Usage
 
-The ussage of data analyzer is can be found [here](../notebooks/data_analyzer.ipynb)
+The usage of data analyzer can be found [here](../notebooks/data_analyzer.ipynb)
 
 ### Customization
 

auto3dseg/docs/ensemble.md

@@ -2,37 +2,32 @@
 
 <div align="center"> <img src="../figures/ensemble.png" width="600"/> </div>
 
-To achieve robust predictions for unseen data, the **auto3dseg** provide a model ensemble module to summarize predictions from various trained models. The module firstly ranks checkpoints of different algorithms based on validation accuracy in each fold of ```N```-fold cross-valition, picks the top-```M``` algorithms from each fold, and creates ensemble predictions using ```MN``` checkpoints. The default ensemble algorithm averages the probability maps (from softmax/sigmoid activations) of candidate predictions.
+To achieve robust predictions for unseen data, the **Auto3dSeg** provides a model ensemble module to summarize predictions from various trained models. The module firstly ranks checkpoints of different algorithms based on validation accuracy in each fold of ```N```-fold cross-validation, picks the top-```M``` algorithms from each fold, and creates ensemble predictions using ```MN``` checkpoints. The default ensemble algorithm averages the probability maps (from softmax/sigmoid activations) of candidate predictions and generates the final outputs.
 
 ### How to Run Model Ensemble Independently
 
-The follow Python script shows how to ensemble predictions from various algorithms using Python class **AlgoEnsembleBuilder**.
+The following Python script shows how to ensemble predictions from various algorithms using the Python class **AlgoEnsembleBuilder**.
 
 ```python
+import os
 from monai.apps.auto3dseg import (
-    BundleGen,
     AlgoEnsembleBestN,
     AlgoEnsembleBuilder,
+    import_bundle_algo_history,
 )
-from monai.apps.auto3dseg.ensemble_builder import AlgoEnsembleKeys
+from monai.utils.enums import AlgoEnsembleKeys
 
-work_dir = "./work_dir"
-data_output_yaml = os.path.join(work_dir, "datastats.yaml")
-data_src_cfg = "./task.yaml"
+# Assuming you have already trained the models
 
-## algorithm generation
-bundle_generator = BundleGen(
-    algo_path=work_dir,
-    data_stats_filename=data_output_yaml,
-    data_src_cfg_name=data_src_cfg,
-)
 
-bundle_generator.generate(work_dir, num_fold=5)
-history = bundle_generator.get_history()
+work_dir = <your_work_dir>  # the algorithm working directory generated by AlgoGen/BundleGen
+input_cfg = <your_task_input_file>  # path to the task input YAML file created by the users
+
+history = import_bundle_algo_history(work_dir, only_trained=True)
 
 ## model ensemble
 n_best = 1
-builder = AlgoEnsembleBuilder(history, data_src_cfg)
+builder = AlgoEnsembleBuilder(history, input_cfg)
 builder.set_ensemble_method(AlgoEnsembleBestN(n_best=n_best))
 ensemble = builder.get_ensemble()
 pred = ensemble()
@@ -47,7 +42,7 @@ for algo in ensemble.get_algo_ensemble():
 
 ### Essential Component for General Algorithm/Mdoel Ensemble
 
-The essential component for model ensemble is the "infer()" function in the "InferClass" class of the script "scripts/infer.py". After class initializaiton of the "InferClass", "infer()" takes image file names as input, and outputs multi-channel probablity maps. And the "infer.py" of different algorithms are located inside their bundle templates. In general, the ensemble module would work for any algorithms as long as the "infer()" function is provided with proper setup.
+The essential component for the model ensemble is the "infer()" function in the "InferClass" class of the script "scripts/infer.py". After class initialization of the "InferClass", "infer()" takes image file names as input, and outputs multi-channel probability maps. And the "infer.py" of different algorithms is located inside their bundle templates. In general, the ensemble module would work for any algorithm as long as the "infer()" function is provided with the proper setup.
 
 ```python
 class InferClass: