Skip to content

Commit f5854ea

Browse files
mingxin-zhengmingxin-zheng
authored
Update autorunner notebook (#1044)
Signed-off-by: Mingxin Zheng <[email protected]> Removed a codeblock that `runner.run` will fail because setting `num_iteration` only doesn't work for all Auto3DSeg algos. Improved markdown and corrected typo. ### Checks <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Notebook runs automatically `./runner [-p <regex_pattern>]` Signed-off-by: Mingxin Zheng <[email protected]>
1 parent ac9b16c commit f5854ea

File tree

1 file changed

+76
-126
lines changed

1 file changed

+76
-126
lines changed

‎auto3dseg/notebooks/auto_runner.ipynb

Lines changed: 76 additions & 126 deletions
Original file line numberDiff line numberDiff line change
@@ -8,24 +8,30 @@
88
"\n",
99
"This notebook will introduce `AutoRunner`, the interface to run the Auto3Dseg pipeline with minimal user inputs.\n",
1010
"\n",
11-
"## 1. Set up environment, imports and datasets\n",
12-
"### 1.1 Set up Environment"
11+
"Specifically, it will show the features below:\n",
12+
"1. Use `AutoRunner` with an input config file `input.yaml` example\n",
13+
"2. How to prepare an `input.yaml`\n",
14+
"3. How to configure the input/ouput folders\n",
15+
"4. How to set the internal parameters of **Auto3DSeg** components\n",
16+
"5. How to apply hyper parameter optimization\n",
17+
"\n",
18+
"## Setup environment"
1319
]
1420
},
1521
{
1622
"cell_type": "code",
17-
"execution_count": 1,
23+
"execution_count": null,
1824
"metadata": {},
1925
"outputs": [],
2026
"source": [
21-
"!python -c \"import monai\" || pip install -q \"monai-weekly[nibabel]\""
27+
"!python -c \"import monai\" || pip install -q \"monai-weekly[nibabel, nni, tqdm, cucim, yaml, optuna]\""
2228
]
2329
},
2430
{
2531
"cell_type": "markdown",
2632
"metadata": {},
2733
"source": [
28-
"### 1.2 Set up imports"
34+
"## Setup imports"
2935
]
3036
},
3137
{
@@ -44,10 +50,9 @@
4450
],
4551
"source": [
4652
"import os\n",
53+
"import tempfile\n",
4754
"import torch\n",
4855
"\n",
49-
"from pathlib import Path\n",
50-
"\n",
5156
"from monai.bundle.config_parser import ConfigParser\n",
5257
"from monai.apps import download_and_extract\n",
5358
"\n",
@@ -59,55 +64,35 @@
5964
"cell_type": "markdown",
6065
"metadata": {},
6166
"source": [
62-
"### 1.3 Download public datasets"
67+
"## Download dataset"
6368
]
6469
},
6570
{
6671
"cell_type": "code",
67-
"execution_count": 3,
72+
"execution_count": null,
6873
"metadata": {},
69-
"outputs": [
70-
{
71-
"name": "stderr",
72-
"output_type": "stream",
73-
"text": [
74-
"Task04_Hippocampus.tar: 27.1MB [00:15, 1.88MB/s] "
75-
]
76-
},
77-
{
78-
"name": "stdout",
79-
"output_type": "stream",
80-
"text": [
81-
"2022-10-18 08:11:37,235 - INFO - Downloaded: Task04_Hippocampus.tar\n",
82-
"2022-10-18 08:11:37,235 - INFO - Expected md5 is None, skip md5 check for file Task04_Hippocampus.tar.\n",
83-
"2022-10-18 08:11:37,236 - INFO - Writing into directory: ..\n"
84-
]
85-
},
86-
{
87-
"name": "stderr",
88-
"output_type": "stream",
89-
"text": [
90-
"\n"
91-
]
92-
}
93-
],
74+
"outputs": [],
9475
"source": [
95-
"root = str(Path(\".\"))\n",
76+
"directory = os.environ.get(\"MONAI_DATA_DIRECTORY\")\n",
77+
"root_dir = tempfile.mkdtemp() if directory is None else directory\n",
78+
"print(root_dir)\n",
79+
"\n",
9680
"msd_task = \"Task04_Hippocampus\"\n",
9781
"resource = \"https://msd-for-monai.s3-us-west-2.amazonaws.com/\" + msd_task + \".tar\"\n",
98-
"compressed_file = os.path.join(root, msd_task + \".tar\")\n",
99-
"if os.path.exists(root):\n",
100-
" download_and_extract(resource, compressed_file, root)\n",
10182
"\n",
102-
"dataroot = os.path.join(root, msd_task)\n",
83+
"compressed_file = os.path.join(root_dir, msd_task + \".tar\")\n",
84+
"dataroot = os.path.join(root_dir, msd_task)\n",
85+
"if os.path.exists(dataroot):\n",
86+
" download_and_extract(resource, compressed_file, root_dir)\n",
87+
"\n",
10388
"datalist_file = os.path.join(\"..\", \"tasks\", \"msd\", msd_task, \"msd_\" + msd_task.lower() + \"_folds.json\")"
10489
]
10590
},
10691
{
10792
"cell_type": "markdown",
10893
"metadata": {},
10994
"source": [
110-
"### 1.4 Prepare a input YAML configuration"
95+
"## Prepare a input YAML configuration"
11196
]
11297
},
11398
{
@@ -117,8 +102,8 @@
117102
"outputs": [],
118103
"source": [
119104
"data_src_cfg = {\n",
120-
" \"name\": msd_task, # optional\n",
121-
" \"task\": \"segmentation\", # optional\n",
105+
" \"name\": msd_task, # optional, it is only for your own record\n",
106+
" \"task\": \"segmentation\", # optional, it is only for your own record\n",
122107
" \"modality\": \"MRI\", # required\n",
123108
" \"datalist\": datalist_file, # required\n",
124109
" \"dataroot\": dataroot, # required\n",
@@ -131,7 +116,7 @@
131116
"cell_type": "markdown",
132117
"metadata": {},
133118
"source": [
134-
"## 2. Run the Auto3DSeg pipeline in a few lines of code\n",
119+
"## Run the Auto3DSeg pipeline in a few lines of code\n",
135120
"\n",
136121
"Below is the typical usage of AutoRunner\n",
137122
"```python\n",
@@ -143,26 +128,14 @@
143128
"\n",
144129
"If the user would like to perform a full training in the tutorial, it is recommended to uncomment the `runner.run()` appended at the end of each code block.\n",
145130
"\n",
146-
"### 2.1 Use the default setting"
131+
"## Use the default setting with the input YAML file"
147132
]
148133
},
149134
{
150135
"cell_type": "code",
151-
"execution_count": 5,
136+
"execution_count": null,
152137
"metadata": {},
153-
"outputs": [
154-
{
155-
"name": "stdout",
156-
"output_type": "stream",
157-
"text": [
158-
"2022-10-18 08:11:37,523 - INFO - ./work_dir does not exists. Creating...\n",
159-
"2022-10-18 08:11:37,524 - INFO - ./work_dir created to save all results\n",
160-
"2022-10-18 08:11:37,524 - INFO - Loading ./input.yaml for AutoRunner and making a copy in /workspace/monai/tutorials-in-dev/auto3dseg/notebooks/work_dir/input.yaml\n",
161-
"2022-10-18 08:11:37,531 - INFO - The output_dir is not specified. /workspace/monai/tutorials-in-dev/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions\n",
162-
"2022-10-18 08:11:37,533 - INFO - Directory /workspace/monai/tutorials-in-dev/auto3dseg/notebooks/work_dir/ensemble_output is created to save ensemble predictions\n"
163-
]
164-
}
165-
],
138+
"outputs": [],
166139
"source": [
167140
"runner = AutoRunner(input=input)\n",
168141
"# runner.run()"
@@ -172,23 +145,14 @@
172145
"cell_type": "markdown",
173146
"metadata": {},
174147
"source": [
175-
"### 2.2 Use the dictionary instead of a YAML file as the input"
148+
"## Use the default setting with the dictionary instead of the YAML file as the input"
176149
]
177150
},
178151
{
179152
"cell_type": "code",
180-
"execution_count": 6,
153+
"execution_count": null,
181154
"metadata": {},
182-
"outputs": [
183-
{
184-
"name": "stdout",
185-
"output_type": "stream",
186-
"text": [
187-
"2022-10-18 08:11:37,674 - INFO - Work directory ./work_dir is used to save all results\n",
188-
"2022-10-18 08:11:37,676 - INFO - The output_dir is not specified. /workspace/monai/tutorials-in-dev/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions\n"
189-
]
190-
}
191-
],
155+
"outputs": [],
192156
"source": [
193157
"runner = AutoRunner(input=data_src_cfg)\n",
194158
"# runner.run()"
@@ -198,8 +162,9 @@
198162
"cell_type": "markdown",
199163
"metadata": {},
200164
"source": [
201-
"## 3 Customize and configure the Auto3Dseg\n",
202-
"### 3.1 Set your working directory"
165+
"## Customize working directory\n",
166+
"`AutoRunner` provides the user interfaces to save all the intermediate and final results in a user-specified location.\n",
167+
"Here we use `./my_workspace` as an example"
203168
]
204169
},
205170
{
@@ -228,9 +193,10 @@
228193
"cell_type": "markdown",
229194
"metadata": {},
230195
"source": [
231-
"### 3.2 Use cached result to save computation time\n",
196+
"## Customize result caching\n",
232197
"\n",
233-
"AutoRunner saves intermediate results by default. The user can choose whether it uses the cached results or restart from scratch.\n",
198+
"AutoRunner saves intermediate results by default to save computation time.\n",
199+
"The user can choose whether it uses the cached results or restart from scratch.\n",
234200
"\n",
235201
"If the users want to start from scratch, they can set `not_use_cache` to True"
236202
]
@@ -265,7 +231,7 @@
265231
"cell_type": "markdown",
266232
"metadata": {},
267233
"source": [
268-
"### 3.3 Output Ensemble Result\n",
234+
"## Customize the output folder to save ensemble result\n",
269235
"\n",
270236
"AutoRunner will perform inference on the testing data specified by the `datalist` in the data source config input. The inference result will be written to the `ensemble_output` folder under the working directory in the form of `nii.gz`. The user can choose the format by adding keyword arguments to the AutoRunner. A list of argument can be found in [MONAI tranforms documentation](https://docs.monai.io/en/stable/transforms.html#saveimage)."
271237
]
@@ -294,8 +260,10 @@
294260
"cell_type": "markdown",
295261
"metadata": {},
296262
"source": [
297-
"## 4 Setting Auto3DSeg internal parameters\n",
298-
"### 4.1 Change the number of folds for cross-validation"
263+
"## Setting Auto3DSeg internal parameters\n",
264+
"`Auto3DSeg` has four steps: data analysis, algorithm generation, training, and ensemble. Users can configure the internal parameters of the `AutoRunner` object to customize some steps in the pipeline.\n",
265+
"\n",
266+
"Below, we begin the experiments with a smaller number of cross-validation folds. The default is 5 in the algorithm but we set it to 2 here:"
299267
]
300268
},
301269
{
@@ -323,41 +291,13 @@
323291
"cell_type": "markdown",
324292
"metadata": {},
325293
"source": [
326-
"### 4.2 Customize traininig parameters by override the default values"
327-
]
328-
},
329-
{
330-
"cell_type": "code",
331-
"execution_count": 11,
332-
"metadata": {},
333-
"outputs": [
334-
{
335-
"name": "stdout",
336-
"output_type": "stream",
337-
"text": [
338-
"2022-10-18 08:11:38,312 - INFO - Work directory ./work_dir is used to save all results\n",
339-
"2022-10-18 08:11:38,314 - INFO - Loading ./input.yaml for AutoRunner and making a copy in /workspace/monai/tutorials-in-dev/auto3dseg/notebooks/work_dir/input.yaml\n",
340-
"2022-10-18 08:11:38,320 - INFO - The output_dir is not specified. /workspace/monai/tutorials-in-dev/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions\n"
341-
]
342-
}
343-
],
344-
"source": [
345-
"runner = AutoRunner(input=input)\n",
346-
"# Note: among the provided bundles, most networks takes \"num_iterations\" to control the training iterations except segresnet\n",
347-
"train_param = {\"num_iterations\": 8}\n",
348-
"runner.set_training_params(params=train_param)\n",
349-
"# runner.run()"
350-
]
351-
},
352-
{
353-
"cell_type": "markdown",
354-
"metadata": {},
355-
"source": [
356-
"#### 4.2.1 A common set of training parameter for all algorithm templates\n",
294+
"## Customize training parameters by override the default values\n",
357295
"\n",
358-
"Note: This is for demo purpose. The user doesn't need to specify this training params.\n",
296+
"`set_training_params` in `AutoRunner` provides an interface to change all algorithms' training parameters in one line. \n",
359297
"\n",
360-
"**Auto3DSeg** uses bundle templates to perform training, validation, and inference. The number of epochs/iterations of training is specified by the config files in each template. While we can override them, it is also noted that some bundle templates may use \"num_iterations\" and other may use \"num_epochs\" to iterate. Below is code-block to convert num_epoch to iteration style and override all algorithms with the same training parameters for 1-GPU/2-GPU machine. "
298+
"Note: **Auto3DSeg** uses bundle templates to perform training, validation, and inference. The number of epochs/iterations of training is specified by the config files in each template. While we can override them, it is also noted that some bundle templates may use `num_iterations` and other may use `num_epochs` to iterate.\n",
299+
"\n",
300+
"For demo purpose, below is code-block to convert num_epoch to iteration style and override all algorithms with the same training parameters for 1-GPU/2-GPU machine. \n"
361301
]
362302
},
363303
{
@@ -384,6 +324,7 @@
384324
" \"num_epochs\": num_epoch,\n",
385325
" \"num_warmup_iterations\": n_iter_val,\n",
386326
"}\n",
327+
"runner = AutoRunner(input=input)\n",
387328
"runner.set_training_params(params=train_param)\n",
388329
"# runner.run()\n"
389330
]
@@ -392,7 +333,9 @@
392333
"cell_type": "markdown",
393334
"metadata": {},
394335
"source": [
395-
"### 4.3 Customize the ensemble method (mean vs. majority voting)"
336+
"## Customize the ensemble method\n",
337+
"\n",
338+
"There are two supported methods: \"AlgoEnsembleBestN\" and \"AlgoEnsembleBestByFold\""
396339
]
397340
},
398341
{
@@ -420,7 +363,7 @@
420363
"cell_type": "markdown",
421364
"metadata": {},
422365
"source": [
423-
"### 4.4 Customize the inference parameters by override the default values"
366+
"## Customize the inference parameters by override the default values"
424367
]
425368
},
426369
{
@@ -454,12 +397,13 @@
454397
"cell_type": "markdown",
455398
"metadata": {},
456399
"source": [
457-
"## 5 Train model with HPO (NNI Grid-search)\n",
458-
"### 5.1 Apply HPO to search hyper-parameter in Auto3DSeg\n",
400+
"## Train model with HPO (NNI Grid-search)\n",
459401
"\n",
460-
"Note: Auto3DSeg supports hyper parameter optimization (HPO) via NNI and Optuna backends. Notebook of how to use these modules can be found in this directory.\n",
461-
"AutoRunner supports NNI backend with a grid search method via automatically generating a the NNI config and run `nnictl` commands in subprocess.\n",
462-
"Note: to run the HPO, you need to ensure the development environment has `nni` package. Please refer to the [MONAI Installation Guide](https://docs.monai.io/en/stable/installation.html#installing-the-recommended-dependencies) for how to install the recommended dependencies."
402+
"**Auto3DSeg** supports hyper parameter optimization (HPO) via `NNI` and `Optuna` backends.\n",
403+
"AutoRunner supports `NNI` backend with a grid search method via automatically generating a the `NNI` config and run `nnictl` commands in subprocess.\n",
404+
"\n",
405+
"Note: to run the HPO, you need to ensure the development environment has `nni` package.\n",
406+
"Please refer to the [MONAI Installation Guide](https://docs.monai.io/en/stable/installation.html#installing-the-recommended-dependencies) for how to install the recommended dependencies."
463407
]
464408
},
465409
{
@@ -488,9 +432,10 @@
488432
"cell_type": "markdown",
489433
"metadata": {},
490434
"source": [
491-
"### 5.2 Override the templated values\n",
435+
"## Override the templated values\n",
436+
"\n",
437+
"The default `NNI` config that `AutoRunner` looks like below. User can override some of the parameters via the `set_hpo_params` interface:\n",
492438
"\n",
493-
"AutoRunner uses the following NNI config in its HPO module\n",
494439
"```python\n",
495440
"default_nni_config = {\n",
496441
" \"trialCodeDirectory\": \".\",\n",
@@ -501,9 +446,7 @@
501446
" \"tuner\": {\"name\": \"GridSearch\"},\n",
502447
" \"trainingService\": {\"platform\": \"local\", \"useActiveGpu\": True},\n",
503448
"}\n",
504-
"```\n",
505-
"\n",
506-
"It can be override by setting the hpo parameters"
449+
"```"
507450
]
508451
},
509452
{
@@ -534,7 +477,14 @@
534477
"cell_type": "markdown",
535478
"metadata": {},
536479
"source": [
537-
"## 6 Conclusion\n",
480+
"For more details about the usage of **Auto3DSeg** HPO features, please check the [Auto3DSeg NNI Notebok](./hpo_nni.ipynb) and [Auto3DSeg Optuna Notebook](./hpo_optuna.ipynb)"
481+
]
482+
},
483+
{
484+
"cell_type": "markdown",
485+
"metadata": {},
486+
"source": [
487+
"## Conclusion\n",
538488
"\n",
539489
"Here we demonstrate how to use the AutoRunner APIs to customize your **Auto3DSeg** pipeline with mininal inputs. Don't forget you need to execute the `run` command to start the training and make everything take effect.\n",
540490
"\n",
@@ -546,7 +496,7 @@
546496
],
547497
"metadata": {
548498
"kernelspec": {
549-
"display_name": "Python 3.8.13 ('base')",
499+
"display_name": "Python 3.8.10 64-bit",
550500
"language": "python",
551501
"name": "python3"
552502
},
@@ -560,12 +510,12 @@
560510
"name": "python",
561511
"nbconvert_exporter": "python",
562512
"pygments_lexer": "ipython3",
563-
"version": "3.8.13"
513+
"version": "3.8.10"
564514
},
565515
"orig_nbformat": 4,
566516
"vscode": {
567517
"interpreter": {
568-
"hash": "d4d1e4263499bec80672ea0156c357c1ee493ec2b1c70f0acce89fc37c4a6abe"
518+
"hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
569519
}
570520
}
571521
},

0 commit comments

Comments
 (0)