Skip to content

Commit 8dab339

Browse files
Nic-Mapre-commit-ci[bot]
Nic-Ma
and
pre-commit-ci[bot]
authored
Add tutorial to extend bundle workflow with event-handler (#1098)
Fixes Project-MONAI/MONAI#5489. ### Description This PR added a tutorial document to extend a bundle workflow with event-handler. ### Checks <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [ ] Notebook runs automatically `./runner [-p <regex_pattern>]` Signed-off-by: Nic Ma <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 7c6ed20 commit 8dab339

File tree

2 files changed

+158
-0
lines changed

2 files changed

+158
-0
lines changed

‎model_zoo/readme.md renamed to ‎model_zoo/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
This folder contains typical usage examples of the bundles in MONAI model-zoo.
2+
13
## Getting Started
24

35
To download and get started with the models, please see also https://monai.io/model-zoo.

‎model_zoo/extend_workflow_with_handler.md

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# Overview
2+
This tutorial shows how to extend the features of workflow in the model-zoo bundles based on `event-handler` mechanism.
3+
Here we try to add the execution time computation logic in the spleen segmentation bundle.
4+
5+
## Event-handler mechanism
6+
The bundles in the `model-zoo` are constructed by MONAI workflow, which can enable quick start of training and evaluation experiments.
7+
The MONAI workflow is compatible with pytorch-ignite `Engine` and `Event-Handler` mechanism: https://pytorch-ignite.ai/.
8+
9+
So we can easily extend new features to the workflow by defining a new independent event handler and attaching to the workflow engine.
10+
11+
### Supported events
12+
Here is all the supported `Event` in MONAI:
13+
| Class | Event name | Description |
14+
| --- | --- | --- |
15+
| ignite.engine.Events | STARTED | triggered when engine's run is started |
16+
| ignite.engine.Events | EPOCH_STARTED | triggered when the epoch is started |
17+
| ignite.engine.Events | GET_BATCH_STARTED | triggered before next batch is fetched |
18+
| ignite.engine.Events | GET_BATCH_COMPLETED | triggered after the batch is fetched |
19+
| ignite.engine.Events | ITERATION_STARTED | triggered when an iteration is started |
20+
| monai.engines.IterationEvents | FORWARD_COMPLETED | triggered when `network(image, label)` is completed |
21+
| monai.engines.IterationEvents | LOSS_COMPLETED | triggered when `loss(pred, label)` is completed |
22+
| monai.engines.IterationEvents | BACKWARD_COMPLETED | triggered when `loss.backward()` is completed |
23+
| monai.engines.IterationEvents | MODEL_COMPLETED | triggered when all the model related operations completed |
24+
| monai.engines.IterationEvents | INNER_ITERATION_STARTED | triggered when the iteration has an inner loop and the loop is started |
25+
| monai.engines.IterationEvents | INNER_ITERATION_COMPLETED | triggered when the iteration has an inner loop and the loop is completed |
26+
| ignite.engine.Events | ITERATION_COMPLETED | triggered when the iteration is ended |
27+
| ignite.engine.Events | DATALOADER_STOP_ITERATION | triggered when dataloader has no more data to provide |
28+
| ignite.engine.Events | EXCEPTION_RAISED | triggered when an exception is encountered |
29+
| ignite.engine.Events | TERMINATE_SINGLE_EPOCH | triggered when the run is about to end the current epoch |
30+
| ignite.engine.Events | TERMINATE | triggered when the run is about to end completely |
31+
| ignite.engine.Events | INTERRUPT | triggered when the run is interrupted |
32+
| ignite.engine.Events | EPOCH_COMPLETED | triggered when the epoch is ended |
33+
| ignite.engine.Events | COMPLETED | triggered when engine's run is completed |
34+
35+
For more information about the `Event` of pytorch-ignite, please refer to:
36+
https://pytorch.org/ignite/generated/ignite.engine.events.Events.html.
37+
38+
Users can also register their own customized `Event` to the workflow engine.
39+
40+
### Develop event handler
41+
A typical handler must contain the `attach()` function and several callback functions to handle the attached events.
42+
For example, here we define a dummy handler to do some logic when iteration started and completed for every 5 iterations:
43+
```py
44+
from ignite.engine import Engine, Events
45+
46+
47+
class DummyHandler:
48+
def attach(self, engine: Engine) -> None:
49+
engine.add_event_handler(Events.ITERATION_STARTED(every=5), self.iteration_started)
50+
engine.add_event_handler(Events.ITERATION_COMPLETED(every=5), self.iteration_completed)
51+
52+
def iteration_started(self, engine: Engine) -> None:
53+
pass
54+
55+
def iteration_completed(self, engine: Engine) -> None:
56+
pass
57+
```
58+
59+
### Get context information of workflow and extend features or debug
60+
Within the handler callback functions, it's easy to get the property objects of `engine` to execute more logic,
61+
like: `engine.network`, `engine.optimizer`, etc. And all the context information are recorded as properties in the `engine.state`:
62+
| Property | Description |
63+
| --- | --- |
64+
| rank | index of current rank in distributed data parallel |
65+
| iteration | index of current iteration |
66+
| epoch | index of current epoch |
67+
| max_epochs | max epoch number to execute |
68+
| epoch_length | iteration number to execute in 1 epoch |
69+
| output | output data of current iteration |
70+
| batch | input data of current iteration |
71+
| metrics | metrics values of current epoch |
72+
| metric_details | details data during metrics computation of current epoch |
73+
| dataloader | dataloader to generate the input data of every iteration |
74+
| device | target device to put the input data |
75+
| key_metric_name | name of the key metric to compare and select the best model |
76+
| best_metric | value of the best metric results |
77+
| best_metric_epoch | epoch index of the best metric value |
78+
79+
Users can also register their own customized properties to the `engine.state`.
80+
81+
To extend features or debug the workflow, we can leverage these information.
82+
For example, here we try to print the `learning rate` value and `current epoch` index within an event callback function:
83+
```py
84+
def epoch_completed(self, engine: Engine) -> None:
85+
print(f"Current epoch: {engine.state.epoch}")
86+
print(f"Learning rate: {engine.optimizer.state_dict()['param_groups'][0]['lr']}")
87+
```
88+
89+
And to extract expected data from the `engine.state.output`, we usually define a `output_transform` callable argument in the handler,
90+
like the existing [StatsHandler](https://docs.monai.io/en/stable/handlers.html#monai.handlers.StatsHandler), [TensorBoardStatsHandler](https://docs.monai.io/en/stable/handlers.html#monai.handlers.TensorBoardStatsHandler), etc.
91+
MONAI contains a convenient utility `monai.handlers.from_engine` to support most of the typical `output_transform` callables.
92+
For more details, please refer to: https://docs.monai.io/en/stable/handlers.html#monai.handlers.utils.from_engine.
93+
94+
## Download example MONAI bundle from model-zoo
95+
```
96+
python -m monai.bundle download --name spleen_ct_segmentation --version "0.1.1" --bundle_dir "./"
97+
```
98+
99+
## Extend the workflow to print the execution time for every iteration, every epoch and total time
100+
Here we define a new handler in `spleen_ct_segmentation/scripts/timer.py` to compute and print the time consumption details:
101+
```py
102+
from time import time
103+
from ignite.engine import Engine, Events
104+
105+
106+
class TimerHandler:
107+
def __init__(self) -> None:
108+
self.start_time = 0
109+
self.epoch_start_time = 0
110+
self.iteration_start_time = 0
111+
112+
def attach(self, engine: Engine) -> None:
113+
engine.add_event_handler(Events.STARTED, self.started)
114+
engine.add_event_handler(Events.EPOCH_STARTED, self.epoch_started)
115+
engine.add_event_handler(Events.ITERATION_STARTED, self.iteration_started)
116+
engine.add_event_handler(Events.ITERATION_COMPLETED, self.iteration_completed)
117+
engine.add_event_handler(Events.EPOCH_COMPLETED, self.epoch_completed)
118+
engine.add_event_handler(Events.COMPLETED, self.completed)
119+
120+
def started(self, engine: Engine) -> None:
121+
self.start_time = time()
122+
123+
def epoch_started(self, engine: Engine) -> None:
124+
self.epoch_start_time = time()
125+
126+
def iteration_started(self, engine: Engine) -> None:
127+
self.iteration_start_time = time()
128+
129+
def iteration_completed(self, engine: Engine) -> None:
130+
print(f"iteration {engine.state.iteration} execution time: {time() - self.iteration_start_time}")
131+
132+
def epoch_completed(self, engine: Engine) -> None:
133+
print(f"epoch {engine.state.epoch} execution time: {time() - self.epoch_start_time}")
134+
135+
def completed(self, engine: Engine) -> None:
136+
print(f"total execution time: {time() - self.start_time}")
137+
```
138+
Then add the handler to the `"train": {"handlers: [...]"}` list of `train.json` config:
139+
```json
140+
{
141+
"_target_": "scripts.timer.TimerHandler"
142+
}
143+
```
144+
145+
## Command example
146+
To run the workflow with this customized handler, `PYTHONPATH` should be revised to include the path to the customized scripts:
147+
```
148+
export PYTHONPATH=$PYTHONPATH:"<path to 'spleen_ct_segmentation/scripts'>"
149+
```
150+
And please make sure the folder `spleen_ct_segmentation/scripts` is a valid python module (it has a `__init__.py` file in the folder).
151+
152+
Execute training:
153+
154+
```
155+
python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf
156+
```

0 commit comments

Comments
 (0)