1014-mlflow-handler-tutorial

[binliunls]

Fixes #1014 .

Description

A tutorial about how to use mlflow handler in monai engine based training process by adding one line in code.

Checks

• Notebook runs automatically ./runner [-p <regex_pattern>]

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[mingxin-zheng]

Thanks @binliunls for creating the PR!

We received the request to simplify usage of experiment management to a few lines of code. Just for brainstorming, is it possible to hide the details as much as possible in the bundle config files, so that the tutorial can be something like:

# we provide a user_config_file.json bundle config in this folder
parser = ConfigParser()
parser.read_config("user_config_file.json")
trainer = parser.get_parsed_content("train#trainer")
MLFlowHandler(...).attach(trainer)
trainer.run()

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

Nic-Ma commented on 2022-10-31T11:24:27Z
----------------------------------------------------------------

Line #1.    def render(image, label, prediction, show=False, out_file=None, colormap="spring"):

I didn't see where you use this function?

Thanks.

[Nic-Ma]

Hi @binliunls @mingxin-zheng ,

I feel lost in this tutorial, it doesn't clearly show how easy to use the MLFlowHandler in MONAI workflow.
In my simple mind, I imagine below things to write:

1. A quick notebook for a typical training example with MONAI workflow, and highlight only 2 line code needed to add the MLFlowHandler in the trainer and evaluator.
2. Clearly show all kinds of possible outputs from the MLFlowHandler, users will use this notebook as a reference manual.
3. And also need a markdown doc to describe how to easily add MLFlowHandler in the bundle config, need to show 2 ways: (1) Add config item, (2) Add in the python parser level as @mingxin-zheng described. You can refer to: https://github.com/Project-MONAI/tutorials/tree/main/model_zoo/adapt_bundle_to_another_dataset

What do you think?

Thanks in advance.

[mingxin-zheng]

Thanks @Nic-Ma for the suggestions. As you mention in the last point, the handlers should be added into the bundle config too.

[Nic-Ma]

Thanks for the quick update, put some comments inline.

1. Your current file names are not clear to me, I would suggest to change the 2 file names to:
   bundle_integrate_mlflow.md
workflow_integrate_mlflow.ipynb

2. And put a README.md file in this folder to describe all the files in it.

3. And please remove the training log in the notebook.

4. I would suggest to change the first part description of the notebook to:
   This tutorial shows how to easily enable theMLFlowbased experiment management in the MONAI workflow. Currently, we provide theMLFlowHandlerto add the experiment management in a MONAI workflow training program with just 1 or 2 lines of code, by adding mlflow handler to the handler list: ...

5. And no need to say the default_experiment things, instead, I would suggest to:
   It supports rich parameters to control the experiment tracking logic, ...

Thanks.

[mingxin-zheng]

I tried to run the tutorial and the code executed smoothly. But I am not finding the experiment log on localhost:5000. Did you see similar issue @binliunls ? I have tried both docker and a local linux machine with Conda.

---

Update: my issue seems fixed after the ui process is killed manually as described here

[mingxin-zheng]

Hi @binliunls @Nic-Ma

I did some quick experiment with the current status of MLFlowHandler. The usage with bundle can be simplified and demonstrated in the following lines. The main idea is to tell users that they can override the mlflow handler we're working on in CLI and Pythonic way.

P.S. By using this code, we will drop the handlers defined in the original train#handlers. We can guide user to other tutorials or create new ones about how to use multilple handlers. Of course, it will be nice if ConfigParser support element appending. Then we can append the mlflow handler to the existing handler list.

import os
from monai.apps import download_and_extract
from monai.bundle import run

root_dir = os.environ.get("MONAI_DATA_DIRECTORY")

resource = "https://msd-for-monai.s3-us-west-2.amazonaws.com/Task09_Spleen.tar"
md5 = "410d4a301da4e5b2f6f86ec3ddba524e"

compressed_file = os.path.join(root_dir, "Task09_Spleen.tar")
data_dir = os.path.join(root_dir, "Task09_Spleen")
if not os.path.exists(data_dir):
    download_and_extract(resource, compressed_file, root_dir, md5)
print(data_dir)

# To enable experiment management, one can override the existing trainer handler in the pythonic way:

handler_override = {
    "train#handlers":
        {
            "_target_": "MLFlowHandler",
            "experiment_name": "spleen_seg_ignite",
            "output_transform": "$monai.handlers.from_engine(['loss'], first=True)"
        }
}

run(
    runner_id="training",
    config_file="./bundle/spleen_segmentation/configs/train.json",
    bundle_root=".",
    dataset_dir=data_dir,
    **handler_override
)

"""
Alternatively, you can also use the command line to override the existing handler, by appending

--train#trainer#train_handlers '$@train#handlers + [{"_target_": "MLFlowHandler","experiment_name":"spleen_seg_ignite","output_transform": "$monai.handlers.from_engine([\"loss\"],first=True)"}]'

So the entire command looks like:

python -m monai.bundle run training \
    --config_file ./bundle/spleen_segmentation/configs/train.json \
    --bundle_root . \
    --dataset_dir /workspace/data/Task09_Spleen \
    --train#trainer#train_handlers '$@train#handlers + [{"_target_": "MLFlowHandler","experiment_name":"spleen_seg_ignite","output_transform": "$monai.handlers.from_engine([\"loss\"],first=True)"}]'
"""

[Nic-Ma]

As the previous comments said: (1) Please remove the training log, (2) Please also add the MLFlowHandler to track the metrics during training.

Thanks.

[Nic-Ma]

I put comments inline.
And I see all your code cells contain many blank lines, please format them:

Thanks.

[Nic-Ma]

Thanks for the quick update.
Looks good to me now.
@wyli Do you have any other comments?

Thanks.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

wyli commented on 2022-12-12T10:47:18Z
----------------------------------------------------------------

"only run 5 epochs" it seems the commands set max_epochs to 10?

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

wyli commented on 2022-12-12T10:47:19Z
----------------------------------------------------------------

the installation command is not consistent with the description

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

wyli commented on 2022-12-12T10:47:20Z
----------------------------------------------------------------

perhaps provide a list of attributes that are tracked? it's unclear what the Git Commit and the artifact's name is.

[binliunls]

Hi @wyli ,
I have updated this PR according to your advices. Could you please review it again to see if it looks fine to you?

Thanks,
Bin