Missing Saved Checkpoints when using Multiple Loggers

[athenawisdoms]

Hello Lightning gods!

When using a single logger (without specifying a logger when creating the Trainer) and also using a ModelCheckpoint callback to automatically save good checkpoints, everything works as expected: The directory created by TensorBoard at <path_to_project>/lightning_logs/<experiment_name>/version_<N> contains hparams.yaml and a directory checkpoints containing several .ckpt files.

checkpoint_callback = ModelCheckpoint(
    monitor="val_loss",
    save_top_k=5,
)
trainer = pl.Trainer(
    gpus=-1,
    accelerator="ddp",
    callbacks=[checkpoint_callback],
)

However, when I added a second logger, the checkpoints directory is no longer find inside the lightning_logs subdirectories. These subdirectories created by TensorBoard only contains hparams.yaml and a events.out.tfevents.* file.

from aim.pytorch_lightning import AimLogger
from pytorch_lightning.loggers import TensorBoardLogger

checkpoint_callback = ModelCheckpoint(
    monitor="val_loss",
    save_top_k=5,
)
tb_logger = TensorBoardLogger("lightning_logs", name="my_experiment")
aim_logger = AimLogger(experiment="my_experiment")
trainer = pl.Trainer(
    gpus=-1,
    accelerator="ddp",
    callbacks=[checkpoint_callback],
    logger=[aim_logger, tb_logger],
)

How can we configure Lightning to also include the checkpoint files in the TensorBoard log directories, just like in the original case?

[awaelchli]

Hi
When using multiple loggers, there is no canonical way to store the checkpoints in subdirs. What Lightning currently does is put the checkpoints one level above in a directory with the names of the loggers concatenated:

This is very reasonable, since both loggers were used to produce the same checkpoints.
There are alternatives, for example saving the checkpoints in the experiment dir of the first logger in the list, or copy the checkpoints to both subdirs, but this is not implemented.