Auto3DSeg multiprocessing error

[Djennifer]

Hi, I am trying to run your minimal input example with my own data. However it seems that I am getting an RuntimeError: received 0 items of ancdata. I have seen this error before when trying to train with MONAI with multiple dataloader workers. I am suspecting that I need to change the workers to 0 but I have no idea how, or if this is the right solution. So I was hoping someone here could lead me down the right path. I copied the full output down here in case someone would like to take a look.

python -m monai.apps.auto3dseg AutoRunner run --input='./input.yaml'
2022-11-29 13:23:58,649 - INFO - Work directory ./work_dir is used to save all results
2022-11-29 13:23:58,649 - INFO - Loading ./input.yaml for AutoRunner and making a copy in /raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/input.yaml
2022-11-29 13:23:58,660 - INFO - The output_dir is not specified. /raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/ensemble_output will be used to save ensemble predictions
2022-11-29 13:23:58,661 - INFO - Found cached results and skipping data analysis...
2022-11-29 13:23:58,662 - INFO - Found cached results and skipping algorithm generation...
2022-11-29 13:23:58,721 - INFO - Launching: python /raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/scripts/train.py run --config_file='/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/configs/transforms_validate.yaml','/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/configs/transforms_train.yaml','/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/configs/transforms_infer.yaml','/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/configs/network.yaml','/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/configs/hyper_parameters.yaml'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/monai/apps/auto3dseg/bundle_gen.py", line 185, in _run_cmd
    normal_out = subprocess.run(cmd.split(), env=ps_environ, check=True, capture_output=True)
  File "/usr/local/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/scripts/train.py', 'run', "--config_file='/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/configs/transforms_validate.yaml','/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/configs/transforms_train.yaml','/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/configs/transforms_infer.yaml','/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/configs/network.yaml','/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/configs/hyper_parameters.yaml'"]' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/site-packages/monai/apps/auto3dseg/__main__.py", line 22, in <module>
    fire.Fire(
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/monai/apps/auto3dseg/auto_runner.py", line 586, in run
    self._train_algo_in_sequence(history)
  File "/usr/local/lib/python3.8/site-packages/monai/apps/auto3dseg/auto_runner.py", line 488, in _train_algo_in_sequence
    algo.train(self.train_params)
  File "/usr/local/lib/python3.8/site-packages/monai/apps/auto3dseg/bundle_gen.py", line 202, in train
    return self._run_cmd(cmd, devices_info)
  File "/usr/local/lib/python3.8/site-packages/monai/apps/auto3dseg/bundle_gen.py", line 190, in _run_cmd
    raise RuntimeError(f"subprocess call error {e.returncode}: {errors}, {output}") from e
RuntimeError: subprocess call error 1: b'2022-11-29 13:24:10.854814: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-29 13:24:11.235078: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2022-11-29 13:24:12.211506: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library \'libnvinfer.so.7\'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
2022-11-29 13:24:12.211611: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library \'libnvinfer_plugin.so.7\'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
2022-11-29 13:24:12.211622: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Traceback (most recent call last):
  File "/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/scripts/train.py", line 439, in <module>
    fire.Fire()
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/raid/djennifer/data/auto3dseg/Task811_VerSe20/work_dir/segresnet2d_0/scripts/train.py", line 239, in run
    for batch_data in train_loader:
  File "/usr/local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
    idx, data = self._get_data()
  File "/usr/local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data
    success, data = self._try_get_data()
  File "/usr/local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
  File "/usr/local/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 289, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/local/lib/python3.8/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/local/lib/python3.8/multiprocessing/reduction.py", line 189, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/local/lib/python3.8/multiprocessing/reduction.py", line 164, in recvfds
    raise RuntimeError(\'received %d items of ancdata\' %
RuntimeError: received 0 items of ancdata
', b'[info] number of GPUs: 1
[info] world_size: 1
train_files: 71
val_files: 18
num_epochs 1040
num_epochs_per_validation 13
[info] training from scratch
[info] amp enabled
----------
epoch 1/1040
learning rate is set to 0.2
[2022-11-29 13:24:16] 1/36, train_loss: 1.1765

[Djennifer]

I fixed it by running the auto3dseg in a python script with the solutions provided in Project-MONAI/MONAI#701 added to the script