Auto3dSeg GPU utilization optimization

[elitap]

Hi there,

I am currently playing arround with the Auto3dSeg module, but get a really bad gpu-utilization. I like to run the pipeline step by step and create the bundles with GPU utlization optimization:

` gpu_customization_specs = {
'universal': {
'num_trials': 8,
'range_num_images_per_batch': [1, 32],
'range_num_sw_batch_size': [1, 32]
}
}

bundle_generator.generate(workdir, num_fold=5, gpu_customization=True,
    gpu_customization_specs=gpu_customization_specs)

`

this helps to increase the memory utilization, but the gpu-utilization is still really bad!

Is there something simple to tweek in order to increase the training speed? Currently I run the models independently on a single GPU each, as the utilization in the multigpu mode is even worse?

Also I wonder why the output of the gpu_customization is a gpu_opt_3gb.yaml file although I run the optimizatin on a 24gb GPU? Well somethimes both a 3gb und a 24gb yaml file are created, I wonder why?

Thank you already :-)

[elitap]

A well and yes, CPU and memory wise there is still some capacity left. Cachrate seem to be already set to 1.0 and num workes is 8, which is in my experience usually enough to keep the GPU busy, using the given augmentations...