visualize ThreadDataLoader and GPU transform in Nsight Systems

[hw-ju]

Hi!
The profiling section in the fast_training_tutorial is very useful!
I'm trying to visualize the GPU transform and ThreadDataLoader in the fast training in NsighT Systems UI. Clicking on the range for GPU transform "RandCrop" doesn't correlate with any kernels on the GPU row, please see the two screenshots below:

I'm wondering if the CUDA API call "CatArrayBatchedCopy" right after the range "RandCrop" is the one responsible for launching the GPU transform?

Another quick question, in fast training, the "dataload" range doesn't project to the GPU row, please see the highlighted area in the screenshot below. Is that because nothing is happening on the GPU during that range (i.e., no host-to-device data movement as in regular PyTorch training)?

[Nic-Ma]

Hi @KumoLiu ,

Could you please help share some comments about this question? I think you are also using the Nsight system recently.

Thanks in advance.

[KumoLiu]

I tried to do some comparative experiments and the results are as follows. The height of the blue bars represents utilization rate.

Transform	RandCropByPosNegLabeld	RandAffined	RandFlipd
GPU	~4ms	~7.5ms	~260μs
CPU	~4ms	~52.5ms	~1.5ms
GPU Usage

From the table, we could see that RandAffined has the relatively most significant speed increase when it is moved to the GPU. I guess the reason for this is that affine has more matrix operations, which is more conducive to parallel computing on the GPU. In comparison, the RandCropByPosNegLabeld is merely a slicing operation which may not be conducive to using the parallel computing of GPU to speed up, but the use of EnsureType also reduces the transfer time between different devices, and the overall speed of training is improved (7s/epoch vs 9s/epoch). I'm not sure for the reason why the GPU utilization will improve after crop in each iteration.
https://github.com/Project-MONAI/MONAI/blob/79b8b0ca2a43586e754a97f0e36ae8e5e551b94b/monai/transforms/croppad/array.py#L1107

Do you have any other insight on this？ @Nic-Ma @wyli @yiheng-wang-nv
Also thanks for the guidance from @yiheng-wang-nv!