FP16 uses lesser GPU memory, but training speed unchanged

[athenawisdoms]

I started exploring FP16 training on my Ubuntu machine with two RTX 2070S GPUS.

dm = ...
model  = ...
trainer = Trainer(accelerator="ddp", precision=16)
trainer.fit(model, dm)

Without FP16, the memory usage on GPU 1 is 3.3 GB and the progress bar reports 15 it/s.
With FP16, the memory usage on GPU 1 is 2.6 GB (21% decrease) but the progress bar reports 16 it/s (7% increase). GPU utilization is 94%.

I am using Pytorch 1.7.0, Pytorch Lightning 1.2.5, Python 3.8, CUDA 10.2.89, CUDNN 7.6.5.

Is this improvement in training speed much lesser than expected? I was expecting about a slightly lesser than 50% reduction in GPU memory, and maybe a 1.5X increase in iterations/s.