GGUF models inference speed - Why is GGUF model inference fast on my Mac but slow on cluster?

[eltonjohnfanboy]

Hi guys!

I've noticed that GGUF model inference is much faster on my Mac M3 compared to my college's cluster, even when I request for 8 or 16 cores. Both systems run the same GGUF model version and dependencies. The inference in the MAC takes seconds while in the cluster it can take up to 1 hour to generate the response.
Are there known issues with GGUF models on certain CPUs? Any help would be greatly appreciated. Thank you!

[metal3d]

On your Mac, you probably compiled llama.cpp with "Metal" support, that makes inferences usings your GPU. On a CPU, it's slower, even using AVX/AVX2 instructions.

[eltonjohnfanboy]

Hi @metal3d!
Thanks for your answer. I was wondering about something. Even when using the llama-cpp-python binding, where you can specify the n_gpu_layers to zero, I still see the same issue happening, it's way faster in MAC than in cluster. Shouldn't the speeds be similar if you specify not to use any GPU layers? Or am I missing something?
Thanks a lot!

[metal3d]

You should set -1 to load everything in GPU with the python binding.

For Mac and benchmark, sorry, I haven't any more information.

[eltonjohnfanboy]

Great! Thanks for you answer @metal3d :))

[jeanjerome]

Thanks a lot @metal3d for the n_gpu_layers=-1 setting to leverage the Apple Silicon GPU. I was starting to get discouraged seeing my CPU at 100% while my GPU was idle. I couldn't find this information anywhere else (llama.cpp + llama-cpp-bindings), or it's very well hidden!