Thoughts on programming Metal for llama.cpp

[sroussey]

I took a swing at converting some stuff from float to float4 etc here:

https://github.com/sroussey/llama.cpp/pull/1/files

But the speed seems to be the same. @ikawrakow have any thoughts?

[ikawrakow]

So, if it doesn't lead to performance improvement, why would we want to change the code? The proposed version is 2-3 times longer than the original and is also more difficult to read (at least according to my personal taste).

[sroussey]

Oh, this is not a pull request to the main repo.

[sroussey]

How do you evaluate how fast/efficient it runs besides just checking eval time?

I’ve noticed you mention memory bandwidth. How do I get that?

[ikawrakow]

To evaluate performance I don't know any way other than to run some inference task and see how long it takes.

For the memory bandwidth I add a return as the very first statement in the dot product kernel(s) and measure how long it takes for the pipeline to run. This gives the time for all other tasks in the compute graph, including time for thread synchronization and such. I then subtract the time measured that way from the time it takes to evaluate the full graph, and this gives me the time taken by the dot products only. To perform all dot products for token prediction, one needs to fetch from memory the entire model exactly once. So, model size divided by time taken in dot products is the data rate one has achieved by a given kernel implementation. To have a specific example, Q4_0 currently takes ~22 ms on my M2 Max with 30 GPU cores for the 7B model. Run time with empty Q4_0 kernel (return as the first statement) is ~9 ms., hence Q4_0 dot products take ~13 ms. Q4_0 model size is 3.8 GB. So, 3.8 GB / 13 ms = 292 GB/s. The M2 Max has a theoretical maximum memory bandwidth of 400 GB/s, but this can only be achieved if using the CPU and the GPU at the same time. GPU alone is I believe limited to ~300 GB/s, so the Q4_0 Metal dot product kernel comes pretty close to the maximum achievable on my laptop.

[sroussey]

Oh, I see. Thanks. That makes a lot of sense.

Have you tried using the xcode profiler for metal? I myself am new to xcode, so I don't know how to set it up.