metal: concurrently dispatch commands #2358
Conversation
Function `ggml_metal_graph_find_concurrency` will run and write commands that can be issued concurrently to metal context `concur_list` array, when `ggml_metal_graph_compute` is called for the first time.
ggml-metal.m
Outdated
| if (!ctx->concur_list_len) { | ||
| ggml_metal_graph_find_concurrency(ctx,gf); | ||
| } |
There was a problem hiding this comment.
Seems like this will break when computing graphs of different topologies.
There was a problem hiding this comment.
I think for now we can assume that during the lifetime of metal ctx, the topology of graph won't change. In future, when we need to change the topology of graph and we also have a mechanism to tell the backend that the graph topology is changed, we can easily add the necessary code to address the updated topology.
There was a problem hiding this comment.
For llama that's true, but there are other users of ggml.
There was a problem hiding this comment.
Maybe instead of letting ggml-metal.m automatically call ggml_metal_graph_find_concurrency, we let llama.cpp decide if we should call ggml_metal_graph_find_concurrency and set the metal_ctx->concur_list?
The logic on ggml-metal.m will be not so intrusive: If metal_ctx->concur_list is set then dispatch ops concurrently, otherwise fallback to the original code path.
This will add backend-specific code in llama.cpp for now, but I imagine that in future we can have a ggml_backend_graph_optimize() interface to unify them.
There was a problem hiding this comment.
Sounds good to me. Eventually we will need a better solution, but for now that should do. @ggerganov
Discussion see #2309
Function
ggml_metal_graph_find_concurrencywill write commands that can be issued concurrently to metal contextconcur_listarray, whenggml_metal_graph_computeis called for the first time.Tested on M1 Max 32c gpu with :
./main -m model_file -n 256 -c 512 -s 123 -p "I believe the meaning of life is" --ignore-eos -ngl 1 --no-mmap -t 8Btw, since we already have much faster kernels for both
Q_Kandnon Q_K, is it a good time to do prompt evaluation on metal? Although current kernels are optimized for mat-vec multiplications, they still provide much better performance than cpu. (EDIT: not true for M1 Pro)