Skip to content

vulkan: enable coopmat2 FA gqa and split_k optimizations more often #12931

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 16, 2025

Conversation

jeffbolznv
Copy link
Collaborator

The grouped query attention optmization doesn't require a power of two ratio, the only thing relying on it was the modulo operation written as bitwise &.

split_k need not depend on gqa_ratio - enable it any time there's only one workgroup in the X dimension. The shader gets the split index from the x coord, and multiple workgroups in the X dimension (pre-split) indicates a larger FA operation that wouldn't need splitting.

Perf results:

before:

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -m C:\models\second-state\StarCoder2-7B-GGUF\starcoder2-7b-Q4_0.gguf -m C:\models\Qwen2-VL-7B-Instruct-IQ4_NL.gguf -m C:\models\Phi-3-mini-4k-instruct-q4.gguf -fa 1 -p 0 -n 8192 --repetitions 1
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4070 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
| model                          |       size |     params | backend    | ngl | fa |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ------------: | -------------------: |
| starcoder2 7B Q4_0             |   3.76 GiB |     7.17 B | Vulkan     |  99 |  1 |        tg8192 |         55.98 ± 0.00 |
| qwen2vl 7B IQ4_NL - 4.5 bpw    |   4.13 GiB |     7.62 B | Vulkan     |  99 |  1 |        tg8192 |         57.88 ± 0.00 |
| phi3 3B Q4_K - Medium          |   2.23 GiB |     3.82 B | Vulkan     |  99 |  1 |        tg8192 |         68.98 ± 0.00 |

after:

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -m C:\models\second-state\StarCoder2-7B-GGUF\starcoder2-7b-Q4_0.gguf -m C:\models\Qwen2-VL-7B-Instruct-IQ4_NL.gguf -m C:\models\Phi-3-mini-4k-instruct-q4.gguf -fa 1 -p 0 -n 8192 --repetitions 1
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4070 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
| model                          |       size |     params | backend    | ngl | fa |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ------------: | -------------------: |
| starcoder2 7B Q4_0             |   3.76 GiB |     7.17 B | Vulkan     |  99 |  1 |        tg8192 |         74.72 ± 0.00 |
| qwen2vl 7B IQ4_NL - 4.5 bpw    |   4.13 GiB |     7.62 B | Vulkan     |  99 |  1 |        tg8192 |         74.87 ± 0.00 |
| phi3 3B Q4_K - Medium          |   2.23 GiB |     3.82 B | Vulkan     |  99 |  1 |        tg8192 |         77.90 ± 0.00 |

(This qwen model seems to be broken at TOT, even with the cuda backend, but the speedup is probably realistic)

The grouped query attention optmization doesn't require a power of two ratio,
the only thing relying on it was the modulo operation written as bitwise &.

split_k need not depend on gqa_ratio - enable it any time there's only one
workgroup in the X dimension. The shader gets the split index from the x coord,
and multiple workgroups in the X dimension (pre-split) indicates a larger
FA operation that wouldn't need splitting.
@jeffbolznv jeffbolznv requested a review from 0cc4m April 13, 2025 16:22
@github-actions github-actions bot added testing Everything test related Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Apr 13, 2025
@0cc4m 0cc4m merged commit 015022b into ggml-org:master Apr 16, 2025
51 checks passed
colout pushed a commit to colout/llama.cpp that referenced this pull request Apr 21, 2025
…gml-org#12931)

The grouped query attention optmization doesn't require a power of two ratio,
the only thing relying on it was the modulo operation written as bitwise &.

split_k need not depend on gqa_ratio - enable it any time there's only one
workgroup in the X dimension. The shader gets the split index from the x coord,
and multiple workgroups in the X dimension (pre-split) indicates a larger
FA operation that wouldn't need splitting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ggml changes relating to the ggml tensor library for machine learning testing Everything test related Vulkan Issues specific to the Vulkan backend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants