vulkan: initial support for IQ1_S and IQ1_M quantizations

[remyoudompheng]

This pull request implements basic support for the remaining I-quants (IQ1_S and IQ1_M).
Performance is not great but similar to IQ2 quantizations.

To avoid spamming shared memory, the IQ1S grid has been compressed to 2 bits per value (4kB shmem size).

ggml_vulkan: 0 = AMD Radeon 780M (RADV GFX1103_R1) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        248.08 ± 0.97 |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         18.49 ± 0.22 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        233.05 ± 2.46 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         16.55 ± 0.17 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        234.95 ± 0.70 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         23.99 ± 0.04 |
| qwen2 7B Q4_K - Medium         |   4.36 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        182.14 ± 2.80 |
| qwen2 7B Q4_K - Medium         |   4.36 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         16.45 ± 0.19 |

Pull request is draft waiting for #11501 and #11502 to be merged

[jeffbolznv]

Works with coopmat2 enabled! Perf is a bit low, but I'll fix it after it's merged.

[remyoudompheng]

I added MMV kernels for the new quants

Some performance figures on Radeon 780M (~70GB/s memory bandwidth). The LLVM/AMDGPU compiler does not like the generic code at all and behaves better with the specialized shader.

Before MMV kernels:

test-backend-ops:
  MUL_MAT(type_a=iq1_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                  3408 runs -   340.90 us/run - 117.44 MFLOP/run - 344.50 GFLOPS
  MUL_MAT(type_a=iq1_m,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                  2556 runs -   407.32 us/run - 117.44 MFLOP/run - 288.32 GFLOPS

legraphista/Qwen2.5-Coder-7B-Instruct-IMat-GGUF

ggml_vulkan: 0 = AMD Radeon 780M (RADV GFX1103_R1) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        248.08 ± 0.97 |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         18.49 ± 0.22 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        233.05 ± 2.46 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         16.55 ± 0.17 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        234.95 ± 0.70 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         23.99 ± 0.04 |
| qwen2 7B Q4_K - Medium         |   4.36 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        182.14 ± 2.80 |
| qwen2 7B Q4_K - Medium         |   4.36 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         16.45 ± 0.19 |

ggml_vulkan: 0 = AMD Radeon 780M (RADV GFX1103_R1 (LLVM 19.1.7)) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: none
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        133.94 ± 1.54 |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         12.71 ± 0.05 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        121.18 ± 0.45 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         10.94 ± 0.14 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        115.58 ± 1.74 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         18.11 ± 0.16 |

After:

test-backend-ops:
  MUL_MAT(type_a=iq1_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                  5964 runs -   192.37 us/run - 117.44 MFLOP/run - 610.48 GFLOPS
  MUL_MAT(type_a=iq1_m,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                  5112 runs -   210.85 us/run - 117.44 MFLOP/run - 556.98 GFLOPS

legraphista/Qwen2.5-Coder-7B-Instruct-IMat-GGUF

ggml_vulkan: 0 = AMD Radeon 780M (RADV GFX1103_R1) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        243.66 ± 2.14 |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         22.97 ± 0.12 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        225.84 ± 1.83 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         21.47 ± 0.22 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        204.98 ± 0.45 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         22.57 ± 0.33 |

ggml_vulkan: 0 = AMD Radeon 780M (RADV GFX1103_R1 (LLVM 19.1.7)) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: none
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        133.59 ± 1.04 |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         20.61 ± 0.07 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        121.55 ± 0.34 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         17.40 ± 0.06 |

[remyoudompheng]

See branch https://github.com/remyoudompheng/llama.cpp/tree/vulkan-iq-mmv for MMV kernels for IQ2 and IQ3 quants

[jeffbolznv]

See branch https://github.com/remyoudompheng/llama.cpp/tree/vulkan-iq-mmv for MMV kernels for IQ2 and IQ3 quants

Very cool! I tested this and it's functionally correct and perf is better on RTX 4070.

I dug into the perf a bit and realized that a significant amount of time is spent in init_iq_shmem since the LUT is so large. I think I had suggested this before, but this more unrollable loop code helps:

    [[unroll]] for (uint i = 0; i < iq2s_grid.length(); i += wgsize.x) {
        iq2s_grid[i + gl_LocalInvocationIndex.x] = iq2s_grid_const[i + gl_LocalInvocationIndex.x];
    }

Even then, it's still expensive and that suggests we should be doing more work per workgroup to amortize the cost. The large shared memory allocation may also limit the number of workgroups that can run concurrently, which argues for using larger workgroups. I verified that increasing NUM_ROWS and workgroup size helps:

before
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 11076 runs -    95.35 us/run - 117.44 MFLOP/run -   1.23 TFLOPS
tot iq-mmv branch
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 12780 runs -    80.79 us/run - 117.44 MFLOP/run -   1.45 TFLOPS
unroll iq_shmem
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 14484 runs -    73.06 us/run - 117.44 MFLOP/run -   1.61 TFLOPS
rm_kq=4
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 17040 runs -    61.34 us/run - 117.44 MFLOP/run -   1.91 TFLOPS
rm_kq=4 and wgsize=64
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 17892 runs -    56.95 us/run - 117.44 MFLOP/run -   2.06 TFLOPS
rm_kq=8 and wgsize=64
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 19596 runs -    51.98 us/run - 117.44 MFLOP/run -   2.26 TFLOPS

You don't need to do all of this at once. I think the unrollable loops is a simple change and should help everywhere. For figuring out the best values for all the knobs we'll need to get more exhaustive data from different HW and also test with real models.

[remyoudompheng]

Rebased and added shmem sizes following #11502

[remyoudompheng]

Rebased and applied suggestions

[netrunnereve]

I don't have time to review this yet but all the tests are passing on my GCN cards.

[0cc4m]

LGTM