Skip to content

Vulkan Mixture of Experts (MoE) support #7628

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jun 3, 2024
Merged

Vulkan Mixture of Experts (MoE) support #7628

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
579f059
Finish Vulkan mul_mat_id implementation
0cc4m May 26, 2024
b4abdbb
Add Vulkan sum_rows and div ops
0cc4m May 26, 2024
45928e8
Fix MUL_MAT_ID matrix matrix shader
0cc4m May 29, 2024
8ecdda1
Merge remote-tracking branch 'origin/master' into 0cc4m/vulkan-moe
0cc4m May 29, 2024
c8f9377
Fix MUL_MAT_ID matrix vector shader dispatch size
0cc4m May 29, 2024
2c3d0b4
Fix MUL_MAT_ID matrix vector shader and dispatch code
0cc4m Jun 1, 2024
6e0e0be
Update Vulkan CPU offload for MUL_MAT_ID
0cc4m Jun 2, 2024
fe3f695
Fix crash when using split mode none and setting a main GPU
0cc4m Jun 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions common/common.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1002,9 +1002,9 @@ bool gpt_params_find_arg(int argc, char ** argv, const std::string & arg, gpt_pa
return true;
}
params.main_gpu = std::stoi(argv[i]);
#ifndef GGML_USE_CUDA_SYCL
fprintf(stderr, "warning: llama.cpp was compiled without CUDA/SYCL. Setting the main GPU has no effect.\n");
#endif // GGML_USE_CUDA_SYCL
#ifndef GGML_USE_CUDA_SYCL_VULKAN
fprintf(stderr, "warning: llama.cpp was compiled without CUDA/SYCL/Vulkan. Setting the main GPU has no effect.\n");
#endif // GGML_USE_CUDA_SYCL_VULKAN
return true;
}
if (arg == "--split-mode" || arg == "-sm") {
Expand All @@ -1030,9 +1030,9 @@ bool gpt_params_find_arg(int argc, char ** argv, const std::string & arg, gpt_pa
invalid_param = true;
return true;
}
#ifndef GGML_USE_CUDA_SYCL
fprintf(stderr, "warning: llama.cpp was compiled without CUDA/SYCL. Setting the split mode has no effect.\n");
#endif // GGML_USE_CUDA_SYCL
#ifndef GGML_USE_CUDA_SYCL_VULKAN
fprintf(stderr, "warning: llama.cpp was compiled without CUDA/SYCL/Vulkan. Setting the split mode has no effect.\n");
#endif // GGML_USE_CUDA_SYCL_VULKAN
return true;
}
if (arg == "--tensor-split" || arg == "-ts") {
Expand Down
Loading
Loading