Skip to content

[Executorch][llama] Set # of threads to use performant cores #2352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 10 commits into from
Closed

[Executorch][llama] Set # of threads to use performant cores #2352

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d6a3897
[Executorch][llama] Set # of threads to use performant cores
kimishpatel Mar 11, 2024
38700a4
Update on "[Executorch][llama] Set # of threads to use performant cores"
kimishpatel Mar 11, 2024
b044bb6
Update on "[Executorch][llama] Set # of threads to use performant cores"
kimishpatel Mar 12, 2024
96cbb29
Update on "[Executorch][llama] Set # of threads to use performant cores"
kimishpatel Mar 14, 2024
32f7a53
Update on "[Executorch][llama] Set # of threads to use performant cores"
kimishpatel Mar 14, 2024
4a55cdc
Update on "[Executorch][llama] Set # of threads to use performant cores"
kimishpatel Mar 15, 2024
cfbe2e3
Update on "[Executorch][llama] Set # of threads to use performant cores"
kimishpatel Mar 15, 2024
991c1ef
Update on "[Executorch][llama] Set # of threads to use performant cores"
kimishpatel Mar 15, 2024
ed02e0c
Update on "[Executorch][llama] Set # of threads to use performant cores"
kimishpatel Mar 15, 2024
08ad650
Update on "[Executorch][llama] Set # of threads to use performant cores"
kimishpatel Mar 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions examples/models/llama2/main.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@

#include <executorch/examples/models/llama2/runner/runner.h>

#if defined(ET_USE_THREADPOOL)
#include <executorch/backends/xnnpack/threadpool/cpuinfo_utils.h>
#include <executorch/backends/xnnpack/threadpool/threadpool.h>
#endif

DEFINE_string(
model_path,
"llama2.pte",
Expand Down Expand Up @@ -45,6 +50,14 @@ int32_t main(int32_t argc, char** argv) {

int32_t seq_len = FLAGS_seq_len;

#if defined(ET_USE_THREADPOOL)
uint32_t num_performant_cores =
torch::executorch::cpuinfo::get_num_performant_cores();
ET_LOG(
Info, "Resetting threadpool with num threads = %d", num_performant_cores);
torch::executorch::threadpool::get_threadpool()->_unsafe_reset_threadpool(
num_performant_cores);
#endif
// create llama runner
::torch::executor::Runner runner(model_path, tokenizer_path, temperature);

Expand Down
2 changes: 2 additions & 0 deletions examples/models/llama2/targets.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ def define_common_targets():
deps = [
"//executorch/examples/models/llama2/runner:runner" + aten_suffix,
"//executorch/extension/evalue_util:print_evalue",
"//executorch/backends/xnnpack/threadpool:threadpool",
"//executorch/backends/xnnpack/threadpool:cpuinfo_utils",
],
external_deps = [
"gflags",
Expand Down