Skip to content

Commit 4694c5a

Browse files
cccclaifacebook-github-bot
authored andcommitted
Add bpe tokenizer
Summary: test it with D66895016 Differential Revision: D67406108
1 parent a63c375 commit 4694c5a

File tree

1 file changed

+1
-0
lines changed

1 file changed

+1
-0
lines changed

examples/qualcomm/oss_scripts/llama3_2/targets.bzl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ def define_common_targets():
2525
"//executorch/extension/module:module",
2626
"//executorch/extension/llm/sampler:sampler",
2727
"//executorch/examples/models/llama/tokenizer:tiktoken",
28+
"//executorch/extension/llm/tokenizer:bpe_tokenizer",
2829
"//executorch/extension/evalue_util:print_evalue",
2930
"//executorch/backends/qualcomm/runtime:runtime",
3031
],

0 commit comments

Comments
 (0)