Skip to content

Commit bc497a0

Browse files
authored
Buckify tokenizers
Differential Revision: D69509028 Pull Request resolved: #8408
1 parent 363d138 commit bc497a0

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

shim/xplat/executorch/build/env_interface.bzl

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,12 @@ _EXTERNAL_DEPS = {
3939
"gtest_aten": "//third-party:gtest_aten",
4040
"libtorch": "//third-party:libtorch",
4141
"libtorch_python": "//third-party:libtorch_python",
42+
# Huggingface Tokenizer
43+
"nlohmann_json": [], # Intentionally not supporting OSS buck build HF tokenizer.
4244
"prettytable": "//third-party:prettytable",
4345
"pybind11": "//third-party:pybind11",
4446
"re2": "//extension/llm/third-party:re2",
47+
"sentencepiece": [], # Intentionally not supporting OSS buck build of sentencepiece.
4548
"sentencepiece-py": [],
4649
# Core C++ PyTorch functionality like Tensor and ScalarType.
4750
"torch-core-cpp": "//third-party:libtorch",

0 commit comments

Comments
 (0)