Skip to content

Commit 37ae388

Browse files
larryliu0820facebook-github-bot
authored andcommitted
Buckify tokenizers (#8408)
Summary: Pull Request resolved: #8408 X-link: pytorch-labs/tokenizers#17 So that it can be used by ET internally Reviewed By: jackzhxng Differential Revision: D69509028
1 parent 8ad15f3 commit 37ae388

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

shim/xplat/executorch/build/env_interface.bzl

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,12 @@ _EXTERNAL_DEPS = {
3939
"gtest_aten": "//third-party:gtest_aten",
4040
"libtorch": "//third-party:libtorch",
4141
"libtorch_python": "//third-party:libtorch_python",
42+
# Huggingface Tokenizer
43+
"nlohmann_json": [], # Intentionally not supporting OSS buck build HF tokenizer.
4244
"prettytable": "//third-party:prettytable",
4345
"pybind11": "//third-party:pybind11",
4446
"re2": "//extension/llm/third-party:re2",
47+
"sentencepiece": [], # Intentionally not supporting OSS buck build of sentencepiece.
4548
"sentencepiece-py": [],
4649
# Core C++ PyTorch functionality like Tensor and ScalarType.
4750
"torch-core-cpp": "//third-party:libtorch",

0 commit comments

Comments
 (0)