Skip to content

Commit b81f83e

Browse files
committed
Remove unused/temporary code
1 parent a0ba217 commit b81f83e

File tree

16 files changed

+1
-1411
lines changed

16 files changed

+1
-1411
lines changed

docs/native-execution.md

Lines changed: 1 addition & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -153,21 +153,6 @@ Torchchat includes both Python and C/C++ implementations of both the
153153
SentencePiece and Tiktoken tokenizers that may be used with the Python
154154
and native execution environments, respectively.
155155

156-
The SentencePiece tokenizer implementations for Python (developed by
157-
Google) and and one of the two the C/C++ implementation (developed by
158-
Andrej Karpathy) use different input formats. The Python
159-
implementation reads a tokenizer specification in `tokenizer.model`
160-
format. The C/C++ tokenizer that reads the tokenizer instructions
161-
from a file in `tokenizer.bin` format.
162-
163-
We include Andrej's SentencePiece converter which translates a
164-
SentencePiece tokenizer in `tokenizer.model` format to `tokenizer.bin`
165-
in the unsupported/llama2.c subdirectory:
166-
167-
```
168-
python3 unsupported/llama2.c//tokenizer.py --tokenizer-model=${MODEL_DIR}/tokenizer.model
169-
```
170-
171156
## Appendix: Native model verification using the Python environment
172157

173158
After exporting a model, you will want to verify that the model
@@ -191,4 +176,4 @@ the exported model works correctly, as a debugging aid if you are
191176
developing your own native execution environment based on the llama
192177
runner provided with torchchat.
193178

194-
[end default]: end
179+
[end default]: end

parking_lot/MISC.md

Lines changed: 0 additions & 18 deletions
This file was deleted.

parking_lot/macos-12-x86.yml

Lines changed: 0 additions & 75 deletions
This file was deleted.

parking_lot/macos12.yml

Lines changed: 0 additions & 79 deletions
This file was deleted.

parking_lot/runner_et.yml

Lines changed: 0 additions & 65 deletions
This file was deleted.

0 commit comments

Comments
 (0)