fix(deps): Add tokenizers to the deps explicitly

It was implicitly being pulled in via lm_eval -> transformers, but it's
better to have it explicit since we use it directly

Branch: GraniteCodeSupport

Signed-off-by: Gabe Goodhart <[email protected]>

Author: gabe-l-hart

install/requirements.txt

@@ -9,6 +9,9 @@ gguf
 # Tiktoken tokenizer for Llama 3 and other advanced models
 tiktoken
 
+# Tokenizers for other non-llama models that use HF tokenizers
+tokenizers
+
 # Miscellaneous
 snakeviz
 sentencepiece