derive tokenizer type from model configuration #409
Conversation
facebook-github-bot
added
the
CLA Signed
| ): | ||
| if model is None: | ||
| ) -> None: | ||
| if not (self.is_sentencepiece or self.tiktoken): |
There was a problem hiding this comment.
Should self.tiktoken -> self.is_tiktoken? (from definition on line 143)
There was a problem hiding this comment.
Yeah but I’m not liking it anyway. I’ll try another way. This one just turns what we have into one horrible mess because it combines everything with everything. That being said, config _only is here to stay
There was a problem hiding this comment.
So if you’re going to get called with that, you can just return an empty model without loading anything other than configs.
can we get trustworthy information whether a model wants tiktoken or sentence from GGUF file?
There was a problem hiding this comment.
It looks like we may be able to extract the name of the tokenizer model: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md#tokenizer
|
dead end for now because it makes the code too messy |
derive tokenizer type from model configuration
introduce config_only option to load_model