Skip to content

Commit d8d2f37

Browse files
Add a log message to better track the when the following line of code is triggered
1 parent b3d022a commit d8d2f37

File tree

1 file changed

+1
-0
lines changed

1 file changed

+1
-0
lines changed

convert_hf_to_gguf.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -529,6 +529,7 @@ def get_vocab_base(self) -> tuple[list[str], list[int], str]:
529529
# used for `\n` / `\t` have been manually added in the added tokens
530530
# To avoid unexpected issues - we make sure to encode single-char tokens
531531
if len(token) == 1:
532+
logger.info("Ecode-Decode special characters using AutoTokenizer")
532533
token = tokenizer.decode(tokenizer.encode(token, add_special_tokens=False))
533534

534535
if tokenizer.added_tokens_decoder[i].special or self.does_token_look_special(token):

0 commit comments

Comments
 (0)