Skip to content

Commit dcd8dfa

Browse files
committed
convert : use a string for the SentencePiece tokenizer path
1 parent 3870164 commit dcd8dfa

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

convert.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -463,7 +463,7 @@ def __init__(self, base_path: Path):
463463
raise FileNotFoundError('Cannot find tokenizer.model')
464464

465465
self.sentencepiece_tokenizer = SentencePieceProcessor()
466-
self.sentencepiece_tokenizer.LoadFromFile(fname_tokenizer)
466+
self.sentencepiece_tokenizer.LoadFromFile(str(fname_tokenizer))
467467
vocab_size = self.sentencepiece_tokenizer.vocab_size()
468468

469469
new_tokens = {id: piece for piece, id in added_tokens.items() if id >= vocab_size}

0 commit comments

Comments
 (0)