Skip to content

Commit 39f852f

Browse files
authored
py : add special tokens in hf_converter for RWKV v6 (#9428)
Signed-off-by: Molly Sophia <[email protected]>
1 parent 2b00fa7 commit 39f852f

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

convert_hf_to_gguf.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2771,6 +2771,8 @@ def set_vocab(self):
27712771
self.gguf_writer.add_tokenizer_model("rwkv")
27722772
self.gguf_writer.add_token_list(tokens)
27732773
self.gguf_writer.add_token_types(toktypes)
2774+
special_vocab = gguf.SpecialVocab(self.dir_model, load_merges=False)
2775+
special_vocab.add_to_gguf(self.gguf_writer)
27742776

27752777
def set_gguf_parameters(self):
27762778
block_count = self.hparams["num_hidden_layers"]

0 commit comments

Comments
 (0)