Skip to content

Commit 5aefbce

Browse files
authored
convert : remove fsep token from GPTRefactForCausalLM (#8237)
The <filename> token used by Refact doesn't serve the same purpose as the <file_separator> from CodeGemma. Signed-off-by: Jiri Podivin <[email protected]>
1 parent 71c1121 commit 5aefbce

File tree

1 file changed

+1
-2
lines changed

1 file changed

+1
-2
lines changed

convert_hf_to_gguf.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1203,11 +1203,10 @@ def set_vocab(self):
12031203

12041204
# TODO: how to determine special FIM tokens automatically?
12051205
special_vocab = gguf.SpecialVocab(self.dir_model, load_merges=False,
1206-
special_token_types = ['prefix', 'suffix', 'middle', 'fsep', 'eot'])
1206+
special_token_types = ['prefix', 'suffix', 'middle', 'eot'])
12071207
special_vocab._set_special_token("prefix", 1)
12081208
special_vocab._set_special_token("suffix", 3)
12091209
special_vocab._set_special_token("middle", 2)
1210-
special_vocab._set_special_token("fsep", 4) # is this correct?
12111210
special_vocab.add_to_gguf(self.gguf_writer)
12121211

12131212
def set_gguf_parameters(self):

0 commit comments

Comments
 (0)