Skip to content

Commit d0b7aec

Browse files
committed
Removing fsep token from GPTRefactForCausalLM
The <filename> token used by Refact doesn't serve the same purpose as the <file_separator> from CodeGemma. Signed-off-by: Jiri Podivin <[email protected]>
1 parent d0a7145 commit d0b7aec

File tree

1 file changed

+1
-2
lines changed

1 file changed

+1
-2
lines changed

convert-hf-to-gguf.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1151,11 +1151,10 @@ def set_vocab(self):
11511151

11521152
# TODO: how to determine special FIM tokens automatically?
11531153
special_vocab = gguf.SpecialVocab(self.dir_model, load_merges=False,
1154-
special_token_types = ['prefix', 'suffix', 'middle', 'fsep', 'eot'])
1154+
special_token_types = ['prefix', 'suffix', 'middle', 'eot'])
11551155
special_vocab._set_special_token("prefix", 1)
11561156
special_vocab._set_special_token("suffix", 3)
11571157
special_vocab._set_special_token("middle", 2)
1158-
special_vocab._set_special_token("fsep", 4) # is this correct?
11591158
special_vocab.add_to_gguf(self.gguf_writer)
11601159

11611160
def set_gguf_parameters(self):

0 commit comments

Comments
 (0)