Skip to content

Commit 9eb0fca

Browse files
author
jaime-m-p
committed
Do not remove space when decoding special tokens
1 parent 44c8648 commit 9eb0fca

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

llama.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18506,7 +18506,7 @@ int32_t llama_detokenize(
1850618506
bool special) {
1850718507
// remove the leading space of the first non-control token
1850818508
static const int attr_special = LLAMA_TOKEN_ATTR_UNKNOWN | LLAMA_TOKEN_ATTR_CONTROL;
18509-
bool remove_space = model->vocab.tokenizer_add_space_prefix;
18509+
bool remove_space = !special && model->vocab.tokenizer_add_space_prefix;
1851018510
int32_t avail = text_len_max;
1851118511
int32_t total = 0;
1851218512

0 commit comments

Comments
 (0)