Skip to content

fix run chat #482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 25, 2024
Merged

fix run chat #482

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 23 additions & 20 deletions runner/run.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -512,6 +512,7 @@ void chat(
int prev_token;
int pos = 0; // position in the sequence
while (pos < steps) {

// when it is the user's turn to contribute tokens to the dialog...
if (user_turn) {
// get the (optional) system prompt at position 0
Expand All @@ -538,19 +539,21 @@ void chat(
}
// render user/system prompts into the Llama 2 Chat schema
if (pos == 0 && system_prompt[0] != '\0') {
const char system_template[] = "<s>[INST] <<SYS>>\n%s\n<</SYS>>\n\n%s [/INST]";
// We do not add <s> because that is added by tokenizer->encode(x, 1, 0)
const char system_template[] = "[INST] <<SYS>>\n%s\n<</SYS>>\n\n%s [/INST]";
snprintf(
rendered_prompt, RENDERED_PROMPT_SIZE-1, system_template, system_prompt, user_prompt);
} else {
// Assistant should produce </s>, so we do not include it in template
// "</s><s>[INST] %s [/INST]" for subsequent user inputs.
const char user_template[] = "<s>[INST] %s [/INST]";
// We do not add <s> because that is added by tokenizer->encode(x, 1, 0)
const char user_template[] = "[INST] %s [/INST]";
snprintf(rendered_prompt, RENDERED_PROMPT_SIZE-1, user_template, user_prompt);
}

// encode the rendered prompt into tokens
prompt_tokens = tokenizer->encode(rendered_prompt, 1, 0);
num_prompt_tokens = prompt_tokens.size();

user_idx = 0; // reset the user index
user_turn = 0;
printf("Assistant: ");
Expand All @@ -566,27 +569,27 @@ void chat(
token = next;
}

// forward the transformer to get logits for the next token
float* logits = forward(transformer, token, pos);
next = sample(sampler, logits);


if (token == EOS_TOKEN) {
user_turn = 1;
pos++;
} else {
// forward the transformer to get logits for the next token
float* logits = forward(transformer, token, pos);
next = sample(sampler, logits);
pos++;

if (user_idx >= num_prompt_tokens && next != EOS_TOKEN && next != SOS_TOKEN) {
// the Assistant is responding, so print its output
std::string piece = tokenizer->decode(token, next);
safe_printf(piece.c_str()); // same as printf("%s", piece), but skips
// "unsafe" bytes
fflush(stdout);
}
if (next == EOS_TOKEN) {
printf("\n");
}
}

if (user_idx >= num_prompt_tokens && token != EOS_TOKEN && next != EOS_TOKEN) {
std::string piece = tokenizer->decode(token, next);
safe_printf(piece.c_str()); // same as printf("%s", piece), but skips
// "unsafe" bytes
fflush(stdout);
}

if (next == EOS_TOKEN) {
printf("\n");
}
pos++;

}
printf("\n");
}
Expand Down