Skip to content

use std::random_device{}() for default random seed #6962

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 29, 2024

Conversation

dwrensha
Copy link
Contributor

Currently, the default random seed is set by time(NULL), which returns an integer number of seconds. If we call llama_sampling_init() many times within a single second, then all of the sampling contexts get the same random seed. Such behavior can come as an unpleasant surprise. I encountered it recently while attempting to generate many samples via ollama, which makes use of an adapted version of llama.cpp/examples/server/server.cpp.

We can avoid the problem by using std::random_device, as is already done in utils.hpp.

Note that on old versions of libstd++ on vxworks and mingw std::random_device was sometimes deterministic, however that was considered a bug and was fixed.

I don't believe there should be any performance issues with using std::random_device{}(). On my Linux machine, it does not even require a system call, I think because it's using one of the special CPU instructions RDSEED or RDRAND.

@dwrensha
Copy link
Contributor Author

I originally opened #6953 proposing to use std::chrono::system_clock_now(), but on further reflection I now like this PR better. Please feel free to accept either one.

@dwrensha
Copy link
Contributor Author

Here's how I tested this PR.
In one terminal, I launched the server example:

$ ./bin/server --model ~/models/llemma_7b.Q4_0.gguf -ngl 15

and in another terminal I repeatedly made requests to the /completion endpoint:

$ for i in {1..100}; do curl -s -X POST "http://localhost:8080/completion" -H "accept: application/json" -H "Content-Type: application/json" -d '{"prompt": "The best ice cream flavor is", "stream":false, "stop": [".", "!"], "n_predict": 10}' | jq ".content"; done

With the current default random seed generator, I get lots of repeat responses, like this:

" chocolate chip"
" chocolate chip"
" chocolate chip"
" chocolate chip"
" chocolate chip"
" "
" "
" "
" "
"\n\\frac{10}{3}"
"\n\\frac{10}{3}"
" The ice cream is good The The The The"
" 4"
" 4"
" 4"
" chocolate chip if your preference is cookie flav"
" chocolate chip if your preference is cookie flav"
"\n @Vanilla Ice @Bubble g"
" the one that your customers like the most"
" the one that your customers like the most"
":\n65,8\n\n###"
" chocolate chip cookie dough ice cream"
" chocolate chip, and the best ice cre"
" chocolate chip, and the best ice cre"
" vanilla"
" vanilla"
"\n 4"
"\n 4"
" cream cheese flavor"
" cream cheese flavor"
" strawberry"
" strawberry"
" strawberry"
...

After the change in the PR, the repeats go away:

" vanilla, but the worst is buttercre"
" chocolate chip cookies and"
" vanilla ice cream"
" chocolate chip"
" peanut butter flavor ice cream"
" chocolate chip"
" chocolate chip, but you The best ice"
" 1 2 3 4 5"
" chocolate chip cookies"
" the best"
" chocolate chip, of course"
" chocolate chip"
" the best flavor for you"
"\navamegh\n\n28\n\n"
" chocolate chip cookie dough ice cream"
" chocolate chip"
" The best ice cream flavor is The best"
" vanilla"
" vanilla, but chocolate is the best"
" strawberry rancher 2 ice cream"
": The best The best flavor of ice cre"
" The The best ice cream flavor is The"
" not vanilla but strawberry ice cream"
" vanilla"
...

@ggerganov ggerganov merged commit 3f16747 into ggml-org:master Apr 29, 2024
@dwrensha dwrensha deleted the random-seed-random-device branch April 29, 2024 13:37
nopperl pushed a commit to nopperl/llama.cpp that referenced this pull request May 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants