How do I use the API?

[vicfic18]

I saw here that a new C-style API is available. How do I use it?
I'm sorry if I sound naive 😅, I just want to test this thing out!

[ggerganov]

Check the examples

[hjubb]

It would be great as someone less familiar with C++ or the overall flow of the model to get a bit more info on the functions inside main.cpp including llama_eval and llama_sample_top_p_top_k and the parameters passed into them actually do. Following the flow in the main method is quite difficult as there are a lot of conditionals for the infinite generation and interactive / instructive modes. I can understand most of it and have tried to replicate it in the most minimal way however the results I'm getting are functionally garbage, which I'm not ruling out as possibly just from the environment and setup I'm running in.

If there was a way to get a single example for something as simple as input prompt = some string, max to generate = numerical value and just spitting out whatever with the other default parameters top_p etc. that would be really helpful. Even if there was a specific commit a while ago where the main.cpp existed more or less in that way it would be a great starting point to better understand how to set it up to consume from the API

[ggerganov]

Yes, I think having a minimal example will be very valuable.
It should be the most minimal and straightforward way to generate tokens from a prompt.
I think it could even have all parameters hardcoded and straight up generate a few tokens and exit.
No colors, no interaction, etc.

Feel free to open an issue or PR

[psugihara]

Note for anyone who finds this. Looks like minimal was implemented as simple here: https://github.com/ggerganov/llama.cpp/blob/master/examples/simple/simple.cpp

While I still agree it would be nice to have a small set of higher level API's, this is a fantastic resource (and so is llama.cpp in general @ggerganov!).

FWIW, my use-case is implementing an in-process version of server.cpp in swift.

[ianscrivener]

@psugihara,
the python library llama-cpp-python accesses the llama.cpp C API - perhaps there are some clues there for you - https://github.com/abetlen/llama-cpp-python/blob/231123ee1e718bcccb022ade87f802e5b0466228/llama_cpp/llama_cpp.py