problem about using this on a website

[okgor81]

If I successfully created a website using this to power an AI chatbot, can other people on other computer use the ai on the website? They say this AI is installed "locally" and I am not sure what that really means.

[ghost]

Hi, to clarify, are you talking about using ./server?

[okgor81]

yeah

[ghost]

It's possible to make ./server available for others with the --host parameter.

By default 127.0.0.1 is your particular pc/device. If someone else types that into their web browser then it'll go to their local host, and not yours.

Others cannot normally connect to the local host on your pc/device.

[FSSRepo]

Currently, llama.cpp is designed to handle only one client at a time. At least the server example is not designed to handle multiple requests simultaneously.

[dharrisonweb]

Would it be a lot of work to allow it to handle multiple requests? I've never used C++ so I genuinely have no idea where to start.

[SlyEcho]

It would be not be super difficult, however there needs to be some consideration into how the resources are shared. The API caches the last result so that means every client would have to have some kind of session token or cookie to know which cache to restore. The cached data is a lot of memory (RAM or VRAM, depending on how it's loaded) that would be quickly eaten by a few additional users. Without caching the only option is to reevaluate the fulll input most of the time, which is a lot slower unless using a fast GPU.