I'm starting to implement LLaVA.cpp, but where?

[monatis]

4
Now that monatis/clip.cpp is working, ı'd like to take the next step to implement multimodal generation models with that. I'm thinking of starting with LLaVA first, and then extending to InstructBLIP.

I'd like to get the opinion of maintainors and the community on where to implement it:

1. as an example in ggml
2. as a fork of llama.cpp, e.g., llava.cpp, or even more ambitiously multimodal.cpp, thinking of the support for InstructBLIP in the future.
3. as a feature in llama.cpp.
4. Any other suggestions.

The 3rd option has the potential to reach the largest audience, but I know that it might make things unnecessarily complicated and hinder the speed of innovations here. What would the ideal path be?

[Green-Sky]

how about 3-ish?
integrating with #1910 , and replacing the .py that pr is depending on would be nice.

[monatis]

Good idea! Maybe I should write in that PR.