Idea: Language translation model on top of LLM model in llama.cpp

[klosax]

The current free LLM models are very good at understanding and generating in English but not other languages. The simple reason is that most available training data is in English. This likely makes the best free LLMs rather inaccessible to the non-english speaking community.

My idea is to run a small but good enough translation model on top of any ordinary LLM. The forward and backward translations could be made seamless. To use this feature you would only need to add the translation model as a parameter. The user could possibly choose one language for input and another for output, and change in the middle of a session if needed.

facebook/nllb have 600M models that can provide translations in 200 languages. If a 4 bit model of nllb-600M works it will likely only use around 200MB of memory, which is nothing compared to the LLM part. I dont know how much work that would be needed to implement support for this model in ggml.

Since my native language is non-english - I would love to see this feature in llama.cpp !

[ianscrivener]

Here's a Python project that looks to be a good start for your project - https://github.com/thammegowda/nllb-serve

[klosax]

Thanks!