What backend to use with AWS EC2 g4dn.2xlarge? #706

Unanswered

jamesbraza asked this question in Q&A

jamesbraza
Sep 13, 2023

I have an AWS EC2 g4dn.2xlarge instance running Ubuntu 22.04.2, with an NVIDIA Tesla T4 GPU and CUDA 11.5. I use Python 3.11.5.

https://github.com/abetlen/llama-cpp-python#installation-with-hardware-acceleration

Describes choosing OpenBLAS, cuBLAS, CLBlast, etc backends for faster processing. My question is:

How does one pick a backend? Are there docs on making a decision somewhere?
What would you recommend for my use case?

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment