Running two different models on one GPU

[daytonturner]

I have an A6000 48GB, and I'd like to be able to serve a quantized Llama-2 and WizardCoder, both of which can easily fit inside the 48GB available, but I'm unsure the best way to go about this - or if its a bad idea for some reason?

Initially, I thought simply running two TGI instances, each pointing to the respective model would be a reasonable approach, but I'm wondering if my assumptions are correct? Any thoughts?

[Narsil]

This is the correct way to go about it. Use --cuda-memory-fraction to ensure deployments don't overlap each other (more free VRAM usually means more users on the same compute therefore tgi by default uses all the available VRAM)

[flozi00]

Attention that it always calculates from the free memory
So using the same command twice will result in using only 0.75% of GPU Vram

First command needs 0.5 memory and second 0.9 or 1