[Docs] Add cloud run example #573
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
|
||
| Some clarifications: | ||
| - We provide the latest official Docker image URI based on the [README](https://github.com/huggingface/Google-Cloud-Containers/blob/main/containers/tei/README.md). | ||
| - We choose to deploy the [IBM granite](https://huggingface.co/ibm-granite/granite-embedding-278m-multilingual) embedding model given its strong multilingual capabilities. One can of course choose any other embedding model from the hub. It's recommended to look for models tagged with either `feature-extraction`, `sentence-similarity` or `text-ranking`. |
There was a problem hiding this comment.
Do you think we could add a link to the MTEB leaderboard too ?
https://huggingface.co/models?library=sentence-transformers&sort=trending
https://huggingface.co/spaces/mteb/leaderboard
| gcloud run deploy $SERVICE_NAME \ | ||
| --image=$CONTAINER_URI \ | ||
| --args="--model-id=$MODEL_ID,--max-concurrent-requests=64" \ | ||
| --set-env-vars=HF_HUB_ENABLE_HF_TRANSFER=1 \ |
There was a problem hiding this comment.
Have you checked if this makes a difference ? I don't think TEI uses that flag at all (it downloads the models directly through Rust at load time)
There was a problem hiding this comment.
This was copied from @alvarobartt's TGI blog, might need to check
Co-authored-by: Nicolas Patry <[email protected]>
There was a problem hiding this comment.
LGTM, I'll try to reproduce it later today, but so far so good! Thanks for handling this @NielsRogge 👏🏻
Co-authored-by: Alvaro Bartolome <[email protected]>
What does this PR do?
This PR adds an example to the docs on how to deploy TEI on Google's Cloud Run.
This is largely a copy of the amazing guide on deploying TGI by @alvarobartt but adapted for TEI.
To do: