feat: concat the adapter id to the model id in chat response #2779
Conversation
router/src/server.rs
Outdated
| // extract model id from request if specified | ||
| let model_id = match model.as_deref() { | ||
| Some("tgi") | None => info.model_id.clone(), | ||
| Some(m_id) => format!("{}+{}", info.model_id, m_id), |
There was a problem hiding this comment.
IMO it should only be google-cloud-partnership/gemma-2-2b-it-lora-jap-en
There was a problem hiding this comment.
@philschmid that case is much more concise, however it leaves out the current model id which could obscure information from the caller. Since the generation is a combination of the model and the adapter it seems to reason to add both ids.
I'm happy to change the id to just the adapter if that makes the most sense, however for the reasons above I think its best to include both
There was a problem hiding this comment.
But its not what openai or vllm does. They return the adapter or the "real" model. google-cloud-partnership/gemma-2-2b-it-lora-jap-en defines the base model in the config.json so i would only go with the adapter id
I would try to stay in sync with what others are doing.
There was a problem hiding this comment.
ah thats a great reason to only use the adapter id, just updated in the latest commit
|
merging as the failing CI is unrelated - and issues have been resolved/approved before the string change |
This PR concatenates the adapter id to the model id when an adapter id is specified in the chat request.
the follwing request
curl http: //localhost:3000/v1/chat/completions \ -X POST \ -H 'Content-Type: application/json' \ -d '{ "model": "google-cloud-partnership/gemma-2-2b-it-lora-jap-en", "messages": [ { "content": "たくさんお金を稼ぎましたか？ Translate: ", "role": "user" } ], "max_tokens": 10, "stream": false }' | jqcurrently returns
{ "object": "chat.completion", "id": "", "created": 1732545871, "model": "google/gemma-2-2b-it", "system_fingerprint": "2.4.2-dev0-native", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Have you made a fortune?" }, "logprobs": null, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 19, "completion_tokens": 7, "total_tokens": 26 } }and with the changes returns an updated model id
{ "object": "chat.completion", "id": "", "created": 1732545871, "model": "google-cloud-partnership/gemma-2-2b-it-lora-jap-en", "system_fingerprint": "2.4.2-dev0-native", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Have you made a fortune?" }, "logprobs": null, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 19, "completion_tokens": 7, "total_tokens": 26 } }