feat: improve temperature logic in chat

[drbh]

This PR adds support for do_sample to chat to enable greedy sampling

[Narsil]

I do not understand this. Why are you adding 1.0 ?

[drbh]

1 is added since openais api is essentially zero indexed. They treat 0 as deterministic and higher values as more random.

In our case 1 is deterministic and greater is more random, so adding 1 allows users to get deterministic output with temp 0, and more random with larger values.

temperature:
What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.

[Narsil]

No it's not.

logits.pow(temperature).

temperature == 0 -> Pretty much forces deteministic sampling (with float issues).
temperature == 1 -> Regular distribution.

[drbh]

makes sense! PR updated to:

1. only set do_sample to false when temperature is 0
2. rescale temperature from 0 to 1 only when temp is 0

I believe this fits the requirements, and enables greedy ONLY when temperature is 0. Please lmk if I should change! 🙏

[drbh]

tiny reproduction script

import requests

headers = {
    "Content-Type": "application/json",
}

data = {
    "model": "tgi",
    "messages": [
        {
            "role": "user",
            "content": "Summarize the main ideas of Jeff Walker's Product Launch Formula into bullet points as it pertains to a growth marketing agency implementing these strategies and tactics for their clients...",
        }
    ],
    "stream": False,
    "max_tokens": 100,
    "temperature": 0,
}

url = "http://localhost:3000/v1/chat/completions"

response = requests.post(url, headers=headers, json=data)

print(response.text)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Narsil]

Isn´t that.

let temperature = if temperature == 0.0{
   None
   } else{ Some(temperature) } 

[Narsil]

Better yet, make the condition == 0.0 common to both do_sample and temperature (makes it more readable imho).

[drbh]

I don't think that exactly equivalent since we need the following:

1. do_sample to be true in all cases except if the incoming temp is 0
2. we need temperature to be a Option<f32> and we need to replace 0 with 1.

most readable/simple I've come up with so far is...

let do_sample: bool = temperature.map_or(true, |t| t != 0.0);
let temperature = temperature.map(|t| if t == 0.0 { 1.0 } else { t });

whatcha think? happy to make any changes!