reset glmedge chat template

[piDack]

Make sure to read the contributing guidelines before submitting a PR

Hi, @ngxson,I’d like to know why the glmedge model needs to be added [gMASK]. I tried using the code below, and through testing, I found that the edge model does not exist. I think we shouldn’t add it, right?

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_PATH = "<path>"

tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = AutoModelForCausalLM.from_pretrained(MODEL_PATH, device_map="auto")

message = [{"role": "user", "content": "hello!"}]

inputs = tokenizer.apply_chat_template(
    message,
    return_tensors="pt",
    add_generation_prompt=True,
    return_dict=True,
).to(model.device)
print(tokenizer.decode(inputs["input_ids"][0],skip_special_tokens=False))

the output is

Also, what are the poor performance issues that arise from this?

[ngxson]

There is a test in llava/test.sh and without adding gmask, the model never response.

I don't know why but tbh the chat template of glmedge is quite confusing.

Did you tested it (glmedge gguf model with vision input) without gmask?

[piDack]

There is a test in llava/test.sh and without adding gmask, the model never response.

I don't know why but tbh the chat template of glmedge is quite confusing.

Did you tested it (glmedge gguf model with vision input) without gmask?

I feel it’s because of the changes made to BOI and EOI in your other PR #13081. Because in all GLM multimodal models, BOI and EOI are written in the weights rather than using <|begin_of_image|> and <|end_of_image|> tokenid directly https://huggingface.co/THUDM/glm-edge-v-2b/blob/main/siglip.py#L53. This is quite strange, but they indeed do it this way.

[ngxson]

Hmm ok I see, I didn't know about this python code. This is quite messy because all other models use boi/eoi as text token.

I'll bring back the boi/eoi embeddings in a follow up PR

[ngxson]

Btw, I'm not sure wht CI failed, let's try to make it green before merging

[piDack]

Hmm ok I see, I didn't know about this python code. This is quite messy because all other models use boi/eoi as text token.

Yes, I feel the confuse too.

[piDack]

Btw, I'm not sure wht CI failed, let's try to make it green before merging

Ok