llama : sanitize tokens in the upper bound

[slaren]

cont #9357

[julmb]

I don't know if it is within the scope of this PR, but it might also be worth looking at this:
https://github.com/ggerganov/llama.cpp/blob/a5b5d9a1014248f385939e6739e9db9de7147e55/src/llama-vocab.cpp#L1700-L1712

It seems that the cache lookup is not protected by the bounds check. At least I am getting segmentation faults when passing negative or too large tokens to llama_token_to_piece.

[julmb]

Actually, I think it might be llama_token_to_piece_impl calling llama_token_get_attr_impl (also not protected by bounds checking) that is causing the segfault, and not the cache lookup:

llama_token_attr llama_token_get_attr_impl(const struct llama_vocab & vocab, llama_token token) {
    GGML_ASSERT(vocab.type != LLAMA_VOCAB_TYPE_NONE);
    return vocab.id_to_token[token].attr;
}

The cache lookup uses std::vector<T,Allocator>::at, which does bounds checking (although I am not sure what happens to C++ exceptions when the function is called from C).

[slaren]

I think it would be good to add bounds checking to more functions. The positions in embedding models also need to be bounds checked, currently it will still crash if the sequence is longer than then model supports. However let's do that in separate PRs.