LongLLaMA: Focused Transformer: Contrastive Training for Context Scaling

[daboe01]

https://www.marktechpost.com/2023/07/10/meet-longllama-a-large-language-model-capable-of-handling-long-contexts-of-256k-tokens/?amp
https://arxiv.org/abs/2307.03170
https://github.com/cstankonrad/long_llama

[ghost]

Hi,

It appears models with Focused Transformer(FoT) improve perplexity as context increases up to 64k, is that right?

[FNsi]

4#