Phi-1 model support

[aseok]

phi-1 attains
pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP, surpassing gpt-3.5 hence all other models on coding .
Some details of model architecture: decoder only transformer model using the FlashAttention implementation of multi-
head attention (MHA). also uses MHA and MLP layers in parallel configuration following
some recent models like CodeGen, PaLM and GPT-NeoX. The archi-tecture for 1.3B parameter phi-1 model consists of 24 layers, hidden dimension of 2048, MLP-inner
dimension of 8192, and 32 attention heads of dimension 64 each.
The smaller 350M parameter phi-
1-small model consists of 20 layers, hidden dimension of 1024, MLP-inner dimension of 4096, and 16
attention heads of dimension 64 each. also uses a rotary position embedding with rotary
dimension 32.
Achieving these claimed results despite being only 1.3b parameters seems so promising. supporting it will make running ggml on huge number of mid low memory smartphones and laptops possible.
Not clear whether model or dataset will be publicly available or not, also should mention that novel approaches for curating dataset seems having major impact of scoring such a results.
Reference:
https://arxiv.org/abs/2306.11644