Quantize llama3 on et export

[lucylq]

• Recommended to quantize for llama models. Update examples to use llama3 rather than stories.
• Update quantization docs; weight + embedding + examples.
• Some quantization schemes do not work for et export (not intended to work); gptq, linear int4, hqq. Specify support.

Test with:

python3 torchchat.py export llama3 --quantize '{"embedding": {"bitwidth": 4, "groupsize":32}, "linear:a8w4dq": {"groupsize" : 256}}' --output-pte-path llama3_q.pte       

Run:

python3 torchchat.py generate llama3 --device cpu --pte-path llama3_8da_qe4_32.pte --prompt "Hello my name is" --tokenizer-path ../llama-models/llama3/tiktokenizer.bin

<|begin_of_text|>Hello my name is Samantha, I am a trained healthcare professional with a passion for delivering exceptional patient care. I am excited to offer my services as a caregiver through this platform. I have extensive experience in providing assistance with daily living activities, such as bathing, dressing, grooming, meal preparation, medication management, and other essential tasks. I am also trained in first aid and CPR. I am available for short-term or long-term positions and can work flexible hours depending on your needs. I am a non-smoker and a non-drinker, and I have a clean driving record. I look forward to working with you and your loved ones.
Hello my name is Samantha, I am a trained healthcare professional with a passion for delivering exceptional patient care. I am excited to offer my services as a caregiver through this platform. I have extensive experience in providing assistance with daily living activities, such as bathing, dressing, grooming, meal preparation, medication management, and other essential tasks. I am also trained in first aid and CPR.
Time for inference 1: 19.38 sec total, 10.32 tokens/sec
Bandwidth achieved: 0.00 GB/s
==========
Average tokens/sec: 10.32
Memory used: 0.00 GB

With the different export quantization schemes in https://github.com/pytorch/torchchat/blob/main/docs/quantization.md
Not tested for aoti.

[kimishpatel]

In summary can you put the repro step or how you tested it?

[iseeyuan]

Could you also upadte examples/models/llama2/README, where for llama3 models, --embedding-quantize 4,32 is suggested to reduce the model size significantly due to the larger vocabulary size.

[lucylq]

Could you also upadte examples/models/llama2/README, where for llama3 models, --embedding-quantize 4,32 is suggested to reduce the model size significantly due to the larger vocabulary size.

thanks Martin, added in pytorch/executorch#3315

[kimishpatel]

python3 torchchat.py export llama3 --quantize '{"linear:gptq": {"groupsize" : 32} }' --output-pte-path llama3_q.pte

Did you try running the generated pte?

[lucylq]

python3 torchchat.py export llama3 --quantize '{"linear:gptq": {"groupsize" : 32} }' --output-pte-path llama3_q.pte

Did you try running the generated pte?

So, the gptq, linear int4, and hqq paths don't export well through executorch. I'll try run the 8da4w+qe file - updated in summary.

[kimishpatel]

@mikekgfb I asked Lucy to make these changes. Quite a bit of stuff toward the latter half of the doc is removed. I think not having a very long doc was better since those were just showcasing example commands on how to invoke each quant scheme. If you differ in the opinion let me know.

Also, is HQQ done or WIP?

[jerryzh168]

did we add eval as well

[mikekgfb]

I'll defer to you for adding @jerryzh168

[jerryzh168]

we probably want to tag hqq with bitwidth as well

[lucylq]

hmn, where is hqq bitwidth added?

[jerryzh168]

looks good overall I think, I think the overall structure is better, we just explain multiple dimensions of config and give a few examples of showing how they can be combined

[mikekgfb]

Please make sure that whatever changes in commands, you made are tested.

Thank you!

[mikekgfb]

I'll defer to you for adding @jerryzh168

[kimishpatel]

Thanks for the changes