Qualcomm AI Engine Direct - support llama3.2 1B/3B with static llama in kv mode

[shewu-quic]

summary:

• Add custom_annotate_llama_last_conv_16a8w
• Add llama.py and runner for llama3.2 1B/3B
• Support model sharding and sha

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6779

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit f2a0383 with merge base dc41596 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[cccclai]

LGTM! Can you share the command line for both AOT and runtime?

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[shewu-quic]

LGTM! Can you share the command line for both AOT and runtime?

Sure,

End-to-end flow for 16a4w llama 3.2 1B/3B instruct

python examples/qualcomm/oss_scripts/llama3_2/llama.py -a ./llama3_2_1B_512 -b build-android -H ${ADB_HOST} -s ${ADB_SERIAL_NUM} -m "SM8650" --checkpoint ${checkpoint} --params ${params} --tokenizer_model ${tokenizer.model}  --prompt "What is 1+1?" --temperature 0 --seq_len 512  --model_size ${1B/3B} --ptq 16a4w

Compile only for 16a4w llama 3.2 1B/3B instruct

python examples/qualcomm/oss_scripts/llama3_2/llama.py -a ./llama3_2_1B_512 -b build-android -H ${ADB_HOST} -s ${ADB_SERIAL_NUM} -m "SM8650" --checkpoint ${checkpoint} --params ${params} --tokenizer_model ${tokenizer.model}  --prompt "What is 1+1?" --temperature 0 --seq_len 512 --model_size ${1B/3B}--compile_only --ptq 16a4w

Inference only for 16a4w llama 3.2 1B/3B instruct

python examples/qualcomm/oss_scripts/llama3_2/llama.py -a ./llama3_2_1B_512 -b build-android -H ${ADB_HOST} -s ${ADB_SERIAL_NUM} -m "SM8650" --checkpoint ${checkpoint} --params ${params} --tokenizer_model ${tokenizer.model}  --prompt "What is 1+1?" --temperature 0 --seq_len 512 --pre_gen_pte ./llama3_2_1B_512 --model_size ${1B/3B} --ptq 16a4w