Qualcomm AI Engine Direct - Optimize static llama phase 2

[shewu-quic]

• Modify custom annotation for kv in llama
• Remove unneccesary reshape op
• Refactor requantize mechanism
  • Support partial of ops to insert to_copy
  • For the same quant_attr share the same to_copy op
• Remove qdq for I/O
• Refine calibration and runner in prefill mode
• (new) Support graph sharding in hybrid mode

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7466

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job

As of commit 9def8ff with merge base 7a2dc47 ():

CANCELLED JOB - The following job was cancelled. Please retry:

• Check Labels (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Hey can you help rebase?

[cccclai]

LGTM and thank you for the refactor!

[shewu-quic]

Hey can you help rebase?

I have rebased.

I add graph sharding support in hybrid mode. This has the most significant impact on performance. When HTP execution time exceeds RPC polling time (10 ms in burst mode), CPU frequency may drop to its minimum. Our experiments indicate that setting num_sharding to 4 yields the best results. However, this depends on the device. Aim to keep the execution time for each partition under 10ms ( RPC polling time).

executorch/backends/qualcomm/runtime/backends/htpbackend/HtpDevice.cpp

Line 262 in 6c9b9b6

rpc_polling_time.rpcPollingTimeConfig = kRpcPollingTimeHighPower;

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.