Qualcomm AI Engine Direct - Requantization Mechanism Implementation

[winskuo-quic]

Summary:

• Implement requantization so mix quantization ( e.g., 8bit quant + 16 bit quant) can be properly delegated to QNN.
• Reusing test_qnn_backend_view_permute_matmul unit test to check mix quantization is working as expected.
• Added etdump logic back to qnn_executor_runner that was deleted unintentionally during this PR: a531ca5#diff-f3647de74042ac9a417e2d4000a6f2db00c22c89fd028e9433d3c79ffb7d56f6
• Refactor common arguments in VIT.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/2823

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 337fc1c with merge base 81a7e88 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[winskuo-quic]

Hi @cccclai,
This PR is used to support mix quantization and fixing minor issues that's causing UT failures.
Please have a look on this PR.
Thanks

[cccclai]

Hey would you mind explaining a bit what does requantization mean and how it's different than mix precision quantization? Does it mean we'll do 8bit quantization first then 16bit (maybe skip the 8bit quantized ops)?

[winskuo-quic]

Hey would you mind explaining a bit what does requantization mean and how it's different than mix precision quantization? Does it mean we'll do 8bit quantization first then 16bit (maybe skip the 8bit quantized ops)?

Hi Chen,
Thanks for the response.
Yes.
Requantization should be the same as mix precision quantization.
There are cases where some ops have output of 8 bit, however, the input only accepts 16bit. In this case, we will requantize the model so that the input can properly accept a 16bit input from the op user. This scenario where we have different quant config is mix quantization.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Hey would you mind explaining a bit what does requantization mean and how it's different than mix precision quantization? Does it mean we'll do 8bit quantization first then 16bit (maybe skip the 8bit quantized ops)?

Hi Chen, Thanks for the response. Yes. Requantization should be the same as mix precision quantization. There are cases where some ops have output of 8 bit, however, the input only accepts 16bit. In this case, we will requantize the model so that the input can properly accept a 16bit input from the op user. This scenario where we have different quant config is mix quantization.

Thank you for the explanation! Is there an example somewhere to show how to do it, maybe a test case?

[winskuo-quic]

Hey would you mind explaining a bit what does requantization mean and how it's different than mix precision quantization? Does it mean we'll do 8bit quantization first then 16bit (maybe skip the 8bit quantized ops)?

Hi Chen, Thanks for the response. Yes. Requantization should be the same as mix precision quantization. There are cases where some ops have output of 8 bit, however, the input only accepts 16bit. In this case, we will requantize the model so that the input can properly accept a 16bit input from the op user. This scenario where we have different quant config is mix quantization.

Thank you for the explanation! Is there an example somewhere to show how to do it, maybe a test case?

Yes.
Please refer to the unit test, TestQNNQuantizedModel.test_qnn_backend_view_permute_matmul, under test_qnn_delegate.py.
https://github.com/pytorch/executorch/pull/2823/files#diff-3a76d2f6f72394bf64270a625d31c16560eb7d3b855297352aae423a27f6f59fR984
It demonstrates how mix quantization works.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Hey do you mind rebasing the change? The CI was broken on the base commit and should be fixed now.

[winskuo-quic]

Hey do you mind rebasing the change? The CI was broken on the base commit and should be fixed now.

Thanks for the friendly reminder. I have just rebased.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@cccclai merged this pull request in 61ad48d.