Qualcomm AI Engine Direct - dynamic shape support #7780

haowhsu-quic · 2025-01-21T06:57:46Z

Summary

dynamic shape related change for QC backend
breakage fix
test cases

Test plan

python backends/qualcomm/tests/test_qnn_delegate.py -k TestQNNQuantizedUtils.test_qnn_backend_dynamic_shape -s $DEVICE_SERIAL -m SM8650 -b build-android/

cc @cccclai @winskuo-quic @shewu-quic

pytorch-bot · 2025-01-21T06:57:50Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7780

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

ROCM Infra failures during checkout of PyTorch

✅ No Failures

As of commit ba46b4f with merge base 62e49ce ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

haowhsu-quic · 2025-01-21T16:19:11Z

Hi @cccclai, this PR help adopt dynamism in QC backend. Qnn_Tensor_t.v1 is now replaced with Qnn_Tensor_t.v2, all the generated context binaries via QC backend will use v2. For previously generated PTEs and QNN context binary direct path will still be compatible.

Thank you.

cccclai · 2025-01-21T20:10:12Z

Thank you! This is definitely a big feature, thank you for adding it.

One question, I remember only some ops enable dynamic shape, like maybe along a specific dimension. How do you error out accordingly in AoT instead of runtime?

haowhsu-quic · 2025-01-22T01:25:01Z

Yes, there will be a following PR using QnnProperty_hasCapability to query if certain feature is available or not.

facebook-github-bot · 2025-01-22T20:59:40Z

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

cccclai · 2025-01-30T06:55:51Z

This PR also needs rebase..

facebook-github-bot · 2025-02-03T03:01:45Z

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

summary: - dynamic shape related change for QC backend - brekage fix - test cases

facebook-github-bot · 2025-02-11T02:45:58Z

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

cccclai · 2025-02-11T05:38:48Z

backends/qualcomm/aot/ir/qcir.fbs

@@ -77,6 +77,7 @@ table QuantizeParam {
 table Tensor {
    name: string;
    shape: [uint];
+    dynamic_dims: [ubyte];


As a heads up, this year we'd need to figure out the story for BC/FC, and afterwards we'd need to make the flatbuffers bc compatible (like adding to the end instead of inserting)

I see, will pay attention to this. I think we also have plan to phase out qcir by replacing online_prepare with QNN DLC, which will mitigate the maintaining effort. And it could be fully deprecated once multi-method RFC comes out.

cccclai

Thank you for adding the dynamic shape support!

cccclai · 2025-02-11T05:39:59Z

examples/qualcomm/utils.py

@@ -134,6 +148,32 @@ def push(self, inputs=None, input_list=None, files=None):
        for file_name in input_files:
            self._adb(["push", file_name, self.workspace])

+        # dynamic shape related
+        with tempfile.TemporaryDirectory() as tmp_dir:
+            if self.expected_input_shape and self.expected_output_shape:


Hmm could you elaborate a bit more regarding what do we compare with for the expected_input_shape and expected_output_shape?

I think the output tensor shape extracted from method in runtime will have the maximum value at dynamic dimension. expected_io_shape is used to get the expected size actually calculated by QNN.
For expected_io_dtype, since we're using quantized graph IO with BuildQuantIo pass in to_executorch. Looks like the output node meta from delegated LoweredModule could not be propagated, I have to introduce data type information for getting the correct number of bytes.

ohh..can you remind the reason to use BuildQuantIo? It has been too long and I might miss some context

In traditional ET QNN path, the graph IO will be inserted with QDQ for graph IO to stay in floating point. However, Q / DQ can only work in static shape now.
I think HTP team is extending the coverage of dynamic ops, I will clean up current workaround in the future if we start to have more supported ops.

Q / DQ can only work in static shape now.

Hmm, does it mean dynamic ops only support fp ops, or do I misunderstand?

Currently only few ops in 16bit fixed point can support dynamic shape. Since Q & DQ are not supported, I have to strip them by using BuildQuantIo pass.

In this case, you may consider this pass so we don't hack too much

executorch/exir/passes/quantize_io_pass.py

Line 4 in ee7d388

Thank you for the information, will try to have a follow up PR to clean it.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 21, 2025

haowhsu-quic force-pushed the dev_dynamic_shape branch 2 times, most recently from 87e5b41 to 404cdef Compare January 21, 2025 15:54

cccclai added partner: qualcomm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Qualcomm release notes: qualcomm Changes to the Qualcomm backend delegate labels Jan 22, 2025

haowhsu-quic force-pushed the dev_dynamic_shape branch from 404cdef to 5588113 Compare February 3, 2025 02:08

digantdesai added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Feb 3, 2025

Qualcomm AI Engine Direct - dynamic shape support

ba46b4f

summary: - dynamic shape related change for QC backend - brekage fix - test cases

haowhsu-quic force-pushed the dev_dynamic_shape branch from 5588113 to ba46b4f Compare February 5, 2025 16:26

cccclai reviewed Feb 11, 2025

View reviewed changes

cccclai approved these changes Feb 11, 2025

View reviewed changes

cccclai merged commit 2e204b9 into pytorch:main Feb 11, 2025
45 of 47 checks passed

Qualcomm AI Engine Direct - dynamic shape support #7780

Qualcomm AI Engine Direct - dynamic shape support #7780

Uh oh!

Conversation

haowhsu-quic commented Jan 21, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

pytorch-bot bot commented Jan 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7780

❗ 1 Active SEVs

✅ No Failures

Uh oh!

haowhsu-quic commented Jan 21, 2025

Uh oh!

cccclai commented Jan 21, 2025

Uh oh!

haowhsu-quic commented Jan 22, 2025

Uh oh!

facebook-github-bot commented Jan 22, 2025

Uh oh!

cccclai commented Jan 30, 2025

Uh oh!

facebook-github-bot commented Feb 3, 2025

Uh oh!

facebook-github-bot commented Feb 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cccclai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

haowhsu-quic commented Jan 21, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jan 21, 2025 •

edited

Loading