Qualcomm AI Engine Direct - Implement sdk profiler and intergrate with Qnn profiler

[shewu-quic]

Summary:

• Implement Qnn Profiler for htp backend
  For now, only support kProfileDetailed to profile the performance of each operator with cycle unit.
  Follow up item: Add more qnn profile item
• Intergrated with sdk profiler
• Add the argument etdump_path to dump etdump which analyzes the contents by INSPECTOR in qnn_executorch_runner
• Add unit test to test profile
• Add export example to generate etrecord

Reproduce commands:

python3 backends/qualcomm/tests/test_qnn_delegate.py TestQNNFloatingPointOperator.test_qnn_backend_conv2d -b /local3/mnt/workspace/shewu/executorch/executorch_shewu/executorch/build_android -s $LANAI1 -H $TWL1 -m SM8650 -r /local3/mnt/workspace/shewu/executorch/executorch_shewu/executorch -a /local3/mnt/workspace/shewu/executorch/unit_test -i /local3/mnt/workspace/shewu/executorch/models/data/ImageNet-Mini/images --enable_profile
# Pull the EtDump
adb pull  /data/local/tmp/qnn_executorch_test/etdump.etdp .
# Run inspector to produce the following table
python3 -m sdk.inspector.inspector_cli --etdump_path etdump.etdp --etrecord_path etrecord.bin

╒════╤════════════════════╤══════════════════════════════════════════════╤═════════════╤═════════════╤═════════════╤═════════════╤═════════════╤═════════════╤════════════╤═══════════════════╤═════════════════════════╕
│    │ event_block_name   │ event_name                                   │    p10 (ms) │    p50 (ms) │    p90 (ms) │    avg (ms) │    min (ms) │    max (ms) │ op_types   │ is_delegated_op   │ delegate_backend_name   │
╞════╪════════════════════╪══════════════════════════════════════════════╪═════════════╪═════════════╪═════════════╪═════════════╪═════════════╪═════════════╪════════════╪═══════════════════╪═════════════════════════╡
│  0 │ Default            │ Method::init                                 │   123.898   │   123.898   │   123.898   │   123.898   │   123.898   │   123.898   │ []         │ False             │                         │
├────┼────────────────────┼──────────────────────────────────────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼────────────┼───────────────────┼─────────────────────────┤
│  1 │ Default            │ Program::load_method                         │   123.926   │   123.926   │   123.926   │   123.926   │   123.926   │   123.926   │ []         │ False             │                         │
├────┼────────────────────┼──────────────────────────────────────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼────────────┼───────────────────┼─────────────────────────┤
│  2 │ Execute            │ Input OpId_2 (cycles)                        │  4018       │  4018       │  4018       │  4018       │  4018       │  4018       │ []         │ True              │ QnnBackend              │
├────┼────────────────────┼──────────────────────────────────────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼────────────┼───────────────────┼─────────────────────────┤
│  3 │ Execute            │ aten_permute_copy_default:OpId_17 (cycles)   │ 16765       │ 16765       │ 16765       │ 16765       │ 16765       │ 16765       │ []         │ True              │ QnnBackend              │
├────┼────────────────────┼──────────────────────────────────────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼────────────┼───────────────────┼─────────────────────────┤
│  4 │ Execute            │ aten_convolution_default:OpId_23 (cycles)    │ 12768       │ 12768       │ 12768       │ 12768       │ 12768       │ 12768       │ []         │ True              │ QnnBackend              │
├────┼────────────────────┼──────────────────────────────────────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼────────────┼───────────────────┼─────────────────────────┤
│  5 │ Execute            │ aten_convolution_default_1:OpId_30 (cycles)  │  9439       │  9439       │  9439       │  9439       │  9439       │  9439       │ []         │ True              │ QnnBackend              │
├────┼────────────────────┼──────────────────────────────────────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼────────────┼───────────────────┼─────────────────────────┤
│  6 │ Execute            │ aten_permute_copy_default_1:OpId_33 (cycles) │  2551       │  2551       │  2551       │  2551       │  2551       │  2551       │ []         │ True              │ QnnBackend              │
├────┼────────────────────┼──────────────────────────────────────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼────────────┼───────────────────┼─────────────────────────┤
│  7 │ Execute            │ OpId_0 (cycles)                              │     0       │     0       │     0       │     0       │     0       │     0       │ []         │ True              │ QnnBackend              │
├────┼────────────────────┼──────────────────────────────────────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼────────────┼───────────────────┼─────────────────────────┤
│  8 │ Execute            │ Output OpId_3 (cycles)                       │  3054       │  3054       │  3054       │  3054       │  3054       │  3054       │ []         │ True              │ QnnBackend              │
├────┼────────────────────┼──────────────────────────────────────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼────────────┼───────────────────┼─────────────────────────┤
│  9 │ Execute            │ DELEGATE_CALL                                │     1.17151 │     1.17151 │     1.17151 │     1.17151 │     1.17151 │     1.17151 │ []         │ False             │ QnnBackend              │
├────┼────────────────────┼──────────────────────────────────────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼────────────┼───────────────────┼─────────────────────────┤
│ 10 │ Execute            │ Method::execute                              │     1.18318 │     1.18318 │     1.18318 │     1.18318 │     1.18318 │     1.18318 │ []         │ False             │                         │
╘════╧════════════════════╧══════════════════════════════════════════════╧═════════════╧═════════════╧═════════════╧═════════════╧═════════════╧═════════════╧════════════╧═══════════════════╧═════════════════════════╛

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/2227

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 2db56a1 with merge base 588c391 ():

NEW FAILURE - The following job has failed:

• pull / unittest / macos (buck2) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Olivia-liu]

Thanks for making and sharing the draft diff! This is great progress. I'll get back to you soon with answers to the questions you asked.

[shewu-quic]

Just a little note for the two questions,

1. Cross-Compiling
   The current SDK build flow presents a bit challenges when it comes to cross-compilation scenarios. As the below, it requires some workarounds to achieve successful builds.
   https://github.com/pytorch/executorch/blob/3e67470ffb948e47d784deb84452d905c5f36728/sdk/CMakeLists.txt#L111 

For ExternalProject_Add, it builds on x86 toolchain but does not install to _host_build folder. 
When add subdirectory for third-party/flatcc, link flatcc, and install flatcc library, it will build with android toolchain and override lib and bin in third-party/flatcc. 
They will confuse the cmake and fail to build due to the same lib and bin under the third-party/flatcc for x86 and android toolchains.

2. The unit of profile item
    Because the unit of some profiling items is cycle in HTP. May I know could we set different units for profiling items with this api?

[Olivia-liu]

Sorry for taking a while to get back to you.

1. Cross-Compiling

Thanks for finding a workaround for it. @tarun292 will verify if the workaround breaks anything else of the sdk, if not, I think we can merge this PR.

2. The unit of profile item

This is something that's on our TODO list but unfortunately there's no way to set different units at this moment.

[shewu-quic]

Thanks for your response. I am looking forward for this feature which set the unit for profile item.
I have rebased my branch.
Could you please help to have a look on the PR, thank you :)

[Olivia-liu]

Hey, could you please make sure you add the comment Dave suggested in the cmake file? Thanks for finding the workaround by the way, we think it's OK to land, but should be well documented. And thanks a lot for doing the integration and sorry for taking a while to review it

[facebook-github-bot]

@Olivia-liu has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[shewu-quic]

Hey, could you please make sure you add the comment Dave suggested in the cmake file? Thanks for finding the workaround by the way, we think it's OK to land, but should be well documented. And thanks a lot for doing the integration and sorry for taking a while to review it

I added it in the cmake file.
Thanks for your effort. It is very nice feature to profile more detailed information.

[facebook-github-bot]

@Olivia-liu has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@Olivia-liu merged this pull request in 0b12daf.