Qualcomm AI Engine Direct - Refine max spill fill buffer setting

[shewu-quic]

• Get required spillFillBufferSize from context binary and set to compiler_spec
• Quantize embedding op in qnn.
• If enable multi-contexts, maxSpillFillBuffer could not set to zero.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5925

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit b974e0a with merge base c06a708 ():

NEW FAILURE - The following job has failed:

• trunk / test-models-macos (cmake, llama2, xnnpack-delegation, macos-m1-stable, 90) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[shewu-quic]

Hi @cccclai,
This PR is to refine how to set spill fill buffer and quantize the embedding op.
The purpose is to run llama3 instruct with 128 seq_len on 12GB device.
Could you please have a look on it? Thank you!

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Hey can you apply this patch to the PR? There are some lint failures.

diff --git a/backends/qualcomm/runtime/backends/QnnBackendCache.h b/backends/qualcomm/runtime/backends/QnnBackendCache.h
index c492a3d1d..074092bc1 100644
--- a/backends/qualcomm/runtime/backends/QnnBackendCache.h
+++ b/backends/qualcomm/runtime/backends/QnnBackendCache.h
@@ -24,8 +24,8 @@ class QnnBackendCache {
     ONLINE_PREPARE = 3,
   };
   explicit QnnBackendCache(const QnnExecuTorchContextBinary& qnn_context_blob)
-      : qnn_context_blob_(qnn_context_blob){};
-  virtual ~QnnBackendCache();
+      : qnn_context_blob_(qnn_context_blob){}
+  virtual ~QnnBackendCache() = 0;
   QnnBackendCache(const QnnBackendCache&) = delete;
   QnnBackendCache(QnnBackendCache&&) = delete;
   QnnBackendCache& operator=(const QnnBackendCache&) = delete;
@@ -57,7 +57,7 @@ class QnnBackendCache {
   virtual Error RetrieveBackendBinaryInfo(
       const QnnSystemContext_BinaryInfo_t* binaryinfo) {
     return Error::Ok;
-  };
+  }

  private:
   Error GetQnnGraphInfoFromBinary();
diff --git a/backends/qualcomm/runtime/backends/htpbackend/HtpBackendCache.h b/backends/qualcomm/runtime/backends/htpbackend/HtpBackendCache.h
index 117704f6e..8bf20af37 100644
--- a/backends/qualcomm/runtime/backends/htpbackend/HtpBackendCache.h
+++ b/backends/qualcomm/runtime/backends/htpbackend/HtpBackendCache.h
@@ -13,9 +13,9 @@ namespace executor {
 namespace qnn {
 class HtpBackendCache : public QnnBackendCache {
  public:
-  HtpBackendCache(const QnnExecuTorchContextBinary& qnn_context_blob)
+  explicit HtpBackendCache(const QnnExecuTorchContextBinary& qnn_context_blob)
       : QnnBackendCache(qnn_context_blob), spill_fill_buf_(0) {}
-  ~HtpBackendCache() {}
+  ~HtpBackendCache() = default;

   uint64_t GetSpillFillBufferSize() {
     return spill_fill_buf_;

[cccclai]

I cherry pick you PR to #5989 and make changes on top of it. Also rebase to solve some linter breakage in your base commit and file #5995 to resolve the another lint issue.

[cccclai]

Do you expect HtpBackendCache do nothing?

[shewu-quic]

Yes, I believe it doesn’t need to do anything further.

[cccclai]

#5989 is merged and your name will be the author for this commit.

[shewu-quic]

I cherry pick you PR to #5989 and make changes on top of it. Also rebase to solve some linter breakage in your base commit and file #5995 to resolve the another lint issue.

Thank you very much!
So, I could close this PR?

[cccclai]

I cherry pick you PR to #5989 and make changes on top of it. Also rebase to solve some linter breakage in your base commit and file #5995 to resolve the another lint issue.

Thank you very much! So, I could close this PR?

Yeah this PR can be closed.