[Executorch][CI] Fix qnn runner ci job scripts

[kimishpatel]

Stack from ghstack (oldest at bottom):

• [Executorch] Add quantized kv cache to oss ci #6997
• -> [Executorch][CI] Fix qnn runner ci job scripts #7049
• [Executorch][custom ops] Change lib loading logic to account for package dir #7038
• [Executorch][BE] Rename sdpa_with_kv_cache.py to custom_ops.py #6996
• [Executorch][llama] Rename update_quantized_cache to update_cache #6914
• Dont quantize the current token for attention #5715
• [Executorch][quant] Optimize per channel dequantize #5670

QNN backend's AOT artifacts were being copied in the source directory instead of
them being installed, as build artifacts, in appropriate package directory.

As a result we had to use PYTHON_PATH=".." in test_llama.sh CI script when
running CI for qnn. This results in executorch having two locations as where
the package is installed. This PR fixes that.

Differential Revision: D66406274

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7049

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Merge Blocking SEVs

There is 1 active merge blocking SEVs. Please view them below:

• (merge blocking) Long CI Queue Times: VolumeLimitExceeded

If you must merge, use @pytorchbot merge -f.

❌ 10 New Failures, 1 Cancelled Job

As of commit 51523e8 with merge base 43555d2 ():

NEW FAILURES - The following jobs have failed:

• Propose to merge ghstack orig PRs to main / Try to create a PR with ghstack /orig branch (gh)
  Process completed with exit code 1.
• pull / test-llama-runner-qnn-linux (fp32, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 0a7cc8e4372b4f1ad9a3beba415ca936bc5eac365b0862ab0c520f92c9bc213e /exec failed with exit code 255
• pull / unittest / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-pybind-build-macos (cmake) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-qnn-model (fp32, dl3) / linux-job (gh)
  RuntimeError: Command docker exec -t 9b091985d8a8fc6ba8fe44216af4ad7060973071014bedc81526c4e52f3a2c6e /exec failed with exit code 1
• trunk / test-qnn-model (fp32, ic3) / linux-job (gh)
  RuntimeError: Command docker exec -t 0149fb93096626189d1a672abad330dc0ad0fe07c98954bc4c9bb52b7ec50ccc /exec failed with exit code 1
• trunk / test-qnn-model (fp32, ic4) / linux-job (gh)
  RuntimeError: Command docker exec -t a26c9b489ce8eb831070b0aa33978a67448e12627160a443220a5ee0a644f0fa /exec failed with exit code 1
• trunk / test-qnn-model (fp32, mv2) / linux-job (gh)
  RuntimeError: Command docker exec -t 2e4f6610c4ef4b9a0d85cfb8c0df04f52e6cfc9ea39ff400127c133dca9c9a2e /exec failed with exit code 1
• trunk / test-qnn-model (fp32, mv3) / linux-job (gh)
  RuntimeError: Command docker exec -t 68ef708bf557975e98bf602f7985fad20e2aa85dc7696492c7e8793101e15040 /exec failed with exit code 1
• trunk / test-qnn-model (fp32, vit) / linux-job (gh)
  RuntimeError: Command docker exec -t e469f09f47ecd48073e38e114f195d9859c59c7a712dc2b11c9f286e23b4d24d /exec failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• trunk / test-models-macos (cmake, vit, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D66406274

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D66406274

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D66406274

[kimishpatel]

I am gonna have to abandon this because imports of model in qnn backend think that examples are part of the pip package of executorch. And to get around that we need to do PYTHON_PATH="EXECUTORCH_ROOT". I really dont like it, but dont have bandwidth to figure out better solution. So I will work around it for the purpose of this stack. cc: @cccclai @larryliu0820

[cccclai]

@kimishpatel hmm how does this stack affect qnn CI job?

[kimishpatel]

@kimishpatel hmm how does this stack affect qnn CI job?

Not directly, but now we are importing quantized_kv_cache unconditionally from export_llama_lib. Because qnn CI stack has 1) PYTHON_PATH=.. which affects python imports and confuses the location of where executorch package is installed and 2) various models in examples/backends/qualcomm treat examples folder (as in import executorch.example.models..) as part of executorch pip package, but we pip install of executorch does not include those, so we have to reply on PYTHON_PATH.

2 in particularly is problematic because I see that pattern in a few places but it is not clear if examples is supposed be part of pip package. @larryliu0820 should confirm

[larryliu0820]

@kimishpatel hmm how does this stack affect qnn CI job?

Not directly, but now we are importing quantized_kv_cache unconditionally from export_llama_lib. Because qnn CI stack has 1) PYTHON_PATH=.. which affects python imports and confuses the location of where executorch package is installed and 2) various models in examples/backends/qualcomm treat examples folder (as in import executorch.example.models..) as part of executorch pip package, but we pip install of executorch does not include those, so we have to reply on PYTHON_PATH.

2 in particularly is problematic because I see that pattern in a few places but it is not clear if examples is supposed be part of pip package. @larryliu0820 should confirm

Ideally we shouldn’t include examples in pip package. Anything reusable should go into extension.Ilm and should not tied to any specific model. For example I’m aware a lot of code useful for llama are still in examples.models.llama. They should really go inside extension.llm for reusability

[kimishpatel]

llama

There much more widepsread use of pattern from executorch.examples.... failures in this particular case are from from executorch.examples.qualcomm and were resolved by use of PYTHON_PATH in the past. But the way it was setup was causing executorch package directory to be listed as both source as well as what is in site-packages

[cccclai]

Seems stale