[Executorch][perf-ci] Fix perf ci

[kimishpatel]

Stack from ghstack (oldest at bottom):

• -> [Executorch][perf-ci] Fix perf ci #8374

Summary:
Previous PR #7927 deecoupled max_seq_length from kv cache. That broke
perf ci workflow. Fix that.

Test Plan:
Trigger it manually and check
apple perf: https://github.com/pytorch/executorch/actions/runs/13267110949
android perf: https://github.com/pytorch/executorch/actions/runs/13267110908

Reviewers:

Subscribers:

Tasks:

Tags:

cc @guangy10 @huydhn @kirklandsign @shoumikhin

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8374

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 553d875 with merge base 78752a0 ():

NEW FAILURE - The following job has failed:

• Apple / build-demo-ios / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[guangy10]

The linked job in the PR summary doesn't run with the SpinQuant and QLora. You need to trigger the job using the model id on Hugging Face:

• https://huggingface.co/meta-llama/Llama-3.2-1B
• https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8
• https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct-QLORA_INT4_EO8

[mergennachin]

Also the readme pages?

https://github.com/pytorch/executorch/blob/main/examples/models/llama/README.md

https://github.com/pytorch/executorch/blob/main/examples/demo-apps/apple_ios/LLaMA/docs/delegates/xnnpack_README.md

https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/LlamaDemo/docs/delegates/xnnpack_README.md

[kimishpatel]

doesn't run with the SpinQuant and QLora

let me do this in follow up PR

Actually let me just do it here

[kimishpatel]

The linked job in the PR summary doesn't run with the SpinQuant and QLora. You need to trigger the job using the model id on Hugging Face:

• https://huggingface.co/meta-llama/Llama-3.2-1B
• https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8
• https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct-QLORA_INT4_EO8

need to trigger the job using the model id on Hugging Face:

What does this mean? Is there description as to how to trigger this. I followed steps here https://github.com/pytorch/executorch/tree/main/extension/benchmark

[kimishpatel]

doesn't run with the SpinQuant and QLora

let me do this in follow up PR

Actually let me just do it here

this is updated. But I think I am gonna have to do one more round of scrubbing in subsequent PRs for various incarnations of llama

[guangy10]

The linked job in the PR summary doesn't run with the SpinQuant and QLora. You need to trigger the job using the model id on Hugging Face:

• https://huggingface.co/meta-llama/Llama-3.2-1B
• https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8
• https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct-QLORA_INT4_EO8

need to trigger the job using the model id on Hugging Face:

What does this mean? Is there description as to how to trigger this. I followed steps here https://github.com/pytorch/executorch/tree/main/extension/benchmark

You need to specify the models you want to benchmark against explicitly, separated by ",". In this case, they are "meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8,meta-llama/Llama-3.2-1B-Instruct-QLORA_INT4_EO8". See the screenshot for example.

Updated the screenshot. You need to run against your branch, not on main.

[guangy10]

The changes look good to me

[kimishpatel]

The linked job in the PR summary doesn't run with the SpinQuant and QLora. You need to trigger the job using the model id on Hugging Face:

• https://huggingface.co/meta-llama/Llama-3.2-1B
• https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8
• https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct-QLORA_INT4_EO8

need to trigger the job using the model id on Hugging Face:

What does this mean? Is there description as to how to trigger this. I followed steps here https://github.com/pytorch/executorch/tree/main/extension/benchmark

You need to specify the models you want to benchmark against explicitly, separated by ",". In this case, they are "meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8,meta-llama/Llama-3.2-1B-Instruct-QLORA_INT4_EO8". See the screenshot for example.

oh you are right. I forgot about that step.