Add an activity for benchmarking only #4443

kirklandsign · 2024-07-29T17:43:59Z

Example usage:

adb shell am start -n com.example.executorchllamademo/com.example.executorchllamademo.Benchmarking --es "model_path" "/data/local/tmp/llama/stories_kv_sdpa_fp32_xnn.pte" --es "tokenizer_path" "/data/local/tmp/llama/tokenizer.bin"

Then

adb shell run-as com.example.executorchllamademo cat files/benchmark_results.txt

See result like

loadStart: 1722275116708
loadEnd: 1722275117629
generateStart: 1722275117629
generateEnd: 1722275118834
tokens/second: 105.445114

Note: We use activity because we assume it has higher RAM priority than a background service.

pytorch-bot · 2024-07-29T17:44:02Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4443

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit 2644fac with merge base d9cfd6a ():

NEW FAILURE - The following job has failed:

Android / test-llama-app (bpe) / mobile-job (android) (gh)
Process completed with exit code 1.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

Apple / test-demo-ios / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2024-07-29T17:57:33Z

@kirklandsign has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Summary: Example usage: ``` adb shell am start -n com.example.executorchllamademo/com.example.executorchllamademo.Benchmarking --es "model_path" "/data/local/tmp/llama/stories_kv_sdpa_fp32_xnn.pte" --es "tokenizer_path" "/data/local/tmp/llama/tokenizer.bin" ``` Then ``` adb shell run-as com.example.executorchllamademo cat files/benchmark_results.txt ``` See result like ``` loadStart: 1722275116708 loadEnd: 1722275117629 generateStart: 1722275117629 generateEnd: 1722275118834 tokens/second: 105.445114 ``` Note: We use activity because we assume it has higher RAM priority than a background service. Differential Revision: D60399589 Pulled By: kirklandsign

facebook-github-bot · 2024-07-29T18:15:52Z

This pull request was exported from Phabricator. Differential Revision: D60399589

Summary: Example usage: ``` adb shell am start -n com.example.executorchllamademo/com.example.executorchllamademo.Benchmarking --es "model_path" "/data/local/tmp/llama/stories_kv_sdpa_fp32_xnn.pte" --es "tokenizer_path" "/data/local/tmp/llama/tokenizer.bin" ``` Then ``` adb shell run-as com.example.executorchllamademo cat files/benchmark_results.txt ``` See result like ``` loadStart: 1722275116708 loadEnd: 1722275117629 generateStart: 1722275117629 generateEnd: 1722275118834 tokens/second: 105.445114 ``` Note: We use activity because we assume it has higher RAM priority than a background service. Differential Revision: D60399589 Pulled By: kirklandsign

facebook-github-bot · 2024-07-29T19:52:06Z

This pull request was exported from Phabricator. Differential Revision: D60399589

facebook-github-bot · 2024-08-08T19:24:07Z

@kirklandsign has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Summary: Example usage: ``` adb shell am start -n com.example.executorchllamademo/com.example.executorchllamademo.Benchmarking --es "model_path" "/data/local/tmp/llama/stories_kv_sdpa_fp32_xnn.pte" --es "tokenizer_path" "/data/local/tmp/llama/tokenizer.bin" ``` Then ``` adb shell run-as com.example.executorchllamademo cat files/benchmark_results.txt ``` See result like ``` loadStart: 1722275116708 loadEnd: 1722275117629 generateStart: 1722275117629 generateEnd: 1722275118834 tokens/second: 105.445114 ``` Note: We use activity because we assume it has higher RAM priority than a background service. Differential Revision: D60399589

facebook-github-bot · 2024-08-08T19:33:13Z

This pull request was exported from Phabricator. Differential Revision: D60399589

guangy10 · 2024-08-09T20:38:30Z

examples/demo-apps/android/LlamaDemo/app/src/main/AndroidManifest.xml

+            android:name=".Benchmarking"
+            android:exported="true">
+            <intent-filter>
+                <action android:name="com.example.executorchllamademo.BENCHMARK" />


Can we name it to be something more generic, e.g. llm benchmark runner

Can we later move the entire app under executorch/extension/llm as an extension for llm benchmarking?

is not a blocker for this PR. Once you addressed 1) this PR should be ready to go

Fixed 1 now. Working on moving it out of llamademoapp and use a separate app (for generic as well)

guangy10 · 2024-08-09T20:41:07Z

...o-apps/android/LlamaDemo/app/src/main/java/com/example/executorchllamademo/Benchmarking.java

+  long loadEnd;
+  long generateStart;
+  long generateEnd;
+  String tokens;


We would want to dump it to a standard and portable format later, e.g. json. Something we can reuse from AIBench.

That's a good idea and I plan to do that on the API as well

There is a task for it T197322159. You can coordinate with Varun on it.

guangy10 · 2024-08-09T20:48:32Z

In a follow up PR, you may want to connect this new apk to android-perf.yml here https://github.com/pytorch/executorch/blob/main/.github/workflows/android-perf.yml#L160-L162 and see if the test-spec could recognize it.

Summary: Example usage: ``` adb shell am start -n com.example.executorchllamademo/com.example.executorchllamademo.Benchmarking --es "model_path" "/data/local/tmp/llama/stories_kv_sdpa_fp32_xnn.pte" --es "tokenizer_path" "/data/local/tmp/llama/tokenizer.bin" ``` Then ``` adb shell run-as com.example.executorchllamademo cat files/benchmark_results.txt ``` See result like ``` loadStart: 1722275116708 loadEnd: 1722275117629 generateStart: 1722275117629 generateEnd: 1722275118834 tokens/second: 105.445114 ``` Note: We use activity because we assume it has higher RAM priority than a background service. Differential Revision: D60399589

facebook-github-bot · 2024-08-09T23:35:31Z

This pull request was exported from Phabricator. Differential Revision: D60399589

Summary: Example usage: ``` adb shell am start -n com.example.executorchllamademo/com.example.executorchllamademo.LlmBenchmarkRunner --es "model_path" "/data/local/tmp/llama/stories_kv_sdpa_fp32_xnn.pte" --es "tokenizer_path" "/data/local/tmp/llama/tokenizer.bin" ``` Then ``` adb shell run-as com.example.executorchllamademo cat files/benchmark_results.txt ``` See result like ``` loadStart: 1722275116708 loadEnd: 1722275117629 generateStart: 1722275117629 generateEnd: 1722275118834 tokens/second: 105.445114 ``` Note: We use activity because we assume it has higher RAM priority than a background service. Differential Revision: D60399589 Pulled By: kirklandsign

facebook-github-bot · 2024-08-09T23:36:56Z

This pull request was exported from Phabricator. Differential Revision: D60399589

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 29, 2024

facebook-github-bot force-pushed the android-benchmarking-activity branch from ebb6936 to 005f88f Compare July 29, 2024 18:15

facebook-github-bot added the fb-exported label Jul 29, 2024

facebook-github-bot force-pushed the android-benchmarking-activity branch from 005f88f to 1dda6b5 Compare July 29, 2024 19:51

facebook-github-bot force-pushed the android-benchmarking-activity branch from 43e80fd to 42dfff6 Compare August 8, 2024 19:33

guangy10 reviewed Aug 9, 2024

View reviewed changes

facebook-github-bot force-pushed the android-benchmarking-activity branch from 42dfff6 to 73316a7 Compare August 9, 2024 23:35

facebook-github-bot force-pushed the android-benchmarking-activity branch from 73316a7 to 2644fac Compare August 9, 2024 23:36

guangy10 approved these changes Aug 9, 2024

View reviewed changes

facebook-github-bot merged commit 440048c into main Aug 12, 2024
41 of 44 checks passed

kirklandsign deleted the android-benchmarking-activity branch August 21, 2024 17:44

Add an activity for benchmarking only #4443

Add an activity for benchmarking only #4443

Uh oh!

Conversation

kirklandsign commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4443

❌ 1 New Failure, 1 Unrelated Failure

Uh oh!

facebook-github-bot commented Jul 29, 2024

Uh oh!

facebook-github-bot commented Jul 29, 2024

Uh oh!

facebook-github-bot commented Jul 29, 2024

Uh oh!

facebook-github-bot commented Aug 8, 2024

Uh oh!

facebook-github-bot commented Aug 8, 2024

Uh oh!

guangy10 Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

guangy10 Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

kirklandsign Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

guangy10 Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

kirklandsign Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

guangy10 Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

guangy10 commented Aug 9, 2024

Uh oh!

facebook-github-bot commented Aug 9, 2024

Uh oh!

facebook-github-bot commented Aug 9, 2024

Uh oh!

Uh oh!

Uh oh!

kirklandsign commented Jul 29, 2024 •

edited

Loading

pytorch-bot bot commented Jul 29, 2024 •

edited

Loading