Define generic Android benchmark metric structure #5332

huydhn · 2024-09-12T23:36:22Z

To be able to display the benchmark results, we need the following information:

About the model
- Name, i.e. mv2
- The backend it uses, i.e. xnnpack
- The quantization (dtype) applied, i.e. q8
About the metric
- Name, i.e. token_per_sec. Note that this needs to be flexible to cover future metrics
- Value
- An optional target (so that we can highlight regression if it happens)
More metadata
- The device name, i.e. samsung
- The device model and its Android version
- More can be included here

I codified these fields in a new BenchmarkMetric class, so that the benchmark results can be expressed as a list of different metrics in the result JSON.

NB: Atm, the information about the model is extracted from its name, i.e. NAME_BACKEND_QUANTIZATION.pte, but it's better to get it from the file itself instead. Achieving this needs a bit more research.

Testing

https://github.com/pytorch/executorch/actions/runs/10843580072

The JSON for llama2:

[
  {
    "actual": 247,
    "arch": "SM-S901U1 / 12",
    "benchmarkModel": {
      "backend": "",
      "name": "llama2",
      "quantization": ""
    },
    "device": "samsung",
    "metric": "model_load_time(ms)",
    "target": 0
  },
  {
    "actual": 367,
    "arch": "SM-S901U1 / 12",
    "benchmarkModel": {
      "backend": "",
      "name": "llama2",
      "quantization": ""
    },
    "device": "samsung",
    "metric": "generate_time(ms)",
    "target": 0
  },
  {
    "actual": 342.69662,
    "arch": "SM-S901U1 / 12",
    "benchmarkModel": {
      "backend": "",
      "name": "llama2",
      "quantization": ""
    },
    "device": "samsung",
    "metric": "token_per_sec",
    "target": 0
  }
]

The JSON for mv2_xnnpack_q8. I keep the average latency here as the final number to show later on the dashboard.

[
  {
    "actual": 91.1,
    "arch": "SM-S908U1 / 12",
    "benchmarkModel": {
      "backend": "xnnpack",
      "name": "mv2",
      "quantization": "q8"
    },
    "device": "samsung",
    "metric": "avg_inference_latency(ms)",
    "target": 0
  }
]

pytorch-bot · 2024-09-12T23:36:25Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5332

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 2ea88c4 with merge base 0d0b14a ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2024-09-13T05:42:13Z

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

huydhn · 2024-09-13T05:43:01Z

For more context, this format is influenced by gpt-fast benchmark format from PyTorch https://github.com/pytorch/pytorch/blob/main/benchmarks/gpt_fast/benchmark.py#L25

kirklandsign · 2024-09-13T06:25:57Z

extension/android/benchmark/app/src/main/java/org/pytorch/minibench/LlmBenchmarkActivity.java

@@ -97,6 +124,7 @@ class StatsInfo {
  long generateStart;
  long generateEnd;
  String tokens;
+  String name;



Maybe we can add a "loadStatus" for

executorch/extension/android/benchmark/app/src/main/java/org/pytorch/minibench/LlmBenchmarkActivity.java

Line 63 in 5c5499a

return;

in case of failure

kirklandsign · 2024-09-13T06:26:18Z

Thank you! That's really clean!

kirklandsign · 2024-09-13T06:27:42Z

.../android/LlamaDemo/app/src/main/java/com/example/executorchllamademo/LlmBenchmarkRunner.java

+  double actual;
+  double target;


nit: actualValue, targetValue to reduce future user questions? Or should we give a comment to pointer to https://github.com/pytorch/pytorch/blob/main/benchmarks/gpt_fast/benchmark.py#L25 lol

We can use any name here I guess. I need to write a script to insert the JSON into the database later, so some transformation can be done at that stage instead

Yeah, when I looked at the example in the PR summary, I was having the same question what "target" refers to. In ET target typically means the target device. So yeah, it would be nice to make the name self-explain

kirklandsign · 2024-09-13T06:29:44Z

extension/android/benchmark/app/src/main/java/org/pytorch/minibench/BenchmarkActivity.java

@@ -46,18 +46,21 @@ protected void onCreate(Bundle savedInstanceState) {
      stats.latency.add(forwardMs);
    }

-    // TODO (huydhn): Remove txt files here once the JSON format is ready


I should probably log the time for module.loadMethod() before first forward() 😢

I see, let me try to copy it from llama and add one here too

kirklandsign · 2024-09-13T06:31:00Z

Feel free to land as is

facebook-github-bot · 2024-09-13T17:18:32Z

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-09-13T17:22:24Z

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

kirklandsign · 2024-09-13T17:27:07Z

extension/android/benchmark/app/src/main/java/org/pytorch/minibench/BenchmarkActivity.java

    Module module = Module.load(model.getPath());
+    stats.loadEnd = System.currentTimeMillis();


Sorry, let's add a line module.loadMethod("forward") after line 52

kirklandsign · 2024-09-13T17:29:57Z

extension/android/benchmark/app/src/main/java/org/pytorch/minibench/BenchmarkActivity.java

@@ -47,7 +47,11 @@ protected void onCreate(Bundle savedInstanceState) {
    // TODO: Format the string with a parsable format
    Stats stats = new Stats();

+    // Record the time it takes to load the model
+    stats.loadStart = System.currentTimeMillis();


Let's use https://docs.oracle.com/javase/7/docs/api/java/lang/System.html#nanoTime() (for all places as well)

guangy10 · 2024-09-13T17:42:18Z

.../android/LlamaDemo/app/src/main/java/com/example/executorchllamademo/LlmBenchmarkRunner.java

+  double targetValue;
+
+  // Let's see which information we want to include here
+  final String device = Build.BRAND;


Device brand may not be very useful, I think more details will be needed, e.g. samsung_s22. More device spec would be helpful as well, e.g. RAM, CPU, OS version, etc. At least RAM I think as RAM would be the bottleneck for most edge models

As the fields here are flexible, we can definitely add more information about the devices, let me try to add as many as I could find (maybe RAM and CPU info). We can have subsequent PR to add more I guess

@kirklandsign Do you know a way to get the commercial name of the device i.e. s22. The model field kind of map to it, i.e. I search for S901U1 and it means S22, but having a more familiar name make it easier

lol, it looks like some additional mapping is needed https://github.com/jaredrummler/AndroidDeviceNames. I think it's better then to do it on the dashboard side in this case (when displaying the device on the dashboard)

@huydhn If gathering any of the additional info is non-trivial, let's leave it for future as we are not shooting to get a perfect metrics measurement by PTC. Let's take the low-hanging fruits and merge this PR. I'd rather prioritize to get the similar structured metrics for iOS app, and get the structured metrics displayed in the CI or in the dashboard(optional if not possible by PTC)

Sounds good, I will update iOS benchmark app after this to generate a similar JSON results

guangy10 · 2024-09-13T17:42:39Z

.../android/LlamaDemo/app/src/main/java/com/example/executorchllamademo/LlmBenchmarkRunner.java

+  final String device = Build.BRAND;
+  // The phone model and Android release version
+  final String arch = Build.MODEL;
+  final String os = "Android " + Build.VERSION.RELEASE;


okay OS version is covered

guangy10 · 2024-09-13T17:43:59Z

.../android/LlamaDemo/app/src/main/java/com/example/executorchllamademo/LlmBenchmarkRunner.java

  long loadStart;
  long loadEnd;
  long generateStart;
  long generateEnd;
  String tokens;
+  String name;


name of what? Can we add a comment or make it self-explain?

Ah, it's the model name, let me update the variable to call it so

guangy10 · 2024-09-13T17:56:29Z

extension/android/benchmark/app/src/main/java/org/pytorch/minibench/BenchmarkMetric.java

+class BenchmarkMetric {
+  public static class BenchmarkModel {
+    // The model name, i.e. stories110M
+    String name;
+    String backend;
+    String quantization;
+
+    public BenchmarkModel(final String name, final String backend, final String quantization) {
+      this.name = name;
+      this.backend = backend;
+      this.quantization = quantization;
+    }
+  }
+
+  BenchmarkModel benchmarkModel;
+
+  // The metric name, i.e. TPS
+  String metric;
+
+  // The actual value and the option target value
+  double actualValue;
+  double targetValue;
+
+  // Let's see which information we want to include here
+  final String device = Build.BRAND;
+  // The phone model and Android release version
+  final String arch = Build.MODEL;
+  final String os = "Android " + Build.VERSION.RELEASE;
+
+  public BenchmarkMetric(
+      final BenchmarkModel benchmarkModel,
+      final String metric,
+      final double actualValue,
+      final double targetValue) {
+    this.benchmarkModel = benchmarkModel;
+    this.metric = metric;
+    this.actualValue = actualValue;
+    this.targetValue = targetValue;
+  }


It's okay that we will have to define a duplicate class for iOS app for now. Later we may want to move it to c++ maybe to make it shareble between the iOS and Android app

facebook-github-bot · 2024-09-13T18:11:05Z

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

guangy10

Looks great! Awesome work @huydhn

facebook-github-bot · 2024-09-13T18:58:19Z

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-09-13T20:19:28Z

@huydhn merged this pull request in 0d1644f.

Summary: Given the way iOS benchmark app measures model load time, inference time, and memory usage using `measureWithMetrics` with `XCTClockMetric` and `XCTMemoryMetric`. I think the easiest way to gather the benchmark metric is to do it after the test finishes and parse the output. In the same spirit as #5332, this PR adds more information about the device so that it can be parsed later. I add the information into the test name, shoumikhin plz let me know if you know a better way to pass this information around. The output looks like this with the information in the test case name, i.e. `test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1` ``` Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' started. /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Peak Physical, kB] average: 125171.731, relative standard deviation: 0.010%, values: [125158.624000, 125175.008000, 125175.008000, 125158.624000, 125191.392000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical_peak, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Physical, kB] average: -29.491, relative standard deviation: -228.792%, values: [-49.152000, -16.384000, 16.384000, 49.152000, -147.456000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Clock Monotonic Time, s] average: 0.160, relative standard deviation: 3.460%, values: [0.163377, 0.165837, 0.164974, 0.152334, 0.154970], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' passed (1.322 seconds). Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' started. /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Peak Physical, kB] average: 127403.280, relative standard deviation: 0.000%, values: [127403.280000, 127403.280000, 127403.280000, 127403.280000, 127403.280000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical_peak, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Physical, kB] average: 0.000, relative standard deviation: 0.000%, values: [0.000000, 0.000000, 0.000000, 0.000000, 0.000000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Clock Monotonic Time, s] average: 0.000, relative standard deviation: 41.029%, values: [0.000001, 0.000001, 0.000001, 0.000001, 0.000001], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' passed (0.132 seconds). ``` Reviewed By: guangy10 Differential Revision: D62902327 Pulled By: huydhn

Summary: Given the way iOS benchmark app measures model load time, inference time, and memory usage using `measureWithMetrics` with `XCTClockMetric` and `XCTMemoryMetric`. I think the easiest way to gather the benchmark metric is to do it after the test finishes and parse the output. In the same spirit as #5332, this PR adds more information about the device so that it can be parsed later. I add the information into the test name, shoumikhin plz let me know if you know a better way to pass this information around. The output looks like this with the information in the test case name, i.e. `test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1` ``` Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' started. /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Peak Physical, kB] average: 125171.731, relative standard deviation: 0.010%, values: [125158.624000, 125175.008000, 125175.008000, 125158.624000, 125191.392000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical_peak, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Physical, kB] average: -29.491, relative standard deviation: -228.792%, values: [-49.152000, -16.384000, 16.384000, 49.152000, -147.456000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:134: Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Clock Monotonic Time, s] average: 0.160, relative standard deviation: 3.460%, values: [0.163377, 0.165837, 0.164974, 0.152334, 0.154970], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 Test Case '-[Tests test_forward_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' passed (1.322 seconds). Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' started. /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Peak Physical, kB] average: 127403.280, relative standard deviation: 0.000%, values: [127403.280000, 127403.280000, 127403.280000, 127403.280000, 127403.280000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical_peak, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Memory Physical, kB] average: 0.000, relative standard deviation: 0.000%, values: [0.000000, 0.000000, 0.000000, 0.000000, 0.000000], performanceMetricID:com.apple.dt.XCTMetric_Memory.physical, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 /Users/huydo/Storage/mine/executorch/extension/apple/Benchmark/Tests/Tests.mm:85: Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' measured [Clock Monotonic Time, s] average: 0.000, relative standard deviation: 41.029%, values: [0.000001, 0.000001, 0.000001, 0.000001, 0.000001], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000 Test Case '-[Tests test_load_models_llama2_iPhone_iPhone14,2_iOS_17.6.1]' passed (0.132 seconds). ``` Pull Request resolved: #5410 Reviewed By: guangy10 Differential Revision: D62902327 Pulled By: huydhn fbshipit-source-id: 9b3de5601ae8625ee6db1d63e43accb5c60ec33b

Define generic Android benchmark metric structure

27c895d

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 12, 2024

huydhn added 2 commits September 12, 2024 18:31

Parse the model name for backend and quantization

57c3c5e

Keep model and Android version

5c5499a

huydhn requested a review from kirklandsign September 13, 2024 05:41

huydhn marked this pull request as ready for review September 13, 2024 05:41

huydhn requested a review from guangy10 September 13, 2024 05:42

kirklandsign reviewed Sep 13, 2024

View reviewed changes

kirklandsign approved these changes Sep 13, 2024

View reviewed changes

kirklandsign reviewed Sep 13, 2024

View reviewed changes

huydhn added 2 commits September 13, 2024 09:39

Address review comments

0488a12

Merge branch 'main' into android-benchmark-json-output

60800bd

Move Android OS version to a separate field

a529c3b

kirklandsign reviewed Sep 13, 2024

View reviewed changes

guangy10 reviewed Sep 13, 2024

View reviewed changes

Use nano second

b2b117e

guangy10 reviewed Sep 13, 2024

View reviewed changes

guangy10 approved these changes Sep 13, 2024

View reviewed changes

Add memory info

a66b834

guangy10 approved these changes Sep 13, 2024

View reviewed changes

Init DeviceInfo

2ea88c4

guangy10 approved these changes Sep 13, 2024

View reviewed changes

facebook-github-bot closed this in 0d1644f Sep 13, 2024

facebook-github-bot added the Merged label Sep 13, 2024

huydhn mentioned this pull request Sep 17, 2024

Add DeviceInfo in iOS benchmark run #5410

Closed

		Module module = Module.load(model.getPath());
		stats.loadEnd = System.currentTimeMillis();

Define generic Android benchmark metric structure #5332

Define generic Android benchmark metric structure #5332

Uh oh!

Conversation

huydhn commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

pytorch-bot bot commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5332

✅ No Failures

Uh oh!

facebook-github-bot commented Sep 13, 2024

Uh oh!

huydhn commented Sep 13, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kirklandsign commented Sep 13, 2024

Uh oh!

kirklandsign Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kirklandsign commented Sep 13, 2024

Uh oh!

facebook-github-bot commented Sep 13, 2024

Uh oh!

facebook-github-bot commented Sep 13, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guangy10 Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 13, 2024

Uh oh!

guangy10 left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 13, 2024

Uh oh!

facebook-github-bot commented Sep 13, 2024

Uh oh!

Uh oh!

huydhn commented Sep 12, 2024 •

edited

Loading

pytorch-bot bot commented Sep 12, 2024 •

edited

Loading

kirklandsign Sep 13, 2024 •

edited

Loading

guangy10 Sep 13, 2024 •

edited

Loading