Skip to content

Commit ed7e65f

Browse files
pytorchbotGuang Yang
pytorchbot
and
Guang Yang
authored
Correct Core ML perf metrics (#6000)
Correct Core ML perf metrics (#5862) Summary: Pull Request resolved: #5862 Reviewed By: huydhn Differential Revision: D63873696 Pulled By: guangy10 fbshipit-source-id: e4cb20c0a87b01d9c78dce085dad6f97bcf03b28 (cherry picked from commit 784eb51) Co-authored-by: Guang Yang <[email protected]>
1 parent 245aa4e commit ed7e65f

File tree

2 files changed

+19
-8
lines changed

2 files changed

+19
-8
lines changed

‎.github/workflows/apple-perf.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ jobs:
7474
# Separate default values from the workflow dispatch. To ensure defaults are accessible
7575
# during scheduled runs and to provide flexibility for different defaults between
7676
# on-demand and periodic benchmarking.
77-
CRON_DEFAULT_MODELS: "stories110M,mv3,ic4,resnet50,edsr,mobilebert,w2l"
77+
CRON_DEFAULT_MODELS: "stories110M,mv3,mv2,ic4,ic3,resnet50,edsr,mobilebert,w2l"
7878
CRON_DEFAULT_DEVICES: "apple_iphone_15"
7979
CRON_DEFAULT_DELEGATES: "xnnpack,coreml"
8080
run: |

‎extension/benchmark/README.md

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ The ExecuTorch project introduces an advanced benchmarking infrastructure design
1414

1515
- **Performance Monitoring**: Stores results in a database with a dashboard for tracking performance and detecting regressions.
1616

17-
> **Disclaimer:** The infrastructure is new and experimental. We're working on improving its accessibility and stability over time. The models performance tuning is still a work in progress. The benchmark numbers presented in this document are not vetted yet and should be treated as preliminary results.
17+
> **Disclaimer:** The infrastructure is new and experimental. We're working on improving its accessibility and stability over time.
1818
1919

2020
## Preliminary Benchmark Results
@@ -27,16 +27,27 @@ Below is a table summarizing some example data points obtained via the infra. Th
2727
| DeepLabV3 (dl3) | QNN FP16 | [182.207](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | [9.759](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | Samsung S22 |
2828
| MobileNetV3 (mv3) | XNN Q8 | [34.024](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | [252.628](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | Samsung S22 |
2929
| MobileNetV3 (mv3) | QNN FP16 | [168.414](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785) | [1.182](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785) | Samsung S22 |
30-
| MobileNetV3 (mv3) | COREML FP16 | [61.583](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158) | [682.197](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158) | Apple iPhone 15 Pro |
31-
| MobileNetV2 (mv2) | XNN Q8 | [14.397](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | [10.796](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | Samsung S22 |
30+
| MobileNetV3 (mv3) | COREML FP16 | [57.372](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930691) | [0.429](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930691) | Apple iPhone 15 Pro |
31+
| MobileNetV2 (mv2) | XNNPACK Q8 | [14.397](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | [10.796](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | Samsung S22 |
3232
| MobileNetV2 (mv2) | QNN FP16 | [136.862](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097) | [0.673](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097) | Samsung S22 |
33-
| InceptionV4 (ic4) | XNN Q8 | [87.617](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | [117.937](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | Samsung S22 |
33+
| MobileNetV2 (mv2) | COREML FP16 | [50.892](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078594) | [0.631](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078594) | Apple iPhone 15 Pro |
34+
| InceptionV4 (ic4) | XNNPACK Q8 | [87.617](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | [117.937](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | Samsung S22 |
3435
| InceptionV4 (ic4) | QNN FP16 | [163.943](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381) | [2.734](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381) | Samsung S22 |
35-
| InceptionV3 (ic3) | XNN Q8 | [60.708](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | [98.390](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | Samsung S22 |
36+
| InceptionV4 (ic4) | COREML FP16 | [118.686](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930140) | [4.289](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930140) | Apple iPhone 15 Pro |
37+
| InceptionV3 (ic3) | XNNPACK Q8 | [60.708](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | [98.390](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | Samsung S22 |
3638
| InceptionV3 (ic3) | QNN FP16 | [134.732](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425) | [1.351](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425) | Samsung S22 |
39+
| InceptionV3 (ic3) | COREML FP16 | [86.728](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078753) | [1.391](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078753) | Apple iPhone 15 Pro |
40+
| DeepLabV3 (dl3) | XNNPACK Q8 | [90.616](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | [666.219](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | Samsung S22 |
41+
| DeepLabV3 (dl3) | QNN FP16 | [182.207](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | [9.759](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | Samsung S22 |
42+
| ResNet50 (resnet50) | XNNPACK Q8 | [55.462](https://github.com/pytorch/executorch/actions/runs/10911725781/job/30285857102) | [37.021](https://github.com/pytorch/executorch/actions/runs/10911725781/job/30285857102) | Apple iPhone 15 Pro |
43+
| ResNet50 (resnet50) | COREML FP16 | [68.943](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930818) | [1.979](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930818) | Apple iPhone 15 Pro |
3744
| TorchVisionVit (vit) | QNN FP16 | [174.430](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | [199.279](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | Samsung S22 |
38-
| ResNet50 (resnet50) | COREML FP16 | [84.453](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731) | [1329.846](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731) | Apple iPhone 15 Pro |
39-
| Wave2Letter (w2l) | XNN Q8 | [33.913](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | [135.584](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | Apple iPhone 15 Pro |
45+
| Wave2Letter (w2l) | XNNPACK Q8 | [33.913](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | [135.584](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | Apple iPhone 15 Pro |
46+
| Wave2Letter (w2l) | COREML FP16 | [109.254](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999931566) | [28.465](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999931566) | Apple iPhone 15 |
47+
| MobileBERT (mobilebert) | XNNPACK FP32 | [26.499](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930558) | [33.978](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930558) | Apple iPhone 15 Pro |
48+
| MobileBERT (mobilebert) | COREML FP16 | [206.202](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930398) | [1.873](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930398) | Apple iPhone 15 Pro |
49+
| EDSR (edsr) | XNNPACK Q8 | [3.190](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929836) | [168.429](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929836) | Apple iPhone 15 Pro |
50+
| EDSR (edsr) | COREML FP16 | [156.075](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929690) | [77.346](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929690) | Apple iPhone 15 Pro |
4051

4152

4253
## Supported Use Cases

0 commit comments

Comments
 (0)