Skip to content

Commit 784eb51

Browse files
Guang Yangfacebook-github-bot
Guang Yang
authored and
facebook-github-bot
committed
Correct Core ML perf metrics (#5862)
Summary: Pull Request resolved: #5862 Reviewed By: huydhn Differential Revision: D63873696 Pulled By: guangy10 fbshipit-source-id: e4cb20c0a87b01d9c78dce085dad6f97bcf03b28
1 parent 34e7ad8 commit 784eb51

File tree

2 files changed

+12
-7
lines changed

2 files changed

+12
-7
lines changed

‎.github/workflows/apple-perf.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ jobs:
7474
# Separate default values from the workflow dispatch. To ensure defaults are accessible
7575
# during scheduled runs and to provide flexibility for different defaults between
7676
# on-demand and periodic benchmarking.
77-
CRON_DEFAULT_MODELS: "stories110M,mv3,ic4,resnet50,edsr,mobilebert,w2l"
77+
CRON_DEFAULT_MODELS: "stories110M,mv3,mv2,ic4,ic3,resnet50,edsr,mobilebert,w2l"
7878
CRON_DEFAULT_DEVICES: "apple_iphone_15"
7979
CRON_DEFAULT_DELEGATES: "xnnpack,coreml"
8080
run: |

‎extension/benchmark/README.md

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ The ExecuTorch project introduces an advanced benchmarking infrastructure design
1414

1515
- **Performance Monitoring**: Stores results in a database with a dashboard for tracking performance and detecting regressions.
1616

17-
> **Disclaimer:** The infrastructure is new and experimental. We're working on improving its accessibility and stability over time. The models performance tuning is still a work in progress. The benchmark numbers presented in this document are not vetted yet and should be treated as preliminary results.
17+
> **Disclaimer:** The infrastructure is new and experimental. We're working on improving its accessibility and stability over time.
1818
1919

2020
## Preliminary Benchmark Results
@@ -25,22 +25,27 @@ Below is a table summarizing some example data points obtained via the infra. Th
2525
|-----------------------|--------------|----------------------|----------------------------|------------------------|
2626
| MobileNetV3 (mv3) | XNNPACK Q8 | [34.024](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | [252.628](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | Samsung S22 |
2727
| MobileNetV3 (mv3) | QNN FP16 | [168.414](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785) | [1.182](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785) | Samsung S22 |
28-
| MobileNetV3 (mv3) | COREML FP16 | [61.583](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158) | [682.197](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158) | Apple iPhone 15 Pro |
28+
| MobileNetV3 (mv3) | COREML FP16 | [57.372](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930691) | [0.429](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930691) | Apple iPhone 15 Pro |
2929
| MobileNetV2 (mv2) | XNNPACK Q8 | [14.397](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | [10.796](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | Samsung S22 |
3030
| MobileNetV2 (mv2) | QNN FP16 | [136.862](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097) | [0.673](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097) | Samsung S22 |
31-
| MobileNetV2 (mv2) | COREML FP16 | [63.019](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269520397) | [551.031](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269520397) | Apple iPhone 15 Plus |
31+
| MobileNetV2 (mv2) | COREML FP16 | [50.892](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078594) | [0.631](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078594) | Apple iPhone 15 Pro |
3232
| InceptionV4 (ic4) | XNNPACK Q8 | [87.617](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | [117.937](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | Samsung S22 |
3333
| InceptionV4 (ic4) | QNN FP16 | [163.943](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381) | [2.734](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381) | Samsung S22 |
34-
| InceptionV4 (ic4) | COREML FP16 | [134.402](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518983) | [1906.966](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518983) | Apple iPhone 15 Pro |
34+
| InceptionV4 (ic4) | COREML FP16 | [118.686](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930140) | [4.289](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930140) | Apple iPhone 15 Pro |
3535
| InceptionV3 (ic3) | XNNPACK Q8 | [60.708](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | [98.390](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | Samsung S22 |
3636
| InceptionV3 (ic3) | QNN FP16 | [134.732](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425) | [1.351](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425) | Samsung S22 |
37-
| InceptionV3 (ic3) | COREML FP16 | [91.808](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518511) | [1500.712](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518511) | Apple iPhone 15 Pro |
37+
| InceptionV3 (ic3) | COREML FP16 | [86.728](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078753) | [1.391](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078753) | Apple iPhone 15 Pro |
3838
| DeepLabV3 (dl3) | XNNPACK Q8 | [90.616](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | [666.219](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | Samsung S22 |
3939
| DeepLabV3 (dl3) | QNN FP16 | [182.207](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | [9.759](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | Samsung S22 |
4040
| ResNet50 (resnet50) | XNNPACK Q8 | [55.462](https://github.com/pytorch/executorch/actions/runs/10911725781/job/30285857102) | [37.021](https://github.com/pytorch/executorch/actions/runs/10911725781/job/30285857102) | Apple iPhone 15 Pro |
41-
| ResNet50 (resnet50) | COREML FP16 | [84.453](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731) | [1329.846](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731) | Apple iPhone 15 Pro |
41+
| ResNet50 (resnet50) | COREML FP16 | [68.943](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930818) | [1.979](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930818) | Apple iPhone 15 Pro |
4242
| TorchVisionVit (vit) | QNN FP16 | [174.430](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | [199.279](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | Samsung S22 |
4343
| Wave2Letter (w2l) | XNNPACK Q8 | [33.913](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | [135.584](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | Apple iPhone 15 Pro |
44+
| Wave2Letter (w2l) | COREML FP16 | [109.254](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999931566) | [28.465](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999931566) | Apple iPhone 15 |
45+
| MobileBERT (mobilebert) | XNNPACK FP32 | [26.499](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930558) | [33.978](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930558) | Apple iPhone 15 Pro |
46+
| MobileBERT (mobilebert) | COREML FP16 | [206.202](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930398) | [1.873](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930398) | Apple iPhone 15 Pro |
47+
| EDSR (edsr) | XNNPACK Q8 | [3.190](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929836) | [168.429](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929836) | Apple iPhone 15 Pro |
48+
| EDSR (edsr) | COREML FP16 | [156.075](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929690) | [77.346](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929690) | Apple iPhone 15 Pro |
4449

4550

4651
## Supported Use Cases

0 commit comments

Comments
 (0)