Skip to content

Upload Apple iOS benchmark results to benchmark database #5982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

Upload Apple iOS benchmark results to benchmark database #5982

wants to merge 4 commits into from

Conversation

huydhn
Copy link
Contributor

@huydhn huydhn commented Oct 8, 2024 •
edited
Loading

Do the same for iOS benchmark, there is no JSON output atm, so I parse the test spec output directly

Testing

https://github.com/pytorch/executorch/actions/runs/11239684907

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Oct 8, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5982

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit c569208 with merge base 57e3c81 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 8, 2024
@huydhn huydhn temporarily deployed to upload-benchmark-results October 8, 2024 17:09 — with GitHub Actions Inactive
@huydhn huydhn requested review from guangy10 and shoumikhin October 8, 2024 20:07
@huydhn huydhn marked this pull request as ready for review October 8, 2024 20:07
@facebook-github-bot
Copy link
Contributor

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

guangy10
guangy10 reviewed Oct 8, 2024
View reviewed changes
.github/workflows/apple-perf.yml Outdated
--output-dir benchmark-results \
--repo ${{ github.repository }} \
--head-branch ${{ github.head_ref || github.ref_name }} \
--workflow-name ${{ github.workflow }} \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
--workflow-name ${{ github.workflow }} \
--workflow-name "${{ github.workflow }}" \

The workflow name must be passed as a string, otherwise you may see invalid arguments error

@guangy10
Copy link
Contributor

guangy10 commented Oct 8, 2024

there is no JSON output atm, so I parse the test spec output directly

Do you plan to add the JSON output?

@huydhn
Copy link
Contributor Author

huydhn commented Oct 8, 2024

Do you plan to add the JSON output?

I will need to sync with @shoumikhin on if it's possible to achieve that with the current form the the benchmark

@guangy10
Copy link
Contributor

guangy10 commented Oct 8, 2024

fyi, #6023 is putting up some fixes to include "backend" and "dtype" in the exported model name. Can't test it e2e for iOS since this PR is still WIP.

@guangy10
Copy link
Contributor

guangy10 commented Oct 8, 2024

@huydhn Can you schedule a run to include a non-genai model w/ coreml delegate, e.g. mv3? The coreml model will be named as mv3_coreml_all_fp16.pte, where "coreml_all" refers to the type name of the delegate and compute unit (e.g. ALL, CPU_ONLY, CPU_AND_GPU, CPU_AND_NE). Our models are taking ALL by default.

@huydhn
Copy link
Contributor Author

huydhn commented Oct 9, 2024 •
edited
Loading

Another test with stories110M and mv2 https://github.com/pytorch/executorch/actions/runs/11245821876 with coreml

@huydhn huydhn had a problem deploying to upload-benchmark-results October 9, 2024 00:18 — with GitHub Actions Failure
@facebook-github-bot
Copy link
Contributor

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@huydhn huydhn temporarily deployed to upload-benchmark-results October 9, 2024 02:50 — with GitHub Actions Inactive
@huydhn huydhn mentioned this pull request Oct 9, 2024
Closed
guangy10
guangy10 reviewed Oct 10, 2024
View reviewed changes
@facebook-github-bot
Copy link
Contributor

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@huydhn merged this pull request in 192ca82.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guangy10 guangy10 guangy10 approved these changes

@shoumikhin shoumikhin Awaiting requested review from shoumikhin

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@huydhn @facebook-github-bot @guangy10