-
Notifications
You must be signed in to change notification settings - Fork 608
Onboard ExecuTorch to benchmark database v3 #7117
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7117
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 8d564f1 with merge base c726a9b ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
with: | ||
benchmark-results-dir: benchmark-results/v2 | ||
dry-run: false | ||
schema-version: v2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't GITHUB_TOKEN
required by v2?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, this token is used to query the job ID, which isn't available as part of the job context. In the old schema, it is supplied by .github/scripts/extract_benchmark_results.py
script using the handmade regex in https://github.com/pytorch/executorch/blob/main/.github/scripts/extract_benchmark_results.py#L24. In the new version, the action uses the token to do it on behalf of the caller, so .github/scripts/extract_benchmark_results.py
would only need to extract the benchmark results from AWS, and there is no need to use any hardcoded logic. I'll eventually clean up the script later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great!
ExecuTorch is currently using v2 schema which doesn't have a dedicated field to store the delegate backend. Onboarding it to v3 will allow the dashboard to have this field and unblock T204741729.
tinyllama_xnnpack+custom+qe_fp32
with + in the delegate backend. This has been fixed on iOS, but I missed this on Android.Testing
The records are available on both v2 and v3 tables, i.e.
select * from oss_ci_benchmark_v3 where workflow_id = 12061317075
(The failure for stories110M is fixed by #7091)