Skip to content

Onboard ExecuTorch to benchmark database v3 #7117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Dec 3, 2024

Conversation

huydhn
Copy link
Contributor

@huydhn huydhn commented Nov 27, 2024

ExecuTorch is currently using v2 schema which doesn't have a dedicated field to store the delegate backend. Onboarding it to v3 will allow the dashboard to have this field and unblock T204741729.

  • I opt to keep both v2 and v3 in the interim. The former can be cleaned up once the dashboard is updated.
  • Fix a small regex bug on Android when parsing tinyllama_xnnpack+custom+qe_fp32 with + in the delegate backend. This has been fixed on iOS, but I missed this on Android.

Testing

The records are available on both v2 and v3 tables, i.e. select * from oss_ci_benchmark_v3 where workflow_id = 12061317075

(The failure for stories110M is fixed by #7091)

Copy link

pytorch-bot bot commented Nov 27, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7117

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 8d564f1 with merge base c726a9b (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 27, 2024
@huydhn huydhn had a problem deploying to upload-benchmark-results November 28, 2024 02:52 — with GitHub Actions Failure
@huydhn huydhn had a problem deploying to upload-benchmark-results November 28, 2024 02:59 — with GitHub Actions Failure
@huydhn huydhn had a problem deploying to upload-benchmark-results November 28, 2024 03:22 — with GitHub Actions Failure
@huydhn huydhn temporarily deployed to upload-benchmark-results November 28, 2024 03:35 — with GitHub Actions Inactive
@huydhn huydhn temporarily deployed to upload-benchmark-results November 28, 2024 03:43 — with GitHub Actions Inactive
with:
benchmark-results-dir: benchmark-results/v2
dry-run: false
schema-version: v2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't GITHUB_TOKEN required by v2?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, this token is used to query the job ID, which isn't available as part of the job context. In the old schema, it is supplied by .github/scripts/extract_benchmark_results.py script using the handmade regex in https://github.com/pytorch/executorch/blob/main/.github/scripts/extract_benchmark_results.py#L24. In the new version, the action uses the token to do it on behalf of the caller, so .github/scripts/extract_benchmark_results.py would only need to extract the benchmark results from AWS, and there is no need to use any hardcoded logic. I'll eventually clean up the script later.

Copy link
Contributor

@guangy10 guangy10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@huydhn huydhn merged commit cf16b85 into main Dec 3, 2024
41 checks passed
@huydhn huydhn deleted the migrate-benchmark-results-v3 branch December 3, 2024 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. topic: not user facing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants