Arm backend: Test TOSA, Ethos-U55 and Ethos-U85 on github #6888

zingo · 2024-11-15T12:27:46Z

This will run some more models in the github test flow and also enable some unit tests to use Corston3x0 FVP.

This structure up the Arm backend testing into separate runable scripts in the same structure as other backends.

pytorch-bot · 2024-11-15T12:27:50Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6888

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit d371bc7 with merge base 241cd0c ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

zingo · 2024-11-16T12:41:32Z

Hi @digantdesai this changes files outside Arm folders and need Meta review

freddan80 · 2024-11-18T08:55:03Z

lgtm, but @digantdesai should have a look as well

zingo · 2024-11-21T12:46:25Z

@pytorchbot label "topic: not user facing"

digantdesai · 2024-11-21T17:25:26Z

Love it. One consideration though, we should try to set this scripts such a way that not just a CI but a human (or a tutorial doc) can use this directly without any changes.

This way there is only one source of truth.

Stamping it to unblock you.

digantdesai · 2024-11-21T17:26:08Z

.github/workflows/pull.yml

+        .ci/scripts/setup-arm_baremetal.sh
+
+        # Run pytest
+        .ci/scripts/test_arm_baremetal.sh test_pytest


Do you know the time increase?

I think currently running with FVP vs without is about 2min 30sec (14ish min vs 16,5ish min ) if OK to always run with FVP then we are happy doing that. I just tried to avoid it in the geneal pull job from my understanding of old discussions :)

But that was before I measured the job diff and spotted that it might have been ok

digantdesai · 2024-11-21T17:27:28Z

.github/workflows/trunk.yml

-        # Run arm unit tests
-        pytest -c /dev/null -v -n auto --cov=./ --cov-report=xml backends/arm/test
+        # Run arm pytest tests with Corstone3x0 FVP testing
+        .ci/scripts/test_arm_baremetal.sh test_pytest_ethosu_fvp


I guess firing fvp a bunch of time is more time consuming compared to running a single large-ish model like mv2?

Maybe, we have a patch to make it run FVP faster ongoing also.

But the we are also adding more FVP testing. Lets see how it pans out a bit over time.

yeah I saw that. Thanks

zingo · 2024-11-21T18:17:34Z

The plan was to make it easy to run. So we can use it in our CI also and other tests. We can describe them in the readme, at least the test script. Do you want the script to move out of the .ci Folders and placed under backend/arm/test or similar place in that in that case?

digantdesai · 2024-11-22T17:34:29Z

Do you want the script to move out of the .ci Folders and placed under backend/arm/test or similar place in that in that case?

I don't have a strong preference on where the script lives, as long as a user can just run it on a shell without hacking it up would be a nice-to-have, esp for Tutorial where we can point and know it will work because it is tested by CI.

.github/workflows/pull.yml

zingo · 2024-11-25T07:25:36Z

I don't have a strong preference on where the script lives, as long as a user can just run it on a shell without hacking it up would be a nice-to-have, esp for Tutorial where we can point and know it will work because it is tested by CI.

Could make since (and I kind of like to) move at least the test part (setup could be CI/user specific, but it could also follow) to something like backends/arm/test

I look into that after some other stuff, depending on your wish of getting this is fast or not I can update this patch (a bit slower) or take it as a later patch if this get merged before.

zingo · 2024-12-17T12:42:13Z

Love it. One consideration though, we should try to set this scripts such a way that not just a CI but a human (or a tutorial doc) can use this directly without any changes.

This way there is only one source of truth.

Stamping it to unblock you.

Updated the patch to it's more easy to run for anyone and added it to our README.md
IT will be nice and more easy now to add more stuff to this over time :)

I don't have a strong preference on where the script lives, as long as a user can just run it on a shell without hacking it up would be a nice-to-have, esp for Tutorial where we can point and know it will work because it is tested by CI.

Moved the script to our testing folder, it makes more sense to have it there and more available for anyone.

This will run some more models in the github test flow and also enable some unit tests to use Corston3x0 FVP. This structure up the Arm backend testing into separate runable scripts in the same structure as other backends. Signed-off-by: Zingo Andersen <[email protected]> Change-Id: I5e11b1aca19460845e330b84d0696513c400c0f0

zingo · 2024-12-20T13:25:17Z

MacOs test error seems unrelated to patch

…-Ethos-U85-on-github

swolchok · 2025-01-14T23:02:03Z

.ci/scripts/setup-arm-baremetal-tools.sh

+
+# NB: This function could be used to install Arm dependencies
+# Setup arm example environment (including TOSA tools)
+git config --global user.email "[email protected]"


why do we need to do this? I tried to run this script to repro a build issue and unknowingly set the name and email incorrectly on several github commits.

Ouch sorry about that.
It just followed along when moved from install_arm() in
.ci/scripts/utils.sh (in the same patch)

I'm not sure if it is needed when running in the CI nowdays, some time back the examples/arm/setup.sh did some git cloning of Vela and other repos and I assume it might have been needed then. Ot it could be related to the patch scripts that was used for Vela patches.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 15, 2024

zingo added partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm ciflow/trunk labels Nov 15, 2024

zingo requested a review from digantdesai November 15, 2024 12:30

zingo force-pushed the Arm-backend-Test-TOSA,-Ethos-U55-and-Ethos-U85-on-github branch from ee45572 to bd5ecd9 Compare November 15, 2024 14:38

zingo removed the request for review from digantdesai November 15, 2024 15:20

zingo force-pushed the Arm-backend-Test-TOSA,-Ethos-U55-and-Ethos-U85-on-github branch 2 times, most recently from f466583 to 2a6eab5 Compare November 15, 2024 15:37

zingo marked this pull request as draft November 15, 2024 20:05

zingo force-pushed the Arm-backend-Test-TOSA,-Ethos-U55-and-Ethos-U85-on-github branch 2 times, most recently from ed5aaba to d6affa5 Compare November 16, 2024 12:13

zingo requested a review from digantdesai November 16, 2024 12:39

zingo marked this pull request as ready for review November 16, 2024 12:40

freddan80 approved these changes Nov 18, 2024

View reviewed changes

pytorch-bot bot added the topic: not user facing label Nov 21, 2024

digantdesai approved these changes Nov 21, 2024

View reviewed changes

malfet reviewed Nov 22, 2024

View reviewed changes

.github/workflows/pull.yml Outdated Show resolved Hide resolved

zingo marked this pull request as draft November 28, 2024 13:08

zingo force-pushed the Arm-backend-Test-TOSA,-Ethos-U55-and-Ethos-U85-on-github branch from d98d5b0 to 510aef3 Compare December 17, 2024 12:35

zingo marked this pull request as ready for review December 17, 2024 12:37

zingo force-pushed the Arm-backend-Test-TOSA,-Ethos-U55-and-Ethos-U85-on-github branch from 510aef3 to 108bd73 Compare December 17, 2024 13:28

zingo marked this pull request as draft December 17, 2024 15:48

zingo force-pushed the Arm-backend-Test-TOSA,-Ethos-U55-and-Ethos-U85-on-github branch 4 times, most recently from 426ce47 to b2d6a38 Compare December 20, 2024 09:27

zingo force-pushed the Arm-backend-Test-TOSA,-Ethos-U55-and-Ethos-U85-on-github branch from b2d6a38 to 1ad6338 Compare December 20, 2024 11:13

zingo marked this pull request as ready for review December 20, 2024 11:43

Merge branch 'pytorch:main' into Arm-backend-Test-TOSA,-Ethos-U55-and…

d371bc7

…-Ethos-U85-on-github

freddan80 merged commit 9a23cff into pytorch:main Jan 8, 2025
137 checks passed

swolchok reviewed Jan 14, 2025

View reviewed changes

zingo deleted the Arm-backend-Test-TOSA,-Ethos-U55-and-Ethos-U85-on-github branch February 13, 2025 12:05

Arm backend: Test TOSA, Ethos-U55 and Ethos-U85 on github #6888

Arm backend: Test TOSA, Ethos-U55 and Ethos-U85 on github #6888

Uh oh!

Conversation

zingo commented Nov 15, 2024

Uh oh!

pytorch-bot bot commented Nov 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6888

✅ No Failures

Uh oh!

zingo commented Nov 16, 2024

Uh oh!

freddan80 commented Nov 18, 2024

Uh oh!

zingo commented Nov 21, 2024

Uh oh!

digantdesai commented Nov 21, 2024

Uh oh!

digantdesai Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

zingo Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

zingo Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

digantdesai Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

zingo Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

zingo Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

digantdesai Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

zingo commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

digantdesai commented Nov 22, 2024

Uh oh!

Uh oh!

zingo commented Nov 25, 2024

Uh oh!

zingo commented Dec 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zingo commented Dec 20, 2024

Uh oh!

Uh oh!

swolchok Jan 14, 2025

Choose a reason for hiding this comment

Uh oh!

zingo Jan 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 15, 2024 •

edited

Loading

zingo Nov 21, 2024 •

edited

Loading

zingo commented Nov 21, 2024 •

edited

Loading

zingo commented Dec 17, 2024 •

edited

Loading

zingo Jan 14, 2025 •

edited

Loading