Add all relevant testcases for Arm Ethos-U85 #5346

zingo · 2024-09-13T15:07:42Z

Add separate tests for Ethos-U85 to all backend operator tests.
Updated ethos-u-vela version to support more operators.

Signed-off-by: Per Åstrand <[email protected]>
Signed-off-by: Tom Allsop <[email protected]>

Add separate tests for Ethos-U85 to all backend operator tests. Signed-off-by: Per Åstrand <[email protected]> Signed-off-by: Tom Allsop <[email protected]> Co-authored-by: Tom Allsop <[email protected]> Change-Id: I923372adec92d095f11cb78aa55b675b1be7f070

pytorch-bot · 2024-09-13T15:07:47Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5346

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 0e56d02 with merge base b2517d6 ():

NEW FAILURE - The following job has failed:

trunk / test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job (gh)
Library not loaded: @rpath/liblz4.1.dylib

This comment was automatically generated by Dr. CI and updates every 15 minutes.

zingo · 2024-09-13T15:08:49Z

@pytorchbot label ciflow/trunk

digantdesai

Let's fix some small issues?

backends/arm/test/ops/test_avg_pool.py

digantdesai · 2024-09-13T15:58:23Z

backends/arm/test/ops/test_conv.py

+            .partition()
+            .check_count({"torch.ops.higher_order.executorch_call_delegate": 1})
+            .check_not(["executorch_exir_dialects_edge__ops_aten_convolution_default"])
+            .to_executorch()


add serialize?

We only serialize if the test should be running on FVP. Since it's time consuming to run them all (and on an additional coming target) we probably want to run on FVP for full models only, but not decided internally yet.

digantdesai · 2024-09-13T15:58:47Z

backends/arm/test/ops/test_conv.py

@@ -315,6 +315,24 @@ def _test_conv2d_u55_BI_pipeline(
            .to_executorch()
        )

+    def _test_conv2d_u85_BI_pipeline(


why we are not sharing across u55/u85 like other places here?

Good point. Will fix.

pull request updated

backends/arm/test/ops/test_depthwise_conv.py

digantdesai · 2024-09-13T15:59:52Z

backends/arm/test/ops/test_expand.py

@@ -104,6 +106,15 @@ def test_expand_tosa_BI(self, test_input, multiples):

    # Expected failure since tosa.TILE is unsupported by Vela.
    @parameterized.expand(Expand.test_parameters)
-    @unittest.expectedFailure
+    @unittest.expectedFailure  # TODO: MLBEDSW-9386


are these tickets public?

No, it's an internal bug that is in progress to be sorted out in Vela.

backends/arm/test/models/test_mobilenet_v2_arm.py

digantdesai · 2024-09-13T16:02:25Z

examples/arm/setup.sh

@@ -216,7 +216,7 @@ function setup_vela() {
    if [[ ! -e ethos-u-vela ]]; then
        git clone https://review.mlplatform.org/ml/ethos-u/ethos-u-vela
        repo_dir="${root_dir}/ethos-u-vela"
-        base_rev=d362f5443f67b1e6213a9d8f124edff758efac96
+        base_rev=fe0eaa55c5ed319f78c01978f3b40eb11a9bcb38


In the summary can we add if there is any major change with this rev? Like one or two line would be helpful to understand the rationale.

Yes, good idea!

digantdesai · 2024-09-13T16:03:31Z

backends/arm/test/ops/test_add.py

+            common.get_u55_compile_spec(permute_memory_to_nhwc=True),
+            test_data,
+        )
+        if common.is_option_enabled("corstone300"):


how would this run u85 bitstream?

The compilespec is for Ethos-U55 and the run_method_and_compare_outpus is only run here for.

Right, but this function can be called for both u55 and u85 IIUC

digantdesai · 2024-09-13T16:05:33Z

backends/arm/test/ops/test_linear.py

@@ -172,9 +176,7 @@ def _test_linear_tosa_u55_BI_pipeline(
            .to_executorch()
            .serialize()
        )
-
-        if common.is_option_enabled("corstone300"):


I am not sure if I am following the pattern where we enable/disable these. Is this operator specific? I would rather tie it to the ip i.e. something like this, hypothetically

if common.is_option_enabled("corstone300") and compile_spec.ip == "u55":

the common.is_option_enabled("corstone300") only says if the FVP is installed on the target and can be run.
Right now it's only added for a few operators, but due to the time it adds to CI we probably want to move away from enabling it for all operator unit tests all the time.

IIUC this FVP is u55 specific while this function can also get u85 compile spec and vela/et can produce u85 binary. I was trying to highlight that the choice of fvp 300 vs 320 should be tied to the compile spec somehow.

…-U85

Signed-off-by: Per Åstrand <[email protected]> Change-Id: Ibee7fa81517bf766a550b97be97e2e3dbaed3a6c

…-U85

…thos-U55 This tests was remporary added but fails again, lets remove it from the testing for now. Signed-off-by: Zingo Andersen <[email protected]> Change-Id: I519a8b13e50e12c4093f1b8e4d80ae35bded23ff

backends/arm/test/ops/test_depthwise_conv.py

…-U85

digantdesai

Stamping to unblock.

facebook-github-bot · 2024-09-17T14:35:11Z

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Change-Id: Iffbed1aea3583a6c725367db852d411cb1eb7783

zingo · 2024-09-23T13:12:39Z

Just a try to rebase it on latest after Fredriks pull request got merged if you did this already on the "imported" side of things please ignore the update.

Signed-off-by: Zingo Andersen <[email protected]> Change-Id: I72baec5ab6cc85cd4f628ab2ea14f1c02c060b66

facebook-github-bot · 2024-09-23T22:23:13Z

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-09-24T04:17:42Z

@digantdesai merged this pull request in f4728f4.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 13, 2024

zingo added the partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm label Sep 13, 2024

pytorch-bot bot added the ciflow/trunk label Sep 13, 2024

zingo changed the title ~~Add all relevant testcases for Ethos-U85~~ Add all relevant testcases for Arm Ethos-U85 Sep 13, 2024

digantdesai reviewed Sep 13, 2024

View reviewed changes

zingo and others added 4 commits September 15, 2024 08:37

Merge branch 'pytorch:main' into Add-all-relevant-testcases-for-Ethos…

4ce29cf

…-U85

Fix tests to address review comments.

6a0fc0e

Signed-off-by: Per Åstrand <[email protected]> Change-Id: Ibee7fa81517bf766a550b97be97e2e3dbaed3a6c

Merge branch 'pytorch:main' into Add-all-relevant-testcases-for-Ethos…

4b9da19

…-U85

Arm backend: Remove testing of dw_conv2d_3x3_1x3x256x256_gp3_st1 on E…

77b12eb

…thos-U55 This tests was remporary added but fails again, lets remove it from the testing for now. Signed-off-by: Zingo Andersen <[email protected]> Change-Id: I519a8b13e50e12c4093f1b8e4d80ae35bded23ff

zingo commented Sep 16, 2024

View reviewed changes

backends/arm/test/ops/test_depthwise_conv.py Show resolved Hide resolved

Merge branch 'pytorch:main' into Add-all-relevant-testcases-for-Ethos…

2a197b1

…-U85

digantdesai approved these changes Sep 17, 2024

View reviewed changes

Increase memory allocated for inputs in case semi hosting is used.

e15095c

Change-Id: Iffbed1aea3583a6c725367db852d411cb1eb7783

zingo and others added 2 commits September 23, 2024 15:49

Merge branch 'main' into Add-all-relevant-testcases-for-Ethos-U85

e958a23

Arm Backend: fix merged typo in test_mobilenet_v2_arm.py

0e56d02

Signed-off-by: Zingo Andersen <[email protected]> Change-Id: I72baec5ab6cc85cd4f628ab2ea14f1c02c060b66

facebook-github-bot closed this in f4728f4 Sep 24, 2024

facebook-github-bot added the Merged label Sep 24, 2024

robell mentioned this pull request Sep 24, 2024

Add New Ethos-U85 support to Arm aot_arm_compiler #5345

Closed

zingo deleted the Add-all-relevant-testcases-for-Ethos-U85 branch September 24, 2024 11:25

Add all relevant testcases for Arm Ethos-U85 #5346

Add all relevant testcases for Arm Ethos-U85 #5346

Uh oh!

Conversation

zingo commented Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5346

❌ 1 New Failure

Uh oh!

zingo commented Sep 13, 2024

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 17, 2024

Uh oh!

zingo commented Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Sep 23, 2024

Uh oh!

facebook-github-bot commented Sep 24, 2024

Uh oh!

Uh oh!

zingo commented Sep 13, 2024 •

edited

Loading

pytorch-bot bot commented Sep 13, 2024 •

edited

Loading

zingo commented Sep 23, 2024 •

edited

Loading