Arm backend: Add initial Llama model test case #8679

mansnils · 2025-02-25T11:16:52Z

Adds Llama model test case for TOSA-0.80+MI.
Handles Add and Mul where inputs have different ranks. New unit test parameters --llama_inputs added, without it test will be skipped. Tested with smaller stories, see examples/models/llama/UTILS.md. Adds get_llama_model() to export_llama_lib used in test case.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218

Adds Llama model test case for TOSA-0.80+MI. Handles Add and Mul where inputs have different ranks. New unit test parameters --llama_inputs added, without it test will be skipped. Tested with smaller stories, see examples/models/llama/UTILS.md. Adds get_llama_model() to export_llama_lib used in test case. Change-Id: I003bbcee8f0cc35193d793a4af9b031453114e71

pytorch-bot · 2025-02-25T11:16:58Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8679

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job

As of commit d523b6f with merge base 022a946 ():

CANCELLED JOB - The following job was cancelled. Please retry:

trunk / test-models-macos (resnet50, xnnpack-quantization-delegation) / macos-job (gh)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

mansnils · 2025-02-25T11:18:13Z

@pytorchbot label partner/arm

pytorch-bot · 2025-02-25T11:18:18Z

Didn't find following labels among repository labels: partner/arm

mansnils · 2025-02-25T11:19:13Z

@pytorchbot label "partner: arm"

mansnils · 2025-02-25T11:19:44Z

@pytorchbot label ciflow/trunk

pytorch-bot · 2025-02-25T11:19:51Z

To add these label(s) (ciflow/trunk) to the PR, please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

pytorch-bot · 2025-02-25T11:51:25Z

To add the ciflow label ciflow/trunk please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

mansnils · 2025-02-25T12:07:07Z

topic: not user facing

mansnils · 2025-02-25T12:07:26Z

topic: "not user facing"

digantdesai · 2025-02-26T08:31:40Z

backends/arm/test/models/test_llama_arm.py

@@ -0,0 +1,125 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.


I will take a look but just a nit
s/test_llama_arm.py/test_llama.py/

I will take a look ...

Merge time? :)

Renamed to test_llama.py

zingo · 2025-02-26T15:27:56Z

MacOs fails are unrelated

digantdesai · 2025-02-27T05:32:59Z

.ci/scripts/setup-arm-baremetal-tools.sh

@@ -9,3 +9,5 @@
 git config --global user.email "[email protected]"
 git config --global user.name "Github Executorch"
 bash examples/arm/setup.sh --i-agree-to-the-contained-eula
+
+./examples/models/llama3_2_vision/install_requirements.sh


is this the best place to put this?

The idea was to have it in place before the testing but thinking about it now I think you are right
Having it as part of all the steps that use pytest in the file
backends/arm/test/test_arm_baremetal.sh
(or create a prepare test step)
Might be better and easier for anyone to just run the test and they will install needed dependencies them self.

backends/arm/operators/op_add.py

digantdesai · 2025-02-27T05:37:35Z

backends/arm/operators/op_slice.py

@@ -32,9 +32,11 @@ def define_node(
        output: TosaArg,
    ) -> None:

+        # See slice_copy_support.py
+        assert len(inputs) == 4 or (len(inputs) == 5 and inputs[4].number == 1)


Nit: add an error message

backends/arm/test/conftest.py

backends/arm/test/models/test_llama.py

digantdesai · 2025-02-27T05:48:33Z

backends/arm/test/models/test_llama.py

+                .check_count({"torch.ops.higher_order.executorch_call_delegate": 14})
+                .to_executorch()
+                .run_method_and_compare_outputs(
+                    inputs=llama_inputs, atol=1.8, rtol=0.01


how did we come up with atol? Also this is running on MI + tosa-ref w/o quantization, right?

The tolerance is really bad here. This is just to get the test passing. It it something we need to investigate. Added a TODO.
Yes this is testing reference vs TOSA without quantization.

mansnils · 2025-03-05T12:00:10Z

ping @digantdesai

shoumikhin · 2025-03-17T03:30:14Z

@digantdesai how do we want to proceed with this?

kimishpatel · 2025-03-17T14:00:56Z

@digantdesai how do we want to proceed with this?

Digant is on pat leave right now, so expect delays

digantdesai · 2025-03-18T18:17:57Z

LGTM.

zingo · 2025-03-19T13:28:42Z

MacOS fails unrelated

kirklandsign · 2025-03-20T18:54:22Z

Sorry @zingo we probably need to revert this and import and run internal tests before landing.

executorch/backends/arm/test:cast_int64_pass - main

executorch/backends/arm/test/conftest.py:47: in pytest_configure
    pytest._test_options["llama_inputs"] = config.option.llama_inputs  # type: ignore[attr-defined]
E   AttributeError: 'Namespace' object has no attribute 'llama_inputs'

This line:

pytest._test_options["llama_inputs"] = config.option.llama_inputs  # type: ignore[attr-defined]

probably need to use getattr to check?

This reverts commit 4c54bab.

larryliu0820 · 2025-03-20T20:38:29Z

@kirklandsign @digantdesai is it possible to add the tests into unittest-buck CI job?

kirklandsign · 2025-03-20T21:07:22Z

@kirklandsign @digantdesai is it possible to add the tests into unittest-buck CI job?

Oh are these tests triggered on GH right now? Seems like some config missing issue?

larryliu0820 · 2025-03-20T21:43:34Z

@kirklandsign @digantdesai is it possible to add the tests into unittest-buck CI job?

Oh are these tests triggered on GH right now? Seems like some config missing issue?

Created #9476 to track this.

Adds Llama model test case for TOSA-0.80+MI. Handles Add and Mul where inputs have different ranks. New unit test parameters --llama_inputs added, without it test will be skipped. Tested with smaller stories, see examples/models/llama/UTILS.md. Adds get_llama_model() to export_llama_lib used in test case.

mansnils requested review from lucylq, jackzhxng and digantdesai as code owners February 25, 2025 11:16

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 25, 2025

pytorch-bot bot added the partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm label Feb 25, 2025

zingo added the ciflow/trunk label Feb 25, 2025

pytorch-bot bot removed the ciflow/trunk label Feb 25, 2025

zingo changed the title ~~Arm: Add initial Llama model test case~~ Arm backend: Add initial Llama model test case Feb 25, 2025

mansnils added the topic: not user facing label Feb 25, 2025

Reduce number of delegates in test_llama_arm.py

e7c117b

zingo added the ciflow/trunk label Feb 26, 2025

mansnils added 2 commits February 26, 2025 09:03

Merge remote-tracking branch 'upstream/main' into topics

3cfda9b

Merge remote-tracking branch 'upstream/main' into topics

8115a6c

digantdesai reviewed Feb 26, 2025

View reviewed changes

mansnils added 3 commits February 26, 2025 11:32

Call llama3_2_vision/install_requirements.sh during Arm backend setup

203cea5

Rename test_llama_arm.py to test_llama.py

afbf5e4

Skip test_llama_tosa_MI if no input parameters are given

25dfa11

digantdesai reviewed Feb 27, 2025

View reviewed changes

Update review comments

ec90735

mansnils requested review from kimishpatel and mergennachin March 11, 2025 08:37

Merge branch 'main' into topics

d523b6f

digantdesai approved these changes Mar 18, 2025

View reviewed changes

zingo merged commit 4c54bab into pytorch:main Mar 19, 2025
162 of 163 checks passed

larryliu0820 added a commit that referenced this pull request Mar 20, 2025

Revert "Arm backend: Add initial Llama model test case (#8679)"

59c350b

This reverts commit 4c54bab.

larryliu0820 mentioned this pull request Mar 20, 2025

Revert "Arm backend: Add initial Llama model test case" #9473

Closed

		@@ -0,0 +1,125 @@
		# Copyright (c) Meta Platforms, Inc. and affiliates.

Arm backend: Add initial Llama model test case #8679

Arm backend: Add initial Llama model test case #8679

Uh oh!

Conversation

mansnils commented Feb 25, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8679

❌ 1 Cancelled Job

Uh oh!

mansnils commented Feb 25, 2025

Uh oh!

pytorch-bot bot commented Feb 25, 2025

Uh oh!

mansnils commented Feb 25, 2025

Uh oh!

mansnils commented Feb 25, 2025

Uh oh!

pytorch-bot bot commented Feb 25, 2025

Uh oh!

pytorch-bot bot commented Feb 25, 2025

Uh oh!

mansnils commented Feb 25, 2025

Uh oh!

mansnils commented Feb 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zingo Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zingo commented Feb 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

digantdesai Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mansnils commented Mar 5, 2025

Uh oh!

shoumikhin commented Mar 17, 2025

Uh oh!

kimishpatel commented Mar 17, 2025

Uh oh!

digantdesai commented Mar 18, 2025

Uh oh!

zingo commented Mar 19, 2025

Uh oh!

Uh oh!

kirklandsign commented Mar 20, 2025

Uh oh!

larryliu0820 commented Mar 20, 2025

Uh oh!

kirklandsign commented Mar 20, 2025

Uh oh!

larryliu0820 commented Mar 20, 2025

Uh oh!

Uh oh!

mansnils commented Feb 25, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Feb 25, 2025 •

edited

Loading

zingo Feb 26, 2025 •

edited

Loading

digantdesai Feb 27, 2025 •

edited

Loading