Rework the Greentea docs #927

janjongboom · 2019-01-21T21:00:39Z

@iriark01 @ARMmbed/mbed-docs As per our discussion on e-mail.

This PR:

Consolidates the two Greentea documents that we had in the tests directory.
Updates all commands to use mbed test instead of the mbedgt command.
Adds examples on how to run the tests.
Adds host test runner documentation.

…amples, move everything into a single document

theotherjimmy

Please prefer to use Mbed CLI over the tools shipped with dependent packages.

theotherjimmy · 2019-01-22T15:20:31Z

docs/tools/testing/testing_greentea.md


-#### Writing tests
+![Exporting to Make](https://s3-us-west-2.amazonaws.com/mbed-os-docs-images/test01.png)


We should promote working with Mbed CLI within it's own documentation where possible. Either this is titled incorrectly, or it should be removed (I can't see the image content)

Hmm, I think I copied something wrong there. The description just shows the tree structure, nothing to do with make. My bad :-)

theotherjimmy · 2019-01-22T15:25:43Z

docs/tools/testing/testing_greentea.md

+
+Where `02400203C3423E603EBEC3D8` and `024002031E031E6AE3FFE3D` are the target IDs of platforms attached to your system.
+
+You can view target IDs using the [`mbed-ls`](https://github.com/ARMmbed/mbed-os-tools/tree/master/packages/mbed-ls) tool (installed with Greentea).


Please use mbed detect instead.

Updated. Copied it from old content but missed this.

bridadan

I really like the format you've put forward here, I think it's definitely an improvement.

The CLI for mbed test is at best described as "finicky", I'm actively working to make it better. I've commented on some of the commands below and some issues I foresee.

bridadan · 2019-01-22T23:15:53Z

docs/tools/testing/testing_greentea.md


-The Greentea tool handles the actual testing process. To read more about this tool, please visit its [GitHub repository](https://github.com/ARMmbed/mbed-os-tools/tree/master/packages/mbed-greentea).
+As specified above, there is a convention where all tests live in the `TESTS/` directory. In the `first-greentea-test` folder create a folder `TESTS/tests/simple-test/`.


I'd suggest the following example folder structure since the example kind of looks like a duplication (double "tests" folders):

Suggested change

As specified above, there is a convention where all tests live in the `TESTS/` directory. In the `first-greentea-test` folder create a folder `TESTS/tests/simple-test/`.

As specified above, there is a convention where all tests live in the `TESTS/` directory. In the `first-greentea-test` folder create a folder `TESTS/test-group/simple-test/`.

bridadan · 2019-01-22T23:18:11Z

docs/tools/testing/testing_greentea.md


-#### Writing tests
+![Tree structure for Greentea tests](https://s3-us-west-2.amazonaws.com/mbed-os-docs-images/test01.png)


Do we usually host these on s3? Perhaps just using an text representation of the structure would be easier. Ex:

project/ TESTS/ test-group/ simple-test/ main.cpp

AccessDenied: https://s3-us-west-2.amazonaws.com/mbed-os-docs-images/test01.png

Yeah, the image is in this pr, but not uploaded to s3 yet. @bridadan Good suggestion, I'll update it to use that.

bridadan · 2019-01-22T23:19:08Z

docs/tools/testing/testing_greentea.md

-void test_success() {
-    TEST_ASSERT(true);
-}
+// this is how a test looks


Nit: differentiate between "test" with "test case"

Suggested change

// this is how a test looks

// this is how a test case looks

bridadan · 2019-01-22T23:20:32Z

docs/tools/testing/testing_greentea.md

-    GREENTEA_SETUP(40, "default_auto");
-    return verbose_test_setup_handler(number_of_cases);
+utest::v1::status_t greentea_setup(const size_t number_of_cases) {
+    // here we specify the timeout (60s) and the host runner (the name of our Python file)


Nit: default_auto is currently provided by htrun, not by a python file within mbed-os.

Suggested change

// here we specify the timeout (60s) and the host runner (the name of our Python file)

// here we specify the timeout (60s) and the host test (a built-in host test or the name of our Python file)

bridadan · 2019-01-22T23:20:58Z

docs/tools/testing/testing_greentea.md

-    return verbose_test_setup_handler(number_of_cases);
+utest::v1::status_t greentea_setup(const size_t number_of_cases) {
+    // here we specify the timeout (60s) and the host runner (the name of our Python file)
+    GREENTEA_SETUP(1*60, "default_auto");


Nit: Ok to drop the multiplication here?

Suggested change

GREENTEA_SETUP(1*60, "default_auto");

GREENTEA_SETUP(60, "default_auto");

bridadan · 2019-01-23T00:21:34Z

docs/tools/testing/testing_greentea.md

+$ mbed test -n tests-mbedmicro-rtos-mbed-mail
+```
+
+When using the `-n` argument, you can use the `*` character at the end of a test name to match all tests that share a prefix. This command executes all tests that start with `tests-mbedmicro-rtos`:


This will now use "globbing" rules, so you can actually sprinkle the * characters any where you like.

bridadan · 2019-01-23T00:23:24Z

docs/tools/testing/testing_greentea.md

+You can limit which boards Greentea uses for testing by using the `--use-tids` argument.
+
+```
+$ mbed test --use-tids 02400203C3423E603EBEC3D8,024002031E031E6AE3FFE3D2


This will currently error unless you add --run.

bridadan · 2019-01-23T00:26:44Z

docs/tools/testing/testing_greentea.md

+|K64F          |K64F[0]              |E:          |COM160      |024002031E031E6AE3FFE3D2 |
+|K64F          |K64F[1]              |F:          |COM162      |02400203C3423E603EBEC3D8 |
+|LPC1768       |LPC1768[0]           |G:          |COM5        |1010ac87cfc4f23c4c57438d |
+--------------+---------------------+------------+------------+-------------------------+


If your current working directly is within an Mbed OS project, the output of mbed detect looks like this:

mbed detect [mbed] Working path "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os" (program) [mbed] Detected K64F, port COM79, mounted F:, interface version 0250: [mbed] Supported toolchains for K64F | Target | mbed OS 2 | mbed OS 5 | uARM | IAR | ARM | GCC_ARM | |---------------|-----------|-----------|------|-----------|-----------|-----------| | K64F | Supported | Supported | - | Supported | Supported | Supported | | RAPIDIOT_K64F | Supported | Supported | - | Supported | Supported | Supported | Supported targets: 2 [mbed] Detected K64F, port COM78, mounted D:, interface version 0251: [mbed] Supported toolchains for K64F | Target | mbed OS 2 | mbed OS 5 | uARM | IAR | ARM | GCC_ARM | |---------------|-----------|-----------|------|-----------|-----------|-----------| | K64F | Supported | Supported | - | Supported | Supported | Supported | | RAPIDIOT_K64F | Supported | Supported | - | Supported | Supported | Supported | Supported targets: 2

Which unfortunately does not list the target id. mbedls will but I know we want to direct folks to use Mbed CLI commands. In this case I don't think we have many options here, any ideas @theotherjimmy?

Oh, huh. Not only does mbed detect omit the information, it also neglects to use unique names. I suppose we'll have to use mbedls until we fix this in mbed detect.

bridadan · 2019-01-23T00:28:19Z

docs/tools/testing/testing_greentea.md

+This creates a human-friendly text summary of the test run.
+
+```
+mbed test --report-text text_report.text


All of these report commands would need the --run command added first, otherwise it would throw an exception:

$ mbed test -n tests-mbed_functional-callback -v --report-json report.json [mbed] Working path "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os" (program) [mbed] Exec "c:\python27\python.exe -u C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\test.py -t GCC_ARM -m K64F --source . --build .\BUILD\tests\K64F\GCC_ARM --test-spec .\BUILD\tests\K64F\GCC_ARM\test_spec.json --build-data .\BUILD\tests\K64F\GCC_ARM\build_data.json -n tests-mbed_functional-callback -v --greentea --report-json report.json" in "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os" usage: test.py [-h] [-m MCU] [--custom-targets CUSTOM_TARGETS_DIRECTORY] [-t TOOLCHAIN] [--color] [--cflags CFLAGS] [--asmflags ASMFLAGS] [--ldflags LDFLAGS] [-c] [--profile PROFILE] [--app-config APP_CONFIG] [-D MACROS] [-j JOBS] [--source SOURCE_DIR] [--build BUILD_DIR] [-l] [-p PATHS] [-f FORMAT] [--continue-on-build-fail] [-n NAMES] [--test-config TEST_CONFIG] [--test-spec TEST_SPEC] [--build-report-junit BUILD_REPORT_JUNIT] [--build-data BUILD_DATA] [-v] [--stats-depth STATS_DEPTH] [--ignore IGNORE] [--icetea] [--greentea] test.py: error: unrecognized arguments: --report-json report.json [mbed] ERROR: "c:\python27\python.exe" returned error. Code: 2 Path: "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os" Command: "c:\python27\python.exe -u C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\test.py -t GCC_ARM -m K64F --source . --build .\BUILD\tests\K64F\GCC_ARM --test-spec .\BUILD\tests\K64F\GCC_ARM\test_spec.json --build-data .\BUILD\tests\K64F\GCC_ARM\build_data.json -n tests-mbed_functional-callback -v --greentea --report-json report.json" Tip: You could retry the last command with "-v" flag for verbose output ---

bridadan · 2019-01-23T00:29:23Z

docs/tools/testing/testing_greentea.md

+}
+```
+
+If you now run `mbed test -l` this will now list only these tests:


You want to use the --run-list option here as well.

Suggested change

If you now run `mbed test -l` this will now list only these tests:

If you now run `mbed test --run-list` this will now list only these tests:

OPpuolitaival · 2019-01-23T10:17:42Z

docs/tools/testing/testing_greentea.md


-#### Writing tests
+![Tree structure for Greentea tests](https://s3-us-west-2.amazonaws.com/mbed-os-docs-images/test01.png)


AccessDenied: https://s3-us-west-2.amazonaws.com/mbed-os-docs-images/test01.png

OPpuolitaival · 2019-01-23T10:18:44Z

docs/tools/testing/testing_greentea.md

@@ -1,6 +1,10 @@
 ## Greentea testing applications

-The way tests are run and compiled in Arm Mbed OS 5 is substantially different from previous versions of Mbed.
+Greentea is the automated testing tool for Arm Mbed OS development. It's a test runner that automates the process of flashing development boards, starting tests, and accumulating test results into test reports. Developers use it for local development as well as for automation in a Continuous Integration environment.


Is there some reason to use capital first letters for "Continous Integration"?

OPpuolitaival · 2019-01-23T10:19:49Z

docs/tools/testing/testing_greentea.md

 }

-// Test cases
+// list of all cases


This is listing test cases which are contained in this file and there are multiple files. "all" sounds me a bit more than one file tests

janjongboom · 2019-01-24T19:35:10Z

@bridadan @OPpuolitaival @ARMmbed/mbed-docs Updated based on feedback.

bridadan

Definitely looking really good, thanks for making those changes. Just a few more comments below.

bridadan · 2019-01-24T20:19:43Z

docs/tools/testing/testing_greentea.md

+This creates a human-friendly text summary of the test run.
+
+```
+mbed test --report-text text_report.text --run


Thinking about this more, none of the report arguments are currently part of the help options in Mbed CLI. This only works because Mbed CLI passes along arguments to the underlying scripts when it doesn't recognize it. How do you feel about removing this reporting section altogether?

bridadan · 2019-01-24T20:20:19Z

docs/tools/testing/testing_greentea.md

+You can limit which boards Greentea uses for testing by using the `--use-tids` argument.
+
+```
+$ mbed test --use-tids 02400203C3423E603EBEC3D8,024002031E031E6AE3FFE3D2 --run


Same thing that I mentioned below about the reporting section, this isn't in the mbed test --help text. How do you feel about removing this?

Well, isn't it great that our docs are more complete than our CLI?

bridadan · 2019-01-24T20:20:45Z

docs/tools/testing/testing_greentea.md

+
+utest::v1::status_t greentea_setup(const size_t number_of_cases) {
+   // here we specify the timeout (60s) and the host runner (the name of our Python file)
+   GREENTEA_SETUP(1*60, "hello_world_tests");


One other 1*60 that can be removed

OPpuolitaival

Looks good

janjongboom · 2019-02-04T19:28:16Z

@bridadan @ARMmbed/mbed-docs Is this good to go now?

janjongboom · 2019-02-08T18:07:34Z

@ARMmbed/mbed-docs Ping.

Edit file, mostly for consistent person and voice across docs.

AnotherButler · 2019-02-08T21:41:12Z

docs/tools/testing/testing_greentea.md

+$ mbed test -t GCC_ARM -m auto -v -n tests-test-group-simple-test
+```
+
+This yields (on a K64F):


Can we please use a different board for our examples?

janjongboom · 2019-02-15T10:56:47Z

@AnotherButler Fixed

…-5-docs into greentea-test-update

AnotherButler · 2019-02-19T16:33:49Z

@theotherjimmy Have your concerns been addressed?

janjongboom · 2019-02-25T09:46:07Z

@theotherjimmy @ARMmbed/mbed-docs ping again.

Addressed in comments.

janjongboom · 2019-02-26T08:45:41Z

@AnotherButler @ARMmbed/mbed-docs Who can merge this?

AnotherButler · 2019-02-26T20:15:27Z

@theotherjimmy I'm going to merge this today or tomorrow. If you have any further concerns, please create a new PR.

janjongboom · 2019-03-07T11:13:59Z

@AnotherButler @ARMmbed/mbed-docs When will this be available?

iriark01 · 2019-03-07T11:19:41Z

5.12 at the end of the month

Apply changes from PR #927 to 5.11.

AnotherButler · 2019-03-08T02:26:25Z

I just applied your changes to 5.11. Let me know if they haven't built by tomorrow.

janjongboom · 2019-03-08T08:42:35Z

@AnotherButler Awesome, thanks.

Rework the Greentea docs, remove obsolete commands, add end-to-end ex…

45f7c9d

…amples, move everything into a single document

janjongboom force-pushed the greentea-test-update branch from cfbd5ed to 45f7c9d Compare January 21, 2019 21:01

iriark01 requested review from theotherjimmy, bridadan and AnotherButler January 22, 2019 13:50

theotherjimmy previously requested changes Jan 22, 2019

View reviewed changes

AnotherButler requested a review from OPpuolitaival January 22, 2019 19:55

Feedback from @theotherjimmy

25c7d07

bridadan suggested changes Jan 23, 2019

View reviewed changes

OPpuolitaival approved these changes Jan 23, 2019

View reviewed changes

Feedback from @bridadan and @OPpuolitaival

6a4e729

bridadan reviewed Jan 24, 2019

View reviewed changes

OPpuolitaival approved these changes Jan 25, 2019

View reviewed changes

1*60 => 60

b44b236

bridadan approved these changes Feb 4, 2019

View reviewed changes

Edit testing_greentea.md

3ae40d2

Edit file, mostly for consistent person and voice across docs.

AnotherButler reviewed Feb 8, 2019

View reviewed changes

Replace K64F with NUCLEO_F411RE in the test runner example

2545e82

Merge branch 'greentea-test-update' of github.com:janjongboom/mbed-os…

c85f64b

…-5-docs into greentea-test-update

AnotherButler approved these changes Feb 25, 2019

View reviewed changes

AnotherButler merged commit 25600c9 into ARMmbed:development Feb 26, 2019

AnotherButler pushed a commit that referenced this pull request Mar 8, 2019

Make content changes to testing_greentea.md

59227ea

Apply changes from PR #927 to 5.11.

AnotherButler pushed a commit that referenced this pull request Mar 8, 2019

Delete greentea.md

2a2c467

Apply changes from PR #927 to 5.11.


		#### Writing tests
		![Exporting to Make](https://s3-us-west-2.amazonaws.com/mbed-os-docs-images/test01.png)


		Where `02400203C3423E603EBEC3D8` and `024002031E031E6AE3FFE3D` are the target IDs of platforms attached to your system.

		You can view target IDs using the [`mbed-ls`](https://github.com/ARMmbed/mbed-os-tools/tree/master/packages/mbed-ls) tool (installed with Greentea).


		The Greentea tool handles the actual testing process. To read more about this tool, please visit its [GitHub repository](https://github.com/ARMmbed/mbed-os-tools/tree/master/packages/mbed-greentea).
		As specified above, there is a convention where all tests live in the `TESTS/` directory. In the `first-greentea-test` folder create a folder `TESTS/tests/simple-test/`.


		#### Writing tests
		![Tree structure for Greentea tests](https://s3-us-west-2.amazonaws.com/mbed-os-docs-images/test01.png)

	// here we specify the timeout (60s) and the host runner (the name of our Python file)
	// here we specify the timeout (60s) and the host test (a built-in host test or the name of our Python file)

	GREENTEA_SETUP(1*60, "default_auto");
	GREENTEA_SETUP(60, "default_auto");

	If you now run `mbed test -l` this will now list only these tests:
	If you now run `mbed test --run-list` this will now list only these tests:

Rework the Greentea docs #927

Rework the Greentea docs #927

Uh oh!

Conversation

janjongboom commented Jan 21, 2019

Uh oh!

theotherjimmy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bridadan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janjongboom commented Jan 24, 2019

Uh oh!

bridadan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OPpuolitaival left a comment

Choose a reason for hiding this comment

Uh oh!

janjongboom commented Feb 4, 2019

Uh oh!

janjongboom commented Feb 8, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janjongboom commented Feb 15, 2019

Uh oh!

AnotherButler commented Feb 19, 2019

Uh oh!

janjongboom commented Feb 25, 2019

Uh oh!