Skip to content

Commit 6a4e729

Browse files
committed
Feedback from @bridadan and @OPpuolitaival
1 parent 25c7d07 commit 6a4e729

File tree

2 files changed

+41
-31
lines changed

2 files changed

+41
-31
lines changed

docs/images/test01.png

-9.48 KB
Binary file not shown.

docs/tools/testing/testing_greentea.md

Lines changed: 41 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
## Greentea testing applications
22

3-
Greentea is the automated testing tool for Arm Mbed OS development. It's a test runner that automates the process of flashing development boards, starting tests, and accumulating test results into test reports. Developers use it for local development as well as for automation in a Continuous Integration environment.
3+
Greentea is the automated testing tool for Arm Mbed OS development. It's a test runner that automates the process of flashing development boards, starting tests, and accumulating test results into test reports. Developers use it for local development as well as for automation in a continuous integration environment.
44

55
Greentea tests run on embedded devices, but Greentea also supports 'host tests'. These are Python scripts that run on a computer and can communicate back to the embedded device. You can for example verify that a value was actually written to the cloud when the device said it did so.
66

@@ -81,9 +81,17 @@ Let's write your first test. Use Mbed CLI to create a new project:
8181
$ mbed new first-greentea-test
8282
```
8383

84-
As specified above, there is a convention where all tests live in the `TESTS/` directory. In the `first-greentea-test` folder create a folder `TESTS/tests/simple-test/`.
84+
As specified above, there is a convention where all tests live in the `TESTS/` directory. In the `first-greentea-test` folder create a folder `TESTS/test-group/simple-test/`.
8585

86-
![Tree structure for Greentea tests](https://s3-us-west-2.amazonaws.com/mbed-os-docs-images/test01.png)
86+
```
87+
first-greentea-test/
88+
└── TESTS/
89+
└── test-group/
90+
└── simple-test/
91+
└── main.cpp
92+
```
93+
94+
*Test structure for Greentea tests*
8795

8896
In this folder, create a file `main.cpp`. In here you can use UNITY, utest and the Greentea Client to write your test:
8997

@@ -95,7 +103,7 @@ In this folder, create a file `main.cpp`. In here you can use UNITY, utest and t
95103

96104
using namespace utest::v1;
97105

98-
// this is how a test looks
106+
// this is how a test case looks
99107
static control_t simple_test(const size_t call_count) {
100108
/* test content here */
101109
TEST_ASSERT_EQUAL(4, 2 * 2);
@@ -104,13 +112,13 @@ static control_t simple_test(const size_t call_count) {
104112
}
105113

106114
utest::v1::status_t greentea_setup(const size_t number_of_cases) {
107-
// here we specify the timeout (60s) and the host runner (the name of our Python file)
108-
GREENTEA_SETUP(1*60, "default_auto");
115+
// here we specify the timeout (60s) and the host test (a built-in host test or the name of our Python file)
116+
GREENTEA_SETUP(60, "default_auto");
109117

110118
return greentea_test_setup_handler(number_of_cases);
111119
}
112120

113-
// list of all cases
121+
// List of test cases in this file
114122
Case cases[] = {
115123
Case("simple test", simple_test)
116124
};
@@ -128,25 +136,25 @@ Let's run this test (tip: to see all tests, run `mbed test --compile-list`):
128136
129137
```
130138
# run the test with the GCC_ARM toolchain, automatically detect the target, and run in verbose mode (-v)
131-
$ mbed test -t GCC_ARM -m auto -v -n tests-tests-simple-test
139+
$ mbed test -t GCC_ARM -m auto -v -n tests-test-group-simple-test
132140
```
133141
134142
This will yield (on a K64F):
135143
136144
```
137145
mbedgt: test suite report:
138-
+--------------+---------------+-------------------------+--------+--------------------+-------------+
139-
| target | platform_name | test suite | result | elapsed_time (sec) | copy_method |
140-
+--------------+---------------+-------------------------+--------+--------------------+-------------+
141-
| K64F-GCC_ARM | K64F | tests-tests-simple-test | OK | 16.84 | default |
142-
+--------------+---------------+-------------------------+--------+--------------------+-------------+
146+
+--------------+---------------+------------------------------+--------+--------------------+-------------+
147+
| target | platform_name | test suite | result | elapsed_time (sec) | copy_method |
148+
+--------------+---------------+------------------------------+--------+--------------------+-------------+
149+
| K64F-GCC_ARM | K64F | tests-test-group-simple-test | OK | 16.84 | default |
150+
+--------------+---------------+------------------------------+--------+--------------------+-------------+
143151
mbedgt: test suite results: 1 OK
144152
mbedgt: test case report:
145-
+--------------+---------------+-------------------------+-------------+--------+--------+--------+--------------------+
146-
| target | platform_name | test suite | test case | passed | failed | result | elapsed_time (sec) |
147-
+--------------+---------------+-------------------------+-------------+--------+--------+--------+--------------------+
148-
| K64F-GCC_ARM | K64F | tests-tests-simple-test | simple test | 1 | 0 | OK | 0.01 |
149-
+--------------+---------------+-------------------------+-------------+--------+--------+--------+--------------------+
153+
+--------------+---------------+------------------------------+-------------+--------+--------+--------+--------------------+
154+
| target | platform_name | test suite | test case | passed | failed | result | elapsed_time (sec) |
155+
+--------------+---------------+------------------------------+-------------+--------+--------+--------+--------------------+
156+
| K64F-GCC_ARM | K64F | tests-test-group-simple-test | simple test | 1 | 0 | OK | 0.01 |
157+
+--------------+---------------+------------------------------+-------------+--------+--------+--------+--------------------+
150158
mbedgt: test case results: 1 OK
151159
mbedgt: completed in 18.64 sec
152160
```
@@ -253,7 +261,7 @@ You see the calls to/from the host through the `greentea_send_kv` and `greentea_
253261
Let's run the test, and see if everything works:
254262
255263
```
256-
$ mbed test -v -n tests-tests-integration-test
264+
$ mbed test -v -n tests-test-group-integration-test
257265
```
258266
259267
### Debugging tests
@@ -320,10 +328,10 @@ This section highlights a few of the capabilities of the Greentea command-line i
320328
321329
#### Listing all tests
322330
323-
You can use the `-l` argument to list all available tests:
331+
You can use the `--compile-list` argument to list all available tests:
324332
325333
```
326-
$ mbed test -l
334+
$ mbed test --compile-list
327335
[mbed] Working path "/Users/janjon01/repos/first-greentea-test" (program)
328336
Test Case:
329337
Name: mbed-os-components-storage-blockdevice-component_flashiap-tests-filesystem-fopen
@@ -335,6 +343,8 @@ Test Case:
335343
...
336344
```
337345
346+
After compilation you can use the `--run-list` argument to list all tests that are ready to be ran.
347+
338348
#### Executing all tests
339349
340350
The default action of Greentea using `mbed test` is to execute all tests that are found. You can also add `-v` to make the output more verbose.
@@ -347,10 +357,10 @@ You can select test cases by name using the `-n` argument. This command executes
347357
$ mbed test -n tests-mbedmicro-rtos-mbed-mail
348358
```
349359
350-
When using the `-n` argument, you can use the `*` character at the end of a test name to match all tests that share a prefix. This command executes all tests that start with `tests-mbedmicro-rtos`:
360+
When using the `-n` argument, you can use the `*` character as a wildcard. This command executes all tests that start with `tests-` and have `-rtos-` in them.
351361
352362
```
353-
$ mbed test -n tests-mbedmicro-rtos-*
363+
$ mbed test -n tests-*-rtos-*
354364
```
355365
356366
You can use a comma (`,`) to separate test names (argument `-n`) and build names (argument `-t`). This command executes the tests `tests-mbedmicro-rtos-mbed-mail` and `tests-mbed_drivers-c_strings` for the `K64F-ARM` and `K64F-GCC_ARM` builds in the test specification:
@@ -364,15 +374,15 @@ $ mbed test -n tests-mbedmicro-rtos-mbed-mail,tests-mbed_drivers-c_strings -t K6
364374
You can limit which boards Greentea uses for testing by using the `--use-tids` argument.
365375
366376
```
367-
$ mbed test --use-tids 02400203C3423E603EBEC3D8,024002031E031E6AE3FFE3D2
377+
$ mbed test --use-tids 02400203C3423E603EBEC3D8,024002031E031E6AE3FFE3D2 --run
368378
```
369379
370380
Where `02400203C3423E603EBEC3D8` and `024002031E031E6AE3FFE3D` are the target IDs of platforms attached to your system.
371381
372-
You can view target IDs using Mbed CLI:
382+
You can view target IDs using [mbed-ls](https://github.com/ARMmbed/mbed-os-tools/tree/master/packages/mbed-ls), which is installed as part of Mbed CLI.
373383
374384
```
375-
$ mbed detect
385+
$ mbedls
376386
+--------------+---------------------+------------+------------+-------------------------+
377387
|platform_name |platform_name_unique |mount_point |serial_port |target_id |
378388
+--------------+---------------------+------------+------------+-------------------------+
@@ -393,31 +403,31 @@ Greentea supports a number of report formats.
393403
This creates an interactive HTML page with test results and logs.
394404
395405
```
396-
mbed test --report-html html_report.html
406+
mbed test --report-html html_report.html --run
397407
```
398408
399409
##### JUnit
400410
401411
This creates an XML JUnit report, which you can use with popular Continuous Integration software, such as [Jenkins](https://jenkins.io/index.html).
402412
403413
```
404-
mbed test --report-junit junit_report.xml
414+
mbed test --report-junit junit_report.xml --run
405415
```
406416
407417
##### JSON
408418
409419
This creates a general JSON report.
410420
411421
```
412-
mbed test --report-json json_report.json
422+
mbed test --report-json json_report.json --run
413423
```
414424
415425
##### Plain text
416426
417427
This creates a human-friendly text summary of the test run.
418428
419429
```
420-
mbed test --report-text text_report.text
430+
mbed test --report-text text_report.text --run
421431
```
422432
423433
### Test specification JSON format
@@ -484,7 +494,7 @@ Place this file in your root folder and name it `test_spec.json`.
484494
}
485495
```
486496

487-
If you now run `mbed test -l` this will now list only these tests:
497+
If you now run `mbed test --run-list` this will now list only these tests:
488498

489499
```
490500
mbedgt: greentea test automation tool ver. 1.2.5

0 commit comments

Comments
 (0)