Skip to content

Icetea added #659

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 51 commits into from
Sep 4, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
2132ceb
Icetea added
May 30, 2018
fa8de42
Copy edit testing_intro.md
Aug 13, 2018
5bdfb85
Complete initial copy edits of testing_icetea.md
Aug 13, 2018
df471aa
Copy edit icetea.md
Aug 20, 2018
8988382
Updated link, will work after the integration PR is merged
Aug 22, 2018
e1e3903
Fix instruction for running MAC tester testcases in driver creation g…
Aug 23, 2018
9d0c80f
Copy edit MeshInterface.md
Aug 24, 2018
97bee7d
Remove icetea.md because not ready
Aug 27, 2018
62958a8
Add icetea.md content
Aug 28, 2018
fcc26d1
Fixes according to comments
Aug 28, 2018
f0eab0a
Update cli-test-debug.md
jen3andruska Aug 28, 2018
f746f54
Update cli-test-debug.md
jen3andruska Aug 28, 2018
781f4e7
Update cli-test-debug.md
jen3andruska Aug 28, 2018
87a423d
Update cli-test-debug.md
jen3andruska Aug 28, 2018
bcc22cf
Change cli-test-debug according to comments
Aug 28, 2018
5b2cb15
Update cli-test-debug.md
jen3andruska Aug 28, 2018
42e2015
Update cli-test-debug.md
jen3andruska Aug 28, 2018
5db3026
Update cli-test-debug.md
jen3andruska Aug 29, 2018
dde79f8
Update icetea.md
jen3andruska Aug 29, 2018
f34bd81
Update icetea.md
jen3andruska Aug 29, 2018
a08ee50
Update icetea.md
jen3andruska Aug 29, 2018
e979e11
Update icetea.md
jen3andruska Aug 29, 2018
94997d1
Updating Icetea.md.
Aug 29, 2018
48737b4
Updating Icetea.md
Aug 29, 2018
495d484
Updating Icetea.md
Aug 29, 2018
f102a1b
Update icetea.md
jen3andruska Aug 29, 2018
61cab4c
Update cli-test-debug.md
Aug 29, 2018
622ff0f
Update icetea.md
jen3andruska Aug 29, 2018
ff04613
Update icetea.md
jen3andruska Aug 29, 2018
46897ac
Update icetea.md
jen3andruska Aug 29, 2018
881ccbd
Update icetea.md
jen3andruska Aug 29, 2018
f2b6db3
Update icetea.md
jen3andruska Aug 29, 2018
59a858f
update cli-test-debug.md
Aug 29, 2018
95a9aac
Update testing_icetea.md
Aug 29, 2018
8d1b6a3
Updated Icetea.md
Aug 29, 2018
ef3e502
Update testing_greentea.md
jen3andruska Aug 30, 2018
03b9602
Update testing_greentea.md
jen3andruska Aug 30, 2018
f329c00
Update testing_greentea.md
jen3andruska Aug 30, 2018
60908f4
Update testing_greentea.md
jen3andruska Aug 30, 2018
5f59df6
Update testing_greentea.md
jen3andruska Aug 30, 2018
8c973fa
Update testing_greentea.md
jen3andruska Aug 30, 2018
d7ebf45
Update testing_greentea.md
jen3andruska Aug 30, 2018
e0d4907
Update testing_greentea.md
jen3andruska Aug 30, 2018
90ef9e3
Update testing_greentea.md
jen3andruska Aug 30, 2018
0e826d8
Update testing_greentea.md
jen3andruska Aug 30, 2018
2c47f44
Update testing_icetea.md
jen3andruska Aug 30, 2018
12bf038
Update testing_icetea.md
jen3andruska Aug 30, 2018
e05475e
Copy edit icetea.md
Aug 31, 2018
8411229
Copy edit cli-test-debug.md
Aug 31, 2018
f2a2532
Make changes from comments in icetea.md
Aug 31, 2018
2bc593c
Make changes from comments to cli-test-debug.md
Aug 31, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -815,11 +815,11 @@
},
{
"type": "markdown",
"url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/testing.md"
"url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/testing_greentea.md"
},
{
"type": "markdown",
"url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/greentea.md"
"url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/testing_icetea.md"
},
{
"type": "markdown",
Expand Down
4 changes: 3 additions & 1 deletion docs/reference/contributing/connectivity/MeshInterface.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,9 @@ The following steps describe how you can create a new RF driver:

1. Check with an RF sniffer tool that you can see RF packets transmitted when you start your device. The 6LoWPAN bootstrap should start with IEEE 802.15.4 Beacon Request packets.

1. Verify the functionality of your implementation using the [Nanostack RF driver test application](https://github.com/ARMmbed/nanostack-rf-driver-tester). (This is currently only available to Mbed OS Partners.)
1. Verify the functionality of your implementation by running the Nanostack RF driver testcase set in the Mbed OS repository:

`mbed test --clean --icetea -t <toolchain> -m <platform> --test-config MAC_TESTER -n address_read_and_write,send_data,send_data_indirect,send_large_payloads,create_and_join_PAN,ED_scan`

### Worker thread for Mbed OS

Expand Down
107 changes: 80 additions & 27 deletions docs/tools/offline/cli-test-debug.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,31 @@

Use the `mbed test` command to compile and run tests.

There are two testing frameworks: Greentea and Icetea. Greentea provides tests designed for driver porting and target verification. Icetea provides and manages tests for multiple devices at the same time. For example, you can test the network setup for a server and multiple clients, simultaneously controlling them from the test environment.

The arguments to `test` are:
* `-m <MCU>` to select a target for the compilation. If `detect` or `auto` parameter is passed, then Mbed CLI will attempt to detect the connected target and compile against it.
* `-t <TOOLCHAIN>` to select a toolchain (of those defined in `mbed_settings.py`, see above), where `toolchain` can be either `ARM` (Arm Compiler 5), `GCC_ARM` (GNU Arm Embedded), or `IAR` (IAR Embedded Workbench for Arm).
* `--compile-list` to list all the tests that can be built.
* `--run-list` to list all the tests that can be run (they must be built first).
* `--compile` to only compile the tests.
* `--run` to only run the tests.
* `-n <TESTS_BY_NAME>` to limit the tests built or run to a comma separated list (ex. test1,test2,test3).
* `--source <SOURCE>` to select the source directory. Default is `.` (the current directory). You can specify multiple source locations, even outside the program tree.
* `--build <BUILD>` to select the build directory. Default: `BUILD/` inside your program.
* `--profile <PATH_TO_BUILD_PROFILE>` to select a path to a build profile configuration file. Example: `mbed-os/tools/profiles/debug.json`.
* `-c or --clean` to clean the build directory before compiling.
* `--test-spec <TEST_SPEC>` to set the path for the test spec file used when building and running tests (the default path is the build directory).
* `-v` or `--verbose` for verbose diagnostic output.
* `-vv` or `--very_verbose` for very verbose diagnostic output.

Invoke `mbed test`:

- `-m <MCU>`: to select a target for the compilation. If the `detect` or `auto` parameter is passed, then Mbed CLI will attempt to detect the connected target and compile against it.
- `-t <TOOLCHAIN>`: to select a toolchain from those defined in `mbed_settings.py`, where `toolchain` can either be `ARM` (Arm Compiler 5), `GCC_ARM` (GNU Arm Embedded), or `IAR` (IAR Embedded Workbench for Arm).
- `--compile-list`: to list all the tests that can be built.
- `--run-list`: to list all the tests that can be run, after they have been built.
- `--compile`: to only compile the tests.
- `--run`: to only run the tests.
- `-n <TESTS_BY_NAME>`: to limit the tests built or run to a comma separated list, for example, `test1, test2, test3`.
- `--source <SOURCE>`: to select the source directory. The default is `.` for the the current directory. You can specify multiple source locations, even outside the program tree.
- `--build <BUILD>`: to select the build directory. The default is `BUILD/` inside your program.
- `--profile <PATH_TO_BUILD_PROFILE>`: to select a path to a build profile configuration file, for example, `mbed-os/tools/profiles/debug.json`.
- `-c or --clean`: to clean the build directory before compiling.
- `--test-spec <TEST_SPEC>`: to set the path for the test specification file used when building and running tests. The default path is the build directory.
- `--build-data <BUILD_DATA>`: dumps build_data to this file.
- `--app-config <APP_CONFIG>`: the path of an app configuration file. The default is to look for `mbed_app.json`.
- `--test-config <TEST_CONFIG>`: the path or Mbed OS keyword of a test configuration file, for example, `ethernet`, `odin_wifi` or `path/to/config.json`.
- `--greentea`: to run Greentea tests. As a default, it only runs Greentea tests.
- `--icetea`: to run Icetea tests. If used without the `--greentea` flag, then it only runs Icetea tests.
- `-v` or `--verbose`: for verbose diagnostic output.
- `-vv` or `--very_verbose`: for very verbose diagnostic output.

To invoke the `mbed test`:

```
$ mbed test -m K64F -t GCC_ARM
Expand Down Expand Up @@ -79,6 +87,33 @@ Test Case:
Path: .\TESTS\functional\test3
```

For Icetea:

```
$ mbed test -m K64F -t GCC_ARM --icetea --compile-list
Available Icetea tests for build 'K64F-GCC_ARM', location 'TEST_APPS'
Test Case:
Name: test_cmdline
Path: ./TEST_APPS/testcases/example/test_cmdline.py
Test applications: ./TEST_APPS/device/exampleapp
Test Case:
Name: UDPSOCKET_BIND_PORT
Path: ./TEST_APPS/testcases/netsocket/SOCKET_BIND_PORT.py
Test applications: ./TEST_APPS/device/socket_app
Test Case:
Name: TCPSOCKET_BIND_PORT
Path: ./TEST_APPS/testcases/netsocket/SOCKET_BIND_PORT.py
Test applications: ./TEST_APPS/device/socket_app
Test Case:
Name: TCPSERVER_ACCEPT
Path: ./TEST_APPS/testcases/netsocket/TCPSERVER_ACCEPT.py
Test applications: ./TEST_APPS/device/socket_app
Test Case:
Name: TCPSOCKET_ECHOTEST_BURST_SHORT
Path: ./TEST_APPS/testcases/netsocket/TCPSOCKET_ECHOTEST_BURST_SHORT.py
Test applications: ./TEST_APPS/device/socket_app
```

You can find the tests that are available for **running** by using the `--run-list` option:

```
Expand All @@ -91,21 +126,38 @@ mbedgt: available tests for built 'K64F-ARM', location '.\build\tests\K64F\ARM'
test 'TESTS-functional-test3'
```

For Icetea:

```
$ mbed test -m K64F -t GCC_ARM --icetea --run-list
Available Icetea tests for build 'K64F-GCC_ARM', location 'TEST_APPS'
test 'UDPSOCKET_BIND_PORT'
test 'TCPSOCKET_BIND_PORT'
test 'TCPSERVER_ACCEPT'
test 'TCPSOCKET_ECHOTEST_BURST_SHORT'
```

### Compiling and running tests

You can specify to only **build** the tests by using the `--compile` option:
You can specify that the tests only **build** by using the `--compile` option:

```
$ mbed test -m K64F -t GCC_ARM --compile
```

You can specify to only **run** the tests by using the `--run` option:
For Icetea, only the test applications are built:

```
$ mbed test -m K64F -t GCC_ARM --compile --icetea
```

You can specify that the tests only **run** by using the `--run` option:

```
$ mbed test -m K64F -t GCC_ARM --run
```

If you don't specify any of these, `mbed test` will first compile all available tests and then run them.
If you don't specify any of these, `mbed test` first compiles all available tests and then runs them.

### Limiting the test scope

Expand All @@ -115,7 +167,7 @@ You can limit the scope of the tests built and run by using the `-n` option. Thi
$ mbed test -m K64F -t GCC_ARM -n TESTS-functional-test1,TESTS-functional-test2
```

You can use the wildcard character `*` to run a group of tests that share a common prefix without specifying each test individually. For instance, if you only want to run the three tests `TESTS-functional-test1`, `TESTS-functional-test2` and `TESTS-functional-test3`, but you have other tests in your project, you can run:
You can use the wildcard character `*` to run a group of tests that share a common prefix without specifying each test individually. For instance, if you only want to run the three tests, `TESTS-functional-test1`, `TESTS-functional-test2` and `TESTS-functional-test3`, but you have other tests in your project, you can run:

```
$ mbed test -m NUCLEO_F429ZI -t GCC_ARM -n TESTS-functional*
Expand Down Expand Up @@ -154,16 +206,17 @@ mbed-os-program
| ....
```

As shown above, tests exist inside `TESTS\testgroup\testcase\` directories. Please note that `TESTS` is a special upper case directory that is excluded from module sources while compiling.
As shown above, tests exist inside `TESTS\testgroup\testcase\` directories. Please note that `TESTS` is a special upper-case directory that is excluded from module sources while compiling.

<span class="notes">**Note:** `mbed test` does not work in applications that contain a `main` function that is outside of a `TESTS` directory.</span>
<span class="notes">**Note:** `mbed test` does not work in applications that contain a `main` function that is outside of a `TESTS` directory.</span>

### Troubleshooting

#### Unable to import Mercurial (mbed.org) programs or libraries.
1. Check whether you have Mercurial installed in your system path by running `hg` in command prompt. If you're receiving "command not found" or a similar message, then you need to install Mercurial, and add it to your system path.
#### Import Mercurial (mbed.org) programs or libraries

1. Check whether you have Mercurial installed in your system path by running `hg` in the command prompt. If you are receiving "command not found" or a similar message, then you need to install Mercurial and add it to your system path.
1. Try to clone a Mercurial repository directly. For example, `hg clone https://developer.mbed.org/teams/mbed/code/mbed_blinky/`. If you receive an error similar to `abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.:590)`, then your system certificates are out of date. You need to update your system certificates and possibly add the host certificate fingerprint of `mbed.com` and `mbed.org`. You can read more about Mercurial's [certificate management](https://www.mercurial-scm.org/wiki/CACertificates).

2. Try to clone a Mercurial repository directly. For example, `hg clone https://developer.mbed.org/teams/mbed/code/mbed_blinky/`. If you receive an error similar to `abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.:590)`, then your system certificates are out of date. You need to update your system certificates and possibly add the host certificate fingerprint of `mbed.com` and `mbed.org`. You can read more about Mercurial's [certificate management](https://www.mercurial-scm.org/wiki/CACertificates).
#### Various issues when running Mbed CLI in the Cygwin environment

#### Various issues when running Mbed CLI in Cygwin environment
Currently Mbed CLI is not compatible with Cygwin environment and [cannot be executed inside it](https://github.com/ARMmbed/mbed-cli/issues/299).
Mbed CLI is not currently compatible with the Cygwin environment and [cannot be executed inside it](https://github.com/ARMmbed/mbed-cli/issues/299).
166 changes: 166 additions & 0 deletions docs/tools/testing/icetea.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
<h2 id="icetea">Icetea test framework</h2>

Icetea is an automated testing framework for Mbed development. It automates the process of flashing Mbed boards, running tests and accumulating test results into reports. Developers use it for local development, as well as automation in a Continuous Integration environment.

When testing Mbed OS, Icetea allows you to execute commands remotely by using the command-line interface in a device under test (DUT). The interface between the test framework and a DUT might be, for example, UART or stdio.

More detailed documentation on the tool is available [in rst format](https://github.com/ARMmbed/icetea/tree/master/doc-source) and [in markdown format](https://github.com/ARMmbed/icetea/tree/master/doc).

### Prerequisites

Icetea supports Linux (Ubuntu preferred), Windows and OS X. Our main target is Linux.

You need `pip` to install Icetea.

Icetea supports both Python 2.7 and 3.5, or later. Some OS specific prerequisites are listed below:

- Linux.
- python-dev and python-lxml:
`sudo apt-get install python-dev python-lxml`
- To run test cases with hardware in Linux, without sudo rights:

```
sudo usermod -a -G dialout username
Log out & log in back to Linux
```

This command adds the user `username` to the `dialout` group and grants the required permissions to the USB ports.

- OS X.
- [XCode developer tools](http://osxdaily.com/2014/02/12/install-command-line-tools-mac-os-x/).
- [MacPorts](https://www.macports.org/install.php).
- [lxml](http://lxml.de/installation.html#installation):
`STATIC_DEPS=true sudo pip install lxml`

- Windows
- python-lxml installation on Windows usually requires build tools. You can, however, install it from prebuilt binaries.
- Search the internet for a binary for your system.
- Navigate to the directory where you downloaded the binary, and install it with `pip install <insert_file_name>`
- You can also follow [these instructions](http://lxml.de/installation.html#installation) instead.

#### Optional

If you wish to decorate your console log with colors, install the `coloredlogs` module by using pip: `pip install coloredlogs`. There have been issues with `coloredlogs` installation on Windows. There are no alternative solutions for this at the moment.

### Installation

`> pip install icetea`

### Use

To display the help page:

`icetea --help`

To list all local test cases from the examples subfolder:

`icetea --list --tcdir examples`

To print the Icetea version:

`icetea --version`

#### Typical use

All of the commands described below might also require other commands, depending on the test case.

**Running test cases by using the `tc` argument**

`> icetea --tc <test case name> --tcdir <test case search path>`

To run all existing test cases from the `examples` folder:

`> icetea --tc all --tcdir examples`

**Running an example test case with hardware**

This example requires a compatible board connected to the computer and an application binary for the board. The test case below is available in [the Icetea GitHub repository](https://github.com/ARMmbed/icetea/blob/master/examples/test_cmdline.py).

`> icetea --tc test_cmdline --tcdir examples --type hardware --bin <path to a binary>`

**Using metadata filters**

To run all test cases with test-type regression in the metadata:

`> icetea --testtype regression --tcdir <test case search path>`

The following metadata filters are available:

- test type (`--testtype`).
- test subtype (`--subtype`).
- feature (`--feature`).
- test case name (`--tc`).
- tested component (`--component`).
- test case folder (`--group`).

**Running a premade suite**

Icetea supports a suite file that describes a number of test cases in `json` format:

`> icetea --suite <suite file name> --tcdir <test case search path> --suitedir <path to suite directory>`

**Enabling debug level logging**

Use `-v` or `-vv` arguments to control logging levels. `-v` increases the framework's logging level to debug (default is info), the level of logging in certain plugins and external components to info (default is warning). `--vv` increases the level of logging on all Icetea loggers to debug.

#### Creating a test case

Icetea test cases are implemented as Python classes that inherit the bench object available in the `icetea_lib.bench` module. The test case needs to have an initialization function that defines the metadata and a case function that implements the test sequence. There are two optional functions: setup and teardown.

An example test case is shown below:

```
"""
Copyright 2017 ARM Limited
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""

from icetea_lib.bench import Bench


class Testcase(Bench):
def __init__(self):
Bench.__init__(self,
name="example_test",
title="Example test",
status="development",
purpose="Show example of a test",
component=["examples"],
type="smoke",
requirements={
"duts": {
'*': {
"count": 1,
"type": "hardware"
}
}
}
)

def setup(self):
# nothing for now
pass


def case(self):
self.command(1, "echo hello world", timeout=5)
self.command("help")

def teardown(self):
# nothing for now
pass
```

### License

For licensing details, please see the [license agreement](https://github.com/ARMmbed/icetea/blob/master/LICENSE).
Loading