Merge pull request #659 from OPpuolitaival/icetea

Icetea added

Author: Amanda Butler

docs.json

@@ -819,11 +819,11 @@
                     },
                     {
                         "type": "markdown",
-                        "url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/testing.md"
+                        "url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/testing_greentea.md"
                     },
                     {
                         "type": "markdown",
-                        "url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/greentea.md"
+                        "url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/testing_icetea.md"
                     },
                     {
                         "type": "markdown",

docs/reference/contributing/connectivity/MeshInterface.md

@@ -66,7 +66,9 @@ The following steps describe how you can create a new RF driver:
 
 1. Check with an RF sniffer tool that you can see RF packets transmitted when you start your device. The 6LoWPAN bootstrap should start with IEEE 802.15.4 Beacon Request packets.
 
-1. Verify the functionality of your implementation using the [Nanostack RF driver test application](https://github.com/ARMmbed/nanostack-rf-driver-tester). (This is currently only available to Mbed OS Partners.)
+1. Verify the functionality of your implementation by running the Nanostack RF driver testcase set in the Mbed OS repository:
+
+   `mbed test --clean --icetea -t <toolchain> -m <platform> --test-config MAC_TESTER -n address_read_and_write,send_data,send_data_indirect,send_large_payloads,create_and_join_PAN,ED_scan`
 
 ### Worker thread for Mbed OS
 

docs/tools/offline/cli-test-debug.md

@@ -2,23 +2,31 @@
 
 Use the `mbed test` command to compile and run tests.
 
+There are two testing frameworks: Greentea and Icetea. Greentea provides tests designed for driver porting and target verification. Icetea provides and manages tests for multiple devices at the same time. For example, you can test the network setup for a server and multiple clients, simultaneously controlling them from the test environment.
+
 The arguments to `test` are:
-* `-m <MCU>` to select a target for the compilation. If `detect` or `auto` parameter is passed, then Mbed CLI will attempt to detect the connected target and compile against it.
-* `-t <TOOLCHAIN>` to select a toolchain (of those defined in `mbed_settings.py`, see above), where `toolchain` can be either `ARM` (Arm Compiler 5), `GCC_ARM` (GNU Arm Embedded), or `IAR` (IAR Embedded Workbench for Arm).
-* `--compile-list` to list all the tests that can be built.
-* `--run-list` to list all the tests that can be run (they must be built first).
-* `--compile` to only compile the tests.
-* `--run` to only run the tests.
-* `-n <TESTS_BY_NAME>` to limit the tests built or run to a comma separated list (ex. test1,test2,test3).
-* `--source <SOURCE>` to select the source directory. Default is `.` (the current directory). You can specify multiple source locations, even outside the program tree.
-* `--build <BUILD>` to select the build directory. Default: `BUILD/` inside your program.
-* `--profile <PATH_TO_BUILD_PROFILE>` to select a path to a build profile configuration file. Example: `mbed-os/tools/profiles/debug.json`.
-* `-c or --clean` to clean the build directory before compiling.
-* `--test-spec <TEST_SPEC>` to set the path for the test spec file used when building and running tests (the default path is the build directory).
-* `-v` or `--verbose` for verbose diagnostic output.
-* `-vv` or `--very_verbose` for very verbose diagnostic output.
-
-Invoke `mbed test`:
+
+- `-m <MCU>`: to select a target for the compilation. If the `detect` or `auto` parameter is passed, then Mbed CLI will attempt to detect the connected target and compile against it.
+- `-t <TOOLCHAIN>`: to select a toolchain from those defined in `mbed_settings.py`, where `toolchain` can either be `ARM` (Arm Compiler 5), `GCC_ARM` (GNU Arm Embedded), or `IAR` (IAR Embedded Workbench for Arm).
+- `--compile-list`: to list all the tests that can be built.
+- `--run-list`: to list all the tests that can be run, after they have been built.
+- `--compile`: to only compile the tests.
+- `--run`: to only run the tests.
+- `-n <TESTS_BY_NAME>`: to limit the tests built or run to a comma separated list, for example, `test1, test2, test3`.
+- `--source <SOURCE>`: to select the source directory. The default is `.` for the the current directory. You can specify multiple source locations, even outside the program tree.
+- `--build <BUILD>`: to select the build directory. The default is `BUILD/` inside your program.
+- `--profile <PATH_TO_BUILD_PROFILE>`: to select a path to a build profile configuration file, for example, `mbed-os/tools/profiles/debug.json`.
+- `-c or --clean`: to clean the build directory before compiling.
+- `--test-spec <TEST_SPEC>`: to set the path for the test specification file used when building and running tests. The default path is the build directory.
+- `--build-data <BUILD_DATA>`: dumps build_data to this file.
+- `--app-config <APP_CONFIG>`: the path of an app configuration file. The default is to look for `mbed_app.json`.
+- `--test-config <TEST_CONFIG>`: the path or Mbed OS keyword of a test configuration file, for example, `ethernet`, `odin_wifi` or `path/to/config.json`.
+- `--greentea`: to run Greentea tests. As a default, it only runs Greentea tests.
+- `--icetea`: to run Icetea tests. If used without the `--greentea` flag, then it only runs Icetea tests.
+- `-v` or `--verbose`: for verbose diagnostic output.
+- `-vv` or `--very_verbose`: for very verbose diagnostic output.
+
+To invoke the `mbed test`:
 
 ```
 $ mbed test -m K64F -t GCC_ARM
@@ -79,6 +87,33 @@ Test Case:
     Path: .\TESTS\functional\test3
 ```
 
+For Icetea:
+
+```
+$ mbed test -m K64F -t GCC_ARM --icetea --compile-list
+Available Icetea tests for build 'K64F-GCC_ARM', location 'TEST_APPS'
+Test Case:
+    Name: test_cmdline
+    Path: ./TEST_APPS/testcases/example/test_cmdline.py
+    Test applications: ./TEST_APPS/device/exampleapp
+Test Case:
+    Name: UDPSOCKET_BIND_PORT
+    Path: ./TEST_APPS/testcases/netsocket/SOCKET_BIND_PORT.py
+    Test applications: ./TEST_APPS/device/socket_app
+Test Case:
+    Name: TCPSOCKET_BIND_PORT
+    Path: ./TEST_APPS/testcases/netsocket/SOCKET_BIND_PORT.py
+    Test applications: ./TEST_APPS/device/socket_app
+Test Case:
+    Name: TCPSERVER_ACCEPT
+    Path: ./TEST_APPS/testcases/netsocket/TCPSERVER_ACCEPT.py
+    Test applications: ./TEST_APPS/device/socket_app
+Test Case:
+    Name: TCPSOCKET_ECHOTEST_BURST_SHORT
+    Path: ./TEST_APPS/testcases/netsocket/TCPSOCKET_ECHOTEST_BURST_SHORT.py
+    Test applications: ./TEST_APPS/device/socket_app
+```
+
 You can find the tests that are available for **running** by using the `--run-list` option:
 
 ```
@@ -91,21 +126,38 @@ mbedgt: available tests for built 'K64F-ARM', location '.\build\tests\K64F\ARM'
         test 'TESTS-functional-test3'
 ```
 
+For Icetea:
+
+```
+$ mbed test -m K64F -t GCC_ARM --icetea --run-list
+Available Icetea tests for build 'K64F-GCC_ARM', location 'TEST_APPS'
+    test 'UDPSOCKET_BIND_PORT'
+    test 'TCPSOCKET_BIND_PORT'
+    test 'TCPSERVER_ACCEPT'
+    test 'TCPSOCKET_ECHOTEST_BURST_SHORT'
+```
+
 ### Compiling and running tests
 
-You can specify to only **build** the tests by using the `--compile` option:
+You can specify that the tests only **build** by using the `--compile` option:
 
 ```
 $ mbed test -m K64F -t GCC_ARM --compile
 ```
 
-You can specify to only **run** the tests by using the `--run` option:
+For Icetea, only the test applications are built:
+
+```
+$ mbed test -m K64F -t GCC_ARM --compile --icetea
+```
+
+You can specify that the tests only **run** by using the `--run` option:
 
 ```
 $ mbed test -m K64F -t GCC_ARM --run
 ```
 
-If you don't specify any of these, `mbed test` will first compile all available tests and then run them.
+If you don't specify any of these, `mbed test` first compiles all available tests and then runs them.
 
 ### Limiting the test scope
 
@@ -115,7 +167,7 @@ You can limit the scope of the tests built and run by using the `-n` option. Thi
 $ mbed test -m K64F -t GCC_ARM -n TESTS-functional-test1,TESTS-functional-test2
 ```
 
-You can use the wildcard character `*` to run a group of tests that share a common prefix without specifying each test individually. For instance, if you only want to run the three tests `TESTS-functional-test1`, `TESTS-functional-test2` and `TESTS-functional-test3`, but you have other tests in your project, you can run:
+You can use the wildcard character `*` to run a group of tests that share a common prefix without specifying each test individually. For instance, if you only want to run the three tests, `TESTS-functional-test1`, `TESTS-functional-test2` and `TESTS-functional-test3`, but you have other tests in your project, you can run:
 
 ```
 $ mbed test -m NUCLEO_F429ZI -t GCC_ARM -n TESTS-functional*
@@ -154,16 +206,17 @@ mbed-os-program
      | ....
 ```
 
-As shown above, tests exist inside `TESTS\testgroup\testcase\` directories. Please note that `TESTS` is a special upper case directory that is excluded from module sources while compiling.
+As shown above, tests exist inside `TESTS\testgroup\testcase\` directories. Please note that `TESTS` is a special upper-case directory that is excluded from module sources while compiling.
 
-<span class="notes">**Note:** `mbed test` does not work in applications that contain a  `main` function that is outside of a `TESTS` directory.</span>
+<span class="notes">**Note:** `mbed test` does not work in applications that contain a `main` function that is outside of a `TESTS` directory.</span>
 
 ### Troubleshooting
 
-#### Unable to import Mercurial (mbed.org) programs or libraries.
-1. Check whether you have Mercurial installed in your system path by  running `hg` in command prompt. If you're receiving "command not found" or a similar message, then you need to install Mercurial, and add it to your system path.
+#### Import Mercurial (mbed.org) programs or libraries
+
+1. Check whether you have Mercurial installed in your system path by  running `hg` in the command prompt. If you are receiving "command not found" or a similar message, then you need to install Mercurial and add it to your system path.
+1. Try to clone a Mercurial repository directly. For example, `hg clone https://developer.mbed.org/teams/mbed/code/mbed_blinky/`. If you receive an error similar to `abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.:590)`, then your system certificates are out of date. You need to update your system certificates and possibly add the host certificate fingerprint of `mbed.com` and `mbed.org`. You can read more about Mercurial's [certificate management](https://www.mercurial-scm.org/wiki/CACertificates).
 
-2. Try to clone a Mercurial repository directly. For example, `hg clone https://developer.mbed.org/teams/mbed/code/mbed_blinky/`. If you receive an error similar to `abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.:590)`, then your system certificates are out of date. You need to update your system certificates and possibly add the host certificate fingerprint of `mbed.com` and `mbed.org`. You can read more about Mercurial's [certificate management](https://www.mercurial-scm.org/wiki/CACertificates).
+#### Various issues when running Mbed CLI in the Cygwin environment
 
-#### Various issues when running Mbed CLI in Cygwin environment
-Currently Mbed CLI is not compatible with Cygwin environment and [cannot be executed inside it](https://github.com/ARMmbed/mbed-cli/issues/299).
+Mbed CLI is not currently compatible with the Cygwin environment and [cannot be executed inside it](https://github.com/ARMmbed/mbed-cli/issues/299).

docs/tools/testing/icetea.md

@@ -0,0 +1,166 @@
+<h2 id="icetea">Icetea test framework</h2>
+
+Icetea is an automated testing framework for Mbed development. It automates the process of flashing Mbed boards, running tests and accumulating test results into reports. Developers use it for local development, as well as automation in a Continuous Integration environment.
+
+When testing Mbed OS, Icetea allows you to execute commands remotely by using the command-line interface in a device under test (DUT). The interface between the test framework and a DUT might be, for example, UART or stdio.
+
+More detailed documentation on the tool is available [in rst format](https://github.com/ARMmbed/icetea/tree/master/doc-source) and [in markdown format](https://github.com/ARMmbed/icetea/tree/master/doc).
+
+### Prerequisites
+
+Icetea supports Linux (Ubuntu preferred), Windows and OS X. Our main target is Linux.
+
+You need `pip` to install Icetea.
+
+Icetea supports both Python 2.7 and 3.5, or later. Some OS specific prerequisites are listed below:
+
+- Linux.
+   - python-dev and python-lxml:
+      `sudo apt-get install python-dev python-lxml`
+   - To run test cases with hardware in Linux, without sudo rights:
+   
+      ```
+      sudo usermod -a -G dialout username
+      Log out & log in back to Linux
+      ```
+   
+      This command adds the user `username` to the `dialout` group and grants the required permissions to the USB ports.
+      
+- OS X.
+   - [XCode developer tools](http://osxdaily.com/2014/02/12/install-command-line-tools-mac-os-x/).
+   - [MacPorts](https://www.macports.org/install.php).
+   - [lxml](http://lxml.de/installation.html#installation):
+      `STATIC_DEPS=true sudo pip install lxml`
+
+- Windows
+   - python-lxml installation on Windows usually requires build tools. You can, however, install it from prebuilt binaries.
+      - Search the internet for a binary for your system.
+      - Navigate to the directory where you downloaded the binary, and install it with `pip install <insert_file_name>`
+   - You can also follow [these instructions](http://lxml.de/installation.html#installation) instead.
+
+#### Optional
+
+If you wish to decorate your console log with colors, install the `coloredlogs` module by using pip: `pip install coloredlogs`. There have been issues with `coloredlogs` installation on Windows. There are no alternative solutions for this at the moment.
+
+### Installation
+
+`> pip install icetea`
+
+### Use
+
+To display the help page:
+
+`icetea --help`
+
+To list all local test cases from the examples subfolder:
+
+`icetea --list --tcdir examples`
+
+To print the Icetea version:
+
+`icetea --version`
+
+#### Typical use
+
+All of the commands described below might also require other commands, depending on the test case.
+
+**Running test cases by using the `tc` argument**
+
+`> icetea --tc <test case name> --tcdir <test case search path>`
+
+To run all existing test cases from the `examples` folder:
+
+`> icetea --tc all --tcdir examples`
+
+**Running an example test case with hardware**
+
+This example requires a compatible board connected to the computer and an application binary for the board. The test case below is available in [the Icetea GitHub repository](https://github.com/ARMmbed/icetea/blob/master/examples/test_cmdline.py).
+
+`> icetea --tc test_cmdline --tcdir examples --type hardware --bin <path to a binary>`
+
+**Using metadata filters**
+
+To run all test cases with test-type regression in the metadata:
+
+`> icetea --testtype regression --tcdir <test case search path>`
+
+The following metadata filters are available:
+
+- test type (`--testtype`).
+- test subtype (`--subtype`).
+- feature (`--feature`).
+- test case name (`--tc`).
+- tested component (`--component`).
+- test case folder (`--group`).
+
+**Running a premade suite**
+
+Icetea supports a suite file that describes a number of test cases in `json` format:
+
+`> icetea --suite <suite file name> --tcdir <test case search path> --suitedir <path to suite directory>`
+
+**Enabling debug level logging**
+
+Use `-v` or `-vv` arguments to control logging levels. `-v` increases the framework's logging level to debug (default is info), the level of logging in certain plugins and external components to info (default is warning). `--vv` increases the level of logging on all Icetea loggers to debug.
+
+#### Creating a test case
+
+Icetea test cases are implemented as Python classes that inherit the bench object available in the `icetea_lib.bench` module. The test case needs to have an initialization function that defines the metadata and a case function that implements the test sequence. There are two optional functions: setup and teardown.
+
+An example test case is shown below:
+
+```
+"""
+Copyright 2017 ARM Limited
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+from icetea_lib.bench import Bench
+
+
+class Testcase(Bench):
+    def __init__(self):
+        Bench.__init__(self,
+                       name="example_test",
+                       title="Example test",
+                       status="development",
+                       purpose="Show example of a test",
+                       component=["examples"],
+                       type="smoke",
+                       requirements={
+                           "duts": {
+                               '*': {
+                                    "count": 1,
+                                    "type": "hardware"
+                               }
+                           }
+                       }
+                       )
+
+    def setup(self):
+        # nothing for now
+        pass
+
+
+    def case(self):
+        self.command(1, "echo hello world", timeout=5)
+        self.command("help")
+
+    def teardown(self):
+        # nothing for now
+        pass
+```
+
+### License
+
+For licensing details, please see the [license agreement](https://github.com/ARMmbed/icetea/blob/master/LICENSE).