Host test autodetection

[PrzemekWirkus]

Description

In order to achieve leaner test suite functionality and in the future decouple functionality such as:

• mbed-enabled board auto-detection - mbed-ls project, former mbed-lmtools.
• Host tests in worskace_tools/host_tests - set of test supervising scripts.
• Test suite CLI (command line interface)
• Building mechanism such as build.py, make.py and singletest.py will be replaced in future with separate project build tool.

mbed-ls - cool mbed-enabled platform auto-detection on your host!

mbed-ls is Python module used to detect and list mbed-enabled devices connected to host computer. package will support Windows 7, Ubuntu and MacOS. It will be delivered as redistributable Python module (package) and command line tool.
Currently mbed-ls module is under development but already decoupled mbed-ls functionality is delivered to mbed SDK's test suite.

You will be able for example to use this tool from command line!

$ mbedls
+---------------------+-------------------+-------------------+--------------------------------+
|platform_name        |mount_point        |serial_port        |target_id                       |
+---------------------+-------------------+-------------------+--------------------------------+
|KL25Z                |I:                 |COM89              |02000203240881BBD9F47C43        |
|NUCLEO_F302R8        |E:                 |COM34              |07050200623B61125D5EF72A        |
+---------------------+-------------------+-------------------+--------------------------------+

New command line switch --auto and --tc are added to workspace_tools/singletest.py script and replace (not completely, you will be still able to use configuration files just because they are more powerful than mbed-ls) existing file based test configuration.

Currently you will execute your automated tests like this:

$ singletest.py -i test_spec.json -M muts_all.json

With mbed-ls tools you will be able to auto-detect all connected to your host devices and just decide which toolchains you want to include.
Below command is replacement for -i and -M combo. Tests will be built using default compiler for each detected mbed-enabled platform.
Note: Now mbed-ls tools will detect devices for you, pair serial ports and disks (mount points) and based on TargetID determine mbed-enabled target generic name.

$ singletest.py --auto

or just for example specify that you want to build tests using default toolchains (per platform) and GCC_ARM for all platforms.

$ singletest.py --auto --tc default,GCC_ARM

New --auto command is compatible with existing command line switches so you can for example:

• Check your auto-detected configuration:

$ singletest.py --auto --config
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_F302R8, port: COM34, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: I:
MUTs configuration in auto-detected:
+-------+-------------+---------------+------+-------+
| index | peripherals | mcu           | disk | port  |
+-------+-------------+---------------+------+-------+
| 1     |             | NUCLEO_F302R8 | E:   | COM34 |
| 2     |             | KL25Z         | I:   | COM89 |
+-------+-------------+---------------+------+-------+

Test specification in auto-detected:
+---------------+-----+------+
| mcu           | ARM | uARM |
+---------------+-----+------+
| KL25Z         | Yes | -    |
| NUCLEO_F302R8 | -   | Yes  |
+---------------+-----+------+

• Check supported toolchains for each platform (with optional filter -f):

$ singletest.py -S -f F302
+---------------+-----------+---------+-----------+--------+--------+-----------+------------+---------------+
| Platform      |    ARM    |   uARM  |  GCC_ARM  | GCC_CS | GCC_CR |    IAR    | GCC_CW_EWL | GCC_CW_NEWLIB |
+---------------+-----------+---------+-----------+--------+--------+-----------+------------+---------------+
| NUCLEO_F302R8 | Supported | Default | Supported |   -    |   -    | Supported |     -      |       -       |
+---------------+-----------+---------+-----------+--------+--------+-----------+------------+---------------+
*Default - default on-line compiler
*Supported - supported off-line compiler

Total platforms: 1
Total permutations: 4

• Test suite will check for you if you configured settings and your toolchain path(s):

$ singletest.py --auto --config --tc all
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_F302R8, port: COM34, mounted: E:
MBEDLS: Detected KL25Z, port: COM89, mounted: I:
MUTs configuration in auto-detected:
+-------+-------------+---------------+------+-------+
| index | peripherals | mcu           | disk | port  |
+-------+-------------+---------------+------+-------+
| 1     |             | NUCLEO_F302R8 | E:   | COM34 |
| 2     |             | KL25Z         | I:   | COM89 |
+-------+-------------+---------------+------+-------+

Test specification in auto-detected:
+---------------+-----+---------+------------+---------------+-----+------+
| mcu           | ARM | GCC_ARM | GCC_CW_EWL | GCC_CW_NEWLIB | IAR | uARM |
+---------------+-----+---------+------------+---------------+-----+------+
| KL25Z         | Yes | Yes     | Yes#       | Yes#          | Yes | -    |
| NUCLEO_F302R8 | Yes | Yes     | -#         | -#            | Yes | Yes  |
+---------------+-----+---------+------------+---------------+-----+------+
Toolchain conflicts:
        # Toolchain GCC_CW_EWL path not found: C:/Freescale/CW MCU v10.3/MCU/ARM_GCC_Support/ewl/lib
        # Toolchain GCC_CW_NEWLIB path not found: C:/Freescale/CW MCU v10.3/Cross_Tools/arm-none-eabi-gcc-4_6_2/bin

• Use test verbose mode -V to see test printouts in real time, choose test(s) you want to execute with -n option and define your favorite toolchain(s) with --tc:

$ singletest.py --auto --tc GCC_ARM -V -n DTCT_1
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_F302R8, port: COM34, mounted: E:
Building library CMSIS (NUCLEO_F302R8, GCC_ARM)
Building library MBED (NUCLEO_F302R8, GCC_ARM)
Building project DETECT (NUCLEO_F302R8, GCC_ARM)
Executing 'python host_test.py -d E: -f "C:\Work\mbed-przemek\build\test\NUCLEO_F302R8\GCC_ARM\DTCT_1\detect.bin" -p COM34 -t 10 -C 1.5 -m NUCLEO_F302R8'
Test::Output::Start
MBED: Instrumentation: "COM34" and disk: "E:"
HOST: Copy image onto target...
HOST: Initialize serial port...
HOST: Reset target...
HOST: Property 'timeout' = '10'
HOST: Property 'host_test_name' = 'detect_auto'
HOST: Property 'description' = 'Simple detect test'
HOST: Property 'test_id' = 'DTCT_1'
HOST: Start test...
{{start}}
HOST: Detecting target name...
MBED: Target 'NUCLEO_F302R8'
HOST: MUT Target name 'NUCLEO_F302R8', expected 'NUCLEO_F302R8'... [OK]
MBED: Test ID 'DTCT_1'
MBED: UUID '04dab79c-a6c8-4bf0-8b1e-3771b4540bb8'

{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F302R8::GCC_ARM::DTCT_1::Simple detect test [OK] in 1.07 of 10 sec
Test summary:
+--------+---------------+-----------+---------+--------------------+--------------------+---------------+-------+
| Result | Target        | Toolchain | Test ID | Test Description   | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------------+-----------+---------+--------------------+--------------------+---------------+-------+
| OK     | NUCLEO_F302R8 | GCC_ARM   | DTCT_1  | Simple detect test |        1.07        |       10      |  1/1  |
+--------+---------------+-----------+---------+--------------------+--------------------+---------------+-------+
Result: 1 OK

Completed in 4.33 sec

Host test script auto-detection

Structure for host tests and C/C++ test source code changed. In the future tests will be deployed together with corresponding modules. For now mbed SDK tests have to improve. We want to remove dependency on workspace_tools/tests.py test meta-data and move test / test runner handshake inside test source code.

• Example target test, look how new macros TEST_* are used to notify host test system about prefered host test handler. Now test will decide which host test should handle it, not extra meta-data.

#include "mbed.h"
#include "test_env.h"

int main() {
    TEST_TIMEOUT(10);
    TEST_HOSTTEST(detect_auto);
    TEST_DESCRIPTION(Simple detect test);
    TEST_START("DTCT_1");

    // Test case body start
    notify_start();
    printf("MBED: Target '%s'\r\n", TEST_SUITE_TARGET_NAME);
    printf("MBED: Test ID '%s'\r\n", TEST_SUITE_TEST_ID);
    printf("MBED: UUID '%s'\r\n", TEST_SUITE_UUID);
    // Test case body end

    TEST_RESULT(true);
}

Note, things like:

• Test case timeout (real time of execution) is defined in macro TEST_TIMEOUT().
• Test case description is encapsulated with macro TEST_DESCRIPTION().
• Preferred host test handler is defined in macro TEST_HOSTTEST().
• Test case result can be send to test suite environment using TEST_RESULT() macro - if applicable. Host test can decide about test case result in many cases.

mbed SDK is now in color!

Colorama is new package dependency added so CLI is now colorful (Hard to say if this is good or bad).
Below sample of new color format:

Multiple boards of the same type

Few boards of the same type can be auto-detected but tests are performed for one device of each type.

$ mbedls
+---------------------+-------------------+-------------------+--------------------------------+
|platform_name        |mount_point        |serial_port        |target_id                       |
+---------------------+-------------------+-------------------+--------------------------------+
|NUCLEO_L152RE        |E:                 |COM9               |07100200860579FAB960EFD7        |
|NUCLEO_L152RE        |F:                 |COM91              |07100200824A7DB5BD2FEB98        |
+---------------------+-------------------+-------------------+--------------------------------+

Test execution in above configuration:

$ singletest.py --auto --tc default,GCC_ARM -n MBED_A1,MBED_34
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected NUCLEO_L152RE, port: COM9, mounted: E:
MBEDLS: Detected NUCLEO_L152RE, port: COM91, mounted: F:
Building library CMSIS (NUCLEO_L152RE, uARM)
Building library MBED (NUCLEO_L152RE, uARM)
Building project TICKER_3 (NUCLEO_L152RE, uARM)
TargetTest::NUCLEO_L152RE::uARM::MBED_34::Ticker Two callbacks [OK] in 12.28 of 15 sec
Building project BASIC (NUCLEO_L152RE, uARM)
TargetTest::NUCLEO_L152RE::uARM::MBED_A1::Basic [OK] in 2.35 of 20 sec
Building library CMSIS (NUCLEO_L152RE, GCC_ARM)
Building library MBED (NUCLEO_L152RE, GCC_ARM)
Building project TICKER_3 (NUCLEO_L152RE, GCC_ARM)
TargetTest::NUCLEO_L152RE::GCC_ARM::MBED_34::Ticker Two callbacks [OK] in 12.98 of 15 sec
Building project BASIC (NUCLEO_L152RE, GCC_ARM)
TargetTest::NUCLEO_L152RE::GCC_ARM::MBED_A1::Basic [OK] in 2.35 of 20 sec
Test summary:
+--------+---------------+-----------+---------+----------------------+--------------------+---------------+-------+
| Result | Target        | Toolchain | Test ID | Test Description     | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------------+-----------+---------+----------------------+--------------------+---------------+-------+
| OK     | NUCLEO_L152RE | uARM      | MBED_34 | Ticker Two callbacks |       12.28        |       15      |  1/1  |
| OK     | NUCLEO_L152RE | uARM      | MBED_A1 | Basic                |        2.35        |       20      |  1/1  |
| OK     | NUCLEO_L152RE | GCC_ARM   | MBED_34 | Ticker Two callbacks |       12.98        |       15      |  1/1  |
| OK     | NUCLEO_L152RE | GCC_ARM   | MBED_A1 | Basic                |        2.35        |       20      |  1/1  |
+--------+---------------+-----------+---------+----------------------+--------------------+---------------+-------+
Result: 4 OK

Completed in 41.66 sec

Testing

Solution tested with 8 of mbed-enabled boards. Both test auto-detection for test_env and unit tests using CppUTest works well.
This solution will be tested with greater number of boards soon, so pelase wait for notification in this pool request.

[0xc0170]

I would use prefix with the name of this test env, it's either part of mbed or any other. The other libraries can easily have similar macros TEST_DESCRIPTION or similar.

[PrzemekWirkus]

Yes, you are right, I will change it to something more mbed specific / descriptive.