test suite: mbed assert detection

[PrzemekWirkus]

Description

Currently when MBED_ASSERT() macro is used in test case source code or mbed SDK code as part of test case we do not detect assertion and test cases fail because of timeout (test case result: TIMEOUT).
We want to detect assertion and end test case as soon as possible.

This small update changes how host test supervising script looks for prints. Now we also check for mbed assertion printout and if found we exit with MBED_ASSERT test case test result.

Testing

We are looking for mbed assert string. Example mbed assert string structure:

mbed assertation failed: _pin[index], file: C:\Work\mbed-przemek\libraries\mbed\common\BusOut.cpp, line 82

Example 1. mbed assert detection missing from host test supervising script:

$ singletest.py --auto -j 8 -n UT_BUSIO -V
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected KL46Z, port: COM40, mounted: I:
Building library CMSIS (KL46Z, ARM)
Building library MBED (KL46Z, ARM)
Building library CPPUTEST (KL46Z, ARM)
Building project BUS (KL46Z, ARM)
Executing 'python host_test.py -d I: -f "C:\Work\mbed-przemek\build\test\KL46Z\ARM\UT_BUSIO\bus.bin" -p COM40 -t 10 -C 4 -m KL46Z'
Test::Output::Start
MBED: Instrumentation: "COM40" and disk: "I:"
HOST: Copy image onto target...
HOST: Initialize serial port...
HOST: Reset target...
HOST: Property 'timeout' = '20'
HOST: Property 'host_test_name' = 'default_auto'
HOST: Property 'description' = 'Unit test'
HOST: Property 'test_id' = 'UT'
HOST: Start test...
TEST(BusOut_digitalout_write, led_1_nc_2_nc_nc_3)
C:\Work\mbed-przemek\libraries\tests\utest\bus\busout_ut.cpp:83: error: Failure in TEST(BusOut_digitalout_write, led_1_nc_2_nc_nc_3)
        CHECK(bus_data[0].read()) failed

 - 17 ms
TEST(BusOut_digitalout_write, led_1_2_3_nc_nc) - 0 ms
TEST(BusOut_digitalout_write, led_1_2_3) - 0 ms
TEST(BusOut_digitalout_write, led_nc)mbed assertation failed: _pin[index], file: C:\Work\mbed-przemek\libraries\mbed\common\BusOut.cpp, line 82
Test::Output::Finish
TargetTest::KL46Z::ARM::UT_BUSIO::BusIn BusOut [MBED_ASSERT] in 1.41 of 20 sec
Test summary:
+-------------+--------+-----------+----------+------------------+--------------------+---------------+-------+
| Result      | Target | Toolchain | Test ID  | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
+-------------+--------+-----------+----------+------------------+--------------------+---------------+-------+
| TIMEOUT     | KL46Z  | ARM       | UT_BUSIO | BusIn BusOut     |        20.0        |       20      |  0/1  |
+-------------+--------+-----------+----------+------------------+--------------------+---------------+-------+
Result: 1 TIMEOUT

Completed in 22.3 sec

Example 2. mbed assert detection added to host test supervising script:

$ singletest.py --auto -j 8 -n UT_BUSIO -V
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected KL46Z, port: COM40, mounted: I:
Building library CMSIS (KL46Z, ARM)
Building library MBED (KL46Z, ARM)
Building library CPPUTEST (KL46Z, ARM)
Building project BUS (KL46Z, ARM)
Executing 'python host_test.py -d I: -f "C:\Work\mbed-przemek\build\test\KL46Z\ARM\UT_BUSIO\bus.bin" -p COM40 -t 10 -C 4 -m KL46Z'
Test::Output::Start
MBED: Instrumentation: "COM40" and disk: "I:"
HOST: Copy image onto target...
HOST: Initialize serial port...
HOST: Reset target...
HOST: Property 'timeout' = '20'
HOST: Property 'host_test_name' = 'default_auto'
HOST: Property 'description' = 'Unit test'
HOST: Property 'test_id' = 'UT'
HOST: Start test...
TEST(BusOut_digitalout_write, led_1_nc_2_nc_nc_3)
C:\Work\mbed-przemek\libraries\tests\utest\bus\busout_ut.cpp:83: error: Failure in TEST(BusOut_digitalout_write, led_1_nc_2_nc_nc_3)
        CHECK(bus_data[0].read()) failed

 - 17 ms
TEST(BusOut_digitalout_write, led_1_2_3_nc_nc) - 0 ms
TEST(BusOut_digitalout_write, led_1_2_3) - 0 ms
TEST(BusOut_digitalout_write, led_nc)mbed assertation failed: _pin[index], file: C:\Work\mbed-przemek\libraries\mbed\common\BusOut.cpp, line 82
Test::Output::Finish
TargetTest::KL46Z::ARM::UT_BUSIO::BusIn BusOut [MBED_ASSERT] in 1.41 of 20 sec
Test summary:
+-------------+--------+-----------+----------+------------------+--------------------+---------------+-------+
| Result      | Target | Toolchain | Test ID  | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
+-------------+--------+-----------+----------+------------------+--------------------+---------------+-------+
| MBED_ASSERT | KL46Z  | ARM       | UT_BUSIO | BusIn BusOut     |        1.41        |       20      |  0/1  |
+-------------+--------+-----------+----------+------------------+--------------------+---------------+-------+
Result: 1 MBED_ASSERT

Completed in 8.36 sec

Test suite regression for one board (expected one assert for UT_BUSOUT unit test)

$ singletest.py --auto -V
Test summary:
+-------------+--------+-----------+-------------+---------------------------------------+--------------------+---------------+-------+
| Result      | Target | Toolchain | Test ID     | Test Description                      | Elapsed Time (sec) | Timeout (sec) | Loops |
+-------------+--------+-----------+-------------+---------------------------------------+--------------------+---------------+-------+
| OK          | KL46Z  | ARM       | DTCT_1      | Simple detect test                    |        0.52        |       10      |  1/1  |
| OK          | KL46Z  | ARM       | EXAMPLE_1   | /dev/null                             |        3.49        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | KL25Z_5     | MMA8451Q accelerometer                |       12.52        |       15      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_10     | Hello World                           |        0.47        |       5       |  1/1  |
| OK          | KL46Z  | ARM       | MBED_11     | Ticker Int                            |       11.43        |       15      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_12     | C++                                   |        1.41        |       10      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_16     | RTC                                   |        4.66        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_2      | stdio                                 |        0.78        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_23     | Ticker Int us                         |       11.44        |       15      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_24     | Timeout Int us                        |        11.6        |       15      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_25     | Time us                               |       11.48        |       15      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_26     | Integer constant division             |        1.39        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_34     | Ticker Two callbacks                  |        11.4        |       15      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_A1     | Basic                                 |        1.39        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_A21    | Call function before main (mbed_main) |        1.45        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_A9     | Serial Echo at 115200                 |        5.84        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | MBED_BUSOUT | BusOut                                |        2.31        |       15      |  1/1  |
| OK          | KL46Z  | ARM       | RTOS_1      | Basic thread                          |        11.4        |       15      |  1/1  |
| OK          | KL46Z  | ARM       | RTOS_2      | Mutex resource lock                   |       11.54        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | RTOS_3      | Semaphore resource lock               |        8.47        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | RTOS_4      | Signals messaging                     |        6.45        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | RTOS_5      | Queue messaging                       |        2.44        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | RTOS_6      | Mail messaging                        |        2.43        |       20      |  1/1  |
| OK          | KL46Z  | ARM       | RTOS_7      | Timer                                 |       11.43        |       15      |  1/1  |
| OK          | KL46Z  | ARM       | RTOS_8      | ISR (Queue)                           |        6.45        |       20      |  1/1  |
| MBED_ASSERT | KL46Z  | ARM       | UT_BUSIO    | BusIn BusOut                          |        1.44        |       20      |  0/1  |
+-------------+--------+-----------+-------------+---------------------------------------+--------------------+---------------+-------+
Result: 1 MBED_ASSERT / 25 OK

Completed in 301.12 sec

[0xc0170]

+1 for this mbed assert detection.

[0xc0170]

Rebase when updating your branch with upstream, the commit merge could be omitted