Singletest option -p to force peripherals

[PrzemekWirkus]

Description

mbed/workspace_tools/singletest.py script can work in --auto mode with mbed-ls module (not yet released officially).
Auto-detection function doesn't allow users to define peripherals from command line. Now option -p can give you this functionality. You can set up for singletest.py global peripheral set which will be added to each auto-detected MUT (Mbed Under Test). This will allow users with farms (test beds) with common peripheral configuration for each device to run also peripheral tests.

• This option can be useful when you connect many devices with the same peripheral profile and you want to execute peripheral tests. Previously you could only execute common tests. Now you can pass to sengletest.py simple peripheral description you would usually pass via option -M and muts_all.json file.
• You can mix new option -p with other options like -n, -O, -j etc.

Examples of -p option use

$ singletest.py --auto -P -p digital_loop,TMP102

$ singletest.py --auto -j 4 -O -P -p SD

Separation between peripheral test mode and common test mode

Our test suite creates logical separation between tests for peripherals (see option -P) and common tests (see option -C) which are tests without any external peripheral dependency like connected via SPI SD card).
But as we said earlier we can mix new option -p with existing test suite options.

When user tries to mix option -p and use option -n (Specify by name test cases to be executed) to define test case names in peripheral test case namespace and common test case names, test suite will add non-peripheral test case names and just execute them.
It shows option -p is not filtering non-peripheral test cases.
Option -P didn't change its meaning. Please use option -P to filter out non-peripheral test cases

See example below where we add use of peripheral SD with new option -p (add SD dependent test case names to global test case namespace) and at the same time we want to run two test cases by name:

• PERF_1 which is dedicated SD peripheral test and
• MBED_11 which is common test (no peripheral required)

See below how we've added SD test case names to test namespace so option -n can catch them and execute.

$ singletest.py --auto -p SD -n MBED_11,PERF_1
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected LPC1768, port: COM77, mounted: H:
Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building project TICKER (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.26 of 15 sec
Building library FAT (LPC1768, ARM)
Building project SD_PERF_STDIO (LPC1768, ARM)
TargetTest::LPC1768::ARM::PERF_1::SD Stdio R/W Speed [OK] in 7.63 of 15 sec
Test summary:
+--------+---------+-----------+---------+--------------------+--------------------+---------------+-------+
| Result | Target  | Toolchain | Test ID | Test Description   | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+---------+--------------------+--------------------+---------------+-------+
| OK     | LPC1768 | ARM       | MBED_11 | Ticker Int         |       12.26        |       15      |  1/1  |
| OK     | LPC1768 | ARM       | PERF_1  | SD Stdio R/W Speed |        7.63        |       15      |  1/1  |
+--------+---------+-----------+---------+--------------------+--------------------+---------------+-------+
Result: 2 OK

Completed in 26.94 sec

See below how we've filtered out all non-peripheral test case names so option -n can now only catch peripheral test case names.

$ singletest.py --auto -p SD -n MBED_11,PERF_1 -P
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected LPC1768, port: COM77, mounted: H:
Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building library FAT (LPC1768, ARM)
Building project SD_PERF_STDIO (LPC1768, ARM)
TargetTest::LPC1768::ARM::PERF_1::SD Stdio R/W Speed [OK] in 7.63 of 15 sec
Test summary:
+--------+---------+-----------+---------+--------------------+--------------------+---------------+-------+
| Result | Target  | Toolchain | Test ID | Test Description   | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+---------+--------------------+--------------------+---------------+-------+
| OK     | LPC1768 | ARM       | PERF_1  | SD Stdio R/W Speed |        7.63        |       15      |  1/1  |
+--------+---------+-----------+---------+--------------------+--------------------+---------------+-------+
Result: 1 OK

Completed in 11.60 sec

Option -C will cancel out all peripheral tests in -n even you specify peripherals using -p:

$ singletest.py --auto -p SD -n MBED_11,PERF_1 -C
.
.
.
Test summary:
+--------+---------+-----------+---------+------------------+--------------------+---------------+-------+
| Result | Target  | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+---------+------------------+--------------------+---------------+-------+
| OK     | LPC1768 | ARM       | MBED_11 | Ticker Int       |       12.22        |       15      |  1/1  |
+--------+---------+-----------+---------+------------------+--------------------+---------------+-------+
Result: 1 OK

Completed in 15.42 sec

Peripheral usage use case

You can connect few boards all equipped with additional SD card peripheral and test for each of this boards SD peripheral related tests. Note that all devices should have common peripheral (SD card connected) and you can't exclude some devices from "I'm having this peripheral(s)" peripheral list.

Example configuration (one LPC1768 device)

$ singletest.py --auto --config
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected LPC1768, port: COM77, mounted: H:
MUTs configuration in auto-detected:
+-------+-------------+---------+------+-------+
| index | peripherals | mcu     | disk | port  |
+-------+-------------+---------+------+-------+
| 1     |             | LPC1768 | H:   | COM77 |
+-------+-------------+---------+------+-------+

Test specification in auto-detected:
+---------+-----+
| mcu     | ARM |
+---------+-----+
| LPC1768 | Yes |
+---------+-----+

Note: Our simple configuration with one LPC1768 connected to host PC.

Building all default tests (no peripheral tests included)

$ singletest.py --auto -j 8 -O
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected LPC1768, port: COM77, mounted: H:
Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building project DETECT (LPC1768, ARM)
Building project DEV_NULL (LPC1768, ARM)
Building project HELLO (LPC1768, ARM)
Building project TICKER (LPC1768, ARM)
Building project CPP (LPC1768, ARM)
Building project RTC (LPC1768, ARM)
Building project STDIO (LPC1768, ARM)
Building project SEMIHOST (LPC1768, ARM)
Building project TICKER_2 (LPC1768, ARM)
Building project TIMEOUT (LPC1768, ARM)
Building project TIME_US (LPC1768, ARM)
Building project DIV (LPC1768, ARM)
Building project TICKER_3 (LPC1768, ARM)
Building project BASIC (LPC1768, ARM)
Building project VTOR_RELOC (LPC1768, ARM)
Building project FILE (LPC1768, ARM)
Building project CALL_BEFORE_MAIN (LPC1768, ARM)
Building project ECHO (LPC1768, ARM)
Building project BUS_OUT (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project BASIC (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project MUTEX (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project SEMAPHORE (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project SIGNALS (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project QUEUE (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project MAIL (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project TIMER (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project ISR (LPC1768, ARM)
Completed in 1.39 sec

Note: We have built only basic tests (just like for option -C)

Using option -p to force peripheral on all auto-detected devices

$ singletest.py --auto -j 8 -O -P -p SD
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected LPC1768, port: COM77, mounted: H:
Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building library FAT (LPC1768, ARM)
Building project SD (LPC1768, ARM)
Building library FAT (LPC1768, ARM)
Building project SD_PERF_STDIO (LPC1768, ARM)
Building library FAT (LPC1768, ARM)
Building project SD_PERF_FHANDLE (LPC1768, ARM)
Building library FAT (LPC1768, ARM)
Building project SD_PERF_FATFS (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building library FAT (LPC1768, ARM)
Building project FILE (LPC1768, ARM)
Completed in 0.51 sec

Note: singletest.py built all tests related to SD card (option -p) and only allowed peripheral tests to be included (already existing option `-P). Option-O`` used to only build, no test execution on hardware.

Building tests for few types of peripherals

$ singletest.py --auto -j 8 -O -P -p digital_loop,TMP102
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected LPC1768, port: COM77, mounted: H:
Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building project I2C_TMP102 (LPC1768, ARM)
Building project DIGITALIN_DIGITALOUT (LPC1768, ARM)
Building project DIGITALINOUT (LPC1768, ARM)
Building project INTERRUPTIN (LPC1768, ARM)
Completed in 0.27 sec

Note: We've only built tests related to digital_loop and TMP102 peripherals.

Let's run some tests related to this configuration:

$ singletest.py --auto -P -p digital_loop,TMP102
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected LPC1768, port: COM77, mounted: H:
Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building project I2C_TMP102 (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_A4::I2C TMP102 [OK] in 2.28 of 10 sec
Building project DIGITALIN_DIGITALOUT (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_A5::DigitalIn DigitalOut [OK] in 2.26 of 10 sec
Building project DIGITALINOUT (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_A6::DigitalInOut [OK] in 2.32 of 10 sec
Building project INTERRUPTIN (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_A7::InterruptIn [OK] in 9.42 of 15 sec
Test summary:
+--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
| Result | Target  | Toolchain | Test ID | Test Description     | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
| OK     | LPC1768 | ARM       | MBED_A4 | I2C TMP102           |        2.28        |       10      |  1/1  |
| OK     | LPC1768 | ARM       | MBED_A5 | DigitalIn DigitalOut |        2.26        |       10      |  1/1  |
| OK     | LPC1768 | ARM       | MBED_A6 | DigitalInOut         |        2.32        |       10      |  1/1  |
| OK     | LPC1768 | ARM       | MBED_A7 | InterruptIn          |        9.42        |       15      |  1/1  |
+--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
Result: 4 OK

Completed in 28.41 sec

Note: We've only executed tests related to digital_loop and TMP102 peripherals.

Option -P for auto-detection doesn't work (if no -p defined)

$ singletest.py --auto -P
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected LPC1768, port: COM77, mounted: H:
Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Test summary:
+--------+--------+-----------+---------+------------------+--------------------+---------------+-------+
| Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+--------+-----------+---------+------------------+--------------------+---------------+-------+
+--------+--------+-----------+---------+------------------+--------------------+---------------+-------+
Result:

Completed in 0.17 sec

Note: By default there are no peripherals attached to MUT because there is no extra peripheral information.

Auto-detection mode --auto silently assumes we are using -C switch to only use default tests (no peripheral association)

$ singletest.py --auto -C
Test summary:
+---------+---------+-----------+-------------+---------------------------------------+--------------------+---------------+-------+
| Result  | Target  | Toolchain | Test ID     | Test Description                      | Elapsed Time (sec) | Timeout (sec) | Loops |
+---------+---------+-----------+-------------+---------------------------------------+--------------------+---------------+-------+
| OK      | LPC1768 | ARM       | DTCT_1      | Simple detect test                    |        1.33        |       10      |  1/1  |
| OK      | LPC1768 | ARM       | EXAMPLE_1   | /dev/null                             |        4.44        |       20      |  1/1  |
.
.
.
| OK      | LPC1768 | ARM       | RTOS_7      | Timer                                 |       12.35        |       15      |  1/1  |
| OK      | LPC1768 | ARM       | RTOS_8      | ISR (Queue)                           |        7.43        |       20      |  1/1  |
+---------+---------+-----------+-------------+---------------------------------------+--------------------+---------------+-------+
Result: 25 OK / 2 TIMEOUT

Completed in 320.48 sec

Note: Option -C used with --auto mode doesn't do anything (assumed silently if no other things specified).

Example

Let's connect two devices with connected peripherals (some peripherals will be shared between this devices).
I've connected:

• LPC1768 with "TMP102", "digital_loop", "port_loop", "analog_loop", "SD" peripherals.
• KL25Z with: "SD", "24LC256", "KL25Z" peripherals.
  Note we share few the same peripherals across those two boards.

Let's first detect them (no peripherals can be detected with --auto option) but I know hardware configuration I'm using:

$ singletest.py --auto --config
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected KL25Z, port: COM89, mounted: E:
MBEDLS: Detected LPC1768, port: COM77, mounted: H:
MUTs configuration in auto-detected:
+-------+-------------+---------+------+-------+
| index | peripherals | mcu     | disk | port  |
+-------+-------------+---------+------+-------+
| 1     |             | KL25Z   | E:   | COM89 |
| 2     |             | LPC1768 | H:   | COM77 |
+-------+-------------+---------+------+-------+

Test specification in auto-detected:
+---------+-----+
| mcu     | ARM |
+---------+-----+
| KL25Z   | Yes |
| LPC1768 | Yes |
+---------+-----+

Now we can just use our experience and mix few options to just execute two tests across all connected platforms:

$ singletest.py --auto -p SD -n MBED_11,PERF_1
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected KL25Z, port: COM89, mounted: E:
MBEDLS: Detected LPC1768, port: COM77, mounted: H:
Building library CMSIS (KL25Z, ARM)
Building library MBED (KL25Z, ARM)
Building project TICKER (KL25Z, ARM)
TargetTest::KL25Z::ARM::MBED_11::Ticker Int [OK] in 11.37 of 15 sec
Building library FAT (KL25Z, ARM)
Building project SD_PERF_STDIO (KL25Z, ARM)
TargetTest::KL25Z::ARM::PERF_1::SD Stdio R/W Speed [OK] in 5.43 of 15 sec
Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building project TICKER (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.24 of 15 sec
Building library FAT (LPC1768, ARM)
Building project SD_PERF_STDIO (LPC1768, ARM)
TargetTest::LPC1768::ARM::PERF_1::SD Stdio R/W Speed [OK] in 7.67 of 15 sec
Test summary:
+--------+---------+-----------+---------+--------------------+--------------------+---------------+-------+
| Result | Target  | Toolchain | Test ID | Test Description   | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+---------+--------------------+--------------------+---------------+-------+
| OK     | KL25Z   | ARM       | MBED_11 | Ticker Int         |       11.37        |       15      |  1/1  |
| OK     | KL25Z   | ARM       | PERF_1  | SD Stdio R/W Speed |        5.43        |       15      |  1/1  |
| OK     | LPC1768 | ARM       | MBED_11 | Ticker Int         |       12.24        |       15      |  1/1  |
| OK     | LPC1768 | ARM       | PERF_1  | SD Stdio R/W Speed |        7.67        |       15      |  1/1  |
+--------+---------+-----------+---------+--------------------+--------------------+---------------+-------+
Result: 4 OK

Completed in 55.57 sec

And execute one test from common test domain (MBED_11, one from peripheral connected to and available on both platforms (PERF_1), and one only for KL25Z platform (KL25Z_4).

$ singletest.py --auto -p SD -n MBED_11,PERF_1,KL25Z_5
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected KL25Z, port: COM89, mounted: E:
MBEDLS: Detected LPC1768, port: COM77, mounted: H:
Building library CMSIS (KL25Z, ARM)
Building library MBED (KL25Z, ARM)
Building project I2C_MMA8451Q (KL25Z, ARM)
TargetTest::KL25Z::ARM::KL25Z_5::MMA8451Q accelerometer [OK] in 12.55 of 15 sec
Building project TICKER (KL25Z, ARM)
TargetTest::KL25Z::ARM::MBED_11::Ticker Int [OK] in 11.37 of 15 sec
Building library FAT (KL25Z, ARM)
Building project SD_PERF_STDIO (KL25Z, ARM)
TargetTest::KL25Z::ARM::PERF_1::SD Stdio R/W Speed [OK] in 5.42 of 15 sec
Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building project TICKER (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.23 of 15 sec
Building library FAT (LPC1768, ARM)
Building project SD_PERF_STDIO (LPC1768, ARM)
TargetTest::LPC1768::ARM::PERF_1::SD Stdio R/W Speed [OK] in 7.65 of 15 sec
Test summary:
+--------+---------+-----------+---------+------------------------+--------------------+---------------+-------+
| Result | Target  | Toolchain | Test ID | Test Description       | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+---------+------------------------+--------------------+---------------+-------+
| OK     | KL25Z   | ARM       | KL25Z_5 | MMA8451Q accelerometer |       12.55        |       15      |  1/1  |
| OK     | KL25Z   | ARM       | MBED_11 | Ticker Int             |       11.37        |       15      |  1/1  |
| OK     | KL25Z   | ARM       | PERF_1  | SD Stdio R/W Speed     |        5.42        |       15      |  1/1  |
| OK     | LPC1768 | ARM       | MBED_11 | Ticker Int             |       12.23        |       15      |  1/1  |
| OK     | LPC1768 | ARM       | PERF_1  | SD Stdio R/W Speed     |        7.65        |       15      |  1/1  |
+--------+---------+-----------+---------+------------------------+--------------------+---------------+-------+
Result: 5 OK

Completed in 73.61 sec