Experimental parallel test execution on MUTs (option --parallel), v2

[PrzemekWirkus]

Description

Added new experimental feature: thread model for function SingleTest::execute() where for each target we have separate thread.

Currently we are only building in parallel mbed SDK and libraries with dependencies.
Each unique MUT (mbed under test) is flashed separately one by one, but with this new feature we are able to flash multiple boards at the same time, run test case on hardware and get results in parallel.

Test execution time is reduced by n where n is number of MUTs (MUT: mbed under test).

Rationale & limitations

We can experiment with usage of --auto and CI systems. Assuming that we are performing testing on n unique mbed platforms we can gain almost n folt test execution acceleration by flashing, executing and grabbing test results from each board at the same time.

Note: this extension works also with --auto feature added by non-public (yet) mbed-ls auto-detection module.

Note: Building still will take some time so we are not accelerating building process which already is accelerated by multi-core compilation feature (-j switch). You can combine -j and --parallel to really messw ith your CPUs silicone ;-)

Note: We want to explore new testing possibilities with existing test suite before we enable them for mbed 3.0 test suite.

Usage

To test in parallel with old sytle of configuration:

$ singletest.py singletest.py -i test_spec.json -M muts_all.json --parallel

To test only detected by mbed-ls boards.

$ singletest.py --auto --tc ARM,GCC_ARM --parallel

Limitations

For now if we connect two boards of the same type, for example two Freedom K46F boards to our system test suite (singletest.py) will assume that we are testing only for one of the targets. Mbed-ls feeds singletest.py with information about all targets but as above only one board of the same type will be tested.
This limitation will be removed in future versions of test suite*.

Using --parallel means that:

• separate thread will be spawned to build mbed SDK and dependencied (libraries)
  for each unique target with all declared toolchains.
  E.g:

        Thread(n==1):   LPC1768:    ARM, uARM, GCC_ARM
        Thread(n==2):   K64F:       ARM, GCC_ARM
        Thread(n==3):   NUCLEO_X:   uARM
        Thread(n==4):   NUCLEO_Y:   uARM
        Thread(n==5):   NUCLEO_Z:   uARM

Example:
n == 5 if 5 types of MUTs connected to host computer:
    E.g. LPC1768, K64F, NUCLEO_X, NUCLEO_Y, NUCLEO_Z.

Thread model for n == 5

    Thread(1): for toolchain in LPC1768[toolchains]
               -> build mbed SDK
                -> build libs
                 -> build project
                  -> run MUT testrunner
                    return test results for LPC1768[toolchain]
    .
    .
    .
    Thread(4): for toolchain in NUCLEO_Z[toolchains]
               -> build mbed SDK
                -> build libs
                 -> build project
                  -> run MUT testrunner
                    return test results for NUCLEO_Z[toolchain]

Done:

1. Added option --parallel.
2. Decoupled execute function so it can be run in parallel with other execute functions.
3. Thread join via Queue, not Thread::Join() to avoid deadlocks or waits for particular thread to finish.
4. All builds are in parallel, but because each target and library for each toolchain have different directory we do not worry about building in parallel and compiler collisions.

Missing:

1. No sync for 'print' (TODO).
2. No sync on test result structures - not needed because we only append to them (?).

Testing

Test MUTs configuration

Configuration, 4 boards; three Nucleo flavors and one Freedom K64F:

$ mbedls
+--------------+------------+------------+-------------------------+
|platform_name |mount_point |serial_port |target_id                |
+--------------+------------+------------+-------------------------+
|K64F          |E:          |COM61       |02400203D94B0E7724B7F3CF |
|NUCLEO_F072RB |H:          |COM84       |073002006230631A5D55F517 |
|NUCLEO_F302R8 |G:          |COM34       |07050200623B61125D5EF72A |
|NUCLEO_F401RE |F:          |COM52       |07200200073E650A385BF317 |
+--------------+------------+------------+-------------------------+

Building stuff first so when I execute parallel tests I do not build stuff (I want to see parallel testing velocity without build time constrains).
Note: singletes.py will still check if mbed SDK and dependencies should be rebuilt but this will be also done when non-parallel test execution is invoked.

$ singletest.py --auto -O
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected K64F, port: COM61, mounted: E:
MBEDLS: Detected NUCLEO_F401RE, port: COM52, mounted: F:
MBEDLS: Detected NUCLEO_F302R8, port: COM34, mounted: G:
MBEDLS: Detected NUCLEO_F072RB, port: COM84, mounted: H:
Building library CMSIS (K64F, ARM)
Building library MBED (K64F, ARM)
Building project DETECT (K64F, ARM)
.
.
.
Building project ECHO (NUCLEO_F072RB, uARM)
Building project BUS_OUT (NUCLEO_F072RB, uARM)
Completed in 7.39 sec

$ singletest.py --auto --config
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected K64F, port: COM61, mounted: E:
MBEDLS: Detected NUCLEO_F401RE, port: COM52, mounted: F:
MBEDLS: Detected NUCLEO_F302R8, port: COM34, mounted: G:
MBEDLS: Detected NUCLEO_F072RB, port: COM84, mounted: H:
MUTs configuration in auto-detected:
+-------+-------------+---------------+------+-------+
| index | peripherals | mcu           | disk | port  |
+-------+-------------+---------------+------+-------+
| 1     |             | K64F          | E:   | COM61 |
| 2     |             | NUCLEO_F401RE | F:   | COM52 |
| 3     |             | NUCLEO_F302R8 | G:   | COM34 |
| 4     |             | NUCLEO_F072RB | H:   | COM84 |
+-------+-------------+---------------+------+-------+

Test specification in auto-detected:
+---------------+-----+------+
| mcu           | ARM | uARM |
+---------------+-----+------+
| K64F          | Yes | -    |
| NUCLEO_F401RE | -   | Yes  |
| NUCLEO_F302R8 | -   | Yes  |
| NUCLEO_F072RB | -   | Yes  |
+---------------+-----+------+

Parallel test execution

$ singletest.py --auto --parallel
.
.
Result: 2 FAIL / 70 OK

Completed in 305.16 sec

Non-parallel test execution

$ singletest.py --auto
.
.
Result: 2 FAIL / 70 OK

Completed in 695.88 sec

Measurement: Parallel vs non-parallel

Test scenario	Board count	Time (sec)
Parallel	4 boards	305.16 sec
Non-parallel	4 boards	695.88 sec
Non-parallel	1 board (K64F)	304.17 sec
Non-parallel	1 board (NUCLEO)	123.44 sec

Execution in parallel (threaded model with --parallel) scheme:

Note: K64F execution is far longer that any Nucleo board, so total time of parallel execution equals to longest test execution streak. In this case it is about 305.16 sec (where 304.17 sec is K64F test execution length):

# +---------------+>
# | NUCLEO_F401RE |> 123.44 sec 
# +---------------+>

# +---------------+>
# | NUCLEO_F401RE |> 123.44 sec
# +---------------+>

# +---------------+>
# | NUCLEO_F401RE |> 123.44 sec
# +---------------+>

# +------------------------------+>
# |             K64F             |> 304.17 sec
# +------------------------------+>

# ------------- time axis (sec) -------------->

Non-parallel execution scheme

Non-parallel execution just takes approximately sum of times for each platform to be executed.

• Measured execution time: 695.88 sec.
• Sum of test execution times per tested board (3 * 123.44 sec + 304.17 sec) = 674.49 sec.
  • which gives around 21.3 sec for test suite non-parallel code.

# +---------------+>
# | NUCLEO_F401RE |> 123.44 sec 
# +---------------+>

#                  +---------------+>
#                  | NUCLEO_F401RE |> 123.44 sec
#                  +---------------+>

#                                  +---------------+>
#                                  | NUCLEO_F401RE |> 123.44 sec
#                                  +---------------+>

#                                                  +-- ~ ------ ~ --+>
#                                                  |   ~  K64F  ~   |> 304.17 sec
#                                                  +-- ~ ------ ~ --+>

# -------------------------- time axis (sec) -------------------------->