Skip to content

Tools: bugfix: singletes.py was not reporting test case build failure #1087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 4, 2015
Merged

Tools: bugfix: singletes.py was not reporting test case build failure #1087

merged 3 commits into from
May 4, 2015

Conversation

PrzemekWirkus
Copy link
Contributor

Description

  1. singletest.py was not reporting in summary table or other reports test case build failure. Thi issue was observed in team's internal CI server.
  2. Added new error code to sibgletst.py called BUILD_FAILURE - used when test case build fails.
  3. Added new summary: Build successful, skipped and failed - the same summary as in build.py.

Example of test case build failure:

.
.
Building project DETECT (K64F, ARM)
Compile: main.cpp
[Error] main.cpp@4:  #35: #error directive: Error added for test
[23:19:24] Error: There were errors while building project DTCT_1
.
.

Example of fixed behavior where singletest.py is fetching information about failed test case build and passing to results tables (also JUnit and HTML reports).
In below example test DICT_1 is failing to build and summary is enriched with BUILD_FAILURE

$ singletest.py --auto -j 8 -n DTCT_1,RTOS_1 --report-html hr.html
MBEDLS: Detecting connected mbed-enabled devices...
MBEDLS: Detected K64F, port: COM61, mounted: I:
Building library CMSIS (K64F, ARM)
Building library MBED (K64F, ARM)
Building library RTOS (K64F, ARM)
Building library RTX (K64F, ARM)
Building project DETECT (K64F, ARM)
Compile: main.cpp
[Error] main.cpp@4:  #35: #error directive: Error added for test
[23:19:24] Error: There were errors while building project DTCT_1
Building project BASIC (K64F, ARM)
TargetTest::K64F::ARM::RTOS_1::Basic thread [OK] in 11.33 of 15 sec
Test summary:
+--------------+--------+-----------+---------+------------------------+--------------------+---------------+-------+
| Result       | Target | Toolchain | Test ID | Test Description       | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------------+--------+-----------+---------+------------------------+--------------------+---------------+-------+
| BUILD_FAILED | K64F   | ARM       | DTCT_1  | Toolchain build failed |         0          |       0       |   -   |
| OK           | K64F   | ARM       | RTOS_1  | Basic thread           |       11.33        |       15      |  1/1  |
+--------------+--------+-----------+---------+------------------------+--------------------+---------------+-------+
Result: 1 BUILD_FAILED / 1 OK

Completed in 20.83 sec

Build successes:
  * ARM::K64F

Build failures:
  * ARM::K64F::DTCT_1

Synd with master (27 April 2015)
Added build report after build completion.

Fixed result from execute() capture when result is NoneType value
@@ -479,17 +518,17 @@ def execute_thread_slice(self, q, target, toolchains, clean, test_ids, build_rep
test_suite_properties['test.path.%s.%s.%s'% (target, toolchain, test_id)] = path

# read MUTs, test specification and perform tests
single_test_result, detailed_test_results = self.handle(test_spec, target, toolchain, test_loops=test_loops)
handle_result = self.handle(test_spec, target, toolchain, test_loops=test_loops)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will fix:

[DEBUG] Command: C:/mbed_tools/ARMCompiler_5.03_117_Windows\bin\fromelf --bin -o C:\Jenkins\jobs\mbed_2.0_build_release_test_publish\workspace\build\test\NRF51822\ARM\DTCT_1\detect.hex C:\Jenkins\jobs\mbed_2.0_build_release_test_publish\workspace\build\test\NRF51822\ARM\DTCT_1\detect.elf
[DEBUG] Return: 0
[DEBUG] Merge SoftDevice file s110_nrf51822_8.0.0_softdevice.hex
Traceback (most recent call last):
  File "./workspace_tools/singletest.py", line 237, in <module>
    singletest_in_cli_mode(single_test)
  File "C:\Jenkins\jobs\mbed_2.0_build_release_test_publish\workspace\workspace_tools\test_api.py", line 1419, in singletest_in_cli_mode
    test_summary, shuffle_seed, test_summary_ext, test_suite_properties_ext, build_report = single_test.execute()
  File "C:\Jenkins\jobs\mbed_2.0_build_release_test_publish\workspace\workspace_tools\test_api.py", line 543, in execute
    self.execute_thread_slice(q, target, toolchains, clean, test_ids, cur_build_report)
  File "C:\Jenkins\jobs\mbed_2.0_build_release_test_publish\workspace\workspace_tools\test_api.py", line 482, in execute_thread_slice
    single_test_result, detailed_test_results = self.handle(test_spec, target, toolchain, test_loops=test_loops)
TypeError: 'NoneType' object is not iterable

when None object is passed as port or mount point from e.g. mbed-ls tools

0xc0170 added a commit that referenced this pull request May 4, 2015
Tools: bugfix - singletest.py was not reporting test case build failure
@0xc0170 0xc0170 merged commit 8dc5e44 into ARMmbed:master May 4, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants