Unit testing framework #7819

lorjala · 2018-08-17T13:51:51Z

Description

Unit testing framework for writing, building and running Mbed OS unit tests for developers and contributors.

Separate test binaries
No HW and SW dependencies for testable code
Linux, Mac OS and Windows

MBEDOSTEST-3

Pull request type

[ ] Fix
[ ] Refactor
[ ] Target update
[x] Feature
[ ] Breaking change

0xc0170 · 2018-08-20T12:06:43Z

UNITTESTS/README.md

+static uint32_t count = 0;
+
+// Test stubs
+osStatus_t osSemaphoreAcquire (osSemaphoreId_t semaphore_id, uint32_t timeout) { return retval; }


examples should follow our coding style (please run astyle on this code and update the example)

0xc0170 · 2018-08-20T12:07:44Z

UNITTESTS/features/netsocket/NetworkInterface/test_NetworkInterface.cpp

+{
+    char *n = 0;
+    EXPECT_EQ(iface->get_mac_address(), n);
+}


No newline in t hese files? Leave one please

0xc0170 · 2018-08-20T12:08:56Z

UNITTESTS/README.md

+### Testing with Python script:
+
+```
+./UNITTESTS/mbed_unittest.py test


Is this how we want to introduce this tools ? Shouldn't this be mbed unittest rather? and this scripts to be moved into tools/ folder

0xc0170 · 2018-08-20T12:13:55Z

UNITTESTS/stubs/mbed_shared_queues.cpp

@@ -0,0 +1,13 @@
+#include "events/mbed_shared_queues.h"


Add license headers to the new files, like this one

0xc0170 · 2018-08-20T12:18:08Z

UNITTESTS/stubs/EventFlags.cpp

@@ -0,0 +1,24 @@
+#include "rtos/EventFlags.h"


ATHandler_stub - has suffix, but some of the stubs do not have, like this one. Is there a naming convention?

Few example unit tests are from features/cellular/UNITTESTS without major changes to demonstrate how to convert existing unit tests to use this framework. So ATHandler_stub for example is from these existing tests which uses that naming convention. These unit tests introduced in this PR are not final in any form including naming. Their purpose is only to showcase the framework before proper unit tests are written/converted.

deepikabhavnani · 2018-08-20T15:04:42Z

UNITTESTS/README.md

+### Testing with Mbed CLI
+
+```
+mbed unittest test


Please describe what will be the behavior of new code when executing mbed compile. All C/CPP files inside unittest folder will not be build or will be skipped during linking step?

I am now reviewing Icetea addition. So back to this mbed unittest comment I had earlier today.
Icetea has it as mbed test -m <target> -t <toolchain> --icetea . This should follow it ? as mbed test -m <target> -t <toolchain> --unittest or just this unittest

Just looking for alignment (icetea / unittest/ app test) - if there can be any. Adding my comment here to this line (hijacking @deepikabhavnani question that is however related)

@OPpuolitaival Please review

No, the unittesting is basically target and toolchain agnostic.
It runs the one provided by the host OS.

So it is so different from what mbed test is targeting, that it requires its own command.

@SeppoTakalo is right, we need separated command because:
"mbed test" - testing code which runs on embedded device and therefore need a target and toolchain
"mbed unittest" - testing code units in host computer using documented compilers

+1, own command

OPpuolitaival · 2018-08-20T16:04:48Z

UNITTESTS/README.md

+### Testing with Mbed CLI
+
+```
+mbed unittest test


@SeppoTakalo is right, we need separated command because:
"mbed test" - testing code which runs on embedded device and therefore need a target and toolchain
"mbed unittest" - testing code units in host computer using documented compilers

OPpuolitaival · 2018-08-20T16:07:05Z

UNITTESTS/README.md

+
+Traditional software testing is defined into three main levels: unit testing, integration testing and system testing. These levels are often pictured as a pyramid to indicate the amount of testing per level.
+```
+^ Testing level


@SeppoTakalo should we move this picture in testing introduction in Handbook?

OPpuolitaival · 2018-08-20T16:07:43Z

UNITTESTS/README.md

+    * GCC 6 or later
+    * MinGW-W64 GCC-6.4.0 or MinGW-W64 GCC-7.3.0 (Windows)
+* CMake 3.0+ installed.
+* Python and pip installed.


Which python versions we support? Need to define. And maybe pip also. Something like python 2.7.x or >3.5

OPpuolitaival · 2018-08-20T16:08:08Z

UNITTESTS/README.md

+    * MinGW-W64 GCC-6.4.0 or MinGW-W64 GCC-7.3.0 (Windows)
+* CMake 3.0+ installed.
+* Python and pip installed.
+* gcovr and optionally virtualenv pip modules installed.


gcovr version definition?

OPpuolitaival · 2018-08-20T16:12:01Z

UNITTESTS/README.md

+mbed unittest test
+```
+
+##### Parameters to mbed unittest test


I don't feel good to list these here because these are coming from mbed-cli and mbed-cli works with any version of Mbed OS. Then we don't have guarantee that these are like this forever.

OPpuolitaival · 2018-08-20T16:14:18Z

UNITTESTS/README.md

+mbed unittest new
+```
+
+##### Parameters to mbed unittest new


Don't list parameters here because there is no guarantee. Just command and why it exist is good enough

SeppoTakalo · 2018-08-21T08:57:01Z

Is it a bit too early to describe mbed unittest command here?
First we need this in,
then we can add the command into mbed-cli.

Also, don't remove anything from the README.md, once ready, we just submit part of it into Mbed OS handbook.

lorjala · 2018-08-21T09:53:16Z

Referencing Mbed CLI here should be ok, because it is mentioned elsewhere in the repo as well(case greentea), and it would not take long to get new mbed-cli release out (PR is ongoing).

SeppoTakalo · 2018-08-21T11:52:58Z

OK.
Thanks.
PR already here ARMmbed/mbed-cli#734

0xc0170

@theotherjimmy Please check the location for the scripts here (in the UNITTESTS folder rather than tools/unittests or similar). If that is fine.

cmonr · 2018-08-24T04:37:27Z

First we need this in, then we can add the command into mbed-cli.

Not really. Both PRs can be active in parallel, and we can even merge one before another, as long as we make sure they're released together. But it looks like y'all are already moving forward with that.

OPpuolitaival · 2018-08-24T12:58:31Z

@cmonr you are right. Mbed-cli need to work for many Mbed OS version which means that it works also without unit tests in repository. PRs can be merged in any order

lorjala · 2018-08-27T11:41:15Z

@0xc0170 @cmonr can you trigger the build?

0xc0170 · 2018-08-27T12:07:14Z

@0xc0170 @cmonr can you trigger the build?

I would like to have first all approvals (currently one requested changes) and someone from @ARMmbed/mbed-os-test reviewed

OPpuolitaival · 2018-08-27T12:15:05Z

@0xc0170 I accepted this

0xc0170 · 2018-08-27T12:16:38Z

/morph build

mbed-ci · 2018-08-27T15:16:44Z

Build : SUCCESS

Build number : 2923
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/7819/

Triggering tests

/morph test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-08-27T17:32:20Z

Exporter Build : SUCCESS

Build number : 2543
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/7819/

jamesbeyond · 2018-08-27T19:33:23Z

UNITTESTS/README.md

+```
+virtualenv pyenv
+pyenv\\Scripts\\activate
+pip install 'gcovr>=4.1'


All looks good, well done! I give a quick try, everything works just fine, apart from this line.
Seems on my windows 10, pip not like the single quotes, but only works with double quotes. please confirm @lorjala

Yes, single quotes do not work when using default command prompt, but they do with powershell.

mbed-ci · 2018-08-28T02:40:56Z

Test : SUCCESS

Build number : 2674
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/7819/2674

lorjala · 2018-08-28T06:46:43Z

@0xc0170 @cmonr This can be merged (trigger the build again if necessary).

cmonr · 2018-08-28T13:57:11Z

/morph build

mbed-ci · 2018-08-28T14:56:51Z

Build : SUCCESS

Build number : 2942
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/7819/

Triggering tests

/morph test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-08-28T16:52:39Z

Exporter Build : SUCCESS

Build number : 2559
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/7819/

mbed-ci · 2018-08-29T07:04:40Z

Test : SUCCESS

Build number : 2691
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/7819/2691

SeppoTakalo approved these changes Aug 17, 2018

View reviewed changes

cmonr requested review from NeilJackson-AlifSemi, a user and sg- August 17, 2018 22:51

cmonr added the needs: review label Aug 17, 2018

0xc0170 requested changes Aug 20, 2018

View reviewed changes

0xc0170 added needs: work and removed needs: review labels Aug 20, 2018

lorjala mentioned this pull request Aug 20, 2018

Support for unit testing ARMmbed/mbed-cli#734

Merged

deepikabhavnani reviewed Aug 20, 2018

View reviewed changes

OPpuolitaival suggested changes Aug 20, 2018

View reviewed changes

lorjala mentioned this pull request Aug 22, 2018

Add documentation for unit testing ARMmbed/mbed-os-5-docs#677

Merged

0xc0170 requested a review from a team August 22, 2018 12:14

cmonr added the risk: G label Aug 23, 2018

0xc0170 approved these changes Aug 23, 2018

View reviewed changes

cmonr added risk: A and removed risk: G labels Aug 24, 2018

Add unit testing framework

4dde13d

OPpuolitaival approved these changes Aug 27, 2018

View reviewed changes

0xc0170 added needs: CI and removed needs: work labels Aug 27, 2018

cmonr added risk: G and removed risk: A labels Aug 27, 2018

cmonr added the release-version: 5.10.0-rc1 label Aug 27, 2018

jamesbeyond reviewed Aug 27, 2018

View reviewed changes

Unittests: update documentation for Windows

4cf98dd

NirSonnenschein added ready for merge and removed needs: CI labels Aug 29, 2018

cmonr merged commit 7ecf111 into ARMmbed:master Aug 29, 2018

0xc0170 removed the ready for merge label Aug 29, 2018

lorjala deleted the unittests branch August 31, 2018 09:21

cmonr removed the risk: G label Sep 2, 2018

theotherjimmy mentioned this pull request Sep 4, 2018

Document Unit Testing ARMmbed/mbed-os-5-docs#703

Merged

Unit testing framework #7819

Unit testing framework #7819

Uh oh!

Conversation

lorjala commented Aug 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Pull request type

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xc0170 Aug 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SeppoTakalo commented Aug 21, 2018

Uh oh!

lorjala commented Aug 21, 2018

Uh oh!

SeppoTakalo commented Aug 21, 2018

Uh oh!

0xc0170 left a comment

Choose a reason for hiding this comment

Uh oh!

cmonr commented Aug 24, 2018

Uh oh!

OPpuolitaival commented Aug 24, 2018

Uh oh!

lorjala commented Aug 27, 2018

Uh oh!

0xc0170 commented Aug 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

OPpuolitaival commented Aug 27, 2018

Uh oh!

0xc0170 commented Aug 27, 2018

Uh oh!

mbed-ci commented Aug 27, 2018

Triggering tests

Uh oh!

mbed-ci commented Aug 27, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbed-ci commented Aug 28, 2018

Uh oh!

lorjala commented Aug 28, 2018

Uh oh!

lorjala commented Aug 17, 2018 •

edited

Loading

0xc0170 Aug 20, 2018 •

edited

Loading

0xc0170 commented Aug 27, 2018 •

edited

Loading