Skip to content

Add documentation for unit testing #677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Sep 4, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -722,6 +722,10 @@
"type": "markdown",
"url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/offline/cli-test-debug.md"
},
{
"type": "markdown",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we've removed this page.

"url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/offline/cli-unittest.md"
},
{
"type": "markdown",
"url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/offline/mbedignore.md"
Expand Down Expand Up @@ -784,6 +788,10 @@
"type": "markdown",
"url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/testing_intro.md"
},
{
"type": "markdown",
"url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/unit_testing.md"
},
{
"type": "markdown",
"url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/testing.md"
Expand Down
58 changes: 58 additions & 0 deletions docs/tools/offline/cli-unittest.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
## Unit testing
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this should be its own page. I think it would make more sense in the CLI test and debug section.


Use the `mbed unittest` command to build and run unit tests, or to generate files for new unit tests.

Build and run unit tests with `mbed unittest`. The arguments are:

* `--skip-build` to skip building unit tests.
* `--skip-run` to skip running unit tests.
* `--clean` to clean previous build data.
* `-d` or `--debug` to prepare debug build.
* `--coverage <TYPE>` to generate code coverage report where TYPE can be "html", "xml" or "both".
* `-m <NAME>` or `--make-program <NAME>` to select which make build tool to use where NAME can be "make", "gmake", "mingw32-make" or "ninja".
* `-g <NAME>` or `--generator <NAME>` to select which CMake generator to use where NAME can be "Unix Makefiles", "MinGW Makefiles" or "Ninja".
* `-r <EXPRESSION>` or `--regex <EXPRESSION>` to run tests matching the regular expression.
* `--build-path <PATH>` to specify build path.
* `-v` or `--verbose` for verbose diagnostic output.

Generate files for a new unit test with `mbed unittest --new <FILE>`.

### Building and running unit tests

You can specify to only **build** the unit tests by using the `--skip-run` option:

```
$ mbed unittest --skip-run
```

You can specify to only **run** the unit tests by using the `--skip-build` option:

```
$ mbed unittest --skip-build
```

If you do not specify any of these, `mbed unittest` will build all available unit tests and run them.

### Running a subset of tests

You can run a **limited set** of unit tests by using the `-r` or `--regex` option. This takes a regular expression, which it compares against the test names. For example to run all cellular unit tests you can specify:

```
$ mbed unittest -r cellular
```

### Getting code coverage

You can generate a code coverage report by using the `--coverage` option. For example to create an html report you can specify:

```
$ mbed unittest --coverage html
```

### Creating new unit tests

All unit tests are under `mbed-os/UNITTESTS` directory. You can **generate** the necessary files for a unit test by using the `--new` option. For example to create the files for `rtos/Semaphore.cpp` you can specify:

```
$ mbed unittest --new rtos/Semaphore.cpp
```
2 changes: 2 additions & 0 deletions docs/tools/testing/testing_intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Testing is a critical step in the development process. The Arm Mbed ecosystem offers several tools to help you test your code.

Unit testing tools can be used to build and run Mbed OS unit tests. Each unit test is built into an isolated test executable using a separate build system and build tools native to the development machine without any hardware or software dependencies. The tool can be used with Mbed CLI using the keyword `unittest`.

Greentea, `htrun` and `mbed-ls` are testing tools written in Python. Greentea tests serve as functional unit tests in C++, as well as integration tests for complex use cases that execute on microcontrollers. The Mbed CLI tool has a verb `test` that drives these tools to form a testing system. These comprise our automated testing framework for Mbed OS development.

The testing system automates the process of flashing Mbed boards, driving the tests and accumulating test results into test reports. Developers and Mbed Partners use this system for local development, as well as for automation in a Continuous Integration environment.
Expand Down
47 changes: 47 additions & 0 deletions docs/tools/testing/unit_testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
## Unit testing

### Introduction

Traditional software testing is defined into three main levels: unit testing, integration testing and system testing. These levels are often pictured as a pyramid to indicate the amount of testing per level.

```
^ Testing level
|
| /\
| / \ System testing
| /----\
| / \ Integration testing
| /--------\
| / \ Unit testing
| /------------\
|
*-------------------> Amount of tests
```

Integration and system testing are performed in an environment where the tests are run with full Mbed OS. Other testing tools for Mbed OS require specific hardware and whole Mbed OS to be built, which means traditional unit testing is not possible.

Unit testing takes place in a build environment where each C/C++ class or module is tested in isolation. This means test suites are built into separate test binaries and all access outside is stubbed to remove dependency of any specific hardware or software combination. This allows the testing to be done quickly using native compilers on the build machine.

### Using unit tests

#### Test code structure

Unit tests are located in Mbed OS repository under `UNITTESTS`. Each unit test should use identical directory tree structure to the file to be tested. This makes it easier to find unit tests for a particular class or a module. For example if the file to be tested is `rtos/Semaphore.cpp` then all the test files should be in `UNITTESTS/rtos/Semaphore` directory.

##### Test discovery

Registering unit tests for running is automatic and handled by the test runner, but test files are not automatically assigned to be built. Unit tests are built using a separate build system, which will search for unit tests under `UNITTESTS` directory.

For the build system to find and build any test suite automatically, a unit test configuration file named `unittest.cmake` is required to be included with each unit test. This configuration file contains a name for the test and other source files required for the build.

##### Test names

Each test suite requires a name to be configured in the `unittest.cmake` file. This name is used for generated files and when running a subset of tests.

#### Building, running and writing tests

Unit testing requires external tools which need to be installed. See the developer [documentation](https://github.com/ARMmbed/mbed-os/blob/master/UNITTESTS/README.md) in GitHub for installation guide.

You can build and run unit tests through Arm Mbed CLI. The tool can also be used to generate new test files. For information on using Mbed CLI, please see the [CLI documentation](/docs/development/tools/arm-mbed-cli.html).

For more information about the framework, the build process or how to write unit tests, see the GitHub documentation.