Skip to content

Support for unit testing #734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Sep 4, 2018
Merged

Support for unit testing #734

merged 7 commits into from
Sep 4, 2018

Conversation

lorjala
Copy link
Contributor

@lorjala lorjala commented Aug 20, 2018

Add support for Mbed OS unit testing.

related mbed-os PR: ARMmbed/mbed-os#7819

Copy link
Contributor

@bridadan bridadan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments to help this command be more consistent with the existing ones.

mbed/mbed.py Outdated
if os.path.exists(tool):
popen([python_cmd, tool]
+ (["--skip-build"] if skip_build else [])
+ (["--skip-run"] if skip_run else [])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using --skip-build and --skip-run, could you use --compile and --run like the existing mbed test command? So it's an "opt-in" instead of "opt-out". It would make the testing experience more parallel with what already exists.

mbed/mbed.py Outdated
@subcommand('unittest',
dict(name='--skip-build', action='store_true', help='skip build step'),
dict(name='--skip-run', action='store_true', help='skip run step'),
dict(name='--clean', action='store_true', help='clean build data'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add the short version -c? This is present in the other commands that build code.

mbed/mbed.py Outdated
dict(name=['-m', '--make-program'], choices=['gmake', 'make', 'mingw32-make', 'ninja'], help='which make program to use'),
dict(name=['-g', '--generator'], choices=['Unix Makefiles', 'MinGW Makefiles', 'Ninja'], help='which CMake generator to use'),
dict(name=['-r', '--regex'], help='run tests matching regular expression'),
dict(name='--build-path', help='specify build path'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please rename this to just --build to be inline with the other mbed commands?

Copy link
Contributor

@bridadan bridadan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making those changes! I have one final question about the default build path.

mbed/mbed.py Outdated
dict(name=['-m', '--make-program'], choices=['gmake', 'make', 'mingw32-make', 'ninja'], help='Which make program to use'),
dict(name=['-g', '--generator'], choices=['Unix Makefiles', 'MinGW Makefiles', 'Ninja'], help='Which CMake generator to use'),
dict(name=['-r', '--regex'], help='Run tests matching regular expression'),
dict(name='--build', help='Build directory. Default: mbed-os/UNITTESTS/build/'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This default build directory is outside of the normal root BUILD directory, is this something that has already been agreed upon? @theotherjimmy

Copy link
Contributor Author

@lorjala lorjala Aug 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since unit tests use completely different build environment, they will not be part of any mbed-os application and the tools used are separate from other mbed-os tools it would be better to keep the build folders separate in my opinion.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lorjala I'm concerned that your build directory will be scanned by the build tools. How are you avoiding that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@theotherjimmy UNITTESTS directory has .mbedignore file, but it could be a possibility to add .mbedignore into the build directory as well during the folder creation if that is the concern.

Copy link
Contributor

@theotherjimmy theotherjimmy Aug 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 That will work. I have the same concern as Brian: why change the default? it has the potential to confuse users, and I can see no upside.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😉 Defaults to BUILD/unittests for now. Let's see how it works.

Copy link
Contributor

@bridadan bridadan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks for the changes!

@AnotherButler
Copy link
Contributor

@theotherjimmy @bridadan Could one of you please merge this?

@screamerbg
Copy link
Contributor

screamerbg commented Sep 3, 2018

@lorjala @theotherjimmy @bridadan @MarceloSalazar Why not have this under mbed test? E.g. PR #731 is introducing a different test framework for system testing under mbed test.

@screamerbg screamerbg self-requested a review September 3, 2018 10:12
Copy link
Contributor

@screamerbg screamerbg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment

@lorjala
Copy link
Contributor Author

lorjala commented Sep 3, 2018

@AnotherButler
Copy link
Contributor

AnotherButler commented Sep 4, 2018

@screamerbg Is this OK to merge?

@theotherjimmy
Copy link
Contributor

@screamerbg

from @OPpuolitaival's linked comment

we need separated command because:
"mbed test" - testing code which runs on embedded device and therefore need a target and toolchain
"mbed unittest" - testing code units in host computer using documented compilers

I think that's a satisfactory explanation. Perhaps you would like to use a different subcommand?

@screamerbg
Copy link
Contributor

@theotherjimmy Yeah. I'm not happy that unittesting is very internal to Mbed OS development as a product, which won't apply for 99.9% of the Mbed OS users.

@theotherjimmy
Copy link
Contributor

@lorjala @OPpuolitaival We have mbed test for everything testing. Greentea tests and Icetea tests are very different. Greentea and Unittests are also very different. Why do unittests get a seperate subcommand and icetea did not? Why is unittesting logically not testing? I think mbed test --unit or mbed test --unittest would be better here.

Copy link
Contributor

@theotherjimmy theotherjimmy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^

@theotherjimmy
Copy link
Contributor

I'm also open to mbed test --host-only or similar.

Copy link
Contributor

@theotherjimmy theotherjimmy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated subcommand.

@screamerbg
Copy link
Contributor

LGTM. Great work @theotherjimmy!

@screamerbg
Copy link
Contributor

@theotherjimmy Merge at will

@screamerbg screamerbg merged commit 29dfd5b into ARMmbed:master Sep 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants