Update examples testing scripts #11710

jamesbeyond · 2019-10-18T12:04:59Z

Description

This PR is update the examples testing scripts.
This script is standalone script purely used by mbed CI test, So this will not change mbed-OS behaviours.

This including following changes:

Add a README file to explain how this script is used in the CI
Removed the dependency of Mercurial, This script will only use Github for check out code
Add a function for the script to create symbolic links to mbed-os.
Add a logging tool, allow the STDOUT to be easy to debug
Update the output message, print a nicer result summary

However, because this scripts changed where to checkout examples, some of the examples will failed to build

mbed-os-example-BLE
mbed-os-example-tls
mbed-os-example-crypto

So temporarily turn off the build tests for these examples. An update on the CI scripts will be raised soon. Once the CI update is in place will enable the build test for these examples ASAP.

Pull request type

[ ] Fix
[ ] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[x] Test update
[ ] Breaking change

Reviewers

@OPpuolitaival @evedon @adbridge @mark-edgeworth @AnotherButler

Release Notes

- remove "mbed" and "test-repo-source" key - add "sub-repo-example" key as an indicator if an example has sub examples - add "subs" key to sotre sub examples names

- add logging function - update clone function - update deploy function - update source function - add symlink function

ciarmcom · 2019-10-18T13:00:27Z

@jamesbeyond, thank you for your changes.
@AnotherButler @evedon @mark-edgeworth @adbridge @OPpuolitaival @ARMmbed/mbed-os-tools @ARMmbed/mbed-os-maintainers please review.

OPpuolitaival

Looks great!

OPpuolitaival · 2019-10-18T12:12:59Z

tools/test/examples/README.md

+* **name** - name of the example, should be the base name of the example repository address, will throw if not match
+* **github** - example github repository address
+* **sub-repo-example** specify if the example repository have sub folder for each examples
+* **subs** - array of sub examples names


Which expected to be as folders in root of the repository?

Yes, as above TLS example, it just going to be benchmark tls-client hashing and authcrypt

Edit file, mostly for formatting and grammar.

jamesbeyond · 2019-10-20T19:14:52Z

Thanks for your updates on the README @AnotherButler

0xc0170

0xc0170 · 2019-10-21T08:43:15Z

tools/test/examples/examples.json

-          "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-tls-client",
-          "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-hashing",
-          "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-authcrypt"
+      "sub-repo-example": true,


Nice to have finally support for these. Can this be automated? If this is true, we walk through the root and if mbed-os.json file found, it is an example for testing ? This way we do not need to define subs (this can be out of date easily as everytime someone adds a new sub example or restructure example, this breaks).

Or any other way to do this without having this "subs" in here?

Removing these lines breaks the release script. I will need to consider the impact before the next release.

Nice to have finally support for these. Can this be automated? If this is true, we walk through the root and if mbed-os.json file found, it is an example for testing ? This way we do not need to define subs (this can be out of date easily as everytime someone adds a new sub example or restructure example, this breaks).

Or any other way to do this without having this "subs" in here?

@0xc0170 That is how the release script currently updates the mbed-os.lib files. It works down the root.

@jamesbeyond Actually looking at the release script more carefully I think this change is ok. The script only looks at the 'name' field so as long as that doesn't change we should be ok!

0xc0170 · 2019-10-21T08:44:37Z

So temporarily turn off the build tests for these examples. An update on the CI scripts will be raised soon. Once the CI update is in place will enable the build test for these examples ASAP.

What is the timeframe for this?

jamesbeyond · 2019-10-21T11:25:59Z

What is the timeframe for this?

Those tests should be turned back on within a week after this PR been merged

evedon · 2019-10-22T08:52:11Z

tools/test/examples/README.md

+The scripts in this folder are used for testing `mbed-os` official examples. It contains the following files:
+
+- `examples.py` - the main script that serves as the command-line interface.
+- `examples.json` - the default configuration file for the script, which contains the information to the examples. If required, you can pass a customized configuration file as an argument to the script.


What do you mean by a customized configuration file? Can you show an example on how to use this feature?

I think that we currently don't have a way to test different profiles (bare metal profile/minimal-printf profile) and/or library configurations, do we? Is this PR the first step towards achieving this?

the json template in the README is an example of configuration. This means if other people want to use this in their own CI, they can do it like such. But regarding to mbed-OS main CI, it always uses the default configuration file.

The new scripts actually allow the minimal-printf to be tested, you can use:

python mbed-os/tools/test/examples/examples.py compile GCC_ARM -m <target> --profile develop mbed-os/tools/profile/extensions/minimal-printf.json

evedon · 2019-10-22T08:54:24Z

tools/test/examples/README.md

+Number of failures = 0
+```
+
+After the compilation stage, a `test_spec.json` file is generated. Later, Greentea tests will consume this file. They test the compiled example on hardware platform. 


Can this test_spec.json also include some specific test configuration settings, for example if we want to test bare metal?

this test_spec.json is generated after the compilation is done. it contains the if where is the binary file, where is the compare_log. in terms of bare-metal profile it need to be passed in to the compilation as mbed_apps.json. If we have the baremetal example, it can test the baremetal example without any issues

adbridge · 2019-10-22T15:11:50Z

General comment. I'm not happy about turning off ci testing of the 3 sets of examples mentioned! These are not really tested anywhere else so this is a dangerous thing to do....

adbridge · 2019-10-22T15:17:07Z

Also the changes you have made to the examples with multiple sub examples will now break the release script. Examples.json is parsed by the release script to work out which examples to update the mbed-os.lib files for....

adbridge · 2019-10-22T15:18:00Z

tools/test/examples/examples.json

-          "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-tls-client",
-          "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-hashing",
-          "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-authcrypt"
+      "sub-repo-example": true,


Removing these lines breaks the release script. I will need to consider the impact before the next release.

jamesbeyond · 2019-10-22T16:59:50Z

General comment. I'm not happy about turning off ci testing of the 3 sets of examples mentioned! These are not really tested anywhere else so this is a dangerous thing to do....

Maybe I didn't made myself clear enough, it just temporary turned off. very soon they will be added back to the compile test. the reason is I need to make any other change to the CI groovy script, once the CI groovy script is done I can re-enable them in mbed-os. then the CI will check those example out from Github instead of Mercurial.

Removing these lines breaks the release script. I will need to consider the impact before the next release.

I wasn't aware of the impacts to the release scripts, let have a catch-up offline regarding this issue. @adbridge

adbridge · 2019-10-23T10:43:09Z

tools/test/examples/examples.json

-          "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-tls-client",
-          "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-hashing",
-          "https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-authcrypt"
+      "sub-repo-example": true,


Nice to have finally support for these. Can this be automated? If this is true, we walk through the root and if mbed-os.json file found, it is an example for testing ? This way we do not need to define subs (this can be out of date easily as everytime someone adds a new sub example or restructure example, this breaks).

Or any other way to do this without having this "subs" in here?

@0xc0170 That is how the release script currently updates the mbed-os.lib files. It works down the root.

@jamesbeyond Actually looking at the release script more carefully I think this change is ok. The script only looks at the 'name' field so as long as that doesn't change we should be ok!

0xc0170

Although less error prone would be : if subs is not empty use it as "subs repo example" otherwise a repo is just one. This would be simplified: https://github.com/ARMmbed/mbed-os/pull/11710/files#diff-e373941cca74280be2386207895bc63aR167

0xc0170 · 2019-10-25T13:01:55Z

CI started

jamesbeyond · 2019-10-25T13:17:50Z

Although less error prone would be : if subs is not empty use it as "subs repo example" otherwise a repo is just one. This would be simplified: https://github.com/ARMmbed/mbed-os/pull/11710/files#diff-e373941cca74280be2386207895bc63aR167

Yeah, I thought about that, but I think the empty value [ ], seems misleading sometimes. Because in other fields, the [ ] means all possible alternatives rather than nothing. So just want to have this explicit filed that shows this example dose contains sub-examples

mbed-ci · 2019-10-25T15:13:44Z

Test run: FAILED

Summary: 3 of 4 test jobs failed
Build number : 1
Build artifacts

Failed test jobs:

jenkins-ci/mbed-os-ci_build-ARM
jenkins-ci/mbed-os-ci_build-GCC_ARM
jenkins-ci/mbed-os-ci_build-IAR

jamesbeyond · 2019-10-25T16:37:08Z

Sorry, forgot to turn-off the build test for NFC example, they are with sub-examples as well. Now updated, Could we start the CI again please? @0xc0170

mbed-ci · 2019-10-28T00:11:30Z

Test run: SUCCESS

Summary: 11 of 11 test jobs passed
Build number : 2
Build artifacts

0xc0170 · 2019-10-28T08:51:30Z

@jamesbeyond Integrated, lets update CI

Update examples testing scripts

Update examples testing scripts (cherry picked from commit 83c4a80)

Qinghao Shi and others added 12 commits October 16, 2019 15:56

EXAMPLES: add test key for remaining examples

f317753

EXAMPLES: update examples.json file

2f6f066

- remove "mbed" and "test-repo-source" key - add "sub-repo-example" key as an indicator if an example has sub examples - add "subs" key to sotre sub examples names

EXAMPLES: add list amd Symlink functions

fa86629

EXAMPLES: improve examples.py cli help messages

640ea73

EXAMPLES: update examples_lib.py

4c5f940

- add logging function - update clone function - update deploy function - update source function - add symlink function

EXAMPLES: update export_repos function

3f0add7

EXAMPLES: update build_repo function

6876362

EXAMPLES: update console output format

cf02be6

EXAMPLES: update comments and fix bugs

88d5079

EXAMPLES: temporarily turn off the complie test for sub example

52b4c39

EXAMPLES: add README file

59a4ad6

EXAMPLES: remove default build profiles

61c170f

ciarmcom requested review from adbridge, AnotherButler, evedon, mark-edgeworth, OPpuolitaival and a team October 18, 2019 13:00

ciarmcom added the needs: review label Oct 18, 2019

OPpuolitaival approved these changes Oct 18, 2019

View reviewed changes

hugueskamba mentioned this pull request Oct 18, 2019

Move script to check for floats and make it Python 2 compatible #11429

Merged

Edit README.md

924787f

Edit file, mostly for formatting and grammar.

jamesbeyond mentioned this pull request Oct 18, 2019

TEST: add atecc608a example to compilation test #11475

Merged

0xc0170 reviewed Oct 21, 2019

View reviewed changes

EXAMPLES: fix a typo in the path

5100bdd

evedon reviewed Oct 22, 2019

View reviewed changes

adbridge requested changes Oct 22, 2019

View reviewed changes

adbridge approved these changes Oct 23, 2019

View reviewed changes

0xc0170 approved these changes Oct 23, 2019

View reviewed changes

0xc0170 added needs: CI and removed needs: review labels Oct 25, 2019

EXAMPLES: temporarily turn off build for NFC examples

a211cf1

0xc0170 added ready for merge and removed needs: CI labels Oct 28, 2019

0xc0170 merged commit 83c4a80 into ARMmbed:master Oct 28, 2019

0xc0170 added release-version: 5.14.2 and removed ready for merge labels Oct 28, 2019

urutva pushed a commit to urutva/mbed-os that referenced this pull request Oct 30, 2019

Merge pull request ARMmbed#11710 from jamesbeyond/exp_github

2b72d27

Update examples testing scripts

urutva pushed a commit to urutva/mbed-os that referenced this pull request Oct 30, 2019

Merge pull request ARMmbed#11710 from jamesbeyond/exp_github

ab1ac95

Update examples testing scripts

urutva pushed a commit to urutva/mbed-os that referenced this pull request Oct 31, 2019

Merge pull request ARMmbed#11710 from jamesbeyond/exp_github

bd51999

Update examples testing scripts (cherry picked from commit 83c4a80)

jamesbeyond deleted the exp_github branch October 31, 2019 23:39

Update examples testing scripts #11710

Update examples testing scripts #11710

Uh oh!

Conversation

jamesbeyond commented Oct 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Pull request type

Reviewers

Release Notes

Uh oh!

ciarmcom commented Oct 18, 2019

Uh oh!

OPpuolitaival left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jamesbeyond commented Oct 20, 2019

Uh oh!

0xc0170 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adbridge Oct 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adbridge Oct 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xc0170 commented Oct 21, 2019

Uh oh!

jamesbeyond commented Oct 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adbridge commented Oct 22, 2019

Uh oh!

adbridge commented Oct 22, 2019

Uh oh!

adbridge Oct 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jamesbeyond commented Oct 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adbridge Oct 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xc0170 left a comment

Choose a reason for hiding this comment

Uh oh!

0xc0170 commented Oct 25, 2019

Uh oh!

jamesbeyond commented Oct 25, 2019

Uh oh!

mbed-ci commented Oct 25, 2019

Uh oh!

jamesbeyond commented Oct 25, 2019

Uh oh!

mbed-ci commented Oct 28, 2019

Uh oh!

0xc0170 commented Oct 28, 2019

Uh oh!

Uh oh!

jamesbeyond commented Oct 18, 2019 •

edited

Loading

adbridge Oct 22, 2019 •

edited

Loading

adbridge Oct 23, 2019 •

edited

Loading

jamesbeyond commented Oct 21, 2019 •

edited

Loading

adbridge Oct 22, 2019 •

edited

Loading

jamesbeyond commented Oct 22, 2019 •

edited

Loading

adbridge Oct 23, 2019 •

edited

Loading