Skip to content

Commit 83c4a80

Browse files
0xc01700xc0170
authored
Merge pull request #11710 from jamesbeyond/exp_github
Update examples testing scripts
2 parents 87a5e47 + a211cf1 commit 83c4a80

File tree

4 files changed

+526
-335
lines changed

4 files changed

+526
-335
lines changed

‎tools/test/examples/README.md

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
# Examples testing script
2+
3+
The scripts in this folder are used for testing `mbed-os` official examples. It contains the following files:
4+
5+
- `examples.py` - the main script that serves as the command-line interface.
6+
- `examples.json` - the default configuration file for the script, which contains the information to the examples. If required, you can pass a customized configuration file as an argument to the script.
7+
- `examples_lib.py` - the library file, which contains the main function of the testing scripts.
8+
9+
## The scripts
10+
11+
`examples.py` provides command-line interface and subcommands that makes easier to test examples. Included subcommands are:
12+
13+
* **import** - imports each of the repos and its dependencies (.lib files) associated with the specific examples name from the .json configuration file. If there is already a clone of the repo, it will first be removed to ensure a clean, up-to-date cloning.
14+
15+
* **clone** - clones each of the repos associated with the specific examples name from the .json configuration file. If there is already a clone of the repo, it will first be removed to ensure a clean, up-to-date cloning.
16+
17+
* **deploy** - if the example directory exists as provided by the .json configuration file, pulls in the examples dependencies by using `mbed-cli deploy`.
18+
19+
* **update** - for each example repo identified in the config .json object, updates the version of `mbed-os` to that specified by the supplied GitHub tag. This function assumes that each example repo has already been cloned.
20+
21+
* **compile** - compiles combinations of example programs, targets and compile chains.
22+
23+
* **export** - exports and builds combinations of example programs, targets and IDEs.
24+
25+
* **list** - displays examples in a configuration file in a table.
26+
27+
* **symlink** - creates a symbolic link to a given `mbed-os` PATH.
28+
29+
For more detailed options, please use `-h` or `--help`.
30+
31+
## The configuration file
32+
33+
Here is the section of default configuration file:
34+
35+
```json
36+
{
37+
"examples": [
38+
{
39+
"name": "mbed-os-example-blinky",
40+
"github": "https://github.com/ARMmbed/mbed-os-example-blinky",
41+
"sub-repo-example": false,
42+
"subs": [],
43+
"features" : [],
44+
"targets" : [],
45+
"toolchains" : [],
46+
"exporters": [],
47+
"compile" : true,
48+
"export": true,
49+
"test" : true,
50+
"baud_rate": 9600,
51+
"compare_log": ["mbed-os-example-blinky/tests/blinky.log"],
52+
"auto-update" : true
53+
},
54+
55+
...
56+
57+
{
58+
"name": "mbed-os-example-tls",
59+
"github": "https://github.com/ARMmbed/mbed-os-example-tls",
60+
"sub-repo-example": true,
61+
"subs": [
62+
"benchmark",
63+
"tls-client",
64+
"hashing",
65+
"authcrypt"
66+
],
67+
"features" : [],
68+
"targets" : ["K66F", "NUCLEO_F429ZI"],
69+
"toolchains" : ["GCC_ARM", "ARM"],
70+
"exporters": [],
71+
"compile" : false,
72+
"export": false,
73+
"test" : false,
74+
"baud_rate": 9600,
75+
"compare_log": [
76+
"mbed-os-example-tls/tests/benchmark.log",
77+
"mbed-os-example-tls/tests/tls-client.log",
78+
"mbed-os-example-tls/tests/hashing.log",
79+
"mbed-os-example-tls/tests/authcrypt.log"
80+
],
81+
"auto-update" : true
82+
}
83+
]
84+
}
85+
86+
```
87+
88+
### Fields
89+
90+
* **name** - name of the example. It should be the base name of the example repository address and will throw if it doesn't match.
91+
* **github** - example GitHub repository address.
92+
* **sub-repo-example** - specifies if the example repository has a subfolder for each example.
93+
* **subs** - array of subexample names.
94+
* **features** - the features that must be in the features array of a target in `targets.json`.
95+
* **baud_rate** - example default baud rate.
96+
* **compare_log** - array of log compared to command-line output during testing. If the example has many subexamples, the order of log should match the order of subexamples.
97+
* **targets** - list of `mbed-os` development boards that run this example.
98+
* **targets** - list of targeted development boards.
99+
* **toolchains** - toolchain to use for compiling.
100+
* **exporters** - allowed exporters.
101+
* **compile** - enables compiling.
102+
* **export** - enables exporting.
103+
* **test** - enables testing.
104+
105+
### Values
106+
107+
`[ ]` means all possible alternatives.
108+
109+
## Typical use
110+
111+
In the Mbed OS CI, we follow the below steps to compile and test Mbed OS examples.
112+
113+
1. Clone `mbed-os` repository to the current folder:
114+
115+
```
116+
git clone https://github.com/ARMmbed/mbed-os.git
117+
```
118+
119+
1. Clone the examples repo to the current folder. Users can pass an `-e` option to the script to filter out the rest of the examples, so the scripts only run on one particular example:
120+
121+
```
122+
python mbed-os/tools/test/examples/examples.py clone
123+
```
124+
125+
1. Create a symbolic link to `mbed-os` for every example. This step lets all the examples share a single `mbed-os` folder, rather than checking out the `mbed-os` folder many times. We highly recommend you pass an absolute path as the argument:
126+
127+
```
128+
python mbed-os/tools/test/examples/examples.py symlink $PWD/mbed-os
129+
```
130+
131+
1. Deploy other dependency libraries:
132+
133+
```
134+
python mbed-os/tools/test/examples/examples.py deploy
135+
```
136+
137+
1. Compile the test for the examples on a specific target:
138+
139+
```
140+
python mbed-os/tools/test/examples/examples.py compile -m <target>
141+
```
142+
143+
After the compile test finished, the scripts print the result table:
144+
145+
```
146+
Passed example compilation:
147+
+---------------------------------+--------+-----------+----------+--------------+
148+
| EXAMPLE NAME | TARGET | TOOLCHAIN | TEST GEN | BUILD RESULT |
149+
+---------------------------------+--------+-----------+----------+--------------+
150+
| mbed-os-example-kvstore | K64F | GCC_ARM | TEST_ON | PASSED |
151+
| mbed-os-example-tls-socket | K64F | GCC_ARM | TEST_ON | PASSED |
152+
| mbed-os-example-blockdevice | K64F | GCC_ARM | TEST_ON | PASSED |
153+
| mbed-os-example-wifi | K64F | GCC_ARM | TEST_OFF | PASSED |
154+
| mbed-os-example-error-handling | K64F | GCC_ARM | TEST_ON | PASSED |
155+
| mbed-os-example-sd-driver | K64F | GCC_ARM | TEST_ON | PASSED |
156+
| mbed-os-example-crash-reporting | K64F | GCC_ARM | TEST_ON | PASSED |
157+
| mbed-os-example-filesystem | K64F | GCC_ARM | TEST_ON | PASSED |
158+
| mbed-os-example-blinky | K64F | GCC_ARM | TEST_ON | PASSED |
159+
| mbed-os-example-bootloader | K64F | GCC_ARM | TEST_OFF | PASSED |
160+
| mbed-os-example-cpu-stats | K64F | GCC_ARM | TEST_ON | PASSED |
161+
| mbed-os-example-sys-info | K64F | GCC_ARM | TEST_ON | PASSED |
162+
| mbed-os-example-attestation | K64F | GCC_ARM | TEST_ON | PASSED |
163+
+---------------------------------+--------+-----------+----------+--------------+
164+
Number of failures = 0
165+
```
166+
167+
After the compilation stage, a `test_spec.json` file is generated. Later, Greentea tests will consume this file. They test the compiled example on hardware platform.

0 commit comments

Comments
 (0)