Skip to content

Commit 66ac315

Browse files
teetak01Yogesh Pande
teetak01
authored and
Yogesh Pande
committed
Add Device Management E2E tests
1 parent 66b0400 commit 66ac315

15 files changed

+778
-0
lines changed

‎TESTS/README.md

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Device Management E2E testing
2+
3+
Device Management E2E tests are platform agnostic. They verify that a target platform can reliably perform basic critical operations.
4+
There are two different test sets:
5+
6+
- Tests for verifying the basic functionality of a target platform.
7+
- Advanced tests for more complex use cases (for example `device_update` test case).
8+
9+
## Requirements
10+
11+
1. [IceTea](https://github.com/ARMmbed/icetea) v1.2.1 or later.
12+
1. [Mbed Cloud SDK for Python](https://cloud.mbed.com/docs/current/mbed-cloud-sdk-python/index.html) v2.0.5 or later.
13+
14+
## Installation
15+
16+
```
17+
pip install icetea mbed-cloud-sdk
18+
```
19+
20+
## Basic usage
21+
22+
1. Register your device to Device Management. For examples, see the [tutorials](https://cloud.mbed.com/docs/current/connecting/device-management-client-tutorials.html).
23+
1. Enter your [device_id](https://cloud.mbed.com/docs/current/connecting/device-identity.html) to `TESTS/E2E/pelion.tc_cfg`.
24+
1. Enter your [API key](https://cloud.mbed.com/docs/current/integrate-web-app/api-keys.html) to `TESTS/E2E/pelion.tc_cfg`.
25+
26+
### Running the test suite
27+
28+
To run the test suite, use the following command:
29+
30+
```
31+
icetea --suite basic_tests.json --suitedir TESTS/E2E/ --tcdir ./TESTS/E2E/ --tc_cfg TESTS/E2E/pelion.tc_cfg
32+
```
33+
34+
If you have prepared a manifest (see below the instructions for `device_update` testcase), you can run the full suite with:
35+
36+
```
37+
icetea --suite full_tests.json --suitedir TESTS/E2E/ --tcdir ./TESTS/E2E/ --tc_cfg TESTS/E2E/pelion.tc_cfg
38+
```
39+
40+
### Running a single test
41+
42+
To run a single test, use the following command:
43+
44+
```
45+
icetea --tc basic_get --tcdir ./TESTS/E2E/ --tc_cfg TESTS/E2E/pelion.tc_cfg
46+
```
47+
48+
## Current tests
49+
50+
| Test name | Main functions | Notes |
51+
| ---------------- | ------------------------------------------ | --------------------------------------|
52+
| `register` | Verify that the device is registered. | |
53+
| `get` | Verify that the device responds to GET. | Uses OMA resource `/3/0/0` |
54+
| `put` | Verify that the device responds to PUT. | Uses custom resource `/3201/0/5853` |
55+
| `post` | Verify that the device responds to POST. | Uses custom resource `/3201/0/5850` |
56+
| `observation` | Verify that the device can send notifications. | Uses custom resource `/3201/0/5853` |
57+
| `deregister` | Verify that the device can deregister. | Uses custom resource `/5000/0/1` |
58+
| `device_update` | Performs the firmware update. | This testcase verifies that the device can perform a firmware update. For this testcase, you need to manually generate an update image and a manifest. |
59+
60+
### Executing `device_update` test case
61+
62+
The test case does not automatically generate an update image or generate a manifest. Read [Update documentation](https://cloud.mbed.com/docs/current/updating-firmware/index.html) for more information.
63+
64+
To run the test case:
65+
66+
1. Compile a new update image.
67+
1. Upload the image to Device Management.
68+
1. Generate a manifest for the update image.
69+
1. Upload the manifest to Device Management.
70+
1. Add the `<manifest-id>` to `TESTS/E2E/pelion.tc_cfg`.
71+
72+
#### Example for executing the `device_update` test case on Mbed OS platform
73+
74+
To prepare the latest Device Management Client example on Mbed OS for testing, run the following commands:
75+
76+
1. `mbed dm init -d arm.com --model-name example-app --force -q`
77+
1. `mbed compile -t <toolchain> -m <MCU>`
78+
1. `mbed dm update prepare`
79+
80+
The last command uploads the `mbed-cloud-client-example_update.bin`, generates a new manifest and uploads the manifest to Device Management. Record the resulting Manifest ID and add it to `TESTS/E2E/pelion.tc_cfg`.
81+
82+
```
83+
[INFO] 2018-12-12 15:06:42 - manifesttool.prepare - Manifest ID: <manifest-id>
84+
```
85+
86+
To execute the test, use the following command:
87+
88+
```
89+
icetea --tc device_update --tcdir ./TESTS/E2E/ --tc_cfg TESTS/E2E/pelion.tc_cfg
90+
```
91+
92+
<span class="notes">**Note:** After each successful update, you need to generate a new manifest for a new firmware update test. Otherwise, the test will pass without actually performing any device side updates. The update campaign ends automatically because the manifest has already been applied to the device.</span>

‎TESTS/__init__.py

Whitespace-only changes.

‎TESTS/basic_os_tests.json

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"testcases": [
3+
{"name" : "register_and_read_id"},
4+
{"name" : "get"},
5+
{"name" : "put"},
6+
{"name" : "post"},
7+
{"name" : "observation"},
8+
{"name" : "deregister"}
9+
]
10+
}

‎TESTS/basic_tests.json

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"testcases": [
3+
{"name" : "register"},
4+
{"name" : "get"},
5+
{"name" : "put"},
6+
{"name" : "post"},
7+
{"name" : "observation"},
8+
{"name" : "deregister"}
9+
]
10+
}

‎TESTS/deregister.py

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
"""
2+
Copyright 2019 ARM Limited
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.
14+
"""
15+
16+
# pylint: disable=missing-docstring,useless-super-delegation
17+
# pylint: disable=line-too-long,method-hidden,relative-import
18+
19+
from icetea_lib.bench import TestStepFail
20+
from mbed_cloud.exceptions import CloudApiException
21+
from pelion_helper import PelionBase
22+
23+
24+
class Testcase(PelionBase):
25+
def __init__(self):
26+
PelionBase.__init__(self,
27+
name="deregister",
28+
title="Example application can deregister",
29+
status="released",
30+
type="acceptance",
31+
component=["mbed_cloud_client_example"])
32+
33+
def setup(self):
34+
super(Testcase, self).setup()
35+
36+
def case(self):
37+
resource_path = '/5000/0/1'
38+
# Deregister device via POST (/5000/0/1)
39+
self.logger.info("Deregister via POST %s", resource_path)
40+
try:
41+
self.connect_api.execute_resource(device_id=self.device_id,
42+
resource_path=resource_path,
43+
timeout=self.restTimeout)
44+
except CloudApiException as error:
45+
raise TestStepFail("POST request failed with %d and msg %s" % (error.status, error.message))
46+
47+
# Verify client is deregistered
48+
self.logger.info("POST done")
49+
50+
self.verify_registration("deregistered")
51+
52+
def teardown(self):
53+
self.connect_api.stop_notifications()

‎TESTS/device_update.py

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
"""
2+
Copyright 2019 ARM Limited
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.
14+
"""
15+
16+
# pylint: disable=missing-docstring
17+
# pylint: disable=line-too-long,method-hidden,relative-import
18+
19+
import time
20+
from icetea_lib.bench import TestStepFail
21+
from pelion_helper import PelionBase
22+
23+
24+
class Testcase(PelionBase):
25+
def __init__(self):
26+
PelionBase.__init__(self,
27+
name="device_update",
28+
title="Example application can perform firmware update",
29+
status="released",
30+
type="acceptance",
31+
component=["mbed_cloud_client_example"])
32+
def setup(self):
33+
super(Testcase, self).setup()
34+
super(Testcase, self).setup_update()
35+
36+
def case(self):
37+
# Create filter for update campaign generation, with automatic startup
38+
update_filter = self.prepare_campaign_filter(state="scheduled")
39+
# Start the update campaign
40+
campaign_id = self.post_campaign(update_filter)
41+
# Wait for the campaign to reach autostopped state
42+
timeout = 600
43+
_endtime = time.time() + timeout
44+
45+
while True:
46+
self.delay(10)
47+
result = self.check_campaign_state(campaign_id)
48+
# Update campaign has stopped
49+
if result[0] == "autostopped":
50+
break
51+
52+
if time.time() > _endtime:
53+
self.logger.error("Campaign in state: %s, Device in state: %s", result[0], result[1])
54+
raise TestStepFail("Campaign did not finish in %d" % timeout)
55+
56+
# Check for final state of the device
57+
final_result = self.check_campaign_state(campaign_id)
58+
if final_result[1] != "deployed":
59+
raise TestStepFail("Device in %s state, expected deployed" % result[1])
60+
61+
self.logger.info("Firmware update finished successfully")
62+
63+
def teardown(self):
64+
self.connect_api.stop_notifications()

‎TESTS/full_tests.json

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"testcases": [
3+
{"name" : "register"},
4+
{"name" : "get"},
5+
{"name" : "put"},
6+
{"name" : "post"},
7+
{"name" : "observation"},
8+
{"name" : "device_update"},
9+
{"name" : "deregister"}
10+
]
11+
}

‎TESTS/get.py

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
"""
2+
Copyright 2019 ARM Limited
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.
14+
"""
15+
16+
# pylint: disable=missing-docstring,useless-super-delegation
17+
# pylint: disable=line-too-long,method-hidden,relative-import
18+
19+
from icetea_lib.bench import TestStepFail
20+
from mbed_cloud.exceptions import CloudApiException
21+
from pelion_helper import PelionBase
22+
23+
24+
class Testcase(PelionBase):
25+
def __init__(self):
26+
PelionBase.__init__(self,
27+
name="get",
28+
title="Example application can perform basic CoAP operation (GET)",
29+
status="released",
30+
type="acceptance",
31+
component=["mbed_cloud_client_example"])
32+
33+
def setup(self):
34+
super(Testcase, self).setup()
35+
36+
def case(self):
37+
resource_path = '/3/0/0'
38+
self.logger.info("Testing GET %s", resource_path)
39+
try:
40+
resource_manufacturer = self.connect_api.get_resource_value(device_id=self.device_id,
41+
resource_path=resource_path,
42+
timeout=self.restTimeout)
43+
except CloudApiException as error:
44+
raise TestStepFail("GET request failed with %d and msg %s" % (error.status, error.message))
45+
46+
self.logger.info("Received value %s for %s", resource_manufacturer, resource_path)
47+
48+
def teardown(self):
49+
self.connect_api.stop_notifications()

0 commit comments

Comments
 (0)