Skip to content

Update CircleCI to run parallel Python jobs #668

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 23, 2018
Merged

Update CircleCI to run parallel Python jobs #668

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b5ff448
Updated CircleCI files:
cmonr May 11, 2018
3a4d2c3
Enabled py3 tests to get help understanding final issues
cmonr May 11, 2018
cf4676d
Re-disable py3 test
cmonr May 11, 2018
ed7e207
Updated py3 test to checkout Mbed OS master branch.
cmonr May 22, 2018
ee4fce9
Updated circle_tests.py to run either py2 or py3 tests from config.yml
cmonr May 22, 2018
9f2c53b
Reintroduced py3 upper version limit
cmonr May 22, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 145 additions & 0 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
version: 2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please update the circle_tests.py script which allows to run all commands from the circle.yml file and emulate circle CI test run?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now works across both versions!

jobs:
py2:
machine: true
steps:
- run: sudo add-apt-repository -y ppa:terry.guo/gcc-arm-embedded
- run: sudo apt-get -y update
- run: sudo apt-get -y install gcc-arm-none-eabi=4.9.3.2015q3-1trusty1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we update this to GCC 6.x? Pretty sure Mbed OS doesn't recommend using 4.9.x anymore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woah. You're right, but I'd prefer to change only what's needed for the topic at hand.

Definitely going to need to submit another PR to get that updated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updating GCC 6.x would only work for newer Mbed OS, but older cases will fail (like Morpheus). And the Morpheus routine is used for comprehensive multi-level dependencies handling, including removal of libraries etc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That make sense, but we should be testing with the recommended toolchain for each Mbed OS release. So if that means installing two different versions of GCC then we may need to do that.


- run: python --version
- run: pip --version
- run: pip install pytest future
- run: git config --global user.email "[email protected]"
- run: git config --global user.name "mbed Test"
- run: echo -e "[ui]\nusername = mbed Test <[email protected]>\n" > ~/.hgrc
- checkout
- run: pip install -e ./

- run: mbed --version
- run: py.test test
- run: mbed toolchain -G GCC_ARM
- run: mbed target -G K64F
- run: mbed config -G protocol ssh
- run: mkdir .tests
- run: cd .tests && mbed new new-test
- run: cd .tests/new-test && mbed ls
- run: cd .tests/new-test && mbed releases -r
- run: cd .tests/new-test && mbed compile --source=. --source=mbed-os/TESTS/integration/basic -j 0
- run: cd .tests/new-test && mbed test --compile -n mbed-os-tests-integration-basic -j 0
- run: cd .tests && mbed import https://developer.mbed.org/teams/Morpheus/code/mbed-Client-Morpheus-hg hg-test
- run: cd .tests/hg-test && mbed ls
- run: cd .tests/hg-test && mbed releases -r
- run: cd .tests/hg-test && mbed update b02527cafcde8612ff051fea57e9975aca598807 --clean
- run: cd .tests/hg-test && mbed update --clean
- run: cd .tests/hg-test && mbed compile -j 0
- run: cd .tests && mbed import https://developer.mbed.org/users/samux/code/USBSerial_HelloWorld bld-test
- run: cd .tests/bld-test && mbed ls
- run: cd .tests/bld-test && mbed releases -r
- run: cd .tests/bld-test/mbed && mbed update 85 --clean
- run: cd .tests/bld-test && mbed update --clean
- run: cd .tests/bld-test && mbed compile -m LPC1768 -j 0
- run: cd .tests && mbed new supported-tests
- run: |-
cd .tests/supported-tests
echo "Testing mbed-os-5.6.0"
mbed update --clean "mbed-os-5.6.0"
mbed compile -S
mbed toolchain -S
mbed target -S
mbed export -S
- run: |-
cd .tests/supported-tests
echo "Testing mbed-os-5.5.0"
mbed update --clean "mbed-os-5.5.0"
mbed compile -S
mbed toolchain -S
mbed target -S
mbed export -S
- run: |-
cd .tests/supported-tests
echo "Testing mbed-os-5.4.0"
mbed update --clean "mbed-os-5.4.0"
mbed compile -S
mbed toolchain -S
mbed target -S
mbed export -S
- run: |-
cd .tests/supported-tests
echo "Testing mbed-os-5.3.0"
mbed update --clean "mbed-os-5.3.0"
mbed compile -S
mbed toolchain -S
mbed target -S
mbed export -S
- run: |-
cd .tests/supported-tests
echo "Testing mbed-os-5.2.0"
mbed update --clean "mbed-os-5.2.0"
mbed compile -S
mbed toolchain -S
mbed target -S
mbed export -S
- run: |-
cd .tests/supported-tests
echo "Testing mbed-os-5.1.0"
mbed update --clean "mbed-os-5.1.0"
mbed compile -S
mbed toolchain -S
mbed target -S
mbed export -S

py3:
machine: true
steps:
- run: sudo add-apt-repository -y ppa:terry.guo/gcc-arm-embedded
- run: sudo apt-get -y update
- run: sudo apt-get -y install gcc-arm-none-eabi=4.9.3.2015q3-1trusty1
- run: pyenv install 3.6.0
- run: pyenv global 3.6.0

- run: python --version
- run: pip --version
- run: pip install pytest future
- run: git config --global user.email "[email protected]"
- run: git config --global user.name "mbed Test"
- run: echo -e "[ui]\nusername = mbed Test <[email protected]>\n" > ~/.hgrc
- checkout
- run: pip install -e ./

- run: mbed --version
- run: py.test test
- run: mbed toolchain -G GCC_ARM
- run: mbed target -G K64F
- run: mbed config -G protocol ssh

- run: mkdir .tests
- run: cd .tests && mbed new new-test
- run: |-
cd .tests/new-test/mbed-os
git checkout master
- run: cd .tests/new-test && mbed ls
- run: cd .tests/new-test && mbed releases -r
- run: cd .tests/new-test && mbed compile --source=. --source=mbed-os/TESTS/integration/basic -j 0
- run: cd .tests/new-test && mbed test --compile -n mbed-os-tests-integration-basic -j 0

- run: cd .tests && mbed import https://developer.mbed.org/teams/Morpheus/code/mbed-Client-Morpheus-hg hg-test
- run: cd .tests/hg-test && mbed ls
- run: cd .tests/hg-test && mbed releases -r
- run: cd .tests/hg-test && mbed update b02527cafcde8612ff051fea57e9975aca598807 --clean
- run: cd .tests/hg-test && mbed update --clean
- run: cd .tests/hg-test && mbed compile -j 0

- run: cd .tests && mbed import https://developer.mbed.org/users/samux/code/USBSerial_HelloWorld bld-test
- run: cd .tests/bld-test && mbed ls
- run: cd .tests/bld-test && mbed releases -r
- run: cd .tests/bld-test/mbed && mbed update 85 --clean
- run: cd .tests/bld-test && mbed update --clean
- run: cd .tests/bld-test && mbed compile -m LPC1768 -j 0

workflows:
version: 2
build:
jobs:
- py2
#- py3
91 changes: 0 additions & 91 deletions circle.yml
View file
Open in desktop

This file was deleted.

70 changes: 47 additions & 23 deletions circle_tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,27 +26,51 @@ def remove_readonly(func, path, _):

shutil.rmtree(directory, onerror=remove_readonly)

# Source tests from yaml config
tests = None
with open("circle.yml", "r") as f:
types = yaml.load_all(f)
for t in types:
for k,v in t.items():
if k == 'test':
tests = v['override']

if tests:
cwd = os.path.abspath(os.path.dirname(__file__))

if os.path.exists(os.path.join(cwd, '.tests')):
rmtree_readonly(os.path.join(cwd, '.tests'))
os.mkdir(os.path.join(cwd, '.tests'))

for cmd in tests:
os.chdir(cwd)
print "\n----------\nEXEC: \"%s\" " % cmd
proc = subprocess.Popen(cmd, shell=True)
proc.communicate()

if proc.returncode != 0:
print "\n------------\nERROR: \"%s\"" % cmd
sys.exit(1)
with open('.circleci/config.yml', 'r') as f:
data = yaml.safe_load(f)

# Read yaml tree
if sys.version_info[0] == 3:
tests = data['jobs']['py3']['steps']
else:
tests = data['jobs']['py2']['steps']

# Filter command list to only contain commands
tests = [item['run'] for item in list(filter(lambda x : type(x) is dict, tests))]

# ... and replace new lines with ampersands
tests = [item.replace('\n', ' && ') for item in tests]

# Exit if no tests are found
if tests == None:
sys.exit(1)

# Ignore all tests found before `pip install -e`
startIndex = -1
for cmd in tests:
if 'pip install -e' in cmd:
startIndex = tests.index(cmd) + 1
break

if startIndex == -1:
sys.exit(1)

# Delete `.test` directory if it exists
cwd = os.path.abspath(os.path.dirname(__file__))

if os.path.exists(os.path.join(cwd, '.tests')):
rmtree_readonly(os.path.join(cwd, '.tests'))
os.mkdir(os.path.join(cwd, '.tests'))

# Run tests
for cmd in tests[startIndex:]:
os.chdir(cwd)
print("\n----------\nEXEC: \"%s\" " % cmd)
proc = subprocess.Popen(cmd, shell=True)
proc.communicate()

if proc.returncode != 0:
print "\n------------\nERROR: \"%s\"" % cmd
sys.exit(1)
6 changes: 3 additions & 3 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Copyright (c) 2016 ARM Limited, All Rights Reserved
# SPDX-License-Identifier: Apache-2.0

# Licensed under the Apache License, Version 2.0 (the "License");
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied.

import os
Expand Down