Travis CI: Stages, OS update, apt-get relief! #9844

cmonr · 2019-02-26T06:48:27Z

Description

A rather large refactor of Travis CI.

Changes:

Addition of build stages.
- Adds some much needed job organization
- Runs jobs in batches, allowing for better parallelism
- Allows for better control over job setup (ie: no need to install python modules when only running a shell command)
Linux OS update.
- All jobs are now running Ubuntu Xenial.
Creation of bash helper functions.
- Helper functions are located in tools/test/travis-ci/functions.sh
- Functions can even be sourced and run locally (with caveats)
- Cleans up Travis CI config (a bit)
Misc changes
- No more apt-get update! 🎉
- Enabled more caching
- Some packages are now being sourced from S3
- Easily updatable GCC archive when the time is right
- Cleaned up some of the larger jobs

PR will be squashed and cleaned up tomorrow, along with the addition of Release Notes.

Pull request type

[ ] Fix
[x] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[ ] Test update
[ ] Breaking change

Reviewers

Release Notes

Travis CI's testing config has been updated with some readability and performance improvements in mind.

Build Stages

Build stages have been added, which helps group similar jobs according to function.

Build stages also enocourage greater build parallelism by limiting the number of concurrent build jobs that a single PR can have, allowing multiple PRs to be tested in parallel instead of wauting in a build queue.

Finally, batching jobs into build stages allows for better job configuration. Some jobs that shouldn't have taken long were doing unneccesary actions such as downloading packages or installing python modules, which has now been corrected.

Python 3.7 by default

Jobs that require Python to run now use Py37 by default.

Ubutnu Xenial

The Linux VM that runs tests has been updated to use Xenail, Travis CI's latest supported Linux OS.

Minimal apt-get usage

Usage of apt-get has been greatly minimized by moving packages over to S3 and using dpkg to install cached packages when needed.

A major pain point would occur when a job would fail because the VM failed to run apt-get update successfully.

Bash functions

Common functions such as downloading packages or updating a job's status have now been collected into a single bash function. No strage shell invocations need to be done in order to use them which was not the case before.

cmonr · 2019-02-26T06:49:02Z

@AnotherButler Feel free to take a look at the docs jobs and lmk what you think.

A PR to https://github.com/ARMmbed/mbed-os-5-docs will come tomorrow as well.

cmonr · 2019-02-26T06:52:58Z

A note: My local fork didn't like running 'coveralls' with Py37. Time debugging that failure was next to nothing. Can take another look at it tomorrow.

0xc0170 · 2019-02-28T12:27:15Z

@cmonr I am moving this to 5.12.1 - this is test config that could land later than 5.12.0 ? As we have high number of PR for 5.12 and t his is not even reviewed.

Or at least landing on master, and each patch release will use old travis config - not ideal.

cmonr · 2019-02-28T17:03:14Z

is this test config that could land later than 5.12.0?

It can, but was hoping to have it in sooner, since this fixes a fair amount of headaches with apt-get, and better supports testing multiple PRs at once.

cmonr · 2019-03-01T00:54:58Z

@0xc0170 I just remembered that a large reason I was trying to get this in for 5.12 was that it was possible that the test names were going to be updated.

0xc0170 · 2019-03-01T12:51:22Z

Usage of apt-get has been greatly minimized by moving packages over to S3 and using dpkg to install cached packages when needed.

What is the lifetime? Previously we had some issue with limited lifetime, solved now?

0xc0170

LGTM otherwise

0xc0170 · 2019-03-01T12:51:49Z

tools/test/travis-ci/functions.sh

@@ -0,0 +1,139 @@
+#!/bin/bash -euf


license header file here

cmonr · 2019-03-07T18:03:35Z

What is the lifetime? Previously we had some issue with limited lifetime, solved now?

@0xc0170 What do you mean by lifetime?

0xc0170 · 2019-03-08T14:40:45Z

@0xc0170 What do you mean by lifetime?

We had previously a problem aws stored only for 3 months or so. if this is not a problem anymore

Added external bash file to source functions from

Primary cause of false Travis CI failures was running 'apt-get update'. Refactored all instances where this was needed with manually fetching and installing dependencies in S3 instace. This also includes GCC installation.

Added {} around each variable. Ported python invocations for simple math opts to coreutil commands

Added comments to functions

0xc0170 · 2019-04-11T11:29:36Z

CI started

mbed-ci · 2019-04-11T13:23:12Z

Test run: FAILED

Summary: 1 of 11 test jobs failed
Build number : 1
Build artifacts

Failed test jobs:

jenkins-ci/mbed-os-ci_exporter

0xc0170 · 2019-04-11T13:24:44Z

Exporters failed with some symlink errors, will restart once travis is fixed

OPpuolitaival · 2019-04-11T13:45:58Z

symlink error comes from untar. Sounds like corrupted package. Probably need to run whole pipeline again

'tr' command wasn't doing any replacement

cmonr · 2019-04-12T01:18:41Z

CI started

mbed-ci · 2019-04-12T03:34:59Z

Test run: FAILED

Summary: 1 of 11 test jobs failed
Build number : 2
Build artifacts

Failed test jobs:

jenkins-ci/mbed-os-ci_exporter

0xc0170 · 2019-04-12T08:15:17Z

Exporters restarted

0xc0170 · 2019-04-12T09:10:46Z

Internal CI symlink error again, aborting the current one and should be fixed in the next one

0xc0170 · 2019-04-12T13:45:54Z

Green 🎉 @cmonr in?

cmonr added needs: review risk: A release-version: 5.12.0-rc1 labels Feb 26, 2019

cmonr requested review from theotherjimmy, OPpuolitaival, kegilbert, geky, SeppoTakalo, kjbracey and a team February 26, 2019 06:48

cmonr changed the title ~~Travis CI: Stages, OS update, apt-get relief~~ Travis CI: Stages, OS update, apt-get relief! Feb 26, 2019

cmonr requested a review from SenRamakri February 26, 2019 07:08

cmonr force-pushed the travisCI-refactor branch 2 times, most recently from 9e1fdeb to 175827e Compare February 28, 2019 01:55

0xc0170 added release-version: 5.12.1 and removed release-version: 5.12.0-rc1 labels Feb 28, 2019

cmonr removed the risk: A label Feb 28, 2019

0xc0170 previously requested changes Mar 1, 2019

View reviewed changes

cmonr added needs: work and removed needs: review labels Mar 5, 2019

cmonr force-pushed the travisCI-refactor branch from b2b1fd5 to 231236c Compare March 7, 2019 18:10

Cruz Monrreal II added 6 commits April 8, 2019 10:45

Grouped jobs into build stages

f9f9b60

Re-enabled PR status reporting.

5f40538

Added external bash file to source functions from

Removed nearly all usage of 'apt-get'.

397ee30

Primary cause of false Travis CI failures was running 'apt-get update'. Refactored all instances where this was needed with manually fetching and installing dependencies in S3 instace. This also includes GCC installation.

Fixed littlefs tests to be Py3 compat

f6b9109

Cleaned up LittleFS and Event Queue tests.

260c97a

Added {} around each variable. Ported python invocations for simple math opts to coreutil commands

Fixed shellcheck issues found in functions.sh

3f76f6e

Added comments to functions

cmonr force-pushed the travisCI-refactor branch from 1a65895 to 3f76f6e Compare April 8, 2019 15:45

cmonr added release-version: 5.12.2 and removed release-version: 5.12.1 labels Apr 9, 2019

0xc0170 added needs: CI and removed needs: work labels Apr 11, 2019

0xc0170 added needs: work and removed needs: CI labels Apr 11, 2019

cmonr force-pushed the travisCI-refactor branch from f987632 to c79bb7e Compare April 11, 2019 15:46

Fixed status reporting.

2a2ec16

'tr' command wasn't doing any replacement

cmonr force-pushed the travisCI-refactor branch from c79bb7e to 2a2ec16 Compare April 11, 2019 15:50

Made typo to unblock final check

17dcbd1

cmonr added needs: CI and removed needs: work labels Apr 11, 2019

cmonr merged commit f1c4a02 into ARMmbed:master Apr 12, 2019

cmonr removed the needs: CI label Apr 12, 2019

Travis CI: Stages, OS update, apt-get relief! #9844

Travis CI: Stages, OS update, apt-get relief! #9844

Uh oh!

Conversation

cmonr commented Feb 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Pull request type

Reviewers

Release Notes

Build Stages

Python 3.7 by default

Ubutnu Xenial

Minimal apt-get usage

Bash functions

Uh oh!

cmonr commented Feb 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmonr commented Feb 26, 2019

Uh oh!

0xc0170 commented Feb 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmonr commented Feb 28, 2019

Uh oh!

cmonr commented Mar 1, 2019

Uh oh!

0xc0170 commented Mar 1, 2019

Uh oh!

0xc0170 left a comment

Choose a reason for hiding this comment

Uh oh!

0xc0170 Mar 1, 2019

Choose a reason for hiding this comment

Uh oh!

cmonr Mar 27, 2019

Choose a reason for hiding this comment

Uh oh!

cmonr commented Mar 7, 2019

Uh oh!

0xc0170 commented Mar 8, 2019

Uh oh!

0xc0170 commented Apr 11, 2019

Uh oh!

mbed-ci commented Apr 11, 2019

Test run: FAILED

Uh oh!

0xc0170 commented Apr 11, 2019

Uh oh!

OPpuolitaival commented Apr 11, 2019

Uh oh!

cmonr commented Apr 12, 2019

Uh oh!

mbed-ci commented Apr 12, 2019

Test run: FAILED

Uh oh!

0xc0170 commented Apr 12, 2019

Uh oh!

0xc0170 commented Apr 12, 2019

Uh oh!

0xc0170 commented Apr 12, 2019

Uh oh!

Uh oh!

cmonr commented Feb 26, 2019 •

edited

Loading

cmonr commented Feb 26, 2019 •

edited

Loading

0xc0170 commented Feb 28, 2019 •

edited

Loading