CI - better coverage report, pypy on Azure, add codeclimate

[gaborbernat]

This now is done. It's a major rewrite and was significantly more troublesome than expected. After all the effort put in, failed to convince either codecov or coveralls to function correctly. So where two does not work, let's use the third option: codeclimate (https://codeclimate.com/github/tox-dev/tox/builds).

This one finally gives up on trying to be too smart (e.g. forcing users to merge reports locally during the build), but work correctly what it does offer:

• great documentation,
• Github App,
• great chrome extension to annotate PRs inline,
• Github comment support.

Does not have branch coverage for now but I don't think we used that much often that feature. With this, we also moved off Travis completely as now pypy tests are also done on Azure Pipelines.

PS. NOTE: we no longer really on a service to report/display code coverage. The CI itself collects all coverage reports, merges them and then displays them as a final step. Codeclimate is just an additional UI to this data, and the coverage gets uploaded to that service too; however, it's also available as part of the CI build log (both full coverage and diff coverage).

[gaborbernat]

@obestwalter @asottile Looks promising. Now it just remains to merge files across machines. Ability to upload artifacts from PRs is predicted for the end of the month, so in the meantime, we'll only get coverage reports if we PR branches of this repo; but not forks. Before merge we just need to configure coveralls to report coverage results as a comment.

[gaborbernat]

This now is done. It's a major rewrite and was significantly more troublesome than expected. After all the effort put in, failed to convince either codecov or coveralls to function correctly. So where two does not work, let's use the third option: codeclimate (https://codeclimate.com/github/tox-dev/tox/builds).

This one finally gives up on trying to be too smart (e.g. forcing users to merge reports locally during the build), but work correctly what it does offer:

• great documentation,
• Github App,
• great chrome extension to annotate PRs inline,
• Github comment support.

Does not have branch coverage for now but I don't think we used that much often that feature. With this, we also moved off Travis completely as now pypy tests are also done on Azure Pipelines.

Builds - tox-dev/tox - Code Climate

PS. NOTE: we no longer really on a service to report/display code coverage. The CI itself collects all coverage reports, merges them and then displays them as a final step. Codeclimate is just an additional UI to this data, and the coverage gets uploaded to that service too; however it's also available as part of the CI build log (both full coverage and diff coverage).

2018-10-03T18:18:07.3350628Z python -m tox -e coverage
2018-10-03T18:18:07.3379833Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/6b13d377-c9ab-4b2f-8ad8-0d34537230a7.sh
2018-10-03T18:18:07.5590051Z coverage create: /home/vsts/work/1/s/.tox/coverage
2018-10-03T18:18:10.5198017Z coverage installdeps: coverage >= 4.4.1, < 5, diff_cover
2018-10-03T18:18:24.4452969Z coverage installed: coverage==4.5.1,diff-cover==1.0.4,inflect==1.0.1,Jinja2==2.10,jinja2-pluralize==0.3.0,MarkupSafe==1.0,Pygments==2.2.0,six==1.11.0
2018-10-03T18:18:24.4455947Z coverage run-test-pre: PYTHONHASHSEED='4214662049'
2018-10-03T18:18:24.4456550Z coverage runtests: commands[0] | coverage erase
2018-10-03T18:18:24.8038698Z coverage runtests: commands[1] | coverage combine
2018-10-03T18:18:24.9931719Z coverage runtests: commands[2] | coverage report -m
2018-10-03T18:18:25.4983423Z Name                       Stmts   Miss Branch BrPart  Cover   Missing
2018-10-03T18:18:25.4994820Z ----------------------------------------------------------------------
2018-10-03T18:18:25.5008702Z src/tox/__main__.py            1      1      0      0     0%   1
2018-10-03T18:18:25.5009872Z src/tox/_pytestplugin.py     276     11     70      7    94%   26, 28, 68, 87, 121, 134, 138, 248-251, 25->26, 27->28, 67->68, 86->87, 137->138, 168->181, 395->392
2018-10-03T18:18:25.5010877Z src/tox/_quickstart.py       128     22     59      7    82%   76, 83, 91, 96-98, 102, 109, 127-138, 142-144, 285-287, 75->76, 82->83, 90->91, 108->109, 126->127, 186->196, 188->187
2018-10-03T18:18:25.5012231Z src/tox/config.py            889     14    339     17    97%   334, 809, 958, 1012, 1128, 1132, 1235, 1445-1446, 1457, 1465, 1522, 1536, 1571, 185->193, 333->334, 561->573, 808->809, 840->842, 954->958, 1011->1012, 1074->exit, 1127->1128, 1129->1132, 1234->1235, 1456->1457, 1464->1465, 1479->1489, 1521->1522, 1535->1536, 1552->1571
2018-10-03T18:18:25.5013175Z src/tox/interpreters.py      129      9     32      5    91%   55-57, 66-67, 82, 85-86, 159, 73->75, 81->82, 158->159, 173->exit, 178->exit
2018-10-03T18:18:25.5014165Z src/tox/package.py           122      7     45      4    93%   28-29, 47-48, 125-129, 27->28, 43->51, 113->116, 124->125
2018-10-03T18:18:25.5015322Z src/tox/result.py             51      1      4      1    96%   15, 14->15
2018-10-03T18:18:25.5016350Z src/tox/session.py           542     26    212     18    94%   53, 188, 208, 217-219, 253, 329, 454-461, 488-490, 495, 528-530, 615-616, 714, 728, 52->53, 186->188, 207->208, 223->exit, 232->234, 239->241, 252->253, 317->exit, 342->exit, 346->exit, 350->exit, 354->exit, 362->exit, 487->488, 494->495, 614->615, 713->714, 727->728
2018-10-03T18:18:25.5017370Z src/tox/venv.py              382     13    150      9    95%   129, 149, 219, 247-252, 402, 474-478, 503-504, 528, 94->98, 143->146, 146->149, 188->187, 273->exit, 401->402, 473->474, 500->464, 527->528
2018-10-03T18:18:25.5018163Z ----------------------------------------------------------------------
2018-10-03T18:18:25.5018555Z TOTAL                       2618    104    933     68    95%
2018-10-03T18:18:25.5019003Z 
2018-10-03T18:18:25.5019408Z 6 files skipped due to complete coverage.
2018-10-03T18:18:25.5125473Z coverage runtests: commands[3] | coverage xml -o /home/vsts/work/1/s/.tox/coverage.xml
2018-10-03T18:18:26.1652847Z coverage runtests: commands[4] | coverage html -d /home/vsts/work/1/s/.tox/htmlcov
2018-10-03T18:18:27.0073446Z coverage runtests: commands[5] | diff-cover --compare-branch origin/master /home/vsts/work/1/s/.tox/coverage.xml
2018-10-03T18:18:28.0643018Z -------------
2018-10-03T18:18:28.0644520Z Diff Coverage
2018-10-03T18:18:28.0655715Z Diff: origin/master...HEAD, staged and unstaged changes
2018-10-03T18:18:28.0656574Z -------------
2018-10-03T18:18:28.0657129Z No lines with coverage information in this diff.
2018-10-03T18:18:28.0657898Z -------------
2018-10-03T18:18:28.0917496Z ___________________________________ summary ____________________________________
2018-10-03T18:18:28.0919145Z   coverage: commands succeeded
2018-10-03T18:18:28.0920344Z   congratulations :)

[obestwalter]

hi @gaborbernat sorry I missed that (like many other things at the end of last year). I am reading through what happened in reverse chronological order at the moment and am completely stumped by the amazing work that you have put in 🙇‍♂️ 👌