Enable Github Action #3862

yutannihilation · 2020-03-06T14:18:36Z

Fix #3760, also fixes #3702.

Major changes

Enable the following GitHub Actions:
- R-CMD-check: Run R CMD check on Windows, macOS, and Ubuntu with the last 5 minor versions of R.
- pr-command: Commenting /document will trigger devtools::document() automatically (example: Update autolayer.r yutannihilation/ggplot2#3 (comment)).
- pkgdown: build the pkgdown site.
~~Remove Travis and AppVeyor settings~~ Stop building pkgdown site by Travis (Now I feel it's safe to leave them for now and remove when we confirm GitHub Actions CI does the right thing. But, pkgdown site should not be built twice, so I tweaked .travis.yml only about this point)

Diffs between the default YAML files are following:

diff --git a/.github/workflows/R-CMD-check.yaml b/.github/workflows/R-CMD-check.yaml
index 0d76f6e0..075d5167 100644
--- a/.github/workflows/R-CMD-check.yaml
+++ b/.github/workflows/R-CMD-check.yaml
@@ -18,18 +18,23 @@ jobs:
       fail-fast: false
       matrix:
         config:
-          - {os: windows-latest, r: '3.6'}
-          - {os: macOS-latest, r: '3.6'}
-          - {os: macOS-latest, r: 'devel'}
-          - {os: ubuntu-16.04, r: '3.2', cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest"}
-          - {os: ubuntu-16.04, r: '3.3', cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest"}
-          - {os: ubuntu-16.04, r: '3.4', cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest"}
-          - {os: ubuntu-16.04, r: '3.5', cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest"}
-          - {os: ubuntu-16.04, r: '3.6', cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest"}
+          - {os: windows-latest, r: '3.6', vdiffr: true}
+          - {os: macOS-latest, r: '3.6', vdiffr: true}
+          - {os: macOS-latest, r: 'devel', vdiffr: false}
+          - {os: ubuntu-16.04, r: '3.2', vdiffr: false, cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest"}
+          - {os: ubuntu-16.04, r: '3.3', vdiffr: false, cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest"}
+          - {os: ubuntu-16.04, r: '3.4', vdiffr: false, cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest"}
+          - {os: ubuntu-16.04, r: '3.5', vdiffr: false, cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest"}
+          - {os: ubuntu-16.04, r: '3.6', vdiffr: true, cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest"}
 
     env:
       R_REMOTES_NO_ERRORS_FROM_WARNINGS: true
       CRAN: ${{ matrix.config.cran }}
+      # don't treat missing suggested packages as error
+      _R_CHECK_FORCE_SUGGESTS_: false
+      # Runs vdiffr test only on the latest version of R
+      VDIFFR_RUN_TESTS: ${{ matrix.config.vdiffr }}
+      VDIFFR_LOG_PATH: "../vdiffr.Rout.fail"
 
     steps:
       - uses: actions/checkout@v2
@@ -40,6 +45,18 @@ jobs:
 
       - uses: r-lib/actions/setup-pandoc@master
 
+      - name: Install XQuartz on macOS
+        if: runner.os == 'macOS'
+        run: brew cask install xquartz
+
+      # To install vdiffr, these three libraries/tools are needed:
+      # - freetype (already installed, needed by systemfonts)
+      # - cairo (not installed, needed by gdtools)
+      # - pkg-config (not installed, needed to set proper build settings)
+      - name: Install pkg-config and cairo on devel macOS
+        if: runner.os == 'macOS' && matrix.config.r == 'devel'
+        run: brew install pkg-config cairo
+
       - name: Query dependencies
         run: |
           install.packages('remotes')
diff --git a/.github/workflows/pkgdown.yaml b/.github/workflows/pkgdown.yaml
index bd7cce52..d80b2236 100644
--- a/.github/workflows/pkgdown.yaml
+++ b/.github/workflows/pkgdown.yaml
@@ -15,6 +15,7 @@ jobs:
         run: |
           install.packages("remotes")
           remotes::install_deps(dependencies = TRUE)
+          remotes::install_github("tidyverse/tidytemplate")
           remotes::install_dev("pkgdown")
         shell: Rscript {0}
       - name: Install package

Minor fixes

To enable these GitHub actions, I needed to do some tweaks

test-conditions.R errors when it runs in covr, trying to parse a non-R script file ggplot2.rdb. Added pattern argument to list.files() so that only R script files are parsed.
test-geom-quantile.R fails on Ubuntu with R >=3.4. This seems because all.equal.tbl_df() doesn't accept tolerance (No tolerance in dplyr's all.equal.tbl_df() dplyr#2264). So, use data.frame instead of tibble to compare them with tolerance.

Test run

I tested this setting on my forked repo and confirmed they work. Note that macOS-latest (3.6) fails only because the token for codecov is not set yet.

https://github.com/yutannihilation/ggplot2/actions/runs/51341080

yutannihilation · 2020-03-06T15:27:57Z

Phew, this seems really challenging... Currently I'm stuck with these issues:

vdiffr tests won't run on macOS (c.f. Visual tests fails on GitHub Actions (macOS) r-lib/vdiffr#80)
Unknown failure somehow related to quantiles

* checking tests ...
  Running ‘testthat.R’
 ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
  ggplot_data[ggplot_data$quantile == 0.5, c("x", "y")] not equal to `pred_rq_test_50`.
  Rows in x but not y: 86, 82, 81, 77, 75, 71, 67, 63, 85, 54, 41[...]. Rows in y but not x: 86, 85, 82, 77, 75, 71, 67, 63, 54, 41, 37[...]. 
  
  ── 3. Failure: geom_quantile matches quantile regression (@test-geom-quantile.R#
  ggplot_data[ggplot_data$quantile == 0.75, c("x", "y")] not equal to `pred_rq_test_75`.
  Rows in x but not y: 92, 89, 84, 79, 75, 68, 64, 62, 36, 32, 27[...]. Rows in y but not x: 92, 89, 84, 79, 75, 68, 64, 62, 36, 32, 27[...]. 
  
  ══ testthat results  ═══════════════════════════════════════════════════════════
  [ OK: 1407 | SKIPPED: 227 | WARNINGS: 0 | FAILED: 3 ]
  1. Failure: geom_quantile matches quantile regression (@test-geom-quantile.R#33) 
  2. Failure: geom_quantile matches quantile regression (@test-geom-quantile.R#41) 
  3. Failure: geom_quantile matches quantile regression (@test-geom-quantile.R#49)

jimhester · 2020-03-06T18:43:55Z

Also @malcolmbarrett was working on this during the tidyverse dev day (https://github.com/malcolmbarrett/ggplot2/pull/1) you might want to pull in his changes here.

malcolmbarrett · 2020-03-06T19:13:53Z

Yes, let me know if I can help in any way. I got most of the way there, but I seemed to be stuck with latticeExtra having this new dep on R 3.6, which cascaded into the <3.6 builds.

clauswilke · 2020-03-06T19:31:59Z

If the comment regarding latticeExtra relates to ggplot2, this is now fixed in master.

malcolmbarrett · 2020-03-06T20:46:41Z

Oh, great. I reran mine with upstream changes but it looks more or less the same as @yutannihilation's now. Since he's working on it, I'll close mine.

If it's of use, here are the most recent GHA outputs.

And the YAML I used:

pkgdown.yml

on:
  push:
    branches: master

name: Pkgdown

jobs:
  pkgdown:
    runs-on: macOS-latest
    steps:
      - uses: actions/checkout@master
      - uses: r-lib/actions/setup-r@master
      - uses: r-lib/actions/setup-pandoc@master
      - name: Install dependencies
        run: |
          Rscript -e 'install.packages("remotes")' \
                  -e 'remotes::install_deps(dependencies = TRUE)' \
                  -e 'remotes::install_github("jimhester/pkgdown@github-actions-deploy")'
      - name: Install package
        run: R CMD INSTALL .
      - name: Deploy package
        run: |
          pkgdown:::deploy_local(new_process = FALSE, remote_url = 'https://x-access-token:${{secrets.DEPLOY_PAT}}@github.com/${{github.repository}}.git')
        shell: Rscript {0}

R-CMD-check.yml

on: push: branches: - master pull_request: branches: - master

name: R-CMD-check

jobs:
R-CMD-check:
runs-on: ${{ matrix.config.os }}

name: ${{ matrix.config.os }} (${{ matrix.config.r }})

strategy:
  fail-fast: false
  matrix:
    config:
    - { os: windows-latest, r: '3.6', vdiff: true }
    - { os: macOS-latest, r: '3.6', vdiff: true }
    - { os: macOS-latest, r: 'devel', vdiff: false }
    - { os: ubuntu-16.04, r: '3.2', cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest", vdiff: false }
    - { os: ubuntu-16.04, r: '3.3', cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest", vdiff: false }
    - { os: ubuntu-16.04, r: '3.4', cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest", vdiff: false }
    - { os: ubuntu-16.04, r: '3.5', cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest", vdiff: false }
    - { os: ubuntu-16.04, r: '3.6', cran: "https://demo.rstudiopm.com/all/__linux__/xenial/latest", vdiff: false }

env:
  R_REMOTES_NO_ERRORS_FROM_WARNINGS: true
  CRAN: ${{ matrix.config.cran }}
  _R_CHECK_FORCE_SUGGESTS_: false
  VDIFFR_RUN_TESTS: ${{ matrix.config.vdiff }}
  VDIFFR_LOG_PATH: "../vdiffr.Rout.fail"

steps:
  - uses: actions/checkout@v1

  - uses: r-lib/actions/setup-r@master
    with:
      r-version: ${{ matrix.config.r }}

  - uses: r-lib/actions/setup-pandoc@master

  - name: Install XQuartz
    if: runner.os == 'macOS'
    run: brew cask install xquartz

  - name: Query dependencies
    run: Rscript -e "install.packages('remotes')" -e "saveRDS(remotes::dev_package_deps(dependencies = TRUE), 'depends.Rds', version = 2)"

  - name: Cache R packages
    if: runner.os != 'Windows'
    uses: actions/cache@v1
    with:
      path: ${{ env.R_LIBS_USER }}
      key: ${{ runner.os }}-r-${{ matrix.config.r }}-${{ hashFiles('depends.Rds') }}
      restore-keys: ${{ runner.os }}-r-${{ matrix.config.r }}-

  - name: Install system dependencies
    if: runner.os == 'Linux'
    env:
      RHUB_PLATFORM: linux-x86_64-ubuntu-gcc
    run: |
      Rscript -e "remotes::install_github('r-hub/sysreqs')"
      sysreqs=$(Rscript -e "cat(sysreqs::sysreq_commands('DESCRIPTION'))")
      sudo -s eval "$sysreqs"
      sudo add-apt-repository ppa:ubuntugis/ppa
      sudo apt-get update
      sudo apt-get install -y libudunits2-dev libproj-dev libgeos-dev libgdal-dev

  - name: Install dependencies
    run: Rscript -e "library(remotes)" -e "update(readRDS('depends.Rds'))" -e "remotes::install_cran('rcmdcheck')"

  - name: Check
    run: Rscript -e "rcmdcheck::rcmdcheck(args = '--no-manual', error_on = 'warning', check_dir = 'check')"

  - name: Upload check results
    if: failure()
    uses: actions/upload-artifact@master
    with:
      name: ${{ runner.os }}-r${{ matrix.config.r }}-results
      path: check

  - name: Test coverage
    if: matrix.config.os == 'macOS-latest' && matrix.config.r == '3.6'
    run: |
      Rscript -e 'covr::codecov(token = "${{secrets.CODECOV_TOKEN}}")'

@malcolmbarrett

Thanks @malcolmbarrett!

yutannihilation · 2020-03-07T01:35:00Z

Oh, sorry for taking your job @malcolmbarrett... I was wondering where is your working branch. Do you want to continue working on this? If so, I'll withdraw mine.

yutannihilation · 2020-03-07T15:52:22Z

This PR is ready for review now.

yutannihilation · 2020-03-08T02:32:21Z

@thomasp85 @clauswilke
Can any of you review this? I think this PR is safe to merge as this only adds GitHub Actions settings and do almost nothing on the current Travis CI and AppVeyor settings (except for removing lines about building pkgdown site). I think we need some time to see if this setting is proper, so let's run both the existing CIs (Travis CI and AppVeypr) and GitHub Actions for a while.

clauswilke

I can't judge the github actions parts, but the rest looks fine to me.

yutannihilation · 2020-03-08T04:10:23Z

Thanks!

@jimhester
Could you set tokens for GitHub pkgdown build and codecov?

thomasp85

lgtm

thomasp85 · 2020-03-09T08:41:25Z

Actually... Will the new pkgdown action push dev versions to the main page? I'm still not completely clear on how this is handled?

@jimhester can we have a pkgdown action that triggers when making a new GitHub release?

yutannihilation · 2020-03-09T09:10:35Z

Whether the built site will go to /dev/ or not is determined by the following rule of pkgdown; the GitHub Action is not related here. It just pushes what is built into gh-pages.

https://pkgdown.r-lib.org/reference/build_site.html#development-mode

yutannihilation · 2020-03-09T09:16:49Z

can we have a pkgdown action that triggers when making a new GitHub release?

It seems we can add this. May I add?

on:
  push:
    tags:        
      - v*

thomasp85 · 2020-03-09T09:23:10Z

If we do we should also make sure that the pkgdown site is build based on the release, rather than current master

yutannihilation · 2020-03-09T11:40:46Z

Sure.

jimhester · 2020-03-09T16:02:30Z

You can have it run on published releases with the following

on:
  release: 
    types: published

But maybe using the tag as @yutannihilation suggested would be an easier approach

jimhester · 2020-03-09T16:04:42Z

.github/workflows/R-CMD-check.yaml

+
+      - name: Test coverage
+        if: matrix.config.os == 'macOS-latest' && matrix.config.r == '3.6'
+        run: covr::codecov(token = "${{secrets.CODECOV_TOKEN}}")


With latest version of covr and changes in codecov.io you no longer need a token here.

Suggested change

run: covr::codecov(token = "${{secrets.CODECOV_TOKEN}}")

run: covr::codecov()

jimhester · 2020-03-09T16:06:31Z

Could you set tokens for GitHub pkgdown build and codecov?

with the dev version of pkgdown you no longer need a token for pkgdown deploys, and with the latest CRAN covr you don't need a token for codecov.io results either, so this should be good to go once you change the codecov line.

yutannihilation · 2020-03-10T13:58:45Z

Wow, thanks for very useful advises! I didn't notice we are free from tokens now. Cool.

yutannihilation · 2020-03-15T12:59:26Z

I'm merging this in the hope that I'll have some time in the coming weeks so I can handle troubles related to this... Please file an issues if you find some strange behavior.

yutannihilation · 2020-03-15T13:04:54Z

Seems something is wrong with crandeps... I'll retry later.

Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
  cannot download all files
Calls: cat ... get_cran_deps -> download_json -> download -> <Anonymous>
In addition: Warning message:
In utils::download.file(url, path, method = download_method(), quiet = quiet,  :
  URL 'https://crandeps.r-pkg.org:443/deps/ggplot2,R,digest,glue,grDevices,grid,gtable,isoband,MASS,mgcv,rlang,scales,stats,tibble,withr,covr,dplyr,ggplot2movies,hexbin,Hmisc,knitr,lattice,mapproj,maps,maptools,multcomp,munsell,nlme,profvis,quantreg,rgeos,rmarkdown,rpart,sf,svglite,testthat,vdiffr,sp': status was '404 Not Found'
Execution halted
##[error]Process completed with exit code 1.

The codecov service seems to have issues with HTTP 2 and projects with lots of data, e.g. ggplot2. Requiring HTTP 1.1 seems to fix the issue. Fixes tidyverse/ggplot2#3862

jimhester · 2020-03-16T14:10:39Z

FWIW I get this same PROTOCOL_ERROR when trying to upload the ggplot2 coverage results locally to codecov.io. So it is not a GitHub Actions specific issue. I have never seen this in any other project, but explicitly setting the POST request to use HTTP 1.1 resolves it. I just pushed a change to covr to use HTTP 1.1 requests unconditionally, which should fix this.

yutannihilation · 2020-03-16T14:45:37Z

@jimhester
Thanks for sharing the information! I guess this issue is triggered only when the request size is huge enough to use multiple streams. So, this seems why we see this error only on ggplot2's repo, which has about 4 million lines of code.

Just curious,

when trying to upload the ggplot2 coverage results locally

do you mean you used your local macOS? If so, this might be a problem with LibreSSL bundled with macOS. This explains the difference that covr::codecov() succeeds on Travis CI.

c.f. https://learnings.bolmaster2.com/posts/curl-openssl-tlsv1.3-on-macos.html

Anyway, sticking with HTTP 1.1 seems a good idea considering those situations.

Enable Github Action

84fb8f8

yutannihilation marked this pull request as ready for review March 6, 2020 14:19

yutannihilation changed the title ~~Enable Github Action~~ WIP: Enable Github Action Mar 6, 2020

yutannihilation added 3 commits March 6, 2020 23:45

Run vdiffr tests only on macOS + 3.6

b58dc28

Use tidytemplate

b0468f9

Add more settings

d7d1946

yutannihilation added 2 commits March 7, 2020 10:25

Install XQuartz

2e40bd5

Improve R-CMD-check.yaml

c2e9975

Thanks @malcolmbarrett!

yutannihilation added 11 commits March 7, 2020 10:54

Try setting tolerance

c68bc98

Use the dev version of covr

cd1b64b

Revert

933458d

Fix covr

ca427f3

Match with $

49b38dd

Install freetype

b6a8478

Fix installing freetype

ab8cb1d

Install pkg-config

3b1edaf

Install cairo as well

78622fc

Do not convert to tibble to compare with tolerance

915c85e

Fix test

cdc50a8

yutannihilation changed the title ~~WIP: Enable Github Action~~ Enable Github Action Mar 7, 2020

yutannihilation added 2 commits March 8, 2020 11:19

Revert .travis.yml and appvayeor.yml

bc8329a

Do not build pkgdown site on Travis

4eb83ff

yutannihilation requested a review from thomasp85 March 8, 2020 02:28

yutannihilation requested a review from clauswilke March 8, 2020 02:28

clauswilke approved these changes Mar 8, 2020

View reviewed changes

thomasp85 added this to the ggplot2 3.3.1 milestone Mar 9, 2020

thomasp85 approved these changes Mar 9, 2020

View reviewed changes

Build pkgdown site on tagging

2591766

Quate *

36bd6f4

jimhester reviewed Mar 9, 2020

View reviewed changes

yutannihilation added 2 commits March 10, 2020 22:52

Use release event instead of tag

949c186

Do not specify token

99dedea

yutannihilation merged commit c8c41cb into tidyverse:master Mar 15, 2020

yutannihilation deleted the ci/issue-3760-github-action branch March 15, 2020 12:59

yutannihilation mentioned this pull request Mar 15, 2020

Temporarily disable Test coverage step on GitHub Action CI #3895

Merged

jimhester added a commit to r-lib/covr that referenced this pull request Mar 16, 2020

Enforce using HTTP 1.1

cc71080

The codecov service seems to have issues with HTTP 2 and projects with lots of data, e.g. ggplot2. Requiring HTTP 1.1 seems to fix the issue. Fixes tidyverse/ggplot2#3862

yutannihilation mentioned this pull request Apr 8, 2020

Various fixes on GitHub Actions CI #3933

Merged

This was referenced Jun 9, 2020

Fix CI failures #4046

Merged

Stop using Travis CI and AppVeyor #4053

Merged

	run: covr::codecov(token = "${{secrets.CODECOV_TOKEN}}")
	run: covr::codecov()

Enable Github Action #3862

Enable Github Action #3862

Uh oh!

Conversation

yutannihilation commented Mar 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Major changes

Minor fixes

Test run

Uh oh!

yutannihilation commented Mar 6, 2020

Uh oh!

jimhester commented Mar 6, 2020

Uh oh!

malcolmbarrett commented Mar 6, 2020

Uh oh!

clauswilke commented Mar 6, 2020

Uh oh!

malcolmbarrett commented Mar 6, 2020

Uh oh!

yutannihilation commented Mar 7, 2020

Uh oh!

yutannihilation commented Mar 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yutannihilation commented Mar 8, 2020

Uh oh!

clauswilke left a comment

Choose a reason for hiding this comment

Uh oh!

yutannihilation commented Mar 8, 2020

Uh oh!

thomasp85 left a comment

Choose a reason for hiding this comment

Uh oh!

thomasp85 commented Mar 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yutannihilation commented Mar 9, 2020

Uh oh!

yutannihilation commented Mar 9, 2020

Uh oh!

thomasp85 commented Mar 9, 2020

Uh oh!

yutannihilation commented Mar 9, 2020

Uh oh!

jimhester commented Mar 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jimhester Mar 9, 2020

Choose a reason for hiding this comment

Uh oh!

jimhester commented Mar 9, 2020

Uh oh!

yutannihilation commented Mar 10, 2020

Uh oh!

yutannihilation commented Mar 15, 2020

Uh oh!

yutannihilation commented Mar 15, 2020

Uh oh!

jimhester commented Mar 16, 2020

Uh oh!

yutannihilation commented Mar 16, 2020

Uh oh!

Uh oh!

yutannihilation commented Mar 6, 2020 •

edited

Loading

yutannihilation commented Mar 7, 2020 •

edited

Loading

thomasp85 commented Mar 9, 2020 •

edited

Loading

jimhester commented Mar 9, 2020 •

edited

Loading