Skip to content

[Github Action] cli-test: improve naming summary #1448

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 5, 2023
Merged

[Github Action] cli-test: improve naming summary #1448

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8da9320
[Github Action] cli-test: improve naming summary
savil Sep 1, 2023
0a31cef
fixes from code review
savil Sep 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/cli-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ jobs:
tests:
uses: ./.github/workflows/cli-tests.yaml
with:
# This will run the basic testscript unit tests on MacOS.
# NOTE: cli-tests will NEVER run project-tests on MacOS.
run-mac-tests: true

edge:
Expand Down
19 changes: 9 additions & 10 deletions .github/workflows/cli-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,6 @@ on:
type: boolean
workflow_dispatch:
inputs:
run-devbox-json-tests:
type: boolean
run-mac-tests:
type: boolean
# run the example tests with DEVBOX_DEBUG=1
Expand Down Expand Up @@ -76,24 +74,25 @@ jobs:
strategy:
matrix:
is-main:
- ${{ github.ref == 'refs/heads/main' }}
- ${{ github.ref == 'refs/heads/main' && 'is-main' || 'not-main' }}
os: [ubuntu-latest, macos-latest]
# This is an optimization that runs tests twice, with and without
# the devbox.json tests. We can require the other tests to complete before
# merging, while keeping the others as an additional non-required signal
run-devbox-json-tests: [true, false]
run-project-tests: ["project-tests", "project-tests-off"]
# Run tests on:
# 1. the oldest supported nix version (which is 2.9.0? But determinate-systems installer has 2.12.0)
# 2. nix version 2.17.0 which introduces a new code path that minimizes nixpkgs downloads.
# 3. latest nix version (currently, that is 2.17.0, so omitted)
nix-version: ["2.12.0", "2.17.0"]
exclude:
- is-main: false
- is-main: "not-main"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that improving the labels is good, but lines like this make my head spin :(

os: "${{ inputs.run-mac-tests && 'dummy' || 'macos-latest' }}"
- is-main: true
run-devbox-json-tests: false
- run-devbox-json-tests: true
- is-main: "is-main"
run-project-tests: "project-tests"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be project-tests-off?

- run-project-tests: "project-tests"
os: macos-latest

runs-on: ${{ matrix.os }}
timeout-minutes: ${{ (github.ref == 'refs/heads/main' || inputs.run-mac-tests) && 37 || 25 }}
steps:
Expand Down Expand Up @@ -131,8 +130,8 @@ jobs:
env:
# For devbox.json tests, we default to non-debug mode since the debug output is less useful than for unit testscripts.
# But we allow overriding via inputs.example-debug
DEVBOX_DEBUG: ${{ (!matrix.run-devbox-json-tests || inputs.example-debug) && '1' || '0' }}
DEVBOX_RUN_DEVBOX_JSON_TESTS: ${{ matrix.run-devbox-json-tests }}
DEVBOX_DEBUG: ${{ (matrix.run-project-tests == 'not-project-tests' || inputs.example-debug) && '1' || '0' }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be project-tests-off?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch!

DEVBOX_RUN_PROJECT_TESTS: ${{ matrix.run-project-tests == 'project-tests' && '1' || '0' }}
# Used in `go test -timeout` flag. Needs a value that time.ParseDuration can parse.
DEVBOX_GOLANG_TEST_TIMEOUT: "${{ (github.ref == 'refs/heads/main' || inputs.run-mac-tests) && '35m' || '30m' }}"
run: |
Expand Down
16 changes: 8 additions & 8 deletions testscripts/testscripts_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ import (
"go.jetpack.io/devbox/testscripts/testrunner"
)

// When true, tests that `devbox run run_test` succeeds on every devbox.json
// When true, tests that `devbox run run_test` succeeds on every project (i.e. having devbox.json)
// found in examples/.. and testscripts/..
const runDevboxJSONTests = "DEVBOX_RUN_DEVBOX_JSON_TESTS"
const runProjectTests = "DEVBOX_RUN_PROJECT_TESTS"

func TestScripts(t *testing.T) {
// To run a specific test, say, testscripts/foo/bar.test.text, then run
Expand All @@ -24,21 +24,21 @@ func TestMain(m *testing.M) {

// TestExamples runs testscripts on the devbox-projects in the examples folder.
func TestExamples(t *testing.T) {
isOn, err := strconv.ParseBool(os.Getenv(runDevboxJSONTests))
isOn, err := strconv.ParseBool(os.Getenv(runProjectTests))
if err != nil || !isOn {
t.Skipf("Skipping TestExamples. To enable, set %s=1.", runDevboxJSONTests)
t.Skipf("Skipping TestExamples. To enable, set %s=1.", runProjectTests)
}

// To run a specific test, say, examples/foo/bar, then run
// go test ./testscripts -run TestExamples/foo_bar_run_test
testrunner.RunDevboxTestscripts(t, "../examples")
}

// TestScriptsWithDevboxJSON runs testscripts on the devbox-projects in the testscripts folder.
func TestScriptsWithDevboxJSON(t *testing.T) {
isOn, err := strconv.ParseBool(os.Getenv(runDevboxJSONTests))
// TestScriptsWithProjects runs testscripts on the devbox-projects in the testscripts folder.
func TestScriptsWithProjects(t *testing.T) {
isOn, err := strconv.ParseBool(os.Getenv(runProjectTests))
if err != nil || !isOn {
t.Skipf("Skipping TestExamples. To enable, set %s=1.", runDevboxJSONTests)
t.Skipf("Skipping TestScriptsWithProjects. To enable, set %s=1.", runProjectTests)
}

testrunner.RunDevboxTestscripts(t, ".")
Expand Down