CMake & build-script: add a new tier of testing, "long tests"

[gribozavr]

What's in this pull request?

We are adding a new tier of tests, "long tests", and moving SIL parsing tests into this tier.

• The behavior of build-script -t is unchanged.
• build-script -T continues to run primary and validation test suites, but without the long tests.
• build-script --long-test runs just the long tests.
• build-script -T --long-test runs all tests.

Tests:

$ ./utils/build-script -R
Building the standard library for: swift-test-stdlib-macosx-x86_64

$ ./utils/build-script -Rt
Building the standard library for: swift-test-stdlib-macosx-x86_64
Running Swift tests for: check-swift-macosx-x86_64

$ ./utils/build-script -RT
Building the standard library for: swift-stdlib-macosx-x86_64
Running Swift tests for: check-swift-validation-macosx-x86_64

$ ./utils/build-script -R --long-test
Building the standard library for: swift-test-stdlib-macosx-x86_64
Running Swift tests for: check-swift-only_long-macosx-x86_64

$ ./utils/build-script -RT --long-test
Building the standard library for: swift-stdlib-macosx-x86_64
Running Swift tests for: check-swift-all-macosx-x86_64

---

Before merging this pull request to apple/swift repository:

• Test pull request on Swift continuous integration.

Triggering Swift CI

The swift-ci is triggered by writing a comment on this PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment that you use. The currently available comments are:

Smoke Testing

Platform	Comment
All supported platforms	@swift-ci Please smoke test
OS X platform	@swift-ci Please smoke test OS X platform
Linux platform	@swift-ci Please smoke test Linux platform

Validation Testing

Platform	Comment
All supported platforms	@swift-ci Please test
OS X platform	@swift-ci Please test OS X platform
Linux platform	@swift-ci Please test Linux platform

Note: Only members of the Apple organization can trigger swift-ci.

[gribozavr]

@swift-ci Please test

[gribozavr]

@shahmishal @modocache Would you mind reviewing?

[modocache]

I don't think you need =1 here, so this should just be long-test. It may not seem like a big deal, but migrating build-script-impl arguments like this to Python ends up being tricky--Python's argparse does not allow numerical values for boolean command-line arguments. For more details, see #1745.

[modocache]

Oh! I see you've addressed this concern in the argument parser directly, with argparse_bool. Thanks for that!! 💯

[gottesmm]

For those who are just listening in, this change should significantly speed up all of our testing. The justification for this change is that these tests are mainly meant to ensure that when frontend engineers change the AST that we can always parse the stdlib.

In terms of getting this information, running these tests once a day should provide the coverage that we need given the amount of times that this type of issues happens.

As a huge benefit all of our normal testing should speed up significantly including the smoke tests. IIRC 1/3 - 1/2 of the smoke test testing time (ignoring building time) is related to running these tests.

[gottesmm]

(What I am trying to say is @gribozavr, go get em!) = ).

[modocache]

LGTM, thanks @gribozavr! It looks like this will turn off long_test by default, which I imagine will speed up all of the CI jobs as an immediate result. @shahmishal, are you planning on running long_test periodically?

Aside: could we run package tests at the same frequency as long_test? It's my understanding that those are also very time-consuming (please let me know if I'm mistaken).

[gottesmm]

@modocache On the test frequency, see my post above.

[gribozavr]

@modocache Thank you for the review!

We are planning on setting up a buildbot for master that runs long tests. I also enabled long tests in the presets that are used to produce packages (just pushed this change).

[gribozavr]

@swift-ci Please test