Don't run the Java 11 CI tests on open PRs

[smarter]

Java 11 only failures are very rare, so running the CI tests on Java 11
for every commit pushed to an open PR doesn't seem worth it. Instead,
only run these tests on merged commits and release builds.

[anatoliykmetyuk]

Every such "merged-only" change increases the probability of putting us in hot water on critical releases. While the rational of speeding up CI is clear, we should think of a different strategy of running this extended set of tests with rare failures. Such, that it won't be a blocker for a release. What about not running these tests on merge but e.g. on nightly? Or a similar schedule?

[smarter]

What about not running these tests on merge but e.g. on nightly? Or a similar schedule?

I don't understand. These tests are run both on merge and on nightly. Running them only on nightly would make it harder to find the PR where they were broken, and I don't see how it would make it easier to do releases.

[anatoliykmetyuk]

I don't see how it would make it easier to do releases.

Merges are the only way to trigger releases. Failed tests block merges. A number of times our releases were blocked because of some SBT test that ran only on merges and was dependent on the version of Dotty. Since the release changed the version, it broke. As far as I remember, there were some other similar cases. If the release (merge) tests do not behave exactly the same way as the PR tests, we are open to surprises.

Running them only on nightly would make it harder to find the PR where they were broken

That's a tradeoff, speed vs precision. I don't think it'll be a big problem if we assume that the tests ran this way fail indeed very rarely.

[smarter]

We shouldn't do releases with failed tests. If we really need to, you can always edit .drone.yml to bypass the need to run the tests to publish the release (just edit the value of publish_on), but this is really not something we should need to do.