[swiftpm][tests] Rely on XCTest for xplat shim

[modocache]

The package manager tests perform some interesting acrobatics in order
to support unit tests on all platforms while also enforcing type safety.
Change this to rely on swift-corelibs-xctest as a shim layer, as that
library intends. This is the first step to using XCTMain() from within
the package manager tests.

swift-corelibs-xctest provides an API for developers to write unit
tests using XCTestCase. Since Swift XCTest cannot automatically detect test
methods like Obj-C XCTest can, Swift XCTest defines XCTestCase as
conforming to XCTestCaseProvider, which defines the allTests public
variable. The shim layer exists in swift-corelibs-xctest, and this is
where is should be.

The package manager currently explicitly marks all of its test cases as
conforming to XCTestCaseProvider. XCTestCaseProvider isn't available on
all platforms, so the package manager tests use macros to conditionally
define it. The logic for this conditional definition is convoluted:

• All test cases conform to XCTestCaseProvider, regardless of platform.
• XCTestCaseProvider.swift defines XCTestCaseProvider as an empty
  protocol if the current platform is OS X (which uses Obj-C XCTest and thus
  doesn't define XCTestCaseProvider). This file is not included in any
  of the test targets, so it is not included when building from the command
  line on OS X or Linux. It is included in the Xcode project, thus
  allowing tests to build in Xcode.
• Each target's main.swift defines XCTestCaseProvider as a protocol
  that defines the allTests public variable, but only if the platform
  is not Linux (on Linux, XCTestCaseProvider is already defined). These
  files are used when building from the command-line, but are not
  included in the Xcode project (thus the Xcode project does not have
  duplicate definitions of XCTestCaseProvider).

The benefit to doing all this is to ensure that all package manager
tests conform to the XCTestCaseProvider protocol, and so developers
are informed at compile-time whether their tests will work on Linux.
This is great, but not worth the complexity above.

[ddunbar]

I am not convinced this isn't worth the complexity, we need adding tests to be as reliable as possible.

I actually want to go the other direction, and have the OS X build do more work to audit that the allTests array is declared correctly.

I would be interested in seeing if there are other ways we can simplify the current implementation though -- I have forgotten why we ended up declaring the protocol in each test target, I think it may have been because we didn't have a simple way to put it somewhere to be shared. We should be able to solve that.

Note also that all of this is hopefully temporary, just until the package manager itself really supports building test targets.

What specific problem are you trying to solve? Just extra complexity?

[modocache]

What specific problem are you trying to solve? Just extra complexity?

My primary goal is to have the test targets use XCTMain(). I'd like to ensure that all clients of swift-corelibs-xctest use that as an entry point. If I can reduce complexity while doing so that'd be great, but I'm beginning to understand the package manager tests are necessarily complex when compared to, say, swift-corelibs-foundation.

I would be interested in seeing if there are other ways we can simplify the current implementation though -- I have forgotten why we ended up declaring the protocol in each test target, I think it may have been because we didn't have a simple way to put it somewhere to be shared. We should be able to solve that.

This pull request was one of several approaches I tried. I also thought of having XCTestProvider be its own package, and have each of the test targets rely on it as a dependency. I can explore this further if you think that's the way to go.

[modocache]

I am not convinced this isn't worth the complexity, we need adding tests to be as reliable as possible.

This may be something we need to discuss further on the mailing list. It would be fantastic if swift-corelibs-xctest could provide a way to write cross-platform tests while also ensuring that developers know when they're writing tests that won't work on Linux (because they don't implement allTests).

[ddunbar]

I think what we hope to do there is get the necessary reflection features in place so that we can have some kind of auto-test discovery. Maintaining allTests isn't intended to be a long term solution...

We can tackle short term fixes to reduce the pain for now, but if we are going to invest serious work then that should be focused at getting the right long term solution in place.

[modocache]

I think what we hope to do there is get the necessary reflection features in place so that we can have some kind of auto-test discovery.

This would be really, really cool. 💯

We can tackle short term fixes to reduce the pain for now, but if we are going to invest serious work then that should be focused at getting the right long term solution in place.

Totally agree. Even in the short- or medium-term, though, I'd like to improve swift-corelibs-xctest, and I view XCTMain() as the canonical entry point into that system. In that vein, I'd like to have the package manager use it as well. Do you agree that this is worth pursuing?

[ddunbar]

Yes, what would a minimal patch to just move to use XCTMain() look like?

[modocache]

Working on it! 🙇

[modocache]

Closing in favor of #66.