[XCTestObservation] Add XCTestSuite announcements #84
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
modocache
force-pushed
the
observation-test-suite
branch
2 times, most recently
from
93fe8df to
2634dc1
Compare
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Mar 28, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
|
This seems like a reasonable first step. I agree generally with your assessment in the second "downside" you mention. The weirdness around having the "All" suite and the bundle suite be peers in a list of root suites instead of having a containment relationship arises from this, I guess. I would definitely like to see continued movement towards having test suites constructed and then used to control the running of the tests instead of just being constructed in order to send the observations. But I am OK with taking this in incremental stages. |
|
@swift-ci please test |
|
Here's the error from Linux CI: I think this is the error @briancroom mentioned in #69 (is there a JIRA issue for this problem?). I'll try to address this in the coming days. |
modocache
force-pushed
the
observation-test-suite
branch
from
2634dc1 to
958582e
Compare
|
@mike-ferris-apple I've resolved the Linux build issue. Could you ask @swift-ci to please test? If the tests pass, I'll merge this to achieve the "incremental stages" you describe (or you can request "@swift-ci please test and merge"). #86 is the realization of the "continued movement towards having test suites constructed and then used to control the running of the tests" you describe. |
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 2, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 2, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 2, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
|
@swift-ci please test |
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 3, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
force-pushed
the
observation-test-suite
branch
from
958582e to
584cb1e
Compare
Apple XCTest's `XCTestObservation` protocol includes methods that announce when an `XCTestSuite` will begin executing, or has finished executing. Add these same announcements to swift-corelibs-xctest. swift-corelibs-xctest did not already defined `XCTestSuite`, so add it. Apple XCTest allows `XCTestObservation` listeners to determine the number of passing and failing tests by accessing an `XCTestSuite`'s `testRun` property, but this is not yet supported in swift-corelibs-xctest.
modocache
force-pushed
the
observation-test-suite
branch
from
584cb1e to
5274cdd
Compare
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 3, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 6, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 6, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 6, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 6, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
pushed a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 6, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 6, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 6, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 6, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 6, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
modocache
added a commit
to modocache/swift-corelibs-xctest
that referenced
this pull request
Apr 7, 2016
A major goal for swift-corelibs-xctest is API parity with Apple XCTest. This adds the largest missing API in swift-corelibs-xctest: `XCTestRun`. In Apple XCTest, `XCTestRun` is responsible for keeping track of the result of a test run. It's an integral part of how Apple XCTest works. swift-corelibs-xctest, on the other hand, used a global array of `XCTRun` structs to keep track of how many tests passed/failed. While it may have been possible to tack on `XCTestRun` to the swift-corelibs-xctest mechanism for failure reporting, this commit instead fully integrates it. As a result, the changes are widespread: gone is `XCTFailureHandler`, `XCTRun`, and other internal structures. In their place, welcome the Apple XCTest public APIs: the `XCTest` abstract class, `XCTestRun`, and its subclasses `XCTestCaseRun` and `XCTestSuiteRun`. In conjunction with the new `XCTestSuite`-related observation methods from swiftlang#84, test reporting is now done exclusively through `XCTestObservation`. As a result, test output is now nearly identical to Apple XCTest.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's in this pull request?
Apple XCTest's
XCTestObservationprotocol includes methods that announce when anXCTestSuitewill begin executing, or has finished executing. Add these same announcements to swift-corelibs-xctest.swift-corelibs-xctest did not already defined
XCTestSuite, so add it.Apple XCTest allows
XCTestObservationlisteners to determine the number of passing and failing tests by accessing anXCTestSuite'stestRunproperty, but this is not yet supported in swift-corelibs-xctest.Why merge this pull request?
XCTestObservationidentical, bringing us closer to our Swift 3 goal of API parity.What are the downsides of merging this pull request?
XCTestSuiteto the codebase, but notXCTestRun. As a result, the most interesting aspect of test announcements (how many tests passed/failed) is still unimplemented. Still, I feel like this is a positive step forward, and one that keeps this pull request small enough to actually review.XCTestSuiteto represent the collection of test cases it executes. As a result, the quality of implementation here leaves much to be desired. On the other hand, refactoring the codebase to seamless integrateXCTestSuitewould involve more widespread changes, which would make this pull request more difficult to review. If you ask me, thanks to the functional test cases, we could merge this, then easily refactor in a subsequent pull request.