Asynchronous testing API [AsyncXCTest 6/6] #43
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,6 +11,12 @@ | |
| // Base class for test cases | ||
| // | ||
|
|
||
| #if os(Linux) || os(FreeBSD) | ||
| import Foundation | ||
| #else | ||
| import SwiftFoundation | ||
| #endif | ||
|
|
||
| /// This is a compound type used by `XCTMain` to represent tests to run. It combines an | ||
| /// `XCTestCase` subclass type with the list of test methods to invoke on the test case. | ||
| /// This type is intended to be produced by the `testCase` helper function. | ||
|
|
@@ -48,6 +54,12 @@ private func test<T: XCTestCase>(testFunc: T -> () throws -> Void) -> XCTestCase | |
| } | ||
| } | ||
|
|
||
| // FIXME: Expectations should be stored in an instance variable defined on | ||
| // XCTestCase, but when so defined Linux tests fail with "hidden symbol | ||
| // isn't defined". Use a global for the time being, as this seems to | ||
| // appease the Linux compiler. | ||
| private var XCTAllExpectations = [XCTestExpectation]() | ||
|
|
||
| extension XCTestCase { | ||
|
|
||
| public var continueAfterFailure: Bool { | ||
|
|
@@ -92,6 +104,8 @@ extension XCTestCase { | |
| } | ||
|
|
||
| testCase.tearDown() | ||
| testCase.failIfExpectationsNotWaitedFor(XCTAllExpectations) | ||
| XCTAllExpectations = [] | ||
|
|
||
| totalDuration += duration | ||
|
|
||
|
|
@@ -122,4 +136,161 @@ extension XCTestCase { | |
|
|
||
| XCTPrint("Executed \(tests.count) test\(testCountSuffix), with \(totalFailures) failure\(failureSuffix) (\(unexpectedFailures) unexpected) in \(printableStringForTimeInterval(totalDuration)) (\(printableStringForTimeInterval(overallDuration))) seconds") | ||
| } | ||
|
|
||
| /// It is an API violation to create expectations but not wait for them to | ||
| /// be completed. Notify the user of a mistake via a test failure. | ||
| private func failIfExpectationsNotWaitedFor(expectations: [XCTestExpectation]) { | ||
| if expectations.count > 0 { | ||
| let failure = XCTFailure( | ||
| message: "Failed due to unwaited expectations.", | ||
| failureDescription: "", | ||
| expected: false, | ||
| file: expectations.last!.file, | ||
| line: expectations.last!.line) | ||
| if let failureHandler = XCTFailureHandler { | ||
| failureHandler(failure) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| /// Creates and returns an expectation associated with the test case. | ||
|
There was a problem hiding this comment. Nice documentation on these! |
||
| /// | ||
| /// - Parameter description: This string will be displayed in the test log | ||
| /// to help diagnose failures. | ||
| /// - Parameter file: The file name to use in the error message if | ||
| /// this expectation is not waited for. Default is the file | ||
| /// containing the call to this method. It is rare to provide this | ||
| /// parameter when calling this method. | ||
| /// - Parameter line: The line number to use in the error message if the | ||
| /// this expectation is not waited for. Default is the line | ||
| /// number of the call to this method in the calling file. It is rare to | ||
| /// provide this parameter when calling this method. | ||
| /// | ||
| /// - Note: Whereas Objective-C XCTest determines the file and line | ||
| /// number of expectations that are created by using symbolication, this | ||
| /// implementation opts to take `file` and `line` as parameters instead. | ||
| /// As a result, the interface to these methods are not exactly identical | ||
| /// between these environments. To ensure compatibility of tests between | ||
| /// swift-corelibs-xctest and Apple XCTest, it is not recommended to pass | ||
| /// explicit values for `file` and `line`. | ||
| public func expectationWithDescription(description: String, file: StaticString = #file, line: UInt = #line) -> XCTestExpectation { | ||
| let expectation = XCTestExpectation( | ||
| description: description, | ||
| file: file, | ||
| line: line, | ||
| testCase: self) | ||
| XCTAllExpectations.append(expectation) | ||
| return expectation | ||
| } | ||
|
|
||
| /// Creates a point of synchronization in the flow of a test. Only one | ||
| /// "wait" can be active at any given time, but multiple discrete sequences | ||
| /// of { expectations -> wait } can be chained together. | ||
| /// | ||
| /// - Parameter timeout: The amount of time within which all expectation | ||
| /// must be fulfilled. | ||
| /// - Parameter file: The file name to use in the error message if | ||
| /// expectations are not met before the given timeout. Default is the file | ||
| /// containing the call to this method. It is rare to provide this | ||
| /// parameter when calling this method. | ||
| /// - Parameter line: The line number to use in the error message if the | ||
| /// expectations are not met before the given timeout. Default is the line | ||
| /// number of the call to this method in the calling file. It is rare to | ||
| /// provide this parameter when calling this method. | ||
| /// - Parameter handler: If provided, the handler will be invoked both on | ||
| /// timeout or fulfillment of all expectations. Timeout is always treated | ||
| /// as a test failure. | ||
| /// | ||
| /// - Note: Whereas Objective-C XCTest determines the file and line | ||
| /// number of the "wait" call using symbolication, this implementation | ||
| /// opts to take `file` and `line` as parameters instead. As a result, | ||
| /// the interface to these methods are not exactly identical between | ||
| /// these environments. To ensure compatibility of tests between | ||
| /// swift-corelibs-xctest and Apple XCTest, it is not recommended to pass | ||
| /// explicit values for `file` and `line`. | ||
| public func waitForExpectationsWithTimeout(timeout: NSTimeInterval, file: StaticString = #file, line: UInt = #line, handler: XCWaitCompletionHandler?) { | ||
| // Mirror Objective-C XCTest behavior; display an unexpected test | ||
| // failure when users wait without having first set expectations. | ||
| // FIXME: Objective-C XCTest raises an exception for most "API | ||
| // violation" failures, including this one. Normally this causes | ||
| // the test to stop cold. swift-corelibs-xctest does not stop, | ||
| // and executes the rest of the test. This discrepancy should be | ||
| // fixed. | ||
| if XCTAllExpectations.count == 0 { | ||
| let failure = XCTFailure( | ||
| message: "call made to wait without any expectations having been set.", | ||
| failureDescription: "API violation", | ||
| expected: false, | ||
| file: file, | ||
| line: line) | ||
| if let failureHandler = XCTFailureHandler { | ||
| failureHandler(failure) | ||
| } | ||
| return | ||
| } | ||
|
|
||
| // Objective-C XCTest outputs the descriptions of every unfulfilled | ||
| // expectation. We gather them into this array, which is also used | ||
| // to determine failure--a non-empty array meets expectations weren't | ||
| // met. | ||
| var unfulfilledDescriptions = [String]() | ||
|
|
||
| // We continue checking whether expectations have been fulfilled until | ||
| // the specified timeout has been reached. | ||
| // FIXME: Instead of polling the expectations to check whether they've | ||
| // been fulfilled, it would be more efficient to use a runloop | ||
| // source that can be signaled to wake up when an expectation is | ||
| // fulfilled. | ||
| let runLoop = NSRunLoop.currentRunLoop() | ||
| let timeoutDate = NSDate(timeIntervalSinceNow: timeout) | ||
| repeat { | ||
| unfulfilledDescriptions = [] | ||
| for expectation in XCTAllExpectations { | ||
| if !expectation.fulfilled { | ||
| unfulfilledDescriptions.append(expectation.description) | ||
| } | ||
| } | ||
|
|
||
| // If we've met all expectations, then break out of the specified | ||
| // timeout loop early. | ||
| if unfulfilledDescriptions.count == 0 { | ||
| break | ||
| } | ||
|
|
||
| // Otherwise, wait another fraction of a second. | ||
| runLoop.runUntilDate(NSDate(timeIntervalSinceNow: 0.01)) | ||
|
There was a problem hiding this comment. @parkera @phausler Here I've implemented asynchronous tests in XCTest using SwiftFoundation's In order to get this to work as it does with Objective-C Foundation, I had to make the following change to SwiftFoundation: diff --git a/Foundation/NSRunLoop.swift b/Foundation/NSRunLoop.swift
index 4434429..6288242 100644
--- a/Foundation/NSRunLoop.swift
+++ b/Foundation/NSRunLoop.swift
@@ -95,10 +95,6 @@ extension NSRunLoop {
return false
}
let modeArg = mode._cfObject
- if _CFRunLoopFinished(_cfRunLoop, modeArg) {
- return false
- }
-
let limitTime = limitDate.timeIntervalSinceReferenceDate
let ti = limitTime - CFAbsoluteTimeGetCurrent()
CFRunLoopRunInMode(modeArg, ti, true)I examined the implementation of There was a problem hiding this comment. Looks like the above works perfectly fine on Linux. Perhaps swift-corelibs-xctest should depend upon Objective-C Foundation when being built for OS X, instead of SwiftFoundation? |
||
| } while NSDate().compare(timeoutDate) == NSComparisonResult.OrderedAscending | ||
|
|
||
| if unfulfilledDescriptions.count > 0 { | ||
| // Not all expectations were fulfilled. Append a failure | ||
| // to the array of expectation-based failures. | ||
| let descriptions = unfulfilledDescriptions.joinWithSeparator(", ") | ||
| let failure = XCTFailure( | ||
| message: "Exceeded timeout of \(timeout) seconds, with unfulfilled expectations: \(descriptions)", | ||
| failureDescription: "Asynchronous wait failed", | ||
| expected: true, | ||
| file: file, | ||
| line: line) | ||
| if let failureHandler = XCTFailureHandler { | ||
| failureHandler(failure) | ||
| } | ||
| } | ||
|
|
||
| // We've recorded all the failures; clear the expectations that | ||
| // were set for this test case. | ||
| XCTAllExpectations = [] | ||
|
|
||
| // The handler is invoked regardless of whether the test passed. | ||
| if let completionHandler = handler { | ||
| var error: NSError? = nil | ||
| if unfulfilledDescriptions.count > 0 { | ||
| // If the test failed, send an error object. | ||
| error = NSError( | ||
| domain: "org.swift.XCTestErrorDomain", | ||
| code: 0, | ||
| userInfo: [:]) | ||
| } | ||
| completionHandler(error) | ||
| } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| // This source file is part of the Swift.org open source project | ||
| // | ||
| // Copyright (c) 2014 - 2016 Apple Inc. and the Swift project authors | ||
| // Licensed under Apache License v2.0 with Runtime Library Exception | ||
| // | ||
| // See http://swift.org/LICENSE.txt for license information | ||
| // See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors | ||
| // | ||
| // | ||
| // XCTestExpectation.swift | ||
| // Expectations represent specific conditions in asynchronous testing. | ||
| // | ||
|
|
||
| /// Expectations represent specific conditions in asynchronous testing. | ||
| public class XCTestExpectation { | ||
| internal let description: String | ||
| internal let file: StaticString | ||
| internal let line: UInt | ||
|
|
||
| internal var fulfilled = false | ||
| internal weak var testCase: XCTestCase? | ||
|
|
||
| internal init(description: String, file: StaticString, line: UInt, testCase: XCTestCase) { | ||
| self.description = description | ||
| self.file = file | ||
| self.line = line | ||
| self.testCase = testCase | ||
| } | ||
|
|
||
| /// Marks an expectation as having been met. It's an error to call this | ||
| /// method on an expectation that has already been fulfilled, or when the | ||
| /// test case that vended the expectation has already completed. | ||
| /// | ||
| /// - Parameter file: The file name to use in the error message if | ||
| /// expectations are not met before the given timeout. Default is the file | ||
| /// containing the call to this method. It is rare to provide this | ||
| /// parameter when calling this method. | ||
| /// - Parameter line: The line number to use in the error message if the | ||
| /// expectations are not met before the given timeout. Default is the line | ||
| /// number of the call to this method in the calling file. It is rare to | ||
| /// provide this parameter when calling this method. | ||
| /// | ||
| /// - Note: Whereas Objective-C XCTest determines the file and line | ||
| /// number the expectation was fulfilled using symbolication, this | ||
| /// implementation opts to take `file` and `line` as parameters instead. | ||
| /// As a result, the interface to these methods are not exactly identical | ||
| /// between these environments. To ensure compatibility of tests between | ||
| /// swift-corelibs-xctest and Apple XCTest, it is not recommended to pass | ||
| /// explicit values for `file` and `line`. | ||
| public func fulfill(file: StaticString = #file, line: UInt = #line) { | ||
| // FIXME: Objective-C XCTest emits failures when expectations are | ||
| // fulfilled after the test cases that generated those | ||
| // expectations have completed. Similarly, this should cause an | ||
| // error as well. | ||
| if fulfilled { | ||
| // Mirror Objective-C XCTest behavior: treat multiple calls to | ||
| // fulfill() as an unexpected failure. | ||
| let failure = XCTFailure( | ||
| message: "multiple calls made to XCTestExpectation.fulfill() for \(description).", | ||
| failureDescription: "API violation", | ||
| expected: false, | ||
| file: file, | ||
| line: line) | ||
| if let failureHandler = XCTFailureHandler { | ||
| failureHandler(failure) | ||
| } | ||
| } else { | ||
| fulfilled = true | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| // This source file is part of the Swift.org open source project | ||
| // | ||
| // Copyright (c) 2014 - 2015 Apple Inc. and the Swift project authors | ||
| // Licensed under Apache License v2.0 with Runtime Library Exception | ||
| // | ||
| // See http://swift.org/LICENSE.txt for license information | ||
| // See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors | ||
| // | ||
| // | ||
| // XCWaitCompletionHandler.swift | ||
| // A closure invoked by XCTestCase when a wait for expectations to be | ||
| // fulfilled times out. | ||
| // | ||
|
|
||
| #if os(Linux) || os(FreeBSD) | ||
| import Foundation | ||
| #else | ||
| import SwiftFoundation | ||
| #endif | ||
|
|
||
| /// A block to be invoked when a call to wait times out or has had all | ||
| /// associated expectations fulfilled. | ||
| /// | ||
| /// - Parameter error: If the wait timed out or a failure was raised while | ||
| /// waiting, the error's code will specify the type of failure. Otherwise | ||
| /// error will be nil. | ||
| public typealias XCWaitCompletionHandler = (NSError?) -> () |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm wondering how true it is that a
masterbuild of Foundation is necessary? I just hacked around a little bit and saw a successful build and test run using the Foundation+CoreFoundation shipped with the latest snapshot. Clearly there may be times where we depend on code only present on Foundationmasterbut I would imagine that won't be too frequent.Do you think it would be reasonable to treat
--foundation-build-diras an optional parameter to the build script, allowing the compiler to find a pre-installed Foundation library from a snapshot? This would simplify the use case where someone just wants to verify that their changes don't breakXCTest's Linux test suite without doing too much extra legwork getting a more complete swift.org development environment set up. The same logic applies to the--swiftcargument too, I think.It'd be pretty neat if people could spin up an environment from a Docker image like this, clone just
swift-corelibs-xctestand run./build_script.py --testwithout having to wait forswiftandllvmto build which (AFAIK?) is currently required when building swift-corelibs-foundation from source.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I specified
masterin the README as a matter of convenience--using the Swift CI we can guarantee that XCTest and Foundationmasterbuild together. Of course, it might be possible for people to contribute by linking the latest XCTestmasteragainst an older Foundation snapshot, but it'll be hard for us to keep track of which versions of XCTest works with which versions of Foundation.I totally agree with your suggestions to make it easier to contribute (and I think it's imperative we keep the barrier to entry as low as possible), but if possible I'd like to leave those as future improvements.