[XCTestCase] Add asynchronous testing API

Mirror the Objective-C XCTest API to add asynchronous testing support,
primarily via two methods:

- `XCTestCase.expectationWithDescription()`
- `XCTestCase.waitForExpectationsWithTimeout()`

These methods' APIs are identical to their Objective-C SDK overlay
counterparts, with the following exceptions:

1. If expectations are created but not waited for, Objective-C XCTest
   generates a test failure. In order to display a failure on the exact
   line of the last expectation that was created, Objective-C XCTest
   symbolicates the call stack. It does so using a private framework
   called `CoreSymbolication.framework`. We don't have this at our
   disposal for swift-corelibs-xctest, so instead we take `file` and
   `line` as parameters, defaulting to `__FILE__` and `__LINE__`. By
   doing so, we provide the same failure highlighting functionality
   while also maintaining a (nearly) identical API.
2. If `waitForExpectationsWithTimeout()` fails, Objective-C XCTest
   generates a test failure on the exact line that method was called,
   using the same technique from (1). For the same reason, this method
   also takes `file` and `line` parameters in swift-corelibs-xctest.
3. `waitForExpectationsWithTimeout()` takes a handler callback, which
   in Objective-C XCTest is passed an instance of `NSError`.
   swift-corelibs-xctest doesn't depend upon swift-corelibs-foundation,
   and so does not have access to `NSError`. Instead, it defines a shim,
   `XCTError`.

Author: modocache

Sources/XCTest/XCTestCase.swift

@@ -11,6 +11,12 @@
 //  Base protocol (and extension with default methods) for test cases
 //
 
+#if os(Linux) || os(FreeBSD)
+    // FIXME: Import swift-corelibs-foundation or swift-corelibs-libdispatch
+#else
+    import Foundation
+#endif
+
 public protocol XCTestCase : XCTestCaseProvider {
     func setUp()
     func tearDown()
@@ -61,6 +67,23 @@ extension XCTestCase {
 
                 tearDown()
 
+                // It is an API violation to create expectations but not wait
+                // for them to be completed. Notify the user of a mistake via
+                // a test failure.
+                if XCTAllExpectations.count > 0 {
+                    let failure = XCTFailure(
+                        message: "Failed due to unwaited expectations.",
+                        failureDescription: "",
+                        expected: true,
+                        file: XCTLatestExpectationLocation.file,
+                        line: XCTLatestExpectationLocation.line)
+                    XCTAllExpectationFailures.append(failure)
+                }
+
+                failures += XCTAllExpectationFailures
+                XCTLatestExpectationLocation = (file: "", line: 0)
+                XCTAllExpectationFailures = []
+
                 totalDuration += duration
 
                 var result = "passed"
@@ -98,4 +121,132 @@ extension XCTestCase {
     public func tearDown() {
 
     }
+
+    /// Creates and returns an expectation associated with the test case.
+    ///
+    /// - Parameter description: This string will be displayed in the test log
+    ///   to help diagnose failures.
+    /// - Parameter file: The file name to use in the error message if
+    ///   this expectation is not waited for. Default is the file
+    ///   containing the call to this method. It is rare to provide this
+    ///   parameter when calling this method.
+    /// - Parameter line: The line number to use in the error message if the
+    ///   this expectation is not waited for. Default is the line
+    ///   number of the call to this method in the calling file. It is rare to
+    ///   provide this parameter when calling this method.
+    ///
+    /// - Note: Whereas Objective-C XCTest determines the file and line
+    ///   number of expectations that are created by using symbolication, this
+    ///   implementation opts to take `file` and `line` as parameters instead.
+    ///   As a result, the interface to these methods are not exactly identical
+    ///   between these environments.
+    public func expectationWithDescription(description: String, file: StaticString = __FILE__, line: UInt = __LINE__) -> XCTestExpectation {
+        let expectation = XCTestExpectation(description: description)
+        XCTAllExpectations.append(expectation)
+        XCTLatestExpectationLocation = (file: file, line: line)
+        return expectation
+    }
+
+    /// Creates a point of synchronization in the flow of a test. Only one
+    /// "wait" can be active at any given time, but multiple discrete sequences
+    /// of { expectations -> wait } can be chained together.
+    ///
+    /// - Parameter timeout: The amount of time within which all expectation
+    ///   must be fulfilled.
+    /// - Parameter file: The file name to use in the error message if
+    ///   expectations are not met before the given timeout. Default is the file
+    ///   containing the call to this method. It is rare to provide this
+    ///   parameter when calling this method.
+    /// - Parameter line: The line number to use in the error message if the
+    ///   expectations are not met before the given timeout. Default is the line
+    ///   number of the call to this method in the calling file. It is rare to
+    ///   provide this parameter when calling this method.
+    /// - Parameter handler: If provided, the handler will be invoked both on
+    ///   timeout or fulfillment of all expectations. Timeout is always treated
+    ///   as a test failure.
+    ///
+    /// - Note: Whereas Objective-C XCTest determines the file and line
+    ///   number of the "wait" call using symbolication, this implementation
+    ///   opts to take `file` and `line` as parameters instead. As a result,
+    ///   the interface to these methods are not exactly identical between
+    ///   these environments.
+    public func waitForExpectationsWithTimeout(timeout: Double, file: StaticString = __FILE__, line: UInt = __LINE__, handler: XCWaitCompletionHandler?) {
+        // Mirror Objective-C XCTest behavior; display a test failure when
+        // users wait without having first set expectations.
+        if XCTAllExpectations.count == 0 {
+            let failure = XCTFailure(
+                message: "call made to wait without any expectations having been set",
+                failureDescription: "API violation",
+                expected: false,
+                file: file,
+                line: line)
+            XCTAllExpectationFailures.append(failure)
+            return
+        }
+
+        // Objective-C XCTest outputs the descriptions of every unfulfilled
+        // expectation. We gather them into this array, which is also used
+        // to determine failure--a non-empty array meets expectations weren't
+        // met.
+        var unfulfilledDescriptions = [String]()
+
+        // FIXME: Ideally we'd poll the expectations frequently and stop
+        //        blocking as soon as they've been met. For the time being,
+        //        however, we wait the full timeout period every time.
+        let semaphore = dispatch_semaphore_create(0)
+        let when = dispatch_time(
+            DISPATCH_TIME_NOW,
+            Int64(timeout * Double(NSEC_PER_SEC)))
+        dispatch_after(when, dispatch_get_global_queue(0, 0)) {
+            for expectation in XCTAllExpectations {
+                if !expectation.fulfilled {
+                    unfulfilledDescriptions.append(expectation.description)
+                }
+            }
+
+            if unfulfilledDescriptions.count > 0 {
+                // Not all expectations were fulfilled. Append a failure
+                // to the array of expectation-based failures.
+                let descriptions = unfulfilledDescriptions.joinWithSeparator(", ")
+                let failure = XCTFailure(
+                    message: "Exceeded timeout of \(timeout) seconds, with unfulfilled expectations: \(descriptions)",
+                    failureDescription: "Asynchronous wait failed",
+                    expected: true,
+                    file: file,
+                    line: line)
+                // FIXME: This should be an instance variable on the test case,
+                //        not a global.
+                XCTAllExpectationFailures.append(failure)
+            }
+
+            // We've recorded all the failures; clear the expectations that
+            // were set for this test case, in order to prepare the next test
+            // case to be run.
+            // FIXME: This should be an instance variable on the test case.
+            //        Once this is the case, there is no longer any need
+            //        to reset it to an empty array here.
+            XCTAllExpectations = []
+            dispatch_semaphore_signal(semaphore)
+        }
+        dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER)
+
+        // The handler is invoked regardless of whether the test passed.
+        if let completionHandler = handler {
+            var error: XCTError? = nil
+            if unfulfilledDescriptions.count > 0 {
+                // If the test failed, send an error object.
+                error = XCTError(
+                    domain: "org.swift.XCTestErrorDomain",
+                    code: 0,
+                    userInfo: [:])
+            }
+            completionHandler(error)
+        }
+    }
 }
+
+// FIXME: These should be instance variables on XCTestCase;
+//        see: https://github.com/apple/swift-corelibs-xctest/pull/40
+private var XCTLatestExpectationLocation: (file: StaticString, line: UInt) = ("", 0)
+private var XCTAllExpectations = [XCTestExpectation]()
+private var XCTAllExpectationFailures = [XCTFailure]()

Sources/XCTest/XCTestExpectation.swift

@@ -0,0 +1,29 @@
+// This source file is part of the Swift.org open source project
+//
+// Copyright (c) 2014 - 2016 Apple Inc. and the Swift project authors
+// Licensed under Apache License v2.0 with Runtime Library Exception
+//
+// See http://swift.org/LICENSE.txt for license information
+// See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
+//
+//
+//  XCTestExpectation.swift
+//  Expectations represent specific conditions in asynchronous testing.
+//
+
+/// Expectations represent specific conditions in asynchronous testing.
+public class XCTestExpectation {
+    internal var fulfilled = false
+    internal let description: String
+
+    internal init(description: String) {
+        self.description = description
+    }
+
+    /// Marks an expectation as having been met. It's an error to call this
+    /// method on an expectation that has already been fulfilled, or when the
+    /// test case that vended the expectation has already completed.
+    public func fulfill() {
+        fulfilled = true
+    }
+}

Sources/XCTest/XCWaitCompletionHandler.swift

@@ -0,0 +1,59 @@
+// This source file is part of the Swift.org open source project
+//
+// Copyright (c) 2014 - 2016 Apple Inc. and the Swift project authors
+// Licensed under Apache License v2.0 with Runtime Library Exception
+//
+// See http://swift.org/LICENSE.txt for license information
+// See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
+//
+//
+//  XCWaitCompletionHandler.swift
+//  A block to be invoked when a call to wait times out or has had all
+// associated expectations fulfilled.
+//
+
+#if os(Linux) || os(FreeBSD)
+    /// On platforms where NSError is not available, this class steps in as a
+    /// similar, but not identical, interface.
+    public class XCTError {
+        /// The domain and code define the error. Domains are described by names
+        /// that are arbitrary strings used to differentiate groups of codes;
+        /// for custom domain using reverse-DNS naming will help avoid
+        /// conflicts. Codes are domain-specific.
+        public let domain: String
+        public let code: Int
+
+        /// Additional info which may be used to describe the error further.
+        public let userInfo: [String: String]
+
+        // An internal initializer that mirrors the Objective-C Foundation
+        // API. This and the typealias below allow us to instantiate an
+        // XCTError easily on any platform.
+        internal init(domain: String, code: Int, userInfo: [String: String]) {
+            self.domain = domain
+            self.code = code
+            self.userInfo = userInfo
+        }
+    }
+
+    /// A block to be invoked when a call to wait times out or has had all
+    /// associated expectations fulfilled.
+    ///
+    /// - Parameter error: If the wait timed out or a failure was raised while
+    ///   waiting, the error's code will specify the type of failure. Otherwise
+    ///   error will be nil.
+    public typealias XCWaitCompletionHandler = (XCTError?) -> ()
+#else
+    import Foundation
+
+    /// A block to be invoked when a call to wait times out or has had all
+    /// associated expectations fulfilled.
+    ///
+    /// - Parameter error: If the wait timed out or a failure was raised while
+    ///   waiting, the error's code will specify the type of failure. Otherwise
+    ///   error will be nil.
+    public typealias XCWaitCompletionHandler = (NSError?) -> ()
+
+    // An alias that allows us to reference an XCTError easily on any platform.
+    internal typealias XCTError = NSError
+#endif