[XCTestCase] Add asynchronous testing API

Mirror the Objective-C XCTest API to add asynchronous testing support,
primarily via two methods:

- `XCTestCase.expectationWithDescription()`
- `XCTestCase.waitForExpectationsWithTimeout()`

These methods' APIs are identical to their Objective-C SDK overlay
counterparts, with the following exceptions:

1. If expectations are created but not waited for, Objective-C XCTest
   generates a test failure. In order to display a failure on the exact
   line of the last expectation that was created, Objective-C XCTest
   symbolicates the call stack. It does so using a private framework
   called `CoreSymbolication.framework`. We don't have this at our
   disposal for swift-corelibs-xctest, so instead we take `file` and
   `line` as parameters, defaulting to `__FILE__` and `__LINE__`. By
   doing so, we provide the same failure highlighting functionality
   while also maintaining a (nearly) identical API.
2. If `waitForExpectationsWithTimeout()` fails, Objective-C XCTest
   generates a test failure on the exact line that method was called,
   using the same technique from (1). For the same reason, this method
   also takes `file` and `line` parameters in swift-corelibs-xctest.
3. `waitForExpectationsWithTimeout()` takes a handler callback, which
   in Objective-C XCTest is passed an instance of `NSError`.
   swift-corelibs-xctest doesn't depend upon swift-corelibs-foundation,
   and so does not have access to `NSError`. Instead, it defines a shim,
   `XCTError`.

In order to implement the asynchronous APIs, swift-corelibs-xctest now
has a dependency upon swift-corelibs-foundation. Bump minimum deployment
targets for XCTest to the same required by Foundation.

Author: modocache

README.md

@@ -24,37 +24,58 @@ The rest of this document will focus on how this version of XCTest differs from
 
 ## Working on XCTest
 
+### On Linux
+
 XCTest can be built as part of the overall Swift package. When following [the instructions for building Swift](http://www.github.com/apple/swift), pass the `--xctest` option to the build script:
 
 ```sh
 swift/utils/build-script --xctest
 ```
 
-If you want to build just XCTest, use the `build_script.py` script at the root of the project. The `master` version of XCTest must be built with the `master` version of Swift.
+If you want to build just XCTest, use the `build_script.py` script at the root of the project. The `master` version of XCTest must be built with the `master` version of Swift. XCTest has a dependency upon Foundation, so you must have built the `master` version of that as well.
 
 If your install of Swift is located at `/swift` and you wish to install XCTest into that same location, here is a sample invocation of the build script:
 
 ```sh
 ./build_script.py \
     --swiftc="/swift/usr/bin/swiftc" \
     --build-dir="/tmp/XCTest_build" \
+    --foundation-build-dir "/swift//usr/lib/swift/linux" \
     --library-install-path="/swift/usr/lib/swift/linux" \
     --module-install-path="/swift/usr/lib/swift/linux/x86_64"
 ```
 
 To run the tests on Linux, use the `--test` option:
 
 ```sh
-./build_script.py --swiftc="/swift/usr/bin/swiftc" --test
+./build_script.py \
+    --swiftc="/swift/usr/bin/swiftc" \
+    --foundation-build-dir "/swift/usr/lib/swift/linux" \
+    --test
+```
+
+You may add tests for XCTest by including them in the `Tests/Functional/` directory. For an example, see `Tests/Functional/SingleFailingTestCase`.
+
+### On OS X
+
+You may build XCTest via the "SwiftXCTest" scheme in `XCTest.xcworkspace`. The workspace assumes that Foundation and XCTest are checked out from GitHub in sibling directories. For example:
+
 ```
+% cd Development
+% ls
+swift-corelibs-foundation swift-corelibs-xctest
+%
+```
+
+Unlike on Linux, you do not need to build Foundation prior to building XCTest. The "SwiftXCTest" Xcode scheme takes care of that for you.
 
 To run the tests on OS X, build and run the `SwiftXCTestFunctionalTests` target in the Xcode workspace. You may also run them via the command line:
 
 ```
 xcodebuild -workspace XCTest.xcworkspace -scheme SwiftXCTestFunctionalTests
 ```
 
-You may add tests for XCTest by including them in the `Tests/Functional/` directory. For an example, see `Tests/Functional/SingleFailingTestCase`.
+When adding tests to the `Tests/Functional` directory, make sure they can be opened in the `XCTest.xcworkspace` by adding references to them, but do not add them to any of the targets.
 
 ### Additional Considerations for Swift on Linux
 

Sources/XCTest/XCTestCase.swift

@@ -11,6 +11,12 @@
 //  Base class for test cases
 //
 
+#if os(Linux) || os(FreeBSD)
+    import Foundation
+#else
+    import SwiftFoundation
+#endif
+
 /// This is a compound type used by `XCTMain` to represent tests to run. It combines an
 /// `XCTestCase` subclass type with the list of test methods to invoke on the test case.
 /// This type is intended to be produced by the `testCase` helper function.
@@ -48,6 +54,14 @@ private func test<T: XCTestCase>(testFunc: T -> () throws -> Void) -> XCTestCase
     }
 }
 
+// FIXME: These should be instance variables defined on XCTestCase, but when so
+//        defined Linux tests fail with "hidden symbol isn't defined". Use
+//        globals for the time being, as these seem to appease the Linux
+//        compiler.
+private var XCTLatestExpectationLocation: (file: StaticString, line: UInt) = ("", 0)
+private var XCTAllExpectations = [XCTestExpectation]()
+private var XCTAllExpectationFailures = [XCTFailure]()
+
 extension XCTestCase {
 
     public var continueAfterFailure: Bool {
@@ -93,6 +107,24 @@ extension XCTestCase {
 
                 testCase.tearDown()
 
+                // It is an API violation to create expectations but not wait
+                // for them to be completed. Notify the user of a mistake via
+                // a test failure.
+                if XCTAllExpectations.count > 0 {
+                    let failure = XCTFailure(
+                        message: "Failed due to unwaited expectations.",
+                        failureDescription: "",
+                        expected: false,
+                        file: XCTLatestExpectationLocation.file,
+                        line: XCTLatestExpectationLocation.line)
+                    XCTAllExpectationFailures.append(failure)
+                }
+
+                failures += XCTAllExpectationFailures
+                XCTLatestExpectationLocation = (file: "", line: 0)
+                XCTAllExpectations = []
+                XCTAllExpectationFailures = []
+
                 totalDuration += duration
 
                 var result = "passed"
@@ -122,4 +154,155 @@ extension XCTestCase {
 
         XCTPrint("Executed \(tests.count) test\(testCountSuffix), with \(totalFailures) failure\(failureSuffix) (\(unexpectedFailures) unexpected) in \(printableStringForTimeInterval(totalDuration)) (\(printableStringForTimeInterval(overallDuration))) seconds")
     }
+
+    internal func expectationWasFulfilledInDuplicate(expectation: XCTestExpectation, file: StaticString, line: UInt) {
+        // Mirror Objective-C XCTest behavior: treat multiple calls to
+        // fulfill() as an unexpected failure.
+        // FIXME: Objective-C XCTest raises an exception for most "API
+        //        violation" failures, including this one. Normally this causes
+        //        the test to stop cold. swift-corelibs-xctest does not stop,
+        //        and executes the rest of the test. This discrepancy should be
+        //        fixed.
+        let failure = XCTFailure(
+            message: "multiple calls made to XCTestExpectation.fulfill() for \(expectation.description).",
+            failureDescription: "API violation",
+            expected: false,
+            file: file,
+            line: line)
+        XCTAllExpectationFailures.append(failure)
+    }
+
+    /// Creates and returns an expectation associated with the test case.
+    ///
+    /// - Parameter description: This string will be displayed in the test log
+    ///   to help diagnose failures.
+    /// - Parameter file: The file name to use in the error message if
+    ///   this expectation is not waited for. Default is the file
+    ///   containing the call to this method. It is rare to provide this
+    ///   parameter when calling this method.
+    /// - Parameter line: The line number to use in the error message if the
+    ///   this expectation is not waited for. Default is the line
+    ///   number of the call to this method in the calling file. It is rare to
+    ///   provide this parameter when calling this method.
+    ///
+    /// - Note: Whereas Objective-C XCTest determines the file and line
+    ///   number of expectations that are created by using symbolication, this
+    ///   implementation opts to take `file` and `line` as parameters instead.
+    ///   As a result, the interface to these methods are not exactly identical
+    ///   between these environments. To ensure compatibility of tests between
+    ///   swift-corelibs-xctest and Apple XCTest, it is not recommended to pass
+    ///   explicit values for `file` and `line`.
+    public func expectationWithDescription(description: String, file: StaticString = #file, line: UInt = #line) -> XCTestExpectation {
+        let expectation = XCTestExpectation(description: description, testCase: self)
+        XCTAllExpectations.append(expectation)
+        XCTLatestExpectationLocation = (file: file, line: line)
+        return expectation
+    }
+
+    /// Creates a point of synchronization in the flow of a test. Only one
+    /// "wait" can be active at any given time, but multiple discrete sequences
+    /// of { expectations -> wait } can be chained together.
+    ///
+    /// - Parameter timeout: The amount of time within which all expectation
+    ///   must be fulfilled.
+    /// - Parameter file: The file name to use in the error message if
+    ///   expectations are not met before the given timeout. Default is the file
+    ///   containing the call to this method. It is rare to provide this
+    ///   parameter when calling this method.
+    /// - Parameter line: The line number to use in the error message if the
+    ///   expectations are not met before the given timeout. Default is the line
+    ///   number of the call to this method in the calling file. It is rare to
+    ///   provide this parameter when calling this method.
+    /// - Parameter handler: If provided, the handler will be invoked both on
+    ///   timeout or fulfillment of all expectations. Timeout is always treated
+    ///   as a test failure.
+    ///
+    /// - Note: Whereas Objective-C XCTest determines the file and line
+    ///   number of the "wait" call using symbolication, this implementation
+    ///   opts to take `file` and `line` as parameters instead. As a result,
+    ///   the interface to these methods are not exactly identical between
+    ///   these environments. To ensure compatibility of tests between
+    ///   swift-corelibs-xctest and Apple XCTest, it is not recommended to pass
+    ///   explicit values for `file` and `line`.
+    public func waitForExpectationsWithTimeout(timeout: Double, file: StaticString = #file, line: UInt = #line, handler: XCWaitCompletionHandler?) {
+        // Mirror Objective-C XCTest behavior; display an unexpected test
+        // failure when users wait without having first set expectations.
+        // FIXME: Objective-C XCTest raises an exception for most "API
+        //        violation" failures, including this one. Normally this causes
+        //        the test to stop cold. swift-corelibs-xctest does not stop,
+        //        and executes the rest of the test. This discrepancy should be
+        //        fixed.
+        if XCTAllExpectations.count == 0 {
+            let failure = XCTFailure(
+                message: "call made to wait without any expectations having been set.",
+                failureDescription: "API violation",
+                expected: false,
+                file: file,
+                line: line)
+            XCTAllExpectationFailures.append(failure)
+            return
+        }
+
+        // Objective-C XCTest outputs the descriptions of every unfulfilled
+        // expectation. We gather them into this array, which is also used
+        // to determine failure--a non-empty array meets expectations weren't
+        // met.
+        var unfulfilledDescriptions = [String]()
+
+        // We continue checking whether expectations have been fulfilled until
+        // the specified timeout has been reached.
+        // FIXME: Instead of polling the expectations to check whether they've
+        //        been fulfilled, it would be more efficient to use a runloop
+        //        source that can be signaled to wake up when an expectation is
+        //        fulfilled.
+        let runLoop = NSRunLoop.currentRunLoop()
+        let timeoutDate = NSDate(timeIntervalSinceNow: timeout)
+        repeat {
+            unfulfilledDescriptions = []
+            for expectation in XCTAllExpectations {
+                if !expectation.fulfilled {
+                    unfulfilledDescriptions.append(expectation.description)
+                }
+            }
+
+            // If we've met all expectations, then break out of the specified
+            // timeout loop early.
+            if unfulfilledDescriptions.count == 0 {
+                break
+            }
+
+            // Otherwise, wait another fraction of a second.
+            runLoop.runUntilDate(NSDate(timeIntervalSinceNow: 0.01))
+        } while NSDate().compare(timeoutDate) == NSComparisonResult.OrderedAscending
+
+        if unfulfilledDescriptions.count > 0 {
+            // Not all expectations were fulfilled. Append a failure
+            // to the array of expectation-based failures.
+            let descriptions = unfulfilledDescriptions.joinWithSeparator(", ")
+            let failure = XCTFailure(
+                message: "Exceeded timeout of \(timeout) seconds, with unfulfilled expectations: \(descriptions)",
+                failureDescription: "Asynchronous wait failed",
+                expected: true,
+                file: file,
+                line: line)
+            XCTAllExpectationFailures.append(failure)
+        }
+
+        // We've recorded all the failures; clear the expectations that
+        // were set for this test case.
+        XCTAllExpectations = []
+
+        // The handler is invoked regardless of whether the test passed.
+        if let completionHandler = handler {
+            var error: NSError? = nil
+            if unfulfilledDescriptions.count > 0 {
+                // If the test failed, send an error object.
+                error = NSError(
+                    domain: "org.swift.XCTestErrorDomain",
+                    code: 0,
+                    userInfo: [:])
+            }
+            completionHandler(error)
+        }
+    }
 }

Sources/XCTest/XCTestExpectation.swift

@@ -0,0 +1,59 @@
+// This source file is part of the Swift.org open source project
+//
+// Copyright (c) 2014 - 2016 Apple Inc. and the Swift project authors
+// Licensed under Apache License v2.0 with Runtime Library Exception
+//
+// See http://swift.org/LICENSE.txt for license information
+// See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
+//
+//
+//  XCTestExpectation.swift
+//  Expectations represent specific conditions in asynchronous testing.
+//
+
+/// Expectations represent specific conditions in asynchronous testing.
+public class XCTestExpectation {
+    internal let description: String
+    internal weak var testCase: XCTestCase?
+
+    internal var fulfilled = false
+
+    internal init(description: String, testCase: XCTestCase) {
+        self.description = description
+        self.testCase = testCase
+    }
+
+    /// Marks an expectation as having been met. It's an error to call this
+    /// method on an expectation that has already been fulfilled, or when the
+    /// test case that vended the expectation has already completed.
+    ///
+    /// - Parameter file: The file name to use in the error message if
+    ///   expectations are not met before the given timeout. Default is the file
+    ///   containing the call to this method. It is rare to provide this
+    ///   parameter when calling this method.
+    /// - Parameter line: The line number to use in the error message if the
+    ///   expectations are not met before the given timeout. Default is the line
+    ///   number of the call to this method in the calling file. It is rare to
+    ///   provide this parameter when calling this method.
+    ///
+    /// - Note: Whereas Objective-C XCTest determines the file and line
+    ///   number the expectation was fulfilled using symbolication, this
+    ///   implementation opts to take `file` and `line` as parameters instead.
+    ///   As a result, the interface to these methods are not exactly identical
+    ///   between these environments. To ensure compatibility of tests between
+    ///   swift-corelibs-xctest and Apple XCTest, it is not recommended to pass
+    ///   explicit values for `file` and `line`.
+    public func fulfill(file: StaticString = #file, line: UInt = #line) {
+        if fulfilled {
+            if testCase == nil {
+                // FIXME: Objective-C XCTest emits failures when expectations
+                //        are fulfilled after the test cases that generated
+                //        those expectations have completed.
+            } else {
+                testCase!.expectationWasFulfilledInDuplicate(self, file: file, line: line)
+            }
+        } else {
+            fulfilled = true
+        }
+    }
+}

Sources/XCTest/XCWaitCompletionHandler.swift

@@ -0,0 +1,27 @@
+// This source file is part of the Swift.org open source project
+//
+// Copyright (c) 2014 - 2015 Apple Inc. and the Swift project authors
+// Licensed under Apache License v2.0 with Runtime Library Exception
+//
+// See http://swift.org/LICENSE.txt for license information
+// See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
+//
+//
+//  XCWaitCompletionHandler.swift
+//  A closure invoked by XCTestCase when a wait for expectations to be
+//  fulfilled times out.
+//
+
+#if os(Linux) || os(FreeBSD)
+    import Foundation
+#else
+    import SwiftFoundation
+#endif
+
+/// A block to be invoked when a call to wait times out or has had all
+/// associated expectations fulfilled.
+///
+/// - Parameter error: If the wait timed out or a failure was raised while
+///   waiting, the error's code will specify the type of failure. Otherwise
+///   error will be nil.
+public typealias XCWaitCompletionHandler = (NSError?) -> ()