[SR-7615] Implement XCTWaiter and missing XCTestExpectation APIs

[stmontgomery]

This brings Corelibs XCTest up-to-date with ObjC XCTest's modern asynchronous waiting APIs.

This has been requested via Swift JIRA (SR-7615, SR-6249). It is effectively a rewrite of this project's asynchronous testing logic to match the behavior of Apple's ObjC framework.

• Implement missing XCTestExpectation APIs including:
  • expectedFulfillmentCount
  • isInverted
  • assertForOverFulfill
• Implement XCTWaiter class and XCTWaiterDelegate protocol
• Publicly expose the XCTNS(Predicate,Notification)Expectation classes
• Re-implement XCTestCase.waitForExpectations(timeout:handler:) to use XCTWaiter and add XCTestCase.wait(for:timeout:)
• Implement "nested waiter" behavior, using a new internal helper class WaiterManager
• Add documentation comments to all public APIs, based on their ObjC counterparts
• Add a utility struct called SourceLocation which represents a file & line
• Update expectation handler block typealiases to match current ObjC names
• Match various API call syntaxes to their ObjC counterparts
• Introduce a subsystemQueue DispatchQueue for synchronization, shared between XCTWaiter and all expectations
• Modify XCTestCase's "un-waited" expectations failure message to note all un-waited on expectations, to match ObjC
• Add many new tests

[stmontgomery]

@swift-ci Please test

[stmontgomery]

@swift-ci Please test

[stmontgomery]

@swift-ci Please test

[briancroom]

@swift-ci please test

[stmontgomery]

Hi @modocache, I believe you've worked extensively on the code being touched here. If you have any time to take a look at these changes and give feedback, I'd appreciate it! Thanks

[briancroom]

This is looking good to me. Thanks a lot for building this out!

[modocache]

This looks great, thanks for implementing this!

I had some questions about the behavior compared to Apple XCTest.

[modocache]

I don't know if we care about the behavior being identical in this regard, but I'll just point out that the precondition used in this pull request would crash the test process, whereas Apple XCTest raises an exception that causes the test with the API violation to fail.

[stmontgomery]

Yeah, thanks, I realize that will be a behavioral difference. And note that XCTestExpectation.assertForOverFulfill is similar in that it causes a fatalError, although I opted to always have it be disabled by default unlike Apple XCTest where it is enabled by default for XCTest-generated instances and disabled otherwise.

There just is not a way I am aware of to fully mimic the behavior of ObjC's exceptions here.

[briancroom]

Ultimately, I suspect we might want to develop the harness to a point that it spawns worker subprocesses to run the actual test code, thus providing a way for test execution to continue and results to not get lost after a failure like this. (Xcode plays a similar role today when running XCTest bundles in that it is able to restart the runner after a crash.)

[modocache]

These are great tests! Thanks for writing them.

I remember Apple XCTest had some interesting behavior where the first of these tests fails with "Failed due to expectations fulfilled in incorrect order", but the second passes:

- (void)test_encforcingTheOrderOfInvertedExpectations_inTheMiddleOfAList_createsAnImpossibleAssertion {
    XCTestExpectation *expectationOne = [self expectationWithDescription:@"one"];
    [expectationOne fulfill];

    XCTestExpectation *invertedExpectationTwo = [self expectationWithDescription:@"inverted two"];
    invertedExpectationTwo.inverted = YES;

    XCTestExpectation *expectationThree = [self expectationWithDescription:@"three"];
    [expectationThree fulfill]; // Fails here: "Failed due to expectation fulfilled in incorrect order"

    [self waitForExpectations:@[expectationOne, invertedExpectationTwo, expectationThree]
    	              timeout:0.1
    	         enforceOrder:YES];
}

- (void)test_encforcingTheOrderOfInvertedExpectations_atTheEndOfAList_passes {
    XCTestExpectation *expectationOne = [self expectationWithDescription:@"one"];
    [expectationOne fulfill];

    XCTestExpectation *expectationTwo = [self expectationWithDescription:@"two"];
    [expectationTwo fulfill];

    XCTestExpectation *invertedExpectationThree = [self expectationWithDescription:@"inverted three"];
    invertedExpectationThree.inverted = YES;

    [self waitForExpectations:@[expectationOne, expectationTwo, invertedExpectationThree]
    	              timeout:0.1
    	         enforceOrder:YES];
}

I'd be curious if Apple XCTest still behaves this way, and if so, maybe we should add a test that swift-corelibs-xctest does the same?

[stmontgomery]

Thanks @modocache!

I found the bug you're referring to (rdar://problem/31657785) and it was fixed in Xcode 9. I confirmed that both tests you pasted above now pass in recent versions of Xcode (using Apple XCTest). And that issue is now covered by the unit test I added in this PR named test_combiningInverseAndStandardExpectationsWithOrderingEnforcement.

[modocache]

Awesome! Thanks for checking that.