Allow differing leading/trailing whitespace when checking test output

[briancroom]

There are two situations lately where I've been finding myself fighting the testing framework's current strictness with respect to whitespace at the beginning and end of lines:

1. Here, I had to include some awkward whitespace after the // CHECK-ALL: prefix, in order to also be able to keep things nicely lined up on lines like this.
2. Whenever dealing with lines like this one, Xcode likes to trim the trailing space, which results in a test failure that is very difficult to understand because the difference between the lines is effectively invisible.

As I see it, we don't have any tests right now for which precise matching of leading/trailing whitespace is important, and I don't foresee any such tests coming up either, so it seems that this change will make the test checker a little easier to work with. On the other hand if there are concerns about this, we could also add a flag for turning this behavior off and on.

[modocache]

I agree with your first point! I also think that FileCheck does the same thing (and we should try to be consistent as much as possible). It'd be nice to confirm this.

My two cents on (2): personally I think the better solution is to put a $ at the end of the line. I also think printing a superfluous space at the end of assertion failures is something we could think about addressing--and it's something I noticed due to the fact that we match trailing whitespace. I'll defer to @mike-ferris-apple on this point, though.

[mike-ferris]

If Xcode's XCTest is outputting extra space at the end of some lines, I wouldn't mind seeing that addressed. Does it? Any specifics as to when it does this would be appreciated, and I can file a radar for the behavior.

I think this particular change would be fine. Especially if it has precedent in FileCheck.

[briancroom]

Any specifics as to when it does this would be appreciated, and I can file a radar for the behavior.

The following test case:

import XCTest
class TestBar: XCTestCase {
    func testBar() {
        XCTAssert(false)
    }
}

produces the following (partial) output:

Test Case '-[XCTestSimpleTests.TestBar testBar]' started.
/Users/pivotal/workspace/Experiments/XCTestSimple/XCTestSimpleTests/XCTestSimpleTests.swift:15: error: -[XCTestSimpleTests.TestBar testBar] : XCTAssertTrue failed - 
Test Case '-[XCTestSimpleTests.TestBar testBar]' failed (0.005 seconds).

Note that the line indicating the failure does contain a trailing space after the hyphen. swift-corelibs-xctest also reproduces this behavior currently.

[briancroom]

Regarding FileCheck's behavior, this bit of documentation seems to confirm what @modocache indicated. (Note that our default matching style appears to be basically equivalent to FileCheck with the --match-full-lines flag)

[briancroom]

Besides rebasing, I've also taken advantage of a comment on the aforementioned FileCheck docs and updated this behavior such that it now fully respects ^ or $ characters if they are included in the check line, allowing explicit checking of leading/trailing whitespace if desired, but ignores such whitespace by default.

I've also updated a few tests to take advantage of the enhanced checker. Note that there are still a couple of tests like this one which explicit check for a trailing space. These will serve us well when we want to get rid of that trailing space (in a separate PR!)

[modocache]

Will the spaces here only match spaces, or will they match tabs as well? Perhaps use \s?

[briancroom]

Yeah, that's a good point. \s is probably more appropriate here considering the way that I've been referencing whitespace generally in the docs for this PR.

OTOH, considering that it's not actually necessary at this moment, and CI has gone green already with the existing code (on Linux at least), how about we make that adjustment separately?

[modocache]

Works for me! 👍

[modocache]

Wow!! Can't argue with that. Sorry to make you jump through so many hoops. I think this is a great solution. Thanks, @briancroom! 💯

Personally I'd prefer to keep a trailing $ in a few places, until we file a radar for the Apple XCTest extra space and get that fixed. But certainly not all tests need them.

@mike-ferris-apple could you kick off the tests so we can merge this?

[mike-ferris]

@swift-ci please test

[briancroom]

Sorry to make you jump through so many hoops.

Not at all! This is exactly the sort of discussion I was hoping would come out of this PR. We all learned a thing or two, got some clarity on a (minor!) bug in XCTest.framework, and ended up with a much more robust patch than my initial naive attempt. Seems like a job well done all around. 🙇