Skip to content

Allow running XCTest bundle with multiple comma-separated tests #268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 12, 2019
Merged

Allow running XCTest bundle with multiple comma-separated tests #268

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,9 @@ The `XCTMain` function does not return, and will cause your test executable to e
* A particular test or test case can be selected to execute. For example:

```
$ ./FooTests Tests.FooTestCase/testFoo # Run a single test case
$ ./FooTests Tests.FooTestCase # Run all the tests in FooTestCase
$ ./FooTests Tests.FooTestCase/testFoo # Run a single test method
$ ./FooTests Tests.FooTestCase # Run all the tests in FooTestCase
$ ./FooTests Tests.FooTestCase/testFoo,Tests.FooTestCase/testBar # Run multiple test methods
```
* Tests can be listed, instead of executed.

Expand Down
18 changes: 9 additions & 9 deletions Sources/XCTest/Private/ArgumentParser.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@ internal struct ArgumentParser {

/// The basic operations that can be performed by an XCTest runner executable
enum ExecutionMode {
/// Run a test or test suite, printing results to stdout and exiting with
/// a non-0 return code if any tests failed. The name of a test or class
/// may be provided to only run a subset of test cases.
case run(selectedTestName: String?)
/// Run tests or test cases, printing results to stdout and exiting with
/// a non-0 return code if any tests failed. The names of tests or test cases
/// may be provided to only run a subset of them.
case run(selectedTestNames: [String]?)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we should prefer to use a Set for this for efficiency when filtering during execution?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will iterate over this array in TestFiltering.selectedTestFilter anyway for mapping into SelectedTests, but using a Set in the filter itself is indeed faster. Fixed.


/// The different ways that the tests can be represented when they are listed
enum ListType {
Expand All @@ -39,9 +39,9 @@ internal struct ArgumentParser {
/// Print Help
case help(invalidOption: String?)

var selectedTestName: String? {
if case .run(let name) = self {
return name
var selectedTestNames: [String]? {
if case .run(let names) = self {
return names
} else {
return nil
}
Expand All @@ -56,7 +56,7 @@ internal struct ArgumentParser {

var executionMode: ExecutionMode {
if arguments.count <= 1 {
return .run(selectedTestName: nil)
return .run(selectedTestNames: nil)
} else if arguments[1] == "--list-tests" || arguments[1] == "-l" {
return .list(type: .humanReadable)
} else if arguments[1] == "--dump-tests-json" {
Expand All @@ -66,7 +66,7 @@ internal struct ArgumentParser {
} else if let fst = arguments[1].first, fst == "-" {
return .help(invalidOption: arguments[1])
} else {
return .run(selectedTestName: arguments[1])
return .run(selectedTestNames: arguments[1].split(separator: ",").map(String.init))
}
}
}
25 changes: 12 additions & 13 deletions Sources/XCTest/Private/TestFiltering.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,20 @@
internal typealias TestFilter = (XCTestCase.Type, String) -> Bool

internal struct TestFiltering {
private let selectedTestName: String?
private let selectedTestNames: [String]?

init(selectedTestName: String?) {
self.selectedTestName = selectedTestName
init(selectedTestNames: [String]?) {
self.selectedTestNames = selectedTestNames
}

var selectedTestFilter: TestFilter {
guard let selectedTestName = selectedTestName else { return includeAllFilter() }
guard let selectedTest = SelectedTest(selectedTestName: selectedTestName) else { return excludeAllFilter() }
guard let selectedTestNames = selectedTestNames else { return includeAllFilter() }
let selectedTests = Set(selectedTestNames.compactMap { SelectedTest(selectedTestName: $0) })

return selectedTest.matches
}

private func excludeAllFilter() -> TestFilter {
return { _,_ in false }
return { testCaseClass, testCaseMethodName in
return selectedTests.contains(SelectedTest(testCaseClass: testCaseClass, testCaseMethodName: testCaseMethodName)) ||
selectedTests.contains(SelectedTest(testCaseClass: testCaseClass, testCaseMethodName: nil))
}
}

private func includeAllFilter() -> TestFilter {
Expand All @@ -47,7 +46,7 @@ internal struct TestFiltering {
}

/// A selected test can be a single test case, or an entire class of test cases
private struct SelectedTest {
private struct SelectedTest : Hashable {
let testCaseClassName: String
let testCaseMethodName: String?
}
Expand All @@ -67,7 +66,7 @@ private extension SelectedTest {
}
}

func matches(testCaseClass: XCTestCase.Type, testCaseMethodName: String) -> Bool {
return String(reflecting: testCaseClass) == testCaseClassName && (self.testCaseMethodName == nil || testCaseMethodName == self.testCaseMethodName)
init(testCaseClass: XCTestCase.Type, testCaseMethodName: String?) {
self.init(testCaseClassName: String(reflecting: testCaseClass), testCaseMethodName: testCaseMethodName)
}
}
6 changes: 3 additions & 3 deletions Sources/XCTest/Public/XCTestMain.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ public func XCTMain(_ testCases: [XCTestCaseEntry]) -> Never {
// - An `XCTestSuite` representing the .xctest test bundle is not included.
let rootTestSuite: XCTestSuite
let currentTestSuite: XCTestSuite
if executionMode.selectedTestName == nil {
if executionMode.selectedTestNames == nil {
rootTestSuite = XCTestSuite(name: "All tests")
currentTestSuite = XCTestSuite(name: "\(testBundle.bundleURL.lastPathComponent).xctest")
rootTestSuite.addTest(currentTestSuite)
Expand All @@ -75,7 +75,7 @@ public func XCTMain(_ testCases: [XCTestCaseEntry]) -> Never {
currentTestSuite = rootTestSuite
}

let filter = TestFiltering(selectedTestName: executionMode.selectedTestName)
let filter = TestFiltering(selectedTestNames: executionMode.selectedTestNames)
TestFiltering.filterTests(testCases, filter: filter.selectedTestFilter)
.map(XCTestCaseSuite.init)
.forEach(currentTestSuite.addTest)
Expand Down Expand Up @@ -118,7 +118,7 @@ public func XCTMain(_ testCases: [XCTestCaseEntry]) -> Never {
> \(exeName) \(sampleTests)
""")
exit(invalidOption == nil ? EXIT_SUCCESS : EXIT_FAILURE)
case .run(selectedTestName: _):
case .run(selectedTestNames: _):
// Add a test observer that prints test progress to stdout.
let observationCenter = XCTestObservationCenter.shared
observationCenter.addTestObserver(PrintObserver())
Expand Down
12 changes: 12 additions & 0 deletions Tests/Functional/SelectedTest/main.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
// RUN: %{swiftc} %s -o %T/SelectedTest
// RUN: %T/SelectedTest SelectedTest.ExecutedTestCase/test_foo > %T/one_test_case || true
// RUN: %T/SelectedTest SelectedTest.ExecutedTestCase > %T/one_test_case_class || true
// RUN: %T/SelectedTest SelectedTest.ExecutedTestCase/test_foo,SelectedTest.ExecutedTestCase/test_bar > %T/two_test_cases || true
// RUN: %T/SelectedTest > %T/all || true
// RUN: %{xctest_checker} -p "// CHECK-METHOD:" %T/one_test_case %s
// RUN: %{xctest_checker} -p "// CHECK-CLASS:" %T/one_test_case_class %s
// RUN: %{xctest_checker} -p "// CHECK-TWO-METHODS:" %T/two_test_cases %s
// RUN: %{xctest_checker} -p "// CHECK-ALL:" %T/all %s

#if os(macOS)
Expand All @@ -14,6 +16,7 @@

// CHECK-METHOD: Test Suite 'Selected tests' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-CLASS: Test Suite 'Selected tests' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-TWO-METHODS: Test Suite 'Selected tests' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-ALL: Test Suite 'All tests' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-ALL: Test Suite '.*\.xctest' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+

Expand All @@ -31,13 +34,18 @@ class ExecutedTestCase: XCTestCase {
// CHECK-CLASS: Test Suite 'ExecutedTestCase' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-CLASS: Test Case 'ExecutedTestCase.test_bar' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-CLASS: Test Case 'ExecutedTestCase.test_bar' passed \(\d+\.\d+ seconds\)
// CHECK-TWO-METHODS: Test Suite 'ExecutedTestCase' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-TWO-METHODS: Test Case 'ExecutedTestCase.test_bar' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-TWO-METHODS: Test Case 'ExecutedTestCase.test_bar' passed \(\d+\.\d+ seconds\)
// CHECK-ALL: Test Suite 'ExecutedTestCase' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-ALL: Test Case 'ExecutedTestCase.test_bar' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-ALL: Test Case 'ExecutedTestCase.test_bar' passed \(\d+\.\d+ seconds\)
func test_bar() {}

// CHECK-CLASS: Test Case 'ExecutedTestCase.test_foo' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-CLASS: Test Case 'ExecutedTestCase.test_foo' passed \(\d+\.\d+ seconds\)
// CHECK-TWO-METHODS: Test Case 'ExecutedTestCase.test_foo' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-TWO-METHODS: Test Case 'ExecutedTestCase.test_foo' passed \(\d+\.\d+ seconds\)
// CHECK-ALL: Test Case 'ExecutedTestCase.test_foo' started at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-ALL: Test Case 'ExecutedTestCase.test_foo' passed \(\d+\.\d+ seconds\)
func test_foo() {}
Expand All @@ -46,6 +54,8 @@ class ExecutedTestCase: XCTestCase {
// CHECK-METHOD: \t Executed 1 test, with 0 failures \(0 unexpected\) in \d+\.\d+ \(\d+\.\d+\) seconds
// CHECK-CLASS: Test Suite 'ExecutedTestCase' passed at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-CLASS: \t Executed 2 tests, with 0 failures \(0 unexpected\) in \d+\.\d+ \(\d+\.\d+\) seconds
// CHECK-TWO-METHODS: Test Suite 'ExecutedTestCase' passed at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-TWO-METHODS: \t Executed 2 tests, with 0 failures \(0 unexpected\) in \d+\.\d+ \(\d+\.\d+\) seconds
// CHECK-ALL: Test Suite 'ExecutedTestCase' passed at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-ALL: \t Executed 2 tests, with 0 failures \(0 unexpected\) in \d+\.\d+ \(\d+\.\d+\) seconds

Expand All @@ -72,6 +82,8 @@ XCTMain([
// CHECK-METHOD: \t Executed 1 test, with 0 failures \(0 unexpected\) in \d+\.\d+ \(\d+\.\d+\) seconds
// CHECK-CLASS: Test Suite 'Selected tests' passed at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-CLASS: \t Executed 2 tests, with 0 failures \(0 unexpected\) in \d+\.\d+ \(\d+\.\d+\) seconds
// CHECK-TWO-METHODS: Test Suite 'Selected tests' passed at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-TWO-METHODS: \t Executed 2 tests, with 0 failures \(0 unexpected\) in \d+\.\d+ \(\d+\.\d+\) seconds
// CHECK-ALL: Test Suite '.*\.xctest' passed at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
// CHECK-ALL: \t Executed 3 tests, with 0 failures \(0 unexpected\) in \d+\.\d+ \(\d+\.\d+\) seconds
// CHECK-ALL: Test Suite 'All tests' passed at \d+-\d+-\d+ \d+:\d+:\d+\.\d+
Expand Down