[SR-1541] Listing test methods

[briancroom]

Add a command line flag for printing a list of the tests in the suite. This is triggered by passing --list-tests or -l to an executable that calls XCTMain.

I'm looking for particular feedback on:

• Is this kind of work good to merge right now? It is certainly a useful, purely additive feature, however it does not directly support the project's Swift 3 goals, nor has it seen discussion on the corelibs-dev or evolution lists.
• How is the output format? It currently looks like: (edit: This has changed, see comments below)

FirstTestCase
  FirstTestCase.test_foo
  FirstTestCase.test_bar
SecondTestCase
  SecondTestCase.test_someMore
  SecondTestCase.test_allTheThings

cc @mike-ferris-apple @ddunbar @aciidb0mb3r

[briancroom]

@swift-ci please test

[aciidgh]

YAYY! awesome.
is there a way to get the module name in the output?
I think the output should match the syntax used for running selected tests i.e. TestModuleName.TestCase/TestMethod

[ddunbar]

I think this is useful functionality (and would like to see a JSON form), but I don't think we should do this without first getting buy in and an implementation to get a compatible experience on OS X. Otherwise we cannot effectively use it as scaffolding for SwiftPM features, for example.

[briancroom]

The latest commit on the branch now print JSON output following the format of the OS X code sample on SR-1541:

{ "testSpecifiers": [
  "ListTests.FirstTestCase/test_foo",
  "ListTests.FirstTestCase/test_bar",
  "ListTests.SecondTestCase/test_someMore",
  "ListTests.SecondTestCase/test_allTheThings"
] }

The previous commit on the branch has the original format, but fixed to include the module name as is expected when filtering to run a single test case or class. Any thoughts on whether it would make sense to support both output formats?

[briancroom]

@swift-ci please test

[ddunbar]

I wonder if we should break down the schema to be more explicit, in that the module name, test suite name, and test name are not mangled together, at least in the schema.

/cc @aciidb0mb3r @modocache @mike-ferris-apple

[ddunbar]

I also suspect we should use a separate verb for list as JSON, it isn't unreasonable for it to have a human readable output mode as a standalone tool (this particularly makes sense in the context of OS X's xctest)

[aciidgh]

I think it makes sense to have better schema when outputting JSON. Maybe something like this?

{
  "module1": {
    "testcase1": [
      "test1",
      "test2"
    ],
    "testcase2": [
      "test1",
      "test2"
    ]
  },
  "module2": {
    "testcase1": [
      "test1",
      "test2"
    ],
    "testcase2": [
      "test1",
      "test2"
    ]
  }
}

[modocache]

Thanks for pursuing this, @briancroom!

I think this is useful functionality (and would like to see a JSON form), but I don't think we should do this without first getting buy in and an implementation to get a compatible experience on OS X.

I would love it if Apple XCTest supported listing tests. Depending on how it was implemented, it might even address my concerns from rdar://26152293.

[modocache]

It would be really cool if, in a future pull request, we can expand the argument parsing here to include -h and --help. SwiftPM also implements its own argument parsing--perhaps we can extract this out into a library of code shared between SwiftPM and corelibs-xctest?

[ddunbar]

I do hope to factor out the argument parser more, although for practical reasons it may end up being intertwined with our own internal infrastructure. The ideal would be for the stdlib to help... :)

[briancroom]

Indeed! Once this goes in I was intending to open a Starter Issue for getting help/usage added.

[briancroom]

I'm definitely in favor of making the JSON output more structured. I'll revisit the output formatting again when I get the chance, but that probably won't be until the middle of next week since I'll be away from my dev machine for the long weekend up here. Any more input on how to make the output useful for consumers would be great in the meantime!

Thanks for the interest and feedback, everyone 😄

[ddunbar]

I believe we have converged on a schema to use for this in:
swiftlang/swift-package-manager#363

Are we all agreed?

[briancroom]

Hot on the heels of swiftlang/swift-package-manager#363 (comment), I have finished updating this branch to include the updated JSON format. The following usages have been added:

$ ./ListTests --list-tests
Listing 4 tests in tmp9Imn34.xctest:

ListTests.FirstTestCase/test_foo
ListTests.FirstTestCase/test_bar
ListTests.SecondTestCase/test_someMore
ListTests.SecondTestCase/test_allTheThings

$ ./ListTests --dump-test-json
{"tests":[{"tests":[{"tests":[{"name":"test_foo"},{"name":"test_bar"}],"name":"ListTests.FirstTestCase"},{"tests":[{"name":"test_someMore"},{"name":"test_allTheThings"}],"name":"ListTests.SecondTestCase"}],"name":"tmp9Imn34.xctest"}],"name":"All tests"}

I'm very much open to a different flag name for the JSON case in particular, if anyone has preferences. Regardless, as @modocache pointed out, we will need to introduce usage instructions on top of this now that more modes are being added.

[briancroom]

@swift-ci please test

[briancroom]

@ddunbar @ankit @modocache Any feedback here? Should we merge this in?

[aciidgh]

looks awesome to me 🤓

[modocache]

Looks good! I'm interpreting @ddunbar's enthusiasm for this direction to indicate the tacit approval of the Apple XCTest team 😁

[aciidgh]

YAYY!
Once swiftlang/swift-package-manager#363 is merged a list test mode can be added to swift test \o/