[SYCL-MLIR] Run all tests in Basic and XFAIL the failed test cases

[whitneywhtsang]

If this is good, then will extend to test cases in other folders.

Signed-off-by: Tsang, Whitney [email protected]

[sommerlukas]

I think it's a good idea to enable more of the tests in the test-suite, so we get an overview of potential regressions through CI.

However, I think there might be a more elegant solution to marking tests that we currently do not yet support as such:
LLVM's lit accepts an --xfail=LIST option, to which you can pass a list of tests that should be considered expected failures.
So instead of modifying 50+ files to add the XFAIL line to each of them, we could have a list of unsupported tests in single location (e.g., CMakeLists.txt) and pass that list to the xfail option of lit.

What do you think?

[etiotto]

I think it's a good idea to enable more of the tests in the test-suite, so we get an overview of potential regressions through CI.

However, I think there might be a more elegant solution to marking tests that we currently do not yet support as such: LLVM's lit accepts an --xfail=LIST option, to which you can pass a list of tests that should be considered expected failures. So instead of modifying 50+ files to add the XFAIL line to each of them, we could have a list of unsupported tests in single location (e.g., CMakeLists.txt) and pass that list to the xfail option of lit.

What do you think?

Personally I think using the --xfail opt is preferable ** only if** we can pass a text file containing the list of tests that do not yet pass, Listing them directly seems cumbersome, given that we have many failing tests currently.

[sommerlukas]

I think it's a good idea to enable more of the tests in the test-suite, so we get an overview of potential regressions through CI.
However, I think there might be a more elegant solution to marking tests that we currently do not yet support as such: LLVM's lit accepts an --xfail=LIST option, to which you can pass a list of tests that should be considered expected failures. So instead of modifying 50+ files to add the XFAIL line to each of them, we could have a list of unsupported tests in single location (e.g., CMakeLists.txt) and pass that list to the xfail option of lit.
What do you think?

Personally I think using the --xfail opt is preferable ** only if** we can pass a text file containing the list of tests that do not yet pass, Listing them directly seems cumbersome, given that we have many failing tests currently.

Agreed. I think it should be possible to initially generate the list/file from the list of failed tests during a run and store it in a separate file.

This file could be included using CMake's include, and the content of that file could be a single variable.

[whitneywhtsang]

LLVM's lit accepts an --xfail=LIST option, to which you can pass a list of tests that should be considered expected failures.

That's what I tried to do first, but the --xfail option doesn't work as expected with more than one test case listed. For example, I tried --xfail='Basic/access_to_subset.cpp;Basic/accessor/accessor.cpp' and --xfail 'Basic/access_to_subset.cpp;Basic/accessor/accessor.cpp'. (The environment variable LIT_XFAIL works as expected.)

That's why I simply use a quick script to add XFAIL to have the failing test.

Debugged the --xfail option to see why it is not working. It works if I invoke the lit command directly, but doesn't work as a cmake command. If using in cmake, need to escape the semicolon.

[sommerlukas]

LGTM, thanks for changing the implementation to use a list.

Would it be possible to list one file per line, to avoid merge conflicts when two people edit the list?

[whitneywhtsang]

Would it be possible to list one file per line, to avoid merge conflicts when two people edit the list?

Yes, done.

[etiotto]

Thanks @whitneywhtsang