[Fortran/gfortran] Correctly handle "shouldfail" tests #57
Conversation
tarunprabhu
requested review from
tblah,
kiranchandramohan,
jeanPerier and
klausler
Fortran/gfortran/CMakeLists.txt
Outdated
| # this when running the test suite. | ||
| if (WIN32) | ||
| # Probably need to check %errorlevel% here. | ||
| message(FATAL_ERROR "Verification not implemented on Windows.") |
There was a problem hiding this comment.
Does this imply that that the test suite cannot be run on windows completely?
If so, would it be possible to just skip the error tests with a warning instead and at least still offer the possibility to run the "positive" tests?
There was a problem hiding this comment.
Does this imply that that the test suite cannot be run on windows completely?
If so, would it be possible to just skip the error tests with a warning instead and at least still offer the possibility to run the "positive" tests?
Yes, the test suite will not run at all on Windows. I put that check there as a safeguard to ensure that if the top-level check for Windows (in a parent directory) is not disabled without also adding the missing parts here. It is not possible to run some subset of tests on Windows.
| all_bounds_1.f90 | ||
| bounds_check_12.f90 | ||
| bounds_check_array_ctor_4.f90 | ||
| bounds_check_fail_3.f90 | ||
| maxloc_bounds_1.f90 | ||
| maxloc_bounds_2.f90 | ||
| maxloc_bounds_3.f90 | ||
| maxloc_bounds_4.f90 | ||
| maxloc_bounds_5.f90 | ||
| maxloc_bounds_7.f90 | ||
| maxloc_bounds_8.f90 | ||
| pack_bounds_1.f90 | ||
| spread_bounds_1.f90 |
There was a problem hiding this comment.
Isn't it possible to now remove these tests from the list instead of moving them to the category of tests for which flang does not emit runtime errors (since I believe it is for these tests)?
There was a problem hiding this comment.
Ah, I misunderstood your comment in the original.
Interestingly, when I enable the test, they fail anyway despite correctly generating a runtime error. This seems to be a bug in the way I check for failures. I will look into this.
Thanks for the catch!
There was a problem hiding this comment.
Interestingly, when I enable the test, they fail anyway despite correctly generating a runtime error. This seems to be a bug in the way I check for failures. I will look into this.
This occurred because calling abort from the test instantly caused the test to be reported as a failure without running the custom verifier which checks for a non-zero return code.
The easiest way to fix this is to use LLVM's not utility. But we cannot rely on the utility being available since we may not have LLVM's build directory when running the test suite and LLVM could have been configured with -DLLVM_INSTALL_UTILS=OFF.
I will look into this. Thanks for taking a look at this patch. |
tarunprabhu
force-pushed
the
shouldfail
branch
from
75a8e07 to
3fbeaf9
Compare
|
Updated the patch to use the not tool that was merged in #60. This resulted in a number of test failures which have been added to the disabled tests list. Several tests also now pass which have been removed from the list. There are probably more tests that can be removed but that will be done in a separate patch. Some tests were also re-classified. |
tarunprabhu
force-pushed
the
shouldfail
branch
from
70c466f to
11f89f9
Compare
Correctly handle the DejaGNU shouldfail annotation which asserts that the test is expected to fail at runtime. Disable tests that now fail. These may be actual bugs in the runtime or a case of flang behaving differently from gfortran and will need to be investigated.
…e the disabled files list.
tarunprabhu
force-pushed
the
shouldfail
branch
from
11f89f9 to
9bfa9e4
Compare
|
Since #97. |
The DejaGNU
shouldfailannotation is now handled correctly. Support for tests that are expected to fail has been added to the build system for the gfortran tests. This currently does not work on Windows, but there are several other issues with building the tests on Windows anyway.A number of tests that were previously passing are failing as a result and those have been disabled. The failures may be a result of an actual bug in the runtime, or a case of flang behaving differently from gfortran. That will need to be investigated.