[Fortran/gfortran] Enable passing tests

[tarunprabhu]

These tests now pass. There are some tests that only pass on certain platforms, so they are left disabled.

[jeanPerier]

Thank you @tarunprabhu for doing this update!

My update for polymorphism in the regression tests match yours, I triaged the remaining 40 tests that were under the polymorphism TODO and do not pass. I will move them to appropriate categories once you merged this.

However, I am seeing 63 failures in other areas when running your patch, I think most of them (61) are related to tests that expect runtime errors, that flang correctly raise, but the test suite does not expect the error (e.g: regression/maxloc_bounds_3.f90). I attached the full list of these 61 tests runtime_error_failures.txt.

I also have a two more failures (regression/quad_1.f90 and regression/ieee/ieee_9.f90) because my build did not use libquadmath (the support for using it was recently added and is optional if not available on the system). I would advise not enabling them unconditionally and to add a test suite option to conditionally enable/disable such tests relying on quad math library (FLANG_RUNTIME_F128_MATH_LIB).

[tblah]

A few tests are failing for me on aarch64 with optimization:

Compile failure (UNREACHABLE executed at llvm-project/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp): internal_dummy_3.f90.

Three runtime failures:

• alloc_comp_class_4.f03
• maxlocval_1.f90
• unpack_bounds_1.f90

[tarunprabhu]

Thank you @tarunprabhu for doing this update!

My update for polymorphism in the regression tests match yours, I triaged the remaining 40 tests that were under the polymorphism TODO and do not pass. I will move them to appropriate categories once you merged this.

However, I am seeing 63 failures in other areas when running your patch, I think most of them (61) are related to tests that expect runtime errors, that flang correctly raise, but the test suite does not expect the error (e.g: regression/maxloc_bounds_3.f90). I attached the full list of these 61 tests runtime_error_failures.txt.

I also have a two more failures (regression/quad_1.f90 and regression/ieee/ieee_9.f90) because my build did not use libquadmath (the support for using it was recently added and is optional if not available on the system). I would advise not enabling them unconditionally and to add a test suite option to conditionally enable/disable such tests relying on quad math library (FLANG_RUNTIME_F128_MATH_LIB).

@jeanPerier, thanks for trying this out. what options do you use to configure flang and the test suite? I see 11 failures on my end, not 61. I can just disable the 61 tests in your list, so we can get this merged in, but I'd prefer to take a closer look at the failures before I do so.

Failed Tests (11):
  test-suite :: Fortran/gfortran/regression/gfortran-regression-compile-regression__oldstyle_5_f.test
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__alloc_comp_class_4_f03.test
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__do_check_1_f90.test
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__pointer_check_11_f90.test
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__pointer_check_1_f90.test
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__pointer_check_2_f90.test
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__pointer_check_3_f90.test
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__pointer_check_4_f90.test
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__pointer_check_6_f90.test
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__random_3_f90.test
  test-suite :: Fortran/gfortran/regression/gfortran-regression-execute-regression__unpack_bounds_1_f90.test

[jeanPerier]

@jeanPerier, thanks for trying this out. what options do you use to configure flang and the test suite? I see 11 failures on my end, not 61. I can just disable the 61 tests in your list, so we can get this merged in, but I'd prefer to take a closer look at the failures before I do so.

I could still reproduce my 61 failures with today's flang, here is my config where FLANG_DIR/LIT_DIR is just my llvm build directory (release build on X86-64):

cmake -G "Unix Makefiles" -DCMAKE_C_COMPILER=$FLANG_DIR/bin/clang \
    -DCMAKE_CXX_COMPILER=$FLANG_DIR/bin/clang++ \
    -DCMAKE_Fortran_COMPILER=$FLANG_DIR/bin/flang-new \
    -DTEST_SUITE_COLLECT_CODE_SIZE:STRING=OFF \
    -DTEST_SUITE_SUBDIRS:STRING="Fortran" \
    -DTEST_SUITE_FORTRAN:STRING=ON \
    -DTEST_SUITE_FORTRAN_ISO_C_HEADER_DIR=$SRC_DIR \
    ../llvm-test-suite
make -j
export NO_STOP_MESSAGE=1
$LIT_DIR/bin/llvm-lit -v -j 80 -o results.json .

I am missing some option here?

[tarunprabhu]

I am missing some option here?

This is odd. It looks pretty straightforward to me. The only difference between this and my configure line is that I don't explicitly specify the ISO_C_HEADER_DIR and I set -DCMAKE_BUILD_TYPE=Release

I tried with -DCMAKE_BUILD_TYPE=Debug and the number of failing tests went down to 3. On the other hand, I was able to reproduce the failures reported by @tblah on AArch64.

@AlexisPerry, could you run the tests with your build. The latest commits in this PR should address the AArch64 failures, but I would like to see what someone else with a (possibly) different build configuration on X86 gets.

[AlexisPerry]

I have a pretty vanilla build of flang and llvm-test-suite and I'm still getting errors building the tests:

[ 66%] Building CXX object MicroBenchmarks/libs/benchmark/test/CMakeFiles/options_test.dir/options_test.cc.o
/vast/home/aperry/llvm-test-suite/MicroBenchmarks/libs/benchmark/test/options_test.cc:68:10: error: variable 'actual_iterations' set but not used [-Werror,-Wunused-but-set-variable]
   68 |   size_t actual_iterations = 0;
      |          ^
1 error generated.
make[2]: *** [MicroBenchmarks/libs/benchmark/test/CMakeFiles/options_test.dir/build.make:76: MicroBenchmarks/libs/benchmark/test/CMakeFiles/options_test.dir/options_test.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:218431: MicroBenchmarks/libs/benchmark/test/CMakeFiles/options_test.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

and

CMakeFiles/Regression-C-float16-smoke.dir/float16-smoke.c.o: In function `printArg':
float16-smoke.c:(.text+0x2): undefined reference to `__extendhfsf2'
CMakeFiles/Regression-C-float16-smoke.dir/float16-smoke.c.o: In function `main':
float16-smoke.c:(.text+0x50): undefined reference to `__extendhfsf2'
float16-smoke.c:(.text+0x5d): undefined reference to `__truncsfhf2'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [SingleSource/Regression/C/CMakeFiles/Regression-C-float16-smoke.dir/build.make:97: SingleSource/Regression/C/Regression-C-float16-smoke] Error 1
make[1]: *** [CMakeFiles/Makefile2:235318: SingleSource/Regression/C/CMakeFiles/Regression-C-float16-smoke.dir/all] Error 2

[tarunprabhu]

I have a pretty vanilla build of flang and llvm-test-suite and I'm still getting errors building the tests:

These are not part of the Fortran tests. You can add the following to your configure line to build only the Fortran tests and ignore the rest.

-DTEST_SUITE_FORTRAN=ON \
-DTEST_SUITE_SUBDIRS="Fortran" \

[jeanPerier]

@tarunprabhu, I think I found the issue. the not utility from my test suite build does not recognize aborts with code 134 as failures, and thus fail to validate tests that are aborting at runtime as expected.

To be more specific, it seems my std::system returns 34304 in this case for which WEXITSTATUS is 134 and WSTOPSIG is 0 (which is what the utility expect to be different from zero when --crash is provided). I am building on an ubuntu machine with libc version 2.35.

llvm-not works for me. And if I change the "not" code to use the APPLE path with spawn and wait, "not" also works for me.

Except that in both case I see only 3 failures, which is less than expected because segfaults are considered a pass by "not" in bounds checking test, while the test intends to check for a clean abort with an error message (this may be the reason why you are seeing different number of failures in debug vs release, in release mode, some of the bad code causing segfaults may be "optimized out" causing not to complain that the test did not fail). I think we should not consider segmentation faults a pass in an xfail test.

$: ./gfortran-regression-execute-regression__maxloc_bounds_1_f90
   fatal Fortran runtime error ... Aborted
$: echo "$?"
  134

$: test-suit-build/tools/not --crash ./gfortran-regression-execute-regression__maxloc_bounds_1_f90
   fatal Fortran runtime error ... Aborted
$: echo "$?"
  1

$: llvm-build/bin/not--crash ./gfortran-regression-execute-regression__maxloc_bounds_1_f90
   fatal Fortran runtime error ... Aborted
$: echo "$?"
  0

All that to say:

• the 11 tests that are also failing for you should not be enabled.
• the other failures I had flagged may be enabled if they do not segfault, but some fix for "not" may be needed so that it reliably detects std::abort. I am still trying to understand why my std::system returns 34304...

[jeanPerier]

I think this is UBUNTU specific. It looks like they have a modified glibc and that std::system does not return the "usual" return codes, but instead a code that is shifted by 8 bits.

I could reproduce on all 5 different ubuntu systems I could get a hand on (with ubuntu version 18, 20 and 22), but I could not reproduce on any other linux distributions I tried.

Could anyone else with an ubuntu machine try to reproduce?

Maybe it would be safer to stay away from std::system (that calls sh) when possible (llvm not uses spwan or exec/fork under the hood via LLVM sys library in https://github.com/llvm/llvm-project/blob/main/llvm/lib/Support/Unix/Program.inc).

[tarunprabhu]

Thanks a lot for the investigation, Jean. That was very helpful.

I think it is safer to disable all 61 tests so we don't encounter failures on Ubuntu machines while I fix the not utility. What do you others think?

[AlexisPerry]

After some reconfiguring, I am now able to build and run the tests. I get 174 failures, and am digging into the details now.

[AlexisPerry]

Ok, 172 of them seem to be that flang-new produces a "Fortran STOP" in the output file that is not in the reference output file.

gfortran-regression-execute-regression__random_3_f90.test produces the error

Fortran STOP: code 1
IEEE arithmetic exceptions signaled: INEXACT

gfortran-regression-execute-regression__where_2_f90.test produces the error

error: child terminated by signal 8

[tarunprabhu]

Ok, 172 of them seem to be that flang-new produces a "Fortran STOP" in the output file that is not in the reference output file.

export NO_STOP_MESSAGE=1
lit ...

This is needed by the FCVS tests. This is documented somewhere but I can't seem to find it now.

[AlexisPerry]

Thank you. That takes care of the bulk of the failures I see. Now I only get the aforementioned other two.

[tarunprabhu]

@tblah, @jeanPerier, @AlexisPerry

Could you run this again and ensure that all tests pass? I have disabled a number of tests that you have identified as failing on various platforms and configurations.

[tblah]

All pass for me, thanks for your work on this.

Please wait for confirmation from everyone else before merging.

[jeanPerier]

Passed: 6205 (100.00%) on my Ubuntu machine.
Thanks @tarunprabhu for addressing my comments and enabling these tests!

[AlexisPerry]

Passed 5184 (100%) on my node.

LGTM!

[tarunprabhu]

Thanks @AlexisPerry, @jeanPerier and @tblah for the reviews and for checking.