[Fortran/gfortran] Initial support to override DejaGNU annotations

The DejaGNU annotations in the gfortran test suite sometimes need to be
overridden, for instance in cases where the one of the compilers generates a
compile error for some non-standard extension while the other does not. This
PR adds support to override the some annotations, such as forcing a test to
pass with flang where it would fail in gfortran.

For now, only some of the behavior can be overridden, but this could be
extended in the future. The documentation was updated to reflect this (some
cleanup of the documentation was also carried out). Similarly, the static
test configuration generation script was updated and cleaned up.

An override file is also provided for two tests that were disabled in
27a6a3a since they are expected to pass in flang. The static test
configuration was also updated to reflect this change.

Author: tarunprabhu

Fortran/gfortran/README.md

@@ -16,7 +16,7 @@ behavior of the binary produced by the compiler.
 Currently, only the _execute_ tests are supported in `regression`. Both
 `compile` and `execute` tests have been enabled in `torture`.
 
-Of the supported tests, a number of tests have been disabled. These are listed 
+Of the supported tests, a number of tests have been disabled. These are listed
 in the `DisabledFiles.cmake` files that can be found in the various
 subdirectories of `Fortran/gfortran`. There are four
 categories of such tests:
@@ -35,12 +35,12 @@ features.
 
 - *Failing*: These tests fail at test-time.
 
-  - For "execute" tests, some crash on execution, others produce
-  incorrect/unexpected output. This could be a result of a bug in the
-  compiler/code generator or the runtime.
+- For "execute" tests, some crash on execution, others produce
+incorrect/unexpected output. This could be a result of a bug in the
+compiler/code generator or the runtime.
 
-  - For "compile" tests, this could be because the compilation succeeds when it
-  is expected to fail, or vice versa.
+- For "compile" tests, this could be because the compilation succeeds when it
+is expected to fail, or vice versa.
 
 Over time, the number of tests in the *unimplemented*, *skipped*, and *failing*
 categories should decrease. Eventually, only the *unsupported* category should
@@ -116,7 +116,7 @@ are unrelated to the gfortran tests here.
 
 Additional denylists for a particular feature can be included by creating
 `DisabledFiles_FEATURE.cmake` files (in the same format as those for the default
-denylists, `DisabledFiles.cmake`), and adding FEATURE to 
+denylists, `DisabledFiles.cmake`), and adding FEATURE to
 `TEST_SUITE_FORTRAN_FEATURES`. Additional compiler flags can be added using
 `CMAKE_Fortran_FLAGS`.
 
@@ -131,8 +131,8 @@ cmake -DTEST_SUITE_FORTRAN_FEATURES=FOO \
 `DisabledFiles_FOO.cmake` files can be created in the appropriate subdirectories
 if enabling the feature/flag results in the failure of tests that otherwise pass.
 Conversely, the feature/flag may cause some disabled tests to pass. These can be
-added to an allowlist file, `EnabledFiles_FOO.cmake` in the corresponding 
-directory. The file must contain a single variable named `ENABLED_FILES` with 
+added to an allowlist file, `EnabledFiles_FOO.cmake` in the corresponding
+directory. The file must contain a single variable named `ENABLED_FILES` with
 the file names of the tests that should be enabled (in the case of multi-file
 tests, this should be the name of the "main" file). An example of such a list is
 below.
@@ -152,8 +152,8 @@ implemented at a steady pace. The relevant tests in this directory should be
 enabled. This would involve building the test suite with one of the
 `TEST_SUITE_FORTRAN_*` flags described above.
 
-The build system uses static test configuration files named `tests.cmake` to be 
-found in the various subdirectories of the test suite. These are generated by 
+The build system uses static test configuration files named `tests.cmake` to be
+found in the various subdirectories of the test suite. These are generated by
 `utils/update-test-config.py`. The configuration files are the result of parsing
 the relevant DejaGNU annotations from the test files and are used by the various
 `CMakeLists.txt` files to set up the tests. These configuration files *must not*
@@ -178,29 +178,169 @@ Each field is described in the table below:
 | `<disabled-on>`| A space-separated list of targets on which the test is disabled. Each element of the list will be a regular expression that is expected to match an LLVM target triple.
 
 The test `kind`'s generally reflect what is being tested. For instance,
-`preprocess` tests only run the preprocessor, `assemble` tests generate assembly 
+`preprocess` tests only run the preprocessor, `assemble` tests generate assembly
 but no object code, the `compile` tests generate object code but do not invoke
 the linker while the `link` tests do invoke the linker. The `run` tests are
 "end-to-end" in that the code is compiled, linked and executed. These tests
 generally examine the output of the execution to ensure that the behavior of the
 generated executable is as expected.
 
-The test files should be kept in sync with gfortran. This has to be done 
-manually. When performing this update, the test configuration files must be 
+The test files should be kept in sync with gfortran. This has to be done
+manually. When performing this update, the test configuration files must be
 regenerated. This can be done by running `update-test-config.py` in the root of
 the test suite. The `-h` switch can be provided to the script for additional
 options.
 
 The test files in `regression` and `torture` *must not* be modified.
 
+### Overriding DejaGNU annotations ###
+
+In some cases, it may be necessary to override the DejaGNU annotations. Some of
+these include:
+
+- To invert the xfail status of a test (which is usually needed when there is
+a difference in behavior between gfortran and flang)
+
+- To selectively enable/disable a test on a particular platform.
+
+- In cases where a warning is expected, to override the gfortran-specific
+warning message with a flang-specific one.
+
+For now, only limited forms of overriding are supported. In particular, we
+do not support overriding warning messages, but that might be supported in the 
+future.
+
+In order to override annotations for tests in a given directory, create a
+file named `override.yaml` in that directory. The format of the file is
+described below. After the file has been populated, the static test 
+configuration files must be updated by running `update-test-config.py` as shown
+below. This will update all the `tests.cmake` files in the test suite.
+
+```
+$ cd /path/to/llvm-test-suite/Fortran/gfortran
+$ ./utils/update-test-config.py
+```
+
+#### override.yaml ####
+
+The `override.yaml` file can only be used to override attributes of tests 
+contained within the directory containing `override.yaml`. In order to override
+attributes of tests in subdirectories, an `override.yaml` file must be created
+in the subdirectory. This file only needs to be created if necessary. Unlike 
+the `DisabledFiles.cmake` files, it does not need to be present if test 
+attributes do not need to be overridden. At a high level, the format of the 
+file is as follows:
+
+```
+---
+"file1":
+  attr-name-1: attr-val-1
+  attr-name-2: attr-val-2
+  ...
+
+"file2":
+  attr-name-1: attr-val-2
+  ...
+
+...
+
+```
+
+Here each "fileN" is the full file name (including the extension) of the test.
+In the case of multi-file tests, this must be the name of the main test file.
+This must not contain paths (relative or absolute).
+
+The following attributes are currently supported.
+
+#### `disabled_on` ###
+
+The value must be a list of strings. Each string describes a platform on which
+the test must be disabled. This will usually be a target triple, and may be a
+regex. 
+
+This can be used to disable tests on platforms on which they have been
+explicitly enabled (this usually occurs when tests are restricted to run only 
+on certain platforms). In such cases, the string that is used to specify the 
+platform must exactly match the string that was used in a DejaGNU annotation. 
+For instance, the test `regression/simd-builtins-1.f90` is explicitly enabled 
+on certain Linux platforms only. This is done in the following annotation:
+
+```
+! { dg-do compile { target i?86-*-linux* x86_64-*-linux* aarch64*-*-linux* } }
+```
+
+In order to disable this test on AArch64-based Linux systems, the following 
+should be added to `regression/override.yaml`:
+
+```
+"simd_builtins-1.f90":
+    disabled_on: ["aarch64*-*-linux*"]
+```
+
+Note that the string in the `disabled_on` list exactly matches that in the 
+DejaGNU annotation. If you are not sure which string to use, check the 
+`enabled_on` column in the entry for the test in the static test configuration
+file, `tests.cmake`, which will be present in the directory containing the test.
+
+If the `enabled_on` column for the test in `tests.cmake` is empty, the test is 
+run on all platforms. In this case, overriding the `disabled_on` attribute is
+perfectly safe and will result in the test being executed on all platforms
+_except_ those in the `disabled_on list`.
+
+*WARNING*: However, if any of the strings in the `disabled_on` list is the sole
+entry in the `enabled_on` column for the test in `tests.cmake`, the test will 
+end up being enabled on _all_ platforms _except_ those in `disabled_on` list. 
+This is a known issue. There is no timeline for a fix for this.
+
+#### `enabled_on` ####
+
+The value must be a list of strings. Each string describes a platform on which
+the test must be enabled. This will usually be a target triple, and may be a
+regex. 
+
+This attribute will nearly always be used to override DejaGNU annotations that
+disable the test on certain platforms. In such cases, the string *must* exactly
+match the string that is present in the test file. For instance, the test
+`regression/chmod_1.f90` is disabled on cygwin using the following annotation:
+
+```
+! { dg-do run { target { ! { *-*-mingw* *-*-cygwin* } } } }
+```
+    
+In order to enable this test on cygwin, the following must be added to
+`regression/override.yaml`:
+    
+```
+"chmod_1.f90":
+    enable_on: ["*-*-cygwin*"]
+```
+
+Note that the string in the value of `enabled_on` exactly matches the string 
+in the `regression/chmod_1.f90`. If you are not sure which string to add to 
+the `override.yaml` file, check the `disabled_on` column in the entry for the
+test in the static test configuration file, `tests.cmake`, which will be 
+present in the directory containing the test.
+
+*WARNING*: If this is used on a test that is not disabled on any platform, it
+will result in the test being enabled _only_ on the platforms specified in the 
+`enabled_on` list.
+
+#### `xfail` ####
+
+The value must be a boolean, i.e. `true` or `false`. If `true`, the test will
+be expected to fail, and if `false`, the test will be expected to pass. If 
+the `xfail` column in the entry for the test in `tests.cmake` is the same as 
+the overridden value provided here, there will be no change in the behavior 
+of the test.
+
 
 ### TODO's ###
 
 If some of the items listed here are implemented, even in part, it should
 allows us to make better use of the test-suite.
 
 Several DejaGNU directives from the test files are either ignored or only
-partially supported - i.e. only a subset of the options specified by the 
+partially supported - i.e. only a subset of the options specified by the
 directive are handled correctly. In some
 cases, those directives check that the language feature/optimization being
 exercised by the tests is actually handled correctly. By ignoring them, we are
@@ -231,8 +371,8 @@ thereby causing the test to pass.
 
 #### `dg-warning` directive ####
 
-Currently, the `dg-warning` directive is ignored. It ought to be possible to 
-treat in a manner similar to `dg-error`.
+Currently, the `dg-warning` directive is ignored. It ought to be possible to
+treat this in a manner similar to `dg-error`.
 
 #### `scan-tree-dump` directive ####
 
@@ -249,24 +389,25 @@ of GCC to an equivalent representation in LLVM IR.
 #### `target` directive ####
 
 The `target` directive is used to restrict tests to run on certain platforms
-and/or systems. The directive can be fairly complex. While in most cases the 
-directive simply consists of a triple specifying the platform on which the 
-test is enabled (or disabled), negations, logical `and` and `or` operations are 
-also permitted. The directives can appear in several places within 
-a test file. In addition to "top-level" directives which control whether or 
-not the entire test is enabled, a `target` directive can also be used to 
-conditionally emit or suppress an error or a warning on certain platforms. 
+and/or systems. The directive can be fairly complex. While in most cases the
+directive simply consists of a triple specifying the platform on which the
+test is enabled (or disabled), negations, logical `and` and `or` operations are
+also permitted. The directives can appear in several places within
+a test file. In addition to "top-level" directives which control whether or
+not the entire test is enabled, a `target` directive can also be used to
+conditionally emit or suppress an error or a warning on certain platforms.
 Currently, there is limited support for these directives.
 
-- Logical operators on `target` directives are not supported. Directives 
+- Logical operators on `target` directives are not supported. Directives
 containing these operators are ignored entirely.
 
-- Only "top-level" directives are handled. `target` directives that appear 
-inside other directives such as `dg-error` or `dg-warning` are ignored. 
+- Only "top-level" directives are handled. `target` directives that appear
+inside other directives such as `dg-error` or `dg-warning` are ignored.
 
 #### Platform-specific disabling of tests ####
 
-Some tests fail on certain platforms but not on others. There is, currently, no
-way to disable these tests on a specific platform and these are disabled 
-everywhere. This is obviously not ideal since the extra coverage that the tests
-provide &mdash; even if on a limited set of platforms &mdash; is desirable.
+Some tests fail on certain platforms but not on others. It is possible to 
+disable such tests on the failing platforms and/or enable them selectively
+only on certain platforms. See the [Overriding
+DejaGNU annotations](#Overriding-DejaGNU-Annotations) section for details on
+how this can be done.

Fortran/gfortran/regression/override.yaml

@@ -0,0 +1,18 @@
+---
+# Overrides for DejaGNU annotations for tests in the directory in which this
+# file is present. To override tests in subdirectories, create/edit an
+# override.yaml file in that subdirectory.
+#
+# Please see gfortran/README.md for instructions on editing this file.
+#
+# When adding a test to this file, please leave a comment describing why the
+# behavior of the test is being overridden.
+
+# The following two tests use ```print <namelist name>```. This is a
+# non-standard extension that is not supported in certain cases in gfortran,
+# but, for now, is always supported in flang,
+"namelist_print_2.f":
+  xfail: false
+
+"print_fmt_2.f90":
+  xfail: false

Fortran/gfortran/regression/tests.cmake

@@ -1946,7 +1946,7 @@ compile;namelist_args.f90;xfail;-std=gnu;;
 compile;namelist_assumed_char.f90;xfail;-std=f95;;
 compile;namelist_blockdata.f;xfail;;;
 compile;namelist_empty.f90;;-std=legacy;;
-compile;namelist_print_2.f;xfail;-std=f95;;
+compile;namelist_print_2.f;;-std=f95;;
 compile;namelist_utf8.f90;;;;
 compile;nan_4.f90;;-std=gnu -fallow-invalid-boz;;
 compile;nan_5.f90;;-fno-range-check;;
@@ -2852,7 +2852,7 @@ compile;predict-3.f90;;-fno-tree-fre -fno-tree-ccp -Og;;
 compile;present_1.f90;xfail;;;
 compile;print_1.f90;xfail;;;
 compile;print_2.f90;xfail;;;
-compile;print_fmt_2.f90;xfail;;;
+compile;print_fmt_2.f90;;;;
 compile;print_fmt_3.f;;;;
 compile;print_fmt_4.f;xfail;;;
 compile;print_fmt_5.f90;xfail;;;