[lit] add --max-retries-per-test execution option

When packaging LLVM we've seen arbitrary tests fail.
It happened sporadically and most of the times the test
do work if they are run a second time on the next day.

The tests themselves were always different and we didn't
know ahead of time which ones we wanted to re-run.
That's we filter-out a lot of `libomp` and `libarcher` tests [1].

This change allows us to set
`LIT_OPTS="--max-retries-per-test=12"`
when running any "check-XXX" build target. Then any lit test
will at most be re-run 12 times, unless there's an `ALLOW_RETRIES:`
in one of the test scripts that's specifying a different value
than `12`. `12` is just an example here, any positive integer
will work.

Please note, that this only adds the possibility to re-run
lit tests. It does not actually do it until the caller specifies
`--maxRetriesPerTest=<POSITIVE_INT>` either on a call to `lit` or
in `LIT_OPTS`.

Also note, that one can still use `ALLOW_RETRIES:` in test scripts
and it will always rule over `--max-retries-per-test`. There's a test
for this that covers this.

[1]: https://src.fedoraproject.org/rpms/llvm/blob/rawhide/f/llvm.spec#_2326

Downstream PR to make use of the `--max-retries-per-test` option: https://src.fedoraproject.org/rpms/llvm/pull-request/434
Downstream ticket: https://issues.redhat.com/browse/LLVM-145

Author: kwk

llvm/utils/lit/lit/cl_arguments.py

@@ -199,6 +199,13 @@ def parse_args():
         "0 means no time limit. [Default: 0]",
         type=_non_negative_int,
     )
+    execution_group.add_argument(
+        "--max-retries-per-test",
+        dest="maxRetriesPerTest",
+        metavar="N",
+        help="Maximum number of allowed retry attempts per test (NOTE: 'ALLOWED_RETRIES:' always takes precedence)",
+        type=_positive_int,
+    )
     execution_group.add_argument(
         "--max-failures",
         help="Stop execution after the given number of failures.",

llvm/utils/lit/lit/main.py

@@ -79,6 +79,9 @@ def main(builtin_params={}):
             )
             lit_config.maxIndividualTestTime = opts.maxIndividualTestTime
 
+    if opts.maxRetriesPerTest is not None:
+        lit_config.test_retry_attempts = opts.maxRetriesPerTest
+
     determine_order(discovered_tests, opts.order)
 
     selected_tests = [

llvm/utils/lit/tests/Inputs/max_retries_per_test/lit.cfg

@@ -0,0 +1,10 @@
+import lit.formats
+
+config.name = "max_retries_per_test"
+config.suffixes = [".py"]
+config.test_format = lit.formats.ShTest()
+config.test_source_root = None
+config.test_exec_root = None
+
+config.substitutions.append(("%python", lit_config.params.get("python", "")))
+config.substitutions.append(("%counter", lit_config.params.get("counter", "")))

llvm/utils/lit/tests/Inputs/max_retries_per_test/test.py

@@ -0,0 +1,22 @@
+# RUN: "%python" "%s" "%counter"
+
+import sys
+import os
+
+counter_file = sys.argv[1]
+
+# The first time the test is run, initialize the counter to 1.
+if not os.path.exists(counter_file):
+    with open(counter_file, "w") as counter:
+        counter.write("1")
+
+# Succeed if this is the fourth time we're being run.
+with open(counter_file, "r") as counter:
+    num = int(counter.read())
+    if num == 4:
+        sys.exit(0)
+
+# Otherwise, increment the counter and fail
+with open(counter_file, "w") as counter:
+    counter.write(str(num + 1))
+    sys.exit(1)

llvm/utils/lit/tests/allow-retries.py

@@ -70,3 +70,14 @@
 #     CHECK-TEST7: # executed command: export LLVM_PROFILE_FILE=
 # CHECK-TEST7-NOT: # executed command: export LLVM_PROFILE_FILE=
 #     CHECK-TEST7: Passed With Retry: 1
+
+# This test checks that a test can be re-run if the "--max-retries-per-test" is specified and otherwise
+# no ALLOW_RETRIES: is used in tests and no config.test_retry_attempts is adjusted in the test suite config.
+# This test checks that the config-wide test_retry_attempts property is used
+# when no ALLOW_RETRIES keyword is present.
+# TODO(kwk): Add test that ensures ALLOW_RETRIES: takes precedence over the --max-retries-per-test option.
+#
+# RUN: rm -f %t.counter
+# RUN: %{lit} %{inputs}/max_retries_per_test/test.py --max-retries-per-test=10 -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST8 %s --color --dump-input=always
+# CHECK-TEST8: SPECIFY ME BETTER
+