Augment uncaught-exception.test fuzzer test to be msvc-compatible

[davidmrdavid]

Today, the uncaught-exception.test fuzzer test checks for the string "libFuzzer: deadly signal" in the program output as the result of an uncaught exception.

Although this is correct for clang, msvc reports a different error message: "libFuzzer: uncaught C++ exception". Since msvc reuses the libFuzzer infrastructure for ASan regression testing, it would help us greatly if the test handled the msvc divergence more gracefully.

This PR: augments this test so check for a different string (namely "libFuzzer: uncaught C++ exception") if the compiler target matches the msvc naming scheme.

I understand if this is outside the scope of support for LLVM as well, and I'm also open for different approaches here. Thanks!

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: David Justo (davidmrdavid)

Changes

Today, the uncaught-exception.test fuzzer test checks for the string "libFuzzer: deadly signal" in the program output as the result of an uncaught exception.

Although this is correct for clang, msvc reports a different error message: "libFuzzer: uncaught C++ exception". Since msvc reuses the libFuzzer infrastructure for ASan regression testing, it would help us greatly if the test handled the msvc divergence more gracefully.

This PR: augments this test so check for a different string (namely "libFuzzer: uncaught C++ exception") if the compiler target matches the msvc naming scheme.

I understand if this is outside the scope of support for LLVM as well, and I'm also open for different approaches here. Thanks!

---

Full diff: https://github.com/llvm/llvm-project/pull/125924.diff

1 Files Affected:

• (modified) compiler-rt/test/fuzzer/uncaught-exception.test (+6-3)

diff --git a/compiler-rt/test/fuzzer/uncaught-exception.test b/compiler-rt/test/fuzzer/uncaught-exception.test
index b055c88f6d9039..d1b98cfb7c74ba 100644
--- a/compiler-rt/test/fuzzer/uncaught-exception.test
+++ b/compiler-rt/test/fuzzer/uncaught-exception.test
@@ -4,7 +4,10 @@
 REQUIRES: windows
 RUN: %cpp_compiler %S/UncaughtException.cpp -o %t-UncaughtException
 
-RUN: not %run %t-UncaughtException 2>&1 | FileCheck %s
+# Clang will fail the test with 'deadly signal', but other compilers may fail with different error messages.
+# For example, msvc fails with 'uncaught C++ exception'. So the error we check depends on the compiler target.
+RUN: not %run %t-UncaughtException 2>&1 | FileCheck %s --check-prefixes=CHECK-CRASH,%if target={{.*-windows-msvc.*}} %{CHECK-MSVC%} %else %{CHECK-ERROR%}
 
-CHECK: ERROR: libFuzzer: deadly signal
-CHECK: Test unit written to ./crash
+CHECK-ERROR: ERROR: libFuzzer: deadly signal
+CHECK-MSVC: ERROR: libFuzzer: uncaught C++ exception
+CHECK-CRASH: Test unit written to ./crash

[davidmrdavid]

@vitalybuka - any chance you could take a look at this one as well (please let me know if I'm not following process here)? I noticed no one got added as a reviewer. Thank you!

[davidmrdavid]

A friendly ping here again :)

[davidmrdavid]

Thanks for the approval!
I don't have merge commit access (though I'd love to work towards that, since I'm looking to upstream more of our changes), so I can't merge this myself :) .

[davidmrdavid]

@vitalybuka - are you able to help me merge this? I notice it's approved, but I don't have commit permissions (though I'd love to gain them eventually).

[davidmrdavid]

@vitalybuka - can I get your help merging this, I don't have commit access (though having it would help me contribute upstream much faster!)? Thanks!

[vitalybuka]

Sorry, I just didn't notice the pings. :(

[thurstond]

The sanitizer-windows buildbot (https://lab.llvm.org/buildbot/#/builders/107/builds/11815) has been red ever since this was merged. It looks like it might be incorrectly comparing clang against the MSVC output?

C:/b/slave/sanitizer-windows/build/stage1/./bin/clang.exe       --driver-mode=g++ -O2 -gline-tables-only -fsanitize=address,fuzzer -IC:/b/slave/sanitizer-windows/llvm-project/compiler-rt\lib\fuzzer  -D_DISABLE_VECTOR_ANNOTATION -D_DISABLE_STRING_ANNOTATION C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\test\fuzzer/UncaughtException.cpp -o C:\b\slave\sanitizer-windows\build\stage1\projects\compiler-rt\test\fuzzer\X86_64DefaultWindowsConfig\Output\uncaught-exception.test.tmp-UncaughtException
# executed command: C:/b/slave/sanitizer-windows/build/stage1/./bin/clang.exe --driver-mode=g++ -O2 -gline-tables-only -fsanitize=address,fuzzer '-IC:/b/slave/sanitizer-windows/llvm-project/compiler-rt\lib\fuzzer' -D_DISABLE_VECTOR_ANNOTATION -D_DISABLE_STRING_ANNOTATION 'C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\test\fuzzer/UncaughtException.cpp' -o 'C:\b\slave\sanitizer-windows\build\stage1\projects\compiler-rt\test\fuzzer\X86_64DefaultWindowsConfig\Output\uncaught-exception.test.tmp-UncaughtException'
# .---command stdout------------
# |    Creating library C:\b\slave\sanitizer-windows\build\stage1\projects\compiler-rt\test\fuzzer\X86_64DefaultWindowsConfig\Output\uncaught-exception.test.lib and object C:\b\slave\sanitizer-windows\build\stage1\projects\compiler-rt\test\fuzzer\X86_64DefaultWindowsConfig\Output\uncaught-exception.test.exp
# `-----------------------------
# RUN: at line 9
not  C:\b\slave\sanitizer-windows\build\stage1\projects\compiler-rt\test\fuzzer\X86_64DefaultWindowsConfig\Output\uncaught-exception.test.tmp-UncaughtException 2>&1 | FileCheck C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\test\fuzzer\uncaught-exception.test --check-prefixes=CHECK-CRASH,CHECK-MSVC
# executed command: not 'C:\b\slave\sanitizer-windows\build\stage1\projects\compiler-rt\test\fuzzer\X86_64DefaultWindowsConfig\Output\uncaught-exception.test.tmp-UncaughtException'
# executed command: FileCheck 'C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\test\fuzzer\uncaught-exception.test' --check-prefixes=CHECK-CRASH,CHECK-MSVC
# .---command stderr------------
# | C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\test\fuzzer\uncaught-exception.test:12:13: error: CHECK-MSVC: expected string not found in input
# | CHECK-MSVC: ERROR: libFuzzer: uncaught C++ exception
# |             ^
# | <stdin>:1:1: note: scanning from here
# | INFO: Running with entropic power schedule (0xFF, 100).
# | ^
# | <stdin>:6:10: note: possible intended match here
# | ==3288== ERROR: libFuzzer: deadly signal
# |          ^
# | 
# | Input file: <stdin>
# | Check file: C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\test\fuzzer\uncaught-exception.test
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |             1: INFO: Running with entropic power schedule (0xFF, 100). 
# | check:12'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
# |             2: INFO: Seed: 3427103997 
# | check:12'0     ~~~~~~~~~~~~~~~~~~~~~~~
# |             3: INFO: Loaded 1 modules (24 inline 8-bit counters): 24 [00007FF7077734C8, 00007FF7077734E0),  
# | check:12'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |             4: INFO: Loaded 1 PC tables (24 PCs): 24 [00007FF70773D960,00007FF70773DAE0),  
# | check:12'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |             5: INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes 
# | check:12'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |             6: ==3288== ERROR: libFuzzer: deadly signal 
# | check:12'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~