[asan] Print diagnostic if unlimited stack size detected

[thurstond]

This adds a diagnostic message if the stack size is unlimited. This would have simplified the diagnosis of
google/sanitizers#856 (comment); we anticipate this may help diagnose future issues too.

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Thurston Dang (thurstond)

Changes

This adds a diagnostic message if the stack size is unlimited. This would have simplified the diagnosis of
google/sanitizers#856 (comment); we anticipate this may help diagnose future issues too.

---

Full diff: https://github.com/llvm/llvm-project/pull/133170.diff

1 Files Affected:

• (modified) compiler-rt/lib/asan/asan_rtl.cpp (+12)

diff --git a/compiler-rt/lib/asan/asan_rtl.cpp b/compiler-rt/lib/asan/asan_rtl.cpp
index 19c6c210b564c..d98b350828448 100644
--- a/compiler-rt/lib/asan/asan_rtl.cpp
+++ b/compiler-rt/lib/asan/asan_rtl.cpp
@@ -451,6 +451,18 @@ static bool AsanInitInternal() {
 
   DisableCoreDumperIfNecessary();
 
+#if SANITIZER_POSIX
+  if (StackSizeIsUnlimited()) {
+    VPrintf(1,
+            "WARNING: Unlimited stack size detected. This may affect "
+            "compatibility with the shadow mappings.\n");
+    // MSan and TSan re-exec with a fixed size stack. We don't do that because
+    // it may break the program. InitializeShadowMemory() will, if needed,
+    // re-exec without ASLR, which solves most shadow mapping compatibility
+    // issues.
+  }
+#endif
+
   InitializeShadowMemory();
 
   AsanTSDInit(PlatformTSDDtor);