[asan] Install pthread_atfork

[vitalybuka]

This prevents deadlocks in forked process
if parent had more then one running threads.

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Vitaly Buka (vitalybuka)

Changes

This prevents deadlocks in forked process
if parent had more then one running threads.

---

Full diff: https://github.com/llvm/llvm-project/pull/75290.diff

6 Files Affected:

• (modified) compiler-rt/lib/asan/asan_fuchsia.cpp (+2)
• (modified) compiler-rt/lib/asan/asan_internal.h (+1)
• (modified) compiler-rt/lib/asan/asan_posix.cpp (+24)
• (modified) compiler-rt/lib/asan/asan_rtl.cpp (+2)
• (modified) compiler-rt/lib/asan/asan_win.cpp (+2)
• (modified) compiler-rt/test/sanitizer_common/TestCases/Posix/fork_threaded.c (+1-1)

diff --git a/compiler-rt/lib/asan/asan_fuchsia.cpp b/compiler-rt/lib/asan/asan_fuchsia.cpp
index 2b15504123bee..12625e9d75833 100644
--- a/compiler-rt/lib/asan/asan_fuchsia.cpp
+++ b/compiler-rt/lib/asan/asan_fuchsia.cpp
@@ -240,6 +240,8 @@ void FlushUnneededASanShadowMemory(uptr p, uptr size) {
 // So this doesn't install any atexit hook like on other platforms.
 void InstallAtExitCheckLeaks() {}
 
+void InstallAtForkHandler() {}
+
 }  // namespace __asan
 
 namespace __lsan {
diff --git a/compiler-rt/lib/asan/asan_internal.h b/compiler-rt/lib/asan/asan_internal.h
index 5b97e77882cd6..2944ebe213b5d 100644
--- a/compiler-rt/lib/asan/asan_internal.h
+++ b/compiler-rt/lib/asan/asan_internal.h
@@ -126,6 +126,7 @@ void *AsanDlSymNext(const char *sym);
 bool HandleDlopenInit();
 
 void InstallAtExitCheckLeaks();
+void InstallAtForkHandler();
 
 #define ASAN_ON_ERROR() \
   if (&__asan_on_error) \
diff --git a/compiler-rt/lib/asan/asan_posix.cpp b/compiler-rt/lib/asan/asan_posix.cpp
index e1f66641617cc..37fca8aea5151 100644
--- a/compiler-rt/lib/asan/asan_posix.cpp
+++ b/compiler-rt/lib/asan/asan_posix.cpp
@@ -148,6 +148,30 @@ void PlatformTSDDtor(void *tsd) {
 }
 #endif
 
+void InstallAtForkHandler() {
+  auto before = []() {
+    if (CAN_SANITIZE_LEAKS) {
+      __lsan::LockGlobal();
+    }
+    // `_lsan` functions defined regardless of `CAN_SANITIZE_LEAKS` and do the
+    // job.
+    __lsan::LockThreads();
+    __lsan::LockAllocator();
+    StackDepotLockAll();
+  };
+  auto after = []() {
+    StackDepotUnlockAll();
+    // `_lsan` functions defined regardless of `CAN_SANITIZE_LEAKS` and do the
+    // job.
+    __lsan::UnlockAllocator();
+    __lsan::UnlockThreads();
+    if (CAN_SANITIZE_LEAKS) {
+      __lsan::UnlockGlobal();
+    }
+  };
+  pthread_atfork(before, after, after);
+}
+
 void InstallAtExitCheckLeaks() {
   if (CAN_SANITIZE_LEAKS) {
     if (common_flags()->detect_leaks && common_flags()->leak_check_at_exit) {
diff --git a/compiler-rt/lib/asan/asan_rtl.cpp b/compiler-rt/lib/asan/asan_rtl.cpp
index b28f9f181239b..a61deed7382b0 100644
--- a/compiler-rt/lib/asan/asan_rtl.cpp
+++ b/compiler-rt/lib/asan/asan_rtl.cpp
@@ -493,6 +493,8 @@ static bool AsanInitInternal() {
     InstallAtExitCheckLeaks();
   }
 
+  InstallAtForkHandler();
+
 #if CAN_SANITIZE_UB
   __ubsan::InitAsPlugin();
 #endif
diff --git a/compiler-rt/lib/asan/asan_win.cpp b/compiler-rt/lib/asan/asan_win.cpp
index d5a30f471e2b0..f16ce677618e4 100644
--- a/compiler-rt/lib/asan/asan_win.cpp
+++ b/compiler-rt/lib/asan/asan_win.cpp
@@ -203,6 +203,8 @@ void InitializePlatformInterceptors() {
 
 void InstallAtExitCheckLeaks() {}
 
+void InstallAtForkHandler() {}
+
 void AsanApplyToGlobals(globals_op_fptr op, const void *needle) {
   UNIMPLEMENTED();
 }
diff --git a/compiler-rt/test/sanitizer_common/TestCases/Posix/fork_threaded.c b/compiler-rt/test/sanitizer_common/TestCases/Posix/fork_threaded.c
index 67d2993d11bf8..1a52702c5de8c 100644
--- a/compiler-rt/test/sanitizer_common/TestCases/Posix/fork_threaded.c
+++ b/compiler-rt/test/sanitizer_common/TestCases/Posix/fork_threaded.c
@@ -1,6 +1,6 @@
 // RUN: %clang -O0 %s -o %t && %env_tool_opts=die_after_fork=0 %run %t
 
-// UNSUPPORTED: asan, hwasan
+// UNSUPPORTED: hwasan
 
 // Forking in multithread environment is unsupported. However we already have
 // some workarounds, and will add more, so this is the test.

[rorth]

Since this patch, all asan tests loop on Solaris. This had been hidden for a bit by an unrelated extended build breakage on the bots, but now every ninja check-all on the Solaris/amd64 bot times out. I could trace this to this patch.

E.g. when running projects/compiler-rt/test/asan/I386SunOSConfig/TestCases/Output/alloca_big_alignment.cpp.tmp, I get the expected output

=================================================================
==22223==ERROR: AddressSanitizer: dynamic-stack-buffer-overflow on address 0xfeffd88a at pc 0x0812907d bp 0xfeffd7f4 sp 0xfeffd7ec
WRITE of size 1 at 0xfeffd88a thread T0

and afterwards the test loops. truss shows an unending series of

22210:	yield()						= 0
22210:	yield()						= 0
22210:	yield()						= 0

and pstack gives

22213:	/var/llvm/local-amd64-release-stage2-A-flang-492214/tools/clang/stage2
 fdfbebc5 yield    (0x8139158, 0x8109558, 0x818a580, 0x0, 0x5dd, 0x8139158) + 15
 0810cd32 __sanitizer::FutexWait(__sanitizer::atomic_uint32_t*, unsigned int) (0xfe00a000, 0xfdebdd56, 0x805ad7c, 0xfdfa0107, 0xfeffc68c, 0x5) + 12
 080f4952 __asan::InstallAtForkHandler()::$_0::__invoke() (0xfde26fc0, 0x7, 0xfe010200, 0xfe010140, 0x7, 0x5) + 12
 fdfa49c8 forkx    (0x0, 0xfe5ad000, 0x89f, 0xfdfa4b8c) + c8
 fdfa4b9d fork     (0x8139158, 0x811563e, 0xfeffc720, 0xfd6007a0, 0x4, 0x8139158) + 1d
 0810ccd2 __sanitizer::internal_fork() () + 12

This seems no wonder given that sanitizer_common/sanitizer_solaris.cpp has

void FutexWait(atomic_uint32_t *p, u32 cmp) {
  // FIXME: implement actual blocking.
  sched_yield();
}

sanitizer_mac.cpp is the same, btw., and even sanitizer_linux.cpp has

#  if !SANITIZER_SOLARIS
void FutexWait(atomic_uint32_t *p, u32 cmp) {
#    if SANITIZER_FREEBSD
  _umtx_op(p, UMTX_OP_WAIT_UINT, cmp, 0, 0);
#    elif SANITIZER_NETBSD
  sched_yield(); /* No userspace futex-like synchronization */
#    else
  internal_syscall(SYSCALL(futex), (uptr)p, FUTEX_WAIT_PRIVATE, cmp, 0, 0, 0);
#    endif
}

so even NetBSD would be affected.

[vitalybuka]

That's bad, we need to fix this.
Do you see how we endup in handler, from internal_fork.
internal_fork claims it will not call handlers.

[rorth]

It took me a bit to notice this snippet in sanitizer_solaris.cpp:

DECLARE__REAL_AND_INTERNAL(int, fork, void) {
  // TODO(glider): this may call user's pthread_atfork() handlers which is bad.
  return _REAL(fork)();
}

which didn't show up in searches for internal_fork.

From what I could learn from libc disassembly and the OpenSolaris sources, the only way to avoid the handlers on Solaris is to invoke the syscall directly. This is highly unportable, however: syscalls are an implemention detail that can (and does) change. There's reasonable hope that this won't happen for the remaining livetime of Solaris 11.4, though.

I'll give such a patch a try...

[rorth]

I'll give such a patch a try...

That would be something like

int internal_fork(void) {
  // Call syscall directly to avoid pthread_atfork handler processing.
  //
  // This is highly unportable on Solaris since syscalls are an implementation
  // detail subject to change.
  return syscall(SYS_forksys, 0, 0);
}

Unfortunately, this fails miserably: at least every asan test that invokes the llvm-symbolizer fails like

==4030==Launching Symbolizer process: /usr/bin/llvm-symbolizer --demangle --inlines --default-arch=i386 
==4031==Waiting on the process failed (errno 10).
==4031==WARNING: external symbolizer didn't start up correctly!

From all I could learn from the OpenSolaris libc code (lib/libc/port/threads/scalls.c (forkx), it seems that there happens so much processing on libc-internal data structures that cannot simply be skipped that there's no reasonable chance to run fork without the handlers. Expecting to be able to seems to be hack that may work on some platforms, but not on others.