[asan][win][msvc] override new and delete and seperate TUs (#68754)

Migrated from: https://reviews.llvm.org/D155879, with some of the
suggestions applied.

PR Description copied from above:

Currently asan simply exports each overridden new/delete function from
the DLL, this works fine normally, but fails if the user is overriding
some, but not all, of these functions. In this case the non-overridden
functions still come from the asan DLL, but they can't correctly call
the user provided override (for example sized op delete should fall back
to scalar op delete, if a scalar op delete is provided). Things were
also broken in the static build because all the asan overrides were
exported from the same TU, and so if you overrode one but not all of
them then you'd get ODR violations. This PR should fix both of these
cases, but the static case isn't really tested (and indeed one such test
does fail) because linking asan statically basically doesn't work on
windows right now with LLVM's version of asan. In fact, while we did fix
this in our fork, it was a huge mess and we've now made the dynamic
version work in all situations (/MD, /MT, /MDd, /MTd, etc) instead.

The following is the description from the internal PR that implemented
most of this feature.

> Previously, operator new/delete were provided as DLL exports when
linking dynamically and wholearchived when linked statically. Both
scenarios were broken. When linking statically, the user could not
define their own op new/delete, because they were already brought into
the link by ASAN. When dynamically linking, if the user provided some
but not all of the overloads, new and delete would be partially hooked.
For example, if the user defined scalar op delete, but the program then
called sized op delete, the sized op delete would still be the version
provided by ASAN instead of falling back to the user-defined scalar op
delete, like the standard requires.

> The change <internal PR number>: ASAN operator new/delete fallbacks in
the ASAN libraries fixes this moving all operator new/delete definitions
to be statically linked. However, this still won't work if
/InferAsanLibs still whole-archives everything since then all the op
new/deletes would always be provided by ASAN, which is why these changes
are necessary.

> With these changes, we will no longer wholearchive all of ASAN and
will leave the c++ parts (the op new/delete definitions) to be included
as a default library. However, it is also necessary to ensure that the
asan library with op new/delete will be searched before the
corresponding CRT library with the same op new/delete definitions. To
accomplish this, we make sure to add the asan library to the beginning
of the default lib list, or move it explicitly to the front if it's
already in the list. If the C runtime library is explicitly provided, we
make sure to warn the user if the current linker line will result in
operator new/delete not being provided by ASAN.

Note that the rearrangement of defaultlibs is not in this diff.

---------

Co-authored-by: Charlie Barto <[email protected]>

Author: barcharcraz

compiler-rt/lib/asan/CMakeLists.txt

@@ -38,9 +38,40 @@ if (NOT WIN32 AND NOT APPLE)
     )
 endif()
 
-set(ASAN_CXX_SOURCES
-  asan_new_delete.cpp
-  )
+if (WIN32)
+  set(ASAN_CXX_SOURCES asan_win_new_delete.cpp)
+else()
+  set(ASAN_CXX_SOURCES asan_new_delete.cpp)
+endif()
+
+if (APPLE)
+  set(ASAN_SOURCES ASAN_CXX_SOURCES)
+endif()
+
+if (WIN32)
+  set(ASAN_STATIC_IMPLIB_SOURCES
+    asan_win_delete_array_thunk.cpp
+    asan_win_delete_array_align_thunk.cpp
+    asan_win_delete_array_align_nothrow_thunk.cpp
+    asan_win_delete_array_nothrow_thunk.cpp
+    asan_win_delete_array_size_thunk.cpp
+    asan_win_delete_array_size_align_thunk.cpp
+    asan_win_delete_scalar_thunk.cpp
+    asan_win_delete_scalar_align_thunk.cpp
+    asan_win_delete_scalar_align_nothrow_thunk.cpp
+    asan_win_delete_scalar_nothrow_thunk.cpp
+    asan_win_delete_scalar_size_thunk.cpp
+    asan_win_delete_scalar_size_align_thunk.cpp
+    asan_win_new_array_thunk.cpp
+    asan_win_new_array_align_thunk.cpp
+    asan_win_new_array_align_nothrow_thunk.cpp
+    asan_win_new_array_nothrow_thunk.cpp
+    asan_win_new_scalar_thunk.cpp
+    asan_win_new_scalar_align_thunk.cpp
+    asan_win_new_scalar_align_nothrow_thunk.cpp
+    asan_win_new_scalar_nothrow_thunk.cpp
+    )
+endif()
 
 set(ASAN_STATIC_SOURCES
   asan_rtl_static.cpp
@@ -83,12 +114,20 @@ SET(ASAN_HEADERS
   asan_thread.h
   )
 
+if (WIN32)
+  list(APPEND ASAN_HEADERS
+    asan_win_new_delete_thunk_common.h
+    asan_win_thunk_common.h
+    )
+endif()
+
 include_directories(..)
 
 set(ASAN_CFLAGS ${SANITIZER_COMMON_CFLAGS})
 set(ASAN_COMMON_DEFINITIONS ${COMPILER_RT_ASAN_SHADOW_SCALE_DEFINITION})
 
 append_rtti_flag(OFF ASAN_CFLAGS)
+append_list_if(MSVC /EHsc ASAN_CFLAGS)
 
 # Silence warnings in system headers with MSVC.
 if(NOT CLANG_CL)
@@ -139,6 +178,15 @@ add_compiler_rt_object_libraries(RTAsan_dynamic
   CFLAGS ${ASAN_DYNAMIC_CFLAGS}
   DEFS ${ASAN_DYNAMIC_DEFINITIONS})
 
+if (WIN32)
+  add_compiler_rt_object_libraries(RTAsan_static_implib
+    ARCHS ${ASAN_SUPPORTED_ARCH}
+    SOURCES ${ASAN_STATIC_IMPLIB_SOURCES}
+    ADDITIONAL_HEADERS ${ASAN_HEADERS}
+    CFLAGS ${ASAN_CFLAGS} ${NO_DEFAULT_LIBS_OPTION}
+    DEFS ${ASAN_COMMON_DEFINITIONS})
+endif()
+
 if(NOT APPLE)
   add_compiler_rt_object_libraries(RTAsan
     ARCHS ${ASAN_SUPPORTED_ARCH}
@@ -239,13 +287,24 @@ else()
     DEFS ${ASAN_COMMON_DEFINITIONS}
     PARENT_TARGET asan)
 
+if(WIN32)
   add_compiler_rt_runtime(clang_rt.asan_static
     STATIC
     ARCHS ${ASAN_SUPPORTED_ARCH}
     OBJECT_LIBS RTAsan_static
+      RTAsan_static_implib
     CFLAGS ${ASAN_CFLAGS}
     DEFS ${ASAN_COMMON_DEFINITIONS}
     PARENT_TARGET asan)
+else()
+  add_compiler_rt_runtime(clang_rt.asan_static
+    STATIC
+    ARCHS ${ASAN_SUPPORTED_ARCH}
+    OBJECT_LIBS RTAsan_static
+    CFLAGS ${ASAN_CFLAGS}
+    DEFS ${ASAN_COMMON_DEFINITIONS}
+    PARENT_TARGET asan)
+endif()
 
   add_compiler_rt_runtime(clang_rt.asan-preinit
     STATIC
@@ -255,6 +314,13 @@ else()
     DEFS ${ASAN_COMMON_DEFINITIONS}
     PARENT_TARGET asan)
 
+
+  if (MSVC AND COMPILER_RT_DEBUG)
+    set(MSVC_DBG_SUFFIX _dbg)
+  else()
+    set(MSVC_DBG_SUFFIX )
+  endif()
+
   foreach(arch ${ASAN_SUPPORTED_ARCH})
     if (COMPILER_RT_HAS_VERSION_SCRIPT)
       add_sanitizer_rt_version_list(clang_rt.asan-dynamic-${arch}
@@ -311,6 +377,27 @@ else()
       DEFS ${ASAN_DYNAMIC_DEFINITIONS}
       PARENT_TARGET asan)
 
+    if(WIN32)
+      set_target_properties(clang_rt.asan${MSVC_DBG_SUFFIX}-dynamic-${arch}
+        PROPERTIES ARCHIVE_OUTPUT_NAME clang_rt.asan_dynamic-${arch}_implib
+                  ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}")
+
+      add_library(clang_rt.asan-${arch}_implib STATIC)
+      target_link_libraries(clang_rt.asan-${arch}_implib RTAsan_static_implib.${arch})
+      add_dependencies(asan clang_rt.asan-${arch}_implib)
+      add_dependencies(clang_rt.asan-${arch}_implib clang_rt.asan${MSVC_DBG_SUFFIX}-dynamic-${arch})
+      get_compiler_rt_output_dir(${arch} IMPLIB_OUTPUT_DIR)
+      set_target_properties(clang_rt.asan-${arch}_implib
+        PROPERTIES ARCHIVE_OUTPUT_NAME clang_rt.asan${MSVC_DBG_SUFFIX}_dynamic-${arch}
+                    ARCHIVE_OUTPUT_DIRECTORY ${IMPLIB_OUTPUT_DIR}
+                    STATIC_LIBRARY_OPTIONS "$<TARGET_LINKER_FILE:clang_rt.asan${MSVC_DBG_SUFFIX}-dynamic-${arch}>")
+      get_compiler_rt_install_dir(${arch} IMPLIB_INSTALL_DIR)
+      install(TARGETS clang_rt.asan-${arch}_implib
+        ARCHIVE DESTINATION ${IMPLIB_INSTALL_DIR}
+        LIBRARY DESTINATION ${IMPLIB_INSTALL_DIR}
+        RUNTIME DESTINATION ${IMPLIB_INSTALL_DIR})
+    endif()
+
     if (SANITIZER_USE_SYMBOLS AND NOT ${arch} STREQUAL "i386")
       add_sanitizer_rt_symbols(clang_rt.asan_cxx
         ARCHS ${arch})

compiler-rt/lib/asan/asan_interface.inc

@@ -187,3 +187,22 @@ INTERFACE_FUNCTION(__asan_update_allocation_context)
 INTERFACE_WEAK_FUNCTION(__asan_default_options)
 INTERFACE_WEAK_FUNCTION(__asan_default_suppressions)
 INTERFACE_WEAK_FUNCTION(__asan_on_error)
+
+#if SANITIZER_WINDOWS
+INTERFACE_FUNCTION(__asan_delete)
+INTERFACE_FUNCTION(__asan_delete_align)
+INTERFACE_FUNCTION(__asan_delete_array)
+INTERFACE_FUNCTION(__asan_delete_array_align)
+INTERFACE_FUNCTION(__asan_delete_array_size)
+INTERFACE_FUNCTION(__asan_delete_array_size_align)
+INTERFACE_FUNCTION(__asan_delete_size)
+INTERFACE_FUNCTION(__asan_delete_size_align)
+INTERFACE_FUNCTION(__asan_new)
+INTERFACE_FUNCTION(__asan_new_align)
+INTERFACE_FUNCTION(__asan_new_align_nothrow)
+INTERFACE_FUNCTION(__asan_new_array)
+INTERFACE_FUNCTION(__asan_new_array_align)
+INTERFACE_FUNCTION(__asan_new_array_align_nothrow)
+INTERFACE_FUNCTION(__asan_new_array_nothrow)
+INTERFACE_FUNCTION(__asan_new_nothrow)
+#endif  // SANITIZER_WINDOWS

compiler-rt/lib/asan/asan_win_delete_array_align_nothrow_thunk.cpp

@@ -0,0 +1,24 @@
+//===-- asan_win_delete_array_align_nothrow_thunk.cc ----------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of AddressSanitizer, an address sanity checker.
+//
+// Windows-specific user-provided new/delete operator detection and fallback.
+//===----------------------------------------------------------------------===//
+#include "asan_win_new_delete_thunk_common.h"
+
+// see diagram in asan_win_new_delete_thunk_common.h for the ordering of the
+// new/delete fallbacks.
+
+// Avoid tailcall optimization to preserve stack frame.
+#pragma optimize("", off)
+void operator delete[](void* ptr, std::align_val_t align,
+                       std::nothrow_t const&) noexcept {
+  // nothrow version is identical to throwing version
+  operator delete[](ptr, align);
+}

compiler-rt/lib/asan/asan_win_delete_array_align_thunk.cpp

@@ -0,0 +1,28 @@
+//===-- asan_win_delete_array_align_thunk.cc ------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of AddressSanitizer, an address sanity checker.
+//
+// Windows-specific user-provided new/delete operator detection and fallback.
+//===----------------------------------------------------------------------===//
+#include "asan_win_new_delete_thunk_common.h"
+
+// see diagram in asan_win_new_delete_thunk_common.h for the ordering of the
+// new/delete fallbacks.
+
+__asan_InitDefine<op_delete_array_align> init_delete_array_align;
+
+// Avoid tailcall optimization to preserve stack frame.
+#pragma optimize("", off)
+void operator delete[](void* ptr, std::align_val_t align) noexcept {
+  if (__asan_InitDefine<op_delete_scalar_align>::defined) {
+    __asan_delete_array_align(ptr, align);
+  } else {
+    operator delete(ptr, align);
+  }
+}

compiler-rt/lib/asan/asan_win_delete_array_nothrow_thunk.cpp

@@ -0,0 +1,23 @@
+//===-- asan_win_delete_array_nothrow_thunk.cc ----------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of AddressSanitizer, an address sanity checker.
+//
+// Windows-specific user-provided new/delete operator detection and fallback.
+//===----------------------------------------------------------------------===//
+#include "asan_win_new_delete_thunk_common.h"
+
+// see diagram in asan_win_new_delete_thunk_common.h for the ordering of the
+// new/delete fallbacks.
+
+// Avoid tailcall optimization to preserve stack frame.
+#pragma optimize("", off)
+void operator delete[](void* ptr, std::nothrow_t const&) noexcept {
+  // nothrow version is identical to throwing version
+  operator delete[](ptr);
+}

compiler-rt/lib/asan/asan_win_delete_array_size_align_thunk.cpp

@@ -0,0 +1,28 @@
+//===-- asan_win_delete_array_size_align_thunk.cc -------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of AddressSanitizer, an address sanity checker.
+//
+// Windows-specific user-provided new/delete operator detection and fallback.
+//===----------------------------------------------------------------------===//
+#include "asan_win_new_delete_thunk_common.h"
+
+// see diagram in asan_win_new_delete_thunk_common.h for the ordering of the
+// new/delete fallbacks.
+
+// Avoid tailcall optimization to preserve stack frame.
+#pragma optimize("", off)
+void operator delete[](void* ptr, size_t size,
+                       std::align_val_t align) noexcept {
+  if (__asan_InitDefine<op_delete_scalar_align>::defined &&
+      __asan_InitDefine<op_delete_array_align>::defined) {
+    __asan_delete_array_size_align(ptr, size, align);
+  } else {
+    operator delete[](ptr, align);
+  }
+}

compiler-rt/lib/asan/asan_win_delete_array_size_thunk.cpp

@@ -0,0 +1,27 @@
+//===-- asan_win_delete_array_size_thunk.cc -------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of AddressSanitizer, an address sanity checker.
+//
+// Windows-specific user-provided new/delete operator detection and fallback.
+//===----------------------------------------------------------------------===//
+#include "asan_win_new_delete_thunk_common.h"
+
+// see diagram in asan_win_new_delete_thunk_common.h for the ordering of the
+// new/delete fallbacks.
+
+// Avoid tailcall optimization to preserve stack frame.
+#pragma optimize("", off)
+void operator delete[](void* ptr, size_t size) noexcept {
+  if (__asan_InitDefine<op_delete_scalar>::defined &&
+      __asan_InitDefine<op_delete_array>::defined) {
+    __asan_delete_array_size(ptr, size);
+  } else {
+    operator delete[](ptr);
+  }
+}

compiler-rt/lib/asan/asan_win_delete_array_thunk.cpp

@@ -0,0 +1,28 @@
+//===-- asan_win_delete_array_thunk.cc ------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of AddressSanitizer, an address sanity checker.
+//
+// Windows-specific user-provided new/delete operator detection and fallback.
+//===----------------------------------------------------------------------===//
+#include "asan_win_new_delete_thunk_common.h"
+
+// see diagram in asan_win_new_delete_thunk_common.h for the ordering of the
+// new/delete fallbacks.
+
+__asan_InitDefine<op_delete_array> init_delete_array;
+
+// Avoid tailcall optimization to preserve stack frame.
+#pragma optimize("", off)
+void operator delete[](void* ptr) noexcept {
+  if (__asan_InitDefine<op_delete_scalar>::defined) {
+    __asan_delete_array(ptr);
+  } else {
+    operator delete(ptr);
+  }
+}

compiler-rt/lib/asan/asan_win_delete_scalar_align_nothrow_thunk.cpp

@@ -0,0 +1,24 @@
+//===-- asan_win_delete_scalar_align_nothrow_thunk.cc ---------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of AddressSanitizer, an address sanity checker.
+//
+// Windows-specific user-provided new/delete operator detection and fallback.
+//===----------------------------------------------------------------------===//
+#include "asan_win_new_delete_thunk_common.h"
+
+// see diagram in asan_win_new_delete_thunk_common.h for the ordering of the
+// new/delete fallbacks.
+
+// Avoid tailcall optimization to preserve stack frame.
+#pragma optimize("", off)
+void operator delete(void* ptr, std::align_val_t align,
+                     std::nothrow_t const&) noexcept {
+  // nothrow version is identical to throwing version
+  operator delete(ptr, align);
+}

compiler-rt/lib/asan/asan_win_delete_scalar_align_thunk.cpp

@@ -0,0 +1,24 @@
+//===-- asan_win_delete_scalar_align_thunk.cc -----------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of AddressSanitizer, an address sanity checker.
+//
+// Windows-specific user-provided new/delete operator detection and fallback.
+//===----------------------------------------------------------------------===//
+#include "asan_win_new_delete_thunk_common.h"
+
+// see diagram in asan_win_new_delete_thunk_common.h for the ordering of the
+// new/delete fallbacks.
+
+__asan_InitDefine<op_delete_scalar_align> init_delete_scalar_align;
+
+// Avoid tailcall optimization to preserve stack frame.
+#pragma optimize("", off)
+void operator delete(void* ptr, std::align_val_t align) noexcept {
+  __asan_delete_align(ptr, align);
+}