[Fuzzer] Enable custom libc++ for Android

[rprichard]

The Android LLVM build system builds the arm64 fuzzer lib without HWASan, but then applications that enable HWASan can generated an object file with a HWASan-ified version of some libc++ symbols (e.g. std::__1::piecewise_construct). The linker can choose the HWASan-ified definition, but then it cannot resolve the relocation from libclang_rt.fuzzer-aarch64-android.a to this symbol because the high bits of the address are unexpectedly set. This produces an error:

relocation R_AARCH64_ADR_PREL_PG_HI21 out of range

Fix this problem by linking a custom isolated libc++ into Android's fuzzer library.

We need to pass through ANDROID_NATIVE_API_LEVEL so that the libc++ for 32-bit Android (API < 24) uses LLVM_FORCE_SMALLFILE_FOR_ANDROID.

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Ryan Prichard (rprichard)

Changes

The Android LLVM build system builds the arm64 fuzzer lib without HWASan, but then applications that enable HWASan can generated an object file with a HWASan-ified version of some libc++ symbols (e.g. std::__1::piecewise_construct). The linker can choose the HWASan-ified definition, but then it cannot resolve the relocation from libclang_rt.fuzzer-aarch64-android.a to this symbol because the high bits of the address are unexpectedly set. This produces an error:

relocation R_AARCH64_ADR_PREL_PG_HI21 out of range

Fix this problem by linking a custom isolated libc++ into Android's fuzzer library.

We need to pass through ANDROID_NATIVE_API_LEVEL so that the libc++ for 32-bit Android (API < 24) uses LLVM_FORCE_SMALLFILE_FOR_ANDROID.

---

Full diff: https://github.com/llvm/llvm-project/pull/70407.diff

2 Files Affected:

• (modified) compiler-rt/cmake/Modules/AddCompilerRT.cmake (+2)
• (modified) compiler-rt/lib/fuzzer/CMakeLists.txt (+2-2)

diff --git a/compiler-rt/cmake/Modules/AddCompilerRT.cmake b/compiler-rt/cmake/Modules/AddCompilerRT.cmake
index 5ed49f0f5588144..9c9d256a58b61be 100644
--- a/compiler-rt/cmake/Modules/AddCompilerRT.cmake
+++ b/compiler-rt/cmake/Modules/AddCompilerRT.cmake
@@ -625,6 +625,8 @@ macro(add_custom_libcxx name prefix)
   set_target_properties(${name}-clobber PROPERTIES FOLDER "Compiler-RT Misc")
 
   set(PASSTHROUGH_VARIABLES
+    ANDROID
+    ANDROID_NATIVE_API_LEVEL
     CMAKE_C_COMPILER_TARGET
     CMAKE_CXX_COMPILER_TARGET
     CMAKE_SHARED_LINKER_FLAGS
diff --git a/compiler-rt/lib/fuzzer/CMakeLists.txt b/compiler-rt/lib/fuzzer/CMakeLists.txt
index a9a10f724d1aa35..fb5adf1e5c9e698 100644
--- a/compiler-rt/lib/fuzzer/CMakeLists.txt
+++ b/compiler-rt/lib/fuzzer/CMakeLists.txt
@@ -59,7 +59,7 @@ CHECK_CXX_SOURCE_COMPILES("
 
 set(LIBFUZZER_CFLAGS ${COMPILER_RT_COMMON_CFLAGS})
 
-if(OS_NAME MATCHES "Linux|Fuchsia" AND
+if(OS_NAME MATCHES "Android|Linux|Fuchsia" AND
    COMPILER_RT_LIBCXX_PATH AND
    COMPILER_RT_LIBCXXABI_PATH)
   list(APPEND LIBFUZZER_CFLAGS -D_LIBCPP_ABI_VERSION=Fuzzer)
@@ -135,7 +135,7 @@ add_compiler_rt_runtime(clang_rt.fuzzer_interceptors
   CFLAGS ${LIBFUZZER_CFLAGS}
   PARENT_TARGET fuzzer)
 
-if(OS_NAME MATCHES "Linux|Fuchsia" AND
+if(OS_NAME MATCHES "Android|Linux|Fuchsia" AND
    COMPILER_RT_LIBCXX_PATH AND
    COMPILER_RT_LIBCXXABI_PATH)
   macro(partially_link_libcxx name dir arch)

[rprichard]

This change is also being tested on Gerrit:
https://android-review.googlesource.com/c/toolchain/llvm_android/+/2806865.

[smeenai]

LGTM

[rprichard]

The fuzzer's custom libc++ is built with -DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY, so the LIBCXX_HAS_PTHREAD_LIB etc feature testing in libcxx/cmake/config-ix.cmake doesn't work. It decides that the rt and pthread libaries do exist, and outputs #pragma comment(lib, "...") directives that break linking.

It looks like I can pass the appropriate variables manually, but we also have OS-specific sections in libcxx and libcxxabi, so I'll add sections for Android and upload a separate PR for that.

[rprichard]

The fuzzer's custom libc++ is built with -DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY, so the LIBCXX_HAS_PTHREAD_LIB etc feature testing in libcxx/cmake/config-ix.cmake doesn't work. It decides that the rt and pthread libaries do exist, and outputs #pragma comment(lib, "...") directives that break linking.

It looks like I can pass the appropriate variables manually, but we also have OS-specific sections in libcxx and libcxxabi, so I'll add sections for Android and upload a separate PR for that.

This change is #70534.

[vitalybuka]

Could you please take a look:
https://lab.llvm.org/buildbot/#/builders/77/builds/31876

[rprichard]

I saw the same unknown type name 'size_t' errors when I turned on COMPILER_RT_USE_LIBCXX for the Android LLVM team's build of LLVM. The fix was to avoid including two copies of libc++ in the include search path:

https://android-review.googlesource.com/c/toolchain/llvm_android/+/2806864

Previously, the Android LLVM (llvm_build/build.py) sometimes passed -nostdinc++ as an option for the compiler-rt runtimes, along with -isystem pointing at a copy of libc++'s headers. That doesn't work because, when COMPILER_RT_USE_LIBCXX is enabled, libFuzzer also adds -nostdinc++ and -isystem pointing at libc++ headers. When both of these are active, there are two copies of libc++'s headers on the search path, which fails, because (from that Gerrit CL):

The stddef.h in the first libc++ will delegate to the stddef.h in
the second, which will see that _LIBCPP_STDDEF_H is already defined,
and then not delegate to the stddef.h in the clang resource directory.
As a result, size_t/ptrdiff_t/etc aren't defined.

I don't see where the sanitizer buildbot is defining COMPILER_RT_USE_LIBCXX, but it looks like my attempt to reproduce the issue locally just finished, so I'll take a look.

[rprichard]

I don't see where the sanitizer buildbot is defining COMPILER_RT_USE_LIBCXX, but it looks like my attempt to reproduce the issue locally just finished, so I'll take a look.

Oh, it defaults to ON.

Anyway, I'm experimenting with an llvm-zorg fix that removes the redundant -stdlib=libc++ -I/$ANDROID_TOOLCHAIN/sysroot/usr/include/c++/v1 flags in $ANDROID_CXX_FLAGS.

[rprichard]

I uploaded llvm/llvm-zorg#66 to remove the redundant options. I tried to run buildbot_android.sh, and it was able to build things, but failed because it couldn't connect to an arm32 device. The #70534 PR is also needed if the buildbot tests libFuzzer, but I think it isn't necessary to set ANDROID_NATIVE_API_LEVEL because the sanitizer buildbots are already targeting API 24, which supports _FILE_OFFSET_BITS=64 on 32-bit targets. (I think I'll merge the #70534 PR tomorrow.)