Use LLVM runtimes build #60993

edymtt · 2022-09-08T09:30:13Z

LLVM plans to remove the legacy methods for building compiler-rt. This change updates build-script to use LLVM_ENABLE_RUNTIMES to build compiler-rt with the just-built clang instead of the legacy methods.

(cherry picked from #58465 to double check my suggestion is sound)

LLVM plans to remove the legacy methods for building compiler-rt. This change updates build-script to use LLVM_ENABLE_RUNTIMES to build compiler-rt with the just-built clang instead of the legacy methods. (cherry picked from commit 6386f60)

edymtt · 2022-09-08T09:30:25Z

@swift-ci please build toolchain

edymtt · 2022-09-08T09:30:33Z

@swift-ci please test

edymtt · 2022-09-08T12:41:21Z

@swift-ci please test

edymtt · 2022-09-08T12:41:31Z

@swift-ci please build toolchain

edymtt · 2022-09-08T12:41:58Z

@swift-ci Please Test Source Compatibility Release

edymtt · 2022-09-08T13:17:36Z

It looks like under Linux we started detecting i386 as an architecture we can build for, and then we stumble finding missing headers/libraries

-- Looking for __i386__
-- Looking for __i386__ - found
-- Builtin supported architectures: i386;x86_64
<snip>
FAILED: CMakeFiles/clang_rt.builtins-i386.dir/absvdi2.c.o 
/home/build-user/build/buildbot_linux/llvm-linux-x86_64/./bin/clang --target=x86_64-unknown-linux-gnu -DVISIBILITY_HIDDEN  -O3 -DNDEBUG  -m32 -std=c11 -fPIC -fno-builtin -fvisibility=hidden -fomit-frame-pointer -MD -MT CMakeFiles/clang_rt.builtins-i386.dir/absvdi2.c.o -MF CMakeFiles/clang_rt.builtins-i386.dir/absvdi2.c.o.d -o CMakeFiles/clang_rt.builtins-i386.dir/absvdi2.c.o -c /home/build-user/llvm-project/compiler-rt/lib/builtins/absvdi2.c
In file included from /home/build-user/llvm-project/compiler-rt/lib/builtins/absvdi2.c:13:
In file included from /home/build-user/llvm-project/compiler-rt/lib/builtins/int_lib.h:93:
In file included from /home/build-user/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0/include/limits.h:21:
/usr/include/limits.h:26:10: fatal error: 'bits/libc-header-start.h' file not found
#include <bits/libc-header-start.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

Trying to figure out why that's the case.

edymtt · 2022-09-28T14:48:38Z

@swift-ci please test

edymtt · 2022-09-28T14:52:18Z

@swift-ci please test Linux

edymtt · 2022-09-30T20:47:33Z

Figured out the failure was due to the fact that the configuration of builtins compiles archives to detect compiler features, and so we assume we can compile for x86 even if we miss the libraries on the bots

https://reviews.llvm.org/D91334

edymtt · 2022-09-30T20:47:39Z

@swift-ci please test

edymtt · 2022-09-30T21:27:17Z

Now hitting some troubles building libclang_rt.tsan.so

/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25
compiler-rt/lib/tsan/rtl/CMakeFiles/clang_rt.tsan-dynamic-x86_64.dir/tsan_interceptors_libdispatch.cpp.o: in function `__tsan::dispatch_callback_wrap(void*)':
tsan_interceptors_libdispatch.cpp:(.text._ZN6__tsanL22dispatch_callback_wrapEPv+0x62): undefined reference to `dispatch_queue_offsets'
/usr/bin/ld: compiler-rt/lib/tsan/rtl/CMakeFiles/clang_rt.tsan-dynamic-x86_64.dir/tsan_interceptors_libdispatch.cpp.o: in function `__interceptor_dispatch_async':
tsan_interceptors_libdispatch.cpp:(.text.__interceptor_dispatch_async+0x8f): undefined reference to `_Block_copy'
/usr/bin/ld: compiler-rt/lib/tsan/rtl/CMakeFiles/clang_rt.tsan-dynamic-x86_64.dir/tsan_interceptors_libdispatch.cpp.o: in function `__tsan::invoke_and_release_block(void*)':
tsan_interceptors_libdispatch.cpp:(.text._ZN6__tsanL24invoke_and_release_blockEPv+0xc): undefined reference to `_Block_release'
<snip>

They will fail linking since we are not building the runtimes with `-Wl,-z,defs`, which will fail in presence of undefined symbols in the final `libclang_rt.tsan` library.

edymtt · 2022-10-03T16:19:36Z

@swift-ci please test

edymtt · 2022-10-03T16:21:09Z

The failure seems to be caused by the fact that with the runtimes builds we are now linking the sanitizer libraries with -Wl,-z,defs, so now the linker complains about undefined symbols related to dispatch/BlocksRuntime.

edymtt · 2022-10-03T19:23:28Z

Getting another regression in Linux when running integration tests

+ python3 /home/build-user/llvm-project/llvm/utils/lit/lit.py . -sv --param package-path=/home/build-user/build/buildbot_linux/none-swift_package_sandbox_linux-x86_64 --param test-exec-root=/home/build-user/build/buildbot_linux/none-swift_package_sandbox_linux-x86_64/tests --param llvm-bin-dir=/home/build-user/build/buildbot_linux/llvm-linux-x86_64/bin
~/swift-integration-tests ~
-- Testing: 31 tests, 31 workers --
<snip>
FAIL: swift-package-tests :: test-codecov-package/test-codecov-package.txt (30 of 31)
******************** TEST 'swift-package-tests :: test-codecov-package/test-codecov-package.txt' FAILED ********************
error: link command failed with exit code 1 (use -v to see invocation)
<snip>
clang version 13.0.0 (https://github.com/apple/llvm-project.git c3aebe50177cd03e8ec426c8e337efd6157b571d)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/build-user/build/buildbot_linux/none-swift_package_sandbox_linux-x86_64/usr/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64
clang-13: error: no such file or directory: '/home/build-user/build/buildbot_linux/none-swift_package_sandbox_linux-x86_64/usr/lib/swift/clang/lib/linux/libclang_rt.profile-x86_64.a'
error: fatalError


--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
********************
Failed Tests (1):
  swift-package-tests :: test-codecov-package/test-codecov-package.txt

edymtt · 2022-10-03T20:21:52Z

This looks related to

swiftlang/llvm-project@32e1ceb
https://discourse.llvm.org/t/rfc-time-to-drop-legacy-runtime-paths/64628/26
https://reviews.llvm.org/D107799

edymtt · 2022-10-03T20:25:17Z

@swift-ci please test Linux

edymtt · 2022-10-03T22:29:56Z

@swift-ci please test Linux

Remove spurious newline

edymtt · 2022-10-04T14:31:37Z

@swift-ci please test

edymtt · 2022-10-04T14:31:49Z

@swift-ci Please Test Source Compatibility Release

edymtt · 2022-10-04T14:31:59Z

@swift-ci please build toolchain

edymtt · 2022-10-04T14:32:40Z

@swift-ci please test Apple Silicon

edymtt · 2022-10-04T14:33:25Z

@swift-ci Please ASAN test

edymtt · 2022-10-04T14:33:53Z

@swift-ci Please test CentOS 7 platform

edymtt · 2022-10-04T14:33:59Z

@swift-ci Please test CentOS 8 platform

edymtt · 2022-10-04T14:34:07Z

@swift-ci Please test Amazon Linux 2 platform

edymtt · 2022-10-04T15:11:36Z

Just realized now that LLVM_ENABLE_RUNTIMES may remain in caches and cause trouble with other PR testing.

Tackling this in #61435

yln · 2022-10-28T18:20:06Z

utils/build-presets.ini

+# We are now inheriting the -Wl,-z,defs linker flags
+# from HandleLLVMOptions.cmake, so the compiler now complains
+# for BlocksRuntime symbols that are undefined
+  -DCOMPILER_RT_INTERCEPT_LIBDISPATCH:BOOL=OFF


This should not be turned off, otherwise we will break TSan. @edymtt

Can we fix the undefined references by linking the blocks runtime -fblocks?

The CMake code here is trying to do this:
https://github.com/llvm/llvm-project/blob/f418f88824905c372cb9252b3f675012a3fc799d/compiler-rt/CMakeLists.txt#L172

I can try explore that -- to the best of my understanding that is not present in CI nodes, and we are building that ourselves when we are building Swift, but likely I'm missing something

edymtt · 2024-01-31T22:56:13Z

Duplicate of #67986

Use LLVM runtimes build

3c2e7b6

LLVM plans to remove the legacy methods for building compiler-rt. This change updates build-script to use LLVM_ENABLE_RUNTIMES to build compiler-rt with the just-built clang instead of the legacy methods. (cherry picked from commit 6386f60)

Update llvm install components

849f858

Don't build builtins for i386 under linux

cbb22f3

Don't build libdispatch support in sanitizers

1de49ef

They will fail linking since we are not building the runtimes with `-Wl,-z,defs`, which will fail in presence of undefined symbols in the final `libclang_rt.tsan` library.

Opt into LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON like LLVM-15 would do

338dead

Disable LLVM_ENABLE_PER_TARGET_RUNTIME_DIR for runtimes

3c68844

edymtt added 4 commits October 4, 2022 07:25

Factor compiler-rt configuration

787a576

Merge branch 'main' into use-runtimes-build-bis

37b8763

Update build-presets.ini

d5c110c

Remove spurious newline

Add missing new lines

585eb6b

Unset old variables that control compiler-rt build

8cc9592

edymtt mentioned this pull request Oct 4, 2022

Unset LLVM_ENABLE_RUNTIMES in CMake caches #61435

Merged

Merge branch 'main' into use-runtimes-build-bis

1719bd7

yln reviewed Oct 28, 2022

View reviewed changes

edymtt marked this as a duplicate of #67986 Jan 31, 2024

edymtt closed this Jan 31, 2024

edymtt deleted the use-runtimes-build-bis branch January 31, 2024 22:56

Use LLVM runtimes build #60993

Use LLVM runtimes build #60993

Uh oh!

Conversation

edymtt commented Sep 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edymtt commented Sep 8, 2022

Uh oh!

edymtt commented Sep 8, 2022

Uh oh!

edymtt commented Sep 8, 2022

Uh oh!

edymtt commented Sep 8, 2022

Uh oh!

edymtt commented Sep 8, 2022

Uh oh!

edymtt commented Sep 8, 2022

Uh oh!

edymtt commented Sep 28, 2022

Uh oh!

edymtt commented Sep 28, 2022

Uh oh!

edymtt commented Sep 30, 2022

Uh oh!

edymtt commented Sep 30, 2022

Uh oh!

edymtt commented Sep 30, 2022

Uh oh!

edymtt commented Oct 3, 2022

Uh oh!

edymtt commented Oct 3, 2022

Uh oh!

edymtt commented Oct 3, 2022

Uh oh!

edymtt commented Oct 3, 2022

Uh oh!

edymtt commented Oct 3, 2022

Uh oh!

edymtt commented Oct 3, 2022

Uh oh!

edymtt commented Oct 4, 2022

Uh oh!

edymtt commented Oct 4, 2022

Uh oh!

edymtt commented Oct 4, 2022

Uh oh!

edymtt commented Oct 4, 2022

Uh oh!

edymtt commented Oct 4, 2022

Uh oh!

edymtt commented Oct 4, 2022

Uh oh!

edymtt commented Oct 4, 2022

Uh oh!

edymtt commented Oct 4, 2022

Uh oh!

edymtt commented Oct 4, 2022

Uh oh!

yln Oct 28, 2022

Choose a reason for hiding this comment

Uh oh!

edymtt Oct 28, 2022

Choose a reason for hiding this comment

Uh oh!

edymtt commented Jan 31, 2024

Uh oh!

Uh oh!

edymtt commented Sep 8, 2022 •

edited

Loading